Senior GCP DevOps & LLMOps Engineer

Job Description: Senior GCP DevOps &LLMOps Engineer

Role Summary

We are looking for an experiencedSenior GCP DevOps &LLMOps Engineer to architect, deploy, andmaintain robust infrastructure supporting enterprise-scale Large Language Model (LLM) and Generative AI applications. In this role, you will bridge the gap between traditional DevOps, platform engineering, and modern machine learning operations.

You will lead the automation of Google Cloud Platform (GCP) environments usingTerraform, design resilient CI/CD pipelines, and implement production-gradeLLMOps architectures—including model serving, inference routing, vector database orchestration, evaluation pipelines, and GPU/TPU resource optimization.

Key Responsibilities

1. Infrastructure as Code (IaC) & Cloud Architecture

• Design, build, andmaintainproduction GCP environments using modular, reusableTerraform (andTerragrunt) code.

• Manage multi-environment cloud resources across VPC networking, Private Service Connect, Cloud Interconnect, IAM, KMS, and Cloud Storage.

• Enforce security best practices, least-privilege IAM policies, Workload Identity Federation, and automated compliance auditing.

2.LLMOps & GenAI Platform Engineering

• Build automated deployment, scaling, and operational infrastructure for generative AI models hosted onVertex AI, GKE (Google Kubernetes Engine), or hybrid inference servers.

• Implementor Understandingend-to-end LLM lifecycle pipelines: prompt management, fine-tuning workloads, automated retrieval-augmented generation (RAG) components, andevaluation harness pipelines.

• Deploy, manage, and scale vector databases (e.g., Vertex AI Vector Search, Pinecone,Qdrant,pgvector) with proper indexing, backup, and latency monitoring.

• Architect andoptimize model-serving engines.

3. CI/CD & Pipeline Automation

• Build and maintain scalable CI/CD pipelines usingCloud Build,GitHub Actions, orGitLab CI for both microservices and machine learning models.

• Establish automated testing for infrastructure changes (e.g.,tflint,tfsec,terratest) and model release pipelines (canary deployments, blue/green rollouts, shadow traffic).

• Package and deploy containerized AI/ML applications using Docker, Artifact Registry, and Helm charts on GKE.

4. Observability, Guardrails & Reliability

• Set up deep observability stacks using Google Cloud Monitoring,OpenTelemetry, Prometheus, and Grafana to track standard SRE metrics (SLIs/SLOs, latency, throughput).

• Implement LLM-specific telemetry: token consumption rates, inference latency, model drift, cost-per-query, and hallucinations/groundedness metrics using tools such asLangSmith, Phoenix/Arize, or Vertex AI Model Monitoring.

• Integrate safety guardrails, prompt sanitization, rate limiting, and fallback/resiliency mechanisms across inference endpoints.

5. FinOps & Compute Optimization

• Monitor andoptimize cloud spending for compute-intensive workloads, specificallyLLM quotamonitoring, spot instances, and dynamic autoscaling.

• Partner with engineering and product teams to project infrastructure costs andestablish budget s and quota guardrails.

Requirements & Qualifications

Must-Have

• Experience: 5+ years in DevOps, SRE, or Cloud Platform Engineering with at least 3+ years dedicated toGoogle Cloud Platform (GCP).

• Terraform Expertise: Advancedproficiency with Terraform (state management, reusable module design, workspaces, policy-as-code with Sentinel or OPA).

• LLMOps /MLOps Background: 1.5+ years of practical experience operationalizing LLMs or ML workloads, including familiarization withVertex AI (Model Garden, Pipelines, Endpoints) or custom model serving on Kubernetes.

• Containerization & Orchestration: Strong background in Docker andGoogle Kubernetes Engine (GKE), including cluster setup, GPU node pools, ingress controllers, and autoscaling (HPA, KEDA).

• CI/CD: Hands-on experience configuring automated pipelines withGitOps patterns (e.g.,ArgoCD, Flux) or modern CI engines (GitHub Actions, Cloud Build).

• Scripting & Languages: Solid programming/scriptingproficiency inPython and Bash/Shell.

• Security & Networking: Strong understanding of GCP VPC architecture, private endpoints, service perimeters (VPC-SC), and Secret Manager.

Nice-to-Have

• Experience deploying agentic frameworks (e.g.,LangChain,LlamaIndex,LangGraph,CrewAI) or Model Context Protocol (MCP) servers in production.

• GCP Certifications (e.g., Google Cloud Certified Professional Cloud DevOps Engineer, Professional Cloud Architect, or Professional Machine Learning Engineer).

• Familiarity with distributed model training and high-throughput inference caching strategies (Redis, Semantic Cache).

• Exposure to regulatory compliance frameworks (SOC2, HIPAA, ISO 27001) in relation to cloud AI systems.

Technical Stack Overview