AI DevOps Engineer

Company Summary

Position Responsibilities

About the Role 

We are seeking a highly motivated AI DevOps Engineer to join Deltek’s growing AI Center of Excellence.The ideal candidate combines deep expertise in LLMOps, MLOps, and DevOps with hands-on experience in Generative AI, Large Language Models (LLMs), SLMs, and cloud computing. 

The AI DevOps Engineer will partner with AI data scientists, architects, AI solutions engineers, and platform engineering teams to deliver production-grade AI services while ensuring security, scalability, governance, observability, and operational excellence. The role also contributes to AI solution development, so that operational practices are shaped by real delivery experience. This role reports to the Senior AI Solutions Architect. 

Key Responsibilities 

LLMOps & Generative AI DevOps 

• Design and operate LLMOps pipelines for prompt management, evaluation, versioning, release, and rollback of LLM-based applications. 

• Run and scale LLM gateways and model-serving layers across providers such as GPT, Claude, Gemini, Llama, and Mistral, including routing, rate limiting, caching, and failover. 

• Operate Retrieval-Augmented Generation (RAG) platforms, including vector database management, embedding refresh, and index lifecycle. 

• Build automated evaluation frameworks to measure quality, groundedness, hallucination rates, safety, latency, and cost per request. 

• Support deployment and monitoring of AI agents and agentic workflows, including MCP and A2A integrations. 

• Track token usage and spend, and optimize model selection, context size, and inference configuration for cost and performance. 

MLOps & Model Lifecycle Management 

• Design and implement end-to-end MLOps pipelines for model training, validation, registration, deployment, monitoring, and retirement. 

• Automate model training, testing, and promotion across development, staging, and production environments. 

• Manage experiment tracking, model registries, feature pipelines, and reproducible environments. 

• Monitor model performance, data drift, concept drift, and operational metrics, and trigger retraining or remediation. 

DevOps, Cloud & Platform Engineering 

• Build and maintain CI/CD pipelines for AI applications, models, and infrastructure using Git-based workflows. 

• Provision and manage cloud infrastructure on AWS (preferred), Azure, or Google Cloud using Infrastructure as Code such as Terraform, CloudFormation, or Bicep. 

• Containerize and orchestrate AI workloads with Docker and Kubernetes, including GPU scheduling, autoscaling, and serverless options. 

• Implement observability for AI platforms: metrics, logs, traces, dashboards, alerting, and on-call runbooks. 

• Drive reliability practices, including incident response, root cause analysis, capacity planning, and disaster recovery. 

• Manage AI platform cost through tagging, budgets, right-sizing, and FinOps reporting. 

AI Solutions Engineering 

• Collaborate with data scientists and solution teams to productionize machine learning, Generative AI, and RAG solutions. 

• Develop Python-based APIs, microservices, and integrations that connect AI services with enterprise systems and data platforms. 

• Contribute to prompt engineering, fine-tuning, and model customization efforts where operational input improves outcomes. 

• Provide reusable templates, SDKs, and reference architectures that help teams adopt AI platform standards quickly. 

AI Governance, Security & Compliance 

• Ensure compliance with Responsible AI, security, privacy, and regulatory requirements across the AI lifecycle. 

• Implement secure design and operation: secrets management, identity and access controls, network isolation, and data protection. 

• Put guardrails in place, such as content filtering, prompt-injection defenses, PII handling, and audit logging. 

Qualifications

Required Qualifications 

Education 

• Bachelor’s or Master’s degree in Computer Science, Artificial Intelligence, Data Science, Engineering, or a related technical field. 

Experience 

• 5+ years of experience in DevOps, platform engineering, software engineering, or machine learning engineering. 

• 2+ years of hands-on experience building and operating MLOps or LLMOps pipelines and platforms in production environments. 

• 2+ years of experience supporting Generative AI, Agentic AI, or AI/ML solutions in production. 

Technical Skills 

MLOps & LLMOps 

• MLflow, Kubeflow, Azure ML, SageMaker, or similar platforms for tracking, registry, and pipelines. 

• LLM evaluation, prompt versioning, and observability tooling, such as LangSmith, Langfuse, or similar. 

• Model serving and inference optimization, such as vLLM, KServe, Triton, or managed endpoints. 

• Monitoring of model quality, data drift, hallucinations, latency, and token cost. 

DevOps & Platform Engineering 

• Docker and Kubernetes (EKS, AKS, or GKE), including Helm and GitOps tools such as Argo CD or Flux. 

• CI/CD automation with GitHub Actions, Azure DevOps, or Jenkins. 

• Infrastructure as Code: Terraform, CloudFormation, or Bicep. 

• Observability stack: Prometheus, Grafana, OpenTelemetry, CloudWatch, Datadog, or similar. 

• Scripting and automation in Python and Bash. 

Programming & Engineering 

• Strong expertise in Python; working knowledge of Java, JavaScript, or similar languages. 

• Experience building APIs and microservices with FastAPI or Flask. 

• Solid understanding of algorithms, data structures, APIs, and software design principles. 

Artificial Intelligence & Generative AI 

• Machine Learning and Deep Learning concepts and frameworks, including PyTorch, TensorFlow, and Scikit-learn. 

• Large Language Models (LLMs) and SLMs, prompt engineering, and fine-tuning concepts. 

• Retrieval-Augmented Generation (RAG), vector embeddings, and semantic search. 

• AI agents and agentic workflows using LangChain, LlamaIndex, Semantic Kernel, MCP, and A2A. 

Data & Cloud Platforms 

• SQL and NoSQL databases; data pipelines, ETL, and data modeling. 

• AWS (preferred), including Bedrock and SageMaker; Azure OpenAI Service; Google Vertex AI. 

• Cloud security fundamentals: IAM, networking, encryption, and secrets management. 

Applicant Privacy Notice

• Career Interests: Engineering
• Position Type: FT
• Travel Requirements: 20%