Infrastructure Engineer — Managed Inference

About the Role

This role sits at the center of a high-stakes inference platform, responsible for designing, operating, and scaling the production serving layer that delivers large language model outputs to real customers at global scale. The work spans multi-cloud Kubernetes operations, GPU systems debugging, observability infrastructure, and reliability engineering, all in service of keeping latency-sensitive AI workloads fast and available around the clock.

The right person is already operating at staff scope in an IC infrastructure role and brings direct, day-to-day hands-on experience running LLM inference systems in production. You know Kubernetes at scale across AWS and GCP, you can debug NVIDIA GPU systems from driver to network fabric, and you have real familiarity with at least one major inference serving framework. Experience with heterogeneous accelerator fleets or open-source inference contributions is a genuine plus, not just a checkbox.

Key Responsibilities

1. Design, operate, and scale highly available Kubernetes clusters across AWS, GCP, and specialized GPU cloud providers to serve global inference demand at 1,000+ tokens per second
2. Own the production serving layer end-to-end, debugging NVIDIA systems including drivers, CUDA, NCCL, node health, and network fabric
3. Partner closely with the model optimization team to ensure kernel-level speed improvements hold up against real traffic, real hardware, and real customers
4. Extend deployment tooling so models ship consistently across NVIDIA, Trainium, and TPU hosts through a unified workflow
5. Build and maintain observability infrastructure tracking time-to-first-token, inter-token latency, throughput, and availability per model, per chip, and per region
6. Own reliability engineering including alerting, automated failover, self-healing infrastructure, and intelligent traffic routing across models, chips, and regions

Required Skills & Experience

• 5+ years of infrastructure engineering experience, currently operating at staff level or above in an IC role
• Direct, hands-on inference infrastructure experience as a core part of daily work (not inference-adjacent data engineering or general-purpose infra)
• Production Kubernetes at scale across multiple cloud providers, specifically AWS and GCP
• GPU and accelerator systems expertise: NVIDIA drivers, CUDA, NCCL debugging, and node health
• Hands-on experience with at least one inference serving framework: vLLM, SGLang, TensorRT-LLM, or NVIDIA Dynamo
• HPC networking experience: InfiniBand, RDMA/EFA, or MPI
• Observability and reliability engineering for latency-sensitive services, including metrics pipelines, alerting, and automated failover

Preferred Skills

• Background at a production-scale inference platform such as Fireworks, BaseTen, Nebius, or Anyscale
• Experience deploying across heterogeneous accelerator fleets including AWS Trainium, Google TPUs, or mixed GPU configurations
• Contributions to open-source inference frameworks such as vLLM or SGLang
• Familiarity with PyTorch, JAX, Docker, Terraform, Prometheus, or Grafana