AI Infra / Engineer - Large-scale Training

Your Mission

AsAI Infra / Engineer — Large-scale Training, you will build the systems that make UMA’s humanoid robot learningfast, scalable, and reliable. You will turn experimental training code into robust execution pipelines that scale across thousands of GPUs and support rapid iteration.

You will work directly with AI researchers, shaping the software used to train our robotic foundation models acrosspretraining, mid-training, and post-training on large-scale multimodal data. You will own the engineering challenges of distributed execution: choosing parallelism strategies, optimizing memory and communication, accelerating critical operations, and keeping long-running jobs reliable..

We already have a working training stack and a strong team behind it. You will build on that foundation, with room to rethink components where needed. Working closely with AI researchers, you will scalepretraining across GPUs and nodes, optimizecompute, memory, and communication, and make long-running training jobs reliable. The goal is toreduce training time and increase experiment throughput, enabling larger models and faster research iteration within our compute budget.

Debugging here goes beyond keeping jobs running. You will understand enough of the learning process to recognize when a numerical issue or distributed implementation bug appears in a loss curve, and work with researchers to resolve it. The systems you build will also support mid-training and post-training as our models evolve.

Key responsibilities:

• Builddistributed pretraining systems across multiple GPUs and nodes, selecting appropriate strategies such as DDP, FSDP, and tensor, pipeline, or expert parallelism as workloads require.

• DevelopRL and post-training execution infrastructure, including distributed rollouts, actor–learner coordination, policy synchronization, replay buffers, and efficient batching.

• Maximize MFU and eliminate bottlenecks: Profile the complete training loop—data loading, decoding, CPU-GPU memory transfers, and compute kernels—eliminating bottlenecks to keep MFU high and ensure accelerators operate at peak efficiency.

• Integrate and benchmarkaccelerated kernels and execution techniques, such as FlashAttention, fused operations, mixed precision, and compilation; develop custom optimizations where they provide clear value.

• Build efficienttraining-time data movement and buffering, including out-of-core storage, caching, prefetching, and asynchronous transfers, in partnership with the Data Infrastructure team.

• Turn research prototypes intomaintainable training systems, with reusable components that support evolving architectures and experimental workflows.

• Diagnosedistributed and numerical failures, including deadlocks, memory leaks, gradient anomalies, precision issues, and inconsistencies between single-device and distributed runs.

• Deep numerical debugging and run determinism: Diagnose complex issues across the stack—from distributed deadlocks and memory leaks to training instabilities and gradient anomalies—while enforcing numerical determinism so that training runs, evaluations, and experiments are strictly reproducible.

• Buildreliable checkpointing and fault-tolerant resumption, preserving model, optimizer, and relevant execution state so interrupted jobs can recover correctly.

• Strengthenobservability and reproducibility through configuration tracking, profiling, run diagnostics, and deterministic execution where practical.

What You Bring to the Table

• Solid experience buildingML training infrastructure, distributed systems, or high-performance ML software.

• StrongPython and PyTorch, with hands-on knowledge of distributed training tools such as DDP, FSDP, or Megatron.

• Experience withmultiprocessing, asynchronous execution, and GPU memory management, and an understanding of how communication and synchronization affect scaling.

• Practical knowledge oflosses, optimizers, gradients, and numerical precision, sufficient to debug training behavior alongside researchers.

• Experience profiling accelerator workloads with tools such asPyTorch Profiler or Nsight Systems, and translating findings into measurable improvements.

• A pragmatic engineering mindset: you can turn experimental code into reliable, scalable systems while keeping research iteration fast.

• Clear communication and a collaborative approach to working with researchers whose requirements evolve quickly.

• Bonus: RL infrastructure, distributed rollouts, replay buffers, or actor–learner architectures.

• Bonus: C++, CUDA, Triton, custom kernels, or compiler-based optimization.

• Bonus: Slurm or Kubernetes, GPU networking such as InfiniBand/RoCE, or storage caching.

• Bonus: contributions to open-source training tools or technical writing.

UMA is aninclusive workplace that values exceptional builders over perfect pedigrees. Whatever your background, identity, or journey, if you don’t meet every criterion but believe you can have an outsized impact here, we strongly encourage you to apply.