A
AI Engineer – Agentic Systems & RLAIF
Apolis
- 🇺🇸 United States
- On-site
- 17 hours ago
- $60 – $65 / hour
- AI
- LangGraph
- LangChain
- calibration
- GCP
- Vertex AI
- GKE
- Cloud Run
- BigQuery
- Pub/Sub
- CI/CD
- IaC
- Incident Response
- LangSmith
- OpenTelemetry
- Python
- RLHF
- IAM
- MLOps
17 hours ago
Work Site: Dallas, TX (Onsite)
**AI Engineer – Agentic Systems & RLAIF**
## **About the Role**
We are building an RLAIF (Reinforcement Learning from AI Feedback) Plane: a governed learning system that turns production evidence from Client's customer-facing agents into better agent decisions—safely, measurably, and reversibly. Today, small errors compound across interconnected agents, missing capabilities and stale business context silently produce fluent-but-wrong answers, and prompt, policy, and model changes lack one governed path to independent proof.
As an AI Engineer you will own the engineering behind that path: detecting capability and context failures before they become customer-visible, converting root-cause analysis into one-factor variants, running them through hard gates and calibrated AI judges, and promoting only what is approved, versioned, monitored, and reversible.
## **What You Will Do**
• **Build the agent runtime.** Design, build, and operate production agents and multi-agent workflows using LangGraph, LangChain, and deep-agent patterns—governed, monitored, and versioned from day one.
• **Instrument for observability.** Implement tracing, telemetry, and failure taxonomies across agents, tools, memory, and knowledge sources so every failure has a reproducible trail back to its root cause.
• **Engineer the evaluation harness.** Own golden sets, offline and online evals, and AI-judge pipelines with order-swap bias control and human calibration loops; report against business outcomes, not just model metrics.
• **Drive harness optimization.** Use observed failures to improve prompts, tools, skills, and rules through one-change-at-a-time experiments with regression-safe release: hard gates, approval, canary, and rollback.
• **Separate the optimization paths.** Keep harness, contextual-policy, and model-level changes distinct and independently evaluated; build the mechanisms that detect capability and knowledge-coverage gaps and propose governed updates.
• **Ship on GCP.** Deploy and scale the RLAIF Plane on Google Cloud (Vertex AI, GKE/Cloud Run, BigQuery, Pub/Sub) with CI/CD, IaC, and cost/latency controls.
• **Collaborate and lead.** Partner with architects, data scientists, SRE, and Client business stakeholders; document decisions, mentor engineers, and raise the bar on agent engineering practice.
## **Key Responsibilities**
- Ingesting and analyzing traces
- Reconstructing the sequence of events, tool calls, and business flow from traces
- Creating prompt/context/routing variants
- Defining and implementing evaluation dimensions
- Building and calibrating AI judges
- Designing scoring and reward-signal frameworks
- Generating autonomous optimization recommendations
- Identifying root causes such as:
- Hallucinations
- Context drift
- Poor prompts
- Misrouting
- Policy/compliance issue
- Safety: Harmlessness, different biases, toxicity
- Missing or underperforming tools
- Taxonomy/ontology gaps
- Re-configuration opportunities (agent) for identification and recommendation and project outcomes.
## **What You Bring (Required)**
• Demonstrable experience building and running AI agents in production—shipped systems serving real users, with ownership of reliability, latency, cost, and incident response.
• Hands-on experience with AI observability and evals: tracing frameworks (e.g., LangSmith, Arize Phoenix, OpenTelemetry), LLM-as-judge design, golden-set curation, and regression testing.
• Deep, practical experience with LangGraph and LangChain, including deep-agent patterns (planning, sub-agents, tool use, memory, human-in-the-loop, and state persistence).
• Strong Python engineering skills; comfortable with async services, APIs, containers, and testing discipline in a CI/CD environment.
• Working understanding of RLAIF/RLHF concepts, prompt and policy optimization, and how to isolate a single causal change in a complex system.
• 5+ years of software engineering experience, including 2+ years focused on LLM or agentic applications.
## **Preferred Qualifications**
• Cloud experience on Google Cloud (Vertex AI, GKE, Cloud Run, BigQuery, Pub/Sub, IAM); GCP certification is a plus.
• Experience with retrieval, enterprise knowledge bases, and memory systems with provenance, trust scoring, and decay.
• Familiarity with governed release practices in regulated or large-enterprise settings (approval workflows, canary, rollback, audit trails).
• Telecom or large-scale customer-service domain exposure; experience with MLOps tooling and feature flags.
AI Engineer – Agentic Systems & RLAIF · Apolis