Senior Observability Engineer (New Relic Consolidation)

We are seeking aSenior Observability Engineer to own the multi-tenant New Relic consolidation layer aggregating telemetry across our agent estate, as part of an Enterprise Agent Development Platform enabling engineering teams to define, orchestrate, deploy, and observe AI agents at scale across the organization.

Responsibilities

• Own and administer the multi-tenant New Relic consolidation layer across the agent estate
• Aggregate telemetry from multiple project tenants using cross-account NRQL queries
• Build and maintain dashboards-as-code and alert-policy-as-code using Terraform and Crossplane
• Configure New Relic alert policies routed to Opsgenie, with future migration to ServiceNow
• Manage OpenTelemetry / ADOT ingestion into New Relic via OTLP and Firehose pipelines
• Design metric dimensions and low-cardinality facets for large-scale agent estates
• Establish cross-account and cross-tenant telemetry aggregation and reporting processes
• Maintain segregation across Technical, Business, and Audit observability models
• Collaborate with engineering teams to standardize agent observability practices across the platform

Requirements

• 5+ years of experience in observability or SRE engineering
• Hands-on experience consolidating observability/telemetry for AI agent estates in a production agentic AI project, such as multi-tenant New Relic aggregation, cost/token attribution for GenAI workloads, or AgentCore GenAI Observability integration
• Expertise in New Relic administration at scale, including management account setup and cross-account configuration
• Proficiency in NRQL for cross-account queries aggregating telemetry from multiple project tenants
• Skills in dashboards-as-code and alert-policy-as-code using Terraform New Relic provider and Crossplane
• Knowledge of New Relic alert policies routed to Opsgenie, with familiarity with ServiceNow
• Understanding of OpenTelemetry / ADOT consumption fundamentals, including OTLP and Firehose ingestion into New Relic
• Competency in metric dimension design for large agent estates, ensuring stable, low-cardinality facets
• Background in operating New Relic or equivalent SaaS observability platforms as a multi-tenant consolidation layer
• Experience working within a Technical / Business / Audit observability segregation model

Nice to have

• Familiarity with AWS Bedrock AgentCore observability, including Transaction Search and GenAI Observability
• Knowledge of Kinesis Firehose metric and log forwarding pipelines from CloudWatch to New Relic
• Skills in cost/token attribution modeling for GenAI workloads
• Understanding of Crossplane-based provisioning of observability resources