Senior Observability Engineer (New Relic Consolidation)

We are seeking an experiencedSenior Observability Engineer to take ownership of the multi-tenant New Relic consolidation layer that aggregates telemetry across our agent estate, as part of an Enterprise Agent Development Platform that empowers engineering teams to define, orchestrate, deploy, and observe AI agents at scale throughout the organization.

Responsibilities

• Operate and administer the multi-tenant New Relic consolidation layer across the agent estate
• Consolidate telemetry from multiple project tenants through cross-account NRQL queries
• Develop and maintain dashboards-as-code and alert-policy-as-code via Terraform and Crossplane
• Set up New Relic alert policies routed to Opsgenie, with future migration to ServiceNow
• Handle OpenTelemetry / ADOT ingestion into New Relic through OTLP and Firehose pipelines
• Architect metric dimensions and low-cardinality facets for large-scale agent estates
• Define cross-account and cross-tenant telemetry aggregation and reporting processes
• Preserve segregation across Technical, Business, and Audit observability models
• Partner with engineering teams to standardize agent observability practices across the platform

Requirements

• 5+ years of experience in observability or SRE engineering
• Hands-on background in consolidating observability/telemetry for AI agent estates in a production agentic AI project, such as multi-tenant New Relic aggregation, cost/token attribution for GenAI workloads, or AgentCore GenAI Observability integration
• Expertise in New Relic administration at scale, covering management account setup and cross-account configuration
• Proficiency in NRQL for cross-account queries that aggregate telemetry from multiple project tenants
• Skills in dashboards-as-code and alert-policy-as-code with Terraform New Relic provider and Crossplane
• Knowledge of New Relic alert policies routed to Opsgenie, along with familiarity with ServiceNow
• Understanding of OpenTelemetry / ADOT consumption fundamentals, covering OTLP and Firehose ingestion into New Relic
• Competency in metric dimension design for large agent estates, ensuring stable, low-cardinality facets
• Background in running New Relic or equivalent SaaS observability platforms as a multi-tenant consolidation layer
• Experience operating within a Technical / Business / Audit observability segregation model
• English proficiency at an Upper-Intermediate level (B2) or higher

Nice to have

• Familiarity with AWS Bedrock AgentCore observability, covering Transaction Search and GenAI Observability
• Understanding of Kinesis Firehose metric and log forwarding pipelines from CloudWatch to New Relic
• Expertise in cost/token attribution modeling for GenAI workloads
• Knowledge of Crossplane-based provisioning of observability resources