NC
Splunk Enterprise Architecture and Design
NR Consulting - India
🇮🇳 India
On-site
1 week ago
- Splunk
- AIOps
- IaC
- CI/CD
- AI
- RAG
- OpenTelemetry
- Terraform
- Ansible
- GitHub Actions
- ArgoCD
- Devops
- ServiceNow
- Jira
- Confluence
- GitOps
- OpenAI
- Anthropic Claude
- Google Gemini
- LangChain
- LlamaIndex
- CrewAI
- Pinecone
- Weaviate
- Chroma
- AWS
- GCP
- CKA
- ITIL
- Vault
- Gemini
- Python
- Incident Management
- eBPF
- Cilium
- Triton
- vLLM
- AI/ML
- Network Security
1 week ago
Project Role Description : Support the operations and/or manage delivery for production systems and services based on operational requirements and service agreement.
Must have skills : Splunk Enterprise Architecture and Design, Event management with AIOPS , Splunk Enterprise Observability & ITSI
Good to have skills : NA
Minimum 5 year(s) of experience is required
Educational Qualification : 15 years full time education
Summary:
A Tools & Platforms Site Reliability Engineer (SRE) ensures the reliability, availability, performance, and continuous improvement of the infrastructure engineering tooling estate — spanning observability platforms, infrastructure-as-code tooling, CI/CD pipelines, ITSM platforms, internal developer portals, secret management, and AI-augmented operations tooling. The role applies a software engineering discipline to platform operations — building automated remediation, establishing SLIs and SLOs for tooling platforms, reducing toil through systematic automation, and owning reliability outcomes end to end across the four tooling pillars.
At Level 7 / 8, this individual operates at the intersection of platform engineering, SRE practice, and AI operations — not just keeping platforms running but continuously raising their reliability ceiling. A distinctive aspect of this role is ownership of LLMOps reliability — ensuring AI-augmented operations tooling (runbook automation pipelines, agentic ITSM workflows, RAG knowledge bases, and AI alert correlation services) meets production-grade SLOs in a regulated financial services environment.
Observability SRE
– ELK/Splunk
– OpenTelemetry
– SLI/SLO/Error Budget
IaC & Automation SRE
– Terraform
– Ansible/Chef
– GitHub Actions/ArgoCD
– HashiCorp Vault
– Policy-as-Code
ITSM & DevOps SRE
– ServiceNow
– xmatters
– Backstage IDP
– Jira/Confluence
– CMDB Reliability
AI Ops SRE
– LLMOps Reliability
– Agentic ITSM SRE
– AI Alert Pipeline SRE
– RAG Platform SRE
– Model Observability
Roles & Responsibilities:
– Own reliability of observability platforms —Splunk— defining and maintaining SLIs, SLOs, and error budgets for metrics pipelines, alerting systems, and dashboard availability across all infrastructure tiers
– Engineer auto-remediation for common observability failures — scraper restarts, index rollover failures, ingest pipeline blockages — reducing MTTR and eliminating repetitive manual toil
– Implement and govern OpenTelemetry instrumentation standards across the infrastructure estate — ensuring telemetry coverage is comprehensive, consistent, and production-grade
– Drive observability-as-code adoption — dashboards, alert rules, SLO definitions, and recording rules version-controlled and deployed through GitOps pipelines with automated testing
– Perform capacity planning and performance analysis for observability platforms — managing cardinality growth, storage retention, query performance, and ingest throughput at scale
– Lead blameless post-mortems for observability platform failures — producing structured RCA with systemic preventive actions that address root causes rather than symptoms
AI-Augmented Operations SRE
– Own reliability of LLMOps pipelines — monitoring model API health (OpenAI, Anthropic Claude, Google Gemini), prompt execution success rates, token consumption, latency SLOs, and cost anomaly alerting for AI-augmented operations tooling
– Engineer reliability for agentic ITSM workflows — LangChain, LlamaIndex, CrewAI — including agent execution health, tool call success rates, human-in-the-loop handoff reliability, and automated failure recovery
– Build observability for RAG knowledge base platforms — vector database (Pinecone, Weaviate, ChromaDB) availability, retrieval latency SLOs, embedding pipeline health, and index freshness monitoring
– Implement AI alert correlation reliability — ensuring LLM-based alert grouping pipelines maintain accuracy and availability SLOs, with fallback to rule-based alerting during AI platform degradation
– Define and enforce LLMOps governance frameworks — prompt version control, model evaluation pipelines, output quality monitoring, and FSI compliance controls (audit logging, data residency) for AI operations tooling
– Lead blameless post-mortems for AI tooling failures — diagnosing model degradation, hallucination events, pipeline failures, and agent workflow breakdowns with preventive actions that meet FSI audit standards
Professional & Technical Skills:
Certifications
-Terraform Associate or Professional
Splunk Professional
-AWS DevOps Engineer Pro or GCP DevOps Engineer
HashiCorp Vault Associate
-Certified Kubernetes Administrator (CKA)
ITIL Foundation or Practitioner
Must-Have Technical Skills
-Observability SRE: Splunk— SLI/SLO/error budget engineering, OpenTelemetry, ELK/Splunk pipeline reliability, and observability-as-code practices
-IaC Reliability: Terraform — state backend health, drift detection automation, module registry SRE, and policy-as-code pipeline reliability across AWS and GCP
-CI/CD SRE: GitHub Actions, ArgoCD — pipeline health monitoring, runner auto-scaling, deployment success rate SLOs, and automated rollback engineering
-Vault Reliability: HA cluster monitoring, seal/unseal automation, certificate lifecycle management, and lease renewal automation for secrets infrastructure
-LLMOps Reliability: Model API health monitoring (OpenAI, Anthropic, Gemini), prompt execution SLOs, token/cost anomaly alerting, and AI pipeline auto-remediation
- RAG Platform SRE: Vector database availability (Pinecone, Weaviate, ChromaDB), retrieval latency SLOs, embedding pipeline health, and index freshness monitoring
-Automation & Toil Reduction: Python — SRE automation scripting, event-driven remediation, infrastructure SDK integration (boto3, GCP client), and operational workflow engineering
-Incident Management: P1/P2 bridge leadership, blameless post-mortems, structured RCA, error budget reviews, and SLA-governed resolution in FSI environments
-Performance & Capacity: Platform capacity trending, SLO burn rate alerting, cardinality management, and proactive capacity interventions across observability and AI tooling
Preferred / Advantageous
– Experience with chaos engineering or game day exercises for platform tooling resilience — validating failure modes in observability, CI/CD, or AI pipeline infrastructure
– Familiarity with eBPF-based observability (Cilium, Pixie) for deep platform telemetry and service mesh reliability engineering
– Exposure to model serving infrastructure — Triton, vLLM, or similar — for AI/ML platform reliability beyond API-based LLM tooling
– Background in SRE or platform engineering within financial services or other highly regulated industries
Additional Information:
– SLOs for all platform pillars — observability, IaC, CI/CD, ITSM, and AI tooling — are consistently met, with error budgets actively managed and reliability improving measurably quarter-on-quarter
– Toil across the tooling estate decreases consistently — manual intervention patterns are replaced by automated, observable workflows and the team's time shifts toward reliability engineering rather than repetitive operations
– Major platform incidents are managed with clear ownership, rapid mobilisation, blameless RCA outputs, and systemic fixes that prevent recurrence
– LLMOps and AI-augmented operations tooling meets production SLOs — model API failures, agent workflow breakdowns, and RAG pipeline degradation are detected early, remediated automatically where possible, and escalated with full context when not
– Engineering teams across Cloud, Network, Security, Database, and Voice towers rely on platform tooling that is observable, self-healing, and consistently available — the Tools & Platforms SRE is the reason it stays that way
- The candidate should have minimum 5 years of experience in Splunk Enterprise Architecture and Design.
- A 15 years full time education is required.
Splunk Enterprise Architecture and Design · NR Consulting - India