Principal Software Engineer

At Cadence, we hire and develop leaders and innovators who want to make an impact on the world of technology.

Senior AI Test / Automation Engineer

Overview

Role: AI Test / Automation Engineer
Location: Bangalore India

Department: AI Engineering / Quality Assurance
Experience Level: Mid to Senior

We are looking for a highly motivatedSeniorAI Test / Automation Engineer to design and scale automated validation frameworks forAI/ML models, LLM-based applications, and agentic systems. This role is critical to ensure that AI solutions meet enterprise standards forquality, reliability, safety, and compliance before and after production deployment.

Key Responsibilities

• Build and maintain AI test automation frameworks for pre-qualification and continuous validation of models and agent workflows
• Develop comprehensive test suites, including:
  • Unit, integration, and end-to-end (E2E)
  • Functional, regression, performance, and safety testing
• Validate AI system behavior, including:
  • Non-deterministic LLM outputs
  • Hallucinations and edge cases
  • Multi-step agent decision-making
• Design and manage evaluation systems:
  • Golden datasets
  • Benchmarking pipelines (accuracy, latency, reliability)
• Automate testing within CI/CD pipelines for model updates, prompt changes, and tool integrations
• Implement observability and telemetry to enable traceability, monitoring, and audit readiness
• Collaborate cross-functionally with ML, MLOps, Product, and Security teams to define quality gates and release criteria
• Track and report quality KPIs, including test coverage, defect leakage, and system reliability
• Drive root-cause analysis and continuous improvement across the AI testing lifecycle

Required Skills

Core Engineering

• Strong programming skills inPython; familiarity withBash, TypeScript, or Go
• Experience withtest automation frameworks such as PyTest, Playwright, Selenium, or Cypress
• Proficiency inCI/CD tools (GitHub Actions, Jenkins, GitLab CI)
• Experience withcloud platforms (AWS, Azure, GCP) andcontainers (Docker, Kubernetes)

AI / ML & Agentic Systems

• Hands-on experience withLLM ecosystems (OpenAI, Anthropic, Bedrock)
• Familiarity with:
  • RAG architectures and vector databases (Pinecone, Weaviate)
  • Agent frameworks (LangChain, LlamaIndex, AutoGen)

AI Testing Techniques

• Experience withnon-deterministic testing approaches (statistical assertions, tolerance thresholds)
• Knowledge ofevaluation methods:
  • LLM-as-a-judge
  • BLEU, ROUGE, semantic similarity scoring
• Experience withprompt and agent regression testing
• Understanding ofAI safety testing, including adversarial testing, bias/fairness validation, and jailbreak detection

Tooling (Preferred)

• AI testing & observability tools:LangSmith, TruLens, Arize, Weights & Biases
• Evaluation tools:DeepEval, Ragas, PromptFoo, Giskard
• Monitoring:Prometheus, Grafana, OpenTelemetry

Soft Skills

• Strong analytical and problem-solving skills
• Excellent communication and cross-functional collaboration
• Data-driven mindset with focus onquality KPIs
• Detail-oriented with a strong bias towardautomation and scalability

Experience Requirements

• 7+ years in QA, SDET, or test automation engineering
• Proven experience building and scalingautomation frameworks
• Hands-on experience withAI/ML systems or LLM-based applications

• Experience testingRAG pipelines or agentic workflows
• Owned end-to-endAI test strategy and architecture
• Definedquality metrics and release gates
• Deliveredscalable validation pipelines for production AI systems
• Supportedaudit and compliance readiness

Preferred

• Experience inenterprise or regulated environments (SOC2, ISO 27001, etc.)
• Exposure to:
  • Shift-left testing practices
  • Production observability and monitoring
  • Chaos or resilience testing

Senior-Level Differentiators

Education

• Bachelor’s or Master’s degree inComputer Science, Software Engineering, or related field

Nice-to-have:

• ISTQB certification
• Cloud/ML certifications (AWS, Azure, GCP)
• AI testing certifications

What Success Looks Like

• AI systems that areaccurate, reliable, and safe
• Fully automated test pipelines integrated into CI/CD
• Measurable improvements indefect leakage and model quality
• Strong observability and auditability across AI systems
• Scalable validation frameworks supporting rapid AI innovation

We’re doing work that matters. Help us solve what others can’t.