QA automation engineer

Role Overview

You will be theQuality Intelligence Architect of Exterview.

As aQA Automation Engineer (10+ Years), you’ll design, implement, and own theautomation and quality orchestration system that validates AI-driven candidate screening, live interviews, scoring agents, and feedback pipelines.

This isquality engineering at scale: validating probabilistic AI outputs, real-time interview flows, event-driven systems, and media-heavy workflows across thousands of concurrent interviews.

You’ll work closely withBackend, AI Engineering, Prompt Engineering, and Frontend teams to ensure quality isbuilt into the system, not tested after the fact.

Execution is tracked viaLinear (tasks), Notion (PRDs & test strategy), and GitHub (automation reviews) ensuring quality decisions are transparent and measurable.

Key Responsibilities

Automation Architecture

• Design and own ascalable, maintainable automation framework for:

  • Web (Next.js)

  • APIs (GraphQL via AppSync, REST)

  • Event-driven and async workflows

• Move QA from test execution toquality architecture ownership.

AI & Probabilistic System Validation

• Validate:

  • Resume parsing accuracy

  • AI scoring consistency

  • Interview logic stability

• Defineassertion strategies for non-deterministic AI outputs.

Real-Time & Media Workflow Testing

• Testlive interview systems:

  • VideoSDK + Tavus video flows

  • Real-time state updates

• Validate video upload, playback, retries, and failure handling.

API, Event & Integration Testing

• BuildAPI-first automation for:

  • GraphQL queries & mutations

  • Lambda & microservice APIs

• Validateevent-driven flows (notifications, interview state transitions).

Performance, Load & Reliability Testing

• Designload and stress tests for:

  • Interview orchestration APIs

  • Video playback & report generation

• Ensure system reliability under peak concurrency.

CI/CD & Quality Gates

• Integrate automation intoGitHub Actions.

• Definerelease quality gates and production sign-off criteria.

• Track flaky tests, failure patterns, and quality metrics.

Cross-Team Quality Leadership

• Translate product requirements intotest strategies.

• Partner with engineering to improvetestability & observability.

• Mentor junior and mid-level QA engineers.

Success Metrics

First 90 Days

• Stable automation framework covering UI + API + AI workflows.

• Flaky test rate reduced to near zero.

• API-level load tests live for interview & video flows.

• Release quality gates enforced in CI/CD.

12 Months

• End-to-end automation coverage for all interview types.

• Predictable, fast release cycles withno QA bottlenecks.

• AI validation framework adopted as product standard.

• QA recognized asproduct quality owner, not gatekeeper.

Must-Haves

• 10+ years inQA Automation / SDET roles

• Strong experience designingautomation frameworks, not just writing scripts

• Expertise inAPI testing (GraphQL & REST)

• Experience testingreal-time and async systems

• Ability to validateAI-driven, non-deterministic outputs

• Strong understanding ofCI/CD, release quality, and production readiness

Nice-to-Haves

• Experience testingAI/ML or agent-based systems

• Media or video workflow testing experience

• Startup or high-scale SaaS background

• Prior ownership of QA strategy for a product

Tech Stack Visibility

• Frontend: Next.js, TypeScript

• Automation: Cypress, Playwright, Jest

• Backend: Node.js, AWS Lambda, Serverless Framework

• APIs: GraphQL (AWS AppSync), REST

• Auth: AWS Cognito (OAuth, OTP)

• Data: MongoDB Atlas

• Media: VideoSDK, Tavus, Amazon S3

• Communication: Twilio (SMS, WhatsApp, IVR)

• Observability: Langfuse, internal monitoring

Assessment (PoC)

Objective

Validate your ability todesign quality systems, not just write test cases.

Challenge Overview

Design and implement anautomation strategy for an AI-driven interview flow.

The system should validate:

• Resume upload & parsing

• AI resume–job match scoring

• Interview scheduling

• Live interview execution

• Interview report generation

Functional Expectations

1. Automation Design

• Cover:

  • One UI flow

  • One GraphQL API flow

  • One real-time or async workflow

• Show clear separation between test layers.

2. AI Validation Strategy

• Define how you:

  • Assert AI scores

  • Detect drift or instability

  • Handle acceptable variance

3. Performance & Reliability

• Include:

  • One API load test

  • Failure / retry validation

• Avoid direct S3 URL testing; use APIs only.

Deliverables

• Automation repo (clean structure)

• Test strategy README

• Example AI assertion logic

• CI pipeline config (GitHub Actions)

• ≤ 5 min Loom walkthrough explaining decisions