DL
AI QA - Turing
Diverse Lynx India
Location not stated
2 months ago
- Test Automation
- AI
- Python
- pytest
- Unit Testing
- Integration Testing
- REST API
- RAG
- Playwright
- OpenTelemetry
- CI/CD
- LangSmith
- MLflow
- LangGraph
- CrewAI
- ADK
- AutoGen
- Pinecone
- Milvus
- Weaviate
- pgvector
- Vertex AI
- Vector Search
- OpenAI
- Gemini
- Azure OpenAI
- Load Testing
2 months ago
Required Skills
- 7+ years of experience in Software QA, Test Automation, or AI Quality Engineering.
- Strong Python programming skills.
- Hands-on experience with Pytest for:
-
- Unit testing
- Integration testing
- API testing
- Regression testing
- Experience testing REST APIs and backend services.
- Experience evaluating LLM-powered applications.
- Hands-on experience evaluating Retrieval-Augmented Generation (RAG) systems.
- Understanding of AI agent evaluation methodologies.
- Experience measuring AI quality using metrics such as:
-
- Hallucination Rate
- Tool Selection Accuracy
- Execution Accuracy
- Precision / Recall
- Latency (P50/P95/P99)
- Token Usage
- Experience with Playwright or similar frontend automation frameworks.
- Experience with OpenTelemetry, tracing, or observability platforms.
- Strong debugging and root cause analysis skills.
- Experience integrating automated tests into CI/CD pipelines.
Nice to Have
- Experience with LangSmith, Langfuse, MLflow, Arize Phoenix, or similar AI observability platforms.
- Experience evaluating multi-agent systems and orchestration frameworks such as LangGraph, CrewAI, Google ADK or AutoGen.
- Experience with vector databases such as Pinecone, Milvus, Weaviate, pgvector, or Vertex AI Vector Search.
- Exposure to OpenAI, Anthropic, Gemini, or Azure OpenAI.
- Experience with performance and load testing tools.
- Prior experience in Banking, Financial Services, or Insurance (BFSI).
AI QA - Turing · Diverse Lynx India