Senior AI QA Engineer with Python (Manual & Automation)

We are seeking aSenior AI QA Engineer to validate the performance of AI-driven video analysis systems, focused on detecting key moments in sports content. The role combines manual and automated testing, working with pre-labeled video assets provided by the customer.

Responsibilities

• Audit live sports games (NBA, MLB, NFL, NHL) to ensure the AI/Inference Service correctly tracks and labels major sports moments such as touchdowns, home runs, and buzzer-beaters as they happen
• Work with timecodes, video frames, transcriptions, and captions, applying sports-related contexts, lingo, and metrics
• Act as the human expert who catches when the AI hallucinates, misinterprets a sports rule, or produces errors
• Collaborate directly with AWS engineers to detail bugs and validate resolutions
• Review AI-generated labels and metadata tags to ensure they make sense for the sport and meet advertising industry standards (IAB rules), keeping content brand-safe for ad placement
• Compare the inference system's outputs against customer-provided labels to determine accuracy and identify missed or incorrectly detected events
• Execute both manual test cases (for nuanced or edge cases) and automated test cases (for validating outputs at scale and ensuring consistency)
• Log discrepancies, defects, and gaps between expected and actual results, and collaborate with the inference/development team to triage and resolve issues
• Maintain clear QC documentation, track accuracy metrics, and help build a smooth testing process that bridges traditional sports broadcasting with new AI technology
• Serve as a communication bridge between the customer and technical teams, clarifying results and expectations
• Repeat validation iteratively as new labeled assets or model updates are provided to ensure ongoing accuracy and reliability

Requirements

• 3+ years of experience in both manual and automation QA, preferably in video-focused or AI-driven environments
• Knowledge of testing LLMs and understanding of their workflow
• Proficiency in automation scripting with Python, Selenium, or similar tools for comparing JSON outputs to ground truth at scale
• Understanding of software development and QA cycles, including defect logging, triage, and reporting
• Ability to interpret labeled data and validate model outputs
• Familiarity with concepts like computer vision and transcript analysis (no need to understand internal model workings)
• Strong communication skills for stakeholder interaction and reporting
• Detail-oriented and iterative approach to testing
• Proficiency in English at an Upper-Intermediate level (B2) or higher

Nice to Have

• Experience with video testing tools or frameworks
• Prior work in sports analytics or media technology