Back to SearchSenior AI QA Engineer with Python (Manual & Automation)
Remote in Colombia, Mexico
Testing of AI-based applications& 6 others
Looking for something else?
Find a vacancy that works for you. Send us your CV to receive a personalized offer.
Find me a job
Choose an option
We are seeking a Senior AI QA Engineer to validate the performance of AI-driven video analysis systems, focused on detecting key moments in sports content. The role combines manual and automated testing, working with pre-labeled video assets provided by the customer.
Responsibilities
- Audit live sports games (NBA, MLB, NFL, NHL) to ensure the AI/Inference Service correctly tracks and labels major sports moments such as touchdowns, home runs, and buzzer-beaters as they happen
- Work with timecodes, video frames, transcriptions, and captions, applying sports-related contexts, lingo, and metrics
- Act as the human expert who catches when the AI hallucinates, misinterprets a sports rule, or produces errors
- Collaborate directly with AWS engineers to detail bugs and validate resolutions
- Review AI-generated labels and metadata tags to ensure they make sense for the sport and meet advertising industry standards (IAB rules), keeping content brand-safe for ad placement
- Compare the inference system's outputs against customer-provided labels to determine accuracy and identify missed or incorrectly detected events
- Execute both manual test cases (for nuanced or edge cases) and automated test cases (for validating outputs at scale and ensuring consistency)
- Log discrepancies, defects, and gaps between expected and actual results, and collaborate with the inference/development team to triage and resolve issues
- Maintain clear QC documentation, track accuracy metrics, and help build a smooth testing process that bridges traditional sports broadcasting with new AI technology
- Serve as a communication bridge between the customer and technical teams, clarifying results and expectations
- Repeat validation iteratively as new labeled assets or model updates are provided to ensure ongoing accuracy and reliability
Requirements
- 3+ years of experience in both manual and automation QA, preferably in video-focused or AI-driven environments
- Knowledge of testing LLMs and understanding of their workflow
- Proficiency in automation scripting with Python, Selenium, or similar tools for comparing JSON outputs to ground truth at scale
- Understanding of software development and QA cycles, including defect logging, triage, and reporting
- Ability to interpret labeled data and validate model outputs
- Familiarity with concepts like computer vision and transcript analysis (no need to understand internal model workings)
- Strong communication skills for stakeholder interaction and reporting
- Detail-oriented and iterative approach to testing
- Proficiency in English at an Upper-Intermediate level (B2) or higher
Nice to have
- Experience with video testing tools or frameworks
- Prior work in sports analytics or media technology