Lead AI Testing Engineer

Remote in Poland

Testing of AI-based applications

Facebook LinkedIn Send via email

Looking for something else?

Find a vacancy that works for you. Send us your CV to receive a personalized offer.

Find me a job

We are looking for a Lead AI Testing Engineer to drive the quality strategy for AI-driven and data migration initiatives spanning knowledge graphs, ontologies and RAG-based LLM features. You will define and execute QA practices across data ingestion, graph layers, rule evaluation engines and AI/LLM components while building automation frameworks that scale evaluation across complex, data-heavy systems.

Responsibilities

Design and automate evaluation of RAG-based and LLM-driven features including grounding, answer accuracy, determinism/reproducibility, precision, recall and hallucination rate
Build test harnesses to scale evaluation beyond human-in-the-loop processes
Create data-driven test suites for underwriting rules and validate rule execution via SPARQL evaluator, arithmetic/threshold logic and multi-condition scenarios
Verify ontology schema correctness and instance accuracy against source data, perform reasoner consistency checks and SHACL validation
Define and drive end-to-end QA strategy covering data ingestion, ontology/graph layer, rule evaluation engine, AI/LLM layer and integration with underwriting solutions
Establish quality gates, acceptance criteria and test coverage models
Maintain automation test suites across knowledge graph, data ontology layer, backend services and frontend layers
Validate end-to-end flows across ontology, rule engine and decision output, perform contract testing and cross-system data consistency validation
Integrate test suites into CI/CD pipelines and define release quality gates
Champion automation-first and shift-left quality practices
Leverage agentic AI and Gen AI tooling in testing and framework development

Requirements

8+ years of experience in software development, testing automation and DevOps
Proficiency in Python test automation using pytest or equivalent, scripting and AI/ML library integration
Expertise in testing Knowledge Graph or Data Ontology solutions using OWL2, RDF and SPARQL
Skills in SHACL validation and graph databases such as GraphDB, including ontology validation, entity/relationship integrity, reasoner consistency checks and graph query correctness
Proven ability to define and drive QA strategy independently for complex data-heavy systems, including building automation frameworks and implementing quality gates
Skills in data validation using SQL and graph query languages such as SPARQL and Cypher, along with understanding of deterministic vs probabilistic systems
Background in backend and API testing (REST) covering data validation, integration and E2E testing
Proficiency in frontend web application test automation using Playwright or equivalent Python-compatible framework
Hands-on experience using coding agents and agentic development daily
Demonstrated experience testing and evaluating RAG-based Gen AI / LLM applications including grounding, answer accuracy and hallucination/determinism checks
Applied knowledge of Gen AI / LLM evaluation frameworks and metrics such as precision, recall, criteria recall and efficiency, with proven ability to automate Gen AI/RAG evaluation at scale
Experience in Agile delivery using Git and Azure DevOps or JIRA
English proficiency at B2 level or higher

Nice to have

Skills in semantic search testing and evaluation
Familiarity with Vector Database integration and retrieval validation
Background in data science covering ML concepts, data pipelines or data engineering collaboration
Proficiency in API tooling such as Postman and Bruno
Experience with quality-gate implementation in CI/CD pipelines such as Azure DevOps or Jenkins