Back to SearchSenior Test Automation Engineer – AI Solutions (Python)
Hybrid in Belarus
Automated Testing in Python
& 6 others
Choose an option
We are looking for a talented and experienced Senior Test Automation Engineer to join our team and help productionize AI solutions leveraging cutting-edge technologies. This is an exciting opportunity to work on AI-driven systems, including Retrieval Augmented Generation (RAG), Large Language Models (LLMs), and AWS Bedrock capabilities. You will play a critical role in ensuring the functionality, performance, scalability, security, and reliability of these innovative solutions.
Responsibilities
- Design and implement comprehensive testing strategies tailored to AI workflows, APIs, and cloud-native solutions
- Develop detailed test plans, test cases, and scripts for AI systems, including data pipelines, knowledge bases, vector databases, and LLM-powered components
- Validate the performance and accuracy of LLM outputs, ensuring compliance with prompt engineering best practices and client-specific requirements
- Test RAG-based architectures, including retrieval processes, embedding validation, and semantic search accuracy
- Ensure seamless integration of data ingestion, cleansing, transformation, and validation pipelines with vector databases and AI models
- Build and maintain test automation frameworks for AI components and data pipelines using industry-standard tools and custom scripts
- Conduct performance, load, and scalability testing for AI/LLM endpoints, RAG workflows, and deployed pipelines
- Test cloud infrastructure deployment (e.g., IaC via Terraform/CloudFormation) and ensure production-readiness for AWS Bedrock, S3, EKS, and vector database configurations
- Identify, log, and manage defects in AI workflows, APIs, data pipelines, and backend systems, collaborating with developers and DevOps teams to resolve issues
- Monitor system reliability, availability, and fault tolerance in cloud environments to ensure production-level readiness
Requirements
- Proficiency in test automation using Python (3+ years of experience)
- Proven experience in QA engineering, with a focus on testing AI systems and cloud-native solutions
- Strong knowledge of testing frameworks, tools, and methodologies for AI workflows, APIs, and data pipelines
- Hands-on experience with LLMs, RAG architectures, and vector databases
- Proficiency in test automation using Python or similar scripting languages
- Expertise in performance and scalability testing for AI systems
- Familiarity with cloud environments, particularly AWS services like Bedrock, S3, and EKS
- Solid understanding of Infrastructure as Code (IaC) tools such as Terraform or CloudFormation
- Excellent problem-solving skills and ability to collaborate effectively with cross-functional teams
Nice to have
- Experience with AWS Bedrock capabilities and generative AI solutions
- Knowledge of semantic search and embedding validation techniques
- Familiarity with continuous integration and delivery pipelines for AI systems
- Background in data engineering or data science