Accountabilities:

AI and LLM Quality Engineering
Design, implement, and optimise testing strategies for AI-powered applications, copilots, chatbots, Retrieval-Augmented Generation (RAG) pipelines, and autonomous AI agents.
Develop AI evaluation (Evals) frameworks to assess hallucinations, groundedness, toxicity, bias, response quality, contextual relevance, prompt regression, consistency, and safety.
Build automated validation pipelines for LLM outputs using structured assertions, semantic scoring, embeddings, and evaluation metrics.
Create synthetic and benchmark datasets to support comprehensive AI testing scenarios.
Test multi-agent orchestration workflows, including agent memory, context retention, planning, and execution.
Validate Model Context Protocol (MCP) integrations and interoperability between AI models and external tools.
Apply Context Engineering, Prompt Engineering, and prompt optimisation techniques to improve AI reliability and output quality.
Evaluate AI system performance, including latency, token consumption, cost efficiency, throughput, and scalability.
Automation Engineering
Develop, maintain, and scale UI and API automation frameworks using Python, preferably with Robot Framework and tools such as Playwright, Selenium, or Cypress.
Automate testing across REST APIs, GraphQL APIs, event-driven systems, streaming applications, webhooks, microservices, and Micro Frontend (MFE) architectures.
Build reusable automation libraries, testing utilities, and AI-assisted quality engineering workflows.
Integrate automated testing into CI/CD pipelines and implement quality gates to improve software delivery reliability.
AI Engineering and Platform Validation
Validate RAG architectures by assessing document chunking strategies, embedding quality, retrieval accuracy, re-ranking, and vector search relevance.
Work with vector databases and search technologies, including Pinecone, Weaviate, ChromaDB, Qdrant, and FAISS.
Build, test, and validate AI agents using frameworks such as LangChain, LangGraph, CrewAI, AutoGen, Semantic Kernel, and OpenAI SDKs.
Test advanced AI capabilities, including tool calling, function calling, agent memory, planning and execution, multimodal workflows, and structured outputs.
DevOps, Observability, and Reliability
Build and maintain automated quality workflows integrated with CI/CD platforms such as GitHub Actions, Jenkins, GitLab CI/CD, and Azure DevOps.
Work with Docker and Kubernetes environments to validate application deployment, scalability, and reliability.
Perform cloud-based testing and deployments across AWS, Azure, Oracle Cloud Infrastructure (OCI), and Google Cloud Platform (GCP).
Validate distributed systems, focusing on reliability, resilience, and performance under different operating conditions.
Monitor AI system behaviour using observability, logging, and distributed tracing tools.
Promote shift-left testing practices and contribute to continuous improvement in quality engineering standards.

Requirements

Technical Experience and Core Skills
Strong programming skills in Python, with the ability to develop and maintain robust automation frameworks and testing solutions.
Solid understanding of software testing principles, quality engineering methodologies, and modern software development practices.
Hands-on experience with AI and LLM testing, Prompt Engineering, Context Engineering, AI evaluation frameworks, RAG systems, and AI agents.
Experience building and maintaining CI/CD pipelines and working with Docker, Kubernetes, and cloud platforms such as AWS, Azure, OCI, or GCP.
Strong expertise in API testing and automation, microservices testing, and UI automation.
Experience with SQL, database testing, and data validation.
Strong debugging, analytical, and troubleshooting abilities, with a structured approach to solving complex technical challenges.
Familiarity with Agile and Scrum methodologies and experience collaborating within cross-functional engineering teams.
Preferred Qualifications and Additional Skills
Experience with Robot Framework and Playwright for automated testing.

How Jobgether works

We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.

We appreciate your interest and wish you the best!

Why Apply Through Jobgether?

Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.

#LI-CL1