Do you want to work out how you actually measure whether an AI system is doing a good job, and then make it better? We're looking for a sharp, analytical data scientist to own the evaluation and quality loop of our AutoQA product. You can be early in your career (recent graduates with strong LLM fundamentals are welcome), we have room and a clear path for you to grow.

In a nutshell

Join a fast-growing SaaS company in an international environment (steep learning curve guaranteed).

Own a high-impact problem: making LLM-powered quality assurance measurably accurate at scale.

Sit at the intersection of AI and CX, working directly with enterprise customers and their real-world QA rubrics.

Grow into a Senior Data Scientist, AI Engineer, or ML Engineer role. We invest in progression.

Enjoy the perks: flexible hours, open holiday policy, an office in the heart of Amsterdam with hybrid flexibility, visa sponsorship, great gear, workations, and team events.

About Kaizo

At Kaizo, we build a performance development and quality platform for customer support teams. Our AutoQA product uses LLMs to review support conversations against each customer's own quality rubric, automatically and at scale. Behind it sits a microservices-based stream processing platform handling over 200 million events per day (Kafka, Kubernetes on Google Cloud, ElasticSearch, MongoDB, BigQuery), and an LLMOps stack built around LangSmith for experimentation, prompt management, and tracing.

The hard part isn't calling an LLM. It's knowing, with evidence, how well the system performs on every customer's unique rubric, and having a reliable, repeatable way to improve it. That's where you come in.

What you'll focus on

Translate customer rubrics into AutoQA instructions. Work with real customer quality criteria and turn them into precise, testable instructions that LLMs can score reliably.

Run experiments that move accuracy. Design and execute evaluation experiments on large, representative datasets using LangSmith and BigQuery, and track quality with our performance metrics.

Build the datasets that make evaluation possible. Curate raw production data into golden datasets with balanced coverage, and generate synthetic data to cover the rare cases that matter most. QA is a discipline of rare occurrences: distributions are skewed, positives are scarce, and resourcefulness beats volume.

Build LLM-as-a-judge pipelines to assess system quality internally and make evaluation repeatable.

Make results actionable. Your experiments should end in a decision: change a prompt, adjust which tools the system uses, surface context the AI is missing, or flag where new capabilities are needed. You'll work with the AI team to ship those decisions.

Evaluate across the full stack. Beyond scoring quality, you'll help validate retrieval (RAG/IR), tool calling, and speech pipelines (transcription and diarization quality).

Join customer calls with the team to understand how QA leaders define quality, and feed what you learn back into the product.

What you'll grow into

Shaping how customers monitor quality themselves, catch drift, and keep their AutoQA setup improving over time.

Smarter categorization and routing of conversations to power analytics and get the right tickets to the right evaluation.

Requirements

A solid understanding of how LLMs work and hands-on experience prompting them for accuracy (coursework, thesis, internships, or side projects all count; production experience is a bonus).

Good applied statistics: experiment design, classifier evaluation, precision/recall trade-offs, and working with heavily imbalanced data.

Strong Python skills and fluency with the standard data toolkit (Pandas, NumPy, Jupyter). SQL is a plus.

An analytical, evidence-first mindset: you'd rather measure than assume.

Product sense and empathy for end users. You'll be building for QA managers and support agents, not just for benchmarks.

Excellent written and verbal communication. You'll present findings to the team and join customer conversations.

0 to 2 years of industry experience. Recent graduates with strong relevant work are encouraged to apply.

A team player who's comfortable wearing multiple hats. We're an early-stage company and things move fast.

Bonus points for

Experience with LangSmith or similar LLMOps/evaluation tooling

Google Cloud Platform, BigQuery, Docker, or Kubernetes

Speech/audio processing or ASR evaluation

Fine-tuning or building synthetic datasets for LLMs

Who you'll work with

You'll join our AI team (two data scientists, an AI engineer, and an ML engineer) and collaborate closely with our data engineers, frontend engineers, designer, product manager, and CX teams. You'll have mentorship from day one and real ownership fast.

What's in it for you?

An office in the heart of Amsterdam, with the flexibility of hybrid working

Visa sponsorship available for eligible candidates

Great office gear: MacBook, tools, desk, chair, whatever you need

Flexible working schedule and an open holiday policy

Fun workations and team events

A clear growth path into Senior Data Scientist, AI Engineer, or ML Engineer roles

Our interview process

We want the process to be useful for you too. Every step is a chance for you to get to know us, our work, and the people you’d work with. Here’s what to expect.

Intro call

A relaxed conversation to get to know each other. We’ll talk about your background and what you’re looking for next. Come with any questions about Kaizo or the team. We’ll tell you the outcome within 3 working days.

20 min, video call

Take-home task

We invite a small group of candidates (usually around 5) to work on a task based on a real problem from our work. You’ll have one week to complete it, so you can fit it around your current commitments. If your solution isn’t quite what we’re looking for this time, we’ll let you know by email with written feedback.

About 1 week to complete, at your own pace

Solution review with the data science team

You’ll present your solution to the AI team and discuss it together, including some whiteboarding. Treat it as a working session with future teammates.

90 min, in our Amsterdam office or remote

Conversation with our Head of Product (Optional)

Sometimes we’d like a second perspective before deciding. In that case, we’ll invite you to talk with our Head of Product.

30–45 min

Offer conversation with HR and our CTO

If we’d like to work with you, you’ll meet our CTO and HR. We’ll go through the role, the package and salary, your growth path, and practical details like your start date and visa sponsorship, if needed.

45 min

Meet the CX team

This role works closely with customers and our CX teams, so you might have a short, informal chat with our Customer Experience team.

30 min

Meet our CEO (Optional)

An introduction to our CEO to talk about where Kaizo is heading and where you fit in.

30 min

Good to know

Timeline: the full process usually takes about three weeks from intro call to offer.

Where you’ll work: we’re hybrid. The AI team is in our Amsterdam office about three days a week.

Questions at any point? Just ask the person you’re talking with. We’re happy to share more.