# Evaluations Engineer at Vals Ai

> Onsite · United States · Unspecified · Software engineering · open. Source: https://alderhq.app/jobs/ashby%3Avals-ai%3A13d02313-e987-46ee-8b1c-426ea58e8bd6 (read 2026-11-22).

## Facts (observed)

- Company: [Vals Ai](https://alderhq.app/companies/vals-ai)
- Location text: San Francisco
- Where it can be done from (inferred): Onsite · United States (evidence: "Location: San Francisco")
- Posted: 2026-06-27 (148 days ago)
- First seen by alder: 2026-11-07 · last verified: 2026-11-15
- Reposted: 0 times · edited: 1 times
- Apply on the employer's site: https://jobs.ashbyhq.com/vals-ai/13d02313-e987-46ee-8b1c-426ea58e8bd6
- Skills listed (inferred): React, Python, Django, AWS, LLMs

## Pay and sponsorship

- Posted pay: $130k–$185k per year (from the listing's pay field)
- Market band for similar posted roles (SF Bay Area, n 2331): p25 $160k, median $200k, p75 $240k

## Description (the employer's text)

### About the Role

We are looking for strong engineers to join our team and own the leaderboards that appear on Vals AI.

You'll be responsible for testing new models against our benchmarks as they're released; covering tasks in law, tax, coding, finance, social mobility, and more. You will analyze error modes of models, evaluate their strengths and weaknesses, and work with our communications team to release results.

Our results are used by startups, enterprises, and research labs alike. We work with all the major foundation model labs, some of the largest financial institutions, and hospital systems in the world. Our work has been featured by the Wall Street Journal, Washington Post, and Bloomberg.

We are building the standard for evaluating the ability of LLMs to perform real-world tasks. You will contribute directly to the leaderboards that make this possible.

### What You’ll Do

Evaluate new LLM model releases across the Vals AI suite of benchmarks

Work directly with both open-source and closed-source foundation model labs in evaluating model performance

Use tools like Docent to analyze common failure modes and patterns in model performance

Work directly with our social media team to post interesting findings and results

Add new models and maintain integrations in our model library

Help improve and maintain the infrastructure we use to run benchmarks (agentic and non-agentic).

This role follows the rhythm of model releases. Expect intense sprints in the days following a major launch, and calmer stretches in between releases.

### Preferred Qualifications

Familiarity with the LLMs: You should already be familiar with the space - the current leading models, relative performance across them, how to use large language models in practice.

Strong engineering fundamentals: You can build and ship quickly with high quality. You should have a track record of building things of significant scope (at jobs, side projects, open source, etc.)

Python expertise: Significant experience in Python, especially in a professional setting.

Team collaboration: Experience working in development sprints, Git workflows, and pull request reviews.

Strong work ethic: Willingness to work long hours during model releases and get high-quality results out under tight deadlines.

Location: We are an in-person team based in San Francisco. We will support your relocation or transportation as needed.

### Nice-to-Haves

Previous experience with benchmarking large language models, or creating benchmarks

Previous experience working at a startup or starting your own company

Technical writing experience and ability

Machine learning research experience

### What We Offer

Highly competitive salary and meaningful ownership. Excellence is well rewarded.

Relocation and transportation support

Health/dental insurance coverage

Lunch and dinner provided, free snacks/coffee/drinks

401K plan

Unlimited PTO

$1,500 housing stipend (within one-mile radius)

### About Us

Founding team: The core methodology behind this platform comes from NLP evaluation research we conducted at Stanford. We raised a $5M seed from some of the top institutional and angel investors in the valley. Our team has prior work experience at NVIDIA, Meta, Microsoft, Palantir and HRT. Collectively, we have over 300 citations in our published work. Our early team includes Stanford PhDs, ex-Jane Street quants, and the first designer at Snorkel.

We recently announced our $40M Series A at a $400M valuation, led by Andreessen Horowitz, with participation from existing investors 8VC, Pear VC, and Bloomberg and new investors Hudson River Trading and NextLadder Ventures.

Tech stack: We use Python for most things at Vals AI. Our platform is built on Django, with a React frontend. All of the infra is on AWS using CDK for IaC.

### What We're Looking For

Learning velocity: The role encompasses a wide variety of tasks. Rather than expecting you to be an expert on Day 1, we are looking for someone who can learn new skills and technologies quickly.

Ownership: Working in a small, talent-dense team, we expect everyone to show initiative to build where it's needed, not where it's asked. We strive for autonomy over consensus.

---
Data: alder (https://alderhq.app). Cite https://alderhq.app/jobs/ashby%3Avals-ai%3A13d02313-e987-46ee-8b1c-426ea58e8bd6 and the date you read it. Observed facts come from the employer's own listing; location reach, level, family and skills are inferences.
