About Judgment

Judgment Labs is building the infrastructure for continual learning in long-horizon AI agents.

We turn long agent trajectories into clean, structured data for evals, labeling, rubric generation, context engineering, and RL workflows. Instead of only showing teams what happened, Judgment helps decide what matters, what should be learned from, and how that learning should flow back into the agent.

We’ve raised $32M across two rounds led by Lightspeed.

The Role

We’re looking for a Backend Engineer to own the systems that ingest, structure, evaluate, and serve agent experience data at production scale.

This role includes the backend and data infrastructure surface area: high-throughput telemetry ingestion, ClickHouse-backed OLAP performance, evaluation pipelines, RabbitMQ/Temporal workflows, multi-tenant scheduling, and product-facing APIs. Some weeks you’ll be deep in distributed systems and query performance. Other weeks you’ll ship a customer-facing feature end to end across backend, frontend, and the data layer.

This is not a narrow API role. The backend is where raw agent trajectories become structured learning data.

What You’ll Do

Design and build backend systems for trace ingestion, trajectory processing, evaluation orchestration, scoring, labeling, rubric generation, and customer-facing analytics.

Own the API surface used by the Judgment platform UI, SDKs, JudgmentHub libraries, MCP server, Slack agent, and customer integrations.

Build and operate the RabbitMQ / Temporal evaluation pipeline, including retry semantics, failure recovery, state reconciliation, and tenant-level scheduling.

Optimize the ClickHouse OLAP layer: schema design, partitioning, skip indexes, full-text-search pruning, query rewrites, deduplication, pagination correctness, and storage growth.

Turn raw spans, conversations, tool calls, scorer outputs, and agent-judge results into clean data models customers can use for evals, labeling, context engineering, and RL workflows.

Ship features end to end, often across Next.js, backend APIs, queues/workflows, and the data layer.

Work directly with customers to understand where their agents fail, what data is useful, and how Judgment should structure that experience for learning.

Roll out safely with feature flags, design docs, code reviews, tests, observability, and production debugging.

Raise the engineering bar through clear interfaces, maintainable systems, thoughtful reviews, and strong ownership.

Interesting Technical Challenges

High-throughput telemetry ingestion: Parse and persist OTEL traces across protobuf and JSON formats at hundreds of thousands of spans per second, writing to ClickHouse while keeping ingest latency low and backpressure graceful as customer traffic spikes.

Petabyte-scale OLAP performance: Design schemas, partitioning, indexes, storage layouts, and query paths so behavioral queries over billions of spans stay fast. Turn real access-pattern analysis into concrete data modeling decisions.

Long-horizon trajectory modeling: Agent workflows are messy: multi-step tasks, tool calls, retries, partial failures, context changes, and unclear outcomes. Build the abstractions that turn those trajectories into structured data for evals, labeling, rubric generation, context engineering, and RL workflows.

Queue- and workflow-driven evaluation: Evaluations fan out across RabbitMQ and Temporal workflows. Getting this right means reasoning about retries, timeouts, idempotent state transitions, exactly-once-ish semantics, and reconciling runs that fail partway so nothing is silently orphaned.

Multi-tenant fairness at scale: A single large customer should not be able to starve everyone else. Build scheduling and execution systems so latency stays predictable across hundreds of teams sharing the same evaluation pipeline.

Near-real-time scoring: Behavioral scorers and agent judges call LLM APIs at scale, so batching, rate-limit management, retry/backoff, failure handling, and cost control are core backend systems problems.