Skip to content

Get blind review from experts
before you trust the agent.

Blind Bench hides each run’s version, model, and harness from reviewers, so experts judge the work and you learn whether it improved.

Get started Try a blind review

You can’t trust the read on your own agents.

  • A senior engineer opens the coding agent’s trace to review the new version. Knowing it’s the new one, they read to confirm it improved — not to find where it broke.

  • An LLM judge scores every run and hands you a winner. It drifts toward its own phrasing, favors what it would have written — and nobody grades the grader.

  • Your sharpest reviewer explains exactly why the agent failed. The note dies in a Slack thread, never re-attached to the trace that produced it.

The closed review loop

From agent run to a verdict you trust.

Bring in what your agent produced, get blind judgment from the people who know the domain, and route their verdict back to where it changes the next run.

01 Bring it in

Bring in what your agent did.

Import Claude Code trajectories, gateway logs, or plain JSON — or run a prompt across models yourself. Your keys stay yours; Blind Bench never holds them.

Trace received Ready

Source artifact

JSONL

support-agent-run.jsonl

24 turns · 3 test cases

Trajectory parsed 24/24
Sensitive fields isolated

Review batch

hidden from reviewers
A

Control

live support agent

v3
B

Candidate

new agent revision

v4
Prompt

You are a customer support agent. Be friendly and professional. Resolve the issue clearly.

Case BB-1842 · BYOK Prepare review
02 Review blind

Reviewers judge without the label.

One link, no account. Candidate and control sit side by side, with the version, model, and harness hidden — so experts weigh the work, not the name.

03 Route it back

Turn the verdict into the next run.

Preferences roll up to standings. Comments re-attach to the trace and feed regression sets, training data, and prompt revisions — each citing the note it answers.

Verdict routed Loop closed

Blind verdict

Candidate +12%

7 of 9 reviewers flagged the same tone issue.

Evidence attached

2 notes
A Tone

“Reads like a corporate form letter.”

B Specificity

“‘I understand your concern’ is filler.”

Prompt revision

v3 → v4

You are a helpful customer support agent.

Write like a helpful coworker, not a corporate robot. Avoid form-letter openers and get to the point warmly.

Reviewer evidence applied
Case BB-1842 · audit trail preserved Next run ready

What Blind Bench does that others don’t.

These tools already run human review. None hide the label from the reviewer or route the verdict back to the artifact. Blind Bench does both.

Annotation queues

in Braintrust, LangSmith, Langfuse

The gap
Capable tools — but every item shows the version and model. That anchors your highest-stakes call.
Blind Bench
Blind by construction. Provenance is stripped at the API boundary, so experts weigh the work, not the label.

LLM-as-judge

Promptfoo, plus your in-house graders

The gap
Fast and cheap — but the judge drifts to its own phrasing, and nothing grades the judge.
Blind Bench
Your experts review the same runs blind, un-anchored by the judge’s score — for calls that decide a release.

Docs, sheets, threads

the ad hoc review most teams run

The gap
No blinding, no structure — the sharpest comment never re-attaches to its version.
Blind Bench
Structured and blind — the verdict routes back to the trace, a regression set, and the revision that cites it.

Know whether your agent actually got better.

Five minutes to set up. One shareable link. BYOK.
Free while we're in beta — say hi and we'll set you up personally.

Get started