Guide for: AI & ML Teams
For AI/ML engineers, product leads, evaluation, and platform teams using LLM judges but still checking the important calls by hand.
Calibrate Judges Against Expert Judgment
PROBLEM:
LLM judges are inconsistent, biased, brittle, and opaque.
We spoke to an AI team at a public company that had built its own internal eval platform, including multiple judges running against reviewed data with measured alignment. They planned to use those judges across a lot more production outputs, but they couldn't trust them.
We hear this a lot. Teams use an LLM judge because reviewing every output by hand doesn't scale. Then they realize the judge is another model with its own prompt, model version, biases, etc.
A few noteworthy problems that come up often:
-
Lack of reliable ground truth
LLM judges need ground truth backed by human annotations or a golden dataset you trust for the use case.
This is what the community calls the "ground truth wall." A stakeholder might ask what the judge is being calibrated against. The answer should come back to manual review.
But in some cases, even human annotations can be difficult to trust. For tasks that require deductive reasoning, two experts may reach different determinations. For ex., for one of our healthcare customers, two doctors often disagree on the same case.
"You can make a case for either one, but we need to maintain at least uniformity in what kind of logic we employ to make that answer."
Those disagreements show where the rubric might be vague, or context is missing, or even a case that is genuinely ambiguous. Teams need to either resolve them before treating the labels as ground truth or calibrate a judge to review those cases as ambiguous.
-
Variability, inconsistency, and reward hacking
The exact same query can get a different score even on consistently reliable ground truth. Judges can be sensitive to prompt wording, model version changes, and certain answer styles. Some of the ways this can manifest in are verbosity, self-enhancement, and position bias. Reward hacking, where a judge finds an easy way to satisfy the rubric without judging the output correctly, is also very common. We call these judges "yes-judges".
These are repeatable failure modes meaning changing the judge prompt may improve agreement in one area while making another worse.
-
Poor judge-human alignment and insufficient diversity
An alignment score between how humans and LLMs judge tells us how often the LLM matched the reviewed/annotated examples. But it only covers the datasets that have human reviews. Typically, this is the harderst part to scale for companies.
Dataset diversity matters more than dataset size. A few hundred diverse examples might be substantially better than a few hundred homogeneous examples.
A static golden dataset also may go stale over time. Teams tune against the same cases, the score improves, and the judge starts overfitting to the benchmark set. That is why new production failures should be continuously incorporated into these datasets.
If procuring diverse enough production samples is hard for you, given your indutry regulations for instance, you can explore synthetic data generation tools that keep your data true to the required structure and desired scenarios.
SOLUTION:
Build clear rubrics. Turn expert review into ground truth. Use it to
calibrate judges, then automate the checks you trust.
What to do
Our recommendation to AI/ML teams is to start with defining clear rubrics and to not rush beyond this crucial step until they are confidently ready. Rubrics define how experts will review the work (AI interactions) and how LLM judges will grade it later.
This is the first step in operationalizing and standardizing your expert reviews. You need to make sure that reviewers get a clear explanation of what needs to be done, and how they need to evaluate and analyze AI's work.
Here are some helpful tips:
- Provide clear instructions. Say what the reviewer should look at, what evidence matters, and how to handle missing context.
- Construct clear dimensions. You should think thoroughly about how you want to use the result of the reviewing process and pick the right dimensions. For ex., a scale of 1-5 might fit best for your task or for some tasks, a binary scale might work better.
- Then add examples. Include good, bad, and ambiguous ones. Examples matter because a rule that looks clear on paper often breaks on the first borderline case. They also give reviewers a shared point of reference, which reduces variation between their reviews.
Treat rubrics as a working standard. When a review exposes a vague instruction or a missing case, improve it, but keep the old version so earlier decisions still make sense.
What to do
The finalized rubrics can be used to review real cases with experts who understand the work. And their decisions become the ground truth used to test the LLM judges.
Getting experts out of their day and into a routine of evals is very difficult and their time expensive, so we need to make the review easy for them. A doctor, lawyer, analyst, or support lead should not need to understand AI traces, their syntax, prompt engineering, or eval tooling before giving useful feedback.
It is a good practice to have more than one expert review the same cases. If they disagree, reading their reasoning might prove insightful. You might need to fix the rubric when the instruction is vague, or add an example when a case is new.
Ideally, a useful review should help with the next similar case, not disappear into a spreadsheet or a one-off conversation. This way, over time, the review processes get faster and more efficient.
The feedback can also reveal a broader failure. Once one reviewer explains what went wrong, it might be prudent to look for other production cases with the same pattern. Some of those failures may have gone unnoticed before.
What to do
To calibrate, you should run the LLM judge on the same cases the experts reviewed. We use the same rubric, with instructions and examples. This gives you a direct comparison between the judge and human ground truth.
You could test different judge prompts on different models and compare their accuracy and judge-human alignment. Then dive deeper into the disagreements to look for behaviour such as false passes, false fails, and reward hacking.
The dataset you run through the judges is critical. You should include normal cases, known failures, unusual inputs, ambiguous cases, and mistakes that would be expensive to miss in production.
Once the judge is aligned, you can be confident it will preserve the expert knowledge behind it. You should be ready to scale automated judgement in production and monitor how it is doing.
What to do
Once an LLM judge has been calibrated against expert review, it can be used for automated evaluations.
Calibration results can help determine what is ready to automated. Cases with strong agreement between the judge and human reviewers may be suitable for automated evaluation. Clear rules may be handled with deterministic guardrails. Ambiguous or high-risk decisions should continue to involve human review.
This should be established as an ongoing process and automated decisions should be continuously sampled and reviewed. Comparing judge decisions with expert judgments helps identify new problems and areas where the evaluation criteria need to be updated.
When experts review a new or disputed case, that decision can be added to the ground-truth dataset and included in future evaluations.
Optional 60–90 second product demo
See where one judge disagrees with the experts.
Start with a rubric and a small set of reviewed traces. Run the judge, open one disagreement, make a change, and run it again.
Video placeholder
Rubric → reviewed traces → judge evaluation → disagreement → new run
FAQ
Frequently asked questions.
- What is ground truth?
- Verified data used as the gold standard to check if an AI's answers are correct. Ground truth
- What does it mean to calibrate an LLM judge?
- Run the judge on cases people have already reviewed. Then compare its verdict with the human verdict. The cases where they disagree tell you what needs work. Judges & Calibration
- What if the human reviewers disagree?
- Read those cases before using the labels as ground truth. The rubric may be vague, the evidence may be incomplete, or the task may allow more than one valid answer. DataFramer shows reviewer agreement by rubric dimension so you can see where this is happening. Reviewer agreement
- How many reviewed examples do we need?
- Start with enough cases to cover normal behavior, known failures, difficult edge cases, and mistakes that would be expensive. If the first 200 examples all look alike, adding another 800 like them won't tell you much about the cases the judge misses.
- What is a good Judge-Human Alignment score?
- There isn't a useful universal cutoff. A team checking marketing copy can tolerate a different miss rate than one checking a regulated recommendation. Decide from the false passes, false failures, and the worst disagreement cases, not the average score alone. Judge-Human Alignment
- Can we use the judge in production?
- A calibrated DataFramer judge can grade one interaction through the API and return the verdict inline. The call and verdict are recorded for review. Keep sampling those decisions and send uncertain or high-risk cases back to experts. Judge API
Let's see where your judge gets it wrong.
Start free, add a set of reviewed traces, and compare your judge with the people who understand the work.