Introduction
Once a team stops trusting leaderboards, the next move is usually to build an evaluation of its own. We’ve argued before that benchmark scores are a poor predictor of production agent performance , and the natural follow-up question is what to measure instead.
Two answers come up almost every time, because they are the two cheapest evals to stand up. The first is a panel of LLM judges: ask several models whether an output is factually supported, and take the majority vote. The second is step-level grading: walk the agent’s trajectory one step at a time, score each step, and blame the first one that looks wrong. Neither needs labeled data, and both can be running by the end of the week.
Three papers posted to arXiv in the past week measured exactly these two signals against causal ground truth. Instead of asking whether the signal looks reasonable, each study intervened on the system, changing a known thing in a known place, and then checked whether the signal detected the change. Both signals came out badly, and the failures are specific enough to be worth understanding before you ship an eval that depends on them.
Judge Panels Agree Most Confidently When They Are Wrong
The appeal of a judge panel is the intuition that independent errors cancel out. Three judges agreeing feels like stronger evidence than one judge deciding, so unanimity gets treated as near certainty.
JuryProbe , by Tianxin Zhou and Ruixi Lin, tests whether those errors are actually independent, and finds that they are not. Reference-free factuality judges share blind spots: when one judge misses a false claim, the others tend to miss the same claim. The paper reports false-negative-only correlations of 0.402 and 0.368 between judges, which means agreement is partly a measure of shared weakness rather than converging evidence.
The consequence shows up in what the authors call false-consensus lift, the degree to which judges agree wrongly more often than chance would predict. They measure it at 3.13x on one evidence type and 18.13x on the other. At 18x, a unanimous panel verdict is close to the opposite of what teams read it as. The cases where every judge agrees are enriched for the errors every judge shares, so the signal you trust most is the one carrying the most correlated error.
What the paper offers in place of throwing the panel out is a routing policy. It uses a small labeled calibration set to estimate consensus risk up front, then sends only the high-risk decisions to judges equipped with trusted references. A fixed rule specified in advance flagged 8 to 10 of 10 splits across three datasets, drove unanimous false consensus to zero, and cost 0.004 in additional false accepts while avoiding 28% of reference acquisitions. That is the useful shape of the result: keep the cheap panel for most traffic, and pay for grounded verification exactly where the panel is least trustworthy.
Step-Level Coverage Works on the First Hop and Nowhere Else
Step-level grading has an even more attractive premise. If you can find the step where the agent went wrong, you know what to fix.
When Failures Propagate , by Lauren Pothuru, tests that premise with an interventional benchmark called AgenticRAG-FP. The method is the part worth copying: inject a certified fault at a chosen hop of a multi-hop retrieval trajectory, re-execute everything downstream, and then score diagnosers against the intervention you know you performed. The study runs 80 three-hop MuSiQue questions through Claude Haiku 4.5.
Coverage-based diagnosis, the standard approach of checking whether each hop retrieved the evidence it needed, scored 0.91 accuracy on faults injected at hop 1 across 43 failed trajectories. On faults injected at hop 2 it scored 0.00 across 36 trajectories, and on hop 3 it scored 0.00 across 21.
The reason for the collapse is more interesting than the numbers. Later hops repair earlier damage without announcing it. A downstream retrieval pulls a document that happens to contain what the broken hop failed to supply, so by the time the trajectory ends, the coverage trace looks intact and the injected fault is invisible. The signal is not noisy at depth, it is blind at depth, and it stays confident while being blind.
A counterfactual probe did considerably better on the same failures. Freezing the upstream hops and varying only the suspected one reached 0.67 accuracy on the 18 depth-2 content-corruption cases where coverage scored 0.00. Asking what changes when you change a step recovers information that asking whether a step looks correct does not.
The Signals That Look Like Credit Are Tracking Fluency
The third paper closes the loop by testing the signals teams use to assign credit to individual steps. Credit Without Ground Truth , by Haiyue Zhang, audits three of them against causal ground truth built from executed replay, meaning the alternative action is actually run rather than estimated: LLM-judge scores, outcome-conditioned logprob ratios, and policy confidence. All three performed at chance level at identifying the steps that causally mattered.
The paper names the confusion driving this cleanly. Existing evaluations measure step correctness, whether a step follows good practice. Causal credit requires step contribution, whether changing the step changes the outcome. Those come apart, and the gap explains a specific bad habit: implicit credit showed a median rank correlation of +0.75 with policy fluency, replicating at +0.70. A step-level score that correlates that strongly with how fluent the model sounds is substantially a fluency detector wearing an importance label.
Two further findings matter for anyone planning to act on step scores. Causal effects are sparse, with only 30.5% of decision points where ground truth is defined carrying a measurable effect, so most of the steps you are diligently scoring do not influence the result. And conditioning on the outcome, the move that intuitively should inject real signal, added a partial correlation of -0.004, which is nothing. A seven-arm training experiment on these signals found no arm that reliably beat the untrained policy.
How Much Weight These Results Carry
These are three recent preprints, and the sample sizes are small. The RAG study rests on 80 questions and a single model, and its own depth-3 corruption analysis has only three failed cases, which the author flags as too few to support conclusions. JuryProbe’s routing gains come from three datasets. None of this has been replicated independently yet.
What makes them worth acting on anyway is that they agree while measuring different systems with different methods, and that the direction of the error is consistent. In all three cases the cheap signal was not merely imprecise, it was confidently wrong in a predictable place: judges most wrong where they agree, coverage most wrong past the first hop, credit most wrong where the model writes well. A signal with a known blind spot is more dangerous than a noisy one, because nothing in the output tells you which region you are in.
Read them as strong evidence that these signals need validating in your own system rather than proof that they never work anywhere.
What This Changes in Practice
Concretely, if you are building or reviewing agent evals:
- Keep end-state grading as the signal of record. Task completion measured against a known answer survived all three of these audits because it is the thing being intervened on rather than a proxy for it.
- Treat unanimity as a flag to investigate rather than a confidence boost, and hold back a small labeled calibration set so you can estimate how correlated your judges actually are.
- Route by risk instead of choosing between a cheap panel and expensive grounded verification. The measured trade of 28% fewer reference lookups for 0.004 in false accepts is the kind of number worth reproducing on your own data.
- Distrust step-level attribution beyond the first hop unless you have validated it with an intervention. Coverage tracing at hop 3 is not weak evidence, it is close to no evidence.
- When you do need to know which step mattered, freeze the rest of the trajectory and vary that one step. It costs a re-execution and it measures contribution instead of appearance.
- Inject faults into your own pipeline on purpose. Everything above comes from studies that broke a system in a known way and checked whether the eval noticed, and that is a test you can run against your existing eval this sprint.
Conclusion
The uncomfortable part of all this is that the two evals a team builds first are the two that these studies found least reliable, and both fail quietly. An eval that misses problems while reporting healthy numbers is worse than no eval, because it converts an open question into a settled one.
Our earlier post argued that outcome metrics beat benchmark scores. These three papers sharpen that considerably: outcome grading is not just the better signal, it is close to the only one here that held up under intervention. The process metrics that feel more diagnostic earn that feeling from their granularity rather than from their accuracy.
If you want a second opinion on whether your evaluation is measuring what you think it is, our team at OmbuLabs.ai can help. Let’s talk . 🤖