Source-linked AI summary
When Is a Multi-Agent Code Judge Actually Grounded? Two Label-Free Measurements, and a Judge That Declines to Guess
Salma Roshdy Aly, Hussein Assaf, Ziad Kobti
TL;DR
A code judge can issue confident verdicts without evidence, while multi-agent verification depends on evidence independent of the answer and different across candidates. The study evaluates MARCH on code judging and uses pipeline logs to identify comparisons it cannot distinguish; gating those comparisons improves accuracy while retaining half.
Problem
Code judges can return confident verdicts without a basis, and evidence that does not differ between candidates cannot distinguish which solution is correct.
Source: p. 1Method
The study runs MARCH unmodified and uses pipeline-log measurements to diagnose indistinguishable comparisons and checker agreement without ground-truth labels.
Source: p. 1Results
MARCH reached 4.4% accuracy versus 43.7% for the same model asked directly, and declared both solutions equally good on 78 to 95% of comparisons.
Source: p. 1Takeaways & Limitations
Gating comparisons with identical proposer questions raises MARCH accuracy from 20.7 to 36.9% while retaining half of comparisons, but remains below direct judging.
Source: p. 1
Abstract
from arXiv · showhide
When one language model judges whether another's code is correct, it does not report the absence of evidence. It returns a confident verdict with reasoning attached, indistinguishable from a verdict it had grounds for. Multi-agent verification, which decomposes a judgment into checkable claims and verifies each against evidence, is a promising response and works well when the evidence is a set of retrieved documents. We argue such methods require two things of their evidence: it must be independent of the answer under review, and it must differ between the two candidates being compared. The second condition holds automatically with retrieved documents and stops holding in code judging. Running MARCH, a published framework unmodified over 80 condition-by-cell measurements on two code judging benchmarks, we find it declares both solutions equally good on 78 to 95% of comparisons, reaching 4.4% accuracy where the same model asked directly reaches 43.7%. Neither easier problems nor a larger judge changes this. Two measurements taken from the pipeline's own logs explain it without needing labels. Gating on one of them, the pipeline declines the comparisons it cannot make and raises its accuracy from 20.7 to 36.9% while still answering half of all comparisons. The contribution is not a more accurate judge, but a label-free way to tell when a judge has no basis for its answer.