The Judge That Could Not Say Incorrect

One of our own benchmarks measures how well a model's written handoff lets a fresh session recover a software project, and it scored its reasoning answers with a model judge. The scorer read each verdict by testing whether the word CORRECT appeared in the judge's reply, and INCORRECT contains CORRECT, so all 1,256 judged answers in the store were marked right. A recount from the stored answer text, judged by Claude Haiku, reclassified 118 of them, 9.4%, as wrong, and a later check by GPT-6 Astra agreed with only 45.1% of the distinct answers behind those downward changes. A rotation between two judges had seemed to validate the judging by agreeing on 706 of 706 answers, but both judges ran through the same parser, so the check measured the bug twice.

A second defect turned out to be larger. When a judge call failed, the scorer fell back to exact string matching against the reference answer, a test that, applied to all 1,667 eligible reasoning scores, agreed with Haiku's recount labels on only 297 (17.8%), with 1,363 false negatives against 7 false positives by those labels. By the same labels, on the 411 scores that took the fallback it corrupted 322, 320 of them correct answers stored as wrong. The two defects therefore ran in opposite directions, and each partly hid the other in any total.

Exposure to the fallback depended on the day a run was made, because judge failures clustered by date. On 24 June 2026 the judge never answered, every eligible reasoning score that day went through string matching, and Gemini 3.1 Pro's whole arm as a handoff writer ran on that day. The ranking of handoff writers had shown a 34.6 point spread from first to last. After the recount the spread was 8.2 points, and its top and bottom matched a ranking built only on factual questions that never reached the judge. The original ranking had largely measured which day each writer's arm happened to run.

The judge's raw replies had never been stored. Each score kept a question id, the reference answer, the given answer, a method label and a single true or false, so the old verdicts could not be parsed again. Every eligible answer was therefore judged again from the stored text, and 1,667 scores reduced to 1,380 distinct combinations of question and answer text. Claude Haiku judged each one through a parser that raises an error instead of guessing. That parser also refused any reply carrying both verdict words, discarding an unambiguous leading INCORRECT whose explanation used the word correct, and a discarded reply was sampled again, not parsed again, so the error has a direction. Three of the 1,380 items were affected, and reading their first replies as the later, fixed parser does moves the benchmark's main effect from 0.1059 to 0.1052. GPT-6 Astra then labelled all 1,380 independently and agreed on 92.3% of them, with a Cohen's kappa of 0.536 and the disagreement concentrated where the first judge had said incorrect.

On the recorded panel of models, the benchmark's main effect survived both recounts, moving from 0.1155 to 0.1059 after the first and to 0.1035 under the second judge, a change of 0.0024 against a tolerance of 0.02 fixed before that number was computed. That pooled effect depends on including the panel's GPT configurations: without them the estimate is −0.0108 after the first recount (p = 0.81) and −0.0059 under the second judge (p = 1.0000). Two of the five claims made from the original scores did not survive: the ranking of handoff writers was retracted, and the explanation of a coaching effect was withdrawn. Two measures never touched the judge, factual accuracy and how often the successor found an answer at all, and both came out identical to the last digit across the recount. That is what a correction confined to the reasoning verdicts should leave behind, though equal totals cannot show that no individual record moved, and neither measure can show the new verdicts right.

The strict parser rejected 0.29% of replies in the recount, which ran the judge with its tool servers stripped. The production judge runs with its full configuration loaded, and its rejection rate has never been measured. The defect inventory, the date breakdown and the cross judge comparison are in the companion method and recount.