Source-linked AI summary
Easy to Catch a Liar, Hard to Clear an Honest One: Language Models Diagnosing a Corrupted Reward Channel from a Verified Record
Arman Nik Khah
TL;DR
A payout swap and a lying reward reporter can produce indistinguishable histories, leaving no way to identify the cause from reports alone. The paper tests frozen language models on a game where a verified record resolves that ambiguity. The models catch lying reporters reliably but often misclassify honest ones, with errors affected by irrelevant prompt features.
Problem
A payout swap and a lying reporter produce identical reports, so experience with the reports alone cannot distinguish the cause.
Source: §1 · pp. 1–2Method
The authors test frozen language models on matched game histories, adding an independent verified record of one round’s actual result beside the reporter’s claim.
Source: §1 · p. 2, §2.2 · p. 3Results
Large models catch lying reporters almost perfectly but clear honest reporters less often, with error rates varying by prompt features that carry no information about the answer.
Source: §1 · p. 2, §4.4 · p. 9Takeaways & Limitations
The tested models use verification to identify a broken reporter reliably but use it unreliably to recognize an honest one.
Source: §1 · p. 2, §8 · p. 13The one-token protocol measures a single-step read; reasoning aloud might improve honest-reporter clearance, which remains a different question.
Source: §7 · p. 12
Abstract
from arXiv · showhide
An agent that learns from rewards has to trust whatever reports those rewards. When the reports suddenly change, either the world changed or the reporter broke. From the reports alone these are indistinguishable, and reinforcement learning theory shows that no amount of further experience separates them. The prescribed escape is richer data about the reporter itself. We ask whether a frozen language model, handed exactly that data, uses it. We build a two-option game in which a payout swap and a lying reporter produce byte-identical histories. Then we add one verified record: an independent check of one round's real result, printed beside what the reporter said about that round. That single line settles the case. We ask three large models, from two families, to answer one question with one letter. Is the reporter honest or lying? They catch a lying reporter almost perfectly. At the 70B class that holds in every condition we tried; the 32B model slips in one wording. They clear an honest reporter far less often, and how often depends on things that should not matter. Averaged over rounds, letters, and wordings, a 72B model calls an honest reporter a liar 38% of the time when nothing has changed at all, and 58% of the time when the payouts moved. A 70B model from a second family calls an honest reporter a liar 26% and 48% of the time. The failure is not one of reading, because in the situation where nothing changed the same models score 0.96 to 1.00 with the answer printed in the prompt. Which surface feature drives it differs by family. For the Qwen models it is which round the record names, and for Llama it is which letter stands for "honest." Adding the record to a prompt that already states the answer makes Llama less likely to give that answer. We had registered a prediction for that 58% before the run: 35%. The failure is larger than we expected.
1. The question
A payout swap and a scorekeeper beginning to lie produce identical reports, so further play cannot distinguish them. The paper tests whether a frozen language model uses one verified record to identify the scorekeeper’s honesty.
1. The question
- A payout swap under an honest scorekeeper and unchanged payouts under a lying scorekeeper produce the same reports indefinitely.Without an assumption about the corruption, reinforcement-learning theory shows that no strategy can do better than guessing [2017].Source: §1 · p. 2
- A verified record pairs an independent statement of one round’s actual result with the scorekeeper’s report for that round.Because a source that starts lying does so on every later round, agreement indicates honesty and disagreement indicates lying.Source: §1 · p. 2
- Large models catch a lying scorekeeper almost perfectly but clear an honest one less often, with errors varying by prompt features that carry no answer information.The two tested model families are affected by different prompt features.Source: §1 · p. 2
2. The game, and the one line that settles it
The game makes payout changes and source dishonesty produce matched histories, then uses a verified record to distinguish them. Its four situations expose why a visible report flip alone cannot identify the cause.
2.1 A worked example
- In the worked example, the record says round 5 actually paid 0, matching Source 7’s report, so the source was honest and the values swapped.The prompt asks whether the source has reported each result accurately or reported its opposite since round 4.Source: §2.1 · p. 3
2.2 The four situations
- The four situations cross whether payouts moved with whether the source lied: nothing moved, honest; points moved, honest; nothing moved, lying; and points moved, lying.The points-moved, honest case tests whether the model blames the messenger when something real changed.Source: §2.2 · p. 3
- Matched pairs of situations produce byte-identical logs, so a report-flip rule is right in two situations and wrong in two.On a balanced design, that rule scores exactly 0.500, and overall accuracy cannot expose it because each matched pair contains one correct and one incorrect case.Source: §2.2 · pp. 3–4
2.3 Why the record is the only tie-breaker
- A verified record is the only tie-breaker between matched histories, and one record suffices because a lying source lies on every later round.Record rounds and values are balanced across worlds, so the model must compare the recorded result with the report rather than answer from the recorded value alone.Source: §2.3 · p. 4
3. How we ask
The evaluation uses one-letter answers across controlled prompt versions, worlds, wordings, and questions. It registers rules for interpreting results, including a heuristic control that fires for every large model tested.
3.1 One letter, no written reasoning
- The model gives one of two answer letters from its next-token distribution, without sampling or written reasoning.The protocol measures a one-step answer rather than what the model can derive with a scratchpad; answer letters are balanced so always emitting one scores 0.500.Source: §3.1 · p. 4
3.2 Five versions of each history
- Five prompt versions control for the presence of the record, explicit disclosure of the answer, and a salience reminder.The answer-printed condition checks whether the model can read the task; answer-printed plus record is the ceiling for the with-record condition.Source: §3.2 · pp. 4–5
3.3 Sixty-four worlds, four wordings, two questions
- The design crosses 64 worlds, four fact-equivalent wordings, five versions, four situations, and two questions, yielding 10,240 prompts per model.Intervals bootstrap over worlds rather than prompts because prompts from one world share a history.Source: §3.3 · p. 5
3.4 Why the second question is asked on the fresh pair
- The fresh-pair question asks which of two new options pays, testing whether the source-honesty verdict transfers to a choice untouched by the record.The record checks an old option, so the path to answering about the fresh pair runs through the source’s honesty.Source: §3.4 · p. 5, §3.3 · p. 5
3.5 Three rules fixed before the runs
- The registered rules void runs that exceed the answer-printed-plus-record ceiling or fail the answer-printed reading floor, and flag a flip heuristic above a 0.25 difference.Rule 3 fires for every large model; with-record scores are therefore read beside heuristic agreement, since a model following the heuristic would score 0.5.Source: §3.5 · pp. 5–6
3.6 Models and cost
- The evaluated models include Qwen2.5-32B-Instruct and Qwen2.5-72B-Instruct, and Llama-3.1-70B-Instruct and Llama-3.1-8B-Instruct.The paper reports rented-GPU compute cost of $13.19.Source: §3.6 · p. 6
4. Results
The models nearly always detect a lying reporter, but they clear an honest reporter less reliably, even when they can read the answer. Irrelevant prompt features affect honest verdicts differently across the two model families, and adding a matching record can further reduce some models’ accuracy.
4.1 The lying reporter is caught
- At the 70B class, lying-reporter accuracy stays at 1.000 for Qwen 72B and at least 0.984 for Llama 70B across four wordings; Qwen 32B drops on narrated prose.Across the two lying situations, Qwen 32B scores 0.934 and 0.938, while Qwen 72B and Llama 70B score 1.000 and 0.996, respectively.Source: §4.1 · p. 6
4.2 The honest reporter is cleared much less often
- The honest-reporter penalty is +0.135 at Qwen 32B, +0.482 at Qwen 72B, and +0.363 at Llama 70B.The respective intervals are [+0.098, +0.174], [+0.406, +0.557], and [+0.287, +0.443].Source: §4.2 · p. 6
4.3 It is inference, not reading
- In nothing moved, honest, answer-printed accuracy is 0.96–1.00, yet with-record accuracy falls by 0.22–0.44, indicating an inference failure rather than a general reading failure.Across all six rows, the loss beyond reading widens from round 5 to round 6, with every interval excluding zero.Source: §4.3 · p. 8
4.4 What moves the honest verdict, and what does not
- Record round drives honest verdicts for Qwen, while the letter denoting “honest” drives them for Llama; neither family’s lying verdict moves by more than 0.035.For Qwen, the round effect is +0.238 to +0.348 in nothing-moved honest cases; for Llama, the letter effect is +0.215 to +0.320.Source: §4.4 · pp. 8–9
4.5 A record that agrees with the reporter raises suspicion anyway
- Across worlds in honest, nothing-moved situations, answer-printed minus answer-printed-plus-record accuracy was +0.004 [+0.000, +0.012] for Qwen 32B, +0.078 [+0.043, +0.121] for Qwen 72B, and +0.254 [+0.176, +0.340] for Llama 70B.After being told the source had not begun lying and shown a record matching its report, Llama said the source was lying 29% of the time; in lying situations, the difference was within 0.012 of zero for every model.Source: §4.5 · p. 10
4.6 Saying the two causes are equally common changes nothing
- Making payout swaps and source lying equally salient changes accuracy by less than the registered +0.05 threshold across all three models.Across all four situations, the changes are −0.034, −0.010, and −0.017 for Qwen 32B, Qwen 72B, and Llama 70B, respectively.Source: §4.6 · p. 10
4.7 The 8B model cannot read the task
- Llama-3.1-8B scores 0.620 with the answer printed on the source question and 0.778 on the fresh-pair question, both below the registered 0.90 bar.By rule 2, none of its other numbers are read.Source: §4.7 · p. 10
4.8 Knowing is not acting
- On histories where Qwen correctly identifies a lying source, it still chooses the source-reported winner on fresh-pair questions more than 80% of the time.Answer-printed accuracy on the fresh-pair question is 0.987–0.994; Llama acts on its diagnosis more often, but still below half.Source: §4.8 · pp. 10–11
5. What was registered, and what happened
The central registered prediction for points moved, honest at Qwen 72B was wrong: the 64-world result remained near the earlier measurement. Applying the registered rules per situation discounts four cells, making the surviving model count depend on how the rules are read.
What was registered, and what happened
- 0.418 [0.316, 0.523] was the 64-world score for points moved, honest at Qwen 72B, near the 16-world 0.422 and below the registered 0.65 prediction.The pre-registered rule treated a score below 0.55 with an interval excluding 0.60 as a real deficit; it fired.Source: §5 · p. 11
- The round split was pre-registered, but its emphasis in Section 4.3 was retrospective, and Section 4.4 addresses a confound in the estimator.The authors report the confound, its fix, and both sets of worlds.Source: §5 · p. 11
- 0.887 at Qwen 72B and 0.742 at Llama 70B are the per-situation Rule 2 failures, both for points moved, honest.At the row level, every large model clears the 0.90 bar; the answer-printed source scores are 0.987, 0.972, and 0.921 for Qwen 32B, Qwen 72B, and Llama 70B, respectively.Source: §5 · p. 11
- Per-situation reading discounts four cells: Qwen 32B passes every cell, while Llama's penalty rests only on the row-level reading.Qwen 72B's nothing-moved, honest situation passes with 1.000 against 0.617; a per-situation reading leaves two models, not three.Source: §5 · p. 12
6. Related work
The paper places its question alongside work on corrupted rewards, model behavior, and prompt sensitivity. It distinguishes its contribution from these lines of research while noting a related shape in its own findings.
Related work
- Everitt et al. [2017] formalize corrupted reward channels and show that, without assumptions about corruption, no agent can generally do better than guessing; they identify richer channel data as a remedy.Bandit work on adversarial corruption bounds regret without asking agents to diagnose corruption, and the authors know of no prior work giving a language model a verified record in this setting.Source: §6 · p. 12
- Schmied et al. [2025] name the knowing-doing gap and find models can state a correct rationale yet act greedily; Section 4.8 reproduces that shape.Related dissociations are also reported for parametric knowledge [Cultural Binding Heads, 2026] and premise correction [Knowing but Not Correcting, 2026].Source: §6 · p. 12
- Option-letter and position sensitivity is documented [Zheng et al., 2024], while the paper's contribution is that these effects act on one of two symmetric judgments, not the other.Other work argues that primacy and anchoring are structural in autoregressive models [Bias by Necessity, 2026; Anchors in the Machine, 2025].Source: §6 · p. 12
- The ordering in Table 2 resembles inverse scaling, a pattern cataloged by McKenzie et al. [2023], but the paper explicitly makes no inverse-scaling claim.Source: §6 · p. 12
7. Limits
The findings are bounded by the tested models, seeds, prompts, and single-step protocol. The authors also identify an unexplained family difference and a confound in one result.
Limits
- Every number comes from one seed of 64 worlds, and the three models that clear the reading bar do not support a claim about model size.Source: §7 · p. 12
- Only four author-chosen wordings were tested; a fifth could shift honest-reporter results, though none of the tested wordings moved 70B-class lying results.Source: §7 · p. 12
- The one-token protocol measures a single-step read; allowing models to reason aloud might improve honest-reporter clearance, but is a different question.The authors state that the single-step read is the one a policy makes.Source: §7 · p. 12
- The authors give no mechanism for why the checked round moves Qwen results while the answer letter moves Llama results.They also note a confound in the salience result.Source: §7 · pp. 12–13
8. What it means
The paper tests whether language models use richer reward-channel data, finding that they use it unreliably in one direction. This can matter for agents acting on their verdicts, although the paper did not observe that action.
What it means
- The three large models reliably lower their honesty estimate when a check fails but do so unreliably when it passes, with results depending on the checked round or answer letter.Source: §8 · p. 13
- An agent acting on a false conviction could invert a channel that was telling the truth; the paper did not observe this downstream action.Section 4.8 reports that the tested models mostly do not act on their verdicts in either direction.Source: §8 · p. 13
- The authors release code, prompts, answer keys, and every scored output.Source: §8 · p. 13
Appendix A. One full prompt
The appendix gives the full prompt for a two-option points game and asks the model to judge whether a source reported results accurately or reported their opposites. A verified record states one round’s actual result independently of the source.
Appendix A. One full prompt
- Players choose between two options each round, with exactly one worth 1 point and the other worth 0.A reporting source reports each round’s result as 1 or 0 points.Source: Appendix A · p. 14
- After round 4, the original options’ point values may swap, the source may begin reporting the opposite of every result, both changes may occur, or neither may occur.If the source begins reporting opposites, it continues doing so on every later round; a new option pair may also be introduced with its own point values.Source: Appendix A · p. 14
- A verified record gives a round’s actual result independently of the source’s report; the example records 0 points for round 5, matching Source 7’s reported 0.Source: Appendix A · p. 14
- The prompt asks whether Source 7 has reported each result accurately or the opposite since round 5, requiring a single-letter answer.The respondent is instructed to reply K for accurate reporting or Q for reporting the opposite, and nothing else.Source: Appendix A · p. 14
Appendix B. Every number
The paper’s figures use the numbers file generated from scored outputs.
Appendix B. Every number
- NUMBERS.md is generated by code/paper_numbers.py from the scored outputs.Source: Appendix B · p. 14
- The paper identifies NUMBERS.md as the only source for every figure.Source: Appendix B · p. 14
- The stated source chain for every figure runs from scored outputs through code/paper_numbers.py to NUMBERS.md.Source: Appendix B · p. 14
Appendix C. Two tables that confirm the lying verdict does not move
The appendix reports two table findings: crossed round-and-letter results in lying situations remain high, while honest-situation results vary with wording more than lying-situation results at the 70B class.
Appendix C. Two tables that confirm the lying verdict does not move
- In Table C1, every lying-situation cell is 0.930–1.000 and every margin is within 0.035 of zero; Qwen 72B is exactly 1.000 in every cell.Source: Table C1 · p. 15
- Table C2 reports that honest-situation source accuracy with the record moves with wording.Source: Table C2 · p. 15
- Table C2 reports that lying-situation source accuracy with the record barely moves with wording at the 70B class.Source: Table C2 · p. 15