Feedback That Backfires in Games: Do LLM Players Repeat the Move They Just Watched Fail?

1 Introduction

• Across eight instruction-tuned models, the record raises failed-move probability on fixed probes (G = −1.38 nats per token), chiefly through the move’s surface form.The effect holds for illegal moves but not losing ones, and is not universal: four models invert clearly while Llama-3.2-1B shows the opposite.Source: §1 · p. 2

2 Related work

• Earlier tool-calling and program-repair work found negative corrective gain, leaving open whether the effect extends to games, which training stage produces it, and whether task training removes it.That work also found that a runtime-generated failure description removed most inversion, while “do not repeat” and clean retry did not.Source: §1 · p. 1, §2 · p. 2
• Prior agent studies identify action repetition and training-time collapse, but do not isolate how a failure record changes the probability of the failed move.The paper distinguishes its inference-time transcript question from RAGEN’s training-time “Echo Trap” and from behavior-level repetition measures.Source: §2 · pp. 2–3
• Game benchmarks report rejected moves without isolating their cause, while context-contamination work motivates clearing failed attempts even though clean retry was previously the worst harness.The paper’s copying account also tests whether an effect attributed to move strings is present before instruction tuning; prior work on negative instructions motivates testing “do not repeat.”Source: §2 · p. 3

3.1 The quantity

• Corrective gain compares a failed move’s summed token log-probability before and after the move and runtime observation are appended; negative gain is feedback inversion.A valence-free observation separates the effect of writing the move from the effect of saying it failed.Source: §3.1 · p. 3

3.2 Two kinds of failure

• Games distinguish illegal moves, which leave the position unchanged, from legal losing moves whose consequences are verified exactly.Connect Four losses require an immediate opponent win; Sokoban deadlocks are checked by exhaustive search from a previously solvable room.Source: §3.2 · pp. 3–4

3.3 Conditions

• Conditions preserve the same prompt, goal, successful prefix, and position, varying only the attempt record; abstract failure descriptions omit the move string.Additional controls include a “do not repeat” instruction, repeated or different failed moves, and RAGEN-native feedback.Source: §3.3 · p. 4

3.4 Which failed move to probe

• Fixed probes perturb a correct move and retain only moves that fail; random failures provide a control, while model-proposed failures serve as a separate robustness check.Shared perturbation items avoid confounding cross-model comparisons with model-specific item difficulty.Source: §3.4 · p. 4

3.5 Mechanism test: notation

• Three notations vary move surface form while holding the board, rules, and messages constant, enabling tests of how notation, token length, and overlap relate to gain.The notations are short codes, verbose phrases, and plain zero-based indices.Source: §3.5 · p. 4

3.6 Formats, and why there are three

• A shared transcript representation renders completion, chat, and RAGEN formats, allowing the same conditions to be tested across model settings.RAGEN prompts are checked against its own prompt-building functions.Source: §3.6 · pp. 4–5

3.7 Scoring and statistics

• Scoring uses the original teacher-forced code and cluster bootstraps over positions; cached scoring is checked against uncached scoring, and the pipeline reproduces the original headline number.The CPU reproduction returns G = −23.92 nats, matching the original; the GPU result also returns −23.92.Source: §3.7 · p. 5

4.1 Games

• The setup covers Sokoban, Connect Four, and Othello, with illegal moves in all three and losing-move probes in Sokoban and Connect Four.Sokoban uses 6 × 6 rooms with one box; the other games use standard 6 × 7 and 8 × 8 boards.Source: §4.1 · p. 5

4.2 Items

• Each game contributes one decision point per position, rendered in every notation; runs score fixed-order prefixes, and probe items balance failure types.The RQ1 and remedy runs use 40 positions for smaller models and 30 for four larger models; OLMo checkpoints use 20 positions and RAGEN-format probes use 60 Sokoban positions. Some failed moves already appeared earlier in the transcript: 15/80 Connect Four and 24/100 Sokoban points.Source: §4.2 · pp. 5–6

4.3 Models

• The study compares eight instruction-tuned models, five OLMo 2 1B post-training checkpoints, and RAGEN’s two default models in its own Sokoban format.The OLMo checkpoints share tokenizer and architecture and differ in post-training; RAGEN’s models are probed before RL training.Source: §4.3 · p. 6

4.4 Hardware and numerics

• All reported scoring and free play ran in float32 on one Tesla T4 in free-tier Google Colab, with TF32 matrix multiplication disabled.Before game scoring, the GPU reproduced the original paper’s number; cached and uncached scoring also had to agree in all three formats.Source: §4.4 · p. 6

5 RQ1: does the failure record backfire in games?

• −1.38 nats per move token: pooled over eight instruction-tuned models, the record raises the failed move’s probability on shared items.The gain is negative on 61% of 840 model-item pairs.Source: §5 · p. 6
• Move-string exposure accounts for Δcopy = +1.34 nats per token, while the failure-message increment is Δsem = +0.04 and its interval includes zero.Figure 1 decomposes the effect into the contribution of writing the move and the additional contribution of stating that it failed.Source: §5 · p. 7, Figure 1 · p. 7
• Four of eight models invert clearly, while Llama-3.2-1B moves in the opposite direction (G = +0.66).Seven models have point estimates below zero, and Qwen3-0.6B has the largest measured inversion (−5.45).Source: §5 · p. 7
• Illegal moves invert clearly (G = −1.61), whereas the pooled result for legal but losing moves (G = −0.39) has an interval including zero.Greedy decoding repeats losing moves less often after the record (21%) than before it (26%); losing moves differ by game, with Connect Four at −0.84 and Sokoban deadlock pushes at +0.23.Source: §5 · p. 7, Figure 2 · p. 8
• On 759 realistic items where the failed move was among the model’s own top-three choices, the record instead raises its probability (G = +1.34).These items differ by model and are not pooled with the shared items; the paper identifies prior plausibility as the distinction between the item types.Source: §5 · pp. 7–8

• After the verbatim record, failed-move log-probability has a prior slope of +0.18 [+0.14, +0.21], versus +0.75 [+0.72, +0.78] for the abstract description.The valence-free record is similar to verbatim (+0.24 [+0.19, +0.28]); the move string, rather than the failure message, accompanies the flattening.Source: §6 · p. 8
• For moves with prior probability below 1%, G is −2.62 [−3.08, −2.16] nats per token, and verbatim-record greedy repetition rises from 0% to 22%.The abstract description leaves greedy repetition at 0% for this group.Source: §6 · p. 8
• For moves with prior probability above 30%, G is +1.68 [+1.46, +1.91], while verbatim-record greedy repetition falls from 91% to 38%.The abstract description lowers repetition to 60%.Source: §6 · p. 8
• Fixed probe items favor unlikely failures: 48% have prior below 1%, compared with 10% above 30%, helping explain why those probes invert.For model-proposed items, the corresponding shares are 2% below 1% and 44% above 30%.Source: §6 · p. 8
• The flattening depends on format: the prior slope is +0.45 [+0.37, +0.55] for OLMo 2 base in completion format and +0.09 [+0.04, +0.13] in RAGEN format.The informative comparison is between writing the move and describing the failure without its move string.Source: §6 · p. 9

• Before post-training, the base model has G = −0.52 [−0.70, −0.34] nats per token, negative on 82% of items.The copy term carries the inversion (+0.49 [+0.31, +0.67]); the failure message adds little (+0.03 [+0.02, +0.05]).Source: §7 · p. 9
• The largest ladder step is at DPO: gain rises by +0.30 [+0.21, +0.38] nats per token to −0.12 [−0.39, +0.15], an interval including zero.The copy term falls by −0.51 [−0.59, −0.42] and is no longer positive.Source: §7 · p. 9, Figure 4 · p. 10
• Supervised fine-tuning changes gain little (+0.10 [−0.00, +0.20]), while RLVR (−0.03 [−0.06, −0.00]) and final Instruct (−0.01 [−0.03, +0.01]) change little after DPO.In the post-trained checkpoints’ chat template, the SFT-to-DPO step is also +0.46 [+0.33, +0.59].Source: §7 · pp. 9–10
• Within this single 1B ladder, preference tuning removes most of the inversion, but one scale and recipe cannot distinguish DPO from its preference data.The five checkpoints share architecture and tokenizer and are compared on the same items.Source: §7 · pp. 9–10

8 Mechanism test: does the notation matter?

• For the two chat models, notation contrasts and length are not distinguishable from zero, while token overlap goes against H3 (+1.12, SE 0.36, p = 0.002).After controlling for prior, overlap increases inversion in the ladder, so its direction depends on that control.Source: §8 · pp. 10–11
• In the ladder, longer moves invert more per token (−0.36, SE 0.11, p < 0.001), while notation and token-overlap effects depend on whether prior probability is controlled.With prior controlled, overlap increases inversion in the ladder (−0.55, SE 0.11, p < 0.001), and verbose notation increases it in the chat models (−2.12, SE 0.34, p < 0.001).Source: §8 · pp. 10–11

9 RQ3: the RAGEN format, before RL

• In RAGEN’s Sokoban format before training, illegal actions invert for both default models: G is −2.08 [−3.24, −0.97] for Qwen2.5-0.5B and −3.82 [−5.54, −2.22] for Qwen2.5-1.5B.RAGEN’s own invalid-action warning, or no message after a blocked move, inverts at least as much as the paper’s feedback.Source: §9 · p. 11, Table 5 · p. 11
• For legal pushes that create deadlocks, G is +0.63 [−0.26, +1.47] and +0.50 [+0.29, +0.75] for the two models, based on only 15 positions.The first interval includes zero.Source: §9 · p. 11
• Whether RL removes the inversion in RAGEN’s training format, or transfers beyond trained games, remains unmeasured.Source: §9 · p. 11

• In the probe, replacing the move with an abstract description lowers its log-probability by −0.83 [−1.08, −0.58] nats, while a bare failure marker lowers it by −0.72 [−0.97, −0.47].A “do not repeat” instruction changes it by only +0.03 [−0.00, +0.06].Source: §10 · p. 11, Figure 5 · p. 12
• Quoting the move in the failure message raises its probability by +0.78 [+0.70, +0.86], and recording it twice raises it by +1.68 [+1.51, +1.86].When a different failing move is recorded, the scored move is less likely than when it is itself recorded (−0.77 [−1.02, −0.54]).Source: §10 · p. 12
• Across eight models, gain is −1.61 [−2.12, −1.14] nats under the verbatim record and −0.39 [−0.54, −0.24] under the abstract description, removing 76% of inversion.The description raises failed-move probability for two models, and it curbs greedy repetition less well for moves likely to be played.Source: §10 · p. 12, Figure 5 · p. 12
• In free play, exact repeats of failed moves average 94% under clean retry, versus 72% verbatim, 71% with the instruction, and 75% with abstract descriptions.A decoder ban reduces exact repeats to 3% and 0%, but illegal moves remain 81% under verbatim and 79% with the ban.Source: §10 · p. 12, Figure 6 · p. 13

11 Discussion

• Writing the failed move into the transcript pulls its probability toward a level set by its presence, raising unlikely moves but lowering likely ones.A failure description without the move string preserves the model’s preference instead.Source: §11 · p. 13
• Fixed probes mainly measure induced repetition, while in games the verbatim record lowers repetition of likely errors relative to deleting the attempt but leaves much repetition intact.The likely-error comparison was not tested in tool calling.Source: §11 · p. 13
• The OLMo 2 base model already inverts through the copy term, while preference tuning removes most of that push; one recipe at one scale cannot isolate DPO from its preference data.The copy term shifts from +0.49 [+0.31, +0.67] at base to −0.23 [−0.51, +0.05] after DPO.Source: §11 · pp. 13–14
• Harnesses should omit the failed move string, avoid relying on “do not repeat” or clean retries, and pair reworded feedback with a decoder constraint.Banning failed strings removed almost all exact repetition in free play, but did not produce legal play at these model sizes.Source: §11 · p. 14
• RAGEN’s default models show an inversion on illegal Sokoban actions before RL, but whether RL removes or works around it was not measured.The proposed training and SFT-control runs were not completed; their configurations and probes remain untested on real runs.Source: §11 · p. 14, §12 · p. 14

12 Limitations

• The stage-wise training result is bounded to models of 135M–1.7B parameters; the planned OLMo 3 7B scale check was not run.The available GPU memory did not fit float32 checkpoints larger than about 3B.Source: §12 · p. 14
• Free-play harness comparisons mostly reflect distributions of illegal moves, not game-winning ability, because 81% of moves were rejected and 68% of episodes hit the step limit.Rollouts used greedy decoding, making clean retry repeat the same move under an identical context.Source: §12 · pp. 14–15
• The counterfactual success message is slightly incoherent after an illegal move, so the decomposition relies on the valence-free reference message.Source: §12 · p. 14

13 Conclusion

• Recording a failed move flattens the model’s preference for it: unlikely moves become more likely, while likely errors become less likely but still repeat under greedy decoding.The move string carries the effect; it appears before post-training, shrinks at preference tuning, and is present in RAGEN’s pre-RL prompt format.Source: §13 · p. 15

Data and code availability

• The repository provides code, probe items, per-item records, free-play transcripts, run metadata, and a research log; one documented command regenerates every number, table, and figure.Source: Data and code availability · p. 15

Use of AI assistance

• Claude was used to write experimental and analysis code and draft and revise the manuscript; the author reviewed and edited the content and takes responsibility.Source: Use of AI assistance · p. 15

A Reproduction

• All reported runs used float32 on one free-tier Tesla T4, with TF32 disabled, and produced 51,726 scored records plus free-play episodes.The repository generates reported values by code and macros rather than manual transcription.Source: A Reproduction · p. 17
• On the original 100 ToolShed items, the CPU reproduction yielded G = −23.92, ∆copy = 21.69, and ∆sem = 2.23 nats, matching the original values.Stored log-probabilities reproduced within 9.5 × 10−5 nats; GPU values were within 3.5 × 10−4 nats of CPU rows.Source: A Reproduction · p. 17
• A second full-pool GPU session was prepared but not run after the free-tier quota ran out; none of its records enter the paper.Source: A Reproduction · p. 17
• Probe records are keyed by model, revision, format, game, notation, item, condition, and candidate; rollout records also include harness, temperature, and seed.Source: A Reproduction · p. 17

B Probe conditions in full

• Probe conditions keep the prompt and position fixed across conditions, while the message set includes valence-free, counterfactual-success, abstract-description, bare-marker, and move-echo variants.Source: Table 8 · p. 18, Table 9 · p. 18

C Messages

• The message table also includes an instruction condition asking the model not to repeat failed moves.Source: Table 9 · p. 18

D Notations

• Table 10 identifies the move notations and states that the RAGEN word form is used only in the RAGEN format.Source: Table 10 · p. 19

E Item statistics

• Table 11 reports item-pool counts by game and source, including perturbation, perturbation-plus-random, and random items.Source: Table 11 · p. 19

F Free play in full

• Table 12 reports free-play results under greedy decoding, defining exact repeat, illegal, wasted, and loop measures.The caption defines exact repeat as a failed move repeating an earlier failed move byte for byte; illegal as the share of all moves rejected by runtime; wasted as rejected moves per episode; and loop as episodes hitting the turn limit with a repeat.Source: Table 12 · p. 20