Acquire, Repair, Preserve: A Diagnosis-Guided Post-Training Recipe for Small-Model Dialogue Game Agents

Nan Li

Source-linked AI summary

Acquire, Repair, Preserve: A Diagnosis-Guided Post-Training Recipe for Small-Model Dialogue Game Agents

Nan Li

arXiv:2608.28458v1Aug 28, 2026cs.CLcs.LG

TL;DR

Interactive dialogue games require models to maintain state, interpret feedback, and act validly across changing constraints, exposing local failures beyond broad knowledge deficits. The paper proposes acquisition through broad SFT, mechanically targeted turn-local repair, and capability-preserving model selection; the submission substantially improves public and in-domain scores while preserving static performance approximately, but transfer remains limited.

Problem
Dialogue-game agents must receive supervision at the particular trajectory decisions where feedback-conditioned progress stops, not merely learn from dialogue-game data broadly.
Method
The recipe combines broad supervised fine-tuning for game participation, turn-local preference pairs for mechanically verifiable Wordle failures, and delta scaling to preserve general capabilities.
Results
The submission raises public clemscore from 10.67 to 38.92 and closed in-domain score from 13.41 to 41.17, while aggregate static performance is approximately preserved at 44.14 versus 44.24 for the baseline.
Takeaways & Limitations
Broad SFT produces the largest improvement, while precise turn-local supervision gives the clearest repair signal and observed transfer is concentrated primarily within the targeted game family.
Takeaways & Limitations
Mechanically verifiable repair is demonstrated only for Wordle-family failures, semantic or strategic failures are untested, and applicability beyond one Qwen3.5-2B base remains unknown.

Abstract

from arXiv · show

Interactive dialogue games test a capability that static benchmarks largely leave implicit: a model must carry state across turns, interpret feedback, and choose valid actions under changing constraints. We study this setting in the LM Playschool Challenge with a 2B open-weight model, and find that many failures are not only broad knowledge failures but also local decision failures: repeated guesses, malformed actions, and violations of feedback that the model has just seen. These diagnostics motivate a training recipe organized around three steps: acquire broad game participation through supervised fine-tuning, repair mechanically verifiable failures within one targeted dialogue-game family using turn-local preference pairs, and preserve general capabilities beyond these dialogue games. In the official final evaluation, our submission improves public clemscore from 10.67 to 38.92 and closed in-domain score from 13.41 to 41.17, while approximately preserving aggregate static performance (44.14 vs. 44.24 for the baseline). Out-of-domain clemscore remains low at 7.88, with the largest gains concentrated in unseen variants of the targeted family. Our results suggest that broad SFT brings most of the model's capability improvement; turn-local supervision can be effective when failure detection is precise, with observed transfer concentrated primarily within-family.

1 Introduction

Dialogue games require models to maintain state, interpret feedback, and choose valid actions under accumulated constraints. The paper addresses local, mechanically checkable decision failures with a staged recipe that improves interaction while approximately preserving static capability.

Dialogue-game success requires maintaining state, interpreting feedback, and choosing valid actions under accumulated constraints.
The central training question is where to place supervision so the model reaches the particular decision that fails.
Small-model interaction is brittle: the official Qwen3.5-2B baseline scores 10.67 clemscore, versus 60.30 for its 27B sibling.
Broad imitation enables game participation, targeted Wordle repair uses turn-local preference pairs, and delta scaling aims to preserve general capabilities.
The final submission reaches 38.92 public clemscore and 41.17 closed in-domain score while aggregate static performance is approximately preserved at 44.14 versus 44.24 for the 2B baseline.
Turn-local preference pairs provide the clearest positive repair signal, whereas whole-dialogue preference tuning destroyed protocol compliance in the experiments.

2 Task and Evaluation Setting

The evaluation setting measures agents in dialogue-game trajectories mediated by programmatic game masters and game-specific scorers. It combines interactive metrics with static-benchmark performance, while official final suites provide the primary comparison because the pinned development split is small and variable.

Playpen organizes rollouts, training data, and online sampling over dialogue-game trajectories, while clembench mediates turns, tracks state, and enforces output protocols.
The official final evaluation contains 502 public, 1,272 closed in-domain, and 360 closed out-of-domain episodes.
Clemscore combines Played, the percentage of non-aborted episodes, with Quality, the task score on played episodes.
Quality is averaged only over games with Played greater than zero, so models can lose score by failing to play or by playing poorly after remaining in the game.
The validation split contains 430 static-benchmark instances from BBH, CLadder, EQ-Bench, IFEval, and MMLU-Pro, aggregated into Statscore.
The pinned public split has only one or two episodes per represented game–experiment combination, so few-point differences require caution and official final results are primary.

3 Our Method

The method acquires broad game participation with SFT, repairs mechanically diagnosable Wordle failures using turn-local preference pairs, and preserves general capability through scaled LoRA updates.

3 Our Method: Successful transcripts support broad SFT, paired alternatives support DPO, and fresh rollouts support online reward learning.
3.1 Acquisition: Phase A trains on success-only interactions balanced to at most 700 examples per game, raising the pinned development score from 13.05 to 43.85.Raw Wordle quality remains zero because the model still repeats rejected guesses and violates immediate feedback constraints.
3.2 Turn-Local Repair: Turn-local DPO pairs share a dialogue state, contrasting a mechanically erroneous completion with a valid alternative from a clean successful episode.Holding the history fixed concentrates the contrast on decision tokens responsible for repeated guesses, wrong lengths, invalid words, or duplicated answer fields.
3.2 Turn-Local Repair: Phase B applies two Wordle repair passes targeting repeated guesses and bad length, then invalid words and duplicated answer fields.The first pass removes repeat loops, while the second corrects newly exposed invalid-word and duplicated-field errors; aggregate effects are sign-inconsistent across evaluation rails.
3.2 Turn-Local Repair: Phase C branches current-model Wordle trajectories and applies DPO to immediate divergences between highest- and lowest-scoring siblings.It uses a frozen reference with β=0.20 to target residual behavior after fixed failure classes are addressed.
3.3 Preserving General Capability: Delta scaling uses W(s) = Wbase + s ∆LoRA and selects s ∈[0, 1] under a paired interactive gate to preserve static capability.At s⋆=0.85, the development mean is 50.43; the official merged model scores 38.92 clemscore and 44.14 statscore versus 44.24 for the baseline.

4 Results

The recipe substantially improves official interactive performance while approximately preserving static performance, but out-of-domain gains are driven by conditional quality on fewer played episodes and concentrate in held-out Wordle variants.

4 Results: 38.92 public clemscore and 41.17 closed in-domain score improve over the 2B baseline, with the submitted checkpoint identified as an fp32-merged model.The official result is reported for the submitted artifact.
4 Results: 44.14 aggregate static performance approximately preserves the 2B baseline’s 44.24.
4 Results: In-domain gains come from both Played and Quality, with Played nearly doubling while Quality also increases.
4 Results: Out-of-domain clemscore rises from 3.72 to 7.88 even as Played falls from 35.19 to 33.08 versus the untrained base.The clemscore increase therefore reflects conditional quality on fewer played episodes.
4 Results: 70.02 of 84.86 effective out-of-domain score movement, or 82.5%, comes from three held-out Wordle-crazy configurations.Other contributions are TA Frozen Lake (+9.17), TA Mastermind (+5.00), and Cryptolect (+0.67).
4 Results: The pinned development trajectory rises from 13.05 through 43.85, 46.83, 45.52, and 48.52 to 50.43 after delta scaling.The paper reports this as a trajectory rather than a causal decomposition because adjacent differences are mostly small and non-Wordle changes are mixed.

5 Discussion

The recipe’s gains are largest when supervision matches the current policy’s deficit: broad imitation improves participation, turn-local preferences repair precise decisions, and delta scaling selects a preservation–improvement trade-off. However, generalization is concentrated within related Wordle variants, with limited absolute out-of-domain capability.

Acquisition: Broad imitation is the largest improvement stage, with gains also appearing in games not targeted by later phases.This breadth is consistent with all-game training creating participation before local repair.
Local repair: Only turn-DPO pass 1 has positive clem marginals on both evaluation rails, at +2.98 and +1.49.Pass 2 and branch-DPO are sign-inconsistent across the submitted-model trajectory and controlled campaign.
Generalization: 82.5% of total positive out-of-domain movement comes from three held-out Wordle-crazy configurations, indicating transfer concentrated within the Wordle family.The targeted Wordle family remains near zero on hidden in-domain episodes.
Locality: Whole-dialogue DPO scored 12.35 and introduced protocol-violating preambles, while filtering stronger chosen responses raised the score to 25.76 but preserved principal abort patterns.The comparison supports turn-locality as a more precise repair regime than preference coverage over the entire remaining dialogue.
Model selection: Delta scaling selects an operating point along a learned LoRA update without adding supervision or taking a gradient step.It addresses preservation separately from acquisition and repair.
Model selection: 44.14 versus the baseline’s 44.24 aggregate static score shows approximate preservation, despite redistribution across components.BBH Quality rises from 0.00 to 30.47, while IFEval falls from 68.52 to 51.85 and EQ-Bench Quality from 64.78 to 56.43.
Follow-up training: None of five post-selection correction-based DPO arms exceeded the 50.43 development endpoint; scores ranged from 44.15 to 48.92.At this pipeline stage, additional correction-based training did not produce a stronger model.
Acquisition: The same Qwen3.5-2B base reaches 46.01 public, 46.57 hidden in-domain, and 15.62 hidden out-of-domain clemscore with SFT-only training.This comparison points to stronger broad supervision as a way to create a better starting policy before local repair.

6 Conclusion

The study supports a scoped recipe for small dialogue-game agents: broad SFT acquires participation, turn-local preferences repair precise within-family failures, and delta scaling preserves gains. The final improvement is substantial in public and hidden in-domain evaluation, but out-of-domain transfer remains limited.

Conclusion: 38.92 public clemscore and 41.17 closed in-domain score rise from 10.67 and 13.41, while aggregate static performance is approximately preserved.The corresponding static scores are 44.14 for the submission and 44.24 for the baseline.
Conclusion: Out-of-domain clemscore remains 7.88, with Played below the untrained base and transfer concentrated in related Wordle variants.These results support acquisition and precise within-family repair rather than broad transfer.

Limitations

The study’s repair claims are bounded by mechanically verifiable Wordle diagnostics, one acquisition-stage parent, uncertain phase attribution, and aggregate rather than component-wise preservation. Evaluation and implementation details further limit comparability and reproducibility.

Scope of mechanically verifiable repair: Repair coverage is restricted to Wordle-family failures with explicit post-feedback state constraints; semantic or strategic failures remain untested.Three held-out Wordle-crazy configurations contribute 82.5% of positive out-of-domain movement, and scale applicability is untested beyond one Qwen3.5-2B base.
Untested dependence on the acquisition-stage parent: The repair phases are evaluated after one capped broad-SFT recipe, so their usefulness under a differently trained acquisition parent remains unknown.The study does not vary acquisition strength or coverage while holding repair data and configuration fixed.
Limits of phase attribution and model selection: Later-phase effects are not causally identifiable because sequential training, repeated selection, evaluation variance, and cross-rail sign inconsistencies confound checkpoint differences.Only turn-DPO pass 1 is positive on both rails; the magnitude and necessity of individual later phases remain uncertain.
Aggregate rather than component-wise preservation: Aggregate statscore preservation does not guarantee preservation of each capability because the selection objective permits component gains to offset losses.BBH Quality rises from 0.00 to 30.47 while IFEval falls from 68.52 to 51.85 and EQ-Bench Quality from 64.78 to 56.43.
Evaluation boundaries: The official evaluation and development results use different dataset and game-tree revisions, while the pinned development split is small and run-to-run variation requires caution.The official evaluation uses snapshot 2dd5a533; development uses dataset revision 557d8caf and clembench commit ed39486, with 67 interactive episodes.

A.5 Controlled Single-Host Ablation

The controlled single-host campaign tests phase additions on a shared evaluation setup, revealing repeat-run variation and inconsistent trade-offs between interactive and static scores. Only turn-DPO pass 1 improves clem consistently across evaluation rails.

Controlled single-host ablation: Only turn-DPO pass 1 is positive on both rails, with clem marginals of +2.98 and +1.49.Pass 2 is −1.31/+1.34 and branch-DPO is +3.00/−2.07 across the two rails.
Controlled single-host ablation: +0.36 clem and +1.34 stat separate repeated turn-DPO pass-1 runs, estimating measurable repeat-run variation.Rows B and C use the same effective parent and configuration.
Controlled single-host ablation: Adding branch-DPO reduces clem by 2.07 while increasing stat by 1.23 in the controlled campaign.This result comes from the Row D to Row E comparison.
Controlled single-host ablation: Across phases, interactive gains generally cost a small amount of static performance, while the phase that costs clem gains stat.The reported trade-off is descriptive rather than a causal attribution of phase effects.

A.6 Abandoned Directions

The paper abandons whole-dialogue preference tuning because it harms protocol compliance, while limited 4B probes show that SFT outcomes depend strongly on the parent and training setup.

Whole-trajectory preference tuning: 12.35 clemscore followed unfiltered whole-dialogue DPO, which caused natural-language preambles and strict-format violations across multiple games.The arm used 3,973 full-dialogue preference pairs.
Whole-trajectory preference tuning: 25.76 clemscore followed source-filtered whole-dialogue DPO, with partial protocol recovery but continued Wordle and Codenames aborts.All six GuessWhat episodes were played, but none was won.
Whole-trajectory preference tuning: Whole-dialogue DPO used existing transcripts rather than live teacher queries, so its failure cannot be attributed to online teacher interaction.The contrast with turn-local supervision motivates held-fixed histories and token-level contrasts.
4B supervised probes: 24.36 clemscore resulted from a matched 4B SFT probe starting at 40.75, whereas success-only SFT on vanilla 4B trajectories reached 35.06.The two probes differ in parent and data-generation setup, so they do not isolate one training factor.
Static-performance outcome: The official static aggregate is 44.14 for the submission versus 44.24 for the baseline, despite component redistribution.BBH Quality rises by 30.47 while IFEval falls by 16.67 and EQ-Bench Quality by 8.35.

B Per-Game Matched Comparison

Matched per-game evaluation separates broad-SFT gains from Wordle-only phases and final scaling, but non-Wordle later-phase changes are descriptive because they rely on few episodes.

Broad-SFT effects: Macro-averaged Played rises from 30.95 to 84.19 between the base and broad-SFT checkpoint.This is the largest suite-level participation change reported in the matched comparison.
Broad-SFT effects: Macro-averaged Quality over played games rises from 42.15 to 52.09 between the base and broad-SFT checkpoint.The comparison evaluates the same checkpoints on one host, harness, dataset revision, and clembench commit.
Checkpoint decomposition: The comparison partitions changes into Base→SFT, SFT→Branch, and Branch→Final, corresponding to broad SFT, Wordle-only phases, and delta scaling plus merging.These columns are designed to make per-game deltas internally comparable.
Interpretive boundary: Later-phase changes outside Wordle are mixed in sign and based on only 3–13 episodes per game, so their interpretation is descriptive.The matched evaluation uses a common host, harness, dataset revision 557d8caf, and clembench commit ed39486.

C Plain-Wordle Behavior by Phase

Plain-Wordle process metrics show that the phases reduced repeated-guess behavior and stabilized protocol compliance, but scaling reversed participation on the fixed probe. The Wordle-focused phases therefore repaired protocol-level failures without solving the underlying guessing task.

Evaluation: Table 14 tracks three public plain-Wordle episodes across the pipeline using process-level metrics rather than game score alone.The table reports development evaluations on the pinned dataset.
Phase B: Mean repetitions fell from 3.0 after broad SFT to 0 after the second Phase B pass.Repetition was unobservable after the first Phase B pass and after scaling because no episode was played.
Phase C: Phase C achieved 100% played episodes, no violated requests, and 100% request success, while conditional Quality remained 0 at every checkpoint with played episodes.The evaluated Wordle-focused phases repaired protocol-level failures on this public probe but did not solve the underlying guessing task.
Scaling: Across repeated same-harness evaluations, the unscaled Phase C checkpoint played all three public episodes, whereas the s=0.85-scaled checkpoint and downstream artifacts aborted all three.Scaling reintroduced a bad-length guess of the kind targeted in the first Phase B pass.