Source-linked AI summary
First Make It Playable, Then Make It Good: Staged Interaction Learning for Small Dialogue-Game Agents
Syed Mahbubul Huq, Pranava Madhyastha
TL;DR
Dialogue-game agents must learn both valid interaction protocols and effective decisions. The paper trains Qwen3.5-2B through successful-trajectory imitation, weighted turn-level SFT, and constrained teacher-guided repair. The final model reaches 57.12 clemscore and 42.68 statscore on public Playpen validation, while official challenge results show a +35.99 Playpen clemscore delta and the second-highest submitted delta.
Problem
Dialogue games require models to maintain valid interaction protocols while choosing appropriate actions, unlike isolated instruction-response training.
Method
The paper uses three stages: successful full-trajectory SFT, weighted turn-level SFT, and teacher-guided judging and repair that preserves original gold actions.
Results
57.12 clemscore and 42.68 statscore are achieved on public Playpen validation; official challenge evaluation reports a +35.99 Playpen clemscore delta and the second-highest submitted delta.
Takeaways & Limitations
Complete trajectories improve playability, while weighted turns and teacher guidance improve decision-making and overall score without requiring heavier replay-repair or hard-example mining.
Takeaways & Limitations
The method does not explore online reinforcement learning, explicit search, or planning, and its smaller out-of-domain gain suggests some improvement remains domain-specific.
Abstract
from arXiv · showhide
We present Qwen-GuidePlay-2B, a 2B-parameter language model for dialogue-game interaction. We fine-tune Qwen3.5-2B using three steps: a) SFT on only successful game trajectories from Playpen, b) weighted turn-level SFT, and c) teacher-guided SFT. The teacher model (which is a larger model) is only used to fix formatting and evaluate examples, but does not create new gold actions. Our final model scores 57.12 clemscore and 42.68 statscore on the public Playpen validation. In the officially released challenge results, our model obtains the second-highest Playpen clemscore delta among submitted systems (which is approximately +36 over its base model). Our findings suggest that imitating full trajectories helps with playability, while turn-level and teacher-guided training usually improve decision-making and increase the overall score. Alternative procedurally heavy approaches like replay-repair and hard-example mining did not help, which suggests that small models are performant simply by using careful curation strategies rather than aggressive changes. We make available both the model and the code for reproducibility.
1 Introduction
Dialogue games require agents to follow interaction protocols while making appropriate decisions, so the paper separates playability from quality and proposes staged training for both.
- Dialogue-game agents must observe states, produce legal actions, receive feedback, and continue under a defined protocol.
- Playability measures completing episodes without aborting, whereas quality measures task performance after valid participation is established.
- The proposed recipe first imitates successful full trajectories, then trains on local state-action examples, and finally applies constrained teacher guidance.
- Successful transcripts combine interactional signals for rule adherence with decision signals for selecting actions in specific dialogue or game states.
- The paper reports complementary roles for successful trajectories and factorised state-action examples, while lightweight teacher guidance repairs examples without creating new gold actions.
2 Method
The method progresses from successful-trajectory imitation to weighted local decisions and constrained teacher-guided repair, while preserving original successful actions as targets.
- Stage 1: Stage 1 fine-tunes Qwen3.5-2B only on successful Playpen trajectories to teach valid interaction structure.Sequences longer than 1,024 tokens are truncated.
- Stage 2: Stage 2 factorises each successful transcript into examples mapping dialogue/game history to the next assistant action.This trains directly on the inference-time local decision problem.
- Stage 2: Turn-level examples receive heuristic weights that prioritize clean, concise, non-leaky completions, with values capped at 1.0.The proxies are computed from completion text alone rather than learned advantages or equilibrium-derived Q-values.
- Stage 2: Stage 2 uses completion-only weighted SFT, with a uniform-weight ablation testing whether gains arise from factorisation alone.
- Stage 3: Stage 3 adds a small amount of teacher-judged, reweighted data and invalid near-miss repairs while keeping Stage 2 rows dominant.The teacher judges examples and generates formatting or protocol errors for repair training.
- Stage 3: The teacher may change an example’s weight but never changes the original successful action used as the training target.Invalid outputs appear only on the input side of repair examples.
- Stage 3: The teacher ratio remains small because heavier replay or repair improved Playpen-specific clemscore while reducing statscore in development runs.
- Implementation: All stages use Qwen3.5-2B with LoRA adapters, later merged into a full model for release.
3 Experimental Setup
Experiments use Playpen training data and public validation, evaluate interactive and static performance, and distinguish official baseline comparisons from local playability analyses.
- Data: All training supervision comes from the Playpen interactions training split, while validation data is reserved for public evaluation and model selection.No private evaluation data is used for training or model selection.
- Data: The data pipeline contains successful trajectories, factorised turn-level rows, teacher-judged rows, and teacher-generated repair rows.
- Data: The training split contains 20,202 successful trajectories across 16 games and yields 105,972 factorised turn-level examples.The final Stage 3 mixture contains 31,100 rows.
- Ablations: The ablations test complete training mechanisms rather than tuned constants, because heuristic and teacher parameters were set once by inspection under a fixed compute budget.
- Evaluation: Evaluation uses Playpen 3.7.0 and the public validation pipeline, covering interactive dialogue games and a static robustness suite.Clemscore measures interactive performance and statscore checks against over-specialisation.
- Evaluation: The official Qwen3.5-2B base scores are 13.05 clemscore and 44.02 statscore, while the local base supports playability and quality analysis.
4 Results and Analysis
The staged recipe substantially improves Playpen performance, with complete trajectories mainly improving playability and later turn-level and teacher-guided stages improving decision quality and precision. The final model also outperforms the strongest non-teacher baseline, while official challenge results show a strong clemscore gain but a statscore decrease.
- Main results: 57.12 clemscore and 42.68 statscore are achieved by the final model on public Playpen validation, exceeding Stage 2 Weighted-turn by +3.80 clemscore and +1.07 statscore.
- Main results: +35.99 Playpen clemscore improvement over Qwen3.5-2B makes the model the second-highest clemscore-delta submission in the official challenge results.On held-out challenge sets, clemscore improves by +32.85 in-domain and +6.53 out-of-domain, while statscore changes by −1.94.
- Playability and quality: Stage 1 raises episodes played from 28.57% for the local base model to 85.60%, indicating that successful full trajectories primarily improve playability.The paper associates this gain with learning Playpen’s action format, turn structure, and stopping behaviour.
- Playability and quality: Stage 2 raises average Quality from 53.46 to 63.65 while maintaining nearly constant playability, exposing local decision signals through factorised state-action examples.
- Playability and quality: The teacher-guided configuration raises average Quality to 68.57 while maintaining similar playability, improving decision precision and strict-format behaviour.
- Playability and quality: The pipeline assigns complementary roles to its stages: complete trajectories support playability, value-weighted turns improve performance, and teacher-guided repair increases precision.
- Ablations and negative results: 57.12 clemscore and 42.68 statscore are reached by the matched teacher-guided model, compared with 54.71 clemscore and 41.24 statscore without teacher-judged and repair examples.Under matched data, the small teacher-guided component improves clemscore by +2.41 and statscore by +1.44.
- Ablations and negative results: Heavier replay-repair, outcome-based weighting, and hard-example mining were diagnostic alternatives rather than exhaustive hyperparameter sweeps.The ablations compare the lightweight teacher-guided mixture with replay-repair training, scaled replay-repair merging, outcome-based weighting, and loss-based hard-example mining.
5 Model Release and Reproducibility
Qwen-GuidePlay-2B is released as a merged 2B Hugging Face checkpoint, with accompanying GitHub code and evaluation configuration details.
- The released model is a full merged 2B Hugging Face checkpoint rather than only a LoRA adapter.
- Reproducibility resources include code on GitHub and evaluation through the local Hugging Face backend.The evaluation follows the public Playpen setup with specified registration, padding, end-of-sequence handling, and generation-configuration settings.
6 Limitations
The method excludes online reinforcement learning, explicit search, and planning, while teacher guidance remains limited to judging and repairing existing successful-trace actions.
- The training method does not explore online reinforcement learning, explicit search, or planning.
- The teacher judges and repairs existing successful-trace actions but does not propose new strategies.Its role is restricted to modifying example treatment and repairing invalid outputs around existing gold actions.
- +32.85 clemscore in-domain versus +6.53 out-of-domain suggests that part of the improvement remains domain-specific.
A.1 Judging prompt
The judging prompt defines a strict JSON-only evaluation of training examples, combining dialogue context, gold answers, and Stage 2 weights with realistic formatting near-misses for repair.
- A.1 Judging prompt: Movement answers use an exact GO: direction format, while completed tasks use exactly DONE.
- A.1 Judging prompt: The contract forbids explanations, markdown, role prefixes, punctuation, stop tokens, multiple actions, and premature or verbose DONE outputs.
- A.1 Judging prompt: The prompt supplies dialogue history, the original gold answer, and the old Stage 2 training weight to a strict data-quality judge.
- A.1 Judging prompt: The teacher must preserve the provided gold target and may generate realistic invalid outputs that are close to, but stricter-invalid than, that target.
- A.1 Judging prompt: The judge scores format, action, task progress, completion, and overall usefulness, then returns a keep/drop decision and reason.
- A.1 Judging prompt: The output contract requires exactly one JSON object, with no markdown or explanation outside JSON.
- A.2 Repair prompt: The repair stage turns each retained invalid output into a training example whose target remains the original gold action.
- A.2 Repair prompt: The repair prompt asks the model to rewrite the invalid answer using the game history and output only one valid final answer.
B Teacher-generation details
Teacher-generation details constrain context length, sampling, candidate generation, repair retention, and reproducibility through a fixed random seed.
- Teacher judgements use Gemma-4-31B-it at temperature 0.1 with a maximum generation length of 700 tokens.
- Serialized dialogue context is capped at 7,000 characters, retaining the most recent context when truncation is necessary.
- The teacher generates at most four invalid-output candidates per judged example, with at most two repair examples retained per source row.
- The procedure uses random seed 28 for reproducibility.
B.1 Teacher-data selection
Teacher-data selection retained high-scoring judged Stage 2 rows, sampled a subset for Stage 3, and separately sampled repair examples while preventing judged rows from entering the ordinary pool.
- 2,943 Stage 2 rows were retained from the teacher-generated judged file, including 2,888 with overall scores of at least 0.70.
- The final Stage 3 mixture samples 1,000 rows from the eligible judged pool.
- 5,208 repair candidates derived from 2,604 Stage 2 source rows yielded 100 randomly sampled repair examples.
- Rows used for teacher judging were excluded before sampling the 30,000 ordinary Stage 2 rows.
B.2 Ablation configurations
The ablation configurations vary training duration, weighting rules, hard-example mining, replay-and-repair, and merge scaling relative to the Stage 2 setup.
- The Weighted-turn 0.50-epoch variant doubles Stage 2 training from 0.25 to 0.50 epoch while keeping the weighted-turn setup.
- Outcome-based weighting replaces the Stage 2 weighting scheme with heuristic weights based on successful and failed or aborted trajectories.
- HEM applies loss-based hard-example mining over Stage 2 examples.
- RR applies the heavier replay-and-repair intervention described in Section 4.3.
- RR-scaled merge does not introduce an additional training object.