Representation-Aligned Auxiliary Supervision for Language Model Adaptation

1 Introduction

• The study asks when environment-derived auxiliary supervision can improve adaptation and how representation compatibility affects its effectiveness.The motivation is that target-specific supervision can be costly, while structured environments offer automatically derived signals such as state properties, action constraints, and transitions.Source: §1 · pp. 1–2
• Semantically equivalent chess inputs can differ sharply in processing: next-state prediction accuracy is 47.4% in ASCII versus 3.9% in FEN.Qwen3-8B often reconstructs an ASCII board from FEN before reasoning.Source: §1 · p. 2, Figure 1 · p. 2
• Environment-derived auxiliary tasks consistently improve chess optimal-move prediction over target-only training, with dynamics-oriented tasks providing the largest gains.These gains can often match those from adding target-task data; static-property and state-manipulation tasks yield smaller, less consistent improvements.Source: §1 · pp. 2–3
• The paper treats representation compatibility as a factor in adaptation and studies how representation choice affects performance across tasks.It characterizes compatibility through controlled experiments and links it to state-understanding and cross-representation transfer effects.Source: §1 · pp. 2–3
• Auxiliary supervision can provide gains beyond simply increasing target-task data, while jointly training on the target and environment-derived objectives.The approach includes tasks predicting state properties, transitions, and action constraints.Source: §1 · pp. 2–3

2 Background and problem setup

• A target task maps an underlying state to an action, and a representation maps that state to an input sequence.Fine-tuning is framed as learning the conditional distribution p(a | s).Source: §2 · p. 3
• Representation compatibility is assessed through downstream performance across representations and the model’s reasoning behavior.A qualitative indicator is whether the model transforms one representation into another; the mechanisms behind representation differences are beyond this work’s scope.Source: §2 · p. 3
• In chess, Qwen3-8B converts FEN into ASCII before reasoning, while ASCII and FEN differ markedly in next-state prediction accuracy.The authors note that possible mechanisms, including pretraining data or architectural biases, are outside the study’s scope.Source: §2 · p. 3
• Auxiliary tasks cover state properties, legal actions, and next states, with next-state prediction modeling transitions and legal-action prediction characterizing valid actions.The tasks can use different state representations; this study instantiates the framework with offline data and supervised fine-tuning.Source: §2 · p. 3

3 Representation-aligned auxiliary supervision

• The approach adapts structured state-to-action tasks by selecting auxiliary-supervision representations using compatibility and jointly training target and environment-derived objectives.Source: §3 · p. 3

3.1 Step 1: Diagnosing representational preferences

• When target-task performance is uniformly low, a structure-preserving probe estimates compatibility; the highest-scoring representation becomes the auxiliary default, with reasoning traces as a complementary signal.Source: §3.1 · p. 4

3.2 Step 2: Constructing environment-derived auxiliary tasks

• Auxiliary tasks use computable environment information: representation-level operations, state-property prediction, and dynamics such as transitions or valid actions, all expressed in selected r*.Source: §3.2 · pp. 4–5

3.3 Step 3: Joint training

• Joint training combines target and auxiliary losses with inputs in r*, keeping the target primary and shaping the same model rather than composing independently learned skills at inference.Source: §3.3 · p. 5

4.1 Experimental setup

• The experiments use chess positions in semantically equivalent FEN and ASCII formats, with optimal-move prediction as the target and automatically derived environment tasks as auxiliary supervision.The primary dataset has 100k board-move pairs, auxiliary conditions use 200k examples, and evaluation uses 5k held-out positions.Source: §4.1 · pp. 5–6

4.2 Environment-derived supervision for target adaptation

• With 100k optimal-move examples, Qwen3-14B accuracy rises from 0.206 to 0.226 (+9.7%) in ASCII and from 0.199 to 0.223 (+12.1%) in FEN with dynamics supervision.Both comparisons use the same target-task data; OM + dynamics adds next-state and legal-move supervision.Source: §4.2 · pp. 6–7, Table 1 · p. 6
• With 100k target examples, ASCII accuracy reaches 0.226 with 200k dynamics examples, comparable to 0.222 with 300k target examples; dynamics also outperforms surface and static tasks under matched budgets.The matched-budget comparison identifies supervision content—not simply additional multitask examples—as relevant to the gains.Source: §4.2 · p. 7, Figure 3 · p. 7

4.3 Representation compatibility and transfer

• For Qwen3-14B, ASCII-trained OM + dynamics scores 0.202 on FEN evaluation, versus 0.166 for FEN-trained OM + dynamics on ASCII evaluation, and exceeds the FEN OM-only score of 0.199.The transfer asymmetry is also observed for Olmo3-7B: ASCII-trained OM + dynamics scores 0.185 on FEN, while FEN-trained OM + dynamics scores 0.077 on ASCII.Source: §4.3 · p. 7
• For Qwen3-8B state understanding, ASCII→ASCII reaches 93.4% exact match versus 61.3% for FEN→FEN, indicating larger representation effects than on in-distribution move prediction.The paper reports broadly similar optimal-move performance across the two in-distribution representations.Source: §4.3 · pp. 7–8, Table 2 · p. 8

4.4 Additional analyses

• In next-state prediction, longer reasoning improves performance under FEN but degrades it under ASCII; using ASCII between FEN input and output also improves on direct FEN→FEN.These comparisons indicate that representation choice affects both useful reasoning length and intermediate representations for state transformations.Source: §4.4 · p. 8
• Next-state and legal-move tasks each modestly improve optimal-move prediction, while their combination yields larger, more consistent gains across representations, especially with ASCII training.The combined tasks provide supervision about both environment dynamics and action constraints.Source: §4.4 · p. 8, Figure 4 · p. 8
• Auxiliary warm-up also improves factual commentary generation: ASCII warm-up followed by ASCII commentary training yields 51.0% claim accuracy and a 27.9% error rate.For commentary trained and evaluated in FEN, ASCII warm-up improves claim accuracy by 10.3 percentage points over no warm-up; FEN warm-up does not improve performance in ASCII.Source: §4.4 · pp. 8–9, Table 3 · p. 9

• Prior chess work spans state tracking [11], policy-language training [12], engine distillation [13], strategy-aware fine-tuning [14], and commentary generation [15].Source: §5 · p. 9
• Other chess studies examine strategic reasoning with RL [16] [17] and planning with learned world models [18].Source: §5 · p. 9
• Representation effects have been used mainly through inference-time prompting or representation design [4] [5] [7] [20], whereas this work uses compatibility to select how auxiliary supervision is expressed during training.Source: §5 · p. 9
• In contrast to RL work using environment-derived supervision to develop richer representations [3] [21] [22], this work studies which signals and representations support language-model adaptation.Source: §5 · p. 9

6 Conclusion

• Dynamics-oriented supervision, including state transitions and action constraints, provides particularly strong gains, while representation choice especially affects state understanding and cross-representation transfer.Source: §6 · p. 9
• Jointly learning environment structure and target behavior may matter: independently learning auxiliary skills did not fully recover the benefits.Source: §6 · p. 9

A.1 Data construction

• The experiments use Lichess Evaluations Database positions analyzed by Stockfish, with task-specific answers and synthetic reasoning traces.Source: §A.1 · p. 12
• Surface tasks manipulate board strings without requiring chess-specific knowledge such as piece identity, move legality, or board geometry.Source: §A.1 · p. 12
• Static tasks query position facts, including piece counts, material balance, piece lists, and which side has more queens and rooks.Source: §A.1 · p. 12
• Legal-move training oversamples positions involving constraints such as checks, pinned pieces, promotion, and en passant to cover these cases.Source: §A.1 · p. 12

• Table 4 summarizes the supervised fine-tuning hyperparameters.Source: Table 4 · p. 12, §A.2 · p. 12
• Most experiments completed within one day on four NVIDIA H100 GPUs, although training time varied across data mixtures.Source: §A.2 · p. 12
• Training used AdamW with β1 = 0.9 and β2 = 0.95, with FlashAttention 2 enabled for all runs.Source: §A.2 · p. 12

B.1 Win probability loss evaluation

• WPL is the absolute win-probability difference between the optimal Stockfish move and the model-selected move, complementing coarse exact-match accuracy.Source: §B.1 · p. 13
• Dynamics supervision reduces mean and median WPL in-distribution for both FEN and ASCII, while ASCII-trained dynamics supervision has lower WPL in FEN than FEN-trained supervision has in ASCII.Source: §B.1 · p. 13

B.2 Mixture of representations

• The mixture experiments compare FEN→FEN, ASCII→ASCII, and FEN→ASCII→FEN training, with additional explicit representation-mapping tasks.Source: §B.2 · p. 13
• 0.016 EM on FEN→ASCII→FEN for FEN + ASCII training, despite strong single-representation performance; explicit mixed-task training improves that task.Source: §B.2 · pp. 13–14, Table 6 · p. 14
• Explicit FEN↔ASCII conversion supervision further improves performance across the evaluation settings.Source: Table 6 · p. 14, §B.2 · p. 14

B.3 Limits of sub-skill composition

• The composition experiment trained conversion and representation-specific reasoning skills separately, then tested whether the model could compose them at inference time.Source: §B.3 · p. 14
• Models trained on constituent skills achieved near-zero exact match on the full FEN→ASCII→FEN task, and conversion skills did not induce ASCII intermediate reasoning.Source: Table 7 · p. 14, §B.3 · p. 14

C.1 Data curation

• Filtering reduced about 25k collected annotations to 10k, then atom-based selection produced 5k training examples.Annotations contradicting engine evaluations were discarded, and examples containing only a single atom were removed.Source: §C.1 · p. 14
• In the example, tool checks verify only one of three sub-claims: queen centralization is correct, but the claimed attack on b4 and e-file infiltration are incorrect.The attack check does not include b4, and the e-file is blocked by Black's e5 pawn.Source: §C.1 · p. 15
• Each training example pairs atoms with fluent commentary, while evaluation uses 50 examples curated by expert human players.Atoms capture single logical units such as tactical consequences, positional assessments, or comparisons to alternatives.Source: §C.1 · pp. 14–15

C.2 Tool-augmented LLM judge

• The judge has two stages: atomic claim decomposition followed by tool-augmented verification.Atoms are divided into independently verifiable sub-claims and tagged to guide verification.Source: §C.2 · p. 15
• A candidate atom is correct only when every sub-claim is verified using deterministic chess-tool outputs rather than the judge's internal chess knowledge alone.The 18-tool suite covers positional queries, attack and defense relations, board geometry, variation legality and resulting positions, and engine analysis; tools use FEN and return structured JSON.Source: Table 8 · p. 15, §C.2 · p. 16

C.3 Error analysis

• The error analysis distinguishes static board-comprehension, multi-step legality, tactical-consequence, and move-quality errors.Figure 5 presents the distribution of these error types across six models.Source: Figure 5 · p. 16, §C.3 · p. 16
• Commentary fine-tuning substantially reduces board-comprehension and tactical errors, while ASCII-aligned auxiliary supervision coincides with more illegal sequences than the base model.The authors associate the latter shift with more extensive move continuations and identify longer-sequence board tracking as a remaining limitation.Source: §C.3 · p. 16

D.1 Optimal-move prediction

• In the illustrated position, the model selects g4 over Bh6 after estimating advantages of about +0.73 and +0.46 pawns, respectively.The model's stated win-rate estimates are 56.68% for g4 and 54.22% for Bh6.Source: §D.1 · p. 17

D.2 Surface task

• The surface task asks the model to copy a specified rank field from a FEN position; the example returns 4r3 for rank 2.The example identifies rank 2 as the seventh rank field in the board string.Source: §D.2 · p. 17

D.3 Static state task

• The static-state example asks which side has more material using standard piece values, and the response reports equal material: 38 for each side..Source: §D.3 · p. 18

D.4 Next-state prediction

• For next-state prediction, the example applies Nxe5 by moving White's knight from f3 to e5 and outputs the updated position.The FEN output is r2qkbnr/ppp1pppp/8/3PN3/6b1/2N5/PPP2PPP/R1BQKB1R.Source: §D.4 · pp. 18–19

D.5 Legal-move prediction

• The legal-move example examines the bishop on e3 and selects Bf4 as a move that improves bishop activity.The listed available diagonal squares include f4, g5, h6, f2, and g1.Source: §D.5 · p. 23

D.6 Representation mapping

• The representation-mapping example converts a FEN position to ASCII by expanding each rank into its board squares.The passage states that uppercase letters represent White pieces, lowercase letters Black pieces, and numbers consecutive empty squares.Source: §D.6 · pp. 23–24

E Limitations

• The study focuses on chess, so generalization to less structured settings remains to be evaluated; compatibility is assessed with empirical proxies rather than a formal measure.The framework also assumes environment-derived auxiliary supervision, whose construction may require additional modeling effort when environment structure is absent.Source: E Limitations · p. 24

F Broader impacts

• Improved adaptation does not guarantee robustness or factual reliability, especially when auxiliary supervision or the environment is imperfectly specified..Source: F Broader impacts · p. 24