PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Collaboration between a language and a non-language agent

Harini S, Somesh Singh, Yaman K Singla, Rajiv Ratn Shah, David Doermann, Balaji Krishnamurthy

arXiv:2609.00474v2cs.CLcs.AI

TL;DR

The paper asks whether verbalizing non-language agents creates a bottleneck for LLM collaboration. It introduces latent state internalization, which projects agent representations into the LLM token stream, and evaluates it with a six-task chess benchmark. LLAMIA-14B matches or exceeds specialists and frontier models, while verbalization debt widens with interaction depth and persists across the evaluated scale range.

  • Problem

    Existing LLM–agent interfaces assume that non-language agents’ rich internal representations can be faithfully verbalized, although important collaborative tasks require information beyond text summaries.

  • Method

    The paper introduces latent state internalization, dynamically projecting a non-language agent’s continuous representations into latent tokens interleaved with language and action tokens, and trains LLAMIA through projector alignment followed by DAPO.

  • Results

    LLAMIA-14B matches or exceeds task specialists and frontier models across all six LLAMIA-BENCH tasks, while the verbalization performance gap widens with interaction depth and resists closing across the evaluated scale and RL-budget ranges.

  • Takeaways & Limitations

    Direct access to latent agent state supports richer multi-step collaboration than verbalized integration within the evaluated benchmark and settings.

  • Takeaways & Limitations

    The evidence is drawn primarily from chess, with only behavior-cloning evidence on Go, and the method requires access to the agent’s internal activations.

Abstract

from arXiv · show

LLMs are increasingly deployed as orchestrators that coordinate specialized subagents to solve complex tasks through natural language. However, in many important domains like game playing and robotics, the strongest available agents are not language models. Integrating non-language agents with LLMs would require \emph{verbalization}: compressing their rich continuous representations into sparse textual summaries at each interaction step. To study whether verbalization constitutes a bottleneck, we introduce \textsc{LLAMIA-Bench}, a suite of six diverse collaborative chess tasks spanning three facets: behavioral imitation, state assessment, and natural-language explanation. Each task instantiates a well-established chess problem that neither the LLM nor the chess engine can solve alone. To solve LLM collaboration with non-language agents, we introduce \emph{latent state internalization}, which projects the subagent's continuous representations directly into the LLM's token stream as learned state tokens, with dynamic re-encoding as actions advance the environment state. Comparing internalization to verbalized integration, our experiments reveal a consistent \emph{verbalization debt}: the performance gap widens throughout training and persists as the LLM scales from 4B to 14B parameters. A single 14B model, \textsc{LLAMIA}, trained with latent state internalization, matches or exceeds task specialists and frontier models including GPT-5.1 with tool access across all benchmark tasks, and generalizes out-of-distribution where task-specific finetunes collapse

1 INTRODUCTION

The paper studies how LLMs can collaborate with non-language agents when natural-language verbalization may discard useful continuous representations. It introduces latent state internalization and LLAMIA-Bench, then reports stronger collaboration and a widening verbalization debt.

  • Motivation: LLM–non-language-agent collaboration is difficult because existing interfaces verbalize rich subagent representations into natural-language summaries.The paper argues that chess-engine representations contain positional structures, long-range tactical motifs, and learned look-ahead that text cannot faithfully preserve.
  • Method: Latent state internalization interleaves language, action, and projected latent-state tokens in one reasoning trace.The LLM requests state evaluations on demand, and a LatentBridge maps subagent activations into k=32 tokens that are dynamically re-encoded after state transitions.
  • Method: LLAMIA trains this interface through supervised projector alignment followed by end-to-end reinforcement learning with DAPO.The subagent remains frozen while the projector and LLM are trained for collaboration.
  • Results: LLAMIA consistently outperforms its verbalized counterpart, with the performance gap widening on deeper multi-step collaboration tasks.The paper calls this gap verbalization debt and identifies verbalization as a bottleneck when non-language agents are used as tools.
  • Evaluation: LLAMIA-Bench is a curated suite of six chess tasks covering behavior cloning, puzzle understanding, commentary, and planning.The benchmark uses chess because it offers diverse tasks, verifiable metrics, strong pretrained agents, and established evaluation protocols.
  • Results: A single LLAMIA-14B matches or exceeds task specialists and frontier models across all six benchmark tasks.Figure 1 reports LLAMIA as the only system scoring on Puzzle Interest, where the engine signal lacks a text surrogate.

2 METHODOLOGY

The method lets an LLM collaborate with a chess engine through tool-mediated state queries, comparing text-only verbalization with learned latent state tokens. Training aligns the projection before jointly optimizing the LLM and projector with reinforcement learning.

  • 2.1 FORMULATION: The LLM queries a pretrained chess engine about current or hypothetical positions through get_policy while language and actions evolve the environment.The engine API also exposes board state, legal moves, and move execution; get_policy supplies the subagent output studied here.
  • 2.1 FORMULATION: Latent state internalization projects each queried engine state into k=32 continuous tokens that are dynamically re-encoded after state transitions.The tokens are appended alongside language and action tokens, allowing the LLM to attend jointly to all three trace components.
  • 2.1 FORMULATION: Verbalization serializes engine outputs as text containing top moves, prior probabilities, and value estimates, while discarding richer latent structure.Both integration variants receive the textual response, but only LLAMIA also receives projected continuous state tokens.
  • 2.2 INTERNALIZATION: LatentBridge maps the engine’s 1024-dimensional latent state into 32 embeddings matching the LLM hidden size.The implementation uses a three-layer GeLU MLP and the engine’s penultimate layer; the token count was selected where performance saturated in a sweep.
  • 2.3 TRAINING: Training first aligns the projector with the LLM frozen, then jointly trains the LLM and projector with DAPO while keeping the subagent frozen.Stage 1 uses self-play state–policy pairs and cross-entropy; Stage 2 masks injected state-token positions and optimizes generated language and action positions.
  • 2.3 TRAINING: The reinforcement-learning objective uses task-specific scalar rewards, and joint optimization learns both what representation to present and when to query the subagent.The tool call is included in the policy’s action space, allowing query timing to be learned end-to-end.

3 EXPERIMENTS AND RESULTS

The experiments evaluate LLAMIA across six collaborative chess tasks and find that latent internalization consistently outperforms verbalized integration, including on out-of-distribution and human-centered evaluations.

  • Main results: LLAMIA-14B achieves the highest score on all six LLAMIA-BENCH tasks and remains competitive with dedicated task-specific finetunes.It stays inside the expert band on in-distribution Maia and surpasses the strongest expert on OOD Wild splits.
  • Main results: Every verbalized system scores ≤12 on Interest, whereas LLAMIA-14B reaches 52, exposing a task signal unavailable through text.Puzzle Interest depends on policy distributions and value gradients across candidate moves.
  • Verbalization debt: The verbalization debt widens during DAPO, reaches 2–3× by convergence, and persists from 4B to 14B.Verbalized reward is partially useful for behavior cloning and difficulty but stays near-flat on interest and commentary.
  • Ablations: Ablations attribute the gain to Lc0’s latent policy and value representations rather than reinforcement learning, model capacity, or token count.Shuffling latent tokens collapses performance toward LLAMIA-Verb, while LLAMIA-SFT recovers most of the debt.
  • Collaboration behavior: LLAMIA adapts collaboration strategy by task, while LLAMIA-Verb uses engine-follow across tasks and requires more subagent calls.At 14B, calls average 1.9 for LLAMIA versus 2.9 for LLAMIA-Verb, with comparable or lower inference cost.
  • Human evaluation: Human evaluations find LLAMIA difficult to distinguish from humans, comparable to Maia* in human-likeness, and preferred for commentary insight.The same latent state is interpreted differently under different Elo instructions.
  • Cross-domain transfer: LLAMIA-Go-14B achieves 48/50 top-1 human move-match at ranks 5k/5d, providing initial evidence of transfer beyond chess.The Go system uses KataGo latent states with the same two-stage DAPO procedure.

4 RELATED WORK

Prior LLM-agent systems primarily communicate through natural language, while related latent or cross-modal approaches do not internalize pretrained specialists’ policy and value representations.

  • Text-mediated collaboration: Text-mediated systems such as ReAct, Toolformer, AutoGen, and HuggingGPT route communication through natural language.This can compress pretrained neural agents’ policy and value representations into only a few tokens.
  • Adjacent approaches: Multi-agent reinforcement learning and homogeneous LLM populations assume jointly trained or homogeneous agents rather than frozen LLMs paired with frozen specialists.Cross-modal systems such as PaLM-E and RT-2 project raw observations, not processed policy/value representations.

5 CONCLUSION

The paper introduces latent state internalization for LLM–agent communication and reports broad performance advantages over verbalization that persist with scale and interaction depth.

  • Contribution: Latent state internalization projects a non-language agent’s continuous representations directly into the LLM’s embedding space.LLAMIA-14B matches or exceeds dedicated task finetunes across LLAMIA-BENCH after projector alignment and DAPO.
  • Conclusion: The verbalization debt widens with interaction depth and on signals that resist text serialization, and does not close with LLM scale or RL budget in the evaluated range.Puzzle interest is given as an example of a signal that resists verbalization.

A.1 TRAINING AND INFERENCE COST

LLAMIA adds modest training overhead while maintaining comparable or lower average inference cost than verbalized integration. Its implementation uses a two-stage training pipeline, a 32-token LatentBridge, and distributed GPU infrastructure.

  • Training cost: Training cost stays within ∼6% of the verbalized pipeline at every scale.At 14B, LatentBridge and Stage-1 alignment add roughly 1–2 GPU-hours.
  • Inference cost: A LLAMIA call uses 182 total tokens, including 32 latent tokens, versus approximately 150 tokens for a verbalized call.LLAMIA invokes the subagent less frequently, offsetting the per-invocation token increase.
  • Inference cost: Latent internalization yields comparable or lower average tokens-per-query and wall-clock latency than verbalized integration.The two interfaces learn different subagent invocation rates during DAPO.
  • Training pipeline: Training has two stages: frozen-LLM projector alignment followed by joint DAPO reinforcement learning, with the subagent frozen throughout.The three-layer LatentBridge maps subagent residual representations into 32 LLM-sized latent tokens anchored by a special state token.
  • Infrastructure: Model and optimizer states are partitioned across 4 nodes of 8 A100 GPUs using PyTorch FSDP and Ray.The rollout vLLM instance and lc0 server fleet share the same GPUs.

B DATASETS & BENCHMARKS

LLAMIA-Bench uses chess to study collaboration between language models and non-language agents across diverse, verifiable tasks. The benchmark spans behavioral imitation, state assessment, comparative explanation, and game-level narrative integration.

  • Task coverage: The benchmark’s themes include behavioral imitation, state assessment, comparative explanation, and game-level commentary.These tasks progressively increase demands on communication, from single-position queries to game-length narratives.
  • Task coverage: LLAMIA-Bench includes signals with partial textual correlates, such as move quality, and signals without straightforward verbal correlates, such as aesthetic interest.Chess supports these tasks through interpretable subagent representations, public databases, and established benchmarks.

B.1 DATASET & METRICS

The benchmark combines established and newly constructed chess datasets with task-specific metrics and out-of-distribution splits. Its commentary task adds a large narrated-game resource designed for coherent multi-turn evaluation.

  • Evaluation protocol: OOD splits shift or exclude the Stage-2 training distribution, testing generalization from internalized representations rather than memorization.Where fixed test splits are unavailable, the evaluation uses random held-out samples.
  • Commentary: Agadmator-2K provides 1,900 narrated games totaling approximately 500 hours for game-level commentary.The task requires coherent narratives across evolving initiative, pawn structure, and time-pressure themes.
  • Commentary: The commentary dataset is constructed by extracting transcripts, aligning timestamps to PGN moves, assigning referenced moves, and retaining only exact move-order matches.Whisper-v3-large and GPT-4o are used in the construction pipeline.
  • Commentary: Game-level commentary is evaluated with G-eval across relevance, completeness, clarity, and fluency, alongside BLEU-2.The judge is grounded by Lc0 engine lines and the ground-truth transcript; G-eval supplies the RL reward.
  • Behavior cloning: Behavior cloning uses move-match accuracy, while reinforcement learning uses normalized top-3 target-move rank because exact-match rewards are too sparse.The top-3 rank reward provides a dense monotone signal consistent with the evaluation objective.
  • Behavior cloning: The MAIA benchmark uses five Elo buckets and reports aggregate behavior-cloning performance as their unweighted average.The test set is player–game disjoint from training data.
  • OOD evaluation: GM-25 evaluates the top 25 historically rated grandmasters as separate OOD behavioral targets without Stage-2 data from their corpora.Generalization must come from internalized representations and cross-Elo behavioral transfer.
  • OOD evaluation: Low-Time and Elo Gap are OOD splits testing behavior under severe clock pressure and opponent skill differences.Low-Time contains 129,000 positions, while Elo Gap contains 34,000 games and 68,000 player-side instances.

B.2.3 PUZZLE UNDERSTANDING

Puzzle understanding evaluates whether LLAMIA internalizes positional representations for human-aligned difficulty and interest prediction, alongside policy-only puzzle solving and move explanation. The benchmark enforces FEN-level separation between training and test positions.

  • Puzzle understanding: Puzzle understanding uses a shared 4-million-puzzle corpus and a stratified 5,000-puzzle test set covering difficulty, tactical theme, and interest.The corpus supplies community-derived labels for difficulty and engagement.
  • Difficulty estimation: Difficulty is measured by Spearman ρ against Glicko-2 ratings derived from human solving attempts.The model receives the puzzle position and solution line and predicts normalized difficulty.
  • Interest estimation: Interest estimation uses Lichess interestingness scores from −100 to +100 and evaluates Spearman ρ on the same stratified test split.The task probes a signal with no straightforward textual correlate, and text-mediated systems collapse on it.
  • Puzzle solving: Policy-only puzzle solving counts a puzzle as solved only when every forced move in the complete solution line is produced in order.Decoding uses a single forward pass per position without search.
  • Puzzle solving: Puzzle-solving accuracy is excluded for GPT-5.1 + Lc0 because live engine delegation would make the metric uninterpretable.All systems with Lc0 access cluster in the 84–94% range, while LLAMIA’s gain over LLAMIA-Verb is modest at 2–3 percentage points.
  • Move annotation: Move annotation evaluates natural-language explanations using BLEU-2, perplexity, and primary G-eval scores across relevance, accuracy, completeness, and fluency.GPT-4o judges annotations using the board FEN, played move, and Lc0-BT4 top-3 engine lines.
  • Evaluation protocol: All benchmark test splits enforce FEN-level disjointness from projector training, reinforcement-learning rollouts, and base-LLM supervised fine-tuning data.The protocol verifies that the intersection between training FENs and each test split is empty.

C.1 FRONTIER LLMS WITH VERBALIZED TOOLS

The study compares frontier LLMs using identical verbalized chess-tool access and evaluates LLAMIA against task-specific experts and interface controls across collaborative tasks. GPT-5.1 is selected as the strongest frontier verbalized baseline, while fixed templates, probes, alternative representations, and no-engine training underperform LLAMIA.

  • Frontier baseline: GPT-5.1 achieves the highest average across task groups and becomes the frontier verbalized baseline.All frontier models use identical Lc0-BT4 tool schemas, prompts, and sampling parameters.
  • Baselines: Task-specific experts define per-task comparison ceilings, while newly introduced tasks have no established prior expert.The benchmark compares LLAMIA with the strongest published or reproducible specialist for each task.
  • Interface controls: A frozen-BT4 probe scores below text-only GPT-5 on every tested task, showing that lightweight direct decoding does not recover the full human-aligned signal.Reported probe results include BC-MAIA 14, Difficulty ρ=0.15, and Interest ρ=−0.07.
  • Interface controls: 29.4 BLEU-2 / 0.31 G-eval from a deterministic template rises only to 32.1 / 0.39 after GPT-5 rewriting, versus 37.5 / 0.55 for verbalized tools and 45.8 / 0.75 for LLAMIA.The comparison indicates that natural-language presentation of engine statistics does not account for LLAMIA’s gains.
  • Benchmark scope: Chess tasks span behavior cloning, puzzle understanding, move annotation, and game-level commentary, with behavior cloning including in-distribution and out-of-distribution splits.Behavior cloning predicts human rather than optimal moves across Elo buckets and Wild splits covering grandmaster play, low time, and rating gaps.

D.4 FULL LLAMIA-BENCH RESULTS TABLE

Across LLAMIA-Bench, latent integration is consistently more effective than verbalized integration, with the largest gains on tasks whose relevant information is poorly captured by text. The benchmark combines ranking, explanation, and collaboration metrics, while reinforcement learning is especially important for multi-step language tasks.

  • Full results: Interest improves from ρ=0.08 to ρ=0.52, Commentary from 0.40 to 0.75, Difficulty from ρ=0.45 to ρ=0.71, Rationale from 33 to 46, and Behavior Cloning from 45 to 53 on MAIA.Wild behavior cloning also rises from 39 to 49; the relative gains are task-specific.
  • Interface comparison: At fixed backbone, subagent, and DAPO recipe, replacing verbalized text with latent tokens lifts every reported task column.The controlled comparison attributes the ordering to the integration interface rather than model, data, reward, or optimization differences.
  • Metrics: Game Commentary is evaluated with G-eval for relevance, completeness, clarity, and fluency, with BLEU-2 reported as a judge-free metric.G-eval also serves as the DAPO reward.
  • Training effects: Latent SFT closes most of the verbalization gap on single-step tasks, whereas DAPO becomes load-bearing on multi-step Annotation and Commentary.On Interest, LLAMIA-SFT-14B reaches ρ=0.48 and reinforcement learning raises it to 0.52.
  • Metrics: The benchmark evaluates puzzle Difficulty and Interest using Spearman’s ρ against human-derived statistics, alongside explanation and behavior-cloning metrics.Puzzle Understanding uses approximately 1,000 Lichess puzzles and separates difficulty from interest.

E.1 EMERGENT COLLABORATION AGENCY

Latent state internalization expands collaboration from choosing when to query an agent to interpreting and composing its internal representations. Experiments connect this capability to emergent multi-step strategies, agent strength, model scale, and an effective 32-token interface.

  • Emergent collaboration agency: Internalization gives the model perception through a 32-token encoding, allowing attention to determine what aspects of the agent state matter for the current task.Verbalization supplies textual answers, whereas internalization exposes the agent’s continuous representational state.
  • Emergent collaboration agency: The five collaboration patterns are engine-follow, consult-then-override, counterfactual query, multi-step lookahead, and abstention.Figure 4 compares how systems allocate these patterns across tasks, and Figure 6 tracks their training evolution.
  • Emergent collaboration agency: Identical prompts, tools, rewards, backbones, and DAPO hyperparameters leave the integration interface as the stated difference behind divergent collaboration behavior.Neither system is explicitly rewarded or curriculum-trained for counterfactual querying or lookahead.
  • Agent strength: Stronger agents monotonically improve LLAMIA across representative tasks and the overall average, and T72 and T80 preserve the same ordering as BT4.Agent strength is measured without search, so the rating reflects information available to internalization.
  • Scale and architecture: Performance increases with both LLM capacity and agent strength, while the 4B-to-14B improvement is 2.4–3.1× larger for Transformer agents than for SE-ResNets.The ratio is largest on Interest at 3.1× and Commentary at 2.7×, and smallest on Behavior Cloning at 2.4×.
  • Projection bandwidth: Performance rises from k=4 to k=32 state tokens and changes little at k=64, making k=32 the selected bandwidth-context trade-off.The largest gains occur between k=4 and k=16.

E.4 INTERFACE ABLATIONS: LATENT-ONLY AND SHUFFLED TOKENS

Interface ablations show that latent tokens carry most of LLAMIA’s benefit, that token identity matters beyond embedding capacity, and that the approach transfers from chess to Go. The evaluation also documents the controlled design and human-study scope.

  • Latent-only ablation: Latent-only nearly matches the full system across all six tasks, with its largest residual gap on behavior cloning.Removing the returned move occasionally causes board-tracking errors, motivating retention of verbalized output.
  • Shuffled tokens: Shuffling more latent tokens monotonically degrades performance toward LLAMIA-Verb despite preserving 32 embeddings.Degradation is fastest on Interest and slowest on behavior cloning.
  • Representation analysis: Layer 14 gives the lowest Stage-1 alignment loss, while blocks 12–14 are jointly best in frozen-agent linear probes.The probes test played move, two-ply best move, puzzle difficulty, and tactical-motif presence.
  • Transfer beyond chess: LLAMIA-Go leads LLAMIA-Verb-Go at 4B, 8B, and 14B, while LLAMIA-Go-8B surpasses the verbalized 14B system.The Go setup keeps the two-stage recipe fixed and changes the specialist and task.
  • Human evaluation: Human studies recruit chess-knowledgeable participants under IRB-approved consent procedures and screen them with a ≥70% calibration-accuracy criterion.Twelve of fourteen recruited participants proceeded, with median calibration accuracy of 75%.

H.5 RESULTS

LLAMIA-14B outperforms its verbalized counterpart in human gameplay, commentary, and complex state assessment, with the largest advantages appearing when strategic information must be integrated across steps.

  • Gameplay: 39% bot-detection accuracy for LLAMIA-14B versus 72% for LLAMIA-Verb-14B placed the internalized model below chance and the verbalized model above chance.LLAMIA-14B had a 61% human-pass rate, compared with 28% for LLAMIA-Verb-14B.
  • Gameplay: Qualitative coding found stylistic cues in 65% of LLAMIA-Verb-14B segments, whereas LLAMIA-14B produced no identifiable cue in 38% of cases.Identified cues for LLAMIA-14B were distributed across tactical and positional categories without a dominant signal.
  • Gameplay: LLAMIA-14B was rated human-like by 65% of participants and preferred as an opponent by 68%, versus 42% and 44% for LLAMIA-Verb-14B.These ratings approached Maia*’s 72% human-likeness score while LLAMIA-14B received no human-move supervision.
  • Commentary: 80.2% of judged commentary pairs favored LLAMIA-14B, whose Insight score exceeded LLAMIA-Verb-14B by 1.70 points, compared with a 1.10-point Accuracy gap.The systems were comparatively close on factual accuracy, while their strategic explanatory quality diverged more strongly.
  • State annotation: ∆r = 0.41 in complex positions, where LLAMIA-14B reached r = 0.69 versus r = 0.28 for LLAMIA-Verb-14B.The gap was only ∆r = 0.03 in simple positions and increased through moderate positions.

I.1 STAGE 1: PROJECTOR ALIGNMENT DATA

Stage 1 aligns a projector with chess-engine internal states, while the broader training pipeline combines supervised alignment, task-specific reinforcement learning, and dynamic latent-state invocation during interaction.

  • Stage 1: Projector alignment: Stage 1 trains the LatentBridge projector Hφ on state–policy pairs from the Lc0-BT4 forward pass.The projector receives penultimate-layer activations hs ∈R1024 and learns to map engine representations into the LLM token space.
  • Stage 1: Projector alignment: Four question types—position evaluation, principal variation, legal moves, and brief description—are sampled uniformly to diversify projector supervision.The positions are paired with questions about the same board state rather than a single fixed output format.
  • Stage 1: Projector alignment: The 5M-position Stage 1 dataset is split by game ID into 4.5M training, 250K validation, and 250K held-out test positions.No game appears in more than one split, preventing train/test leakage across positions from the same game.
  • Stage 2: Task-specific reinforcement learning: Stage 2 applies DAPO rollouts to approximately 850K task-specific examples spanning behavior cloning, puzzle understanding, move annotation, and game commentary.The shared pipeline uses the same tool catalogue and a common <invoke> tool-call format across task families.
  • Interaction pipeline: The internalized trace interleaves language tokens, action tokens, and latent state tokens, with k=32 state tokens appended after state evaluation.The state is dynamically re-encoded after each board-mutating action, while read-only tool calls do not trigger re-encoding.
Loading 2609.00474v2…