PaperlayerView exact revision on arXiv

Source-linked AI summary

Language-based Trial and Error Falls Behind in the Era of Experience

Haoyu Wang, Guozheng Ma, Shugang Cui, Yilun Kong, Haotian Luo, Li Shen, Mengya Gao, Yichao Wu, Xiaogang Wang, Dacheng Tao

arXiv:2601.21754v3cs.AI

TL;DR

LLMs remain limited on unseen non-linguistic environments because exploration through high-dimensional token generation makes learning environmental dynamics computationally expensive. SCOUT assigns exploration to lightweight scouts, distills their trajectories into the LLM, and applies multi-turn RL; Qwen2.5-3B-Instruct reaches 0.86 average score versus 0.60 for Gemini-2.5-Pro.

  • Problem

    LLMs struggle on unseen non-linguistic tasks because they must learn environmental dynamics from scratch while exploring through computationally expensive, high-dimensional token generation.

  • Method

    SCOUT decouples exploration from exploitation by using lightweight scouts to learn environmental dynamics, textualizing their trajectories for SFT, and refining the LLM with multi-turn RL.

  • Results

    0.86 average score is achieved by Qwen2.5-3B-Instruct with SCOUT, compared with 0.60 for Gemini-2.5-Pro, while SCOUT outperforms tested baselines across symbolic and spatial tasks.

  • Takeaways & Limitations

    SCOUT indicates that lightweight sub-scale exploration can let smaller LLMs efficiently exploit task-specific dynamics on unseen environments.

  • Takeaways & Limitations

    Experiments cover models from 0.5B to 3B and mainly use multi-turn PPO; larger models, other model types, and alternative RL algorithms remain unvalidated.

Abstract

from arXiv · show

While Large Language Models (LLMs) excel in language-based agentic tasks, their applicability to unseen, nonlinguistic environments (e.g., symbolic or spatial tasks) remains limited. Previous work attributes this performance gap to the mismatch between the pretraining distribution and the testing distribution. In this work, we demonstrate the primary bottleneck is the prohibitive cost of exploration: mastering these tasks requires extensive trial-and-error, which is computationally unsustainable for parameter-heavy LLMs operating in a high dimensional semantic space. To address this, we propose SCOUT (Sub-Scale Collaboration On Unseen Tasks), a novel framework that decouples exploration from exploitation. We employ lightweight "scouts" (e.g., small MLPs) to probe environmental dynamics at a speed and scale far exceeding LLMs. The collected trajectories are utilized to bootstrap the LLM via Supervised Fine-Tuning (SFT), followed by multi-turn Reinforcement Learning (RL) to activate its latent world knowledge. Empirically, SCOUT enables a Qwen2.5-3B-Instruct model to achieve an average score of 0.86, significantly outperforming proprietary models, including Gemini-2.5-Pro (0.60), while saving about 60% GPU hours consumption.

1. Introduction

SCOUT addresses the difficulty of unseen non-linguistic tasks by separating efficient environmental exploration from LLM exploitation. Lightweight scouts generate trajectories that warm up LLMs, which then refine their capabilities through multi-turn reinforcement learning.

  • SCOUT framework: The generated trajectories are textualized and used for supervised fine-tuning, after which multi-turn reinforcement learning aligns the LLM with the unseen task.This pipeline transfers scout-learned task dynamics into the LLM and lets it focus on exploiting the newly learned knowledge.
  • Motivation: Unseen spatial, symbolic, and long-horizon tasks require LLMs to internalize environmental dynamics rather than directly use pretrained world knowledge.Existing pretraining is insufficient because real-world environments involve unbounded complexity.
  • Motivation: LLM exploration is inefficient because each action requires a forward pass through billions of parameters while the model searches a vocabulary exceeding 30,000 tokens.This high-dimensional semantic search is wasteful for tasks with small, discrete action spaces.
  • SCOUT framework: SCOUT decouples exploration from exploitation by using lightweight scouts, such as small MLPs or CNNs, to learn task dynamics and generate expert trajectories.Their low parameter count and high inference speed support rapid reinforcement-learning-based exploration.
  • Results: Across symbolic and dynamic worlds, SCOUT enables Qwen2.5-3B-Instruct to reach an average score of 0.86, versus 0.60 for Gemini-2.5-Pro.The evaluated tasks include FrozenLake, Sokoban, Sudoku, 2048, and Rubik’s Cube.

2. Sub-Scale Collaboration On Unseen Tasks

SCOUT separates environmental exploration from LLM exploitation by training lightweight scouts, converting their trajectories into language dialogues, and then applying SFT and multi-turn RL to the LLM.

  • 2.1. Preliminaries: SCOUT models LLMs and scouts with separate MDPs, giving LLMs language-augmented context while scouts learn directly from symbolic states.Scouts infer transition dynamics through extensive trial-and-error without linguistic descriptors.
  • 2.3. Distillation Stage: A trajectory transformation function converts symbolic scout experiences into multi-turn language dialogues, enabling SFT to teach the LLM unseen task dynamics.The transformed dataset leaves thought content blank and grounds symbolic dynamics in language descriptions.
  • 2.2. Exploration Stage: Lightweight scouts use discrete-action reinforcement learning, such as DQN or PPO, to explore unseen environments and maximize cumulative reward.Their low-dimensional parameter space and lack of token-generation overhead enable much higher interaction throughput than LLMs.
  • 2.2. Exploration Stage: After convergence, the better scout policy generates expert trajectories that form the dataset for subsequent LLM training.The trajectories capture high-reward regions and learned transition dynamics.
  • 2.4. Evolving Stage: Multi-turn RL then optimizes the LLM directly in the interactive environment, encouraging planning thoughts and refining or activating the distilled capabilities.In Sudoku, the score rises from 0.29 after distillation to 0.97 after subsequent RL.

3. Experiment Setup

The experiments evaluate SCOUT across six unseen symbolic, dynamic, long-horizon, and spatial tasks using Qwen-2.5 models and comparisons with open and proprietary baselines.

  • Models: The trained models primarily use the Qwen-2.5 series and are compared with RAGEN, State Estimation RL, SPA, and proprietary models.The comparison includes GPT-4o-mini, DeepSeek-V3, GPT-OSS-120B, GPT-5-nano, and Gemini-2.5-Pro.
  • Tasks: The study introduces six tasks: Bandit, FrozenLake, Sokoban, Sudoku, 2048, and Rubiks’ Cube.The suite includes long-horizon symbolic and symbolic spatial environments.
  • Tasks: Task difficulty varies through environmental dynamics, puzzle complexity, rotation count, and the need for spatial imagination.FrozenLake can become slippery, Sokoban can use more boxes, and Rubiks’ Cube can use more rotations.
  • Training Settings: Training uses 200 multi-turn PPO steps, LLaMA-Factory for SFT, and CleanRL as a reference implementation for scout training.These settings cover the evolving, distillation, and exploration stages respectively.
  • Evaluation: Evaluation uses RAGEN’s default codebase for both local and API-accessed models to support a common comparison procedure.The same codebase evaluates trained LLMs and proprietary models accessed through APIs.

4. Experimental Results and Findings

Across symbolic, spatial, long-horizon, and sequential multi-task settings, SCOUT combines lightweight scouts with LLM adaptation to improve performance, efficiency, robustness, and retention of prior skills.

  • Scout Comparison: Scout-DQN achieves superior or equal best performance in 4 of 10 detailed tasks, including 1024 versus PPO’s 512 in one task.Scout-DQN ties in 2 tasks and trails in 4, while Scout-PPO remains competitive on FrozenLake Slippery and Sudoku.
  • Exploration Efficiency: Multi-turn PPO activates capabilities learned from scout trajectories, allowing SCOUT language models to surpass their scout teachers.The results attribute the main bottleneck to inefficient initial exploration rather than insufficient reasoning capacity.
  • GPU Cost Analysis: 24.0 GPU hours for Direct PPO versus 9.6 GPU hours for SCOUT represents an approximately 60% reduction on Rubiks’ Cube Rotation3.SCOUT delegates initial exploration to a lightweight scout and uses GPU resources for knowledge transfer and activation.
  • Robustness Study: With a sub-optimal scout, SCOUT improves Rubiks’ Cube Rotation3 from 0.34 to 0.60 and Sudoku from 0.55 to 0.98.The study reports that subsequent evolution can exceed the performance of the low-quality teacher.
  • Multi-task Sequential RL: In sequential multi-task RL, SCOUT reaches an average score of 0.91 while preserving Bandit at 1.0 and FrozenLake at 0.89 after Sudoku training.Direct Sequential RL improves only from 0.19 to 0.37 after five tasks, whereas SCOUT retains prior skills while learning new ones.

5. Conclusion

SCOUT addresses exploration inefficiency and dimension mismatch in unseen, non-linguistic tasks by combining lightweight scout exploration with LLM reasoning and multi-turn RL. Across symbolic and spatial tasks, it outperforms existing baselines.

  • SCOUT targets exploration inefficiency and dimension mismatch as key barriers to LLM agents mastering unseen, non-linguistic tasks.
  • SCOUT decouples exploration from exploitation by combining lightweight scout networks with the reasoning capabilities of LLMs.
  • Scout trajectories are distilled into LLMs before multi-turn reinforcement learning evolves them on unseen tasks.
  • SCOUT significantly outperforms existing baselines across symbolic and spatial tasks, including long-horizon 2048 and Rubiks’ Cube challenges.

Impact Statement

The paper presents SCOUT as an efficiency-oriented framework for unseen environments. It links this approach to reduced computational demands and broader access to capable agents, while noting standard ethical monitoring for deployment.

  • SCOUT improves the efficiency of LLM agents in unseen environments by offloading computationally expensive exploration to lightweight neural networks.
  • The approach reduces the energy footprint required to train competent agents, with positive implications for Green AI.
  • The paper reports that smaller models such as 3B-parameter systems can outperform larger proprietary models through effective collaboration.
  • The authors state that autonomous-agent deployment warrants standard ethical monitoring in real-world automated systems.

B. Limitations

The evaluation covers model sizes from 0.5B to 3B and Qwen and LLaMA model types, but does not establish performance for larger or other model families. Multi-turn PPO is the main RL algorithm, and some tasks degrade after further training.

  • The experiments validate SCOUT from 0.5B to 3B across Qwen and LLaMA models, leaving larger models and other model types untested.
  • The study mainly uses multi-turn PPO, while the effectiveness of alternative RL algorithms such as GRPO remains unvalidated here.
  • Some tasks show performance degradation after several RL training steps, making stabilization of multi-turn RL an essential area for improvement.
  • The related-work discussion situates SCOUT among LLM-environment interaction, deep RL exploration, and large-small model collaboration approaches.

D.1. Models, Datasets, Tasks

The evaluation spans symbolic, spatial, stochastic, planning, combinatorial, and long-horizon tasks using small neural scouts and multiple LLM baselines. The task suite is designed to represent out-of-distribution environments for language agents.

  • Models: SCOUT uses Qwen2.5 instruct models from 0.5B to 3B and supplements them with LLaMA3.1-1B-It to test different model types.
  • Models: The baselines include GPT-4o-mini, DeepSeek-V3, Gemini-2.5-Pro, GPT-5-nano, and GPT-OSS-120B.
  • Models: MLPs are used for Bandit, 2048, FrozenLake, Rubiks’ Cube, and Sudoku, while CNNs are used for Sokoban; these scouts are about 1.0 × 10^-5 B parameters.
  • Tasks: The study focuses on out-of-distribution symbolic and spatial tasks composed mainly of symbols or numbers and introduces 2048 and Rubiks’ Cube.
  • Tasks: The task suite includes Bandit, FrozenLake, Sokoban, Sudoku, 2048, and Rubik’s Cube, covering reward selection, navigation, planning, constraint satisfaction, long horizons, and spatial restoration.
  • Tasks: Table 8 compares average state-representation perplexity for symbolic tasks with standard language-based agent tasks to quantify distribution shift.

D.2. Experiment Settings

The experiments use supervised fine-tuning and multi-turn PPO, with specified training configurations for each stage.

  • SFT uses LLaMA-Factory with a 4096 cutoff length, batch size 64, three epochs, cosine scheduling, and a 0.1 warm-up ratio.Full fine-tuning uses learning rate 1e-5 on 8 H100 devices.
  • Multi-turn PPO uses RAGEN with a 0.28 clip ratio and 0.25 rollout filter ratio.Checkpoints are trained for 200 steps with maximum model length 16384.
  • The PPO setup requests one action per turn and permits up to 25 turns, except for 2048, which uses up to 1k turns.The 2048 setting also includes an in-context sliding window with 5 dia.

E. Extra Experimental Results

The extra-results section reports detailed scout curves, sequential-RL results, task prompts, and state-estimation prompts across the unseen-task environments.

  • Extra Experimental Results: Detailed Scout-DQN and Scout-PPO results cover six unseen tasks, while sequential-RL results are reported in Table 9.The sequential-RL results correspond to Figure 3 in the main context.
  • System Prompts: The task prompts specify operational rules and output formats for Bandit, 2048, FrozenLake, Sokoban, Rubik’s Cube, and Sudoku.They describe actions, observations, puzzle objectives, and required think/answer formatting.
  • System Prompts: FrozenLake includes separate static and slippery settings, with the latter warning that the agent may slide into an unintended tile.Both prompts represent observations with grids and zero-indexed coordinates.
  • State Estimation Prompts: The 2048 example predicts that sliding Right merges equal tiles and creates a 4 at position (0,3).The example starts with 2s at (0,0) and (0,1), plus a 4 at (3,3).
  • State Estimation Prompts: The Rubik’s Cube prompt requires predicting how a rotation changes face rows before producing an action.The example predicts the resulting front and right top rows after rotating U clockwise.

G.1. Scout Architecture

The scout architectures include MLP and CNN variants, and a Textualizer converts scout trajectories into language trajectories through deterministic serialization.

  • Scout-DQN Architecture: Scout-DQN provides MLP and CNN architectures, including a dueling convolutional Q-network variant.The CNN uses convolutional layers followed by a fully connected layer with 512 units.
  • Scout-PPO Architecture: Scout-PPO provides MLP and CNN agent architectures with separate actor-network components.The MLP implementation uses repeated linear layers, while the CNN implementation includes convolutional layers and an actor output layer.
  • Textualizer: The Textualizer transforms scout trajectories Dscout into language-based trajectories DLLM by expressing states, feedback, and actions as natural-language descriptions.Scouts interact directly with the original Gym-style environments.
  • Textualizer: The transformation deterministically substitutes symbolic states and actions into canonical task templates without adding task structure, transition rules, or planning heuristics.Table 11 illustrates the mapping for Sokoban.
Loading 2601.21754v3…