PaperlayerView exact revision on arXiv

Source-linked AI summary

Expanding the Action Space of LLMs to Reason Beyond Language

Zhongqi Yue, Weishi Wang, Yundaichuan Zhan, Juncheng Li, Daniel Dahlmeier, Fredrik D. Johansson

arXiv:2510.07581v2cs.LG

TL;DR

LLM interactions with external environments commonly rely on predefined text patterns and external parsers. ExpA gives models direct routing and environment-specific actions, while EARL uses counterfactual rollouts to promote exploration; it outperforms vocabulary-constrained baselines on multi-turn tasks and discovers an efficient four-element sorting algorithm.

  • Problem

    External-environment actions are often encoded as predefined text patterns that require parsing and routing, coupling interaction to language reasoning.

    Source: §1 · p. 2, §3 · p. 4

  • Method

    ExpA adds direct routing and environment-specific actions, and EARL uses counterfactual rollouts to encourage exploration of rarely invoked interactions.

    Source: §1 · p. 2

  • Results

    EARL outperforms vocabulary-constrained baselines on multi-turn tasks, including a 26.3% gain on Countdown.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    In partially observed sorting, the method achieves perfect Sort-4 accuracy and discovers an efficient algorithm.

    Source: §1 · p. 2, §6 · p. 10

    Evaluation is limited to the Qwen-2.5 family up to 7B parameters; scaling properties remain unknown, and complex multimodal environments are underexplored.

    Source: A LIMITATIONS AND FUTURE DISCUSSIONS · p. 17

Abstract

from arXiv · show

Large Language Models (LLMs) are powerful reasoners in natural language, but their actions are typically confined to outputting vocabulary tokens. As a result, interactions with external environments -- such as symbolic operators or simulators -- must be expressed through text in predefined formats, parsed, and routed to external interfaces. This overloads the model's language with both reasoning and control duties, and requires a hand-crafted parser, external to the LLM. To address this, we decouple environment interactions from language by internalizing them in an Expanded Action space (ExpA), beyond the vocabulary. The model starts reasoning in the default language environment, but may trigger routing actions and switch to an external environment at any time. From there, the model can only invoke environment-specific actions, receive feedback from the environment, and potentially route back to language as a result. To promote effective exploration of the expanded action space and new environments, we introduce ExpA Reinforcement Learning (EARL) with counterfactual policy optimization. On tasks requiring multi-turn interactions and contingent planning, EARL outperforms strong baselines with vocabulary-constrained actions. It performs robustly across calculator-based multi-task learning and, in the partially observed sorting problem, achieves perfect Sort-4 accuracy while self-discovering an efficient algorithm competitive with classical designs.

1 INTRODUCTION

ExpA separates language reasoning from environment interaction through direct routing and environment-specific actions. EARL uses counterfactual rollouts to encourage exploration, and the paper reports gains over vocabulary-constrained baselines on interactive tasks.

1 INTRODUCTION

  • The approach targets end-to-end training without external parsers, supervised tool-call data, or predefined language patterns for interaction.The stated aims also include keeping interactions under the model’s control and supporting RL on base models.Source: §1 · p. 2
  • ExpA lets the model route directly to an external environment and use its actions, separating interaction from language reasoning.Environment-specific actions produce observations in vocabulary tokens before control returns to the language environment.Source: §1 · p. 2
  • EARL encourages exploration by comparing rewards from an original rollout and a counterfactual rollout that forces a plausible routing action.The forced routing point is identified where the model assigns high probability to the routing description token.Source: §1 · p. 2
  • EARL outperforms vocabulary-constrained baselines on multi-turn tasks, including a 26.3% gain on Countdown.In partially observed sorting, it achieves perfect Sort-4 accuracy while discovering an efficient algorithm.Source: §1 · p. 2

2 RELATED WORK

Prior approaches commonly express environment actions as predefined text patterns that are parsed and routed. This work instead expands LLM action spaces to support language reasoning alongside sequential interaction with multiple environments.

2 RELATED WORK

  • Many existing approaches encode task-specific actions as predefined text patterns for parsing and routing, and some require human-crafted tool-use demonstrations.Such designs rely on instruction-following and pretrained knowledge for action execution, and performance can be sensitive to prompt variations.Source: §2 · p. 3
  • Multimodal token-space expansion often uses large-scale demonstrations, while action adaptors constrain a model to learned actions for one environment.The paper distinguishes its setting as language reasoning alongside sequential interaction with multiple external environments.Source: §2 · p. 3
  • Unlike work on growing action spaces within a single environment, this paper studies LLMs reasoning in language while interacting sequentially with multiple environments.The related-work passage positions this as a more general and challenging setting.Source: §2 · p. 3

3 FORMULATION

The formulation models an LLM as acting in a language environment and one or more external environments, with external state only partially observed. ExpA adds direct routing and environment-specific actions to the available action space.

3 FORMULATION

  • The global state combines language-token history, the active environment, and external latent state; only the latent state is partially observed.The agent’s history includes language tokens and token-based descriptions of environment observations.Source: §3 · p. 3
  • The task reward may be set in the language environment, as with a calculator, or defined by an external environment, as in sorting through comparisons and swaps.The two settings differ in whether the external environment supplies the task and reward.Source: §3 · p. 3
  • Language-only interaction encodes external actions in predefined text patterns, preventing intermediate feedback from changing actions within a pattern.The paper notes this may hamper performance and credit assignment.Source: §3 · p. 4
  • ExpA adds a routing action for each external environment and environment-specific actions; observations return in language tokens, and an exit flag can route control back.Rollouts begin in the language environment, where the policy can generate a vocabulary token or route to an external environment.Source: §3 · p. 4

4 EXPA REINFORCEMENT LEARNING (EARL)

EARL trains a policy over expanded actions with counterfactual policy optimization, pairing factual trajectories with trajectories that force routing. It initializes new actions from their descriptions and adapts updates according to factual rollout rewards.

4 EXPA REINFORCEMENT LEARNING (EARL)

  • CPO optimizes factual and counterfactual rollout pairs for each initial state, with each counterfactual forcing a routing action at a plausible intermediate step.The update for each pair is conditioned on all rollouts.Source: §4 · pp. 4–5

4.1 A POLICY OVER THE EXPANDED ACTION SPACE

  • The expanded policy head scores vocabulary and environment actions, while new-action weights are initialized from descriptions to align them with pretrained linguistic counterparts.For a single-token description, the new action can use that token's pretrained weight directly.Source: §4.1 · p. 5

4.2 COUNTERFACTUAL ROLLOUTS TO ENCOURAGE INVOKING NEW ACTIONS

  • Counterfactual rollouts force routing at language-environment steps weighted by the pretrained next-token likelihood of the routing description, supporting zero-RL training.They reuse the factual rollout prefix, intervene with routing, and continue the rollout through the interaction algorithm.Source: §4.2 · p. 5

4.3 UPDATE FUNCTION

  • When all factual rollout rewards are nonpositive, EARL encourages missing interactions through counterfactual branches; otherwise, it uses group-relative advantage to exploit successful strategies.Each update applies a standard PPO-style clipped, KL-regularized rule using a trajectory and its advantage.Source: §4.3 · pp. 5–6

5 EXPERIMENT

The experiments evaluate calculator-based multi-task learning and sorting tasks that require multi-turn interaction and contingent planning. EARL performs strongly across Calc-Bench tasks and achieves perfect Sort-4 accuracy while learning an efficient sorting strategy.

5.1 EXPERIMENTAL SETUP

  • The evaluation covers Calc-Bench arithmetic, Countdown, GSM8K∗, and Count, plus hidden-number sorting through compare and swap interactions.Countdown and sorting require adapting actions to intermediate observations; sorting is a partially observed problem with penalties for excessive comparisons and swaps.Source: §5.1 · p. 6

5.2 EXPERIMENTAL RESULTS ON CALC-BENCH

  • 10.46 absolute EM gain overall and 17.55 EM gain on Countdown are the reported best gains for EARL on Calc-Bench.The benchmark jointly trains models on all Calc-Bench tasks to assess shared representation learning.Source: §5.2 · p. 7, Table 2 · p. 7
  • EARL uses a moderate number of calculator interactions but produces more planning-related language; Prompt+GRPO often resorts to brute-force trials, while CPO and SFT+GRPO can hallucinate calculator inputs.The authors suggest that SFT-learned patterns transfer poorly to diverse instances and that language reasoning and action learning may interfere in CPO.Source: §5.2 · pp. 7–8
  • On GSM8K∗, correct rollouts link fewer response tokens with stronger performance, and the two CPO methods outperform the two GRPO methods.The comparison includes EARL and Prompt+CPO versus SFT+GRPO and Prompt+GRPO.Source: Figure 4 · p. 7, §5.2 · p. 8
  • On Countdown, CPO consistently outperforms GRPO across settings and converges faster, as indicated by the CPO and GRPO training curves.The ablation compares CPO with GRPO, base with instruction-tuned models, and prompted with unprompted environments.Source: §5.2 · p. 8, Figure 5 · p. 8, Table 4 · p. 8
  • ExpA models succeed without environment prompts through weight initialization and counterfactual rollouts, while base models remain competitive where Prompt+GRPO degrades without instruction tuning.The authors identify this as potential for Zero-RL training of interactive agents and scalability to settings with many environments.Source: §5.2 · p. 8

5.3 EXPERIMENTAL RESULTS ON SORTING

  • EARL achieves perfect accuracy across Sort-4 swap levels and beats the best Sort-5 baseline by up to 21% in a stratum and more than 10% overall.Prompt+CPO accuracy declines as the required number of swaps increases.Source: §5.3 · p. 8, Figure 6 · p. 9
  • EARL and EARL* match the optimal number of swaps and closely approach the optimal number of comparisons, outperforming insertion sort and GCC’s built-in routine.EARL* prunes redundant comparisons, uses A as a pivot, and applies MIN SWAP(R) based on accumulated comparison results.Source: §5.3 · p. 9, Table 5 · p. 9

6 CONCLUSION

The conclusion presents ExpA and EARL as a framework for LLMs to interact with external environments, reporting gains on interactive tasks and an efficiently learned four-element sorting algorithm.

6 CONCLUSION

  • EARL outperforms strong baselines on multi-turn tasks, with gains in challenging contingent-planning settings and consistent improvements in multi-task and continual learning.The framework also removes reliance on external parsers and supports end-to-end training.Source: §6 · p. 9
  • “Optimal” denotes the theoretical minimum over swaps and comparisons combined, while GCC is optimized for processor efficiency rather than minimizing raw operations.Source: §6 · p. 9
  • EARL discovers an efficient four-element sorting algorithm, which the conclusion presents as a notable result of the framework.The algorithm uses A as a pivot, adds comparisons when needed, then minimizes swaps.Source: §6 · pp. 9–10

A LIMITATIONS AND FUTURE DISCUSSIONS

The paper identifies empirical and design-scope limitations, then describes the expanded-action policy and its update procedure. The policy conditions action probabilities on history and the active environment, while the update uses rollout data and PPO-style regularization.

A LIMITATIONS AND FUTURE DISCUSSIONS

  • Evaluation is limited to the Qwen-2.5 family up to 7B parameters; scaling, optimal initialization of novel actions for continual learning, and complex multimodal environments remain unexamined.Source: A LIMITATIONS AND FUTURE DISCUSSIONS · p. 17

C POLICY ON EXPANDED ACTION SPACE

  • The policy applies a linear-softmax layer to the LLM’s encoding of history and restricts the softmax to actions available in the active environment.The action set in the language environment includes routing actions and language actions; external environments use their environment-specific actions.Source: C POLICY ON EXPANDED ACTION SPACE · p. 17

D UPDATE RULE

  • The update objective uses rollout trajectories and importance-sampling ratios with PPO-style clipping and KL regularization.Rollouts are collected with the fixed old policy, while updates optimize the current policy against those sampled actions.Source: D UPDATE RULE · p. 17
  • EARL omits updates at positions corresponding to observations returned by external environments and computes KL loss over the original vocabulary V.Source: D UPDATE RULE · p. 18

E DATASET DETAILS

The appendix describes task datasets, baselines, and implementation support. Calc-Bench spans four task types, sorting uses a curriculum, and the baseline suite includes prompt-based RL and GPT-4o.

E DATASET DETAILS

  • Calc-Bench contains Arithmetic, Countdown, Count, and rewritten GSM8K tasks, generated with task-specific control over complexity, number format, and structure.Arithmetic training has 1,000 samples with up to four operations, while its 2,000-example test set includes up to six operations and more natural-language paraphrases.Source: E DATASET DETAILS · p. 18
  • Sorting training first uses simpler ordering tasks, then sorting examples spanning Sort-2 through Sort-5; the test set is split evenly between Sort-4 and Sort-5.The ordering set has 20,000 training examples; the sorting set has 80,000.Source: E DATASET DETAILS · p. 18

F BASELINE DETAILS

  • The baselines include GPT-4o prompted to generate responses, SFT+GRPO and Prompt-GRPO, and Prompt-CPO, which replaces GRPO with CPO.Countdown SFT solutions are generated with GPT-4o and retained when correct within 1K tokens.Source: F BASELINE DETAILS · pp. 18–19

G IMPLEMENTATION DETAILS

G.1 CALC-BENCH IMPLEMENTATION DETAILS

The appendix supplies calculator and sorting training configurations alongside task prompts. Its examples show how calculator access and compare/swap tools are presented to the model.

G.1 CALC-BENCH IMPLEMENTATION DETAILS

  • The Calc-Bench configurations pair calculator-enabled Prompt+GRPO runs with EARL runs using the calculator and TRPO advantage estimation.The supplied configurations include 3B and 7B Qwen-2.5-Instruct runs.Source: §G.1 · pp. 19–21

G.2 SORTING IMPLEMENTATION DETAILS

  • Sorting configurations give both Prompt+GRPO and EARL access to compare and swap tools, with GRPO and TRPO advantage estimators respectively.The shown runs use Qwen-2.5-7B-Instruct.Source: §G.2 · pp. 22–23

G.3 PROMPTS

  • The prompt examples present calculator access for Arithmetic and Countdown, compare and swap tools for sorting, and a Count-task question about digit frequency.Figure 7’s caption identifies bars as Arithmetic Operation OOD and lines as Arithmetic Digit OOD for Qwen-2.5-0.5B-Instruct.Source: §G.3 · pp. 25–26, Figure 7 · p. 27

H.1 CALC-BENCH

Calc-Bench results show EARL's learning efficiency across most tasks, with a collective-learning exception and benefits for a lightweight model; the appendix also reports robustness across environment configurations. On Sort-4, EARL induces a deterministic decision tree whose redundant comparisons can be pruned.

H.1 CALC-BENCH

  • EARL improves arithmetic for Qwen-2.5-0.5B-Instruct, matching SFT on Operation OOD and outperforming it on Digit OOD.The model has almost zero inherent arithmetic knowledge without fine-tuning; SFT gains substantially on both OOD settings.Source: §H.1 · p. 27
  • On three Calc-Bench tasks, EARL shows efficient and effective learning, supporting robust math reasoning with a better trade-off; collective tool learning shows a similar trend across methods.Figure 8 plots EM score over training steps.Source: §H.1 · pp. 27–28, Figure 8 · p. 28
  • The environment-configuration ablation reports that EARL achieves the best robustness across the tested configurations.Table 7 presents ablation EM results for different environment configurations.Source: §H.1 · p. 28, Table 7 · p. 28

H.2 SORTING

  • On Sort-4, EARL induces a deterministic decision tree under greedy decoding, and pruning its redundant comparisons yields the simplified EARL* variant.Figure 9 marks the redundant comparisons in red.Source: §H.2 · p. 28, Figure 9 · p. 29

I.1 PLANNING PHRASES ANALYSIS

The Countdown analysis finds that counterfactual policy optimization broadens planning-phrase use, while EARL makes the greatest use of diverse, conditional phrases to adapt subsequent planning.

I.1 PLANNING PHRASES ANALYSIS

  • Prompt+GRPO uses a narrow range of planning phrases, while SFT+GRPO expands phrase diversity but yields only a 6.4% overall improvement.The authors suggest supervised fine-tuning may introduce subjective biases that limit effective exploration.Source: §I.1 · p. 29
  • CPO increases planning phrases by 64.3% over SFT+GRPO and 74.8% over Prompt+GRPO.The analysis describes this as a broader range of transition options expressed in natural language.Source: §I.1 · p. 29
  • EARL has the highest planning-phrase utilization and diversity, using conditional phrases after incorrect intermediate responses to refine or restart plans.Examples include “not close,” “close to,” and “far from,” followed by a refined or different combination or approach.Source: §I.1 · p. 29

I.2 ACTION INITIALIZATION ANALYSIS

Initialization affects whether EARL-CPO learns to use tools: full initialization learns tool invocation rapidly and earns the highest rewards, while no initialization nearly settles on language-only behavior.

I.2 ACTION INITIALIZATION ANALYSIS

  • Full initialization rapidly learns tool use and achieves the highest rewards; no initialization almost never uses tools, while partial initialization is less effective.Figure 10 compares training reward and tool-use count across initialization strategies; the partial variant initializes only calculator button actions from natural-language descriptions.Source: §I.2 · p. 29, Figure 10 · p. 30

J CASE STUDY

The case studies show contrasting examples across methods on arithmetic tasks: EARL and Prompt+GRPO produce correct answers in the examples shown, while Prompt+CPO and SFT+GRPO also have failed examples.

J CASE STUDY

  • EARL produces correct answers for the shown targets -5666.125 and 8888 using the provided numbers.The examples receive a reward of 1.0 for the 8888 answer.Source: J CASE STUDY · pp. 30–31
  • Prompt+CPO's shown attempt for -5666.125 ends at -5666 with reward 0.0, and its 8888 example also ends without a correct answer.The latter example includes an unclosed-parenthesis error.Source: J CASE STUDY · pp. 31–32
  • Prompt+GRPO gives correct answers for the shown 8888 and -403 targets.Both examples use the listed numbers and receive reward 1.0.Source: J CASE STUDY · p. 32

SFT+GRPO

  • In two SFT+GRPO arithmetic examples, the 66432 response uses (692 × 8) × 6 × 2 despite operands [692, 8, 6, 6], while its -793 response evaluates to -705.The respective rewards are 1.0 and 0.0.Source: SFT+GRPO · pp. 32–33

Related papers

Loading 2510.07581v2…