PaperlayerView exact revision on arXiv

Source-linked AI summary

Model What Matters: Decision-Guided Selective World Modeling for Language Agents

Maolin Ran, Minda Chen, Di Huang, Wei Xiong, Fei Huang, Weiwen Liu

arXiv:2610.19642v1cs.AIcs.CLcs.LG

TL;DR

RL task rewards evaluate success without directly supervising action-conditioned environment transitions, and existing co-training often targets complete observations uniformly. DGS-WM selects decision-relevant observation content and adapts its supervision strength to prediction confidence. It reports consistent improvements across tasks, model scales, and RL algorithms, with more stable and effective supervision than full-observation methods.

  • Problem

    Task rewards typically evaluate task success without directly supervising action-conditioned environment transitions, while co-training methods often use complete observations as uniform targets.

    Source: §1 · pp. 1–2

  • Method

    DGS-WM selects observation units by their support for subsequent policy responses, adapts supervision strength to prediction confidence, and jointly optimizes the resulting world-modeling loss with RL.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    DGS-WM consistently improves across benchmarks, model scales, and RL algorithms and is more stable and effective than full-observation methods; on Qwen3-1.7B, single-hop QA average EM is 45.77%, improving over ECHO by 1.45%.

    Source: §4.2 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    Decision-guided selection makes on-policy RL trajectories a simple, practical source of world-modeling supervision without additional teacher models, human annotations, or inference overhead.

    Source: §5 · p. 8

    The response-support score assumes that greater consistency with the true response implies greater decision support, although the authors characterize this relation as correlational rather than causal.

    Source: §6 · p. 8

Abstract

from arXiv · show

Reinforcement learning provides a direct mechanism for optimizing task objectives in large language model-driven agents, but task rewards are typically sparse trajectory-level signals that only evaluate task success, without directly supervising action-conditioned environment transitions, leaving the model with limited explicit understanding of action consequences and environment dynamics. Recent work introduces auxiliary supervision through policy and world modeling co-training, yet typically treats complete environmental observations uniformly as world modeling targets, overlooking the varying decision relevance of observation content. We propose DGS-WM (Decision-Guided Selective World Modeling), a decision-guided policy-world modeling co-training approach for language agents. DGS-WM employs a Decision-Guided Observation Selection component that filters observation units with decision value as world modeling targets based on their support for subsequent policy responses, and a Confidence-Adaptive Weighting component that adaptively adjusts supervision strength according to the model's prediction confidence on selected targets. Together, these components form a decision-aware auxiliary world modeling loss, jointly optimized with the RL loss on the same policy model. Across multiple benchmarks covering single-hop/multi-hop question answering and interactive web tasks, DGS-WM achieves consistent improvements under different model scales and RL algorithms, demonstrating more stable and effective performance compared to full-observation world modeling methods.

1 Introduction

RL rewards task success but do not directly supervise action-conditioned environment transitions, while existing co-training typically targets complete observations uniformly. DGS-WM selects decision-relevant observation content and adjusts its supervision strength, with reported improvements across tasks, model scales, and RL algorithms.

1 Introduction

  • Task rewards typically evaluate success without directly supervising action-conditioned transitions, leaving rollout action–observation relationships without independent training signals.The cited discussion describes this limitation for trajectory- or outcome-level rewards.Source: §1 · p. 1
  • Environmental observations can mix decision-relevant information with content unrelated to the task, so the paper argues against treating each complete observation as a uniform target.The proposed supervision granularity is within observations, retaining content valuable for future decisions.Source: §1 · p. 2
  • DGS-WM selects observation content valuable for future decisions, adapts supervision strength to prediction confidence, and jointly trains the policy with RL objectives.Figure 1 contrasts selective targets with typical co-training that supervises all observation content equally.Source: §1 · p. 2, Figure 1 · p. 2
  • DGS-WM improves consistently across tasks, model scales, and RL algorithms, while ECHO is unstable in some settings and can underperform pure RL baselines.The evaluation covers single-hop and multi-hop question answering and web interaction.Source: §1 · p. 2

2 Related Work

Related work covers methods for optimizing LLM-agent policies and for learning environment dynamics. DGS-WM extends policy-world-model co-training by selecting which feedback content receives world-model supervision.

2.1 Reinforcement Learning for LLM Agents

  • RL-agent research spans policy updates, efficiency and stability improvements, and finer-grained credit assignment; process supervision can add annotation, estimation, or data-construction costs.The paper says its world-modeling supervision requires no additional annotations and can combine with on-policy RL algorithms such as GRPO.Source: §2.1 · p. 3

2.2 World Modeling for LLM Agents

  • World modeling has progressed from simulation and planning to staged world-model-then-RL training and joint policy-world-model co-training, which typically targets complete next observations.DGS-WM retains co-training but selects feedback content valuable for future decisions.Source: §2.2 · p. 3

3 Methodology

DGS-WM selects observation content according to its support for subsequent responses, weights prediction supervision by confidence, and jointly trains the policy with RL.

3.1 Problem Formulation and Auxiliary World Modeling

  • RL optimizes from trajectory rewards, while auxiliary world modeling predicts subsequent environmental observations from interaction history and actions.Source: §3.1 · pp. 3–4

3.2 Decision-Guided Observation Selection

  • Observations are split into semantically complete candidates; units whose response-support scores exceed an observation-specific mean-plus-κ-standard-deviation threshold become world modeling targets.Source: §3.2 · pp. 4–5

3.3 Confidence-Adaptive Weighting

  • Confidence-Adaptive Weighting assigns smaller supervision weights to high-confidence targets and preserves stronger learning signals for less-mastered targets.Source: §3.3 · p. 5

3.4 Joint Optimization

  • The decision-aware world modeling loss combines selected targets and target-level weights, normalizes by selected-token count, and is jointly optimized with RL; empty target batches yield zero auxiliary loss.Source: §3.4 · p. 5

4 Experiments

The experiments compare DGS-WM with prompting, RL-only, and full-observation world-modeling baselines across QA and web tasks. DGS-WM improves over ECHO across task groups, model scales, and tested RL algorithms; ablations and parameter sensitivity examine its components and settings.

4.1 Experimental Setup

  • The evaluation covers multi-hop QA, single-hop QA, and WebShop, comparing prompting, RL-only, and RL-augmented world-modeling methods.QA uses EM; WebShop reports success rate and score. Primary experiments use GRPO, with PPO, RLOO, and GMPO in cross-algorithm experiments.Source: §4.1 · p. 6

4.2 Main Results

  • DGS-WM outperforms ECHO across single-hop QA, multi-hop QA, and WebShop at both Qwen3 scales, while ECHO can underperform GRPO.On Qwen3-1.7B multi-hop QA, ECHO scores 17.71% average EM versus GRPO’s 18.88%.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 Generalization Across Models and Algorithms

  • Across all six additional model-and-algorithm configurations, DGS-WM outperforms both the RL-only baseline and its ECHO-augmented variant.The tests cover three additional models under GRPO and three additional RL algorithms on Qwen2.5-1.5B.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 Ablation Study

  • Full DGS-WM achieves 45.77% single-hop and 22.06% multi-hop average EM, and removing either component lowers performance.Removing observation selection lowers the averages by 1.32% and 1.75%; removing confidence weighting lowers them by 0.95% and 0.40%.Source: §4.4 · pp. 7–8, Table 3 · p. 8

4.5 Parameter Sensitivity Analysis

  • Combined QA average first rises and then falls as κ varies, peaking at κ = 0.5; confidence weighting performs best at γ = 1.The sensitivity analysis uses Qwen3-1.7B.Source: §4.5 · p. 8

5 Conclusion

The conclusion argues that observation content differs in value for future decisions, so world-modeling targets should be selected for decision relevance. DGS-WM combines that selection with confidence-adaptive supervision and shows stable improvements across the evaluated tasks, models, and RL algorithms.

5 Conclusion

  • Only decision-relevant observation content should serve as world-modeling prediction targets because observations can also contain content unrelated to the current task.The paper frames this as addressing non-uniform supervision value in auxiliary world modeling.Source: §5 · p. 8
  • DGS-WM selects supervisory content based on subsequent policy responses and adjusts supervision strength according to prediction confidence.The conclusion identifies Decision-Guided Observation Selection and Confidence-Adaptive Weighting as its components.Source: §5 · p. 8
  • Across single-hop QA, multi-hop QA, and interactive web shopping, DGS-WM improves consistently across model scales and RL algorithms without teacher models, human annotations, or inference overhead.The conclusion characterizes selective world modeling as a practical source of auxiliary supervision from on-policy trajectories.Source: §5 · p. 8

6 Limitations

DGS-WM has limitations in how it segments observations and estimates their decision value. Meaningful units for complex feedback remain unresolved, and the response support score reflects correlation rather than causation.

6 Limitations

  • Observation-unit design depends on environment structure: QA uses retrieval-feedback sentences, while WebShop uses page structure.Identifying meaningful units in complex or unstructured feedback remains open.Source: §6 · p. 8
  • The response support score treats higher likelihood of the true subsequent response as greater decision support, an assumption the authors characterize as correlational, not causal.Counterfactual reasoning or causal interventions could help identify feedback that truly affects decisions.Source: §6 · p. 8

A Implementation Details

The experiments cover Open-Domain Question Answering and Interactive Web Shopping.

A Implementation Details

A.1 Datasets

The evaluation uses open-domain QA with retrieval and interactive WebShop shopping tasks. The appendix describes their datasets, held-out splits, and task constraints.

A.1.1 Open-Domain QA

  • Open-domain QA requires agents to answer without pre-given documents by interacting with an external retrieval system.Training mixes NQ and HotpotQA; 27,648 of 169,615 samples are stratified for RL training, with 1,024 for validation.Source: §A.1.1 · p. 10
  • QA evaluation spans single-hop and multi-hop benchmarks, including NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle.Each question allows at most five turns, with search returning the top three documents from the Wiki18-based retrieval system.Source: §A.1.1 · p. 10

A.1.2 Interactive Web Shopping (WebShop)

  • WebShop simulates e-commerce with 1.18M products and 12,087 crowd-sourced shopping instructions.The agent must browse and purchase an item matching user requirements; 10,587 training instructions are used, with 1,000 validation and 500 test instructions.Source: §A.1.2 · p. 10
  • WebShop tasks allow at most 15 steps, using search, click, or buy actions; success requires the correct item, matching attributes, and a price within budget.Source: §A.1.2 · p. 10

A.2 Training Details

Training uses several base models and GRPO as the primary RL algorithm. DGS-WM hyperparameters govern observation selection and confidence weighting, while all tasks use sparse binary rewards.

A.2.1 Model and LoRA Configuration

  • Main experiments use Qwen3-1.7B and Qwen3-8B, with Qwen2.5-1.5B, Llama-3.2-3B, and Qwen2.5-7B used for cross-model generalization.LoRA is applied to attention projection matrices and all linear layers with rank 32 and alpha 32.Source: §A.2.1 · pp. 10–11

A.2.2 Reinforcement Learning Hyperparameters

  • GRPO is the primary RL algorithm, trained with batch size 144, rollout groups of eight, and one PPO epoch.The setup also specifies rollout temperature 1.0, greedy validation, AdamW, and a 5 × 10^-6 learning rate.Source: §A.2.2 · p. 11

A.2.3 DGS-WM Hyperparameters

  • Observation selection uses threshold τt = µt + 0.5σt, and confidence weighting uses a linear decay exponent γ = 1.0.A candidate unit must exceed the current observation’s mean response support score by half a standard deviation.Source: §A.2.3 · p. 11

A.2.4 Reward Function

  • All tasks use sparse binary rewards: success receives 1 and failure receives 0.QA and WebShop each apply a −0.01 penalty per invalid action.Source: §A.2.4 · p. 11

B.1 QA Tasks: Sentence Splitting

The appendix defines task-aware observation units for QA documents and WebShop pages, then illustrates DGS-WM selecting a small subset of sentences for a correct single-hop answer.

B.1 QA Tasks: Sentence Splitting

  • QA observations are split at sentence-ending punctuation followed by whitespace and a capital letter or digit, with abbreviations and single-letter names protected from splits.Source: §B.1 · p. 11
  • Each resulting sentence becomes a candidate observation unit with paragraph, rank, sentence-index, and original-character-position metadata.Source: §B.1 · p. 11

B.2 WebShop: Semantic UI Block Identification

  • WebShop pages are classified as search results, product, search home, or done, and page-specific blocks group product cards, purchase details, search controls, and feedback.Source: §B.2 · pp. 11–12
  • For unrecognized WebShop pages, buttons remain whole-line units while plain text uses the QA sentence-splitting rules.Source: §B.2 · p. 12

B.3 Granularity Design Rationale

  • Task-aware units balance decision precision with semantic integrity: sentences preserve document propositions, while UI blocks retain functional coherence.Coarse units mix relevant content with irrelevant context; overly fine units can become uninformative fragments.Source: §B.3 · p. 12

C Case Studies

  • The case studies mark selected world-modeling prediction targets in bold and dark green within real training-log observations.Source: C Case Studies · p. 13

C.1 Single-Hop Question Answering (Qwen3-1.7B)

  • Selection reduced 18 candidate sentences to 2 selected sentences (11.1%); the subsequent policy response was Lieder, which the cited passage marks with a check.Source: §C.1 · p. 13

C.2 Multi-Hop Question Answering (Qwen3-8B)

The case studies show DGS-WM selecting a small portion of observations while preserving correct responses, from multi-hop QA to WebShop product selection and confirmation.

C.2 Multi-Hop Question Answering (Qwen3-8B)

  • For the multi-hop question, DGS-WM selected two key sentences from 27 candidates, and the policy correctly answered “Greenwich Village.”The question requires linking the film director identified in one document with her location in another.Source: §C.2 · p. 14

C.3 Product Search (Qwen3-8B, WebShop)

  • In product search, the policy selected Product 2, a long-lasting women’s fragrance gift set priced at $64.99, below the $80 task limit.Source: §C.3 · p. 15

C.4 Product-Detail Confirmation (Qwen3-8B, WebShop)

  • The policy chose to buy, and the interpretation says it checked set contents, longevity, and price while ignoring secondary content such as brand story, ingredient details, and recommended products.The $64.99 set includes Eau de Parfum 50ml, Body Lotion 75ml, and Travel Spray 10ml; its formula is long-lasting (8-10 hours).Source: §C.4 · p. 16
  • Across the cases, selection rates of 7.4–12.5% substantially reduce world-modeling supervision while preserving correct policy responses.Source: §C.4 · p. 16
Loading 2610.19642v1…