Source-linked AI summary
Anticipatory Planning for Multimodal AI Agents
Yongyuan Liang, Shijie Zhou, Yu Gu, Hao Tan, Gang Wu, Franck Dernoncourt, Jihyung Kil, Ryan A. Rossi, Ruiyi Zhang
TL;DR
Most multimodal agents remain reactive, limiting coherent planning for long-horizon, multi-step tasks. TraceR1 trains anticipatory trajectory reasoning through two-stage reinforcement learning and improves planning stability, execution reliability, and generalization across GUI and tool-use benchmarks.
Problem
Most multimodal agents choose actions from current observations without anticipating long-term consequences, limiting reliable multi-step planning.
Source: §1 · p. 1Method
TraceR1 combines trajectory-level reinforcement learning for globally coherent forecasts with grounded fine-tuning using execution feedback to improve action feasibility.
Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 4Results
Takeaways & Limitations
Anticipatory trajectory reasoning is presented as a key principle for multimodal agents that must reason, plan, and act over long horizons.
Source: §1 · p. 2, §5 · p. 8Short-horizon updates provide local corrections but cannot reshape the agent’s understanding of long-term feasibility or task structure.
Source: §5 · pp. 8–9
Abstract
from arXiv · showhide
Recent advances in multimodal agents have improved computer-use interaction and tool-usage, yet most existing systems remain reactive, optimizing actions in isolation without reasoning about future states or long-term goals. This limits planning coherence and prevents agents from reliably solving high-level, multi-step tasks. We introduce TraceR1, a two-stage reinforcement learning framework that explicitly trains anticipatory reasoning by forecasting short-horizon trajectories before execution. The first stage performs trajectory-level reinforcement learning with rewards that enforce global consistency across predicted action sequences. The second stage applies grounded reinforcement fine-tuning, using execution feedback from frozen tool agents to refine step-level accuracy and executability. TraceR1 is evaluated across seven benchmarks, covering online computer-use, offline computer-use benchmarks, and multimodal tool-use reasoning tasks, where it achieves substantial improvements in planning stability, execution robustness, and generalization over reactive and single-stage baselines. These results show that anticipatory trajectory reasoning is a key principle for building multimodal agents that can reason, plan, and act effectively in complex real-world environments.
1. Introduction
Existing multimodal agents often act reactively from current observations, making long-horizon tasks difficult when actions have delayed effects. TraceR1 addresses this with anticipatory trajectory reasoning and grounded execution refinement, improving planning stability and robustness across GUI and tool-use benchmarks.
- Most multimodal agents choose actions from current observations without anticipating long-term consequences, causing divergence in multi-step environments.Source: §1 · p. 1
- TraceR1 combines trajectory-level reinforcement learning for globally consistent multi-step plans with grounded fine-tuning based on executable tool-agent feedback.The first stage forecasts future action trajectories; the second refines step-level accuracy and feasibility.Source: §1 · p. 1
- TraceR1 improves planning stability, execution robustness, and generalization across seven GUI and multimodal tool-use reasoning benchmarks.Source: §1 · pp. 1–2
- The framework forecasts future actions and step-level instructions while executing only the first predicted step, retaining later steps as lookahead.Source: Figure 1 · p. 2
- Its capability is comparable to proprietary systems, while significantly outperforming open-source baselines on OSWorld-Verified and AndroidWorld.It also demonstrates strong reasoning and execution reliability on GAIA and GTA.Source: §1 · p. 2
2. Related Work
Related work spans structured GUI-agent pipelines, generalist multimodal agents, and tool-use systems. TraceR1 differs by emphasizing anticipatory organization of future tool-use behavior rather than primarily reactive execution coordination.
Agent frameworks
- GUI agent frameworks combine reasoning modules with grounding and execution components, often relying on proprietary planners for high-level action proposals.Source: §2.1 · p. 2
Generalist agents
- Generalist agents extend vision-language models to computer use, GUI control, and diverse agentic tasks through unified perception, reasoning, and interaction pipelines.Source: §2.1 · p. 2
TraceR1’s distinction
- TraceR1’s training framework uses trajectory-level alignment rewards in Stage 1 and tool-agent execution feedback for grounded reinforcement fine-tuning in Stage 2.Source: Figure 2 · p. 3
Tool-usage multimodal agents
- Tool-use research improves multimodal instruction following, tool composition, and executable control, but primarily emphasizes execution reliability or reactive coordination.Source: §2.2 · p. 3
TraceR1’s distinction
- TraceR1 instead trains agents to anticipate and organize future tool-use behaviors, using grounded feedback mainly for execution validation.Source: §2.2 · p. 3
3. Methodology
TraceR1 trains multimodal agents to anticipate short-horizon trajectories while grounding execution through tool feedback. Its two-stage framework combines globally consistent trajectory planning with step-level refinement for executable actions.
Agent formulation
- TraceR1 conditions action prediction on the instruction, current observation, and a compact K-step interaction history.The history stores abstracted summaries of past observations and actions rather than raw screenshots.Source: §3 · p. 3
Anticipatory trajectory optimization
- Stage 1 uses trajectory-level reinforcement learning to align predicted future trajectories with reference continuations before execution.The model predicts a short-horizon rollout, and the reward evaluates the entire predicted sequence against the ground-truth continuation.Source: §3 · p. 3, §3.1 · p. 3, §3.2 · p. 4
Motivation
- Trajectory-level optimization addresses the limitation of supervised fine-tuning, which emphasizes local likelihoods and does not enforce global rollout consistency.The paper identifies redundant or unstable rollouts as consequences of this local training objective.Source: §3.1 · pp. 3–4
Anticipatory trajectory optimization
- Discounted per-step rewards combine action alignment with a repetition penalty to promote coherent, non-redundant trajectories.The temporal discount factor weights future rewards, while the repetition term penalizes repeated or cyclic actions.Source: §3.1 · pp. 3–4
Grounded reinforcement fine-tuning
- Stage 2 executes only the first predicted action through a frozen tool agent and uses grounded feedback to refine step-level accuracy and feasibility.GUI grounding uses coordinate matching, whereas tool-calling steps use answer matching.Source: §3.2 · p. 4
Inference
- At inference, TraceR1 predicts several future actions, executes the first, receives updated feedback, and replans iteratively.This plan–act loop preserves anticipatory structure while adapting to new environment states.Source: §3.3 · p. 4
4. Experiment
TraceR1 is evaluated across seven GUI and multimodal tool-use benchmarks, with results showing strong task performance and clear benefits from grounded second-stage training and bounded anticipatory planning.
4.1. Setup
- TraceR1 is evaluated across seven benchmarks spanning online GUI interaction, offline GUI evaluation, and multimodal tool-use reasoning.The benchmark suite includes AndroidWorld, OSWorld-Verified, AndroidControl-High, GUI-Odyssey, Multimodal-Mind2Web, GAIA, and GTA.Source: §4.1 · p. 5
4.2. Main Results on GUI Environments
- 30.9% success rate improves UI-TARS-1.5-7B from 27.4% to 30.9% on OSWorld-Verified.Qwen3-VL-32B-Thinking also rises from 35.6% to 41.2%, corresponding to a 15.7% relative gain.Source: §4.2 · p. 6
- More than 40% higher AndroidControl-High performance than GUI-R1 and InfiGUI-R1 establishes a new state of the art among open-source GUI agents.The paper attributes these gains to translating high-level task instructions into fine-grained action instructions more reliably.Source: §4.2 · p. 6
4.3. Main Results on General Tool-use Scenarios
- +8.7 answer-accuracy improvement over Qwen3-VL-8B on GAIA accompanies strong GTA tool-execution performance.GTA evaluates answer accuracy, tool selection and summarization, and executable generated code.Source: §4.3 · p. 6, Table 3 · p. 7
4.4. Ablations and Discussions
- Roughly 6% average performance drops when Stage 2 is removed, indicating that execution feedback stabilizes long-horizon plan generation.Without grounding, predicted actions may be infeasible or assume nonexistent tools; execution outcomes provide corrective signals.Source: §4.4 · p. 7, Figure 3 · p. 7
- Performance improves with longer predictive horizons initially but drops beyond T > 10 as uncertainty accumulates and credit assignment becomes unstable.The repetition penalty λrep and temporal discount factor γ also consistently improve trajectory-aware reinforcement learning when retained.Source: §4.4 · pp. 7–8
5. Conclusion
TraceR1 combines trajectory-level optimization with grounded execution refinement to support anticipatory planning across GUI and tool-use environments. The paper concludes that this approach improves stability, reliability, generalization, and temporally extended decision making, while identifying limits in short-horizon updates.
Conclusion
- TraceR1 couples trajectory-level optimization with grounded execution refinement across GUI and tool-use environments.The framework is presented as a training recipe for anticipatory planning in dynamic, state-changing environments.Source: §5 · p. 8
- The approach achieves substantial gains in planning stability, execution reliability, and generalization.Source: §5 · p. 8
- Anticipatory trajectory reasoning is highlighted as a general principle for coherent, temporally extended decisions across heterogeneous interaction modalities.Source: §5 · p. 8
Limitations and future work
- Short-horizon updates provide local corrections but cannot reshape the agent’s understanding of long-term feasibility or task structure.The paper suggests multiround or hierarchical planning with memory, internal-state, or world-model updates as a future direction.Source: §5 · pp. 8–9
A.1. Training Hyperparameters
The experiments use shared GRPO and optimization configurations for both reinforcement-learning stages, differing only in their reward definitions.
Training configuration
- Stage 1 and Stage 2 share identical training configurations, with reward definitions as the only difference.Source: §A.1 · p. 12
A.2. Reward Formulation Details
TraceR1’s reward formulation combines accuracy and format rewards, emphasizing trajectory alignment, valid output structure, temporal consistency, coverage, and repetition avoidance.
Reward components
- The total reward R is a weighted sum of the accuracy reward Racc and format reward Rfmt, with λfmt = 0.1.Source: §A.2 · p. 12
Format Reward (Rfmt)
- The format reward measures the ratio of valid steps containing required XML tags and JSON keys.Required fields include screenshot abstraction, action with action type, and status.Source: §A.2 · p. 12
Trajectory Alignment Accuracy (Racc)
- Trajectory alignment evaluates predicted and reference action types and status fields rather than relying on standard exact matching.Source: §A.2 · p. 12
- A position penalty of |i −j| × 0.1 is applied, and matches are accepted only when the adjusted score exceeds 0.5.Source: §A.2 · p. 12
- The discounted alignment score uses γ = 0.8, subtracts 0.15 for each unmatched action, and normalizes by the reference trajectory’s maximum discounted return.Source: §A.2 · p. 12
- A repetition penalty counts three consecutive actions of the same type as a repetition to discourage stuck loops.Source: §A.2 · p. 12
Design rationale
- The normalized alignment score encourages following the correct high-level plan before refining parameters in Stage 2.Source: §A.2 · p. 12