Source-linked AI summary
Anticipatory Planning for Multimodal AI Agents
Yongyuan Liang, Shijie Zhou, Yu Gu, Hao Tan, Gang Wu, Franck Dernoncourt, Jihyung Kil, Ryan A. Rossi, Ruiyi Zhang
TL;DR
Most multimodal agents remain reactive, limiting coherent planning for long-horizon, multi-step tasks. TraceR1 trains anticipatory trajectory reasoning through two-stage reinforcement learning and improves planning stability, execution reliability, and generalization across GUI and tool-use benchmarks.
Problem
Most multimodal agents choose actions from current observations without anticipating long-term consequences, limiting reliable multi-step planning.
Method
TraceR1 combines trajectory-level reinforcement learning for globally coherent forecasts with grounded fine-tuning using execution feedback to improve action feasibility.
Results
Across seven GUI and multimodal tool-use benchmarks, TraceR1 achieves substantial improvements in planning stability, execution robustness, and generalization, with performance comparable to proprietary systems.
Takeaways & Limitations
Anticipatory trajectory reasoning is presented as a key principle for multimodal agents that must reason, plan, and act over long horizons.
Takeaways & Limitations
Short-horizon updates provide local corrections but cannot reshape the agent’s understanding of long-term feasibility or task structure.
Abstract
from arXiv · showhide
Recent advances in multimodal agents have improved computer-use interaction and tool-usage, yet most existing systems remain reactive, optimizing actions in isolation without reasoning about future states or long-term goals. This limits planning coherence and prevents agents from reliably solving high-level, multi-step tasks. We introduce TraceR1, a two-stage reinforcement learning framework that explicitly trains anticipatory reasoning by forecasting short-horizon trajectories before execution. The first stage performs trajectory-level reinforcement learning with rewards that enforce global consistency across predicted action sequences. The second stage applies grounded reinforcement fine-tuning, using execution feedback from frozen tool agents to refine step-level accuracy and executability. TraceR1 is evaluated across seven benchmarks, covering online computer-use, offline computer-use benchmarks, and multimodal tool-use reasoning tasks, where it achieves substantial improvements in planning stability, execution robustness, and generalization over reactive and single-stage baselines. These results show that anticipatory trajectory reasoning is a key principle for building multimodal agents that can reason, plan, and act effectively in complex real-world environments.
1. Introduction
Existing multimodal agents often act reactively from current observations, making long-horizon tasks difficult when actions have delayed effects. TraceR1 addresses this with anticipatory trajectory reasoning and grounded execution refinement, improving planning stability and robustness across GUI and tool-use benchmarks.
- Most multimodal agents choose actions from current observations without anticipating long-term consequences, causing divergence in multi-step environments.
- TraceR1 combines trajectory-level reinforcement learning for globally consistent multi-step plans with grounded fine-tuning based on executable tool-agent feedback.The first stage forecasts future action trajectories; the second refines step-level accuracy and feasibility.
- TraceR1 improves planning stability, execution robustness, and generalization across seven GUI and multimodal tool-use reasoning benchmarks.
- The framework forecasts future actions and step-level instructions while executing only the first predicted step, retaining later steps as lookahead.
- Its capability is comparable to proprietary systems, while significantly outperforming open-source baselines on OSWorld-Verified and AndroidWorld.It also demonstrates strong reasoning and execution reliability on GAIA and GTA.
2. Related Work
Related work spans structured GUI-agent pipelines, generalist multimodal agents, and tool-use systems. TraceR1 differs by emphasizing anticipatory organization of future tool-use behavior rather than primarily reactive execution coordination.
- Agent frameworks: GUI agent frameworks combine reasoning modules with grounding and execution components, often relying on proprietary planners for high-level action proposals.
- Generalist agents: Generalist agents extend vision-language models to computer use, GUI control, and diverse agentic tasks through unified perception, reasoning, and interaction pipelines.
- TraceR1’s distinction: TraceR1’s training framework uses trajectory-level alignment rewards in Stage 1 and tool-agent execution feedback for grounded reinforcement fine-tuning in Stage 2.
- Tool-usage multimodal agents: Tool-use research improves multimodal instruction following, tool composition, and executable control, but primarily emphasizes execution reliability or reactive coordination.
- TraceR1’s distinction: TraceR1 instead trains agents to anticipate and organize future tool-use behaviors, using grounded feedback mainly for execution validation.
3. Methodology
TraceR1 trains multimodal agents to anticipate short-horizon trajectories while grounding execution through tool feedback. Its two-stage framework combines globally consistent trajectory planning with step-level refinement for executable actions.
- Agent formulation: TraceR1 conditions action prediction on the instruction, current observation, and a compact K-step interaction history.The history stores abstracted summaries of past observations and actions rather than raw screenshots.
- Anticipatory trajectory optimization: Stage 1 uses trajectory-level reinforcement learning to align predicted future trajectories with reference continuations before execution.The model predicts a short-horizon rollout, and the reward evaluates the entire predicted sequence against the ground-truth continuation.
- Motivation: Trajectory-level optimization addresses the limitation of supervised fine-tuning, which emphasizes local likelihoods and does not enforce global rollout consistency.The paper identifies redundant or unstable rollouts as consequences of this local training objective.
- Anticipatory trajectory optimization: Discounted per-step rewards combine action alignment with a repetition penalty to promote coherent, non-redundant trajectories.The temporal discount factor weights future rewards, while the repetition term penalizes repeated or cyclic actions.
- Grounded reinforcement fine-tuning: Stage 2 executes only the first predicted action through a frozen tool agent and uses grounded feedback to refine step-level accuracy and feasibility.GUI grounding uses coordinate matching, whereas tool-calling steps use answer matching.
- Inference: At inference, TraceR1 predicts several future actions, executes the first, receives updated feedback, and replans iteratively.This plan–act loop preserves anticipatory structure while adapting to new environment states.
4. Experiment
TraceR1 is evaluated across seven GUI and multimodal tool-use benchmarks, with results showing strong task performance and clear benefits from grounded second-stage training and bounded anticipatory planning.
- 4.1. Setup: TraceR1 is evaluated across seven benchmarks spanning online GUI interaction, offline GUI evaluation, and multimodal tool-use reasoning.The benchmark suite includes AndroidWorld, OSWorld-Verified, AndroidControl-High, GUI-Odyssey, Multimodal-Mind2Web, GAIA, and GTA.
- 4.2. Main Results on GUI Environments: 30.9% success rate improves UI-TARS-1.5-7B from 27.4% to 30.9% on OSWorld-Verified.Qwen3-VL-32B-Thinking also rises from 35.6% to 41.2%, corresponding to a 15.7% relative gain.
- 4.2. Main Results on GUI Environments: More than 40% higher AndroidControl-High performance than GUI-R1 and InfiGUI-R1 establishes a new state of the art among open-source GUI agents.The paper attributes these gains to translating high-level task instructions into fine-grained action instructions more reliably.
- 4.3. Main Results on General Tool-use Scenarios: +8.7 answer-accuracy improvement over Qwen3-VL-8B on GAIA accompanies strong GTA tool-execution performance.GTA evaluates answer accuracy, tool selection and summarization, and executable generated code.
- 4.4. Ablations and Discussions: Roughly 6% average performance drops when Stage 2 is removed, indicating that execution feedback stabilizes long-horizon plan generation.Without grounding, predicted actions may be infeasible or assume nonexistent tools; execution outcomes provide corrective signals.
- 4.4. Ablations and Discussions: Performance improves with longer predictive horizons initially but drops beyond T > 10 as uncertainty accumulates and credit assignment becomes unstable.The repetition penalty λrep and temporal discount factor γ also consistently improve trajectory-aware reinforcement learning when retained.
5. Conclusion
TraceR1 combines trajectory-level optimization with grounded execution refinement to support anticipatory planning across GUI and tool-use environments. The paper concludes that this approach improves stability, reliability, generalization, and temporally extended decision making, while identifying limits in short-horizon updates.
- Conclusion: TraceR1 couples trajectory-level optimization with grounded execution refinement across GUI and tool-use environments.The framework is presented as a training recipe for anticipatory planning in dynamic, state-changing environments.
- Conclusion: The approach achieves substantial gains in planning stability, execution reliability, and generalization.
- Conclusion: Anticipatory trajectory reasoning is highlighted as a general principle for coherent, temporally extended decisions across heterogeneous interaction modalities.
- Limitations and future work: Short-horizon updates provide local corrections but cannot reshape the agent’s understanding of long-term feasibility or task structure.The paper suggests multiround or hierarchical planning with memory, internal-state, or world-model updates as a future direction.
A.1. Training Hyperparameters
The experiments use shared GRPO and optimization configurations for both reinforcement-learning stages, differing only in their reward definitions.
- Training configuration: Stage 1 and Stage 2 share identical training configurations, with reward definitions as the only difference.
A.2. Reward Formulation Details
TraceR1’s reward formulation combines accuracy and format rewards, emphasizing trajectory alignment, valid output structure, temporal consistency, coverage, and repetition avoidance.
- Reward components: The total reward R is a weighted sum of the accuracy reward Racc and format reward Rfmt, with λfmt = 0.1.
- Format Reward (Rfmt): The format reward measures the ratio of valid steps containing required XML tags and JSON keys.Required fields include screenshot abstraction, action with action type, and status.
- Trajectory Alignment Accuracy (Racc): Trajectory alignment evaluates predicted and reference action types and status fields rather than relying on standard exact matching.
- Trajectory Alignment Accuracy (Racc): A position penalty of |i −j| × 0.1 is applied, and matches are accepted only when the adjusted score exceeds 0.5.
- Trajectory Alignment Accuracy (Racc): The discounted alignment score uses γ = 0.8, subtracts 0.15 for each unmatched action, and normalizes by the reference trajectory’s maximum discounted return.
- Trajectory Alignment Accuracy (Racc): A repetition penalty counts three consecutive actions of the same type as a repetition to discourage stuck loops.
- Design rationale: The normalized alignment score encourages following the correct high-level plan before refining parameters in Stage 2.