PaperlayerView exact revision on arXiv

Source-linked AI summary

FocusVLA: Future-Intent Guided Dynamic View Routing for Vision-Language-Action Models

Huixin Xu, Ziya Yu, Peng Li

arXiv:2610.31423v1cs.ROcs.AI

TL;DR

Existing VLA models use complementary multi-view inputs but rarely adapt their visual dependence to changing manipulation stages. FocusVLA predicts future intent to guide step-wise view routing during action generation. It outperforms static multi-view baselines across simulation and real-world tasks.

  • Problem

    Existing VLA models rarely account for visual dependencies changing across manipulation stages, although robust manipulation benefits from complementary global and local views.

    Source: §1 · p. 2

  • Method

    FocusVLA predicts future action intent from multimodal observations and short-term action history to generate step-wise view-routing weights during action generation.

    Source: §1 · p. 2

  • Results

    FocusVLA consistently outperforms static multi-view baselines across diverse task horizons and physical scenarios.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Future-intent-guided routing adapts view usage across task stages, supporting global planning and local interaction.

    Source: §5 · p. 9

    Future intent is only one possible representation of manipulation progress; richer progress-aware signals and cross-embodiment routing remain directions for exploration.

    Source: §6 · p. 9

Abstract

from arXiv · show

Vision-language-action (VLA) models have become an important paradigm for robotic manipulation by unifying visual perception, language understanding, and action generation. Since reliable manipulation requires sufficient observation of objects, end-effectors, and their spatial relationships, multi-view inputs are widely used to provide complementary spatial information. However, our experiments show that existing VLA models do not fully exploit such multi-view complementarity. The policy tends to persistently rely on the wrist view, lacking the ability to dynamically adjust global-local view dependency across manipulation stages, which limits stage adaptability and robustness. To address this issue, we propose FocusVLA, a stage-aware multi-view routing framework. FocusVLA predicts future interaction intent from historical actions and uses it to guide dynamic view routing during action generation. By adaptively coordinating global and local observations across different task stages, FocusVLA improves multi-view utilization. On RoboTwin 2.0, FocusVLA improves the average success rate over the strongest baseline $\pi_{0.5}$ by 10.10 pp. In real-world evaluation, FocusVLA improves the success rate from 71.7\% to 85.0\%, achieving a 13.3\% gain over $\pi_{0.5}$. Moreover, the intent prediction and dynamic routing modules of FocusVLA can be integrated into existing VLA frameworks without modifying their core architectures.

1 Introduction

Static multi-view fusion can preserve an early view preference even after manipulation requirements change, creating stage–view misalignment. FocusVLA uses predicted future intent to route views step by step, and is evaluated in simulation and real-world tasks.

1 Introduction

  • FocusVLA predicts future action intent from multimodal observations and short-term action history, then uses it to generate action-step-wise routing weights.The framework coordinates visual features from different views during action generation without modifying the visual encoder, language model, or action-generation backbone.Source: §1 · p. 2
  • Static fusion can preserve early-formed view dependence after visual requirements change, causing stage–view misalignment.The paper identifies this as a temporal failure of static multiview VLA policies.Source: §1 · p. 2
  • FocusVLA is evaluated on all 50 RoboTwin simulation tasks and six real-world manipulation tasks, with improved adaptability and robustness over static multi-view baselines.The real-world tasks include diverse object configurations, viewpoint variations, and interaction requirements.Source: §1 · p. 2

2 Related Works

Prior VLA systems commonly fuse visual observations, language, and robot states into a fixed context for action generation, despite stage-dependent visual needs. FocusVLA instead routes views during action generation, with future-intent-guided, step-specific routing distinguishing it from earlier multi-view fusion approaches.

2.1 Vision-Language-Action Models.

  • VLA models commonly encode visual observations and fuse them with language instructions and robot states as context for action generation.The paper notes that this fixed pipeline rarely accounts for visual dependencies changing across manipulation stages.Source: §2.1 · pp. 2–3
  • FocusVLA introduces an architecture-agnostic routing module on the action-generation side to modulate global and local visual context without modifying existing VLA architectures.This targets stage-dependent visual needs rather than changing action decoders or scaling VLA backbones.Source: §2.1 · p. 3
  • Figure 2 presents the fusion pathway of multi-view perception in VLA models.The described pipeline encodes visual observations, then fuses them with language and robot states before action generation.Source: Figure 2 · p. 3, §2.1 · p. 3

2.2 Multi-View Perception and Dynamic Visual Routing.

  • Unlike BFA’s current-view allocation and pre-policy fusion, FocusVLA predicts H step-specific future routes and fuses views within action generation.FocusVLA also derives task-agnostic labels from trajectories and adds chunk-level intent conditioning, which BFA lacks.Source: §2.2 · p. 3

3 Method

The analysis identifies stage-insensitive view dependency in multi-view policies and motivates FocusVLA, which predicts future intent to route views during action generation. Its components use step-wise routing and joint action-intent training without modifying the pretrained backbone.

3.1 Motivation: Diagnosing Stage-Invariant View Dependency in Multi-View VLA

  • Failed rollouts show stronger dominant-view reliance and less temporal adjustment than successful rollouts, while local-view preference alone is not necessarily harmful.The analysis uses attention ratios for global and aggregated local views, plus Dominance Margin and temporal variation of local-view reliance.Source: §3.1 · pp. 3–4
  • During global-planning stages, masking the local view lowers success from 85.0% to 20.0%, a larger drop than masking the global view to 65.0%.The stage-conditioned masking experiment uses the π0.5 multiview baseline on Stack Blocks Three.Source: §3.1 · p. 4, Table 1 · p. 4

3.2 The FocusVLA Model Architecture

  • FocusVLA predicts future interaction intent from multimodal context and action history, producing a chunk-level intent token and step-wise routing weights without changing the pretrained backbone.The weights dynamically coordinate global and local visual contexts during action generation.Source: §3.2 · p. 4

3.2.1 Future Intent Bridge

  • The Future Intent Bridge summarizes recent action dynamics and uses cross-modal attention to predict step-wise intent across the future action horizon.The intent sequence is pooled into a chunk-level token and converted into routing weights; interaction-oriented intent concerns local visual evidence, while its complement concerns workspace-level motion.Source: §3.2.1 · pp. 4–5

3.2.2 Action-Side Dynamic Routing

  • Action-side routing fuses global and local action contexts according to each future step’s predicted weights, then injects the fused context into action features for refinement.The routing is performed in action generation rather than by modifying multi-view representations inside the visual encoder.Source: §3.2.2 · pp. 5–6
  • On 50 RoboTwin 2.0 tasks, Table 2 reports that FocusVLA achieves the best overall performance and consistent improvements across diverse tasks.The tasks are grouped by horizon.Source: Table 2 · p. 6

3.3 Training Objective

  • Training jointly supervises action generation and future-intent prediction, first warming up the intent branch before joint optimization with occasional teacher forcing.The action loss compares denoised outputs with future action targets; the intent loss uses binary labels.Source: §3.3 · p. 6

4 Experiment

The experiments evaluate FocusVLA on simulation and real-world tasks, unseen spatial layouts, stage-wise view routing, and ablations of its design. Comparisons use shared demonstrations, optimization steps, test seeds, and evaluation protocols.

4 Experiment

  • The evaluation compares models trained on the same demonstrations and optimization steps under identical test seeds and protocols, without additional data or tuning.The ablations used nine tasks, two training seeds, 100 trials per task per seed, and 20K training steps.Source: §4 · p. 6, §4.5 · p. 8

4.1 Simulation Evaluation

  • 76.92% average success on RoboTwin 2.0’s 50 tasks outperformed the strongest baseline by 10.10 percentage points.Evaluation used 100 trials per task.Source: §4.1 · p. 7

4.2 Real-World Evaluation

  • 85.0% average success across six real-world tasks exceeded π0.5 by 13.3 percentage points and π0 by 25.6 percentage points.Each task had 30 trials; FocusVLA performed best on all six.Source: §4.2 · p. 7, Figure 5 · p. 7

4.3 Execution Adaptability under Spatial Rearrangement

  • Under unseen spatial layouts, FocusVLA was more stable than static multi-view fusion baselines.Rearranged object positions required adapting target localization, approach paths, and fine-grained interactions.Source: §4.3 · p. 7, Figure 6 · p. 7

4.4 Visualization Analysis

  • On Stack Blocks Three, routing shifted from global views during early localization and planning toward local views for grasping and alignment.During stacking, routing weights became more balanced.Source: §4.4 · p. 8, Figure 7 · p. 8

4.5 Ablation Study

  • 73.7% mean success for Full exceeded Learned Static by 6.2 pp, Reactive-Vision by 16.3 pp, and Time+State by 15.2 pp.The comparisons test static preferences, current vision, and time/state cues against multimodal future intent.Source: Table 3 · p. 8, §4.5 · p. 9
  • Removing explicit future-intent supervision reduced success to 55.7%, 18.0 pp below Full; random view masking trailed Full by 19.1 pp.The authors conclude that gains are not explained merely by additional routing-network capacity or generic view regularization.Source: Table 3 · p. 8, §4.5 · p. 9
  • Removing the chunk-level intent token reduced performance from 73.7% to 63.3%, a 10.4 pp drop.Step-wise routing allocates views for future actions, while the chunk-level token gives the action generator an overall task-progress estimate.Source: Table 3 · p. 8, §4.5 · p. 9

5 Conclusion

FocusVLA uses future action intent to guide step-wise view routing, and the experiments support this as a strategy for improving multi-view VLA adaptability and robustness.

5 Conclusion

  • FocusVLA predicts future action intent to coordinate global and local visual features during action generation, mitigating view-dependency inertia from static fusion.Experiments and visualizations report improved performance across diverse horizons and physical scenarios, with view use adapting across stages.Source: §5 · p. 9

6 Limitations

The conclusion identifies richer progress signals and cross-embodiment evaluation as directions for future work.

6 Limitations

  • Future work could explore progress signals beyond future intent and extend routing to diverse robots, camera configurations, and real-world environments.Suggested signals include explicit subgoals, object-level interaction states, task-phase descriptions, and language-grounded intermediate plans.Source: §6 · p. 9

A. Automatic Intent Label Construction

Intent labels are generated automatically from demonstration trajectories by segmenting arm motion into interaction and global-movement phases. Gripper closures provide direct interaction seeds when available; otherwise, motion-based heuristics identify interaction candidates and refine them.

A. Automatic Intent Label Construction

  • When gripper signals are available, valid closing runs become interaction seeds; otherwise, motion thresholds and low-speed segments drive candidate detection.Closing runs require at least four consecutive frames, exclude closures before frame 12, and receive a 16-frame lookback.Source: A. Automatic Intent Label Construction · pp. 14–15
  • Motion-based candidates are retained when low-speed segments are surrounded by high-speed motion, with path length and dominant-axis checks filtering out transport.Press-like and handover-like cases receive additional candidate rules; nearby windows are merged, then high-speed movement and short pauses are reclassified or removed.Source: A. Automatic Intent Label Construction · p. 15
  • Binary frame-wise labels mark local interaction as 1 and global movement as 0 after deterministic segmentation and cleanup.The same procedure is applied to every episode, making label generation reproducible.Source: A. Automatic Intent Label Construction · pp. 14, 16

B. Training and Evaluation Configurations

Training uses separate multi-task policies for simulation and real-world benchmarks, with intent prediction warmed up before joint optimization. The Future Intent Bridge predicts future interaction demand and converts it into action-step routing signals, which Action-Side Dynamic Routing applies during action generation.

B. Training and Evaluation Configurations

  • Each domain is trained separately with 50 demonstration trajectories per task, a joint policy per benchmark, and 100,000 optimization steps on 8 NVIDIA A100 80GB GPUs.Evaluation uses task-specific test episodes and checkpoints from the corresponding domain-specific training.Source: B. Training and Evaluation… · p. 16
  • Intent prediction is warmed up before joint training, and a class-balanced weighted binary cross-entropy addresses the lower frequency of interaction frames.The combined objective adds intent loss to action-generation loss, weighted by λintent.Source: B. Training and Evaluation… · p. 16

C. Future Intent Bridge Details

  • The Future Intent Bridge uses language queries over visual and state-history features to predict a step-wise intent sequence across the future horizon.Each predicted probability estimates whether a future action step requires local interaction-oriented visual evidence; the sequence also yields a chunk-level intent token and step-wise routing weights.Source: C. Future Intent Bridge Details · pp. 16–17

D. Action-Side Dynamic Routing Details

  • Action-Side Dynamic Routing uses action features to attend separately to global and local visual features, then fuses the contexts for each future step.A gated residual connection injects the fused context into action features, allowing view dependence to shift across manipulation stages while preserving the pretrained visual backbone.Source: D. Action-Side Dynamic Routing Details · p. 17

E. BFA-Related Controls

Matched controls attribute distinct gains to anticipatory step-wise routing, action-side fusion, trajectory-derived labels, and chunk-level intent conditioning. Comparisons use the same four-task subset and evaluation protocol.

Prediction horizon

  • Full exceeds Reactive-Vision by 18.2 pp and Reactive (Hr=1) by 24.0 pp, supporting multimodal future-intent conditioning and anticipatory step-specific routing.Reactive-Vision uses current vision; Reactive (Hr=1) retains multimodal conditioning but shares one route across the chunk.Source: E. BFA-Related Controls · p. 17

Fusion site

  • Full exceeds Feature-side by 12.5 pp, isolating the benefit of fusing views within action generation rather than before the policy.The control fixes routing, intent supervision, and the chunk-level token while changing the fusion location.Source: E. BFA-Related Controls · p. 18, Table 4 · p. 18

Label construction

  • Full exceeds BFA-style Labels by 7.5 pp, while human-agreement evaluation also supports the accuracy of task-agnostic trajectory labels.The paper-based BFA-style baseline uses GPT-4o arm-state annotations and task-specific mappings.Source: E. BFA-Related Controls · p. 18, Table 4 · p. 18

Intent token

  • Adding the intent token improves Reactive (Hr=1) over Reactive w/o Token by 5.7 pp and Full over Routing Only by 8.0 pp.These matched comparisons indicate benefits for both chunk-shared and step-wise routing.Source: E. BFA-Related Controls · p. 18, Table 4 · p. 18

Summary

  • The controls report distinct gains from anticipatory routing, action-side fusion, trajectory-derived supervision, and chunk-level intent conditioning.All variants use the same four-task subset and evaluation protocol.Source: E. BFA-Related Controls · pp. 17–18

F. Intent-Label Evaluation

Trajectory-derived intent labels align more closely with human annotations than the BFA-style alternative, and the Future Intent Bridge predicts held-out intent with reported F1 and AUROC. Control success rates change by at most 4.0 pp across label definitions and by at most 4.5 pp under moderate inference-time intent corruption.

Human agreement

  • 82.9% F1 for task-agnostic trajectory labels exceeds paper-based BFA-style labels by 17.8 points against human annotations.The authors state that these labels avoid offline VLM annotations and task-specific rule mappings.Source: F. Intent-Label Evaluation · p. 18, Table 5 · p. 19

Intent-prediction accuracy

  • 78.4% F1 at a 0.5 threshold and 91.2% AUROC are reported for held-out intent prediction by the Future Intent Bridge.Most inspected errors slightly shift transition boundaries rather than confuse interaction stages.Source: F. Intent-Label Evaluation · p. 18

Robustness to label definitions

  • Success rates are 75.5% and 74.0% under loose and strict label variants, versus 78.0% under default labels, a maximum change of 4.0 pp.The loose and strict variants alter 45.2% and 27.6% of labels relative to the default definition.Source: F. Intent-Label Evaluation · p. 18

Robustness to prediction errors

  • With 10% and 20% of predicted intents corrupted, success rates are 73.5% and 75.5%, differing from the uncorrupted result by at most 4.5 pp.The authors conclude that FocusVLA remains effective under moderate prediction errors; generalization to unseen camera layouts remains future work.Source: F. Intent-Label Evaluation · p. 19

G. Architecture and Implementation Details

FocusVLA augments π0.5 with a Future Intent Bridge and an action-side router. The Bridge predicts intent for 30 future action steps, while the router uses corresponding weights to fuse global and local action contexts.

Architecture

  • The Future Intent Bridge encodes current state, the previous 30 actions and their temporal differences, and multimodal context to predict intent scores for 30 future action steps.A 2,048-D state token and history token are processed with four-head cross-attention; masked-mean pooling and an MLP produce the scores.Source: G. Architecture and Implementation… · p. 19
  • The action-side router uses cross-attention to construct separate global and local action contexts, then fuses them at each predicted step using its corresponding routing weight.A chunk-level intent token also conditions overall action generation.Source: G. Architecture and Implementation… · p. 19

Training and computational overhead

  • 92.78 ms per chunk is FocusVLA’s latency versus 79.59 ms for π0.5, an increase of 13.19 ms (16.6%) at batch size 1 on one NVIDIA A100.FocusVLA adds 70.13M parameters; training uses a 5K-step intent-only warm-up and anneals teacher forcing from 0.77 to 0.3 between steps 5K and 15K, disabling it at inference.Source: G. Architecture and Implementation… · p. 19

Bounded routing weights

  • Routing weights stay strictly within (0.25, 0.75), with complementary weights summing to one and allowing up to a 3:1 preference while retaining at least 25% of the other view.The shared task-invariant mapping uses sample- and step-specific intent predictions; 64.19% of held-out αg values remain strictly interior at reporting precision.Source: G. Architecture and Implementation… · p. 19

Related papers

Loading 2610.31423v1…