PaperlayerView exact revision on arXiv

Source-linked AI summary

VisualThink-VLA: Visual Intermediate Reasoning for Effective and Low-Latency Vision-Language-Action Policies

Mingjian Gao, Wenqiao Zhang, Yuqian Yuan, Yang Dai, Binhe Yu, Zheqi Lv, Haoyu Zheng, Jiaqi Zhu, Zhiqi Ge, Zixuan Wan, Siliang Tang, Yueting Zhuang

arXiv:2605.30011v1cs.CVcs.AI

TL;DR

Existing reasoning-augmented VLA policies face an accuracy-efficiency trade-off because textual traces are weakly grounded and slow, while dense visual information can introduce interference. VISUALTHINK-VLA uses compact routed visual evidence and achieves a better success-latency trade-off across evaluations, supporting visual-space reasoning as a practical substrate for embodied control.

  • Problem

    Reasoning-augmented VLA policies need a minimal visual reasoning interface that preserves task-relevant structure without textual latency or interference from excessive auxiliary information.

  • Method

    VISUALTHINK-VLA conditions a frozen VLA backbone on sparse, task-adaptively routed visual evidence states and provides VisualEvidence-Kit for route supervision and audit.

  • Results

    Across manipulation, simulation, long-horizon, and closed-loop evaluations, VisualThink-VLA improves the success-latency trade-off while selecting only a small number of evidence channels.

  • Takeaways & Limitations

    Compact structured evidence in visual space is a more practical intermediate reasoning substrate for embodied control than long textual reasoning traces.

  • Takeaways & Limitations

    The evaluation covers several public benchmarks and a focused real-robot tabletop suite, while tactile, force, audio, memory, and broader deployment settings remain untested.

Abstract

from arXiv · show

Recent work has begun to equip vision-language-action (VLA) policies with explicit intermediate reasoning. In embodied control, however, textual chain-of-thought is a poor fit: irrelevant or weakly textual information can interfere with action prediction, while autoregressive text decoding adds too much latency for real-time closed-loop execution. We present VISUALTHINK-VLA, a visual intermediate-reasoning framework for accurate, low-latency VLA policies. Our bootstrapping philosophy is to guide action with effective visual thinking: VISUALTHINK-VLA bootstraps action prediction through a compact visual-evidence interface that preserves spatial precision while avoiding decoding overhead. Besides, to further improve performance and efficiency, VISUALTHINK-VLA adopts a tailored selective routing mechanism to learn the visual evidence tokens, enabling low-latency inference while preserving high-capacity specialization. We also introduce VisualEvidence-Kit, a supervision-and-audit resource centered on a VisualEvidence-Agent that constructs a 754.7k VLA instructions VisualEvidence-Set for route supervision and counterfactual faithfulness tests. Across multiple benchmarks and real-robot evaluation, VISUALTHINK-VLA achieves the highest success rate on most benchmarks while reducing the multi-second latency of reasoning-augmented baselines to the sub-second regime. For example, on BridgeData V2, it reduces step latency from 8.377,s with ECoT to 0.367,s, achieving a 22.8 times speedup.

1 Introduction

VISUALTHINK-VLA addresses the accuracy-efficiency limits of reasoning-augmented VLA policies with a sparse, routed visual-evidence interface for frozen backbones. It combines adaptive evidence orchestration with route-grounded supervision and audit resources to improve control while keeping reasoning lightweight.

  • Motivation: Existing reasoning-augmented VLAs face an accuracy-efficiency trade-off because textual chain-of-thought is weakly grounded and autoregressive decoding is costly, while dense visual side information can interfere.These limitations motivate a minimal interface that preserves decision-relevant visual structure without long text traces or dense always-on information.
  • Visual intermediate-reasoning interface: VisualThink-VLA conditions action decoding on compact routed visual cues, enabling frozen VLAs to think with images without long text traces or dense auxiliary perception.The module is designed as a general plug-and-play visual reasoning interface while leaving the underlying backbone unchanged.
  • Evidence orchestration: Task-adaptive routing selects relevant evidence channels, while soft-hard collaborative optimization and FULLSOFT teacher distillation stabilize sparse inference.Channel screening removes two low-utility channels, leaving a four-channel dense teacher that informs the sparse policy.
  • Supervision and audit: 754.7k visual-thinking VLA instructions form VisualEvidence-Set, built by VisualEvidence-Agent through evidence extraction, channel-grounded trace construction, and human review.The resource supports route supervision and faithfulness diagnostics across task stages, action primitives, evidence dependence, and difficulty.
  • Evaluation: Across real-world manipulation, simulation, long-horizon tasks, and closed-loop execution, VisualThink-VLA improves the success-latency trade-off over reasoning-augmented VLA baselines.It returns textual-reasoning latency to the sub-second control regime while preserving nearly the performance of dense visual evidence with few selected channels.

2 Related Work

Prior VLA research builds on language-conditioned robot policies and increasingly adds explicit reasoning or stronger grounding. Related visual-control work motivates object-centric, geometry-aware evidence and image-based intermediate reasoning for spatial decisions.

  • VLA Reasoning Policies: Representative VLA systems include OpenVLA, Octo, GR-2, and π0, building on language-conditioned robot learning over Open X-Embodiment and RT-X.
  • VLA Reasoning Policies: Recent reasoning-augmented VLA methods add explicit intermediate reasoning or stronger grounding, including ECoT for textual reasoning traces.
  • Visual Evidence and Routing: Depth and segmentation enter the initial visual-cue candidate bank but are screened out from the default operational setting.
  • Visual Evidence and Routing: Object-centric and geometry-aware representations support robot learning and visuomotor control, while image-based intermediate reasoning may better suit spatial decisions than text-space planning.

3 Method

VisualThink-VLA replaces textual intermediate reasoning with a compact, routed visual-evidence interface for action prediction. It combines selective routing, lightweight visual-state composition, blended training, and distillation while keeping auxiliary supervision training-only.

  • Visual evidence interface: VisualThink-VLA avoids prompt-text evidence injection by conditioning the frozen VLA backbone on learned visual evidence states.The interface preserves visual intermediate reasoning without free-form textual rationales.
  • Evidence channels: The evidence bank has six candidate channels; two low-utility channels are screened out, leaving four operational channels for task-adaptive routing.The channels encode bbox location, edge geometry, motion, instruction-grounded relations, monocular depth, and object regions as compact vectors.
  • Selective routing: At inference, the router activates only selected channels, exposing the action decoder to routed evidence states rather than the full operational bank.This selective interface is the method’s main efficiency mechanism and uses the model’s own router prediction at inference.
  • Visual state composition: The Visual State Composer projects routed channel vectors into learned conditioning states before the frozen VLA backbone decodes actions.It adds a lightweight learned interface without online image editing or textual-rationale generation.
  • Training objectives: During training, blended routing improves robustness, while logits distillation transfers dense four-channel FullSoft capacity to the sparse route with λdistill = 0.2 and τ = 1.5.VisualEvidence-Kit route targets and auxiliary route/trace supervision regularize training, but these components are not used at inference.

4 VisualEvidence-Kit

VisualEvidence-Kit is a route-grounded supervision and audit resource comprising a VisualEvidence-Agent and the resulting VisualEvidence-Set. The kit generates governed visual-evidence instructions for route supervision, training, and counterfactual faithfulness auditing while avoiding inference-time dependence on agent traces.

  • Kit overview: VisualEvidence-Kit couples a VisualEvidence-Agent with a VisualEvidence-Set to derive evidence instructions from manipulation trajectories and support route supervision and counterfactual auditing.The resource is designed for policies that reason through sparse visual evidence.
  • Agent pipeline: The VisualEvidence-Agent transforms raw frames and trajectory metadata into governed route-grounded instructions through evidence extraction, route and utility assessment, trace construction, and human review.The pipeline produces route targets, utility ranks, structured traces, and filtered labels.
  • Agent pipeline: Human review verifies instruction consistency and filters unreliable labels, separating noisy supervision from high-confidence training or audit targets.The resulting supervision remains aligned with VisualThink-VLA’s visual interface, while traces are reserved for training and audit.
  • Dataset structure: Each VisualEvidence-Set instruction contains observations, context, feature manifests, supervised route targets, counterfactual channel utilities, and channel-grounded trace fields.Instructions are organized by scene domain, action category, manipulation stage, and routed visual evidence type.
  • Dataset subsets: 754.7k instructions comprise the Gold-Faithfulness subset for high-reliability counterfactual audit experiments, alongside Full-Clean and HQ-Trace subsets for statistics, training, and refinement.Full-Clean supports corpus statistics and weighted training, while HQ-Trace retains reliable structured traces.
  • Training and audit: VisualEvidence-Set supplies route and trace supervision during training, but neither the agent nor trace fields is required at inference; diagnostics test grounding in actually routed evidence.The audit evaluates whether visual thinking reflects routed evidence rather than a post-hoc explanation.

5 Experimental Setup

The experimental setup evaluates VisualThink-VLA across five multi-dataset control benchmarks, a route-grounded audit resource, and a separate real-robot tabletop deployment. It compares textual, image-grounded, and method-family baselines using success, latency, completion-time, and audit metrics.

  • Benchmark Roles: The multi-dataset control benchmark covers BridgeData V2, Fractal, RoboTurk, LIBERO, and UT Austin MUTEX.VisualEvidence-Kit supplies the route-grounded audit layer, while real-robot evaluation uses a separate tabletop platform.
  • Baselines: External comparisons include textual traces, image-grounded reasoning, and stronger reasoning-policy variants, alongside BaseVLA, FullSoft, and VisualThink-VLA.The method-family comparison reports a frozen BaseVLA baseline, dense FullSoft reference, and sparse VisualThink-VLA variant.
  • Compared Systems: BASEVLA-only uses a frozen 7B-parameter OpenVLA backbone without external evidence, whereas FULLSOFT enables all four channels and VisualThink-VLA activates a sparse routed subset.The sparse variant uses a task-adaptive router to select channels per sample.
  • Real-Robot Protocol: The real-robot protocol uses a desk-mounted PIPER NERO (7F) 7-DoF arm with a fixed external RGB camera and controller-mediated executable commands.Tasks include language-specified multi-object pick-place, relation-sensitive placement, contact-sensitive reorientation, and two-stage compositional manipulation.
  • Metrics: Primary evaluation reports success_rate, avg_- step_latency_s, avg_completion_time_s, and a route-grounded VisualEvidence-Kit audit score.avg_- step_latency_s is batch-1 wall-clock observation-to-action time after warm-up.

6 Results and Analysis

VisualThink-VLA delivers a favorable accuracy-latency trade-off across control benchmarks, preserves gains across multiple VLA backbones, and attributes improvements to its structured, selectively routed visual-evidence interface. It achieves sub-second inference while retaining strong success performance and avoiding the overhead of textual or dense evidence reasoning.

  • Primary Control Comparison: VisualThink-VLA improves success on seven of eight benchmarks and reduces BridgeData V2 latency from 8.377 s with ECoT to 0.367 s while improving success.Compared with dense FullSoft, it preserves comparable accuracy while reducing latency on every benchmark.
  • Primary Control Comparison: Textual reasoning baselines improve selected tasks but remain outside the sub-second regime, whereas routed visual evidence stays near the low-latency OpenVLA-family region.Figure 5 presents these relationships as success-latency frontiers and shows that sparse routing preserves most dense visual-evidence gains.
  • Backbone Portability: Test-set gains are preserved across all three representative VLA backbones with only modest latency increases.The result supports VisualThink-VLA as a general visual reasoning layer rather than an OpenVLA-specific modification.
  • Internal Interface Comparison: 90.10% average success versus 89.83% for FullSoft, while latency falls from 0.470 s to 0.395 s with VisualThink-VLA.Prompt-text evidence improves over BaseVLA but remains slow, while heavy dense evidence uses all six candidate channels and has higher latency with lower average success.
  • Internal Interface Comparison: The initial six-channel evidence bank can be reduced to a four-channel operational subset without losing the main control gains.Dense screened candidates do not justify their extra runtime cost, and prompt-text evidence partially recovers performance while departing from visual intermediate reasoning.

Task family Evidence Metric BASEVLA FULLSOFT VISUALTHINK-VLA

On real-robot tabletop tasks, VISUALTHINK-VLA outperforms BASEVLA across all four task families and improves over FULLSOFT on three, while remaining more efficient. VisualEvidence-Kit audits show strong route-grounded visual evidence and phase-dependent routing rather than fixed channel selection.

  • Real-Robot Evaluation: VISUALTHINK-VLA outperforms BASEVLA on all four task families and improves over FULLSOFT on three.The exception is contact-sensitive reorientation, where always-on four-channel dense evidence is slightly stronger.
  • VisualEvidence-Kit Analysis: 0.840 route-grounded audit score and 0.984 utility overlap make VISUALTHINK-VLA strongest on the held-out audit set, versus 0.391 for prompt-text evidence.Prompt-text evidence exposes no visual route.
  • VisualEvidence-Kit Analysis: Stage-wise routing changes evidence use with manipulation phase rather than selecting a fixed shortcut channel.BBox dominates during approach, motion rises sharply at grasp, and edge remains consistently active.

7 Ablation Studies

Ablations show that complementary visual evidence channels and sparse orchestration jointly improve VLA performance and efficiency. Structured, channel-grounded supervision primarily strengthens interpretability and faithfulness over unconstrained rationales.

  • Feature Ablations: Removing any retained evidence channel reduces success, with BridgeData V2 most sensitive to bbox, motion, and relation and LIBERO-Long to edge removal or narrow pairs.The task-dependent drops indicate that the routed evidence bank is complementary rather than dominated by one cue.
  • Orchestration Ablations: Soft-hard collaborative routing raises success from 82.86% to 84.19% on BridgeData V2 and from 85.62% to 93.82% on LIBERO-Long.These gains are measured against hard sparse masks.
  • Orchestration Ablations: Teacher-student optimization further raises success to 89.46% on BridgeData V2 and 94.87% on LIBERO-Long, while adaptive routing selects fewer channels at comparable success.The task-adaptive router is comparable to the sequence-text gate while improving efficiency.
  • Interpretability Ablations: Removing route supervision reduces route alignment from 0.929 to 0.758, while removing trace supervision reduces utility mention from 0.984 to 0.738, with moderate success drops.These supervision components mainly improve faithfulness rather than raw success.
  • Interpretability Ablations: Replacing channel-grounded targets with free-form rationales lowers route alignment to 0.052 and utility mention to 0.039, supporting structured, inspectable supervision.The interpretability ablations favor channel-grounded supervision over unconstrained explanatory text.

8 Conclusion

VisualThink-VLA introduces a sparse image-grounded evidence interface for frozen VLA policies that reduces control-time overhead while retaining much of dense evidence’s benefit and supporting real-robot closed-loop control. Its results indicate that compact structured visual evidence is more practical for embodied control than long textual reasoning traces.

  • Conclusion: VisualThink-VLA introduces a sparse image-grounded evidence interface for frozen VLA policies.The interface provides structured visual evidence for action prediction without changing the frozen policy backbone.
  • Conclusion: The routed OpenVLA variant preserves much of dense soft evidence’s benefit while reducing control-time overhead.Selective routing keeps visual evidence compact while retaining substantial performance benefits.
  • Conclusion: The routed variant remains effective in a real-robot closed-loop setting.The conclusion reports effectiveness beyond benchmark evaluation, under embodied closed-loop control.
  • Conclusion: Compact structured evidence in visual space is presented as a more practical intermediate reasoning substrate than long textual reasoning traces for embodied control.The conclusion contrasts visual-space evidence with lengthy textual reasoning for practical embodied-control use.

9 Limitations · Appendix · A VisualEvidence-Kit Workflow

The paper identifies scope limitations in its visual sensing and evaluation coverage, then documents the VisualEvidence-Kit’s governance workflow, evidence construction, and progressively stricter trace subsets. The appendix consolidates implementation, evaluation, qualitative, and audit details supporting the VisualEvidence-Set.

  • 9 Limitations: The implementation currently emphasizes visual channels for manipulation, leaving tactile, force, audio, and longer-term memory signals for future extensions.The evaluation also covers several public benchmarks and a focused real-robot tabletop suite, while broader embodiments, workspaces, and longer-horizon tasks remain future tests of generality.
  • 9 Limitations: Broader deployments across more embodiments, workspaces, and longer-horizon tasks would further test the generality of the routed evidence interface.This limitation accompanies the current evaluation scope of several public benchmarks and a focused real-robot tabletop suite.
  • Appendix: The appendix documents VisualEvidence-Set governance, evidence-channel construction, the real-robot task suite, and qualitative and audit details.These materials provide supporting implementation and evaluation details.
  • A VisualEvidence-Kit Workflow: The VisualEvidence-Agent derives candidate evidence channels, aggregates route targets and utility signals, then sends records through manual review and annotation before exporting governed benchmark entries.The exported entries include channel-grounded trace fields.
  • A VisualEvidence-Kit Workflow: Full-Clean removes invalid or inconsistent records for broad statistics and weighted training, HQ-Trace retains higher-quality traces for refinement, and Gold-Faithfulness supports high-confidence counterfactual audits.These subsets form progressively stricter governance levels for raw route-grounded traces.
  • A VisualEvidence-Kit Workflow: The six-channel candidate bank is extracted offline before policy inference, with bbox encoding coarse object location and target extent and edge capturing contours and local shape discontinuities.bbox uses open-vocabulary detection with Grounding DINO / OWL-ViT-style localization, while edge uses SAM2 mask proposals and boundary rasterization.

B Real-Robot Task Suite … G AI Assistant Use

The supplementary sections detail the real-robot task suite, routed-evidence case studies, VisualEvidence-Kit auditing, human review, artifact governance, and AI-assistant use. Together, they specify the deployment setting, evidence analysis, review practices, dataset constraints, and preparation support.

  • B Real-Robot Task Suite: The real-robot suite evaluates target localization under distractors, instruction-grounded spatial relations, contact-sensitive pose change, and stage-dependent evidence.The platform is a desk-mounted PIPER NERO 7-DoF arm with a fixed external RGB camera.
  • C Qualitative Case Studies: In lever rotation, relation evidence identifies the lever orientation and enables the correct downward rotation instead of the base policy’s wrong direction.The routed visual evidence changes action-relevant perception in a representative closed-loop case.
  • C Qualitative Case Studies: In bowl placement, edge evidence sharpens the contact and grasp boundary, illustrating how routed evidence supports manipulation-relevant perception.The case studies are presented as qualitative examples of routed visual evidence in representative closed-loop cases.
  • D VisualEvidence-Kit Audit and Faithfulness Details: VisualEvidence-Agent builds the VisualEvidence-Set for trace-quality summaries, while the kit audits route exposure and evidence overlap with useful channels.The audit layer complements stage-conditioned routing visualizations and uses the axes stage, primitive, evidence dependence, and difficulty.
  • E Human Review Protocol: Human reviewers checked route-grounded records, evidence-channel labels, and trace quality using task-level guidelines for VLA supervision, audit, and diagnostic evaluation.Reviewers were laboratory students familiar with robot manipulation; no crowdworking platform or monetary payment was used.
  • E Human Review Protocol: The review collected no personal data or human-subject behavioral data, so no separate ethics-board protocol was sought.Reviewed content consisted of robot observations, task instructions, generated evidence labels, and audit fields.
  • F Artifact Licenses: External datasets, VLA backbones, comparison systems, and perception tools are used under their original release conditions for evaluation, comparison, adaptation, and auxiliary evidence derivation.VisualEvidence-Kit and VisualEvidence-Set are intended only for VLA supervision, audit, and diagnostic evaluation.
  • F Artifact Licenses: The work contains robot observations, task instructions, routed evidence labels, and audit records, but no personal conversations, user profiles, human-subject records, or identifying information.The dataset also introduces no offensive-content annotations or offensive textual content.
Loading 2605.30011v1…