Source-linked AI summary
DIVA: Dual-Space Intent-Aware Visual Attenuation for Vision-Language-Action Policies
Kaixi Feng, Guoheng Sun, Ziyao Wang, Yexiao He, Zheyu Shen, Ang Li
TL;DR
Dense-token VLA policies preserve scene context but lack an explicit mechanism to regulate visual-token influence. DIVA anchors patch relevance using task intent and visual evidence, then attenuates visual representations before and within the backbone. It improves OpenVLA-OFT across simulation and real-world settings, with 98.0% average success on LIBERO.
Problem
Dense-token VLA policies lack an explicit mechanism to regulate how strongly individual visual tokens influence policy computation.
Source: Introduction · p. 1Method
DIVA combines task intent with visual evidence to estimate patch-wise relevance anchors and applies them to projected tokens and visual hidden states in two spaces.
Source: Introduction · p. 1Results
DIVA improves OpenVLA-OFT across LIBERO, zero-shot LIBERO-Plus, and real-world manipulation tasks, reaching 98.0% average success on LIBERO.
Source: Introduction · p. 1, Evaluation Results · p. 5Takeaways & Limitations
The results support non-destructive visual-state strength control as a practical design for robust VLA policies.
Source: Conclusion · p. 7A single fixed relevance budget may be suboptimal when task-relevant evidence varies in scale, clutter, or spatial extent across observations.
Source: Conclusion · p. 7
Abstract
from arXiv · showhide
Vision-language-action (VLA) policies typically feed dense visual patch tokens into a language-action backbone, preserving scene context but offering no explicit mechanism to regulate how strongly different visual tokens influence policy computation. We introduce DIVA, a Dual-Space Intent-Aware Visual Attenuation module with an anchor-then-attenuate design. DIVA combines high-level task intent with low-level visual evidence to estimate patch-wise relevance anchors, then applies them in two complementary spaces: it reweights projected visual tokens before backbone entry and persistently attenuates low-relevance visual states within the backbone. DIVA preserves the full visual token sequence and requires no external grounding supervision. On LIBERO, DIVA improves OpenVLA-OFT from 96.6% to 98.0% average success and raises its zero-shot LIBERO-Plus score from 69.6 to 72.6. Real-world experiments further show consistent gains under task-irrelevant visual perturbations, supporting the robustness of intent-aware visual attenuation beyond simulation.
Introduction
Dense-token VLA policies preserve scene information but lack an explicit way to regulate visual-token influence. DIVA estimates patch-wise relevance from task intent and visual evidence, then softly attenuates representations before and within the backbone; it improves the baseline across evaluated settings.
Introduction
- Dense-token VLA interfaces lack an explicit mechanism to regulate visual-token influence, allowing task-irrelevant objects to impair action prediction.The resulting question is how to attenuate weakly relevant evidence while preserving context needed for manipulation.Source: Introduction · p. 1
- Soft attenuation reduces weak tokens’ contribution without removing them, since peripheral evidence may still support spatial reasoning, obstacle awareness, or safe interaction.The aim is graded influence across visual tokens rather than a binary scene partition.Source: Introduction · p. 1
- Input-space attenuation initializes patch-aligned strength, while latent-space attenuation sustains it as later backbone operations may dilute a one-time adjustment.The relevant later operations include attention, feed-forward transformations, and residual updates.Source: Introduction · p. 1
- DIVA combines task intent and low-level visual evidence to estimate patch-wise anchors, then attenuates representations in input and latent spaces.It preserves the full token sequence and requires no external grounding labels.Source: Introduction · pp. 1–2
- DIVA improves the baseline across LIBERO, zero-shot LIBERO-Plus, and real-world manipulation tasks.Controlled ablations validate the complementary roles of high- and low-level evidence in anchoring and of input- and latent-space attenuation.Source: Introduction · p. 1
Related Work
Related work guides visual relevance through explicit localization, implicit attention-side modulation, or token selection. DIVA instead derives anchors from policy representations and softly attenuates the full visual sequence.
Explicit grounding and input-side localization
- Explicit grounding can use detectors, masks, boxes, crops, or prompts [34]; localization errors or narrow crops may discard context needed for spatial reasoning and collision avoidance.DIVA instead derives relevance anchors from the policy’s own language and visual representations.Source: Related Work · p. 2
Implicit grounding and attention-side visual modulation
- Implicit methods guide visual focus through attention or auxiliary supervision [24], [19], [29] [32], but attention-side control may be reshaped by later feed-forward and residual updates.Gaze-supervised variants also require additional annotations.Source: Related Work · p. 2
Visual token pruning and selection
- Token pruning can improve efficiency, [9], [12], [20] [26], but hard removal may discard weakly salient regions that encode layout, obstacles, or target context.DIVA preserves the full visual sequence and softly attenuates low-relevance representations instead.Source: Related Work · p. 2
Method
DIVA grounds patch-wise relevance anchors in task intent and localized visual evidence, then uses those anchors to attenuate visual tokens before and throughout backbone processing. Its training combines the base action-prediction objective with a relevance-budget loss.
Overview
- DIVA reuses task-conditioned patch relevance anchors to reweight projected visual tokens before backbone entry and persistently attenuate low-relevance hidden states across layers.The visual token layout remains intact, and the anchors require no external grounding labels.Source: Overview · p. 2, Dual-Space Visual Attenuation · p. 4
Grounded Text Pooling
- Grounded Text Pooling weights instruction tokens by localized patch support, distinguishing concepts grounded in a subset of patches from tokens merely compatible with the whole image.It contrasts local responses with a uniform visual baseline before converting grounded scores into pooling weights.Source: Grounded Text Pooling · p. 3
Intent-Aware Visual Relevance Anchoring
- DIVA combines task intent with patch-level visual evidence, then contextualizes task-conditioned patches so each relevance anchor reflects local appearance and relationships among scene elements.Independent sigmoid predictions allow multiple patches spanning a relevant object or interaction region to receive high scores.Source: Intent-Aware Visual Relevance Anchoring · p. 4
Dual-Space Visual Attenuation
- The same anchors control input-token strength once and repeatedly modulate evolving visual states, preserving high-relevance states while retaining reduced, nonzero context for low-relevance states.The latent path uses θ = 0.3 and λ = 0.5 throughout; states above the threshold remain intact.Source: Dual-Space Visual Attenuation · p. 4
Training Objective
- DIVA trains with the base action-prediction loss plus a relevance-budget loss that discourages assigning high relevance to every visual token.The experiments use target average relevance ρ = 0.4 and budget weight η = 0.01.Source: Training Objective · pp. 4–5
Experiments
DIVA improves LIBERO and zero-shot LIBERO-Plus results and completes more trials than OpenVLA-OFT across the reported real-world task–condition pairs. Ablations and analyses examine its anchoring signals, dual-space attenuation, and budget–threshold settings.
Experiments
- Evaluation uses one jointly trained policy across 40 LIBERO tasks, with 2,000 total trials; LIBERO-Plus is evaluated zero-shot on 10,030 variants.Baseline, DIVA, and ablation variants use matched optimization budgets; DIVA activates its modules only during the final 10K simulation-training steps.Source: Experiments · p. 5
Evaluation Results
- 98.0% average success, with DIVA outperforming OpenVLA-OFT across all four LIBERO suites; its task-weighted LIBERO-Plus score rises from 69.6 to 72.6.Gains span all reported perturbation factors, including camera, light, noise, background, layout, and language perturbations.Source: Table 1 · p. 5, Evaluation Results · p. 5, Table 2 · p. 6
Ablation Studies
- Grounded Text Pooling performs best among anchoring variants, while explicit-only and implicit-only attenuation each improve over baseline but fall short of the full design.Visual-only anchoring substantially underperforms task-conditioned variants.Source: Table 3 · p. 6, Ablation Studies · p. 6
Analysis
- 0.4/0.3 budget–threshold settings achieve 98.0% average success; reducing them to 0.2/0.1 lowers success to 96.0%, while 0.8/0.7 yields 97.0%.Across timesteps, anchors shift from object localization toward gripper–target interaction; mean anchors range from 0.36 to 0.46, with 41–54% of visual tokens below threshold.Source: Table 4 · p. 6, Analysis · pp. 6–7, Figure 4 · p. 7
Real-World Experiments
- DIVA completes more trials than OpenVLA-OFT in every evaluated real-world task–condition pair, with higher success under unseen-object distractors and background shifts.Stacking benefits most from unseen-object distractors; the sequential-button task is the most challenging.Source: Real-World Experiments · pp. 5–6, Figure 3 · p. 7
Conclusion
DIVA combines intent-conditioned relevance anchoring with attenuation in input and latent spaces while preserving the full visual token sequence. It improves OpenVLA-OFT under standard and perturbed conditions, though fixed global relevance budgets may not suit every observation.
Conclusion
- DIVA combines intent-conditioned relevance anchoring with dual-space attenuation while preserving the full visual token sequence.Grounded Text Pooling integrates task intent with localized visual evidence, and shared anchors regulate representations entering and persisting through policy computation.Source: Conclusion · p. 7
- Across LIBERO, zero-shot LIBERO-Plus, and physical tasks, DIVA improves OpenVLA-OFT under standard and perturbed conditions.Ablations confirm complementary roles for anchoring and dual-space attenuation.Source: Conclusion · p. 7
- Fixed global relevance budgets may mismatch observations, over-supporting small targets or underrepresenting large objects and extended interaction regions.The mismatch can introduce mild optimization noise; future work will explore input-adaptive budgets.Source: Conclusion · p. 7
A Implementation Details
DIVA adds an anchor-then-attenuate pathway to OpenVLA-OFT, estimating patch relevance and applying shared anchors before and within the backbone. Its two-stage training initializes from a task-trained baseline and uses matched total training budgets.
A Implementation Details
- OpenVLA-OFT uses a shared SigLIP–DINOv2 visual encoder, Llama-2 7B language-embedding space, proprioception projector, and MLP action head.Its bidirectional attention decodes action chunks in parallel; the implementation retains the continuous L1 action objective.Source: A Implementation Details · p. 10
- Grounded Text Pooling weights instruction tokens by localized visual support, then a self-attention block and sigmoid head predict one relevance anchor per visual patch.The task-conditioned sequence uses a 512-wide block with eight attention heads; object or interaction-region patches can receive high relevance simultaneously.Source: A Implementation Details · p. 10
- Input attenuation scales each projected token by its anchor, while latent attenuation reuses anchors at every layer and softly interpolates below-threshold states toward anchor-scaled values.States at or above the threshold remain unchanged, and the full visual-token sequence is preserved.Source: A Implementation Details · p. 10
- DIVA training initializes from task-trained OpenVLA-OFT and matches baseline total budgets: 160K steps in simulation and 30K on the physical robot.Simulation training uses a 150K-step baseline followed by 10K DIVA steps; physical training uses 20K baseline steps followed by 10K DIVA steps.Source: A Implementation Details · p. 10
B Real-World Experiment Details
The physical experiments use synchronized wrist and third-person RGB views with robot proprioception. Four task datasets are converted to the OpenVLA-OFT-compatible RLDS representation and standardized before training.
B Real-World Experiment Details
- Physical experiments use a UFACTORY UF850 arm, with demonstrations collected by VR teleoperation from synchronized wrist and fixed third-person RealSense RGB cameras.Both cameras record at 1920×1080 and 30 FPS; robot transitions are logged at a nominal 10 Hz, and the policy receives both views plus proprioception.Source: B Real-World Experiment Details · p. 10
B.1 Dataset Preparation
- The raw demonstrations preserve both camera images and robot state at every transition before the four task datasets are converted to OpenVLA-OFT-compatible RLDS.The resulting RLDS transition contains two RGB observations, six joint angles, a 7D relative action, and the language instruction.Source: §B.1 · pp. 10–11
- Each camera stream is center-cropped to a square field of view and resized to 224×224 before training.The 7D action comprises TCP translation, wrapped Euler-angle rotation, and a binary gripper command; a no-op filter retains gripper changes.Source: §B.1 · p. 11
B.2 Real-World Task Suites
The real-world task suites span pick-and-place, language-conditioned object selection, and long-horizon action sequencing. Their instructions specify objects, colors, or action order, with several tasks requiring visual-state interpretation.
B.2 Real-World Task Suites
- The task suites span simple pick-and-place, language-conditioned object selection, and long-horizon action sequencing, including picking the instructed cube.Source: §B.2 · p. 11
- The cube-selection task requires identifying the instructed cube among three visually similar candidates and placing it in the cup.Source: §B.2 · p. 11
- The cup-stacking task binds both color expressions to the intended cups before stacking the specified source cup on the target.Source: §B.2 · p. 11
- Sequential button pressing requires pressing three instructed colored buttons in order while inferring progress from the current visual state.The task dataset contains 50 demonstrations.Source: §B.2 · p. 11
- Color-conditioned placement requires picking the instructed object and placing it in the bowl specified by the instruction.The task dataset contains 51 demonstrations.Source: §B.2 · p. 11
B.3 Evaluation Conditions
Physical trials compare task execution across standard and visually shifted conditions, while representative trajectories show how the robot acts across selected tasks.
B.3 Evaluation Conditions
- Each task is tested under the Standard Setup, Unseen Objects, and Unseen Background conditions, with 25 physical trials per task–condition pair.Success requires completing the full instructed task within the evaluation horizon.Source: §B.3 · p. 11
B.4 Execution Trajectories
- Representative trajectories cover cube placement, cup stacking, and button sequencing in the standard setup, plus cup stacking with unseen distractors and backgrounds.Each is shown across available saved queries, up to eight policy-query steps.Source: §B.4 · p. 11
C Simulation Benchmark Details
The simulation evaluation uses the standard 40-task LIBERO split and zero-shot LIBERO-Plus, which adds controlled perturbations to the same four suites.
C.1 LIBERO
- The standard LIBERO evaluation uses 40 tasks across Spatial, Object, Goal, and ten long-horizon tasks from LIBERO-100.LIBERO contains 130 tasks overall, including three controlled ten-task suites and LIBERO-100.Source: §C.1 · p. 11
C.2 LIBERO-Plus
- LIBERO-Plus extends the four suites to 10,030 task variants assigned across seven controlled perturbation factors.The factors alter camera viewpoint, robot state, language, lighting, background, sensor noise, or object layout.Source: §C.2 · p. 11
- The zero-shot protocol trains only on standard LIBERO demonstrations, with no LIBERO-Plus perturbation data used for training or adaptation.Each variant is executed once, and the overall score is weighted across all 10,030 variants rather than averaged across factor scores.Source: §C.2 · pp. 11–12
D Additional Qualitative Results
The qualitative visualizations trace DIVA's relevance and attenuation over policy-query steps, showing how retained regions change with task progress and visual shifts.
Reading the trajectories
- The visualizations pair original RGB observations with explicit input-space anchors and spatial renderings of latent attenuation across sampled policy-query steps.Warm colors indicate larger relevance anchors; lighter latent patches indicate stronger attenuation.Source: D Additional Qualitative Results · p. 12
Qualitative progression
- Across LIBERO trajectories, relevance shifts from candidate objects and targets toward the manipulated object and active interaction region as execution progresses.Long-horizon tasks show corresponding shifts between successive subgoals.Source: D Additional Qualitative Results · p. 12, Figure 5 · p. 13, Figure 6 · p. 14
Interpreting the two attenuation spaces
- Explicit attenuation sets projected visual-token strength before backbone entry, while latent attenuation repeatedly acts only on below-threshold states.The latent view depicts persistent attenuation rather than a second relevance prediction; retained regions include the manipulated object, target, gripper, and active contact region.Source: D Additional Qualitative Results · p. 12
Behavior under visual shifts
- Under unseen objects and backgrounds, instructed objects and interaction regions remain emphasized while distractors and surrounding visual changes are attenuated.The OOD examples depict real-world cup stacking across eight policy-query steps.Source: D Additional Qualitative Results · p. 12, Figure 8 · p. 16
Real-world trajectories
- Real-world standard trajectories show instructed-cube placement, cup stacking, and sequential button pressing across available saved queries.The figure presents RGB, explicit attenuation, and latent attenuation rows.Source: Figure 7 · p. 15