PaperlayerView exact revision on arXiv

Source-linked AI summary

IVRA: Improving Visual-Token Relations for Robot Action Policy with Training-Free Hint-Based Guidance

Jongwoo Park, Kanchana Ranasinghe, Jinhyeok Jang, Cristina Mata, Yoo Sung Jang, Michael S Ryoo

arXiv:2601.16207v2cs.RO

TL;DR

Flattening visual patches into a 1D token sequence weakens spatial cues needed for precise manipulation. IVRA uses encoder-derived affinity hints injected into selected language-model layers at inference time, improving diverse VLA baselines across simulated and real-robot tasks.

  • Problem

    Flattening visual patches into a 1D sequence weakens local correlations, blurs object boundaries, and makes fine-grained spatial relationships harder to recover for manipulation.

  • Method

    IVRA injects patchwise affinity information from the built-in vision encoder into selected visual tokens in a language-model layer, leaving text tokens untouched and requiring no retraining.

  • Results

    IVRA consistently improves diverse VLA baselines across 2D and 3D simulated benchmarks and real-robot tasks, including a 1.1% increase in OpenVLA’s LIBERO overall average.

  • Takeaways & Limitations

    The method provides a lightweight, training-free enhancement for preserving instance-level cues important to object localization, grasping, placement, and multi-step manipulation.

Abstract

from arXiv · show

Many Vision-Language-Action (VLA) models flatten image patches into a 1D token sequence, weakening the 2D spatial cues needed for precise manipulation. We introduce IVRA, a lightweight, training-free method that improves spatial understanding by exploiting affinity hints already available in the model's built-in vision encoder, without requiring any external encoder or retraining. IVRA selectively injects these affinity signals into a language-model layer in which instance-level features reside. This inference-time intervention realigns visual-token interactions and better preserves geometric structure while keeping all model parameters fixed. We demonstrate the generality of IVRA by applying it to diverse VLA architectures (LLaRA, OpenVLA, and FLOWER) across simulated benchmarks spanning both 2D and 3D manipulation (VIMA and LIBERO) and on various real-robot tasks. On 2D VIMA, IVRA improves average success by +4.2% over the baseline LLaRA in a low-data regime. On 3D LIBERO, it yields consistent gains over the OpenVLA and FLOWER baselines, including improvements when baseline accuracy is near saturation (96.3% -> 97.1). Code and visualizations are available at: jongwoopark7978.github.io/IVRA

I. INTRODUCTION

VLA models often flatten visual patches into 1D sequences, weakening spatial and instance-level cues needed for precise manipulation. IVRA restores these cues by injecting encoder-derived affinity hints into language-model layers without modifying the main pipeline.

  • I. INTRODUCTION: Flattening 2D patch grids into 1D visual-token sequences weakens local correlations and can blur object boundaries.This makes object boundaries, color-shape relations, and fine-grained spatial relationships harder to recover for manipulation.
  • I. INTRODUCTION: IVRA extracts local-similarity affinity maps from the model’s encoder and injects them into deeper language-model layers at inference time.The hints reweight flattened visual tokens according to spatial correlations without modifying the main components.
  • I. INTRODUCTION: IVRA improves instance-level recognition and localization across simulated and real-world manipulation tasks.Real-world pick-and-place tasks especially require discriminating precise boundaries and attributes such as color and shape.
  • I. INTRODUCTION: The approach avoids large-scale retraining, specialized data collection, and external spatial modules while applying across diverse VLA systems.The introduction positions encoder-derived affinity hints as a lightweight way to recover local structure in existing architectures.

B. Affinity Hints and Instance-Level Feature Enhancement

Prior multimodal methods preserve spatial information through specialized training, architectural changes, or auxiliary components. IVRA instead uses internal vision-encoder correlations to enrich object-level detail during inference while retaining semantic alignment.

  • B. Affinity Hints and Instance-Level Feature Enhancement: Affinity-based multimodal methods emphasize object boundaries and spatial configurations by incorporating patch-wise correlations into token sequences.Related approaches include hint tokens and visual prompts for highlighting regions or resolving object references.
  • B. Affinity Hints and Instance-Level Feature Enhancement: Robotic systems such as CLIPort bridge semantic and positional information, but commonly rely on extra task-specific training or specialized architecture modifications.This contrasts with IVRA’s lightweight integration into existing VLA pipelines.
  • B. Affinity Hints and Instance-Level Feature Enhancement: IVRA requires no external affinity module and linearly mixes original features with affinity-pooled features to preserve semantics while enriching object-level detail.Its training-free design differs from strategies that retrain complete pipelines on large-scale datasets.

III. METHODOLOGY

IVRA restores 2D spatial structure by deriving affinity hints from a frozen vision encoder and integrating affinity-guided pooling into selected language-model layers. The method represents patch connectivity compactly through pairwise feature similarity.

  • III. METHODOLOGY: IVRA derives affinity hints from affinity maps extracted from the VLA’s frozen vision encoder.Patch features are taken from an intermediate encoder layer to capture local relationships more effectively.
  • III. METHODOLOGY: For an image divided into N patches, the method computes an N × N affinity matrix from pairwise normalized patch-feature similarity.Higher affinity indicates that two patches likely belong to the same object or share visual similarity.
  • III. METHODOLOGY: The affinity map acts as a patchwise connectivity prior that retains 2D spatial layout in a compact representation.This prior supplies local structure that is weakened when visual patches are flattened into a sequence.

B. Affinity-Guided Visual Token Pooling

IVRA applies affinity-guided pooling to flattened visual tokens immediately before a selected language-model self-attention block. It updates only visual tokens, mixing each with visually correlated neighbors to restore local coherence.

  • B. Affinity-Guided Visual Token Pooling: The method identifies the indices of visual tokens inserted at the <image> position and leaves text tokens untouched.This confines the intervention to the visual-token subsequence within the language-model input stream.
  • B. Affinity-Guided Visual Token Pooling: Immediately before self-attention, IVRA refines each visual token by mixing neighboring tokens according to affinity scores.Strongly correlated patches reinforce one another’s features through weighted pooling.
  • B. Affinity-Guided Visual Token Pooling: Affinity-guided token mixing restores some 2D structure lost during flattening while preserving the unchanged textual-token pathway.The operation reweights visual-token contributions using patchwise spatial correlations.

C. Integration into VLA Models

IVRA integrates affinity-guided pooling and token mixing into selected LLM layers during inference, while leaving text tokens and the base model unchanged. In low-data VIMA evaluation, LLaRA+IVRA consistently outperforms LLaRA across the four tasks.

  • Integration: Affinity-guided pooling updates flattened visual tokens using patch similarities before self-attention in selected LLM layers.The pipeline starts from frozen vision-encoder patch embeddings, flattens them into LLM tokens, and applies weighted pooling at chosen layers.
  • Integration: Token mixing forms each final visual token by linearly blending its pooled and original representations.This convex combination injects object-aware evidence while preserving the original token’s semantics.
  • Integration: The updated visual tokens continue through layer normalization, self-attention, and later LLM transformations, while text tokens remain unmodified.Output representations are then used to generate policy actions or textual responses, depending on the VLA setup.
  • VIMA evaluation: LLaRA uses 80k trajectories, approximately 12% of VIMA’s 660k expert trajectories, and evaluates each task over 20 randomized seeds.The four partitions are Novel Task, Novel Object, Object Combination, and Object Place; Novel Task is de-emphasized because of missing original-dataset data.
  • VIMA results: LLaRA+IVRA consistently improves over LLaRA and outperforms both LLaRA and VIMA across the four VIMA generalization tasks.The table reports average success rates and characterizes the result as robust instance-level generalization.

B. LIBERO Simulated Environment

On LIBERO’s 3D manipulation suites, IVRA improves OpenVLA and FLOWER across architectures, task categories, and baseline accuracy regimes. The gains include average-success improvements for OpenVLA and FLOWER even when baseline performance is near saturation.

  • Benchmark: LIBERO evaluates 3D language-conditioned manipulation across Goal, Object, Spatial, and Long suites, with LIBERO-90 additionally used for FLOWER comparisons.The suites stress target variation, object-centric manipulation, spatial relations, and multi-step temporal composition.
  • OpenVLA: OpenVLA+IVRA raises overall LIBERO average success from 76.5% to 77.6%, a +1.1% improvement without retraining or base-model modification.IVRA also surpasses Diffusion Policy and Octo by +5.2% and +2.5% in average success under the same protocol.
  • FLOWER: FLOWER+IVRA improves Task-90 from 93.4% to 96.0%, a +2.6% gain, and Task-Object from 99.3% to 99.9%, a +0.6% gain.The method is applied plug-and-play with identical baseline hyperparameters and no additional training.
  • FLOWER: FLOWER’s overall success rises from 96.3% to 97.1%, a +0.8% improvement despite near-saturated baseline accuracy.The reported gains are consistent across every evaluated setting.
  • Generalization: Across VIMA and LIBERO, IVRA generalizes across 2D and 3D inputs, OpenVLA and FLOWER architectures, and mid-50% to high-90% baseline regimes.All gains use a lightweight inference-time modification with no retraining.

C. Real World Environment

The real-world evaluation uses a calibrated robot arm and fixed overhead camera to test zero-shot manipulation on four novel-object tasks with increasing attribute and localization demands.

  • Environment: The setup uses a gripper-equipped robot arm under a fixed RGB camera, with image-to-action mapping established by calibration.The policy was trained purely on synthetic data and evaluated on four real-world tasks.
  • Tasks: T1 tests selecting a specified object and placing it fully inside a pan.The target is chosen from nine toy items randomly placed on a tabletop.
  • Tasks: T2 tests identifying an object matching a reference toy’s color before placing it in the pan.The reference color is selected from yellow, orange, and green.
  • Tasks: T3 tests target localization under clutter, where touching neighboring distractors constitutes failure.Distractors are placed close to the target horizontally and vertically.
  • Tasks: T4 tests comparative height understanding by requiring selection of a short or long object.Multiple objects with varying lengths are randomly arranged in the scene.
  • Results: +10% on T1 and up to +30% on T2-T4: LLaRA+IVRA outperforms vanilla LLaRA across all four real-world tasks.The evaluation uses a low-data zero-shot setting with LLaRA pretrained on only 1.2% of VIMA training data.

D. Qualitative Results

Qualitative affinity maps show that IVRA produces more coherent object-level visual representations than the baseline across real and simulated tasks.

  • Visualization: Figure 1-(b) compares input images with affinity maps before and after IVRA, using a red reference point and brighter regions for stronger affinity.The visualization links improved map coherence to object-level recognition.
  • Affinity maps: Before IVRA, baseline affinity maps are noisy or incomplete, especially for similarly colored objects or reference points on edges.These maps make object boundaries difficult to delineate.
  • Affinity maps: After IVRA, affinity maps more clearly highlight individual objects instead of scattering activations through the background.The finer-grained delineation appears in both real and simulated environments.
  • Performance impact: Sharper boundary recognition supports precise interactions such as grasping and placement, including a +30% increase over LLaRA in real-world T2.The largest VIMA gains occur on the more difficult Novel Task, where object-specific token features are most essential.

V. VISUALIZATION OF REAL WORLD EXPERIMENTS

Real-world trajectory visualizations compare LLaRA+IVRA with original LLaRA across one moderate and three challenging manipulation tasks, alongside reported overhead and ablation summaries.

  • Trajectory comparisons: Figures 4a, 4b, 5a, and 5b place LLaRA+IVRA trajectories in the top row and original LLaRA trajectories in the bottom row.The figures cover T1 through T4, with T1 moderate and T2-T4 challenging.
  • Trajectory comparisons: In T1, both policies pick up the correct object, while the other tasks examine color matching, cluttered localization, and relative height.These visualizations emphasize differences in instance-level selection and localization demands.
  • Interpretation: The real-world and simulation results together indicate stronger instance-level understanding and more accurate localization and attribute-based selection with IVRA.Reported gains are particularly associated with Localization, Object Height, and Object Color Matching tasks.
  • Overhead: Table IV reports marginal runtime overhead and no additional parameters for IVRA.The provided passage identifies the table as a summary of runtime and parameter overhead.
  • Ablations: Table V summarizes ablations over layer placement, consecutive-layer count, within-layer location, and token-mixing coefficient across four VIMA tasks.The blue row marks the final chosen setting.

VI. COMPLEXITY ANALYSIS & ABLATIONS

IVRA adds little computational overhead, while ablations identify a moderate affinity-token mixture as the most balanced setting across VIMA tasks.

  • Complexity: IVRA adds 3% latency and no parameters when tested on LLaRA with one NVIDIA RTX 6000 Ada GPU.The method does not use external modules.
  • Ablation design: The ablation varies affinity pooling placement, consecutive layers, within-layer location, and token-mixing coefficient across NT, NO, OC, and OP.NT, NO, OC, and OP denote Novel Task, Novel Object, Object Combination, and Object Placement.
  • Token mixing: λ = 0.3 achieves the highest overall average and strongest NO result at 61.3%, while remaining competitive on OP/OC at 73.1%/70.4%.The authors select λ = 0.3 as the balanced setting.
  • Token mixing: Larger λ values slightly benefit NT at 30% but degrade NO and OC, whereas λ = 0 without affinity injection is uniformly worse.The reported λ = 0 range is 22.5–69.6%.

VII. CONCLUSION

IVRA restores spatial structure in VLA models through a lightweight, training-free inference-time intervention. Across 2D and 3D benchmarks and real-robot tasks, it improves strong VLA baselines as a practical drop-in enhancement.

  • IVRA injects encoder-derived affinity hints into a selected intermediate language-model layer to reweight flattened visual tokens using patchwise correlations.This inference-time technique preserves instance-level cues such as object boundaries and attribute relations.
  • IVRA consistently improves LLaRA, OpenVLA, and FLOWER across VIMA, LIBERO, and real-robot tasks.The reported evaluation spans both 2D and 3D benchmarks as well as real robotic settings.
  • The method provides a practical drop-in enhancement for fine-grained grounding and action generation in multimodal robot policies.
  • The paper supports faithful replication through detailed methodology, experimental setup, evaluation protocols, and publicly available models and code.
Loading 2601.16207v2…