PaperlayerView exact revision on arXiv

Source-linked AI summary

PAIR: Bridging Perception and Action in Vision-Language-Action Models

Kaixi Feng, Guoheng Sun, Ang li

arXiv:2610.09016v1cs.AI

TL;DR

Continuous-action VLA models must bridge visual-language semantics and executable actions, but this transition is often implicit under final-action supervision. PAIR learns a shared intermediate representation by aligning perception features with expert-action structure and using Bridge Tokens to initialize action decoding. Across simulation and real-world evaluations, PAIR improves the evaluated VLA models; across seven real-world tasks, OpenVLA-OFT’s average success rate rises from 51.4% to 65.0%.

  • Problem

    Continuous-action VLAs often leave the transition from visual-language semantics to executable action semantics implicit and supervise it mainly through the final action-prediction loss.

    Source: §1 · p. 2

  • Method

    PAIR aligns task-relevant visual-language features with expert-action latents to form Bridge Tokens, then injects them into action decoding as an intermediate representation.

    Source: §1 · p. 2

  • Results

    51.4% to 65.0%: across seven real-world tasks, PAIR raises OpenVLA-OFT’s average success rate, alongside gains on LIBERO, LIBERO-Plus, and CALVIN.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support explicit perception–action intermediates as a practical design principle for VLA policies.

    Source: §5 · p. 9

Abstract

from arXiv · show

Vision-language-action (VLA) models map visual observations and language instructions to continuous robot actions. This task requires a transition from representations that describe the scene and instruction to representations that support action generation. Many continuous-action VLAs leave this transition implicit and supervise it mainly through the final action-prediction loss. We introduce PAIR, a framework that learns a shared perception-action representation between these two spaces. During training, a Masked Action Autoencoder encodes expert action chunks into horizon-aligned Action Latent Tokens. A Bridge Module extracts task-relevant features from the current visual-language representations. PAIR aligns these features with the Action Latent Tokens to form Bridge Tokens that preserve task information and capture the structure of expert actions. The Bridge Tokens are then projected into the action-token space and injected into the initial Action Tokens, providing an action-ready starting point for Action Expert refinement. At inference, the autoencoder is removed, and the Bridge Tokens are generated only from the current observation and instruction. Experiments on LIBERO, LIBERO-Plus, and CALVIN ABC-D show gains for the evaluated OpenVLA-OFT and VLA-Adapter models. On LIBERO-Plus, PAIR raises VLA-Adapter's success rate from 59.1% to 64.2%. On CALVIN, it increases VLA-Adapter's average completed sequence length from 4.42 to 4.53. Across seven real-world tasks, PAIR raises OpenVLA-OFT's success rate from 51.4% to 65.0%. Representation analyses show that Bridge Tokens retain task information while making continuous-action information accessible before Action Expert refinement. These results support a shared intermediate representation as a useful interface between perception and action in continuous-action VLAs.

1 INTRODUCTION

Continuous-action VLAs must translate visual-language semantics into executable action representations, but this transition is often left implicit. PAIR learns a perception-derived bridge shaped by expert actions and injects it into action decoding.

1 INTRODUCTION

  • The perception–action representation gap arises because visual-language representations describe the world and task, while action representations specify temporally ordered, executable control.Existing VLAs rely primarily on final action supervision, leaving action tokens to discover both which information matters for control and how to organize it into an action-ready representation.Source: §1 · pp. 1–2
  • PAIR aligns perception-derived, task-relevant features with expert-action latents to create Bridge Tokens that combine task semantics with executable action structure.A Masked Action Autoencoder encodes expert action chunks into Action Latent Tokens whose reconstruction objective preserves temporal and executable structure; a Bridge Module extracts features from current visual-language representations.Source: §1 · pp. 2–3
  • PAIR injects projected Bridge Tokens into the initial Action Tokens, giving the Action Expert an action-aware starting point for refinement.The action-decoding process can refine this prior rather than building an action-ready representation from the final action objective alone.Source: §1 · pp. 2–3
  • PAIR improves results across benchmark and real-world settings for OpenVLA-OFT and VLA-Adapter.VLA-Adapter's average success rate rises from 97.3% to 98.4% on LIBERO and from 59.1% to 64.2% on LIBERO-Plus; average completed sequence length rises from 4.42 to 4.53 for VLA-Adapter and from 4.28 to 4.40 for OpenVLA-OFT on CALVIN ABC→D. Across seven real-world tasks, OpenVLA-OFT's average success rate rises from 51.4% to 65.0%.Source: §1 · p. 2
  • Representation analyses show that Bridge Tokens retain task context while exposing action structure before Action Expert refinement.The framework uses paired robot demonstrations to integrate perception-side task semantics with action-side executable structure.Source: §1 · pp. 2–3

2 RELATED WORK

PAIR is positioned as a learned perception–action interface rather than a bridge defined by a fixed physical abstraction or a method that only structures action representations. Its content combines task semantics from visual–language context with executable structure from expert actions, before Action Expert refinement.

Intermediate representations for perception-to-action learning

  • Motion-, geometry-, and affordance-based methods define intermediate representations through physical abstractions, whereas PAIR learns bridge content jointly from task context and expert-action structure.The cited approaches encode trajectories, future scene changes, geometry, or possible interactions and robot configurations; PAIR does not prescribe a particular physical meaning for its bridge.Source: §2 · p. 3

Latent action representations and tokenization

  • LAPA, UniVLA, FAST, and VQ-VLA structure latent actions or action tokens, while PAIR learns a perception-derived representation shaped by expert actions before Action Expert refinement.LARA aligns latent visual dynamics with intermediate Action Expert features during generation; PAIR instead makes its bridge available before refinement begins.Source: §2 · p. 3

3 METHOD

PAIR learns horizon-wise Bridge Tokens by aligning task information from visual-language context with executable structure from expert actions, then uses them to prime action generation. At inference, the Bridge Module produces these tokens from the current observation and instruction without the autoencoder.

3.1 MASKED ACTION AUTOENCODER

  • The Masked Action Autoencoder learns executable action structure by reconstructing clean trajectories from randomly masked and noise-perturbed action chunks, then supplies frozen reference tokens during PAIR training.Its temporal modeling encourages recovery from dependencies across action steps rather than copying individual values.Source: §3.1 · pp. 3–4

3.2 BRIDGE MODULE

  • The Bridge Module uses learnable queries to extract horizon-aligned task context, then aligns its tokens with expert-action tokens so the sequence carries task semantics and executable information.Cross-attention gathers visual-language information, while self-attention coordinates slots across the action horizon.Source: §3.2 · pp. 4–5

3.3 BRIDGE INJECTION

  • A projector maps Bridge Tokens into the Action Token space, and an input-dependent token-wise gate regulates their contribution when they are injected at the Action Expert’s entrance.This gives Action Tokens an explicit intermediate state to refine through the expert’s original computation.Source: §3.3 · p. 5

3.4 TRAINING

  • Training first learns action structure with the autoencoder, then freezes it while jointly optimizing the VLA policy and Bridge Module with action-prediction and weighted alignment objectives.At inference, the Bridge Module derives tokens from visual-language representations and Bridge Injection primes the Action Tokens for continuous action prediction.Source: §3.4 · p. 5

4 EXPERIMENTS

Across simulation and real-robot evaluations, PAIR improves performance; analyses show Bridge Tokens encode task and action structure before refinement, while ablations favor alignment and entrance injection.

4.1 EXPERIMENTAL SETUP

  • Evaluations use VLA-Adapter on LIBERO/Plus, both backbones on CALVIN ABC→D, and OpenVLA-OFT on real-robot tasks, with 50 LIBERO rollouts per task and 1,000 CALVIN five-task chains.LIBERO-Plus tests LIBERO-trained checkpoints zero-shot under distribution shifts. Real-robot evaluation covers seven tasks over 20 trials each, totaling 140 trials.Source: §4.1 · p. 6, §4.3 · p. 8

4.2 SIMULATION RESULTS

  • PAIR raises LIBERO average success to (98.4±0.3)% and CALVIN average sequence length to 4.40 ± 0.04 for OpenVLA-OFT and 4.53 ± 0.03 for VLA-Adapter.On LIBERO, VLA-Adapter improves from 97.3% to (98.4±0.3)%, with gains across all four suites. On CALVIN, five-task success rises by 3.9 and 3.7 points for OpenVLA-OFT and VLA-Adapter, respectively, with larger margins as more consecutive tasks are required.Source: Table 1 · p. 6, §4.2 · pp. 6–7, Table 3 · p. 7
  • PAIR raises VLA-Adapter’s LIBERO-Plus overall success from 59.1% to (64.2 ± 0.4)% and improves all seven OOD categories.The largest category gains are camera viewpoint (+8.1 points), robot initial state (+7.8), and noise (+6.7); the LIBERO-to-LIBERO-Plus success-rate drop narrows from 38.2 to 34.2 points.Source: Table 2 · p. 6, §4.2 · pp. 6–7

4.3 REAL-WORLD EXPERIMENTS

  • PAIR improves OpenVLA-OFT’s real-world success from 51.4% to 65.0% across seven tasks, with gains on every task.The largest reported gain is water pouring (+20 points); object placement and drawer loading each improve by 15 points.Source: Figure 4 · p. 8, §4.3 · p. 8

4.4 ANALYSIS

  • Bridge Tokens achieve 87.8% ± 4.5% task accuracy and 91.5% ± 2.2% action-cluster accuracy, and reach action-chunk R2 = 0.878 before the Action Expert.Their action geometry matches ground-truth actions with RSA 0.878 and top-10 neighbor overlap 80.4%, versus 0.014 and 21.6% after shuffling.Source: Figure 5 · p. 8, §4.4 · p. 9

4.5 ABLATION STUDIES

  • On LIBERO, entrance injection achieves 98.4% average success, while alignment weight λalign = 0.1 also reaches 98.4%.Entrance injection exceeds end, middle, and all-layer injection; without alignment, average success is 97.2%, and λalign = 0.01 reaches 97.8%.Source: Table 4 · p. 9, §4.5 · p. 9, Table 5 · p. 9

5 CONCLUSION

PAIR introduces an explicit intermediate representation that bridges perception and action in continuous-action VLAs. Its Bridge Tokens combine visual-language context with action structure to prime Action Tokens before Action Expert refinement.

5 CONCLUSION

  • Across LIBERO, LIBERO-Plus, CALVIN, and seven real-world tasks, PAIR improves two VLA backbones in manipulation, zero-shot generalization, long-horizon execution, and physical control.The conclusion reports gains across these settings without specifying numerical results.Source: §5 · p. 9
  • Bridge Tokens combine visual-language context with action structure and prime Action Tokens at the entrance of the Action Expert.This provides an explicit intermediate representation for the perception-to-action transition.Source: §5 · p. 9
  • Representation analyses show that the learned bridge carries task and action information, while ablations identify action alignment and early injection as key ingredients.The conclusion presents explicit perception-action intermediates as a practical design principle for VLA policies.Source: §5 · p. 9

AI USE STATEMENT

The authors used large language models for language polishing and clarity, while producing and verifying the scientific work themselves and reviewing all AI-assisted text.

AI USE STATEMENT

  • Large language models aided language polishing and manuscript clarity, while the authors produced and verified the scientific ideas, design, implementation, analysis, results, and conclusions.The authors reviewed all AI-assisted text and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A ADDITIONAL ABLATIONS · B ADDITIONAL ANALYSIS · B.1 LAYERWISE PROBE PROTOCOL

The additional LIBERO ablations favor horizon-aligned learned action tokens, adaptive bridge injection, and autoencoder training with noise. The layerwise analysis specifies frozen-representation probes comparing Bridge Tokens with action-token baselines.

A ADDITIONAL ABLATIONS

  • 98.4% average success with Action Latent Token alignment exceeds 97.1% with raw-action alignment, improving all four suites and gaining 1.8 points on Long.The comparison uses VLA-Adapter on the four LIBERO suites.Source: A ADDITIONAL ABLATIONS · p. 12, Table 6 · p. 12
  • 98.4% success with learnable injection exceeds 98.0% and 98.1% with fixed strengths of 1.0 and 0.5, respectively.This ablation evaluates injection strength on the four LIBERO suites.Source: A ADDITIONAL ABLATIONS · p. 12, Table 7 · p. 12
  • 97.6% to 98.4% average success results from adding noise during Action Autoencoder training, with gains on every LIBERO suite.The paper attributes the benefit to denoising that yields action structure useful under input corruption.Source: A ADDITIONAL ABLATIONS · p. 12, Table 8 · p. 12

B.1 LAYERWISE PROBE PROTOCOL

  • The layerwise protocol uses frozen LIBERO-Spatial representations from 500 balanced training frames, comparing eight pre-LLM Bridge Tokens with Action Latent Tokens and baseline tokens across 24 Transformer blocks.Comparisons use the same frames, action targets, and trajectory-grouped folds; each example includes an eight-step, seven-dimensional ground-truth action chunk.Source: §B.1 · p. 12
  • Task and action classification use five trajectory-grouped folds, with task identities or eight KMeans action clusters as labels, respectively.The action clusters use standardized ground-truth chunks, K = 8, 20 initializations, and seed 7; a multinomial logistic-regression probe uses C = 1, and results report mean balanced accuracy and fold standard deviation.Source: §B.1 · p. 12
  • Continuous-action prediction uses ridge regression with α = 10 on the same trajectory-grouped folds and reports varianceweighted R2.Targets are standardized using the training fold; representations are flattened, standardized, and reduced to 50 principal components when input dimension exceeds 128, with preprocessing fit on each fold's training portion.Source: §B.1 · p. 12

B.2 TASK AND ACTION INFORMATION ACROSS DEPTH … C IMPLEMENTATION AND EVALUATION DETAILS

Across depth, Bridge Tokens make task and action-pattern information available before Action Expert refinement, and injected Action Tokens are more readable for continuous actions than baseline tokens. Additional geometry analyses show action structure in Bridge representations at both sample and cluster levels.

B.2 TASK AND ACTION INFORMATION ACROSS DEPTH

  • Before Action Expert refinement, Bridge Tokens reach 87.8% task accuracy and 91.5% action-cluster accuracy; baseline Action Tokens end at 89.6% and 89.1%, respectively.Baseline task accuracy reaches 97.0% after the first block, while action-cluster accuracy develops more gradually. The figure shows baseline scores across layers 0–24 and the pre-LLM Bridge Token score.Source: Figure 6 · p. 13, §B.2 · p. 13
  • Bridge Tokens combine task information near the baseline’s late-layer level with action-cluster accuracy above its final value before refinement begins.The baseline makes task identity accessible after the first block, while action-cluster structure accumulates more gradually across later blocks.Source: §B.2 · p. 13

B.3 CONTINUOUS-ACTION DECODING ACROSS DEPTH

  • At the Transformer entrance, injected PAIR Action Tokens score 47.3% versus 20.1% for baseline tokens; at the final block, they score 76.4% versus 72.7%.Scores decode the full eight-step ground-truth action chunk from frozen tokens, using 1,000 shared LIBERO-Spatial examples and five trajectory-grouped held-out folds.Source: §B.3 · p. 13, Figure 7 · p. 14

B.4 ADDITIONAL ACTION-GEOMETRY EVIDENCE

  • Bridge Token and ground-truth action-cluster spaces retain positive similarity for A0–A1 and A3–A5, and negative similarity for A2–A3, while other matrix differences reflect perceptual and task context.A0–A7 denote unsupervised clusters, not named behaviors.Source: Figure 8 · p. 14, §B.4 · p. 15
  • Linear CKA between Bridge Token and action representations is 0.887, compared with 0.148 under shuffled correspondence, with task-wise values from 0.865 to 0.909.Together with distance-ranking and neighborhood results reported in the main text, these metrics indicate action structure at sample and cluster levels.Source: Figure 8 · p. 14, §B.4 · p. 15
  • The geometry analysis averages scores across ten tasks using a deterministic 500-frame subset, with two ordered frames from each of 25 trajectories per task.Representations and actions were standardized, centered within task, and L2-normalized; the permutation reference shuffled complete two-frame trajectory groups within each task 200 times while retaining internal order.Source: §B.4 · p. 14

C.1 TRAINING HYPERPARAMETERS

The section specifies dataset- and stage-specific training settings for PAIR, along with the Action Autoencoder and Bridge Module configurations used during training.

C.1 TRAINING HYPERPARAMETERS

  • OpenVLA-OFT’s learning rate remains constant because its decay milestone follows the 50k-step run.Table 9 is labeled as listing default training hyperparameters; its caption also defines batch as the effective batch size across four GPUs.Source: Table 9 · p. 15
  • The Action Autoencoder trains for 50k steps on demonstrations for each benchmark or separately for each real-world task.It uses LIBERO demonstrations for LIBERO and zero-shot LIBERO-Plus, and CALVIN demonstrations for CALVIN. Both stages use AdamW; the autoencoder uses weight decay 10−4 and gradient clipping at 1.0, while policy training uses bfloat16, image augmentation, and λalign = 0.1.Source: §C.1 · p. 15
  • The Action Autoencoder uses a two-layer Transformer encoder and one-layer decoder, with action steps randomly masked at probability 0.5 and Gaussian noise of standard deviation 0.05.Both models use eight Bridge Tokens, formed from horizon-aligned queries through cross-attention and self-attention blocks with eight heads. Alignment and injection use two-layer GELU MLPs; injection is scaled by a zero-initialized per-token gate using layer normalization, a scalar linear projection, and tanh.Source: §C.1 · p. 15

C.2 SIMULATION BENCHMARKS

The simulation benchmarks evaluate task-conditioned manipulation on LIBERO, zero-shot robustness on LIBERO-Plus, and execution in unseen CALVIN environment D. Their protocols measure suite success, perturbation generalization, and both task completion and sustained sequence execution.

C.2 SIMULATION BENCHMARKS

  • LIBERO evaluates four task-conditioned manipulation suites over 50 rollouts per task from official initial states, reporting each suite’s and average success rate.The Action Autoencoder and VLA-Adapter policy are trained on LIBERO data; observations are 256×256 pixels. The suites vary spatial relationships, objects, goals, and action-sequence length.Source: §C.2 · p. 15
  • LIBERO-Plus evaluates LIBERO-trained checkpoints zero-shot across seven perturbation categories, measuring per-category and overall success as generalization beyond training conditions.Perturbations cover camera viewpoint, robot initial state, language instruction, lighting, background, sensor noise, and object layout.Source: §C.2 · p. 15
  • CALVIN uses the ABC→D split, training policies in environments A–C and executing 1,000 five-task chains in unseen environment D.OpenVLA-OFT executes each eight-step action chunk open-loop before replanning; evaluation reports success at completing one to five consecutive tasks and average completed sequence length.Source: §C.2 · pp. 15–16

C.3 REAL-WORLD TASKS AND EVALUATION

The section describes a seven-task real-world evaluation using a shared robot and observation setup, with 20 trials per task and macro-averaged results. The task suite spans target selection, sequential control, and extended physical interaction.

C.3 REAL-WORLD TASKS AND EVALUATION

  • The tasks cover target selection, relative-object reasoning and alignment, ordered button pressing, and contact-rich or articulated manipulation.The suite includes object and cube placement, cup stacking, button pressing, board wiping, water pouring, and drawer loading; Table 10 provides task definitions and dataset sizes.Source: §C.3 · p. 16, Table 10 · p. 16
  • Figure 9 shows five chronological frames from demonstrations of each of the seven tasks, with labels identifying manipulation stages.Source: Figure 9 · p. 17
Loading 2610.09016v1…