PaperlayerView exact revision on arXiv

Source-linked AI summary

BridgeWAM: Bridging Video and Action Experts via Latent Bridge Queries

Jiaxin Fang, Ruitong Liu, Zeyu Lai, Xingyu Wang, Mingyuan Jia, Pingyue Sheng, Weikang Wan, Mengdi Xu, Yuxiang Gao

arXiv:2610.14746v1cs.ROcs.CV

TL;DR

WAMs commonly use dense, layer-wise coupling that exposes action prediction to irrelevant visual features and requires matched expert depths. BridgeWAM connects video and action experts through 32 Latent Bridge Queries, with joint supervision and a lightweight action head. The paper reports task evaluations and ablations supporting the architecture, while noting difficulty with complex, long-horizon real-world tasks and instability inherited from Fast-WAM.

  • Problem

    Dense, layer-wise MoT coupling exposes action prediction to irrelevant visual features and requires matched expert depths, limiting selective information use and adding redundant computation.

    Source: §1 · p. 2

  • Method

    BridgeWAM uses 32 Latent Bridge Queries to aggregate multimodal priors into a shared latent space, where joint video–action supervision shapes video representations and the Action Expert selectively retrieves information.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5

  • Results

    The paper reports simulation and real-world task evaluations and ablation studies examining task performance, generalization, and computation for BRIDGEWAM.

    Source: §1 · p. 2

  • Takeaways & Limitations

    In the evaluated settings, the compact LBQ interface and decoupled expert depths support a balance of task performance, generalization, and inference efficiency.

    Source: §5 · p. 10

    During real-world robot training, BRIDGEWAM struggles to fit more complex, long-horizon manipulation tasks and retains instability observed in its Fast-WAM foundation.

    Source: §5 · p. 11

Abstract

from arXiv · show

World Action Models (WAMs) commonly couple video and action experts through dense, layer-wise Mixture-of-Transformers (MoT) attention, exposing action prediction to irrelevant visual features while enforcing matched expert depths. This coupling hinders selective visual-feature use and introduces redundant computation. We introduce Bridge-of-Experts (BoE), instantiated as BridgeWAM, which aggregates visual, language, and proprioceptive priors through 32 Latent Bridge Queries (LBQs) into a shared, limited latent space. Joint video-action supervision lets action gradients guide the Video Expert toward representations better suited for action prediction, while the Action Expert learns to selectively retrieve task-relevant information through cross-attention. Decoupling expert depths and reusing LBQs during action denoising yield a 5.06B model with a 30-layer Video Expert and a two-layer Action Expert, achieving 98.25% success on LIBERO and 72.24% pooled success on LIBERO-Plus, demonstrating effective task execution and strong generalization with an ultralight action head. Ablations support joint representation adaptation and unified LBQ conditioning. Relative to Fast-WAM, BridgeWAM reduces single-sample inference FLOPs by 28.72% in simulation profiling and real-robot latency to 77.1 ms (a 2.47-fold speedup). Qualitative experimental results further indicate preferential attention to target objects and task destinations across perturbations.

1 INTRODUCTION

WAMs need precise, robust, low-latency action prediction, but dense layer-wise MoT coupling exposes action prediction to irrelevant visual features and forces matched expert depths. BridgeWAM uses a compact LBQ interface and decoupled experts to support selective retrieval and lightweight action decoding.

1 INTRODUCTION

  • WAMs must predict precise action chunks, remain robust to scene and object changes, and operate at low enough latency for closed-loop control.Recent WAMs couple Video and Action Experts through MoT attention, but their information-transfer approach remains largely unchanged.Source: §1 · p. 2
  • Dense MoT coupling exposes action layers to task-relevant and redundant visual features while forcing matched expert depths, adding parameters and inference overhead.Prior work cited in the passage associates attention to task-irrelevant regions with interference in task execution and weaker generalization under visual shifts.Source: §1 · p. 2
  • BridgeWAM combines a 30-layer Video Expert with a two-layer Action Expert, using LBQs for cross-attention retrieval and self-attention to coordinate action chunks.The 32 LBQs aggregate visual, language, and proprioceptive priors; action-loss gradients flow through them into the Video Expert.Source: §1 · p. 2, Figure 2 · p. 3
  • BoE uses 32 LBQs to aggregate visual, language, and proprioceptive priors into a limited latent space, where action supervision shapes action-relevant video representations.The Action Expert selectively retrieves information from the LBQs rather than directly mixing video and action tokens.Source: §1 · p. 2

2 RELATED WORK

Prior work transfers information between vision and action through several WAM coupling strategies, while learnable query interfaces offer a compact alternative for extracting task-relevant information. LBQs extend query-based transfer by learning shared video–action conditioning under joint supervision.

Generalist Vision–Language–Action Policies

  • RT-1 and RT-2 scale behavior cloning and vision–language pretraining, while OpenVLA, π0, π0.5, and GR00T N1 use large pretrained backbones with varied action decoders.These action-centric policies learn physical dynamics implicitly from robot-action demonstrations, motivating WAMs that explicitly model visual scene evolution.Source: §2 · p. 3

World Action Models and Video–Action Coupling

  • WAMs couple modalities through shared attention, direct video K/V injection, predicted visual transitions, or temporally decoupled planning and execution.DreamZero jointly denoises video and action latents; LingBot-VA models actions from visual transitions; Fast-WAM omits future-video generation at inference.Source: §2 · p. 3

Cross-Modal Representation Transfer with Learnable Queries

  • MetaQuery and GenCeption use learnable queries for compact information extraction, while LBQs learn shared video–action conditioning under joint video and action supervision.The Video Expert writes action-relevant physical representations into the shared space, and the Action Expert selectively retrieves information for control under changing physical conditions.Source: §2 · p. 3

3 METHODOLOGY

The methodology frames embodied policies by their conditional inputs and outputs, then replaces direct video–action coupling with a compact LBQ interface. A two-layer alternating cross–self-attention Action Expert reads that interface to predict action flow.

3.1 PRELIMINARIES

  • VLA, WM, joint WAM, IDM WAM, Fast-WAM, and BRIDGEWAM differ in whether they predict actions, future observations, or both, and in which observations and conditions they use.The compared MoT-based WAM designs couple experts through shared attention, requiring matched Video and Action Expert depths.Source: §3.1 · p. 4
  • The Video Expert maps the current observation, language instruction, robot state, and 32 learnable queries to a fixed-size set of input-conditioned LBQ representations for Action Expert conditioning.This interface aligns visual, semantic, and proprioceptive information under joint video and action supervision without passing dense video tokens directly to the action branch.Source: §3.1 · p. 4

3.2 LEARNING AND READING THE LATENT BRIDGE

  • During training, future-video queries read video and LBQ keys/values, while LBQ updates exclude future-video keys/values; both video and action losses update the bridge pathway.Action-loss gradients flow through the LBQs into the Video Expert, while the LBQs are computed from current context once and reused throughout action denoising at inference.Source: §3.2 · pp. 4–5, §3.1 · p. 4

3.3 ALTERNATING CROSS–SELF ACTION HEAD

  • The two-layer Action DiT alternates cross-attention, which retrieves LBQ information, with self-attention, which coordinates actions within the chunk.Learned absolute positions distinguish action slots before cross-attention; 1D RoPE encodes relative positions between actions in self-attention.Source: §3.3 · pp. 5–6
  • Table 1 reports task success rates across the LIBERO Spatial, Object, Goal, and Long-Horizon suites and their average; Table 2 covers seven LIBERO-Plus perturbations and their average.The LIBERO-Plus table also indicates whether embodied pre-training is used.Source: Table 1 · p. 6, Table 2 · p. 6

4 EXPERIMENTS

The experiments evaluate BridgeWAM across simulation and real-robot tasks, ablations, efficiency profiling, and visual-routing analyses. Results report strong benchmark performance and reduced inference cost, while ablations examine joint adaptation, latent conditioning, and expert capacity.

4.2 QUANTITATIVE EVALUATION IN SIMULATION

  • 98.25% on LIBERO and 72.2% on LIBERO-Plus, where BridgeWAM raises Fast-WAM’s average from 49.5% to 72.2%.It also averages 100% on LIBERO-Object and 53.7% on LIBERO-Pro.Source: §4.2 · p. 7
  • 90% on randomized Stack Bowls Three, 98% on Place Object Stand, and 100% on Move Pillbottle Pad in RoboTwin 2.0.Handover Mic, Lift Pot, and Open Laptop are near-perfect in both evaluated settings.Source: §4.2 · p. 7, Table 3 · p. 7

4.3 QUANTITATIVE EVALUATION ON REAL ROBOTS

  • Real-robot evaluations report that the LBQ interface transfers beyond simulation and maintains stable, action-relevant representations under visual variation, observation noise, and execution disturbances.The four tasks use staged scoring, crediting completed operation stages.Source: Figure 4 · p. 7, §4.3 · p. 8

4.4 ABLATION STUDIES AND EFFICIENCY

  • 85.50% on LIBERO and 25.48% on LIBERO-Plus for frozen-video conditioning, below the full model’s 98.25% and 72.24%.The comparison supports joint adaptation of video representations with LBQ-conditioned action learning.Source: §4.4 · p. 8
  • The MoT-style readout, attending to text and state through cross-attention, averages 97.80% on LIBERO and 60.48% on LIBERO-Plus; direct text-state-LBQ inputs average 97.20% and 59.29%, respectively.Dedicated LBQ cross-attention more effectively supports action prediction under the evaluated distribution shifts. Both text-state-LBQ-input rates are below the full model.Source: §4.4 · p. 9
  • 1.460 T FLOPs and 88.05 ms H20Z latency for BridgeWAM, compared with Fast-WAM’s 2.048 T FLOPs and 405.91 ms under the reported profiling protocol.Measurements use bfloat16, cached LIBERO input, a 32-step action horizon, and 10 denoising steps.Source: §4.4 · p. 9, Table 6 · p. 9
  • BRIDGEWAM IDM shows small but consistent improvements across all five video-generation metrics over Fast-WAM IDM on LIBERO-Plus replay clips.The evaluation uses 1,415,986 shared clips and assesses complete generated video clips.Source: Table 7 · p. 9, §4.4 · p. 10

4.5 QUALITATIVE ANALYSIS

  • Figure 5 contrasts Fast-WAM’s layer-averaged attention with BridgeWAM’s final-layer LBQ-mediated routing across six visual perturbations.The resulting maps are architecture-specific routing visualizations, not strict causal attributions or matched-layer comparisons.Source: §4.5 · p. 10, Figure 5 · p. 10

5 CONCLUSION

The conclusion presents Bridge-of-Experts as a compact latent interface that separates video and action computation, supporting selective retrieval and lightweight action decoding. It also identifies difficulty with complex long-horizon tasks and instability inherited from Fast-WAM.

5 CONCLUSION

  • BoE uses learnable LBQs to aggregate multimodal information for selective Action Expert retrieval while allowing action supervision to shape video representations.Decoupled expert depths permit lightweight action decoding without reducing Video Expert depth.Source: §5 · p. 10
  • Training on real-world robot tasks struggles to fit more complex, long-horizon manipulation tasks, and BRIDGEWAM retains instability observed in its Fast-WAM foundation.Future work will target long-horizon learning and a more stable, capacity-limited latent space.Source: §5 · p. 11

AI USE STATEMENT

  • The authors report using LLMs to polish author-written text for clarity, fluency, and readability, while retaining responsibility for the manuscript’s claims and results.They say citations were manually added and the related-work discussion was drafted by the authors before language polishing.Source: AI USE STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

  • The manuscript identifies architecture, training objectives, evaluation protocols, action-head initialization, ablation variants, real-robot scoring, and efficiency profiling details as reproducibility documentation.These materials are located in the main text and appendices specified in the statement.Source: REPRODUCIBILITY STATEMENT · p. 11

A ADDITIONAL EXPERIMENTAL DETAILS

The appendix states that its implementation and evaluation details complement the main text and that ablations generally follow the full-model protocol.

A ADDITIONAL EXPERIMENTAL DETAILS

  • Unless otherwise stated, ablations share the full BRIDGEWAM model’s data splits, optimizer, numerical precision, denoising schedule, and checkpoint-selection rule.Source: A ADDITIONAL EXPERIMENTAL DETAILS · p. 15

A.1 EXPERIMENTAL DESIGN DETAILS

The experimental-design details describe pretrained backbone components, task-specific module initialization, and the learnable query bank used by default.

A.1 EXPERIMENTAL DESIGN DETAILS

  • The 30-layer Video Expert uses pretrained Wan2.2-TI2V-5B weights, while the VAE encoder and decoder are pretrained and frozen.Text conditioning uses UMT5-XXL; by default, training reads precomputed text embeddings.Source: §A.1 · p. 15
  • The Action Expert is a two-layer, 1024-hidden-dimension Action DiT with alternating cross-attention and self-attention, initialized from dimension-adapted Wan-derived checkpoint layers.Its action timestep and text projections are loaded from the same preprocessed checkpoint.Source: §A.1 · p. 15
  • The action input and output modules, LBQ-to-action projector, and proprioceptive encoder use default PyTorch linear-layer random initialization.Source: §A.1 · p. 15
  • The 32 LBQs are persistent trainable embeddings initialized from a zero-mean Gaussian distribution with standard deviation approximately 0.01804.They are not resampled at each forward pass.Source: §A.1 · p. 15

A.2 ABLATION PROTOCOL AND VARIANT DEFINITIONS

The ablations vary how visual features adapt, how actions read conditioning, and how video and action generation are coupled. They compare these variants with the unified LBQ interface used by BridgeWAM.

Ablation protocol

  • The ablation set studies action-head capacity, three interface variants, and two alternative video–action generation schemes, with Table 4 variants retaining the default backbone and nominal two-layer Action Expert.The depth study varies the Action Expert across 4, 8, 16, and 30 layers while holding the Video Expert, LBQ count, training schedule, and evaluation protocol fixed.Source: §A.2 · p. 15

Frozen-Video-DiT

  • The frozen-video variant’s LIBERO-Plus average is 25.48%, versus 72.24% for the full model, supporting joint adaptation of video representations and action learning in this setting.It freezes Fast-WAM Video DiT weights and conditions the Action Expert through LBQs.Source: §A.2 · pp. 15–16

MoT-readout

  • MoT-readout’s LIBERO-Plus average falls to 60.48%, below the full model’s dedicated LBQ cross-attention readout.The variant concatenates LBQ and action keys/values in mixed self-attention and reads text–state keys/values in cross-attention.Source: §A.2 · p. 16

Text-state-LBQ-input

  • Text-state-LBQ-input reaches 59.29% on LIBERO-Plus, below the full model’s conditioning exclusively through LBQs.Its action queries read concatenated text, state, and LBQ keys/values.Source: §A.2 · p. 16

Joint

  • Joint propagates video, LBQ, and action states together during denoising, repeatedly applying Action DiT layer 0 before layer 1 performs action self-attention.Text–state conditioning enters on the Video/LBQ side; action-loss gradients update the Action Expert, while video and LBQs receive both video and action supervision.Source: §A.2 · p. 16

IDM

  • IDM first denoises video, then conditions action generation on 32 LBQs representing the full video, reusing that context throughout action denoising.The reported LIBERO-Plus result uses a full-frame prefill at t = 0 on the final video latent; this inference implementation also runs the VAE decoder.Source: §A.2 · p. 16, Table 8 · p. 16

Interpretation

  • Together, the ablations support learning task conditioning within the Video Expert and consolidating it into a fixed-size LBQ space, but do not causally establish the proposed explanation for weaker variants.The proposed explanation is that mixing representation spaces can cause distribution mismatch and attention competition.Source: §A.2 · pp. 16–17

A.3 DETAILED RESULTS ON SIMULATION BENCHMARKS

The detailed simulation results report pooled success across benchmark variants and also expose performance differences across perturbations and tasks. These breakdowns qualify the aggregate rates.

LIBERO-Plus

  • 72.24% pooled success across 10,030 LIBERO-Plus task variants; lighting changes are highest at 92.38% and camera viewpoint changes lowest at 56.10%.The 7,246 successful episodes yield the pooled rate; background texture and robot initial-state perturbations at difficulty level 5 yield 30.14% and 30.48%, respectively.Source: Table 9 · p. 18, §A.3 · p. 18

LIBERO-Pro

  • 53.72% overall success across 10,000 LIBERO-Pro episodes, with language, environment, and object perturbations at 97.60%, 82.70%, and 82.00%, versus 1.35% for swap and 4.95% for task perturbations.All 200 task variants are complete; each suite–perturbation combination contains ten tasks evaluated with 50 trials each.Source: Table 10 · p. 19, §A.3 · p. 19

RoboTwin 2.0

  • 90.03% pooled success across 10,000 RoboTwin 2.0 episodes: 89.84% on clean episodes and 90.22% on randomized episodes.Success is 100% in both settings on Grab Roller, Handover Mic, and Lift Pot, but 32%/31% on Hanging Mug and 50%/56% on Open Microwave (clean/randomized).Source: Table 11 · p. 20, §A.3 · p. 20

A.4 REAL-ROBOT SCORING PROTOCOL

The real-robot protocol scores partial completion of four manipulation tasks on the PiPER X, including corresponding OOD settings. Each satisfied task criterion contributes to a trial score.

Scoring protocol

  • Each numbered task criterion contributes one point when satisfied and zero otherwise; a trial score sums the points.Table 12 defines the task-specific criteria and the maximum score per trial.Source: §A.4 · p. 21, Table 12 · p. 21

Task criteria

  • Closing the laptop has one completion criterion, while drink, pen, and mouse pick-and-place separately assess grasping, transport, or placement.The maximum scores are 3 for drink, 3 for pen, and 4 for mouse tasks.Source: §A.4 · p. 21

Evaluation scope

  • The evaluation covers four PiPER X manipulation tasks and their corresponding OOD settings, using task-specific staged scoring.Source: §A.4 · p. 21

A.5 FRAME-BY-FRAME COMPARISONS ON REAL ROBOTS

Figure 6 presents frame-by-frame comparisons on four real-robot tasks. It places Fast-WAM above BridgeWAM and orders each sequence from left to right.

Figure 6

  • Figure 6 compares Fast-WAM in the upper row with BridgeWAM in the lower row for each task.Source: Figure 6 · p. 22
  • The four panels show closing a laptop and picking and placing a drink, a pen, and a mouse.Source: Figure 6 · p. 22
  • Frames progress from left to right within each panel.Source: Figure 6 · p. 22

A.6 ADDITIONAL QUALITATIVE EXAMPLES

The appendices compare visual-routing patterns across additional placement tasks and perturbations, expand ablation results, and document simulation and real-robot inference timings.

A.6 ADDITIONAL QUALITATIVE EXAMPLES

  • Across six additional placement examples, BRIDGEWAM exhibits more spatially differentiated routing than Fast-WAM across background, lighting, robot-state, viewpoint, object-layout, and sensor-noise perturbations.The routing maps visualize preferences, not causal contributions of individual visual features or task-success measurements.Source: §A.6 · pp. 23–24, Figure 7 · p. 23, Figure 8 · p. 23

A.7 COMPLETE ABLATION RESULTS

  • Complete LIBERO ablations cover all 40 tasks with 50 episodes per task, while LIBERO-Plus breakdowns cover 10,030 task variants across suites and perturbations.Pooled LIBERO-Plus rates use episode counts rather than averages of displayed percentages.Source: §A.7 · p. 25
  • The appendix reports IDM (x0 prefill) and IDM (last video step) evaluations separately, although both use the same checkpoint.The last-video-step evaluation uses LBQs from the final denoising forward pass without an extra x0 prefill.Source: §A.7 · p. 25
  • Additional ablation tables expand results for LBQ sequence lengths from 1 to 64 and report IDM video-generation quality by suite and perturbation.The video-generation evaluation uses complete nine-frame clips, including the reconstructed conditioning frame.Source: §A.7 · pp. 31, 36, Table 21 · p. 31, Table 26 · p. 36

A.8 LATENCY MEASUREMENT DETAILS

  • 88.05 ms vs 405.91 ms (4.61×) for BRIDGEWAM vs Fast-WAM; BRIDGEWAM also reaches 3.65× over Faster-WAM and 77.1 ms real-robot inference.Faster-WAM was measured in its native nine-frame configuration, while the other models processed one observed latent frame.Source: §A.8 · p. 38
Loading 2610.14746v1…