PaperlayerView exact revision on arXiv

Source-linked AI summary

LayerRoute: Action-Conditioned Mixture-of-Layers Routing for Vision-Language-Action Policies

Zheng Lu, Haoran Liao, Wanqi Zhong, Yunhe Ni, Lijie Wang, Xingjie Fan, Zhisheng Chen, Yantang Qu, Meijia Chen, Tianyu Xin, Zirui Song, Yiming Li

arXiv:2609.06079v1cs.AIcs.LG

TL;DR

Existing VLA interfaces expose fixed VLM layers and implicitly propagate intermediate action states, limiting adaptive representation access for manipulation. LayerRoute routes cached VLM layers and rereads earlier action states, consistently improving StarVLA-π and π0.5 across simulation and real-world benchmarks, including a 7.2-point LIBERO Long gain.

  • Problem

    Existing VLA interfaces use fixed VLM layer assignments and residual-only action-state propagation, limiting adaptive access to representations needed across manipulation tasks.

  • Method

    LayerRoute conditions mixtures of cached VLM layers on current action states and rereads earlier intermediate action representations.

  • Results

    LayerRoute consistently improves both VLA backbones across simulation and real-world benchmarks, including a 7.2-point Long gain for StarVLA-π on LIBERO.

  • Takeaways & Limitations

    Adaptive access to VLM layers and earlier action states provides complementary benefits across VLA architectures and task settings.

  • Takeaways & Limitations

    LayerRoute adds parameter and inference-latency overhead, is evaluated on only StarVLA-π and π0.5, and may require architecture-specific integration.

Abstract

from arXiv · show

Vision-Language-Action (VLA) policies leverage pretrained vision-language models (VLMs) to guide action generation for robot control. VLMs provide hierarchical visual-semantic representations that evolve across layers, from local visual geometry to abstract, language-aligned semantics; different manipulation tasks may therefore require different mixtures of layer representations. Meanwhile, the action module maintains intermediate representations that evolve throughout action computation and may provide useful information for subsequent decisions. However, existing VLA interfaces offer limited flexibility in representation access: VLM information is exposed through fixed layer assignments for each action layer, while intermediate action states are only propagated implicitly through residual streams without explicit reuse. We introduce LayerRoute, an action-conditioned representation routing interface that enables adaptive access to VLM layers and action representations. The Layer Mixture Router dynamically forms mixtures of cached VLM representations, while Action-State Reread reuses earlier action representations. Across diverse simulation and real-world benchmarks, LayerRoute consistently improves StarVLA-$π$ and $π_{0.5}$, achieving up to 7.2 gains on LIBERO Long with only 0.31% / 3.87% additional parameters. Ablation studies validate the benefit of action-conditioned layer routing, while routing analyses reveal structured allocation patterns across action layers and task settings.

Introduction

LayerRoute addresses fixed VLM-layer exposure and indirect access to earlier action states by routing representations adaptively for manipulation decisions. It improves multiple VLA backbones and environments, while ablations and routing analyses support the mechanism and reveal structured allocation patterns.

  • Motivation: Different manipulation tasks require different combinations of fine-grained visual cues and abstract semantic representations.Grasp localization, gripper–object alignment, and contact-sensitive placement depend more on fine-grained evidence, whereas instruction grounding, object-role disambiguation, and subgoal selection rely more on abstract semantics.
  • Motivation: Standard residual propagation prevents later action blocks from directly revisiting earlier intermediate action states, constraining long-horizon manipulation decisions.Earlier states are progressively transformed and integrated into current representations, leaving later blocks with only indirect access.
  • Method: LayerRoute adaptively accesses VLM layers and earlier action states through action-conditioned Layer Mixture Routing and Action-State Reread.The router selects VLM representations across depth based on the current action state, while Action-State Reread retrieves earlier action representations.
  • Results: 7.2 and 3.6 Long success-rate gains, and 2.3 and 1.3 Avg4 gains, are reported for StarVLA-π and π0.5, respectively.LayerRoute also transfers to SimplerEnv and RoboCasa-GR1, with gains of up to 5.6 points on SimplerEnv WidowX and improvements across robot settings.
  • Analysis: Routing ablations validate the mechanism, while analyses reveal layer-allocation patterns across action layers and task settings.The reported analyses examine how representations are allocated across different action layers and task conditions.

Related Work

Prior VLA policies differ in their action-decoding interfaces, while recent methods adapt visual features, computation depth, or cross-depth pathways. LayerRoute instead targets depth-aware VLM-state access and direct reuse of intermediate action states across the VLM–action boundary.

  • VLA Action Generation and Representation Interfaces: VLA policies decode actions through discrete language-model tokens, diffusion action heads, or flow-matching action experts, with π0.5 also using tokenized-action pretraining.RT-2 and OpenVLA use discrete tokens; Octo uses diffusion; π0 and π0.5 use flow-matching experts.
  • Adaptive Representation Access and Routing: Prior adaptive-access methods select, prune, or reuse visual features and cross-timestep context, whereas LayerRoute accesses VLM states by depth and directly reuses action states during computation.The cited methods include instruction-relevant feature selection, visual-token aggregation and pruning, stable key–value reuse, and execution-history aggregation.
  • Layerwise Representations and Cross-Layer Reuse: Layerwise analyses establish layer depth as a representation axis, while other methods adapt computation depth or learn cross-depth pathways rather than routing representations across the VLM–action boundary.LayerRoute additionally enables token-wise reuse of earlier action states.

Background and Formulation

LayerRoute formulates VLA representation access as adaptive readout over cached VLM layers conditioned on current action states. It also enables explicit rereading of earlier action states and evaluates the formulation across two representative VLA architectures.

  • Layer-Wise VLM Representation Access: Pretrained VLM hidden states form a multi-layer cache whose token-aligned representations can be combined across depth.The cache includes states at boundaries n ∈ {0, …, N}, with n = 0 denoting the embedded input.
  • Layer-Wise VLM Representation Access: Static interfaces access fixed VLM layers, whereas LayerRoute adapts layer combinations according to the current action state.At action layer s, the available VLM layers are specified by C_s, and the current action state is a(s).
  • Action-State Access: Action-State Reread directly reuses earlier action states, overcoming reliance on residual propagation for later-layer access.Residual propagation progressively transforms and integrates earlier states, while rereading exposes those intermediate representations explicitly.
  • Architectural Instantiations: LayerRoute is instantiated in StarVLA-π, pairing Qwen3-VL with an action DiT, and π0.5, pairing PaliGemma with a flow-matching action expert.These architectures provide two VLA designs for examining representation access.

Method

LayerRoute lets each selected action layer adaptively mix cached VLM representations using the current action state, while Action-State Reread explicitly reuses earlier action states. In StarVLA-π, these mechanisms provide adaptive VLM access across 18 cross-attention layers and action-state reuse throughout 36 DiT blocks.

  • Layer Mixture Router: The Action-Conditioned Layer Mixture Router predicts a sample-wise distribution over accessible VLM depths from the current action state and candidate VLM representations.It mixes same-position VLM tokens with the predicted depth weights and uses the result as cross-attention context.
  • Layer Mixture Router: The router pools and projects action and VLM summaries into a shared d_sel = 256 space, converts similarities into depth weights, and aggregates candidate states.The predicted distribution is shared across all VLM token positions for each sample and selected action layer.
  • Action-State Reread: Action-State Reread aggregates the current action state with selected earlier action states, enabling each action token to reuse intermediate representations.Although the learned query is shared within a layer, compatibility scores vary by token position, producing token-specific source-state weights.
  • StarVLA-π implementation: In StarVLA-π, six retained VLM depths are exposed to all 18 cross-attention layers of the 36-layer DiT, replacing fixed VLM representations with action-conditioned mixtures.The router conditions depth selection on the current action state at every cross-attention layer.
  • StarVLA-π implementation: Action-State Reread applies to all 36 DiT blocks, including self-attention blocks, and recombines current and earlier action representations before the feedforward network.LayerRoute is also extended to π0.5 with a backbone-specific implementation.

Experiments

LayerRoute is evaluated across two VLA backbones, three simulation benchmarks, multiple VLM backbones, and controlled routing ablations. It consistently improves performance, while analyses show benefits from adaptive VLM-layer access and action-state rereading.

  • Experimental scope: Experiments examine generality across VLA and VLM backbones, the two representation reads, and learned structured routing behavior.The evaluation uses LIBERO, SimplerEnv, and RoboCasa-GR1, covering tabletop, visual and environmental variation, and household manipulation settings.
  • Cross-backbone evaluation: Across StarVLA-π and π0.5, LayerRoute improves LIBERO Avg4 and Long performance.StarVLA-π gains 2.3 and 7.2 points on Avg4 and Long, while π0.5 gains 1.3 and 3.6 points, respectively.
  • Cross-benchmark evaluation: LayerRoute consistently improves StarVLA-π and π0.5 across SimplerEnv and RoboCasa-GR1.SimplerEnv gains are 1.6/2.5 and 2.1/5.6 points on Overall/WidowX success; RoboCasa-GR1 macro-average gains are 1.2 and 5.5 points, with 0.31% and 3.87% parameter overheads.
  • VLM-backbone sensitivity: LayerRoute improves LIBERO Avg4 across all four tested VLM backbones, with gains ranging from 1.2 to 3.3 percentage points.The action architecture and training configuration are fixed while only the VLM backbone supplying multilayer representations changes.
  • Routing ablations: The complete routing interface achieves the best Avg4 and Long performance across both backbones.Compared with the best single-read variant, it improves Avg4/Long by 1.6/1.8 points on StarVLA-π and 0.9/1.8 points on π0.5, showing complementary benefits from VLM-layer routing and action-state rereading.

Layer-Routing Analysis

LayerRoute’s routing analysis shows that action-conditioned layer selection improves performance and learns increasingly selective, stage- and task-dependent mixtures of VLM representations. Routing stabilizes by 20K steps while shifting from shallow preferences early in action computation toward more selective deeper allocations later.

  • Action-conditioned routing: 98.0 Avg4 and 95.6 Long are achieved by action-guided routing, exceeding action-independent routing by 0.8 points on both metrics, fixed-depth by 1.6/1.4, and uniform routing by 3.8/3.4.The comparisons indicate that conditioning layer-wise routing on the current action representation provides an effective routing signal.
  • Training dynamics: At 20K steps, mean normalized entropy decreases from 1.000 at initialization to 0.805, while normalized entropy deficit increases from 0.000 to 0.195.Both measures stabilize near 20K, indicating increasingly selective VLM-depth allocation without collapse to a single layer.
  • Action-stage routing: Normalized entropy deficit rises from 0.126 in early action layers to 0.205 in middle layers and 0.253 in late layers, with mean Jensen–Shannon separation MDS = 0.205.The largest pairwise separation is between early and late groups, at 0.248.
  • Task-suite routing: Spatial tasks favor shallow VLM layers with a shallow-mass margin of 0.211, while Object tasks favor V3–V4 with a middle-mass margin of 0.198.Object-task routing gradually shifts toward deeper layers across router bins.
  • Task-suite routing: Goal tasks exhibit the largest expected VLM layer index across router bins, whereas Long tasks use a broader mixture.These suite-specific patterns show that routing preferences vary with task setting.

Real-World Robot Experiments

Real-world experiments with a Franka arm show that LayerRoute improves π0.5 manipulation success across three diverse physical tasks. Under matched fine-tuning and evaluation protocols, gains reach +8.0 percentage points.

  • Experimental Setup: Experiments use a Franka robotic arm with a gripper across cube pick-and-place, board wiping, and table cleaning tasks.The tasks test pose estimation and placement, contact-rich tool motion, and longer-horizon tool use.
  • Experimental Protocol: Each task uses 50 demonstrations, 20K fine-tuning steps for both policies under identical settings, and 50 independent physical trials.The comparison is between π0.5 and π0.5 + LayerRoute.
  • Results: +8.0, +4.0, and +6.0 percentage points improve success rates on PnP, Wipe board, and Clean Table, respectively.These gains compare π0.5 + LayerRoute against π0.5 in real-world manipulation.

Conclusion

LayerRoute addresses fixed representation interfaces in VLA action computation with action-conditioned routing over VLM and action representations. It dynamically accesses cached VLM layers and revisits intermediate action states beyond fixed layer exposure and residual-only propagation.

  • Framework: LayerRoute introduces an action-conditioned mixture-of-layers routing framework for adaptive access to VLM and action representations.The framework is motivated by fixed representation interfaces in current VLA action computation.
  • Representation routing: The Layer Mixture Router routes over cached VLM layer representations during action computation.This enables dynamic access beyond fixed layer exposure.
  • Representation routing: LayerRoute revisits intermediate action states instead of relying only on residual-only propagation.Together with VLM-layer routing, this expands the information available during action computation.

Appendix · Experimental Setup

The appendix details LayerRoute’s controlled experimental setup, including matched training conditions, preserved data protocols, joint fine-tuning, reproducible multi-seed evaluation, and separate routing diagnostics.

  • Experimental Setup: All component variants within each backbone share identical training and evaluation settings, while routed StarVLA-π controls retain Action-State Reread and vary only VLM-layer routing.This isolates the effect of the VLM-layer routing strategy.
  • Experimental Setup: LayerRoute preserves each backbone’s baseline datasets, mixtures, preprocessing, normalization, and sampling strategy without adding demonstrations or task-specific samples.The StarVLA-π mixtures include 1,693 LIBERO, 140,404 SimplerEnv, and 24,000 RoboCasa-GR1 trajectories, excluding auxiliary VLM instruction data.
  • Experimental Setup: Training uses eight NVIDIA H200 GPUs with per-device batch size 32, jointly fine-tuning the VLM, action module, router, reread module, and fusion parameters.All VLM parameters are updated during fine-tuning.
  • Experimental Setup: 30K, 40K, and 100K optimization steps are used for LIBERO, SimplerEnv, and RoboCasa-GR1, respectively, with a 5K-step warmup for both backbones.These are fixed training budgets for the three benchmark families.
  • Experimental Setup: Three independent runs per benchmark configuration use seeds 7, 42, and 41, with mean performance reported across runs.Global and rank-specific seeds initialize the relevant Python, NumPy, PyTorch, CUDA, router, fusion, and Action-State components deterministically.
  • Experimental Setup: Routing statistics largely stabilize by the 20K-step diagnostic checkpoint, which supplies the subsequent analyses in Figures 3–5.These routing diagnostics are separate from benchmark-specific endpoint evaluations, whose task-level results are reevaluated under the reported protocols.

Implementation Details

LayerRoute preserves native action objectives while inserting Layer Mixture Router and Action-State Reread at backbone-specific locations. The implementation uses six cached VLM states as the practical routing configuration, with distinct π0.5 and StarVLA-π fusion paths and joint fine-tuning of the pretrained components.

  • Backbone-specific implementation: Both implementations preserve native action objectives and insert Layer Mixture Router and Action-State Reread at backbone-appropriate locations.
  • Backbone-specific implementation: π0.5 reads selected memory through a separate action-to-VLM cross-attention adapter, whereas StarVLA-π supplies routed memory directly to native cross-attention.Both paths jointly fine-tune the pretrained VLM, action module, router, Action-State Reread, and backbone-specific fusion parameters.
  • VLM-layer granularity: Six cached VLM states form the practical routing configuration, while the 18-candidate variant provides finer depth access but has less stable optimization, added latency, and no consistent performance gains.The six candidates are retained after Qwen3-VL layers [5, 11, 17, 23, 29, 35] and exposed to all 18 action-DiT cross-attention routing sites.
  • Optimization details: Both implementations use bfloat16 mixed precision and AdamW with (β1, β2) = (0.9, 0.95), ϵ = 10^-8, and gradient clipping at 1.0.The π0.5 run uses per-device batch size 32 and weight decay 10^-8; StarVLA-π uses DeepSpeed ZeRO-2, weight decay 10^-8, and learning rate 2.5×10^-5.
  • Fusion and action-state reread: 3.87% additional parameters are introduced relative to the backbone by the projections, per-site routers, normalization layers, and Action-State Reread.In π0.5, the adapter leaves the VLM-prefix input unchanged, feeds its output into native prefix/suffix joint attention, and applies Action-State Reread before the action FFN.

Compute Environment … Limitations

The paper specifies controlled, common-protocol evaluations across simulation and real-world settings, with routing ablations and diagnostics isolating how action-conditioned access to VLM and action representations affects performance. It also reports implementation overheads and limits evaluation scope to two backbones, highlighting integration and deployment constraints.

  • Compute Environment: Both backbone implementations and simulator clients run in isolated environments connected through the existing policy server, preserving native training and evaluation protocols.LIBERO uses 50 episodes per task across 40 tasks, while SimplerEnv follows its public protocols and five repetitions per setting.
  • Component Ablations: All component variants use identical data, optimization budgets, evaluation protocols, and VLM updates, enabling controlled comparisons of routing and rereading components.The ablation routers access six cached VLM states from Qwen3-VL layers [5, 11, 17, 23, 29, 35].
  • Routing-Policy Controls: The initial-state and current-state routes differ only in router query state while sharing the same six-state VLM memory bank and Action-State Reread module.This design isolates the effect of router query state; the static-mixture comparison does not, because it disables Action-State Reread.
  • Routing-Policy Controls: 0.8 points: the current-state route outperforms the action-independent query control on both Avg4 and Long.Uniform, shallow, middle, and deep routing controls respectively use equal weights or predefined VLM-layer groups.
  • Routing Diagnostics: Routing diagnostics average distributions across router layers and suites, normalize over VLM layers, and organize sites into six three-block bins from E1 through L2.The resulting profiles characterize aggregate allocation patterns across training diagnostics rather than endpoint evaluation checkpoints.
  • Reporting Conventions and Data Sources: Success rate is the primary metric, with LIBERO additionally reporting unweighted Avg4, SimplerEnv reporting Overall and WidowX rates, and RoboCasa-GR1 reporting a 24-task macro-average.Within each backbone, component variants use identical training and evaluation settings.
  • Real-World Experimental Details: Real-world evaluation uses a fixed 10 Hz Franka tabletop setup and tests PnP, Wipe board, and Clean Table with 50 demonstrations and 50 trials per task.The tasks span grasping and placement, contact-rich interaction, and longer-horizon tool-use coordination.
  • Limitations: LayerRoute adds parameter and inference-latency overhead, was evaluated only on StarVLA-π and π0.5, and requires access to intermediate VLM and action states.These requirements may constrain resource-limited deployment and require architecture-specific integration for new VLA designs.
Loading 2609.06079v1…