PaperlayerView exact revision on arXiv

Source-linked AI summary

Rewiring Semantics, Dynamics, and Control: A Simple yet Effective Action-Centric Tri-Stream Transformer

Shuang Luo, Yilun Kong, Yunpeng Qing, Yihang Jiao, Zhi Hou, Shunyu Liu, Xiaogang Wang, Dacheng Tao

arXiv:2610.11416v1cs.ROcs.AI

TL;DR

VLM backbones offer limited physical-dynamics priors, while replacing them with WMs can fail to preserve task-level semantic reasoning; combining both as complementary guidance for action generation remains challenging. ACT3 uses separate semantic, dynamics, and action streams, with layerwise action attention integrating the context streams while they retain independent forward computation. On RoboCasa’s 24 kitchen tasks, ACT3 achieves 71.2% success versus 59.4% for the in-house π0.5 baseline, using the same downstream demonstrations and available camera views.

  • Problem

    VLM backbones offer limited physical-dynamics priors, while replacing them with WMs can fail to preserve task-level semantic reasoning; combining both as complementary guidance for action generation remains challenging.

    Source: §1 · pp. 1–2

  • Method

    ACT3 uses separate semantic, dynamics, and action streams, with layerwise action attention integrating the context streams while they retain independent forward computation.

    Source: §1 · p. 2

  • Results

    On RoboCasa’s 24 kitchen tasks, ACT3 achieves 71.2% success versus 59.4% for the in-house π0.5 baseline, using the same downstream demonstrations and available camera views.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    The findings suggest that global task semantics and local predictive dynamics can be combined through action generation without directly coupling the context streams.

    Source: §5 · p. 9

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have emerged as a prominent framework for complex robotic manipulation, building on the strong semantic understanding of pretrained Vision-Language Models (VLMs). However, such VLM backbones offer insufficient physical dynamics priors, which limits the generalization capabilities of robot policies. Recent efforts therefore integrate video-generation World Models (WMs) into robot policies through various strategies, using predictive dynamics to facilitate action generation. Despite these advances, harnessing semantic understanding and dynamics prediction as complementary guidance for action generation remains challenging. In this paper, we introduce $\mathrm{ACT}^3$, a simple yet effective Action-Centric Tri-Stream Transformer that fuses semantic and dynamics information into control actions while preserving the distinct roles of context streams. Specifically, $\mathrm{ACT}^3$ enables the dedicated action expert to access VLM and WM representations through layerwise attention, with each backbone attending only within its own stream. This straightforward interaction design maintains independent forward propagation in the context streams while allowing both backbones to be updated through control supervision. Experiments on both simulated and real-world robotic manipulation benchmarks show that the proposed $\mathrm{ACT}^3$ yields results superior to its counterparts.

1 INTRODUCTION

VLM-based policies offer strong semantic understanding but limited physical-dynamics priors, while WM-centered policies can sacrifice task-level semantic reasoning. ACT3 integrates both sources through the action expert while keeping the context streams independent.

1 INTRODUCTION

  • VLM pretraining emphasizes semantics and provides limited physical-dynamics priors, a limitation that can hinder policy generalization in manipulation.Effective manipulation requires both task semantics and consideration of physical consequences.Source: §1 · p. 1
  • Replacing a VLM backbone with a video-generation WM can strengthen dynamics prediction while failing to preserve the task-level semantic reasoning needed for global objectives.The passage notes that stronger prediction of local state transitions does not necessarily retain semantic reasoning for coordinated manipulation.Source: §1 · p. 1
  • Existing multi-stream designs often couple semantic and dynamics representations beyond the action pathway, motivating direct integration through the action expert.The paper notes that this additional coupling is not necessarily beneficial for control.Source: §1 · p. 2
  • ACT3 gives the action expert sole responsibility for integrating semantic and dynamics context through layerwise attention, while both context streams remain independently computed and jointly optimized.The VLM supplies context for global task objectives, and the WM supplies predictive context for local state transitions.Source: §1 · p. 2
  • Experiments on simulated and real-world manipulation benchmarks demonstrate ACT3’s effectiveness, with controlled ablations supporting action-centric interaction.The paper reports that ACT3 improves on the π0.5 baseline and counterparts.Source: §1 · p. 2

2 RELATED WORKS

Prior VLA approaches draw on VLM semantics, while WAMs use video-generation world models for predictive dynamics; neither direction alone necessarily supplies both kinds of context. ACT3 focuses on information flow among specialized semantic, dynamics, and action streams.

2 RELATED WORKS

  • VLM-centric policies leverage broad semantic knowledge but inherit limited physical-dynamics priors from vision-language pretraining.The related-work section situates this limitation within VLA models and the π series.Source: §2 · p. 3
  • WAMs exploit dynamics learned from video-generation world models, but stronger dynamics modeling does not necessarily preserve task-level semantic reasoning for goal-directed manipulation.The paper contrasts shared-backbone approaches with specialized multi-stream architectures that keep semantic, dynamics, and action pathways separate.Source: §2 · p. 3
  • Existing multi-stream methods use semantic-dynamics coupling, dynamics-action coupling, or feature conditioning; ACT3 instead makes the action stream the sole integration interface.In ACT3, action queries access both context streams through layerwise attention, and both backbones are jointly optimized under action supervision.Source: §2 · p. 3

3 METHOD

ACT3 combines semantic, dynamics, and action streams, using the action stream as the interface between two independently computed context pathways. Its method specifies how these contexts are constructed and attended to, how the streams are jointly trained, and how cached contexts are used at inference.

3.1 OVERVIEW

  • ACT3 encodes multi-view observations and the instruction as semantic context, uses a dynamics stream for anticipated local state transitions, and generates continuous action chunks through flow matching.The policy predicts a short-horizon action chunk and executes a prefix before replanning from updated visual observations.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 ACTION-CENTRIC TRI-STREAM TRANSFORMER

  • At every transformer layer, action queries attend to semantic, action, and dynamics representations, while each context stream attends only within its own stream.The semantic stream processes multi-view images and the instruction; the dynamics stream samples future latents from the base-camera image and instruction, using independently drawn noise during training and inference. Demonstrated futures are used only for auxiliary prediction supervision.Source: §3.2 · pp. 4–5

3.3 TRAINING AND INFERENCE

  • Training combines flow-matching action supervision with an auxiliary future-latent prediction loss, and inference caches contexts while integrating action velocity from noise.The auxiliary branch shares dynamics-transformer parameters but does not feed the action expert; cached semantic and dynamics contexts stay fixed during action integration and refresh at the next query.Source: §3.3 · p. 5

4 EXPERIMENTS

Experiments evaluate ACT3 on simulated and real-world manipulation, including transfer under distribution shifts, and examine how stream interaction, layer access, and dynamics supervision affect performance.

4.1 EXPERIMENTAL SETUP

  • The study evaluates RoboCasa’s 24 kitchen tasks, 40 jointly trained LIBERO tasks, and direct transfer to LIBERO-Plus, comparing ACT3 with π0.5 and other baselines.The policies use LIBERO-trained checkpoints for LIBERO-Plus without fine-tuning; both are evaluated on the same 10,030 perturbed instances.Source: §4.1 · p. 6

4.2 TASK EXECUTION AND ROBUSTNESS

  • ACT3 improves RoboCasa (71.2% vs 59.4%) and LIBERO average (98.6% vs 96.9%), and reaches 82.4% vs 68.4% on LIBERO-Plus transfer.The LIBERO-Plus comparison uses the same 10,030 instances without additional fine-tuning; improvements span all four LIBERO suites and all four LIBERO-Plus suites.Source: §4.2 · p. 6, Table 2 · p. 7, Table 1 · p. 7

4.3 ACTION-CENTRIC INTERACTION

  • The attention-topology comparison finds the highest success with ACT3 and the lowest when all three streams attend to one another.The reported ordering is consistent with avoiding additional cross-stream dependencies during context construction.Source: §4.3 · p. 7, Table 3 · p. 7
  • 71.2% success with layerwise access to both streams exceeds 65.8%, 63.5%, and 64.6% under final-layer K/V reuse.The reuse controls apply to the VLM only, WM only, or both streams, respectively; among tested attention topologies, ACT3 achieves the highest success rate.Source: §4.3 · p. 7, Table 3 · p. 7

4.4 GENERATED CONTEXT AND DYNAMICS SUPERVISION

  • Generated-future context increases RoboCasa success from 61.3% to 65.8% without prediction supervision and from 66.0% to 71.2% with it.Prediction supervision also benefits current-frame control, where no generated future reaches the action stream.Source: §4.4 · p. 8, Figure 4 · p. 8

4.5 ANALYSIS OF ACTION-CENTRIC ATTENTION

  • Figure 5 reports task-equal mean post-minus-pre changes in action-query attention allocation, with 95% bootstrap intervals.The supplied figure description does not state the direction or magnitude of any event-specific change.Source: Figure 5 · p. 8

4.6 REAL-WORLD EXPERIMENTS

  • ACT3 achieves 90.0% success on Bowl Stacking, 83.3% on Block Collection, and 63.3% on Flower Arrangement, exceeding π0.5 on each task.The respective π0.5 success rates are 86.7%, 73.3%, and 46.7%; each task was evaluated over 30 trials.Source: §4.6 · p. 9, Table 4 · p. 9

5 CONCLUSION

ACT3 combines semantic understanding and predictive dynamics for robot control through action-centric attention while keeping the context streams’ forward computation independent. Experiments support this design within robotic manipulation, the scope of the current evaluation.

5 CONCLUSION

  • ACT3 integrates semantic and predictive-dynamics context through layerwise attention in the action expert while keeping the context streams’ forward computation independent.Joint optimization of action generation and dynamics prediction lets both context representations and their integration adapt to the control objective.Source: §5 · p. 9
  • Experiments on simulated and real-world manipulation support combining global task semantics and local predictive dynamics through action generation without directly coupling the context streams.Source: §5 · p. 9
  • The evaluation is limited to robotic manipulation; extending the design to navigation and other embodied tasks is left to future work.Source: §5 · p. 9

A ADDITIONAL DISCUSSION OF MULTI-STREAM ARCHITECTURES

The discussion distinguishes ACT3 from multi-stream alternatives by where semantic, dynamics, and action representations interact. The appendices document dataset preparation, model and gradient paths, optimization, and checkpoint choices.

A ADDITIONAL DISCUSSION OF MULTI-STREAM ARCHITECTURES

  • ACT3 differs from semantic-dynamics coupling, dynamics-action coupling, and feature conditioning by integrating independently constructed context streams only through action queries.Unlike feature conditioning, ACT3 exposes corresponding layers directly to each action layer and keeps both context backbones trainable through action supervision.Source: A ADDITIONAL DISCUSSION OF… · p. 15

B.1 DATASETS AND PREPROCESSING

  • Training uses a 40-task LIBERO dataset with 1,693 episodes and 24-task RoboCasa data with 1,199 episodes; proprioceptive state is disabled in the main experiments.The LIBERO episode count is after preprocessing; RoboCasa uses successful human-demonstration trajectories.Source: §B.1 · pp. 15–16

B.2 MODEL INITIALIZATION AND GRADIENT PATHS

  • The semantic and action streams start from π0.5 and the dynamics stream from Cosmos; action-loss gradients reach shared Cosmos parameters through future-context sampling and feature extraction.The supervised video branch uses demonstrated futures, but inference does not access future observations.Source: §B.2 · pp. 16–17

B.3 OPTIMIZATION AND CHECKPOINT SELECTION

  • Training uses AdamW with a shared 5×10−5 learning rate, batch size 256, and 10,000 updates take approximately 23 hours on eight H100 GPUs.Main RoboCasa checkpoints use 20,000 updates, while LIBERO runs use 30,000; the reported ablations use matched 20,000-update RoboCasa checkpoints.Source: §B.3 · p. 17

B.4 EVALUATION AND STATISTICAL REPORTING

The evaluation uses matched conditions within each benchmark and reports mean success rates over rollouts. ACT3’s measured query latency is higher than π0.5’s on both benchmarks.

B.4 EVALUATION AND STATISTICAL REPORTING

  • LIBERO and RoboCasa main experiments and ablations use three evaluation seeds, with 6,000 and 3,600 trials per seed, respectively.LIBERO uses 50 trials per task across 40 tasks; RoboCasa uses ten trials in each of five scenes across 24 tasks.Source: §B.4 · p. 17
  • Both policies are evaluated on the same 10,030 perturbed LIBERO-Plus instances.Source: §B.4 · p. 17
  • 423/418 ms is ACT3’s mean latency on LIBERO/RoboCasa, versus 161/160 ms for π0.5.Timing includes future generation and context extraction but excludes input/output transforms and data transfers; published Cosmos Policy timings are not a controlled comparison.Source: §B.4 · p. 18

C.1 ATTENTION-TOPOLOGY ABLATIONS

The ablations vary attention topology, context-stream adaptation, future-context source, and dynamics-prediction supervision. They report gains for joint adaptation and near-future targets, while privileged self-distillation does not improve on ACT3.

C.1 ATTENTION-TOPOLOGY ABLATIONS

  • ACT3’s MACT topology keeps context streams independent; MSD lets dynamics queries read semantic tokens, while MSD+AD adds bidirectional dynamics–action attention and Mfull allows all streams to attend to one another.These masks modify the last 18 Cosmos blocks interleaved with semantic and action layers; the first ten blocks remain independent.Source: §C.1 · p. 18

C.2 JOINT ADAPTATION OF THE CONTEXT STREAMS

  • 71.2% success for joint adaptation exceeds 67.1% with prediction-only Cosmos, 62.3% with frozen Cosmos, and 32.7% with a frozen VLM.The variants retain the generated-future context; prediction-only Cosmos blocks action-loss gradients through its keys and values.Source: Table 7 · p. 19, §C.2 · p. 19

C.3 CURRENT AND GENERATED CONTEXT

  • Both generated-future context and prediction supervision improve success when the other factor is held fixed, with point estimates suggesting approximately additive benefits.The authors associate generated-future context with anticipated local state changes and suggest prediction supervision may improve control-relevant representations even with current-frame context.Source: §C.3 · p. 20

C.4 SAMPLING, SUPERVISION, AND ACTION INTEGRATION

  • The dynamics stream samples future latents, extracts action context from the sampled future slice, and shares parameters with a separate auxiliary prediction branch.Action-loss gradients reach the dynamics parameters through sampling and context extraction; the auxiliary branch trains on demonstrated futures and is omitted at inference.Source: §C.4 · pp. 20–21
  • At inference, action-flow integration starts from Gaussian noise, reuses cached contexts across Euler steps, and outputs the final state as the predicted action chunk.The default uses H = 10, B = 32, and SA = 10; caches refresh for the next policy query.Source: §C.4 · p. 21

D.1 TEMPORAL SAMPLING OF FUTURE TARGETS

  • Consecutive near-future targets achieve the highest success, while later consecutive and approximately uniform targets perform worse.The comparison keeps four future images, the action target, and the execution prefix fixed; the authors suggest nearby transitions are more relevant to receding-horizon control.Source: §D.1 · pp. 21–22, Table 8 · p. 22

D.2 EXPLORATORY PRIVILEGED SELF-DISTILLATION

  • Neither privileged self-distillation variant improves on ACT3, and success is lower at λKD = 0.5 than at 0.1.The authors suggest shared-parameter teacher guidance may be limited and that demonstrated-future and sampled-future contexts can support different actions.Source: Table 9 · p. 22, §D.2 · p. 22

D.3 DETAILS OF ACTION-CENTRIC ATTENTION ANALYSIS

The analysis measures how action-query attention across semantic, action, and world-model streams changes around simulator-defined interaction events. It uses rollouts from six RoboCasa manipulation tasks.

D.3 DETAILS OF ACTION-CENTRIC ATTENTION ANALYSIS

  • The analysis includes 300 rollouts and 21,307 policy queries from six RoboCasa tasks, including 210 successful and 90 unsuccessful rollouts.Source: §D.3 · pp. 22–23
  • Events are defined from simulator state using thresholds for proximity, contact, grasp, object motion, and articulation onset.For example, grasp requires bilateral finger contact in two consecutive control states.Source: §D.3 · p. 23
  • Attention is measured at the final action-flow evaluation by summing visible-key probabilities for VLM, action, and WM context, then comparing bins before and after events.Measurements average across 18 layers, eight heads, and action positions 0–4; confidence intervals use 1,000 whole-rollout bootstrap resamples within each task.Source: §D.3 · p. 23

E LIBERO-PLUS EVALUATION AND DETAILED RESULTS

On LIBERO-Plus, ACT3 outperforms the baseline across every reported perturbation category and every annotated difficulty level. Viewpoint changes remain the lowest-scoring category for both policies.

E LIBERO-PLUS EVALUATION AND DETAILED RESULTS

  • ACT3 improves over the baseline in all seven perturbation categories, with the largest gains for robot initial states and sensor noise.Viewpoint changes remain the lowest-scoring category for both policies despite improvement; background textures, lighting, and object layouts show smaller gains.Source: E LIBERO-PLUS EVALUATION AND DETAILED… · p. 23
  • ACT3 outperforms the baseline at every annotated difficulty level, and its advantage persists through the most difficult group.Success rates decrease with difficulty for both policies.Source: E LIBERO-PLUS EVALUATION AND DETAILED… · p. 23, Table 11 · p. 24

F.1 SETUP AND TRAINING

Each task and method uses a separately trained policy, with a fixed training budget and a randomized 30-trial evaluation protocol.

F.1 SETUP AND TRAINING

  • Training uses 40,000 optimizer updates, a global batch size of 64, and an action horizon of H = 50.The learning rate warms up to 2.5×10−5 over 1,000 updates before cosine decay to 2.5×10−6.Source: §F.1 · p. 24

F.2 EVALUATION PROTOCOL

  • π0.5 and ACT3 are evaluated for 30 trials per task, with initial object configurations independently randomized across trials and methods.Success rate is the percentage of successful trials out of 30, using task-specific criteria.Source: §F.2 · p. 24

F.3 TASKS AND METRICS

The real-world task criteria require stable bowl placement, completed block transfers, and ordered flower arrangement, with full success defined by task-specific endpoints.

Bowl Stacking

  • Bowl Stacking requires all designated bowls to form the target stack after release and remain stable through the final observation interval.An initial base bowl, when present, is excluded from required placements and partial-completion counts.Source: §F.3 · p. 24

Block Collection

  • Block Collection requires all designated blocks to remain inside the target bucket after release through the final observation interval.Partial completion counts blocks retained at termination, not transfer attempts.Source: §F.3 · p. 24

Flower Arrangement

  • Flower Arrangement requires completed pouring before flower insertion; full success means all target flowers are inserted, released, and stable with the vase upright.Insertion before pouring is an ordering violation and earns no credit for later milestones.Source: §F.3 · p. 24

F.3 TASKS AND METRICS

  • Figure 7 pairs initial and final observations for the three tasks; its endpoint images do not show Flower Arrangement’s intermediate water-transfer stage.The final images show stacked bowls, collected blocks, and flowers upright in a vase.Source: §F.3 · p. 24, Figure 7 · p. 25

Related papers

Loading 2610.11416v1…