PaperlayerView exact revision on arXiv

Source-linked AI summary

CoRe-WAM: Correspondence-Aligned Temporal Residuals for World Action Models

Bin Zhou, Jialong Liu, Jianan Wang, Changhao Chen, Kani Chen

arXiv:2609.27314v2cs.RO

TL;DR

Fixed-location comparisons can mix different scene content when objects or the camera move, complicating the use of visual history for action generation. CoRe-WAM aligns historical features before differencing and adds validity-gated residuals alongside current visual conditioning. Across 50 RoboTwin 2.0 tasks, it improves clean success over Motus under a 5,000-update budget.

  • Problem

    Comparing features at the same image location can mix different scene content when objects or the camera move, leaving how policies should relate current and historical visual evidence an open question.

    Source: §I · pp. 1–2

  • Method

    CoRe-WAM transports historical pretrained visual features to current locations before signed differencing, then adds validity-gated residuals alongside current-observation conditioning.

    Source: §I · pp. 1–2

  • Results

    92.22% clean success across 50 RoboTwin 2.0 tasks under a 5,000-update budget is 3.56 points above Motus.

    Source: §I · p. 2

  • Takeaways & Limitations

    CoRe-WAM’s modular temporal interface preserves the existing policy architecture and is demonstrated on physical robot control with Astribot S1.

    Source: §I · p. 2, §V · p. 8

    Online correspondence estimation and temporal feature processing add inference overhead, motivating work on lightweight estimation and feature reuse to reduce latency and improve closed-loop responsiveness.

    Source: §V · p. 8

Abstract

from arXiv · show

Comparing current and past observations helps robots understand scene changes and select subsequent actions during manipulation. However, comparing visual features at the same image location can mix different scene content when objects or the camera move. We introduce CoRe-WAM, a world-action model that incorporates correspondence-aligned visual changes through a parameter-efficient temporal interface. Its TraceDelta module uses correspondences from a frozen tracking model to transport historical visual features to current locations before computing signed differences in a shared pretrained feature space. Correspondence thus determines which historical content is compared with the present, rather than entering the policy as a separate trajectory representation. A lightweight adapter converts these differences into validity-gated residuals that supplement current visual conditioning, allowing the policy to use recent changes alongside current-scene information. Built on Motus, CoRe-WAM keeps the pretrained backbone weights frozen and optimizes 1.59 million parameters. With a 5,000-update adaptation budget, CoRe-WAM achieves 92.22% clean success across 50 RoboTwin 2.0 tasks, 3.56 percentage points above Motus; on randomized evaluation, it achieves 89.60% success, a 2.58-point gain. Integrating TraceDelta into a StarVLA-based policy improves clean success from 58.10% to 67.62%, supporting transfer of the temporal interface beyond Motus.

I. INTRODUCTION

CoRe-WAM addresses how a policy should compare current and historical visual evidence so observed change can inform action generation. It aligns historical features before differencing and adds the resulting residual alongside current-observation conditioning.

I. INTRODUCTION

  • Fixed-grid feature differences can mix scene evolution with changes in which content occupies a location when objects or the camera move.Supplying history alone does not define a meaningful temporal comparison.Source: §I · p. 1
  • CoRe-WAM transports historical pretrained features to current locations, computes signed differences, and injects validity-gated residuals while retaining current-observation conditioning.The lightweight adapter adds residuals to head-camera tokens after the frozen visual-language projector.Source: §I · p. 2
  • The Motus implementation keeps pretrained visual encoder and expert weights fixed, optimizing approximately 1.59 million parameters; TraceDelta also raises StarVLA clean success from 58.10% to 67.62%.The StarVLA result provides evidence of transfer beyond Motus.Source: §I · p. 2
  • 92.22% clean success across 50 RoboTwin 2.0 tasks under a 5,000-update budget is 3.56 points above Motus and 2.06 points above Raw Delta.On Astribot S1, CoRe-WAM achieves 28/30 successes on an in-distribution task and 25/30 on an OOD task.Source: §I · p. 2

II. RELATED WORK

CoRe-WAM addresses how a policy relates its current observation to observed history, rather than adding another future-prediction target. It constructs correspondence-aligned residuals from available observations to supplement visual conditioning.

A. World-Action Models and Predictive Representations

  • CoRe-WAM constructs correspondence-aligned temporal residuals from observed history instead of adding another future-prediction target.The residuals supplement visual conditioning while retaining the backbone’s native prediction objectives.Source: A. World-Action Models and Predictive… · p. 2
  • This approach poses a complementary question to predictive world-action models: how a policy should relate its current observation to observed history.The related work describes world-action models as connecting visual prediction with action generation.Source: A. World-Action Models and Predictive… · p. 2

B. Motion and Trajectory Representations for Control

Prior approaches use trajectories or learned motion representations to connect visual observations and robot behavior. CoRe-WAM instead uses correspondence to transport historical visual features before differencing, then supplies the change as an additive policy-conditioning residual.

B. Motion and Trajectory Representations for Control

  • Prior motion approaches predict trajectories, learn latent motion for pseudo-action labels, retrieve historical motion features, or transfer trajectory structure through auxiliary supervision.These examples include Any-point Trajectory Modeling, CoMo, MotionVLA, and 4D-WAM.Source: B. Motion and Trajectory… · p. 2
  • TraceDelta uses correspondence as a transport operator over historical VLM features, determining which historical content is compared with the present.Unlike retrieved motion features or trajectory-target supervision, the resulting change is expressed in pretrained visual-feature space.Source: B. Motion and Trajectory… · p. 2

C. Correspondence and Aligned Feature Comparison

  • CoRe-WAM brings alignment-before-comparison to robot-policy conditioning: correspondence transports historical features to current spatial support before a signed difference is mapped through a validity-gated adapter.Unlike change-localization outputs, the result is an additive residual for action generation, while the direct current-observation path remains intact.Source: C. Correspondence and Aligned Feature… · p. 3

III. METHOD

  • The Motus instantiation retains current-observation conditioning and adds correspondence-aligned visual change through alignment, signed differencing, and validity-gated residual conditioning.Its understanding, video-generation, and action experts interact through joint attention.Source: §III · p. 3

A. Overview and Formulation

The policy takes a current observation, robot state, and instruction and produces an action chunk, with a historical observation used to form temporal evidence. TraceDelta transports historical features to current token locations, compares aligned features, and adds gated residuals to selected current tokens.

A. Overview and Formulation

  • At time t, the policy receives current observation Ot, robot state st, and instruction ℓ, and produces action chunk At.A historical observation Oh is additionally used, with h = t − s and s the observation stride.Source: A. Overview and Formulation · p. 3
  • The same frozen vision-language encoder processes current and historical observations, whose spatial feature grids are denoted Ft and Fh.The original understanding input is the frozen projector applied to current features.Source: A. Overview and Formulation · p. 3
  • TraceDelta maps feature differences into the projected interface and adds residuals only at head-camera positions, preserving the direct current-observation pathway.The residual is added to the original understanding input after the frozen visual-language projector.Source: A. Overview and Formulation · p. 3

B. Trajectory-Grounded Feature Transport

  • Frozen Trace Anything estimates dense pixel trajectories in a shared 3D coordinate system to establish which historical content corresponds to each current visual token.For current token centers, the trajectory is evaluated at historical time and matched to the historical dense 3D field.Source: B. Trajectory-Grounded Feature Transport · pp. 3–4
  • The matched historical pixel is converted to a feature-sampling coordinate, accounting for resizing, cropping, and camera placement in the packed visual grid.The resulting backward map supports bilinear sampling of the estimated historical counterpart rather than same-coordinate content.Source: B. Trajectory-Grounded Feature Transport · p. 4
  • A binary validity mask admits only finite correspondences within the designated camera region; invalid coordinates contribute nothing to residual injection.The mask specifies admissible sampling support, not a learned semantic score.Source: B. Trajectory-Grounded Feature Transport · p. 4
  • Trajectory coordinates and trajectory-encoder features are not appended as policy tokens; correspondence determines where historical visual features are sampled.The transported feature represents the estimated historical counterpart of the current region.Source: B. Trajectory-Grounded Feature Transport · p. 4

C. Signed Temporal Differencing

Fixed-grid feature differences can mix visual change with changing spatial occupancy when objects or the camera move. TraceDelta instead compares current features with transported historical counterparts, preserving signed change in a shared feature space.

C. Signed Temporal Differencing

  • When objects or the camera move, same-location differences can mix changes in visual features with changes in spatial occupancy.A fixed-grid location may contain different scene content across observations.Source: C. Signed Temporal Differencing · p. 4
  • TraceDelta compares each current feature with its transported historical counterpart after establishing correspondence.This aligns the historical content before measuring feature change.Source: C. Signed Temporal Differencing · p. 4
  • Both operands remain in the same frozen feature space, and retaining the sign preserves the direction of feature change rather than displacement.The difference measures feature change after correspondence; identity correspondence reduces it to Raw Delta and does not generally make it zero.Source: C. Signed Temporal Differencing · p. 4

D. Validity-Gated Visual Conditioning

A lightweight adapter maps aligned feature changes into the policy representation, and validity gating makes the resulting residual selective. The residual augments current head-camera conditioning while the current-observation path remains intact.

D. Validity-Gated Visual Conditioning

  • A lightweight adapter maps aligned feature change into the Motus understanding dimension.The implementation uses dimensions 2048 → 256 → 512.Source: D. Validity-Gated Visual Conditioning · p. 4
  • Sampling validity gates the adapted residual, and post-adapter masking makes invalid locations contribute zero despite nonzero adapter biases.The residual is then scattered into native head-camera positions.Source: D. Validity-Gated Visual Conditioning · p. 4
  • The residual augments projected current-observation tokens at selected head-camera positions, with the current path intact.In the evaluated configuration, the 60-token region is a 6 × 10 grid at indices 10–69; other positions receive no direct residual.Source: D. Validity-Gated Visual Conditioning · p. 4

E. Training and Causal Inference

CoRe-WAM adapts with pretrained backbone weights fixed and uses online correspondence at inference when history is available. The reported experiments cover RoboTwin, StarVLA-based transfer, temporal-component controls, and physical deployment.

E. Training and Causal Inference

  • Approximately 1.59 million parameters are optimized while the pretrained backbone weights remain fixed.The optimized components are the temporal adapter, action-side low-rank updates, and existing action decoder.Source: E. Training and Causal Inference · p. 4
  • At inference, frozen Trace Anything estimates correspondence from current and past observations; without history, the temporal branch is bypassed.Training and inference share VLM preprocessing and head-camera token mapping.Source: E. Training and Causal Inference · p. 4

IV. EXPERIMENTS

  • The experiments examine RoboTwin performance, StarVLA-based transfer, temporal-component controls, and physical deployment on Astribot S1.Source: §IV · p. 4

A. Experimental Setup

CoRe-WAM achieves higher reported success than Motus on clean and randomized RoboTwin 2.0 evaluations across 50 tasks. The evaluation also reports setup details and task-level gains across manipulation behaviors.

A. Experimental Setup

  • The benchmark contains 50 manipulation tasks; adaptation uses clean data and 5,000 updates, with the pretrained backbone frozen.The evaluation averages success rates equally across tasks; published references retain their source protocols.Source: A. Experimental Setup · p. 5

B. Performance on RoboTwin 2.0

  • 92.22% clean success and 89.60% randomized success across 50 RoboTwin 2.0 tasks exceed Motus by 3.56 and 2.58 pp, respectively.Both results are aggregated across two seeds.Source: B. Performance on RoboTwin 2.0 · p. 5
  • Selected-task gains are +50 pp for Move Can Pot, +19 pp for Scan Object, +16 pp for Handover Mic, 13 pp for Dustbin, and 10 pp each for Place A2B Left and Place Bread Skillet.Source: B. Performance on RoboTwin 2.0 · p. 5
  • Figure 3 illustrates histories from hammering, handover, articulation, and stacking as examples of changing visual content available to TraceDelta.Source: B. Performance on RoboTwin 2.0 · p. 5

C. Transferability Evaluation

TraceDelta improves clean success when transferred to StarVLA and, on Motus, outperforms temporal and current-feature controls. Its correspondence-aligned residual adds a small trainable pathway while keeping the pretrained backbone frozen.

C. Transferability Evaluation

  • 9.52 percentage points of clean success improvement transfers TraceDelta to StarVLA, raising success from 58.10% to 67.62% under matched clean-data adaptation.The StarVLA variants are both adapted; the comparison does not reuse a Motus-trained checkpoint.Source: C. Transferability Evaluation · p. 5, Table III · p. 6

D. Ablation Study

  • 91.34% clean and 87.82% randomized success for Current-only exceed LoRA-only by 0.44 and 0.52 pp, while Raw Delta trails Current-only by 1.18 and 0.16 pp.Current-only tests added visual-adapter capacity without history; Raw Delta compares features at fixed grid locations.Source: D. Ablation Study · p. 7
  • 92.22% clean and 89.60% randomized success put complete TraceDelta 2.06 and 1.94 pp above Raw Delta, and 0.88 and 1.78 pp above Current-only.Correspondence alignment changes historical sampling without increasing adapter dimensions or policy input tokens.Source: D. Ablation Study · p. 7, Table IV · p. 7
  • 1,594,127 parameters—approximately 0.02% of the policy—are optimized for the complete Motus-based model, with the pretrained backbone and correspondence estimator frozen.The temporal adapter and residual scale account for 660,225 parameters; action LoRA adds 917,504 and the existing action decoder 16,398.Source: D. Ablation Study · p. 7, Table VI · p. 7

E. Real-World Deployment

On Astribot S1, CoRe-WAM completes both an in-distribution placement task and an out-of-distribution sequence more often than Motus. The deployed policy uses three RGB views and robot state, predicts action chunks, and reobserves between executed segments.

Deployment setup

  • The policy observes head, left, and right RGB views plus a 31-dimensional measured robot state, then predicts 48-step Cartesian action sequences.Denormalization and observed-chassis-pose augmentation convert its 31-dimensional predictions into the robot’s 34-dimensional command interface.Source: E. Real-World Deployment · p. 7, Figure 4 · p. 8

Physical tasks and success

  • CoRe-WAM succeeds in 28/30 in-distribution plush-toy placements (93.3%) and 25/30 OOD sequences (83.3%), versus Motus at 20/30 (66.7%) and 18/30 (60.0%).Success requires completing the full instruction; each task uses 30 trials.Source: E. Real-World Deployment · p. 8, Table VII · p. 8

Inference setup

  • Approximately 510 ms is the typical warm inference latency, excluding network communication and robot execution.Five flow-denoising steps generate 48-step action sequences for a 1.6 s horizon, executed in short segments with new observations between segments.Source: E. Real-World Deployment · p. 8

V. CONCLUSION

CoRe-WAM uses correspondence-aligned visual changes in a parameter-efficient world-action model while retaining current-observation conditioning. Its online correspondence and temporal feature processing introduce inference overhead.

V. CONCLUSION

  • CoRe-WAM aligns historical features before comparison and integrates the resulting changes through a lightweight residual interface while retaining current-observation conditioning.The model uses a frozen pretrained backbone and optimizes a compact set of temporal and action-side parameters.Source: §V · p. 8
  • Its modular design preserves the existing policy architecture, facilitating adaptation to new manipulation settings without full-model fine-tuning.Source: §V · p. 8
  • Online correspondence estimation and temporal feature processing introduce inference overhead, motivating work on latency and closed-loop responsiveness.Source: §V · p. 8
Loading 2609.27314v2…