PaperlayerView exact revision on arXiv

Source-linked AI summary

Register-Routed Delayed Fusion: Rewiring Shortcut-Prone Observation Fusion in Visuomotor Imitation

Jieting Long, Weidong Cai, Weiming Zhi

arXiv:2610.02813v1cs.RO

TL;DR

Dense fusion lets compact cues influence visual representations from the first encoder layer, raising the question of whether controlling that route improves visuomotor behavior. RRDF stages exchange through registers while retaining compact action conditioning; it matches or improves nominal performance and is favored in the tested shifted evaluations.

  • Problem

    Dense fusion allows action-correlated compact cues to influence spatial visual representations early, motivating the question of whether controlling this route improves visual responsiveness and policy behavior.

    Source: §I · p. 1

  • Method

    RRDF masks direct compact–visual attention and uses an isolate–collect–route schedule for register-mediated exchange while preserving compact conditioning for the action generator.

    Source: §I · pp. 1–2

  • Results

    Across evaluated simulation and real-robot tasks, RRDF matches or improves nominal dense-fusion performance and achieves higher success under the tested appearance and position shifts.

    Source: §I · p. 1, §VI · p. 8

  • Takeaways & Limitations

    The results support controlling cross-modal propagation while retaining compact action conditioning, with backbone and action-generation transfer supporting RRDF’s modular flexibility.

    Source: §VI · p. 8

Abstract

from arXiv · show

Visuomotor imitation policies combine high-dimensional visual observations with compact signals such as proprioception, and their fusion topology determines when and through which tokens these streams interact. In dense token fusion, visual tokens may attend directly to compact tokens from the first encoder layer, allowing action-predictive compact cues to influence spatial visual representations early in their formation. We ask whether controlling this route improves visual responsiveness and policy behavior. We introduce Register-Routed Delayed Fusion (RRDF), which masks direct compact-visual attention and stages cross-modal interaction through a learned register workspace. Its isolate-collect-route schedule protects an early stream-separated prefix and later permits only register-mediated exchange, while compact conditioning remains available to the native action generator. Across five simulation tasks and three real-robot tasks, RRDF matches or improves dense ACT under nominal conditions. Appearance-shift evaluations on four simulation tasks and held-out-position evaluations on three real-robot tasks also favor RRDF. Phase-matched input probes show lower measured state-to-image sensitivity, while ablations indicate that adding registers alone does not reproduce the full performance gain. These results support controlling cross-modal propagation while retaining compact action conditioning.

I. INTRODUCTION

The paper asks whether controlling when compact signals enter visual representations can improve visuomotor behavior while preserving compact action conditioning. It proposes RRDF and evaluates it on nominal and shifted tasks, alongside sensitivity probes and topology ablations.

I. INTRODUCTION

  • The motivation is that action-correlated compact cues can influence spatial visual representations from the first dense-fusion layer, potentially encouraging reliance on those cues over visual evidence.The authors relate this concern to causal confusion and reported proprioceptive dominance in vision–proprioception policies.Source: §I · p. 1
  • RRDF blocks direct compact–visual attention and stages exchange through learned registers, while retaining compact conditioning for the native action generator.Its isolate–collect–route schedule protects an early stream-separated prefix before later register-mediated interaction.Source: §I · pp. 1–2
  • Across five simulation and three real-robot tasks, RRDF matches or improves nominal dense-fusion performance, while tested appearance and position shifts favor RRDF.The evaluation also includes phase-matched input-sensitivity probes and fusion-topology ablations.Source: §I · pp. 1–2
  • The contributions include characterizing permitted encoder dependencies and showing lower measured state-to-image sensitivity, while register capacity alone does not explain the gains.The paper evaluates nominal performance on five simulation and three real-robot tasks, with appearance shifts on four simulation tasks and held-out positions on three real-robot tasks.Source: §I · p. 2

II. RELATED WORK

Prior work studies multimodal token fusion, shortcut learning, and learned bottleneck or register tokens. This paper frames layer-wise communication routes between explicit modality-token groups as a design variable.

II. RELATED WORK

  • Prior work addresses multimodal fusion through shared feature representations, shared Transformer token spaces, and constrained communication, while shortcut-learning remedies modify data, observation design, regularization, or optimization.The related work also identifies proprioceptive dominance as a concern and positions RRDF as controlling encoder communication while retaining compact action conditioning.Source: §II · p. 2

A. Multimodal Observation Tokenization

  • The analysis represents camera observations and proprioceptive state as spatial visual tokens and compact conditioning tokens, with tokens from multiple time steps concatenated for observation histories.Compact tokens may include proprioception and optional policy-internal conditioning.Source: A. Multimodal Observation Tokenization · pp. 2–3

B. Multimodal Fusion Topologies

  • Feature fusion combines modality-level features into a shared condition, whereas dense token fusion preserves explicit modality tokens and permits unrestricted within- and cross-group communication.The paper represents depth-dependent, potentially directional communication with a routing matrix.Source: B. Multimodal Fusion Topologies · p. 3

IV. REGISTER-ROUTED DELAYED FUSION

RRDF uses depth-dependent attention masks to prevent direct compact–visual exchange and route later cross-modal influence through registers.

IV. REGISTER-ROUTED DELAYED FUSION

  • RRDF keeps direct compact–visual attention masked throughout the fusion encoder.The direct cross-group edges are excluded rather than merely down-weighted.Source: §IV · p. 3
  • During the protected prefix, the primary streams cannot read the register workspace.Source: §IV · p. 3
  • Later cross-modal exchange occurs only through register-mediated routes.Source: §IV · p. 3

A. Register-Augmented Fusion Encoder

RRDF appends learned registers to the fusion encoder and partitions them for source-specific access before forming a shared workspace. The resulting conditioning memory is passed to the action generator through its native interface.

A. Register-Augmented Fusion Encoder

  • A bank of learned registers is appended to the encoder sequence and divided into compact-facing and visual-facing subsets.Source: A. Register-Augmented Fusion Encoder · p. 3
  • During isolation, each register subset attends only to its corresponding modality and registers; from collection onward, both subsets share routing permissions.The partition defines routing permissions without assigning fixed semantic roles, and the registers are learned internal states rather than observations, action queries, or prediction targets.Source: A. Register-Augmented Fusion Encoder · p. 3
  • The encoder supplies its final conditioning memory through the action generator’s native interface, retaining compact tokens as conditioning without changing the generator architecture.Source: A. Register-Augmented Fusion Encoder · p. 3

B. Delayed Fusion via Isolate-Collect-Route

The fusion encoder uses an isolate–collect–route schedule to delay primary-stream exchange until registers have jointly collected information. Direct compact–visual attention remains controlled while the register workspace mediates later communication.

B. Delayed Fusion via Isolate-Collect-Route

  • An additive attention mask implements group-level routing and combines with native padding, temporal, or causal masks.The routing groups are compact tokens, compact-facing registers, visual-facing registers, and visual tokens.Source: B. Delayed Fusion via… · p. 3
  • Figure 4 depicts nominal, bright-floor, and recolored observations and encodes RRDF gains as hatched extensions above dense-fusion success bars.Geometry and dynamics are unchanged across appearance shifts.Source: Figure 4 · p. 4
  • The schedule uses Liso isolation layers, Lcol collection layers, and Lroute routing layers, with at least one collection and one routing layer.The default topology mediates communication in both directions; Ldelay = Liso + Lcol sets when jointly informed register states become accessible, and R sets mediated interface width.Source: B. Delayed Fusion via… · p. 4
  • Isolation, collection, and routing progress from source-specific register accumulation to a shared workspace and then register-mediated redistribution.Primary streams remain self-only during collection; during routing they may read the full register bank, while direct compact–visual attention stays masked.Source: B. Delayed Fusion via… · p. 4

C. Structural Properties of the Fusion Topology

The topology excludes direct compact–visual edges and routes cross-modal influence through registers. During the isolation and collection prefix, visual-token states are computationally independent of the current compact input under stated architectural conditions.

C. Structural Properties of the Fusion Topology

  • Table I reports simulation performance as success rate or Push-T score, with best-of-final-ten and final-ten summaries.Figure 5 presents matched dense-fusion and RRDF rollouts for three real-robot behaviors in nominal and held-out-position conditions, marking representative dense-fusion failures and RRDF successes.Source: Table I · p. 4, Figure 5 · p. 5
  • Direct compact–visual attention is excluded architecturally at every fusion-encoder layer, while masked edges exclude dependencies and unmasked edges merely permit them.Cross-modal influence within the encoder must therefore pass through at least one register state.Source: C. Structural Properties of the Fusion… · pp. 4–5
  • Visual-token states are independent of the current compact input through isolation and collection, for fixed parameters, group-separable tokenizers, and standard Transformer blocks with masked self-attention as the only cross-token operation.This is forward-input independence, not statistical independence or a guarantee against shortcut learning; images may still encode robot configuration.Source: C. Structural Properties of the Fusion… · p. 4
  • The earliest register-mediated compact-to-visual dependency enters the visual stream at Ldelay+1, when a routing layer makes jointly informed register states accessible.The reverse path is analogous.Source: C. Structural Properties of the Fusion… · p. 5

V. EXPERIMENTS AND RESULTS

The experiments assess task performance, input sensitivity, and fusion-topology ablations. RRDF is instantiated in ACT and compared with dense fusion while matching non-fusion modules, data, and optimization settings.

V. EXPERIMENTS AND RESULTS

  • The evaluation covers performance under nominal and shifted observations, representation- and action-level input sensitivity, and register-capacity and communication-schedule effects.The sensitivity probes characterize responses to specified input changes rather than directly identifying shortcut use.Source: §V · p. 5
  • RRDF is instantiated in ACT and compared against dense fusion with non-fusion modules, data, and optimization settings matched.The action generator remains unchanged.Source: §V · p. 5
  • The default four-layer encoder uses (Liso, Lcol, Lroute) = (1, 1, 2), with two registers in simulation and four on the real robot.Compact tokens comprise proprioceptive state and policy-latent tokens; registers are evenly split between compact-facing and visual-facing subsets.Source: §V · p. 5

A. Simulated Experiments

RRDF matches or improves dense fusion on nominal simulated tasks and has higher observed performance under appearance shifts and real-robot task conditions. The real-robot evaluation includes held-out placements, while an illustrative rollout shows recovery after a placement error.

A. Simulated Experiments

  • The simulation benchmarks include four manipulation tasks measured by success rate and Push-T measured by its maximum normalized target-coverage reward during a rollout.For manipulation tasks, each checkpoint uses three matched seeds with 50 rollouts each; Push-T uses 150 scene-initialization seeds with one rollout each.Source: A. Simulated Experiments · p. 5
  • RRDF matches or improves dense fusion at the selected best checkpoint and improves every final-ten-checkpoint average on the simulated tasks.Gains are largest on Square and Stack-D1; Coffee-D2 remains challenging for both methods.Source: A. Simulated Experiments · p. 5
  • RRDF achieves higher observed success in all eight evaluated task–appearance-shift conditions.Rendering alone was modified, with task dynamics and rollout initializations matched across variants.Source: A. Simulated Experiments · pp. 5–6

B. Real-World Experiments

  • RRDF has higher success on all six real-robot task–condition pairs, with wider held-out-position gaps for Soap Pick-up and Plate Stacking.Toolbox remains the most challenging task.Source: B. Real-World Experiments · p. 6
  • In the Toolbox rollout, RRDF reopens the lid after a failed closure, corrects placement, and completes the task, while the dense baseline remains stuck.Figure 6 marks the baseline failure and RRDF’s subsequent corrective actions.Source: B. Real-World Experiments · p. 6, Figure 6 · p. 6

C. State-to-Image and Action Sensitivity

Phase-matched interventions probe state- and image-induced changes in visual representations and action outputs. RRDF shows lower measured state-to-image sensitivity than dense fusion, while proprioception remains available to the action generator.

C. State-to-Image and Action Sensitivity

  • The visual-token change measure is normalized by nominal activation scale, and larger state-induced values indicate stronger state influence on spatial visual representations.The analysis also compares state-induced change with image-induced change because small state effects alone could reflect general input insensitivity.Source: C. State-to-Image and Action Sensitivity · p. 6
  • RRDF has zero state-induced change in spatial visual tokens throughout the protected prefix, and lower state-to-image sensitivity than dense fusion after register-mediated readback begins.The comparison uses representation change and the state-to-image ratio Γℓ at matched depths.Source: C. State-to-Image and Action Sensitivity · p. 7
  • RRDF substantially reduces dense fusion’s high relative action sensitivity to state rather than matched image interventions.The policy latent is fixed to its prior mean, while state or image inputs are replaced using samples from another episode at the same normalized trajectory phase.Source: C. State-to-Image and Action Sensitivity · pp. 6–7
  • Proprioceptive state tokens remain directly available to the action generator despite RRDF’s reduced measured state-to-image sensitivity.The probes characterize responses to specified input changes rather than directly identifying shortcut use.Source: C. State-to-Image and Action Sensitivity · p. 7

D. Architectural Ablations

The ablations distinguish the effects of delaying direct fusion, using a register workspace, and choosing how registers communicate. RRDF’s default schedule has the highest observed success, while gains persist across tested register counts and readback settings.

D. Architectural Ablations

  • Adding registers to dense fusion yields only a small performance change, while delaying direct compact–visual fusion without registers produces a much larger improvement.Register-mediated variants achieve lower ΓL sensitivity while improving performance.Source: D. Architectural Ablations · p. 7
  • The full schedule achieves the highest mean success, although simplified isolation-only and collection-only schedules also improve over dense fusion.Collection-only yields the lowest state sensitivity without the highest success, so minimum sensitivity alone is not a sufficient model-selection criterion.Source: D. Architectural Ablations · p. 7
  • Source-specific register access improves performance and reduces compact-to-image sensitivity compared with R-Shared; the full schedule combines source-specific accumulation with joint collection.R-Coupled and R-RestrCol retain most of the gain while changing register interactions across schedule stages.Source: D. Architectural Ablations · p. 7
  • Register counts R = 2–6 form a broad high-performing region with no consistent benefit from further increasing workspace width, and gains persist through the no-encoder-readback endpoint R2D4.R2D2 has the highest observed success and is retained as default for its early protected prefix and later register-mediated feedback.Source: D. Architectural Ablations · p. 7, Figure 8 · p. 7
  • Both comparisons on Square favor RRDF over dense spatial-token fusion when using DINOv2-Base and over Diffusion Policy’s native feature concatenation.The comparisons extend the evaluation beyond default ACT.Source: Table IV · p. 7, Table V · p. 7, D. Architectural Ablations · p. 8

VI. CONCLUSION

RRDF delays register-mediated feedback while preserving compact action conditioning. Across the evaluated simulation and real-robot tasks, it matches or improves nominal performance and achieves higher success under tested shifts.

VI. CONCLUSION

  • RRDF masks direct compact-visual exchange and delays register-mediated feedback while preserving compact action conditioning.The source-partitioned attention schedule is presented as a modular observation-fusion component.Source: §VI · p. 8
  • Across evaluated simulation and real-robot tasks, RRDF matches or improves nominal performance and achieves higher success under tested appearance and position shifts.Source: §VI · p. 8
  • Ablations favor structured fusion over registers alone, and phase-matched probes show lower measured state-to-image sensitivity.Backbone and action-generation transfer results support compatibility beyond the default ACT configuration.Source: §VI · p. 8

Related papers

Loading 2610.02813v1…