PaperlayerView exact revision on arXiv

Source-linked AI summary

Identifying Habit, Physics, and Nuisance in Robot World Models

Jinting Hang, Zhenhui Cai

arXiv:2609.09210v1cs.ROcs.CV

TL;DR

The paper addresses the ambiguity between operator habit, shared physics, and observation nuisance in multimodal teleoperation predictions. It formalizes their separation with an SCM and complementary interventions, then freezes shared dynamics while adapting a thin interface. Across StackCube, DROID, and RH20T, this improves low-shot transfer over scratch training and preserves cleaner dynamics under corrupted adaptation data, while extending to pixel and multi-view settings.

  • Problem

    Multimodal next-observation predictions can entangle operator action-selection habit, shared physics, and observation nuisance, making adaptation brittle.

  • Method

    The paper uses an SCM, action and appearance interventions, habit-aware reverse scoring, and freeze-and-adapt training for a thin interface over shared dynamics.

  • Results

    Across StackCube, DROID, and RH20T, freezing shared dynamics improves low-shot transfer over scratch training and preserves cleaner dynamics under corrupted adaptation data.

  • Takeaways & Limitations

    Separating habit, physics, and nuisance supports disciplined few-shot adaptation without rewriting shared dynamics when the structural exclusions hold.

  • Takeaways & Limitations

    The evidence is limited by proxy interventions, inconclusive cross-task habit transfer, simulated closed-loop evaluation, and the absence of on-robot validation.

Abstract

from arXiv · show

Teleoperated demonstrations are often multimodal even when the underlying dynamics are nearly deterministic given the executed action. We argue that this multimodality typically mixes three factors--operator habit in action selection, shared physics, and observation nuisance--and that entangled next-observation predictors absorb all three. We formalize the split with a structural causal model a=g(h,z,u), z'=f(z,a), o=r(z,c), and test it with complementary interventions: replacing or shuffling actions at fixed state sharply increases next-state error, whereas appearance and camera changes should not; habit-aware reverse scoring improves ranking of feasible pasts without rewriting the dynamics. The associated adaptation rule is to freeze a shared physics readout and update only a thin interface. On StackCube, DROID, and RH20T this rule improves low-shot transfer relative to training from scratch, retains cleaner dynamics under corrupted adaptation data, and extends from proprioception to pixel observations with multi-view and multi-step checks. We do not equate latent actions with operator habit, and we do not target large-scale video generation benchmarks.

I. INTRODUCTION

The paper frames multimodal teleoperation predictions as an identification problem involving operator habit, shared physics, and observation nuisance. It proposes SCM-based interventions and thin-interface adaptation to separate these factors without rewriting shared dynamics.

  • Motivation: Teleoperation multimodality can reflect operator action-selection habits and observation changes rather than stochastic physics.Lighting, background, and camera pose alter pixels without changing robot configuration or mechanical consequences.
  • Structural causal model: The SCM assigns habit effects to actions, nuisance effects to rendering, and shared dynamics to z′ = f(z, a).The model excludes direct habit-to-state and nuisance-to-dynamics paths.
  • Adaptation: When the graph and state representation are adequate, few-shot adaptation freezes shared dynamics and updates only a thin policy or residual interface.The contribution includes evaluating freeze, finetune, and scratch protocols across simulation, proprioception, and pixel settings.
  • Interventions: The paper organizes physics-facing readouts around next-state error and habit-facing readouts around policy residuals and ranking.The intervention design explicitly separates these readout families.
  • Interventions: Fixed-state action replacement or shuffling probes physics, while appearance and viewpoint changes test nuisance stability.Operator swaps provide a stronger within-task habit proxy, and near-zero actions probe autonomous drift.

III. METHOD

The method compares freeze-and-adapt, joint finetuning, and scratch training, while pixel models use a structural joint readout with action-sensitive training objectives. A joint-facing target is used to prevent latent representations from becoming action-insensitive.

  • Adaptation protocol: At matched shot budgets, adaptation compares freezing a pretrained f* while training g, finetuning f and g jointly, and training both from scratch.The protocol is applied to StackCube, DROID, and RH20T settings.
  • Pixel model: For DROID images, the structural readout encodes observations into z, probes current joints, and predicts next joints through an action-conditioned residual.The formulation is z = Enc(o), ĵ = probe(z), and ĵ′ = ĵ + delta(z, a).
  • Training objective: Training aligns the probe with current joints, regresses next joints through delta, and uses a shuffled-action hinge to increase prediction error.An optional appearance-consistency term can be added, and high-motion transitions are retained to reduce idle-frame dominance.
  • Failure prevention: Optimizing only latent consistency can produce nearly action-insensitive representations, so the method uses a joint-primary target to avoid that collapse.Under latent-only training, appearance interventions may look harmless while wrong actions barely increase error.

C. Multi-Step Prediction

Multi-step evaluation unrolls the structural predictor over action sequences and contrasts true, shuffled, and appearance-perturbed inputs. The intended signature is accumulating action error without converting appearance nuisance into apparent dynamics, while habit priors improve reverse ranking.

  • Multi-step prediction: The predictor is unrolled for horizons H, comparing true action sequences with shuffled actions and appearance-perturbed observations.The evaluation tests whether intervention effects persist over multiple prediction steps.
  • Multi-step prediction: Action errors should accumulate with horizon, whereas appearance perturbations should not produce comparable error growth.The design treats horizon-dependent action sensitivity as distinct from nuisance stability.
  • Reverse scoring: Reverse scoring compares physics-only candidate-past scores with scores augmented by correct or incorrect habit priors.A useful prior should improve ranking without moving the ground truth outside the physics-feasible set.

E. Scope

The paper distinguishes operator habit from latent actions recovered from video and limits its claims to the evaluated intervention, adaptation, and dataset settings. Its experiments use multi-operator proprioception, exterior-camera DROID data, and fixed-budget comparisons across seeds.

  • Scope: Latent-action video methods address a different object from operator habit, and the paper does not study theory-of-mind models or large video-generation leaderboards.External video-action adapters appear only as a boundary case.
  • Datasets: RH20T provides multi-operator proprioceptive transitions, while DROID supplies exterior RGB, 7-DoF actions, joint states, and a second camera for evaluation.StackCube uses the ManiSkill Physical Kernel protocol.
  • Evaluation protocol: DROID pixel, multi-step, downstream, and reverse-scoring experiments report mean±std over seeds {0, 1, 2} under fixed episode and epoch budgets.Proprioceptive and Kernel experiments use the same freeze, finetune, and scratch template.
  • Metrics: The evaluation measures intervention error ratios, operator identification, reverse-scoring mass and win rates, low-shot adaptation error, and StackCube stacked success.These metrics cover physics sensitivity, habit ranking, transfer, and closed-loop behavior.

V. HABIT VERSUS PHYSICS

Action interventions sharply expose physics errors, while freezing the shared dynamics and adapting only the interface improves low-shot StackCube transfer over joint finetuning and scratch training.

  • Replacing the true action increases one-step z_p error by about 10.4×, while idle actions remain concentrated at fixed initial state.This supports action-conditioned branching from habit-induced action mixtures.
  • Freeze+g reaches 0.22±0.09 stacked success versus 0.10±0.07 for scratch across eight seeds, winning on 7/8 seeds.With demonstrations scaled by a×2, freeze+g remains ahead at 0.24±0.09 versus 0.13.

B. DROID Proprioception

DROID proprioception shows strong sensitivity to wrong actions and supports in-domain freeze+g adaptation, while pooled dynamics generalize across users more reliably than single-user fitting.

  • Wrong-action spread is about 0.28 versus 0.02 for true actions, and freeze+g improves on scratch at N=32 with error 0.00030 versus 0.00039.Retraining an in-domain residual model restores motion sensitivity and low-shot freeze gains across N ∈{16, . . . , 256}.
  • A leave-one-user pooled dynamics model approaches same-user error at fall/fself ≈0.38, whereas fitting on one other user is unstable at about 3.9×.
  • Pairing the true state with another operator’s action increases MSE by roughly 1362×, essentially matching unstructured action shuffle at 1352×.The result shows that operator actions can be manipulated within the task while holding the state context fixed.
  • Freezing f under a×2 over-forced demonstrations preserves clean-physics next-state error by about two orders of magnitude relative to finetuning f.Finetuning absorbs the corrupted habit into the learned law.

D. Synthetic Flow Matching and Training Objectives

Synthetic flow matching shows that action interventions reveal dynamics sensitivity, but diagnostic sensitivity alone does not guarantee useful few-shot transfer. Habit-aware scoring improves ranking while preserving the distinction between habit and physics.

  • Training objectives: A mid-strength wrong-action hinge preserves both strong action sensitivity and low-shot freeze+g gains, whereas an over-hard hinge can destroy transfer.In-domain robot models recover both properties, showing that the training objective must balance diagnostic sensitivity with adaptation quality.
  • Synthetic flow matching: Shuffling actions increases conditional flow-matching loss by about 60×, while freezing the physics velocity field and adapting a thin g helps for N ∈{5, . . . , 100}.The toy result reproduces the paper’s qualitative freeze-and-adapt pattern.
  • Multi-step checks: Wrong-action ratios rise from 1.9±0.2 to 13.7±0.6 as H increases from 1 to 10, while appearance ratios remain near one.Longer horizons amplify action mistakes without making photometric nuisance resemble dynamics.
  • Reverse scoring: Correct habit priors increase ground-truth posterior mass by 0.040±0.002, while the correct user’s model beats a wrong user’s on 0.649±0.015 of trials.Direct user identification remains near chance, so reverse scoring is used for ranking rather than hard operator recognition.

C. Pixel Low-Shot Adaptation

Pixel adaptation benefits from freezing the encoder and probe while updating only a thin residual interface, improving low-shot performance over scratch. The section also identifies interface, objective, representation, and habit-transfer boundaries for this prescription.

  • Pixel low-shot adaptation: Freezing Enc and the probe while adapting only delta improves over scratch at low N and remains competitive with full finetuning.The result is reported in Fig. 9 and Table V for pixel low-shot next-joint prediction.
  • Observation channel: A proprioceptive oracle reaches about 0.007 next-joint MSE, indicating that pixels serve as an observation channel into the same structural target rather than replacing joint physics.This comparison links pixel adaptation to the underlying joint-prediction target.
  • Failure boundaries: An overly coarse RH20T command interface can make freeze+g ineffective even when oracle-action-conditioned dynamics remain accurate.This failure diagnoses the interface rather than disproving shared physics.
  • Failure boundaries: Aggressively maximizing wrong-action ratios can harm low-shot freeze+g, whereas moderate hinges and in-domain robot models preserve both properties.Action sensitivity is therefore not sufficient by itself for transfer quality.
  • Representation boundary: Frozen simulator encoders on DROID images remain only partially action-sensitive, while proprioception, in-domain dynamics, and joint-primary pixel models carry the robot evidence.Representation provenance affects whether freezing is useful.

VIII. RELATED WORK

The paper situates its contribution among world models, latent-action and inverse-dynamics methods, imitation and adapter approaches, and invariant or causal learning. It distinguishes executed actions from latent actions and frames freezing as a targeted alternative to entangled adaptation.

  • World models and visual dynamics: World-model and visual-foresight methods learn future predictions for control, whereas this paper asks when multi-operator, multi-camera data justify freezing a shared f.The contrast centers on separating shared dynamics from observation and operator variation.
  • Latent actions and inverse dynamics: Latent-action methods recover controllable bottlenecks from video, but this paper keeps executed actions as intervention handles and treats latent actions as distinct from operator habit.The distinction defines the paper’s intervention object.
  • Imitation, preferences, and adapters: Preference-conditioned, hierarchical, and parameter-efficient adapter methods address heterogeneous demonstrations, but seldom publish freeze-f diagnostics under nuisance interventions.Thin interfaces are presented as practical cousins of adapters, not as equivalent methods.
  • Invariance, causality, and corpora: Domain randomization, view-invariant learning, and causal evaluation address nuisance variation, while this work evaluates those concerns on manipulation resources including DROID and RH20T.The paper combines intervention-based analysis with public manipulation corpora.

IX. LIMITATIONS

The paper concludes that habit, shared physics, and observation nuisance can be separated approximately, but freezing dynamics is justified only when the relevant exclusions and interfaces hold. Its evidence spans simulation, proprioception, and pixels while documenting identifiable failure cases.

  • Conclusion: The proposed SCM and interventions support approximate separation of operator habit, shared physics, and observation nuisance.The conclusion frames the separation as approximate rather than universal.
  • Conclusion: Freezing a shared dynamics or state readout and adapting a thin interface is presented as more disciplined than rewriting the law when exclusions hold.The supporting evidence spans simulation, multi-operator proprioception, and DROID pixels.
  • Conclusion: The paper explicitly includes cases where the prescription fails for identifiable reasons, rather than treating freezing as universally appropriate.This bounds the conclusion to settings where the structural assumptions and interfaces are adequate.
Loading 2609.09210v1…