PaperlayerView exact revision on arXiv

Source-linked AI summary

CST-WM: A Causally Structured World Model for Embodied Visual Tracking

Junyi Hu, Shuaihang Yuan, Jiazhao Liang, Yi Fang

arXiv:2609.06302v2cs.CVcs.RO

TL;DR

Logged tracking data can make actions spuriously predict target evidence, leaving a gap in planning for target loss and counterfactual actions. CST-WM structures the transition so actions affect future evidence through robot motion and views, then plans over imagined target evidence. It improves tracking outcomes and counterfactual action ranking across the reported settings.

  • Problem

    Logged actions correlate with target position and evidence, while reactive trackers offer little basis for choosing multi-step recovery actions and generic action-conditioned models can learn a direct action-to-evidence shortcut.

    Source: §1 · pp. 1–2

  • Method

    CST-WM separates target-evidence, robot, and observation branches, blocks direct current-action input to target evidence, and uses rollout-based planning over detector-distilled evidence from imagined views.

    Source: §1 · p. 2, §3.2.2 · p. 4, §3.2.3 · p. 4

  • Results

    CST-WM improves following, safety, and re-acquisition across EVT-Bench and Habitat 3.0; counterfactual ranking consistency is 0.68 against 0.55 and re-acquisition is 0.76 against 0.61 for the transition without the mask.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    One model supports stable following and re-acquisition, and the reported real-robot evaluation shows it holds under occlusion, distractor crossing, and fast motion.

    Source: §1 · p. 2, §5 · p. 9

    CST-WM inherits detector errors, keeps no target identity, does not track a hidden target past the horizon, infers target motion from one current view, and trades planning horizon and candidates for time.

    Source: §5 · pp. 9–10

Abstract

from arXiv · show

Embodied visual tracking requires a robot to choose actions that keep a moving target observable at a suitable distance, and to recover it after occlusion, out-of-view drift, or distractor crossings. We cast the task as planning over future target evidence with an action-conditioned world model. In logged tracking data, however, the behavior policy's actions are correlated with where the target is, so a generic predictor can learn a shortcut: it writes the current action directly into its prediction of target evidence, instead of letting the action affect that evidence only by moving the robot and changing what it observes. We call this failure causal hallucination; the resulting rollouts look plausible but rank candidate actions for the wrong reason. We propose CST-WM, a causally structured world model whose state is split into target-evidence, robot, and observation branches. Its transition removes the same-step edge from action to target evidence but keeps the path through robot motion and the resulting views, so candidate actions are still distinguished by their predicted ego-motion. With rollout-based model-predictive control, a single model handles both steady following and re-acquisition after target loss. On EVT-Bench and Habitat 3.0, covering standard tracking, target-loss recovery, and cross-dataset transfer, CST-WM improves following, distance-range control, safety, and re-acquisition over reactive trackers and world-model baselines, and removing the action mask causes the largest drop in re-acquisition among our ablations. Offline, CST-WM has lower multi-step rollout error, and its ranking of candidate actions agrees better with the simulator's. On a Unitree Go2 quadruped, CST-WM succeeds in 20 of 30 real-world trials under occlusion, distractor crossing, and fast motion, against 14 for TrackVLA.

1 INTRODUCTION

Logged tracking data can make actions spuriously predict target evidence, undermining action choices for recovery. CST-WM blocks that direct path while preserving action effects through robot motion and future views, and improves tracking outcomes across benchmarks.

1 INTRODUCTION

  • Logged actions correlate with target evidence, so a predictor can learn a direct action-to-evidence shortcut that appears plausible but misranks unchosen actions during planning.The discrepancy is difficult to detect on logged action sequences and emerges for counterfactual candidates.Source: §1 · pp. 1–2
  • CST-WM separates target evidence, robot motion, and observation dynamics, excluding the current action from the target-evidence update while retaining its path through robot motion and resulting views.The action can still distinguish candidate sequences through their predicted ego-motion and views.Source: §1 · p. 2, Figure 1 · p. 2
  • A compact signal for target visibility and apparent scale lets one rollout-based planner support stable following and re-acquisition when the target is hidden.Candidates are ranked by whether their imagined views bring the target back.Source: §1 · p. 2
  • CST-WM improves following, distance-range control, safety, and re-acquisition on EVT-Bench and Habitat 3.0, with higher rollout fidelity and better counterfactual action ranking.The evaluation also includes structural diagnostics, planning cost, and real-robot trials.Source: §1 · p. 2

2 RELATED WORK

Embodied tracking methods strengthen following and robustness, but commonly map recent observations to actions rather than comparing future target observability. World models permit multi-candidate rollouts; CST-WM constrains how actions influence predicted target evidence.

2.1 EMBODIED VISUAL TRACKING

  • Embodied tracking must preserve visibility, regulate distance, and remain safe; detection-plus-planning systems degrade when targets are occluded or visually ambiguous in clutter.The literature includes reactive, offline-RL, occlusion-aware, social-aware, and vision-language-action approaches.Source: §2.1 · pp. 2–3
  • Reactive tracking methods, including occlusion-aware and vision-language-action variants, strengthen robustness but mainly map recent observations to actions rather than comparing future observability and scale.This subsection contrasts reactive policies with multi-step action comparison.Source: §2.1 · p. 3

2.2 WORLD MODELS

  • World models roll out latent futures to compare candidate actions; CST-WM adds a constraint that excludes the current action from target evidence while retaining its route through robot and observation branches.Related work includes Navigation World Models for egocentric planning and WoMAP's detector-distilled reward head for active object localization.Source: §2.2 · p. 3

3 METHOD

CST-WM plans from compact target evidence and predicted observations, with the current action affecting future evidence through robot motion rather than direct injection. Its structured transition, detector-distilled evidence head, and rollout planner support following and recovery, including when the target is hidden.

3.1 PROBLEM DEFINITION

  • The task chooses actions from egocentric observations to keep a moving target visible at a 1–3 m following distance, judging actions by what the robot sees afterward.The target moves independently and may be hidden by the scene.Source: §3.1 · p. 3

3.2.1 STRUCTURED STATE

  • The state combines detector-derived visibility and box-area evidence, a frozen-VAE scene latent, and robot pose integrated from past actions.The evidence provides observability and distance information without privileged geometry, and the transition conditions on the current step only.Source: §3.2.1 · p. 3

3.2.2 WHERE THE ACTION MAY ENTER

  • The current action is excluded from the target-evidence representation, while a detector-distilled head reads imagined evidence from observation-branch tokens.The action retains a physical route through robot motion and predicted views, so candidates that move the camera differently can receive different evidence.Source: §3.2.2 · p. 4

3.2.3 TRANSITION ARCHITECTURE

  • Each transition layer updates target evidence first without the action, then robot tokens carrying the action, and finally observation tokens conditioned on both branches.This construction leaves no direct action-to-evidence path while allowing action effects through robot motion and future observations.Source: §3.2.3 · pp. 4–5

3.2.4 TRAINING

  • Training generates only the next observation latent and jointly trains an evidence head against detector readings of real next frames.The evidence loss reaches all three branches through the observation branch; an auxiliary distance-range loss also trains the model.Source: §3.2.4 · p. 5

3.3 PLANNING WITH THE WORLD MODEL

  • CEM evaluates imagined action sequences for visibility, reference scale, valid actions, and safe predicted states, executes the best first action, and replans.When the target is hidden, candidates still predict different views; candidates that bring it back collect visibility and scale rewards, while failure to recover within the horizon leaves only penalties.Source: §3.3 · p. 5

4 EXPERIMENTS

Across scene-disjoint EVT-Bench and Habitat 3.0 tests of standard tracking, target-loss recovery, and transfer, CST-WM outperforms reactive trackers and world-model baselines. Offline analyses assess rollout fidelity, candidate ranking, and the action mask’s structural effects.

4.2 QUANTITATIVE COMPARISON

  • Across EVT-Bench STT, Habitat 3.0 tracking, and EVT-to-Habitat transfer, CST-WM leads the reported comparisons on core tracking, distance, safety, and success metrics.On EVT-Bench STT it reaches 88.7 SR and 83.4 TR; on Habitat 3.0 it reports F 0.53, DRS 0.70, CR 0.27, and ES 0.61. Transfer gains are +0.05 F, +0.07 DRS, +0.07 ES, and −0.05 CR.Source: Table 1 · p. 6, §4.2 · p. 6

4.4.1 MULTI-STEP ROLLOUT FIDELITY

  • Across horizons of 1, 5, 10, 20, and 40 steps, CST-WM has lower latent prediction error and higher target-visibility AUROC, with the gap growing at longer horizons.The comparison measures latent prediction error and target-visibility AUROC against simulated futures.Source: Figure 4 · p. 6, §4.4.1 · p. 7

4.3 TARGET LOSS AND RE-ACQUISITION

  • Across all four target-loss conditions, CST-WM has the highest Re-acq., Post-F, and Rec-ES, and the shortest TTR.Its Re-acq. margin over adapted NWM is 0.09 under short occlusion and 0.15–0.16 under the other conditions; TTR is 6.4–10.6 steps versus 8.1–14.0.Source: §4.3 · p. 6, Table 2 · p. 7

4.4.2 PLANNING-VALUE CONSISTENCY

  • CST-WM’s candidate-action ranking agrees best with simulator outcomes, although predicting logged futures more accurately without the mask does not preserve counterfactual ranking.Simulator agreement is Spearman ρ 0.68 versus 0.47 for adapted NWM and 0.52 for the TrackVLA-style predictor; without the mask, logged-future visibility AUROC is 0.701 versus 0.675, but ranking is ρ 0.55 versus 0.68 with the mask.Source: Table 3 · p. 7, §4.4.2 · p. 7

4.5 STRUCTURAL AND INVARIANCE ANALYSIS

  • With the humanoid still and the robot moving, CST-WM keeps target evidence more stable (0.98 versus 0.61) while predicted views follow the viewpoint (0.84 versus 0.78).The action-dependence diagnostics also show near-zero action influence on target evidence for CST-WM, as required by the mask.Source: §4.5 · pp. 7–8

4.6 RUNTIME AND REAL-WORLD TRACKING

  • CST-WM succeeds in 8, 6, and 6 of 10 real-world trials across occlusion, distractor crossing, and fast motion, versus TrackVLA’s 4, 6, and 4.The default planning budget takes 267 s from observation to action; the fast budget takes 22.5 s and the robot budget less than 8.2 s.Source: §4.6 · pp. 8–9, Table 5 · p. 8

4.7 ABLATION STUDY

  • Removing only the target-evidence mask causes the largest ablation drop in re-acquisition (0.76 to 0.61) and planning-value consistency (0.68 to 0.55).A single-branch transition lowers following and consistency; dropping the distance-aware loss lowers all four reported metrics by 0.04–0.05, while a deterministic predictor weakens recovery.Source: §4.7 · p. 9, Table 6 · p. 9, Table 7 · p. 9
  • Among deployable target-evidence read-outs, the jointly trained head leads on every metric, including re-acquisition at 0.76, and reads an imagined step in 0.018 s.The alternatives report re-acquisition of 0.43 for a frozen transition, 0.57 for a separate latent head, and 0.46 for decode-and-detect.Source: Table 8 · p. 9, §4.7 · p. 9

5 CONCLUSION

CST-WM plans tracking over imagined target evidence, using a masked transition so actions affect later evidence through robot motion and resulting views. It improves tracking-related outcomes across simulation benchmarks and real-robot trials, with limits from detector-derived evidence and sequential planning.

5 CONCLUSION

  • CST-WM improves following, safety, and re-acquisition on EVT-Bench and Habitat 3.0, transfers between them, and succeeds on a real robot.Under paired simulator interventions, its candidate-action ranking agrees better with simulated outcomes than the unmasked transition (ρ 0.68 against 0.55), and re-acquisition is 0.76 against 0.61.Source: §5 · p. 9
  • The transition masks the current action from target evidence while allowing it to affect later evidence through robot motion and the views produced.A detector-distilled head reads target evidence from each imagined future’s predicted view.Source: §5 · p. 9
  • CST-WM relies on detector-derived evidence, has no target identity, and does not track a hidden target beyond its planning horizon.Its transition conditions on the current view only, so target motion is inferred from one frame.Source: §5 · p. 9
  • Sequential denoising passes make the robot’s planning budget trade off horizon and candidate count against time.Source: §5 · p. 10

A METHOD DETAILS

The method details describe training signals for distance and target evidence, candidate-sequence optimization, and motion and safety constraints. The algorithm summaries outline model training and receding-horizon planning.

A METHOD DETAILS

  • The auxiliary distance-aware loss trains a lightweight head on the target-evidence branch to predict whether simulator-relative distance lies within the following range.Distance labels are used only in training; the planner does not use the head.Source: A METHOD DETAILS · p. 14
  • The evidence head is trained against GroundingDINO readings of confidence and normalized target area, then read from final-pass observation tokens at each imagined step.The head and its loss are added after a 12k-step denoising warm start.Source: A METHOD DETAILS · p. 14
  • At each CEM iteration, the sampling distribution is updated using the indices of the top-K candidates.Source: A METHOD DETAILS · p. 14
  • The admissible action set enforces platform velocity and smoothness limits; safe rollouts use a robot-branch clearance proxy, not obstacle distance.The proxy uses a 0.20 m safety margin and predicted displacement over one control period.Source: A METHOD DETAILS · p. 14

C DETAILED EXPERIMENTAL SETTING

The experimental setting evaluates standard tracking, target-loss recovery, and cross-dataset transfer using egocentric observations and platform actions. It specifies scene-disjoint data splits, matched world-model training controls, reference systems, and a separate real-robot protocol.

C DETAILED EXPERIMENTAL SETTING

  • EVT-Bench evaluates single-target tracking, while Habitat 3.0 adds richer geometry, more frequent occlusion, target-loss recovery, and transfer from EVT-Bench training.Test-time inputs are egocentric observations and platform actions, without humanoid state, future trajectory, or distance.Source: C DETAILED EXPERIMENTAL SETTING · p. 14
  • The training algorithm encodes observations and detector evidence, applies the masked transition, and updates the model using denoising, evidence-head, and auxiliary distance losses.Source: C DETAILED EXPERIMENTAL SETTING · p. 15
  • The planning algorithm scores imagined evidence across CEM action-sequence rollouts, selects elites to update the sampling distribution, and executes the first action of the best sequence.Source: C DETAILED EXPERIMENTAL SETTING · p. 15
  • Transitions are built from consecutive simulator frames, with scene-disjoint train, validation, and test splits; learned models share data and training settings, and ablations change one component.Checkpoints use validation following and distance-range scores, with lower collision rate breaking ties; learned models use three seeds.Source: C DETAILED EXPERIMENTAL SETTING · p. 15
  • Evaluation uses task-specific tracking, following, distance-range, safety, episode-success, and recovery metrics, plus offline rollout-fidelity and candidate-ranking diagnostics.Recovery metrics include re-acquisition success, time to re-acquire, post-re-acquisition following, and recovery episode success.Source: C DETAILED EXPERIMENTAL SETTING · p. 15
  • Reactive policies provide system-level references, while adapted NWM and the TrackVLA-style predictor provide world-model comparisons; the adapted NWM shares the planner budget and observation interface.The key transition difference is that these predictors condition target-related tokens on action, whereas CST-WM masks that branch.Source: C DETAILED EXPERIMENTAL SETTING · p. 16, Table 10 · p. 17
  • The OA-VAT comparison uses its released PID variant on EVT-Bench because its diffusion-planner checkpoint is unavailable.The evaluation uses the simulator’s initial target box as the reference and scores episodes with EVT-Bench evaluation code.Source: C DETAILED EXPERIMENTAL SETTING · pp. 16–17
  • The real-world protocol runs 10 trials per scenario across occlusion, distractor crossing, and fast motion; success requires following the target to the course end without collision or losing it for more than 6.8 s.Planning uses T=4, N=24, K=8, M=2 and 10 sampling steps on an offboard RTX 6000 Ada, taking 1.4 s per step; the deployment evidence head is further trained on 100k real and 100k EVT-Bench and Habitat 3.0 frames, with the transition frozen.Source: C DETAILED EXPERIMENTAL SETTING · p. 17

D.1 STRUCTURAL DIAGNOSTICS

The structural diagnostics examine action dependence in target representations and how masking depth changes that dependence. They also describe tests of representation stability and viewpoint sensitivity when robot motion is held fixed.

D.1 STRUCTURAL DIAGNOSTICS

  • With the humanoid held still, the diagnostics measure target-evidence stability through latent and robot-state changes, and view sensitivity through observation-latent variation under viewpoint change.CST-WM’s target-evidence action dependence is zero by construction, so its rows check the mask implementation.Source: §D.1 · pp. 17–18
  • The adapted NWM and leaky variants depend on action both in and out of view, and action dependence correlates negatively with episode tracking performance (−0.36 to −0.53).Table 11 breaks the Jacobian analysis down by target visibility and reports its correlation with per-episode performance.Source: Table 11 · p. 18, §D.1 · p. 18
  • Mean action-dependence Jacobian values are 0.19 without a mask, 0.33 with a shallow mask, 0.09 with a partial mask, and 0.04 with a deeper mask.Table 12 compares masks of increasing depth.Source: Table 12 · p. 18, §D.1 · p. 18

D.2 READING IMAGINED TARGET EVIDENCE

The read-out experiments compare imagined target evidence with visibility-changing events offline and test alternative evidence sources in closed-loop planning. They also measure how target representations vary when actions are swapped at a fixed state.

D.2 READING IMAGINED TARGET EVIDENCE

  • Table 13 measures target-representation variation across action swaps at a fixed state using variance and mean pairwise W1 distance.Source: Table 13 · p. 18
  • The offline protocol uses 192 held-out starts and eight-step rollouts; the masked and unmasked models share training conditions, differing in whether the target-evidence branch can attend to the action token.Starts include cases where visibility changes within eight steps and randomly chosen starts; read-outs use shared sampled latents.Source: §D.2 · p. 18
  • On visibility-change steps, visibility AUROC is 0.467 with masking and 0.446 without, versus 0.530 for decoded-view detection and 0.417 for the separate latent head.Copying current evidence is wrong on these steps by definition.Source: §D.2 · p. 18
  • The closed-loop Habitat 3.0 protocol compares evidence read-outs while keeping the full transition fixed, including true-distance, rendered-view, decoded-view, and learned-head references.The true-distance reference substitutes simulator distance for apparent scale; rendered-view candidates are executed from saved simulator states.Source: §D.2 · p. 19

D.3 TARGET-EVIDENCE SIGNAL AND SEEDS

The target-evidence signal tracks relative distance across much of the following range, and its distance-aware training improves cross-humanoid correlations. Runtime details report planning-step costs and timing protocols for model comparisons.

D.3 TARGET-EVIDENCE SIGNAL AND SEEDS

  • Figure 7 shows the detector score tracking relative distance across most of the following range, with saturation very close to the target.The signal is used by the planner as a distance cue.Source: Figure 7 · p. 19, §D.3 · p. 19
  • Across three Habitat 3.0 standard-tracking seeds, full-model results vary by at most 0.02, compared with a 0.12 F gap to the strongest Table 1 baseline.Source: Table 15 · p. 19, §D.3 · p. 19
  • Across humanoids, the distance-aware loss raises signal–distance Spearman correlation from 0.71 to 0.83 and improves all four downstream metrics.Source: Table 14 · p. 19, §D.3 · p. 19

D.4 RUNTIME DETAILS

  • Planning-step timings are measured from observation to action on an RTX A6000, with the robot budget also timed on an RTX 6000 Ada; denoising passes run sequentially over candidate batches.GroundingDINO and encoding take 0.1 s, the evidence head takes under 0.02 s, and denoising accounts for the remaining time.Source: §D.4 · pp. 19–20, Table 16 · p. 20
  • For batch sizes of 64 and above, compilation is not faster; a batch-one denoising pass takes 24.7 ms, and the six-step trajectory takes 1.53 s.Source: §D.4 · p. 20
  • The NWM timing protocol measures one trajectory at batch size one, while the reported 4-bit timing is estimated by dividing six-step time by four rather than running quantization.NWM distills to six steps; CST-WM's six-step timing uses a six-step DDIM sampler.Source: Table 17 · p. 20, §D.4 · p. 20

E LIMITATIONS

CST-WM uses target evidence as a planning proxy rather than recovering the hidden target state, and its evaluation and detector impose scope boundaries.

E LIMITATIONS

  • Target evidence summarizes observability and apparent scale rather than the hidden target trajectory; re-acquisition means the target returns in imagined views within the planning horizon.If no candidate recovers it within that horizon, the planner acts on validity and safety terms until new evidence arrives.Source: E LIMITATIONS · p. 21
  • Evidence inherits GroundingDINO errors, and the model has no persistent target identity, so similar-looking people or distractors can redirect planning.The stated detector weaknesses include partial-visibility confidence, small or distant targets, and near-range area saturation.Source: E LIMITATIONS · p. 21
  • EVT-Bench evaluation covers single-target tracking; its distracted and ambiguous tasks, which require maintaining identity among similar people, are outside scope.Source: E LIMITATIONS · p. 21
Loading 2609.06302v2…