PaperlayerView exact revision on arXiv

Source-linked AI summary

LPA-CWM: A Learned Physical Adjudicator for Motion Reasoning with Counterfactual World Models

Kunwei Wu, Xiang Liu, Guocai Yao, Junming Chen, Zhikang Chen, Min Zhang, Pengwei Wang, Sen Cui

arXiv:2609.14073v2cs.CVcs.AIcs.RO

TL;DR

Uniform CWM aggregation weights responses equally, although candidate responses can be inconsistent, and existing video benchmarks do not directly measure recovered point correspondence. LPA-CWM learns relative reliability weights for frozen-CWM candidates, and CMC evaluates dynamic tracks across multiple dimensions. Across DAVIS, Kinetics, and RoboTAP, LPA-CWM improves all main CMC measures over Uniform CWM, with relative DCAavg gains of 18.1%–60.0%.

  • Problem

    Uniform CWM aggregation treats responses as equally reliable despite inconsistent candidates, while existing video benchmarks do not directly measure point correspondence recovered through CWM interventions.

    Source: §1 · p. 2

  • Method

    LPA-CWM learns relative reliability weights for competing frozen-CWM candidates from dense MOVi-F trajectories, while CMC evaluates localization, completeness, visibility, and continuity on dynamic tracks.

    Source: §1 · p. 2

  • Results

    LPA-CWM improves all main CMC measures over Uniform CWM across DAVIS, Kinetics, and RoboTAP, with relative DCAavg gains of 18.1%–60.0%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CMC complements TAP-Vid First by explicitly measuring completeness on moving tracks, and LPA-CWM improves tracking accuracy and motion completeness within the CWM family.

    Source: §4.3 · p. 8

    LPA can only reweight motion evidence already present in the frozen candidate set; missing or systematically incorrect motion requires improved candidate generation or adaptation of the underlying CWM.

    Source: §5 · p. 9

Abstract

from arXiv · show

Counterfactual world models (CWM) extract motion from pretrained video predictors by comparing factual and intervened predictions, but uniform aggregation weights responses equally without explicitly incorporating physical priors. Our key insight is to incorporate physical priors into candidate reliability learning, motivating LPA-CWM with a lightweight Learned Physical Adjudicator (LPA). Trained on dense MOVi-F trajectories, the 3.0M-parameter LPA compares visual context and response structure across an unordered candidate set to predict relative weights; windowed localization and one paired re-evaluation recover motion with the CWM frozen. Existing video-level benchmarks do not directly assess motion correspondence, where low localization error can conceal missing trajectory segments. We introduce Completeness-aware Motion Correspondence (CMC), a ground-truth-anchored protocol jointly measuring localization, completeness, visibility, and continuity, counting missing predictions as failures on visible dynamic points. Across DAVIS, Kinetics, and RoboTAP, LPA-CWM improves all main CMC measures over Uniform CWM, with relative gains of 18.1%--60.0% in average Dynamic Correspondence Accuracy ($\mathrm{DCA}_{\mathrm{avg}}$), and improves TAP-Vid First tracking accuracy (overview: https://LPA-CWM.github.io).

1 Introduction

LPA-CWM learns to weight competing frozen-CWM motion candidates using physical motion priors, while CMC evaluates recovered dynamic tracks beyond localization alone. Across three datasets, LPA-CWM improves the main CMC measures over Uniform CWM.

1 Introduction

  • APA-CWM uses NSG-derived spatiotemporal consistency around candidate response peaks to produce closed-form reliability weights as a training-free physical reference.LPA-CWM instead learns relative weights by comparing visual evidence, response morphology, shared context, and candidate statistics.Source: §1 · p. 2
  • The approach treats multi-mask aggregation as candidate reliability estimation because mask responses can disagree and uniform weighting can dilute useful motion evidence.The paper identifies insufficient use of physical priors as part of this limitation.Source: §1 · p. 2
  • The 3.0M-parameter LPA learns relative candidate reliability from dense MOVi-F trajectories, then weights frozen-CWM responses before windowed localization and paired re-evaluation.The CWM remains unchanged; training uses candidate endpoint errors to define relative reliability targets.Source: §1 · p. 2
  • Existing video-level benchmarks do not directly assess point correspondence recovered through CWM interventions, and low error on returned predictions can hide missing trajectory segments.Smooth trajectories may still follow incorrect motion.Source: §1 · p. 2
  • CMC jointly evaluates localization, completeness, visibility, and continuity, counting missing predictions as failures on ground-truth-visible dynamic points.Track-level success also requires sufficient coverage and bounded localization error.Source: §1 · p. 2
  • LPA-CWM improves all main CMC measures over Uniform CWM across DAVIS, Kinetics, and RoboTAP, with relative DCAavg gains of 18.1%–60.0%.Dedicated trackers remain stronger on some localization or visibility measures, while LPA-CWM leads in completeness and continuity and achieves the best main CMC results on Kinetics.Source: §1 · p. 2

2 Related Work

Related work covers counterfactual motion extraction, diffusion-based correspondence signals, external flow and tracking methods, and benchmarks for motion consistency. LPA-CWM focuses on estimating the relative reliability of responses already produced by a frozen CWM.

2 Related Work

3 Method

The method keeps the counterfactual world model frozen and generates multiple motion candidates for each query. It compares analytic and learned candidate weighting, then localizes and refines motion from the weighted responses.

3.1 Counterfactual Motion Candidates from a Frozen World Model

  • A frozen CWM produces candidate response maps and endpoints for a query, whose disagreement motivates candidate-specific rather than uniform weights.The method combines the candidate maps after weighting.Source: §3.1 · pp. 3–4, §3 · p. 3

3.2 Analytic Candidate Weighting

  • APA-CWM averages an NSG-derived score around each candidate peak, then clips and normalizes scores into training-free reliability weights.The NSG cue is a heuristic, not a calibrated probability of correct correspondence.Source: §3.2 · pp. 4–5

3.3 Learned Physical Adjudicator

  • LPA encodes local and full responses, frame context, and candidate statistics, then uses a set encoder to compare candidates and predict weights.Dense MOVi-F trajectory endpoints provide soft relative-reliability targets; training matches those targets and encourages accurate weighted endpoints and high scores for the lowest-error candidates.Source: §3.3 · p. 5

3.4 Weighted Aggregation and Localized Decoding

  • The weighted response map is localized with two Gaussian-window updates that suppress distant peaks, then one fresh paired CWM evaluation yields the refined endpoint.The initial APA or LPA weights are not reused for the paired re-evaluation.Source: §3.4 · p. 6, Figure 3 · p. 6

4 Experiments

Experiments test learned candidate weighting against analytic and external baselines under CMC and TAP-Vid First, and examine which components contribute. The CWM variants share the predictor, intervention generator, evaluation queries, and mask-sampling protocol.

4.1 Experimental Setup

  • Uniform, APA, and LPA use the same frozen predictor, intervention generator, queries, and mask sampling; only LPA is trained on MOVi-F trajectories.The setup fixes M = 10 and R = 1 and evaluates 90 videos and approximately 1,300 trajectories across three datasets.Source: §4.1 · pp. 6–7

4.3 TAP-Vid First and Qualitative Results

  • LPA-CWM has the lowest displayed mean EPE in all four qualitative comparisons and remains stable under large motion and appearance changes.Figure 4 compares five methods on identical frames and visible queries across manipulation, human-action, and animal-motion scenes.Source: Figure 4 · p. 7, §4.3 · p. 8

4.2 Completeness-aware Motion Correspondence

  • CMC measures localization, completeness, visibility, and continuity on dynamic tracks, counting missing predictions on visible points as failures.Figure 5 summarizes DAVIS CMC results; higher values are better.Source: §4.2 · p. 7, Figure 5 · p. 7
  • Relative to Uniform CWM, DCAavg improves by 22.11, 11.66, and 7.69 percentage points on DAVIS, Kinetics, and RoboTAP, respectively.Relative to APA-CWM, the corresponding gains are 3.19, 5.70, and 3.35 points.Source: §4.2 · p. 7
  • Across three datasets, LPA-CWM leads all seven CMC measures and all main TAP-Vid First measures within the CWM family.Dedicated trackers remain stronger on some conventional tracking measures, particularly on DAVIS.Source: §4.2 · p. 7, Table 1 · p. 8, §4.3 · p. 8

4.4 Ablation Studies

  • The best accuracy–cost setting is M = 10, R = 1; M = 20 raises latency without improving accuracy, and R = 2 does not further reduce EPE.The sweep uses 40 fixed MOVi-F query groups.Source: §4.4 · p. 8
  • Removing candidate-set interaction or visual cues reduces performance; coordinate or best-candidate supervision improves over KL matching alone, and the full objective performs best.Windowed localization improves every weighting rule.Source: §4.4 · p. 8, Table 2 · p. 9
  • LPA reduces candidate-space EPE toward the best-of-10 oracle, favors more accurate candidates, and more than doubles APA’s top-1 hit rate.Figure 6 reports candidate-space weighted-coordinate EPE and weight versus ground-truth candidate rank on MOVi-F validation videos.Source: §4.4 · p. 8, Figure 6 · p. 9

5 Conclusion and Limitations

The paper presents learned reliability weighting and CMC for motion reasoning with a frozen CWM, reporting improvements over Uniform CWM and APA-CWM across datasets and evaluation protocols. It also identifies limits in candidate coverage, training and evaluation scope, and inference cost.

  • LPA-CWM learns relative reliability across competing CWM candidates, while APA-CWM provides a training-free analytic reference; CMC evaluates localization, completeness, visibility, and continuity.Source: §5 · p. 9
  • Across DAVIS, Kinetics, and RoboTAP, LPA-CWM consistently improves over Uniform CWM and APA-CWM under both CMC and TAP-Vid First.Ablations confirm contributions from candidate interaction, trajectory supervision, and windowed localization.Source: §5 · p. 9
  • LPA can only reweight evidence already in the frozen candidate set; synthetic MOVi-F training, evaluation within one CWM family, and added inference computation constrain the current scope.The authors point to broader data and model evaluation and more efficient inference as areas for extension.Source: §5 · p. 9

AI Use Statement

The authors describe AI assistance across research and manuscript work and state that they reviewed the work and verified the reported results.

  • AI tools assisted with drafting, literature and baseline discovery, ideation, planning, code, result interpretation, and documentation; the authors reviewed procedures and verified reported results.Source: AI Use Statement · p. 10

Reproducibility Statement

The reproducibility materials describe training, evaluation, and implementation controls. They specify held-out checkpoint selection, shared evaluation conditions, and additional protocol and ablation details in the appendices.

  • LPA is trained exclusively on MOVi-F with disjoint training and validation videos; checkpoint selection uses held-out MOVi-F videos.Source: Reproducibility Statement · p. 10
  • The CWM predictor and intervention generator remain frozen, and evaluation on DAVIS, Kinetics, and RoboTAP uses no dataset-specific adaptation.Source: Reproducibility Statement · p. 10
  • Methods use the same evaluation subsets and queries, while CWM variants share a frozen candidate generator and mask-sampling protocol; appendices document datasets, metrics, adapters, and ablation procedures.Source: Reproducibility Statement · p. 10

Ethics Statement

The paper notes that real-world videos may depict people and calls for privacy-respecting use. It also identifies limits on trajectory reliability and specifies safeguards for safety-critical use and artifact sharing.

Ethics Statement

  • Real-world videos may depict people, so downstream applications should respect privacy and avoid non-consensual monitoring.Source: Ethics Statement · p. 10
  • LPA reweights existing motion candidates but cannot guarantee correct trajectories when those candidates are unreliable.Source: Ethics Statement · p. 10
  • High completeness or continuity alone does not establish accurate correspondence or physical validity.Source: Ethics Statement · p. 10
  • Safety-critical robotic applications require independent validation and appropriate safeguards.Source: Ethics Statement · p. 10
  • Use and redistribution of datasets, pretrained models, and derived artifacts should follow their respective licenses and usage conditions.Source: Ethics Statement · p. 10

A Implementation Details

The implementation keeps the CWM predictor and intervention generator fixed, while LPA learns candidate reliability from visual, response, and scalar features. Inference combines weighting and windowed localization with a configurable paired re-evaluation stage.

A Implementation Details

  • The predictor and intervention generator stay fixed during LPA training and inference; CWM uses a 256 × 256 grid with 32 × 32 masks, hiding about 89.9% of target patches.Source: A Implementation Details · p. 15
  • LPA has 3,000,257 trainable parameters and encodes local candidate patches, global responses, frame-pair context, and candidate statistics before set-level scoring.The candidate statistics describe displacement, response concentration, ambiguity, photometric change, and local edge structure.Source: A Implementation Details · p. 15, Table A.1 · p. 15
  • The 16 candidate statistics are standardized using MOVi-F training-split statistics, with signed-log compression applied to dimensions 3–10.Source: A Implementation Details · p. 16
  • LPA is trained on complete sets of 10 candidates with randomly permuted order, and checkpoint selection uses weighted-coordinate EPE on held-out MOVi-F videos.Source: A Implementation Details · p. 16
  • APA-CWM derives normalized reliability weights from an NSG-based score map, whereas LPA-CWM does not use the diffusion score map.APA averages aligned scores within a radius-six neighborhood around candidate peaks.Source: A Implementation Details · pp. 16–17, Figure A.1 · p. 17
  • With the default R = 1, the initial endpoint centers a full-field paired re-evaluation; the fresh responses are uniformly aggregated without rerunning APA or LPA.Source: A Implementation Details · pp. 16–17
  • For R > 1, later re-evaluation passes may use progressively smaller crops, while the reported trajectories are assembled from independent query-to-target predictions.Source: A Implementation Details · p. 17
  • Windowed localization applies two Gaussian reweighting steps to the response field, then passes its discrete argmax to the re-evaluation stage.Source: A Implementation Details · p. 17

B Datasets and Evaluation Protocols

Evaluation spans real-world video and robot-manipulation benchmarks, using fixed query tracks and no evaluation-dataset adaptation. The protocol follows TAP-Vid First and reports both benchmark tracking metrics and complementary CMC measures.

B Datasets and Evaluation Protocols

  • The evaluation covers DAVIS, Kinetics, and RoboTAP, spanning real-world video and embodied manipulation settings.DAVIS emphasizes deformation, occlusion, distractors, and fast motion; Kinetics includes diverse actions and camera motion.Source: B Datasets and Evaluation Protocols · p. 17
  • Across 90 videos and nearly 1,300 annotated trajectories, all methods use the same fixed evaluation subsets and query tracks, with no evaluation-dataset adaptation.Source: B Datasets and Evaluation Protocols · p. 18
  • Following TAP-Vid First, each query begins at its first visible frame, and missing predictions on annotated visible points remain evaluation failures.CWM variants share locked videos, frames, queries, target frames, a frozen candidate generator, and mask-sampling protocol.Source: B Datasets and Evaluation Protocols · p. 18
  • TAP-Vid metrics include Average Jaccard, thresholded point accuracy, endpoint distance, and occlusion accuracy; CMC separately examines localization, completeness, visibility, and continuity.Source: B Datasets and Evaluation Protocols · p. 18

C Completeness-aware Motion Correspondence

CMC evaluates motion correspondence on ground-truth dynamic tracks, explicitly accounting for missing visible-point predictions. The section defines its evaluation dimensions and threshold checks, then describes how external estimators are adapted for comparison.

C Completeness-aware Motion Correspondence

  • CMC selects dynamic tracks from ground truth and keeps missing predictions on visible points in the accuracy denominator.Metrics are computed per video and averaged with equal video weight.Source: C Completeness-aware Motion… · p. 18
  • Completeness is measured with coverage and Complete@0.8, while CTS combines a coverage requirement with bounded mean localization error and can still allow 20% of a track to be absent.Source: C Completeness-aware Motion… · pp. 18–19
  • Visibility uses balanced accuracy and occlusion F1; continuity uses longest continuous coverage and reports fragmentation within ground-truth-visible segments.Source: C Completeness-aware Motion… · p. 19
  • 59.42 to 58.30 DCAavg across motion thresholds and 58.79 to 56.29 CTS@(4,c) across coverage thresholds; LPA-CWM remains best throughout.The figure marks the default threshold settings; method ordering remains stable around Lmotion = 4 pixels and c = 0.8.Source: Figure C.1 · p. 20, C Completeness-aware Motion… · p. 20

D Adapting External Estimators

  • External estimators share query identities and target frames but retain method-specific temporal inference; missing predictions enter DCA as failures.Per-video metrics are aggregated with equal video weight, without selecting points or videos by predicted error, coverage, or visibility.Source: D Adapting External Estimators · pp. 20–21
  • Visibility adaptation differs by estimator: CoTracker3 uses its native visibility thresholded at 0.9, while out-of-frame or non-finite flow predictions are treated as invisible.Source: D Adapting External Estimators · pp. 20–21

E Ablation Protocols

The ablation protocols select the deployed mask-count and re-evaluation setting, define how inference cost and candidate weighting are assessed, and separate weighting from localization choices. Additional qualitative results compare five methods on the same frame pairs and query identities.

E Ablation Protocols

  • M = 10, R = 1 is selected for the main comparison from a joint sweep over mask counts and paired re-evaluation depths.The sweep uses 40 query groups from 10 held-out MOVi-F videos and nested prefixes of the same ordered mask set.Source: E Ablation Protocols · p. 21, Figure E.1 · p. 21
  • Candidate-weight analysis ranks each ten-candidate group by ground-truth endpoint EPE and measures top-1 oracle hits and effective support.Effective support is 1 for one-hot weights and 10 for uniform weights; the analysis uses the MOVi-F validation split for checkpoint selection.Source: E Ablation Protocols · p. 22
  • 2.1363 pixels is the final endpoint EPE for the deployed setting, while the 2.41-pixel candidate-space surrogate is not directly comparable.The analyses use different validation subsets and endpoint definitions.Source: E Ablation Protocols · p. 22
  • The inference ablation crosses three initial weighting schemes with standard or windowed localization, then applies the same one-pass paired re-evaluation.The six configurations separate initial candidate weighting from localization; re-evaluation uniformly aggregates fresh responses without rerunning the weighting method.Source: E Ablation Protocols · p. 22
  • Architecture variants remove candidate-set interaction or raw visual encoders, while objective variants isolate the contributions of coordinate and best-candidate supervision.Objective rows are reported under both CMC and TAP-Vid First on the same DAVIS evaluation set, queries, and initial mask protocol.Source: E Ablation Protocols · p. 22

F Additional Qualitative Results

  • Figure F.1 compares five methods on fourteen additional cases, using identical frame pairs and query identities, with continuous lines and mean EPE badges.The displayed columns are SEA-RAFT, CoTracker3, Uniform CWM, APA-CWM, and LPA-CWM.Source: Figure F.1 · p. 23
Loading 2609.14073v2…