PaperlayerView exact revision on arXiv

Source-linked AI summary

GAUGE: Group-Wise View-Inconsistency Rectification for Feed-Forward 4D Tracking

Zhuoqian Feng, Weixing Chen, Ziliang Chen, Yang Liu, Liang Lin

arXiv:2609.32596v1cs.CVcs.AI

TL;DR

Feed-forward 4D trackers retain structured residual error after global alignment, including group-wise scale bias concentrated along the view direction. GAUGE is a training-free post-hoc correction that groups points and estimates per-frame radial scales and group translations from sparse metric anchors. On dynamic query points, it reduces endpoint error across eight trackers.

  • Problem

    Feed-forward trackers produce substantial residual error after global alignment, and its structure is not captured by a single global scale.

    Source: §1 · pp. 1–2

  • Method

    GAUGE groups points using motion direction and spatial connectivity, then estimates per-frame radial scales and group translations from sparse metric anchors without training.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    15.1% to 62.6%: GAUGE lowers endpoint error on dynamic query points across eight trackers compared with uncorrected predictions.

    Source: §5.2 · p. 7

  • Takeaways & Limitations

    The measured residual structure can be corrected post hoc with four degrees of freedom per group and frame using 1% to 5% metric anchors.

    Source: §1 · p. 2

    GAUGE requires true-scale anchors, and its gains disappear beyond about 20 cm of anchor noise; it assumes rigid groups and solves per frame, leaving non-rigid deformation, sensor noise, and long-sequence drift out of scope.

    Source: §6 · p. 9

Abstract

from arXiv · show

Feed-forward models regress dense 3D point trajectories directly from monocular video, yet the residual after global alignment is substantial and lacks a structural explanation. Measured on dynamic query points across models and datasets, the error concentrates along the view direction, while the scale correction each motion group requires differs. The predicted displacement direction nevertheless supports reliable grouping, with a median angle far below the 90{\deg} random baseline. The systematic part of the residual is therefore a family of radial degrees of freedom per motion group, along directions 2D observations cannot constrain. We call it group-wise view inconsistency. We present GAUGE (Group-wise Adaptive Unsupervised Gauge Estimation), a training-free and model-agnostic post-hoc module. It recovers motion groups from direction consistency and spatial connectivity, then estimates a per-frame radial scale and group-level translation from 1% to 5% metric anchors, four degrees of freedom per group and frame. On dynamic query points of eight trackers, including D4RT, 4RC and SM4RT, our correction lowers endpoint error by 15.1% to 62.6% over the uncorrected predictions, while spending the same anchors on gradient fine-tuning improves the same models by only -1.1% to 15.2%. Code is publicly available at https://github.com/HCPLab-SYSU/GAUGE.

1 INTRODUCTION

Feed-forward 4D trackers are fast but retain structured residual error after global alignment, motivating a post-hoc correction that uses sparse metric anchors. GAUGE groups points by motion direction and estimates per-group radial scales and translations, reducing dynamic-point endpoint error across trackers.

1 INTRODUCTION

  • 15.1% to 62.6%: GAUGE lowers endpoint error on dynamic query points of eight trackers, using four degrees of freedom per group and frame.The correction uses 1% to 5% metric anchors; the same anchor budget spent on gradient fine-tuning yields gains of -1.1% to 15.2% on the same models.Source: §1 · p. 2
  • The residual is structured: it concentrates along the view direction, and radial scale corrections are shared within motion groups rather than explained by one global factor.On dynamic query points, 61.6% to 77.7% of residual energy falls along the view direction; a global factor can worsen results.Source: §1 · pp. 1–2
  • GAUGE recovers motion groups from tracker outputs and applies a per-frame radial scale plus a group-level translation using a small set of metric anchors.The grouping uses flow-direction consistency and spatial proximity, without extra supervision.Source: §1 · p. 2

2 RELATED WORK

Related work spans feed-forward trajectory prediction, unsupervised motion grouping, and sparse metric anchoring. GAUGE operates post hoc on predicted trajectories and tailors its correction to their group-wise error structure.

2.1 FEED-FORWARD 3D MOTION PREDICTION

  • GAUGE acts directly on feed-forward 3D trajectories, unlike methods that add test-time optimization or update model weights.D4RT, 4RC and V-DPM regress dense trajectories; SM4RT also represents dynamics with explicit SE(3) motion bases.Source: §2.1 · p. 2

2.2 UNSUPERVISED MOTION GROUPING

  • Unlike grouping methods framed as standalone segmentation, GAUGE groups points by flow-direction agreement so four degrees of freedom per group can describe their correction.Related approaches include weakly or self-supervised scene-flow grouping and rigid-body clustering on 3D representations.Source: §2.2 · p. 3

2.3 SPARSE METRIC ANCHORING

  • GAUGE extends sparse metric anchoring from single-frame depth to temporal trajectories, modeling per-frame drift with radial degrees of freedom for each motion group.The related work contrasts this with fitting a global low-dimensional basis map to sparse anchors.Source: §2.3 · p. 3

3 ERROR STRUCTURE OF FEED-FORWARD 4D TRACKERS

The measured residuals show reliable predicted directions, group-aligned scale bias, and concentration along the view direction. These properties explain why global-scale and unconstrained higher-degree corrections are inadequate.

3.2 DIRECTION ACCURACY, GROUP STRUCTURE AND RADIAL CONCENTRATION

  • Predicted flow directions outperform the roughly 90° random baseline, while 61.6% to 77.7% of dynamic-point residual energy lies along the view direction.Across eight models, the fraction of per-point optimal radial scale explained by motion groups is 15.1% to 76.7%, exceeding a size-preserving permutation null in most settings.Source: §3.2 · pp. 3–4

3.3 FAILURE OF GLOBAL SCALE CORRECTION

  • A single global scale leaves substantial error, while a seven-degree-of-freedom Sim(3) fit under the same grouping and anchors is worse than no correction on some models.The extra rotation and anisotropic scale act along directions already constrained by 2D correspondences.Source: §3.3 · p. 4
  • Each frame and motion group has one radial scale unobservable from 2D projection, whereas the other three translational degrees of freedom are only weakly constrained.A rigid group contributes one radial scale regardless of its point count, so correction can only be partial.Source: §3.3 · p. 4

4 GAUGE

GAUGE is a post-hoc geometric head that groups query points, allocates sparse anchors, and estimates a low-degree-of-freedom correction for each group. It leaves tracker weights unchanged and handles special or degenerate groups separately.

4 GAUGE

  • GAUGE addresses structured residuals with grouping, sparse anchors for metric scale, and per-group radial correction.It modifies no tracker weights and maps globally aligned predicted trajectories to corrected trajectories.Source: §4 · p. 4, §4.1 · p. 4

4.1 MINIMAL TRACKER INTERFACE

  • The tracker interface uses predicted trajectories and visibility, camera centers, and ground-truth positions for a small number of query-point anchors.The geometric head is post hoc and requires no tracker retraining.Source: §4.1 · p. 4, Figure 3 · p. 5

4.2 UNSUPERVISED MOTION GROUPING

  • Grouping targets approximately rigid motion: direction-consistent points are linked within spatial neighborhoods, then components are spatially split or merged to address distant objects and fragments.Static points join the world-fixed class; independent-dynamic points do not take part in correction.Source: §4.2 · p. 5

4.3 SPARSE ANCHOR ALLOCATION

  • The anchor budget is 5% of query points, allocated by group size and mean displacement; this allocation changes endpoint error by 24.3 and 30.2 percentage points versus ungrouped uniform allocation.Per-group allocations are rounded and capped by group size.Source: §4.3 · p. 5

4.4 GROUP-WISE GEOMETRIC CORRECTION

  • Ordinary co-moving groups are rescaled radially about the camera center and receive one group-level translation.Per-frame scales use visible anchors, with under-supported frames interpolated and filled using the group-level scale; translation absorbs the residual rigid bias.Source: §4.4 · pp. 5–6

4.5 SPECIAL GROUPS AND DEGENERATE CASES

  • Static and low-parallax special groups use an in-group anchor as the scaling center because camera-centered radial scaling can be ill-posed for them.On failure, correction falls back to Umeyama Sim(3) or pure translation, and groups that fail both remain uncorrected.Source: §4.5 · p. 6

5 EXPERIMENTS

Across two benchmarks, GAUGE reduces dynamic-query endpoint error, and the experiments compare it with test-time fine-tuning and ablate grouping, correction degrees of freedom, and anchor choices. They also examine when gains weaken and what residual error remains outside the correction family.

5.1 EXPERIMENTAL SETUP

  • The evaluation spans PointOdyssey and WorldTrack, using DQS for dynamic query points and Full for all first-frame-visible queries, including static background.The eight evaluated trackers cover feed-forward, depth-assisted, and camera-motion-aware paradigms; the default anchor budget is 5%.Source: §5.1 · p. 6

5.2 MAIN RESULTS

  • Under DQS, endpoint error drops on all five test splits for all eight models: 15.1% to 62.6% on PointOdyssey and 4.9% to 76.3% on WorldTrack subsets.Table 2 reports PointOdyssey results, while Table 3 reports unweighted WorldTrack averages over four subsets.Source: §5.2 · p. 7, Table 2 · p. 7, Table 3 · p. 7

5.3 COMPARISON WITH TEST-TIME OPTIMIZATION

  • Across five supported base models, GAUGE gains 25.2% to 57.6%, versus -1.1% to 15.2% for fine-tuning with the same information.SM4RT's position error is unchanged under fine-tuning; its Full-protocol gains are larger than under DQS, consistent with improvement concentrated on static background.Source: §5.3 · p. 7, Figure 6 · p. 8

5.4 ABLATION STUDIES

  • Combining direction clustering with spatial merging gives the best grouping result, while differences among grouping strategies are smaller than the benefit of grouping.Under DQS, unsupervised grouping beats oracle instance grouping on seven of eight models, with a median relative change of +7.3%; under Full, oracle grouping is better on five models, with a median relative change of -14.2%.Source: §5.4 · p. 8, §5.5 · p. 8
  • Per-frame scale is the decisive added degree of freedom because a group-level constant cannot capture time-varying drift; the full correction also outperforms scaling about an in-group anchor or one shared form.Error decreases monotonically as degrees of freedom increase under the camera-center parameterization.Source: §5.4 · p. 8
  • Endpoint error decreases with larger anchor budgets, but high-budget results saturate; the 5% choice is based on per-anchor efficiency rather than accuracy.Average accumulated per-anchor efficiency falls from 22.8 at 1% to 6.0 at 5% and 0.7 at 50%.Source: §5.4 · p. 8

5.6 ROBUSTNESS TO IMPERFECT AND NON-GROUND-TRUTH ANCHORS

  • At 10 cm anchor noise, corrected error remains better than baseline, but the gain disappears at 20 cm; anchor-error structure also matters beyond its magnitude.Visual SLAM keyframes yield 20.6% gain with 8.57 cm error, while single-frame depth yields 10.1% gain with 8.50 cm error because its errors contaminate the radial scale.Source: §5.6 · p. 9, Table 5 · p. 9

5.7 ERROR BUDGET AND FAILURE PREDICTION

  • The characterized error family explains at most 34.1% of total error, and GAUGE recovers 28.5 percentage points, or 75.6% of that upper bound.Only between-group scale variance predicts gain (r = 0.61, n = 58, permutation test p < 0.0001); non-rigid deformation and sensor noise remain outside the four degrees of freedom.Source: §5.7 · p. 9

6 CONCLUSION AND FUTURE WORK

The conclusion presents GAUGE as a training-free correction for structured, view-direction residual error in feed-forward 4D tracking. It reports lower dynamic-query endpoint error and identifies anchor noise, non-rigid deformation, sensor noise, and long-sequence drift as boundaries.

6 CONCLUSION AND FUTURE WORK

  • GAUGE corrects structured residual bias with four degrees of freedom per group and frame, reducing dynamic-query endpoint error by 15.1% to 62.6%.The correction is training-free and model-agnostic; its gains are reported across eight trackers and benchmarks.Source: §6 · p. 9
  • The method requires true-scale anchors, and its gain disappears beyond about 20 cm of anchor noise.The authors identify non-rigid deformation, sensor noise, and long-sequence drift as out of scope because groups are assumed rigid and estimates are solved per frame.Source: §6 · p. 9
  • The complete correction-head implementation and pipeline pseudocode are provided in the supplementary material, and the code is publicly available.The implementation depends on numpy and scipy and runs on a single CPU core.Source: REPRODUCIBILITY STATEMENT · p. 10

A METHOD DETAILS AND FORMAL STATEMENTS

GAUGE groups trajectories, allocates metric anchors, and applies group-specific corrections. Its formal analysis distinguishes a strictly unobservable radial scale from translations that are observable but weakly constrained, while the appendix defines evaluation metrics and implementation settings.

A.1 COMPLETE PIPELINE

  • GAUGE groups points using agreement between motion directions and spatial connectivity, then allocates anchors and applies per-group corrections; ungrouped points and skipped groups retain the baseline.The complete pipeline takes aligned trajectories, visibility, camera centers, and anchor ground-truth trajectories as input and returns corrected trajectories.Source: §A.1 · pp. 14–15
  • For co-moving groups, correction estimates per-frame radial scales and then a group translation; static or low-parallax groups scale about an in-group anchor, falling back to translation when scale cannot be estimated.Missing radial scales are interpolated, smoothed with a Gaussian kernel at σ = 2, and can fall back to αglobal; the general pipeline also has Sim(3) and translation fallbacks for degenerate fits.Source: §A.1 · pp. 15–16

A.2 FORMAL STATEMENTS

  • Radial distance from a point to the camera center is unobservable in a single frame, whereas the observation constrains its angular relation to the view direction.The per-group family has G(T + 3) parameters: its zero-translation subfamily preserves projections, while translation changes projections but is weakly constrained for distant, low-parallax groups.Source: §A.2 · pp. 16–17
  • The analysis does not claim tangential unbiasedness or full gauge freedom; metric training supervision partly pins scale, so it predicts partial rather than complete correction.The paper reports a quantitative measurement of this upper bound in Section 5.7.Source: §A.2 · p. 17

A.3 DEFINITIONS OF THE THREE STRUCTURAL METRICS

  • Between-group scale variance measures how much group labels explain variation in points’ optimal radial scales, while radial energy and flow direction error characterize error direction and direction-signal quality.Measurements use dynamic query points with ground-truth displacement of at least τm; static background points carry no scale information.Source: §A.3 · p. 17
  • 15.1% to 76.7% is the measured between-group scale variance across eight models and two protocols; threshold changes preserve the qualitative metric conclusions but not model rankings.Changing τdir and minimum group size also makes endpoint error fluctuate by -10.1% to +14.7% relative to the default configuration.Source: §A.3 · p. 17

A.4 COMPUTATION OF THE EVALUATION METRICS

  • EPE is median error magnitude in meters over valid ground-truth points and frames; APD measures threshold-based positional accuracy, while AJ jointly scores positional accuracy and visibility.APD uses depth-adaptive thresholds, and predictions are evaluated at native metric scale without the TAPVid-3D reference implementation’s global scale alignment.Source: §A.4 · p. 18

A.5 HYPERPARAMETERS

  • The default anchor budget is ρ = 0.05, and the correction head uses trajectories, visibility, camera centers, and anchor ground truth without depth maps, optical flow, or base-model features.Confidence is used only for DQS query screening, while ground-truth positions are used only for anchor sampling and fitting.Source: §A.5 · p. 18

B.1 MAIN RESULTS AND STRUCTURAL METRICS

The appendix provides complete numerical results for the main comparisons and structural metrics, but the supplied passages identify the tables’ scope rather than their cell values.

B.1 MAIN RESULTS AND STRUCTURAL METRICS

  • Table A2 compares base predictions, test-time optimization, and GAUGE across three metrics and five models; Figure 5 plots its relative endpoint-error improvement column.The passage identifies these comparison settings but provides no cell values.Source: §B.1 · p. 18, Table A2 · p. 20
  • Table A1 reports per-subset WorldTrack results for eight models under DQS and Full protocols, with Table 3 giving DQS subset averages.ADTs and PStudio lack depth ground truth, so depth-dependent models are marked with dashes.Source: Table A1 · p. 19
  • Table A3 reports three structural metrics for all eight trackers, using 50% as the isotropic radial-energy baseline and about 90° as the random flow-direction baseline.The supplied passage gives the baselines, not the table’s per-model values.Source: §B.1 · p. 18, Table A3 · p. 20

B.2 ABLATIONS AND ANCHOR SENSITIVITY

The appendix catalogs anchor-sensitivity and grouping comparisons, while the design discussion reports allocation-control differences and explains choices that avoid unsupported or higher-capacity corrections.

B.2 ABLATIONS AND ANCHOR SENSITIVITY

  • Anchor-sensitivity tables cover correction under noisy anchors, seven anchor-budget levels, unsupervised versus oracle grouping, resampling dispersion, and per-sequence improvement distributions.The supplied passages identify these table scopes but do not provide their cell values.Source: §B.2 · p. 20, Table A4 · p. 20, Table A5 · p. 21

B.3 EFFICIENCY

  • Table A9 reports post-processing time per sequence under DQS and Full protocols, measured on a single CPU core without GPU, autograd, or an optimizer.The supplied passage describes the timing setup but gives no timing values.Source: §B.3 · p. 22, Table A9 · p. 22

C.1 DESIGN CHOICES

  • The design discussion rejects rotation and tangential scale as unsupported, assigns time-varying drift to per-frame scale and time-independent residual bias to translation, and uses medians to resist extreme depth ratios.The median has slightly lower statistical efficiency, and the authors report that this choice lets the 10 cm anchor-noise setting beat baseline.Source: §C.1 · p. 22
  • 24.3 and 30.2 percentage points separate motion-weighted allocation from uniform and ungrouped allocation; co-frame versus cross-frame sampling differs by 1.6 and 0.2 percentage points.The authors conclude that using a common frame is not a critical design choice.Source: §C.1 · p. 22
  • Higher-capacity variants—including depth-dependent or anisotropic scale, metric-targeted correction, and iterative correction—either worsened error or fit validation data but degraded on test data.Source: §C.1 · p. 22

C.2 ABLATION CONSISTENCY ON A SECOND MODEL (4RC)

On 4RC, grouping remains better than no grouping, but the contribution of group-level scale is model-dependent. The appendix also bounds the role of input depth, checks motion-screening bias, and describes the scale-variance null test.

C.2 ABLATION CONSISTENCY ON A SECOND MODEL (4RC)

  • 0.1370 endpoint error without grouping is better than the 0.1421 baseline on 4RC, while group-level scale plus translation gives 0.1377 versus 0.1354 for translation alone.The authors report that grouping remains clearly better than no grouping, and that the 4RC gain comes mainly from per-frame scale.Source: §C.2 · p. 23, Table A10 · p. 23

C.3 DEPTH INPUT VERSUS POST-HOC CORRECTION

  • TAPIP3D endpoint error is 0.1548 with ground-truth depth and 1.6705 with predicted depth; GAUGE helps with both inputs but does not change the depth-imposed accuracy ceiling.The authors state that four degrees of freedom cannot characterize error introduced by depth noise, so GAUGE cannot replace a reliable depth source.Source: §C.3 · p. 23, Table A11 · p. 23

C.4 GROUND-TRUTH MOTION CONTROL

  • The median difference between predicted-motion and ground-truth-motion screening gains is -1.91 percentage points across eight models, so predicted-motion screening does not systematically inflate the gain.The largest difference is on SM4RT at 58.82 percentage points, where selected point sets overlap by only 0.16.Source: §C.4 · pp. 23–24, Table A12 · p. 24

C.5 NULL TEST DETAILS

  • The scale-variance null test permutes point-to-group labels while preserving group sizes, recomputes the null distribution 100 times per setting, and compares observed values with its 95th percentile.Table A13 reports observed and null statistics; its excess-over-p95 column is computed per sequence before aggregation.Source: §C.5 · p. 24, Table A13 · p. 24

C.6 ALL EVALUATED SETTINGS

This section describes the evaluated benchmark settings and examines how structural metrics relate to endpoint-error improvement. It also identifies the settings included in the plotted analysis.

C.6 ALL EVALUATED SETTINGS

  • The two real-world WorldTrack test sets, Aria Digital Twin and Panoptic Studio, use annotations converted from TAPVid-3D camera-frame trajectories with camera extrinsics into world coordinates.Source: §C.6 · p. 24
  • Between-group scale variance correlates with endpoint-error improvement (r = 0.61, n = 58, p < 0.0001), while radial energy and flow direction error are not significant.Their correlations are r = 0.078 (p = 0.56) and r = 0.058 (p = 0.67), respectively.Source: §C.6 · p. 25
  • Figure A1 plots relative endpoint-error improvement against between-group scale variance across 58 model, split, and protocol points, with a least-squares fit and 95% confidence band.Source: Figure A1 · p. 25, §C.6 · p. 25
  • Figure A2 is a heatmap of relative endpoint-error improvement.Source: Figure A2 · p. 25

C.7 COMPLETE ERROR BUDGET TABLE

The complete error-budget appendix reports ranges for three medians and identifies the upper-bound comparison in Table A14. Additional figures show D4RT’s anchor-budget curves.

C.7 COMPLETE ERROR BUDGET TABLE

  • The three medians reported in Section 5.7 have ranges of 7.1% to 82.6%, -2.8% to 62.6%, and -11.5% to 92.5%, respectively.Source: §C.7 · p. 25
  • Table A14 compares the upper bound from fitting per-group, per-frame radial scale and translation on all ground-truth points with GAUGE using 5% anchors.Source: Table A14 · p. 26

C.8 ADDITIONAL FIGURES

Related papers

Loading 2609.32596v1…