PaperlayerView exact revision on arXiv

Source-linked AI summary

StreamRig: Exploiting Intra-Rig Geometry for Streaming Multi-Camera Odometry

Yufei Wei, Shuhao Ye, Qi Wang, Xin Zheng, Qing Huang, Rong Xiong, Yue Wang

arXiv:2609.40244v1cs.CVcs.RO

TL;DR

Streaming odometry must exploit calibrated multi-camera geometry while keeping temporal inference affordable and learning under limited rig data. StreamRig combines a frozen multi-view foundation model with a compact back-end trained first on group relocalization and then on causal rig sequences. Across four rigs, it achieves lower translation and rotation drift than evaluated non-oracle methods.

  • Problem

    Streaming models process monocular inputs, while calibrated rig odometry must jointly use synchronized views and learn motion from limited rig data.

    Source: §I · p. 1

  • Method

    StreamRig keeps a calibrated multi-view 3D foundation model frozen and trains a compact pose back-end through group relocalization followed by causal rig training.

    Source: §I · p. 2, C. Training · p. 4

  • Results

    Across four rigs, StreamRig achieves the lowest translation and rotation drift among evaluated non-oracle methods.

    Source: §I · p. 2, B. Main Results · p. 5

  • Takeaways & Limitations

    The results support joint rig perception and temporal history as complementary sources of odometry accuracy, with compact modules making their combination affordable.

    Source: §V · p. 8

Abstract

from arXiv · show

Mobile robots and vehicles carry synchronized multi-camera rigs, yet many streaming 3D foundation models are designed for monocular input, leaving efficient use of rig geometry a challenge. We present StreamRig, a freeze-and-stream framework that builds causal streaming odometry for calibrated rigs on a frozen multi-view 3D foundation model. The frozen front-end jointly perceives the synchronized views using rig calibration. A Rig-Resampler compresses their features, a CausalBridge applies causal attention with a key-value cache, and a lightweight head regresses rig poses. A periodic re-anchoring protocol supports stable pose estimation over long sequences. Only these modules are trained, 74.6M parameters in total, with relative poses as the sole supervision. Our two-stage training strategy combines group relocalization pretraining with causal rig training to transfer the geometric priors of the frozen front-end and the alignment ability of the pretrained modules to streaming odometry. We evaluate on NCLT, TartanGround, KITTI-360, and our self-collected humanoid-robot dataset ZJH, where training uses only simulation and real-world evaluation is zero-shot. Across all four datasets, StreamRig achieves lower translation and rotation drift than the evaluated non-oracle monocular streaming and rig-aware offline models, while maintaining low inference cost. Ablations and controlled camera-count experiments identify the sources of these gains. We further examine how longer training windows affect inference over longer horizons. Code has been released at https://github.com/WeiYuFei0217/StreamRig.

I. INTRODUCTION

Calibrated multi-camera rigs offer joint geometric observations, but streaming odometry must use them without excessive temporal state and learn reliably from limited rig data. StreamRig combines a frozen multi-view front-end with a compact, pretrained causal pose back-end.

I. INTRODUCTION

  • Streaming monocular models process rig views as separate frames, increasing per-step tokens and cached state without explicitly encoding known relations among synchronized cameras.A rig-level streamer must jointly use those relations while avoiding carrying all image tokens through temporal history.Source: §I · p. 1
  • Learning both geometric perception and temporal reasoning from limited rig data is difficult, while freezing a pretrained front-end alone does not teach a pose back-end to use its representation.Rig datasets often have sparse LiDAR depth and limited layout diversity; the paper reports that a randomly initialized back-end learns motion poorly from causal rig sequences.Source: §I · p. 1
  • StreamRig adapts pretrained geometry with a compact pose back-end, first trained on group relocalization and then on causal rig sequences.Sampling groups across sessions and trajectories exposes the back-end to diverse geometries and displacements beyond one fixed rig layout.Source: §I · pp. 1–2
  • The framework jointly encodes calibrated synchronized images, compresses their features, reasons causally over them, and periodically re-anchors pose estimates.Only the compact back-end is trained, using relative-pose supervision alone.Source: §I · p. 2
  • Evaluation covers NCLT, TartanGround, KITTI-360, and a humanoid-robot rig, with humanoid training performed in simulation and real-world evaluation conducted zero-shot.The experiments include camera-count comparisons, ablations, and studies of training-window length and usable anchor range.Source: §I · p. 2
  • Across four rigs, StreamRig has the lowest translation and rotation drift among evaluated non-oracle methods; five-camera inference also uses less memory and has lower latency than monocular CUT3R.Figure 1 separately illustrates closer NCLT trajectory tracking and reports lower ATE, memory use, and latency for five-camera StreamRig than monocular CUT3R.Source: Figure 1 · p. 1, §I · p. 2

II. RELATED WORK

Prior work separately addresses calibrated multi-camera geometry and causal streaming, leaving a gap in streaming odometry for calibrated rigs. StreamRig applies a frozen multi-view model to this setting and estimates poses relative to a periodically advanced anchor.

II. RELATED WORK

  • Rig-aware optimization and fixed-set 3D foundation models handle calibrated views, while streaming 3D models retain temporal history but process one image per step without known calibration.The paper identifies the lack of prior adaptation of a frozen multi-view foundation model to causal odometry with calibrated rigs.Source: §II · p. 2

III. METHOD

  • StreamRig keeps the multi-view foundation model frozen and trains three lightweight modules to turn per-rig perception into causal state and odometry, using relative poses as sole supervision.The overview shows causal attention over compressed rigs, cached earlier keys and values, and a streaming loop that periodically re-anchors.Source: §III · p. 2, Figure 2 · p. 3
  • Each query pose is estimated relative to an anchor rig, whose reference-camera frame advances as the stream proceeds.The mapping predicts the current reference camera's pose in the anchor reference-camera frame in one feed-forward pass.Source: §III · p. 3

A. Rig Perception and Rig-Resampler

StreamRig forms one geometry-grounded observation from each calibrated rig, then compresses each camera’s features before temporal reasoning.

A. Rig Perception and Rig-Resampler

  • MapAnything encodes each synchronized rig once using its images, intrinsics-derived ray directions, and rig extrinsics.Its alternating-attention transformer exchanges information across cameras to produce geometry-grounded patch features.Source: A. Rig Perception and Rig-Resampler · p. 3
  • Calibration enters StreamRig through joint rig perception; the trainable modules do not receive extrinsics or assume a camera layout.Frozen features are cached offline for training, and other multi-view front-ends accepting images, intrinsics, and relative poses can supply features.Source: A. Rig Perception and Rig-Resampler · p. 3
  • The Rig-Resampler uses shared learnable queries and two cross-attention layers to reduce each camera to 16 latent tokens.For five cameras, this compresses 1280 patch tokens to 80, targeting motion and scale cues rather than dense reconstruction.Source: A. Rig Perception and Rig-Resampler · p. 3

B. CausalBridge and Pose Head

The CausalBridge maintains compressed rig history with causal attention and per-query anchor snapshots. A pose head reads bridge outputs to estimate each query rig’s pose relative to the anchor.

B. CausalBridge and Pose Head

  • Across rigs, CausalBridge attention is causal, while each query receives its own anchor snapshot so anchor and query can attend to one another without future observations.The query can also attend to intervening rigs after the anchor; outputs depend only on the anchor through the current rig.Source: B. CausalBridge and Pose Head · p. 3
  • The bridge uses no temporal position code; order comes from the causal mask and token content, permitting longer input windows while anchor distance remains bounded by training displacements.The paper tests adding rotary temporal embeddings.Source: B. CausalBridge and Pose Head · p. 3
  • For each query camera, rotation and translation queries attend to anchor reference-camera tokens and that camera’s tokens, then MLPs decode rotation and translation.Other cameras serve as auxiliary targets whose poses follow from the odometry pose and known calibration, requiring no extra labels.Source: B. CausalBridge and Pose Head · pp. 3–4

C. Training

Training adapts the frozen geometric representation through group-relocalization pretraining followed by causal rig training. Relative poses provide the sole supervision, with mixed window lengths and calibration perturbations shaping training.

Relative-pose supervision

  • Training uses relative-pose targets at query offset d, with a rotation loss and displacement-normalized translation loss.The rotation term is an RMS chordal loss; ground-truth displacement normalization balances near and far translation targets while retaining metric-scale supervision.Source: C. Training · p. 4

Two-stage training

  • Stage I pretrains the modules on paired view groups, then Stage II continues from those weights on causal rig windows with clean extrinsics.Stage I perturbs input extrinsics by 1.5° rotation and 0.1 m translation while keeping targets clean; Stage II changes the inputs and supervision but not the modules or their parameters.Source: C. Training · p. 4

Window and stride sampling

  • Window lengths are sampled from 6 to 48 with probability proportional to length, while frame stride is sampled uniformly up to a platform-specific maximum.This mixed sampling preserves long-range coverage while giving short-range targets greater relative weight.Source: C. Training · p. 4

Training budget

  • The trainable modules contain 74.6 million parameters, or 11.7% of the 638-million-parameter system.Stage II uses 576,000 windows; with frozen features cached, NCLT training costs about 48 GPU-hours on RTX 4090 GPUs with 24 GB.Source: C. Training · p. 4

D. Streaming Inference and Re-anchoring

For each new rig, StreamRig uses cached bridge keys and values to produce a pose from the new rig and its anchor snapshot. Re-anchoring bounds the active state, while an optional long-history mode trades short-range accuracy and growing state for retained history.

Per-step inference

  • Cached keys and values from earlier rigs let one bridge pass over the new rig and its anchor snapshot produce the pose estimate.The backbone perceives each new rig once, and the Rig-Resampler compresses it before its tokens join the active window.Source: D. Streaming Inference and Re-anchoring · p. 4

Re-anchoring protocol

  • Re-anchoring every N rigs makes the last rig of each block the next anchor and composes the trajectory from anchor-to-anchor poses.The state holds at most N compressed rigs, or N CL tokens; short blocks accumulate more hop errors, while long blocks can exceed the learned translation range.Source: D. Streaming Inference and Re-anchoring · p. 4

Optional long-history mode

  • In optional long-history mode, rigs more than eight steps behind retain only four latents per camera, while the cache drops the remaining keys and values.This mode preserves addressable history at the same per-step latency, but state still grows with sequence length and pruning costs short-range accuracy; re-anchoring remains the default.Source: D. Streaming Inference and Re-anchoring · p. 4

IV. EXPERIMENTS

Experiments evaluate four calibrated rigs with causal full-sequence trajectories and dataset-specific splits. Comparisons include monocular streaming and rig-aware offline methods.

A. Setup

  • Evaluation spans NCLT, TartanGround, KITTI-360, and ZJH, including zero-shot real-world testing after simulation-only training on ZJH.The rigs range from co-centered or co-located surround cameras to stereo pairs with side or oblique cameras; KITTI-360 training and evaluation use separate drives.Source: A. Setup · p. 4
  • Every method runs causally over each complete sequence, anchors to its first pose, composes its own predicted trajectory, and re-anchors every N sampled rigs or frames.N=2 gives a pairwise chain, while larger N retains a longer state; ground-truth poses are used only for evaluation except for the MA-AB oracle.Source: A. Setup · p. 4
  • Table I covers streaming odometry on four rigs, while Table II scopes a KITTI-360 comparison across mono, stereo, and full-rig inputs.The KITTI-360 camera-subset comparison changes only which cameras enter the frozen front-end and reports full-drive ATE after SE(3) or Sim(3) alignment.Source: Table I · p. 5, Table II · p. 5, A. Setup · p. 5
  • Evaluation reports relative translation and rotation drift plus full-sequence ATE, with alignment and scale handling varying by method and dataset.KITTI-360 metrics pool both drives weighted by path length; ZJH uses a shared test stride and one global scale per sequence for every method.Source: A. Setup · p. 5
  • Baselines include target-fine-tuned CUT3R and STream3R, released-weight LingBot-Map and HorizonStream, and target-trained or oracle rig-aware methods.Rig-aware comparisons include Reloc3R, Rig3R, MapAnything variants, and G2G-chain.Source: A. Setup · p. 5

B. Main Results

StreamRig combines rig-aware perception with temporal state and achieves the lowest translation and rotation drift among non-oracle methods across all four rigs. The results also show how performance varies by dataset and re-anchoring interval.

Main Results

  • Across all four rigs, StreamRig combines joint rig perception with temporal state and achieves the lowest translation and rotation drift among non-oracle methods.Figure 3 shows example trajectories after alignment to a common start.Source: B. Main Results · p. 5, Figure 3 · p. 6
  • On NCLT, G2G-chain drifts less than every monocular streamer, while StreamRig’s long-range anchor comparisons avoid composing estimates at every hop; MA-AB still drifts 46%.NCLT includes two 6 km sessions with indoor-outdoor transitions, lighting changes, motion blur, and pedestrians.Source: B. Main Results · p. 5
  • On TartanGround, StreamRig leads the comparison, combining the rig perception carried by the frozen front-end with the history that improves on a single pair.G2G-chain drifts far less than target-split-retrained Reloc3R and Rig3R; fine-tuned CUT3R and STream3R reach a similar level.Source: B. Main Results · p. 5
  • Figure 4 plots translation and rotation drift against re-anchoring interval; translation error is U-shaped for all three models, with lowest drift at intermediate blocks.Stars mark intervals beyond training limits, and top bars show estimated RTX 4090 GPU-hour training costs.Source: Figure 4 · p. 6
  • On KITTI-360, the full rig leads among StreamRig’s camera configurations on every metric, while STream3R has the lowest ATE across Table I methods.The stereo pair draws level with CUT3R when StreamRig uses only the front camera; adding two side cameras widens the field of view.Source: B. Main Results · pp. 5–6
  • On the real ZJH humanoid, StreamRig has the lowest translation drift, rotation drift, and ATE; its rotation drift is 63% below the second-best method, CUT3R.Training uses simulation only, with real-robot evaluation performed zero-shot.Source: B. Main Results · p. 6

C. Re-anchoring Distance and Training Horizon

Re-anchoring interval has a U-shaped drift trade-off, and longer training windows extend the usable anchor range at increased training cost. Ablations examine temporal coding, initialization, loss balancing, anchor interaction, and the frozen front-end.

C. Re-anchoring Distance and Training Horizon

  • Translation drift is U-shaped in re-anchoring distance, reaching its standard-model minimum near N=24 before rising modestly at N=48 and sharply at N=64.The sharp rise at N=64 is beyond the standard model’s training horizon; emitted displacements saturate as test targets exceed the training range.Source: C. Re-anchoring Distance and Training… · p. 6
  • Training to [6, 256] and [6, 512] rigs shifts the large-block rise outward, but does not improve minimum drift or guarantee low error throughout the training horizon.Estimated RTX 4090 training costs rise from 48 GPU-hours to 270 and 710, respectively.Source: C. Re-anchoring Distance and Training… · p. 6

D. Ablations

  • Removing Stage I raises translation drift by approximately 4.6–14.9×; removing rotary temporal coding brings no gain and increases translation drift by up to 8%.The authors attribute the initialization result to the need to teach the back-end to align geometric features before causal adaptation.Source: Table III · p. 7, D. Ablations · p. 7
  • Fixed 48-rig windows and unnormalized translation loss under-train near targets; mixed windows and displacement normalization counter this imbalance.The fixed window costs 15 to 30% across three rigs, while no normalization costs up to twice that; up-to-scale supervision also has higher translation drift on all three rigs.Source: Table III · p. 7, D. Ablations · p. 7
  • Replacing the per-query anchor snapshot with a static anchor costs 9 to 38% across three rigs.The table varies one factor at a time; the snapshot lets anchor and query attend to each other through the bridge.Source: Table III · p. 7, D. Ablations · p. 7
  • Halving latents from 16 to 8 costs up to 18% in translation; doubling to 32 keeps translation within a few percent and lowers rotation drift by about a quarter on NCLT and TartanGround.The 32-latent setting uses about three times the training compute, so the authors retain 16 latents per camera.Source: Table III · p. 7, D. Ablations · p. 7
  • Alternative frozen front-ends converge and stream, while their ordering tracks how directly they expose intra-rig geometry to the readout, not model size.The comparison is on NCLT.Source: Table IV · p. 7, D. Ablations · p. 7

E. Compute and Memory Budget

StreamRig’s five-camera inference has lower measured latency and memory use than the fastest monocular streamer, with most latency spent on perception. Re-anchoring, rather than compression alone, bounds memory.

E. Compute and Memory Budget

  • 26.2 ms and 2.6 GiB for a five-camera StreamRig rig compare with 31.3 ms and 3.9 GiB for one CUT3R frame.All methods use the same GPU and resident inputs, measured from RGB to pose over 512 arrivals.Source: E. Compute and Memory Budget · p. 7
  • The frozen backbone and Rig-Resampler take 24.9 of 26.2 ms, while the CausalBridge and pose head take about 1.3 ms; re-encoding both rigs in a pairwise chain costs 1.9 times more.A cached-feature chain matches StreamRig’s work at N=2, so the reported gain over that chain is accuracy at equal cost, not speed.Source: E. Compute and Memory Budget · pp. 7–8
  • Optional long-history mode keeps per-step latency unchanged but grows to 3.0 GiB after 512 rigs; bounded memory comes from re-anchoring, not compression alone.Figure 1’s runtime costs were measured separately on an RTX 4090 with BF16 and batch size one; its LiDAR points are visualization only, not odometry inputs.Source: E. Compute and Memory Budget · pp. 7–8

V. CONCLUSION

StreamRig combines joint rig perception with temporal history for odometry, using a compact back-end and group-relocalization initialization. Its re-anchoring study finds that longer training windows extend the usable anchor range; revisit retrieval and joint pose–geometry refinement remain future work.

V. CONCLUSION

  • StreamRig’s results support joint rig perception and temporal history as complementary sources of odometry accuracy.The Rig-Resampler and CausalBridge make the combination affordable, while group relocalization supplies alignment initialization for limited rig data.Source: §V · p. 8
  • Longer training windows extend the usable anchor range, and effective streaming combines compact geometric observations with an appropriate re-anchoring interval.Source: §V · p. 8
  • Same-session revisit retrieval is not quantitatively evaluated, and its acceptance thresholds still require per-session tuning.Source: §V · p. 8
Loading 2609.40244v1…