PaperlayerView exact revision on arXiv

Source-linked AI summary

UniT: Unified Geometry Learning with Group Autoregressive Transformer

Haotian Wang, Yusong Huang, Zhaonian Kuang, Hongliang Lu, Xinhu Zheng, Meng Yang, Gang Hua

arXiv:2605.21131v1cs.CV

TL;DR

Geometry perception models have not jointly supported online, offline, multimodal, long-horizon, and metric-scale capabilities. UniT addresses this gap with a Group Autoregressive Transformer, bounded queue-style memory, anchor-free relational modeling, scale-adaptive geometry learning, and modal attention, achieving state-of-the-art performance across ten benchmarks and seven tasks.

  • Problem

    Existing geometry-perception approaches rely on incompatible assumptions for online and offline view configurations, while long-horizon memory and metric-scale generalization remain challenging.

  • Method

    UniT treats groups of sensor observations as autoregressive units, varying group size across online and offline inference while combining anchor-free modeling, queue-style KV caching, scale-adaptive loss, and modal attention.

  • Results

    UniT achieves state-of-the-art unified geometry perception across ten benchmark datasets spanning seven representative tasks and diverse view, modality, scale, and sequence settings.

  • Takeaways & Limitations

    A single feed-forward model can support diverse geometry-perception tasks while accommodating arbitrary view configurations and modality combinations, metric-scale estimation, and long-horizon scalability.

Abstract

from arXiv · show

Recent feed-forward models have significantly advanced geometry perception for inferring dense 3D structure from sensor observations. However, its essential capabilities remain fragmented across multiple incompatible paradigms, including online perception, offline reconstruction, multi-modal integration, long-horizon scalability, and metric-scale estimation. We present UniT, a unified model built upon a novel Group Autoregressive Transformer, which reformulates these seemingly disparate capabilities within a single framework. The key idea is to treat groups of sensor observations as the basic autoregressive units and predict the corresponding point maps in an anchor-free and scale-adaptive manner. More specifically, diverse view configurations in both online and offline settings are naturally unified within a single group autoregression process. By varying the group size, online mode operates over multiple autoregressive steps with single-frame groups, whereas offline mode aggregates a multi-frame group in a single forward pass. Meanwhile, a queue-style KV caching mechanism ensures bounded autoregressive memory over long horizons. This is enabled by reducing long-range dependencies on early frames through anchor-free relational modeling, thereby allowing outdated memory to be discarded on the fly. To improve metric-scale generalization across scenes, a scale-adaptive geometry loss is further introduced within this framework. It couples relative geometric constraints with a partial absolute scale term, implicitly regularizing global scale and inducing a progressive transition from scale-invariant geometry to metric-scale solutions. Together with a dedicated modal attention module for integrating auxiliary modalities, UniT achieves state-of-the-art performance in unified geometry perception, as validated on ten benchmarks spanning seven representative tasks.

I. INTRODUCTION

Existing geometry-perception capabilities remain fragmented across incompatible online, offline, multimodal, long-horizon, and metric-scale paradigms. UniT unifies them through group autoregression, bounded memory, scale-adaptive learning, and modal attention, achieving state-of-the-art performance across seven tasks and ten benchmarks.

  • Existing feed-forward models fragment five capabilities across incompatible paradigms: online inference, offline reconstruction, multimodal fusion, long-horizon scalability, and metric-scale estimation.
  • Group Autoregressive Transformer treats sensor-observation groups as autoregressive units and predicts corresponding point maps anchor-free and scale-adaptively.Group size controls how many frames are jointly processed in each forward pass.
  • Group size unifies inference behaviors: single-frame groups produce sequential online processing, whereas full-sequence groups recover offline parallel inference.
  • Queue-style KV caching bounds long-horizon memory at O(Q) by discarding outdated entries after a fixed queue capacity is exceeded.Anchor-free relational modeling reduces dependence on distant early frames, enabling this memory policy.
  • Scale-Adaptive Geometry Loss couples relative constraints with a partial absolute-scale term, encouraging progression from scale-invariant geometry toward metric-scale solutions.The authors describe this progression as an automatic curriculum that improves training stability.
  • UNIT integrates heterogeneous sensor modalities with Modal Attention and achieves state-of-the-art unified geometry perception across ten benchmarks covering seven representative tasks.The tasks include reconstruction, pose estimation, depth estimation, long-horizon perception, multimodal reconstruction, and depth completion.

II. RELATED WORK

Related work has expanded feed-forward geometry perception from point-map reconstruction to parallel, online, multimodal, metric-scale, and long-horizon settings. These advances motivate a unified model that combines flexible view configurations, auxiliary modalities, and scalable temporal reasoning.

  • Point-map methods support multi-view reconstruction, camera pose estimation, and video or monocular depth estimation, while correspondence and graph-based methods improve multi-view geometry.
  • Transformer-based parallel processing enables multiple viewpoints to be handled in one forward pass, motivating broader 3D foundation models.
  • Online methods maintain recurrent or explicit spatial memory to incorporate streaming observations, but expanding memory can increase computational overhead.
  • Multimodal approaches incorporate camera parameters and depth maps, with representative systems combining multimodal inputs and metric-scale estimation.
  • UniT’s architecture processes image groups with modal, frame, and global attention, using bidirectional attention within groups and causal attention across groups.The architecture decodes global point maps from fused tokens and uses an anchor-free camera head.
  • Long-horizon methods use overlapping subsequences or memory-compression strategies, broadening applicability while introducing redundant computation or altered memory representations.

III. METHOD

The method models geometry perception as autoregressive prediction of point maps from image and optional multi-modal observations. Group autoregression unifies online and offline inference by varying the number of jointly processed viewpoints.

  • Geometry Perception: Geometry perception predicts a sequence of target point maps conditioned on past and current observations.The conditional formulation can include optional depth maps, camera intrinsics, and camera extrinsics.
  • Autoregression: The autoregressive process predicts point maps sequentially through next-frame prediction in the online setting.Accumulated predictions form the target point-map sequence.
  • Group Autoregression: Group autoregression extends the autoregressive unit from one observation to a group of point maps processed together.Each group contains G viewpoints jointly observed at the same time step.
  • Group Autoregression: When G = 1, the formulation gives sequential online inference; when G = N, it gives single-step offline inference without temporal dependency.Intermediate group sizes unify configurations ranging from monocular video to multi-view reconstruction.

B. Group Autoregressive Transformer

The Group Autoregressive Transformer combines group-level causal attention, multi-modal token fusion, and queue-style memory to support unified geometry perception. It processes observations bidirectionally within groups while preserving causality across groups and integrates auxiliary modalities through spatially aligned attention.

  • Architecture: The transformer modifies alternating attention by adding temporal causality, group-level autoregression, and flexible multi-modal conditioning.Global attention attends only to observations up to the current time step, while groups are processed bidirectionally.
  • Group Causal Connection: Group causal attention is bidirectional within each group and causal across groups, masking tokens from future groups.Varying G from 1 to N supports arbitrary view configurations with synchronized cameras.
  • Modal Attention: Modal attention fuses image and auxiliary tokens at aligned spatial positions, explicitly injecting pixel-wise correspondence into multi-modal interactions.Point tokens encode depth and local ray maps, while pose tokens encode 12D camera extrinsics.
  • Modal Attention: The modal attention modules are inserted at four stages and account for about 3% of total parameters.Their effectiveness is evaluated through ablation studies.

C. Queue-Style KV Caching

UniT uses anchor-free relational modeling to make autoregressive memory independent of distant early frames, enabling queue-style KV caching for bounded long-horizon inference. The cache discards outdated entries after a fixed capacity while preserving relative geometric relationships.

  • Anchor-free relational modeling removes the autoregressive dependency on the first frame, allowing outdated KV-cache entries to be discarded.Geometric relationships are represented as relative transformations that can be stored independently in cache entries.
  • The camera head is redesigned from four forward passes to one, reducing its computational cost by 75% and simplifying KV-cache management.
  • Relative transformations let the cache operate without relying on early frames, distinguishing the method from memory compression techniques.The authors describe queue-style caching as orthogonal to existing acceleration methods.
  • A queue-style KV cache with fixed capacity Q bounds memory usage and computational complexity by O(Q), rather than allowing growth with sequence length.The mechanism maintains constant computational complexity per autoregressive step over long sequences.

D. Scale-Adaptive Geometry Loss

UniT uses a scale-adaptive geometry loss that combines scale-invariant constraints with partial absolute-scale supervision, progressively guiding learning toward metric-scale geometry without over-constraining regression.

  • Scale-Adaptive Geometry Loss: The loss couples scale-invariant local point-map and camera constraints with a partial absolute-scale term to regularize global scale.The ground-truth depth map normalizes the relative constraint, while confidence-aware regression supplies the absolute component.
  • Scale-Adaptive Geometry Loss: The scale-invariant formulation admits predictions matching ground truth up to shared scale, including camera rotations, point maps, and translations.The stated optimal solution satisfies ˆRi = Ri, ˆPi/ˆs = Pi/s, and ˆTi/ˆs=Ti/s.
  • Scale-Adaptive Geometry Loss: Adding an absolute point-map term drives the predicted scale toward the ground-truth scale and yields metric-scale translations and global point maps.The paper states that ˆs is driven toward s, followed by ˆTi = Ti and ˆXi = Xi after scale-invariant geometry converges.
  • Scale-Adaptive Geometry Loss: Empirically, the design produces an automatic curriculum in which the model learns scale-invariant geometry before gradually recovering metric scale.The paper reports that this behavior avoids over-constraining metric-scale regression.
  • Geometric Regularization: A shuffled normal loss regularizes global geometric consistency across views, while the regular normal loss enforces local consistency on local point maps.The shuffled loss applies normal supervision to randomly shuffled pixels across frames and remains anchor-free through the camera head.

E. Implementation Details

UniT is trained on heterogeneous metric-scale data with simulated modality combinations and varied group sizes, while implementation changes reduce overhead and support diverse view configurations.

  • Training Data: The training collection aggregates 21 public metric-scale datasets spanning indoor, outdoor, object-centric, human-centric, real-world, and synthetic settings.The stated diversity covers scene geometries, camera types, motion patterns, and scale distributions.
  • Multimodal Sampling: Training samples image-only sequences with probability 10% and mixed multimodal sequences with probability 90%, independently sampling each modality at 50%.The optional modalities are depth maps, camera intrinsics, and camera extrinsics.
  • Training Data: Depth-pattern simulation spans uniform densities from 0% to 100%, LiDAR patterns from 1 to 128 beams, SIFT feature points, and super-resolution grids with factors 1 to 16.These patterns are used to support depth maps from diverse sensors.
  • Training Details: Training uses dynamic sequence lengths of 12–24 and randomly samples group size G from 1 to 24 to accommodate different view configurations.The model is trained for 80K iterations at resolution 518 with randomly sampled aspect ratios in [0.33, 1.0].
  • Architecture Changes: UniT removes auxiliary VGGT prediction heads and replaces quaternion rotations with continuous 9D rotations orthogonalized using SVD.The stated purpose of removing the heads is to eliminate unnecessary computational overhead.

A. Experiment Setting

The experiments evaluate UniT across ten benchmarks and seven geometric perception tasks under scale-invariant and metric-scale settings, comparing it with feed-forward baselines spanning different paradigms.

  • Tasks and Datasets: The evaluation covers seven tasks: multi-view reconstruction, camera pose, video depth, monocular depth, long-horizon perception, multimodal reconstruction, and depth completion.The datasets include scene-level, object-centric, synthetic, real-world, and multimodal settings.
  • Tasks and Datasets: Multimodal reconstruction combines arbitrary depth-map, intrinsic, and extrinsic configurations, while depth completion tests four sparse depth patterns.These evaluations use 7-Scenes, ETH3D, ScanNetv2, Sintel, KITTI, and NYUv2.
  • Evaluation Protocol: Metrics include point-map accuracy, completion, and normal consistency; pose trajectory and relative errors; depth errors, δ<1.25 accuracy, and average rank.All metrics are reported in meters under the applicable alignment settings.
  • Evaluation Protocol: Scale-invariant evaluations apply Sim(3) or median alignment, whereas metric-scale evaluations disable scale adjustment.This protocol separates relative-geometry assessment from metric-scale estimation.
  • Baselines: Baselines span offline scale-invariant models, offline multimodal metric-scale models, RNN-like online inference, and autoregressive online inference.The compared models include VGGT, π3, MapAnything, DepthAnything3 (Nested), CUT3R, and StreamVGGT.
  • Efficiency Evaluation: Computational complexity is evaluated using parameters, frames per second, and maximum GPU memory at resolution 448 × 224 with sequence length 50.Online UniT is reported with queue capacities Q=1, Q=N/3, and Q=N.

B. Multi-View Reconstruction

Multi-view reconstruction is evaluated across 7-Scenes, NRGBD, and DTU under scale-invariant and metric-scale settings, where UniT achieves strong cross-setting rankings and accurate metric-scale geometry.

  • Results: UniT ranks first in scale-invariant online, metric-scale online, and metric-scale offline settings, and second in scale-invariant offline reconstruction.The comparison is reported in Tab. III across the three reconstruction datasets.
  • Qualitative Results: Qualitative point clouds are displayed at consistent scene scales, and the results consistently show more accurate metric-scale geometry estimation for UniT.The point clouds are presented in raw form without alignment or filtering.
  • Related Evaluation: Camera pose estimation shows the same ranking pattern, with first place in scale-invariant online, metric-scale online, and metric-scale offline settings and second in scale-invariant offline.The results are reported in Tab. IV and are described as capturing metric-scale trajectories while remaining competitive on relative pose relationships.

D. Video Depth Estimation

UNIT performs strongly across video-depth settings, including scale-invariant and metric-scale online and offline evaluation. Its long-horizon design combines hybrid inference with cached memory to avoid offline methods' quadratic-sequence limitation.

  • UNIT performs best in scale-invariant online, metric-scale online, and metric-scale offline settings, while remaining competitive in scale-invariant offline evaluation.
  • UNIT remains the top-performing method for metric-scale monocular depth estimation and ranks second in the scale-invariant setting.
  • On NRGBD sequences, offline methods have an advantage before out-of-memory issues, but quadratic complexity prevents DepthAnything3 from handling longer sequences.With 300 frames, DepthAnything3 achieves approximately half the ATE error of CUT3R but cannot process longer sequences.
  • UNIT supports hybrid offline-online inference by processing shorter sequences offline, switching to online inference for longer sequences, and reusing offline KV-cache memory.The described setup uses offline inference for the initial 150 frames before continuing online with cached memory.
  • UNIT achieves the best performance in most multi-modal combinations, demonstrating flexibility with auxiliary depth and camera modalities.MapAnything performs better when all modalities are available, possibly because it is trained from scratch with multi-modal inputs.

H. Depth Completion

UNIT performs robustly in depth completion across varied sparse sensor patterns and benefits from its unified training design. Ablations further show that scale-adaptive losses and larger groups improve performance while maintaining stability across configurations.

  • The evaluation uses four sparse-input patterns: uniform sampling, random LiDAR beams, SIFT-based SfM points, and super-resolution grids.The patterns vary in density, beam count, or downsampling factor, and results are reported in metric scale.
  • UNIT ranks first across all evaluated depth-completion scenarios and remains robust across different sparsity patterns.Training simulates multiple sparse patterns with varying densities to reduce the train-test gap across sensor configurations.
  • Applying direct ℓ1 regression in metric-scale space causes a clear drop in metric-scale performance, whereas the scale-adaptive design substantially improves convergence.
  • Larger group sizes consistently improve performance because more frames can interact through bidirectional attention within each group.The model remains stable across a broad range of configurations, supporting the flexibility of the unified design.
  • UNIT is presented as a unified feed-forward geometry model supporting arbitrary view configurations, modality combinations, metric-scale estimation, and long-horizon scalability.
Loading 2605.21131v1…