3AM: 3egment Anything with Geometric Consistency in Videos

Yang-Che Sun; Cheng Sun; Chin-Yang Lin; Fu-En Yang; Min-Hung Chen; Yen-Yu Lin; Yu-Lun Liu

Source-linked AI summary

3AM: 3egment Anything with Geometric Consistency in Videos

Yang-Che Sun, Cheng Sun, Chin-Yang Lin, Fu-En Yang, Min-Hung Chen, Yen-Yu Lin, Yu-Lun Liu

arXiv:2601.08831v5Jan 13, 2026cs.CV

TL;DR

Large viewpoint changes expose the limits of appearance-based VOS, while existing 3D approaches require explicit geometry or costly processing. 3AM integrates MUSt3R geometric features into SAM2 and trains with field-of-view-aware sampling, achieving strong wide-baseline tracking while using RGB input at inference. Its geometric assumption is limited for substantially dynamic objects, for which it falls back to SAM2.

Problem
Appearance-based VOS struggles to maintain object identity under large viewpoint changes, while 3D methods require explicit 3D inputs or processing.
Method
3AM fuses multi-level MUSt3R geometric features with SAM2 appearance features and uses field-of-view-aware sampling to learn spatially consistent correspondences.
Results
90.6% IoU and 71.7% Positive IoU are achieved on the ScanNet++ Selected Subset, improving over SAM2Long by +15.9 and +30.4 points, respectively.
Takeaways & Limitations
3AM provides geometry-consistent tracking across wide-baseline views using only RGB input at inference while retaining SAM2’s handling of dynamic video scenarios through fallback.
Takeaways & Limitations
Substantial object motion can break the assumption of consistent geometric locations, so the method falls back to SAM2 when motion is detected.

Abstract

from arXiv · show

Video object segmentation methods like SAM2 achieve strong performance through memory-based architectures but struggle under large viewpoint changes due to reliance on appearance features. Traditional 3D instance segmentation methods address viewpoint consistency but require camera poses, depth maps, and expensive preprocessing. We introduce 3AM, a training-time enhancement that integrates 3D-aware features from MUSt3R into SAM2. Our lightweight Feature Merger fuses multi-level MUSt3R features that encode implicit geometric correspondence. Combined with SAM2's appearance features, the model achieves geometry-consistent recognition grounded in both spatial position and visual similarity. We propose a field-of-view aware sampling strategy ensuring frames observe spatially consistent object regions for reliable 3D correspondence learning. Critically, our method requires only RGB input at inference, with no camera poses or preprocessing. On challenging datasets with wide-baseline motion (ScanNet++, Replica), 3AM substantially outperforms SAM2 and extensions, achieving 90.6% IoU and 71.7% Tracking Recall on ScanNet++'s Selected Subset, improving over state-of-the-art VOS methods by +15.9 and +30.4 points. Project page: https://jayisaking.github.io/3AM-Page/

1 Introduction

3AM addresses the difficulty of maintaining object identity under large viewpoint changes by combining geometry-aware features with SAM2’s appearance-based tracking. It targets cross-view consistency without requiring explicit 3D supervision at inference and reports strong performance on wide-baseline videos.

Motivation: 2D VOS methods rely on appearance features and can fail when objects undergo large viewpoint changes, disappear temporarily, or resemble distractors.This limitation is important for robust scene understanding in dynamic 3D environments.
Motivation: 3D segmentation methods improve consistency but typically require explicit 3D inputs, camera poses, depth, masks, or offline processing.These requirements can limit practical deployment and promptable, efficient tracking.
Approach: 3AM fuses multi-level MUSt3R geometric features with SAM2 appearance features through a lightweight Feature Merger for geometry-consistent recognition.Its training strategy uses field-of-view-aware sampling to select spatially consistent object regions.
Results: 90.6% IoU and 71.7% Positive IoU are achieved by 3AM on wide-baseline ScanNet++ and Replica evaluation, compared with SAM2Long’s 74.7% and 41.3%.The reported improvements are +15.9 and +30.4 points.
Results: 3AM improves over state-of-the-art VOS methods on challenging viewpoint changes while retaining competitive performance on conventional VOS benchmarks.The framework also supports cross-view consistency without 3D supervision at inference.

2 Related Work

Related work spans memory-based 2D VOS, 3D instance segmentation, and end-to-end 3D-aware reconstruction. These approaches improve tracking or geometric consistency but leave gaps in viewpoint robustness, promptability, streaming, or explicit 3D requirements.

2D Video Object Segmentation: Memory-based VOS architectures improve object association through spatio-temporal attention, feature retrieval, and more efficient memory mechanisms.SAM2 provides a promptable streaming-memory foundation for this line of work.
2D Video Object Segmentation: Large viewpoint changes, occlusion, and disappearance-reappearance events still degrade SAM2 and related 2D VOS methods.Appearance-only correspondence is insufficient when visual appearance changes substantially.
3D Instance Segmentation: 3D instance segmentation methods predict masks from point clouds or lift and merge per-view masks across views.These paradigms provide geometric representations but can produce view-inconsistent or fragmented instances when cross-view consistency is not enforced.
End-to-End 3D-Aware Methods: End-to-end 3D-aware methods infer geometric structure from 2D inputs and increasingly combine geometry with semantic representations.This line includes dynamic reconstruction, Gaussian splatting, multi-view fusion, and semantic scene modeling.
End-to-End 3D-Aware Methods: PanSt3R integrates MUSt3R features for joint geometry and panoptic segmentation but is incompatible with promptable backbones and requires offline access to entire sequences.These constraints prevent streaming use.

3 Method

3AM augments SAM2 with MUSt3R features and geometry-aware training-frame selection to maintain object identity across viewpoint changes. A sequence-level fallback addresses substantial object motion, where static geometric assumptions may fail.

Pipeline: 3AM extracts complementary SAM2 appearance features and MUSt3R geometry-aware features before merging them for segmentation and memory processing.The merged representation is decoded into masks and encoded for future-frame memory.
Feature Merging: Intermediate MUSt3R layers are selected because early features preserve semantic correspondence while deeper features become more geometry-oriented and less semantically aligned.Multi-level sampling captures complementary semantic and geometric information.
Feature Merging: The Feature Merger combines sampled MUSt3R layers through cross-attention and convolutional refinement with SAM2’s segmentation-aware appearance features.The resulting Fmerged representation provides more reliable localization under severe viewpoint changes.
Training Frame Sampling: SAM2’s memory capacity of at most eight slots motivates careful training-frame sampling across diverse camera viewpoints.Sampling must expose viewpoint variation without creating contradictory supervision.
Training Frame Sampling: Naïve random sampling can pair distant regions of the same large object, creating contradictory geometric supervision and ambiguous learning.This object-spanning problem affects examples such as beds, tables, and cabinets.
Training Frame Sampling: Field-of-view-aware sampling retains candidate frames whose masked 3D points overlap the reference camera frustum, avoiding spatially inconsistent views while preserving occlusion variation.The filter uses camera poses and depth during training and does not remove partially occluded frames.
Dynamic Object Fallback: Substantial object motion can violate the method’s geometric-location assumption, so 3AM reverts to the original SAM2 pipeline when motion is detected.The fallback is applied at sequence level because unobserved motion outside the camera frustum is inherently ill-posed.

4 Experiments

The experiments evaluate 3AM under large viewpoint changes, object reappearance, and conventional VOS settings using complementary tracking metrics. Across ScanNet++, Replica, and online 3D instance segmentation, 3AM achieves strong performance and outperforms prior methods.

Evaluation Setup: 3AM is evaluated on wide-baseline camera motion, object disappearance and reappearance, and conventional VOS benchmarks.ScanNet++ and Replica provide substantial camera trajectory variation for testing viewpoint robustness.
Metrics: Tracking Recall measures accuracy when the object is present, while Accuracy measures overlap-conditioned performance on visible frames.IoU is computed over all frames, including object-absent frames, to reflect overall stability.
2D Evaluation: 90.61% IoU, 71.68% Tracking Recall, and 77.37% Accuracy are achieved on the challenging ScanNet++ Selected Subset.The subset emphasizes frequent object reappearance associated with severe viewpoint variation.
2D Evaluation: 3AM reaches 0.8119 IoU, 0.6381 Tracking Recall, and 0.6793 Accuracy on Replica, outperforming SAM2, SAM2Long, and DAM4SAM.Replica is characterized by large viewpoint variations.
3D Evaluation: 3AM achieves the highest online 3D instance-segmentation performance, with AP 47.3, AP50 59.7, and AP25 75.3.MUSt3R’s stable object alignment supports reliable cross-view correspondence, while CUT3R obtains Tracking Recall 0.2751.

5 Conclusion

3AM integrates 3D-aware features into SAM2 to improve viewpoint-robust video object segmentation while retaining RGB-only inference. On wide-baseline datasets, it substantially outperforms state-of-the-art VOS methods.

5 Conclusion: 3AM combines SAM2 with 3D-aware features for geometry-consistent tracking using only RGB input at inference.The method uses a Feature Merger and field-of-view-aware sampling, with a fallback to SAM2 for dynamic video scenarios.

A Overview

The supplementary material provides implementation, training, segmentation, and qualitative comparison details supporting the main manuscript.

A Overview: The supplement covers network architectures, FoV sampling, datasets, class-agnostic instance segmentation, and qualitative video tracking comparisons.It also includes qualitative comparisons for class-agnostic instance segmentation.

B Implementation Details

The implementation merges selected multi-level MUSt3R features with SAM2 image features while preserving SAM2’s shallow features for mask decoding.

B Implementation Details: SAM2.1-Large is the baseline, with multi-scale Hiera features used for frame embeddings and shallow features retained for mask decoding.Memory attention uses sinusoidal absolute positional embeddings and 2D RoPE.
B Implementation Details: MUSt3R features from the encoder and decoder layers [4, 7, 11] combine semantic cues with increasingly geometry-aware signals.The MUSt3R memory mechanism and view-coverage-based memory selection remain unchanged.
B Implementation Details: The Feature Merger uses positional embeddings, self-attention, cross-attention, and feed-forward refinement to form a coarse geometry-aware feature.PE3D comes from MUSt3R point and ray maps, while PE2D comes from MUSt3R’s 2D positional encoding.
B Implementation Details: The refined MUSt3R feature is concatenated with SAM2’s Hiera feature to produce Fmerged for memory attention and memory encoding.Shallow Hiera features remain untouched so the mask decoder can preserve high-resolution segmentation cues.

C.1 Field-of-View Sampling

Field-of-view-aware sampling supports geometric correspondence learning, but applying it to every batch harms SAM2’s inherited within-view feature matching. Training therefore balances cross-view consistency with the original matching ability.

C.1 Field-of-View Sampling: 100% FOV-aware sampling degrades SAM2’s original feature-matching ability.Applying FOV filtering to every batch over-regularizes training toward cross-view matching.
C.1 Field-of-View Sampling: 3AM retains SAM2’s original loss design for mask, IoU, and occlusion prediction.Mask prediction combines focal and dice losses, while the IoU and occlusion heads use L1 and cross-entropy losses, respectively.
C.1 Field-of-View Sampling: Mask prompts are used for ScanNet++ and ASE, while MOSE supports point, box, and mask prompts.The restriction reflects degraded 2D masks and large viewpoint variations introduced during training.

C.3 Datasets

The datasets combine synthetic and real-world settings with varying geometry, motion, and annotation characteristics. The class-agnostic segmentation pipeline propagates masks, lifts them into 3D, and merges fragments using spatial and temporal evidence.

C.3 Datasets: ASE contains 100,000 procedurally generated indoor scenes with simulated sensors, 3D layouts, trajectories, and aligned annotations.The dataset supports large-scale 3D scene understanding, detection, and tracking when real-world labeled data is scarce.
C.3 Datasets: MOSE contains 2,149 clips, 5,200 objects, 36 categories, and 431,725 masks featuring occlusion, disappearance, reappearance, and small targets.It is used to preserve the model’s core VOS capability.
C.3 Datasets: Training on highly dynamic and diverse datasets such as SA-V Train causes practical instability, so MOSE supplies more controlled motion variation.This retains dynamic-object supervision while supporting stable learning.
C.3 Datasets: ScanNet++ provides RGB-D reconstructions, camera poses, dense trajectories, geometric annotations, and substantial viewpoint diversity.Its projected 2D masks contain reprojection noise, and pose annotations may omit consecutive frame stretches.
C.3 Datasets: For class-agnostic 3D instance segmentation, SAM2 generates keyframe masks that 3AM propagates before back-projecting pixels with depth and camera pose.Boundary artifacts are reduced with reprojection scoring and slight mask erosion.
C.3 Datasets: Per-frame 3D fragments are merged using 3D point-cloud overlap and 2D temporal overlap, then duplicate assignments are resolved by superpoint majority voting.Temporal evidence benefits from recognizing objects again after reappearance.

E Qualitative Comparison

The qualitative evaluation presents visual comparisons of VOS methods and additional class-agnostic instance-segmentation results. The cited material identifies the evaluated figures but does not state a specific visual winner.

E Qualitative Comparison: Figures 8–14 provide video object tracking visualizations comparing different VOS methods.The supplied text does not specify a particular outcome from these comparisons.
E Qualitative Comparison: Figure 15 provides additional visual results for class-agnostic instance segmentation.The supplied text does not state a specific comparison outcome.