Source-linked AI summary
Easi3R: Estimating Disentangled Motion from DUSt3R Without Training
Xingyu Chen, Yue Chen, Yuliang Xiu, Andreas Geiger, Anpei Chen
TL;DR
Dynamic video reconstruction remains difficult because object motion violates static-scene assumptions and entangles camera and object motion. Easi3R adapts DUSt3R attention at inference time without training, and outperforms trained or finetuned methods across dynamic reconstruction tasks.
Problem
Dynamic SfM struggles with moving objects because they violate geometric consistency and entangle camera and object motion, while existing solutions rely on extensive training data or task-specific priors.
Method
Easi3R decomposes DUSt3R attention maps to segment dynamic objects, then reweights attention in a second inference pass for 4D reconstruction and camera motion recovery without training.
Results
Easi3R performs robustly across camera pose estimation, dynamic object segmentation, and dynamic-scene pointcloud reconstruction, surpassing trained or finetuned methods in most cases.
Takeaways & Limitations
DUSt3R attention contains exploitable spatial and temporal motion information, enabling robust training-free 4D reconstruction at almost no additional cost.
Takeaways & Limitations
Registration assumes reliable pairwise reconstruction and can fail when dynamic content occupies a considerable portion of the image.
Abstract
from arXiv · showhide
Recent advances in DUSt3R have enabled robust estimation of dense point clouds and camera parameters of static scenes, leveraging Transformer network architectures and direct supervision on large-scale 3D datasets. In contrast, the limited scale and diversity of available 4D datasets present a major bottleneck for training a highly generalizable 4D model. This constraint has driven conventional 4D methods to fine-tune 3D models on scalable dynamic video data with additional geometric priors such as optical flow and depths. In this work, we take an opposite path and introduce Easi3R, a simple yet efficient training-free method for 4D reconstruction. Our approach applies attention adaptation during inference, eliminating the need for from-scratch pre-training or network fine-tuning. We find that the attention layers in DUSt3R inherently encode rich information about camera and object motion. By carefully disentangling these attention maps, we achieve accurate dynamic region segmentation, camera pose estimation, and 4D dense point map reconstruction. Extensive experiments on real-world dynamic videos demonstrate that our lightweight attention adaptation significantly outperforms previous state-of-the-art methods that are trained or finetuned on extensive dynamic datasets. Our code is publicly available for research purpose at https://easi3r.github.io/
1. Introduction
Dynamic Structure-from-Motion remains difficult because moving objects violate static-scene assumptions and entangle object and camera motion. Easi3R addresses this with training-free attention adaptation of DUSt3R for segmentation, pose estimation, and dynamic 4D reconstruction.
- Dynamic video reconstruction remains challenging because static-scene SfM assumptions fail when objects move.
- Object and camera motions become entangled in dynamic videos, causing erroneous camera pose estimation.
- Easi3R uses DUSt3R attention maps to isolate dynamic, under-observed, and textureless regions for long-horizon object segmentation.
- Easi3R is a training-free method that jointly supports dynamic segmentation, dense point-map reconstruction, and robust camera pose estimation.
- Across three task categories, Easi3R performs well on diverse datasets and surpasses several methods trained on dynamic data.
2. Related Work
Prior dynamic reconstruction methods commonly add motion, depth, segmentation, or geometric components to systems designed for static scenes. Easi3R instead adapts DUSt3R at inference time without fine-tuning, pretraining for segmentation, or optical flow.
- SfM and SLAM: SfM and SLAM provide established foundations for structure and camera pose estimation but degrade under limited parallax and ill-posed conditions.
- SfM and SLAM: DUSt3R provides a robust Transformer-based multi-view stereo foundation but assumes primarily static scenes.
- Pose-free Dynamic Scene Reconstruction: Dynamic-scene methods incorporate pose estimation, map management, semantic segmentation, optical flow, geometric constraints, or generative priors.
- Easi3R: Easi3R takes a training-free, plug-in adaptation path that requires no fine-tuning and adds almost no computational cost.
- Motion Segmentation: Unlike methods relying on optical flow or segmentation pretraining, Easi3R extracts dynamic segmentation from pretrained 3D reconstruction models.
3. Method
Easi3R adapts DUSt3R at inference time by aggregating cross-attention across spatial and temporal dimensions, then using the resulting maps to disentangle dynamic regions and improve reconstruction. The method processes video with sliding temporal windows and applies attention-based reweighting before a second inference pass.
- 3.1. DUSt3R with Dynamic Video: Easi3R processes videos with a sliding temporal window, infers DUSt3R pairwise, and globally aligns the resulting pointmaps.The window contains temporally neighboring frame pairs, reducing redundancy because video connectivity is known.
- 3.1. DUSt3R with Dynamic Video: Dynamic objects can cause DUSt3R registration to fail because its static-scene epipolar matching policy is disrupted when moving content occupies many pixels.The global alignment assumes reliable pairwise reconstruction and registration of global content.
- 3.2. Secrets Behind DUSt3R: DUSt3R uses weight-sharing ViT encoders and two decoders whose self- and cross-attention exchange information between the reference and source branches.The network predicts pointmaps in the reference-view coordinate space from the two image branches.
- 3.2. Secrets Behind DUSt3R: Easi3R aggregates decoder cross-attention maps across spatial and temporal dimensions to expose token influence and motion-related patterns.Averaged maps capture overall source-to-reference contribution across decoder layers, while temporal aggregation uses pairwise attention statistics.
- 3.2. Secrets Behind DUSt3R: The method uses inverse reference attention and its variation to highlight low-attention, under-observed, textureless, and camera-motion-related regions.The reference view serves as the registration standard and is assumed to be static.
- 3.3. Dynamic Object Segmentation: Easi3R derives dynamic object segmentation from the element-wise product of complementary attention maps and reweights the reference decoder to suppress dynamic-source contributions.The reweighting uses an outer product of complementary masks and is followed by a second inference pass to enhance reconstruction quality.
4. Experiments
Easi3R is evaluated for dynamic segmentation, camera pose estimation, and 4D point-cloud reconstruction using DUSt3R and MonST3R backbones. Across these tasks, it improves robustness and often outperforms trained or fine-tuned dynamic-scene methods without additional fine-tuning.
- Evaluation setup: Easi3R is evaluated on dynamic object segmentation, camera pose estimation, and 4D reconstruction using DUSt3R- and MonST3R-based settings.The experiments include DAVIS segmentation, DyCheck, ADT, and TUM-dynamics pose estimation, plus DyCheck point-cloud reconstruction.
- Dynamic object segmentation: The enhanced segmentation setting uses method outputs as prompts for SAM2, while direct evaluation also reports network outputs without SAM2.The two settings are denoted w/ and w/o SAM2 in Table 1.
- Dynamic object segmentation: Easi3R accurately segments dynamic objects in textureless regions, small objects, and casual motions, while competing methods can miss or over-segment them.MonST3R struggles in textureless regions because it relies on optical flow, whereas DAS3R tends to over-segment.
- Camera motion: Easi3R provides more accurate and robust camera pose and trajectory estimation across DUSt3R and MonST3R backbones, including settings with and without optical flow.The reported improvement is achieved with only a few lines of code and can exceed models using optical-flow priors.
- 4D reconstruction: Easi3R improves point-cloud reconstruction and is comparable to or better than most baselines, including CUT3R trained on extensive datasets.Qualitative comparisons identify reduced misalignment and entanglement through attention-guided segmentation, attention re-weighting, and segmentation-aware global alignment.
5. Conclusion
The conclusion presents Easi3R as a training-free DUSt3R adaptation that disentangles attention maps for dynamic segmentation, camera recovery, and robust 4D reconstruction. The method achieves strong results with almost no additional cost and outperforms state-of-the-art methods in most cases.
- Method: Easi3R adapts DUSt3R’s spatial and temporal attention to achieve training-free, robust 4D reconstruction.The method decomposes attention maps to isolate textureless, under-observed, and dynamic-object components.
- Method: Attention-map decomposition enables dynamic object segmentation, followed by attention reweighting for 4D reconstruction and camera-motion recovery.The second inference pass adds almost no additional cost on top of DUSt3R.
- Results: Easi3R outperforms state-of-the-art methods in most evaluated cases and is proposed as an example of attention-map disentanglement for other tasks.The conclusion frames the findings as potentially useful beyond dynamic 4D reconstruction.
Supplementary Material
The supplementary document adds method details, ablation studies, limitation analysis, and qualitative results. It also directs readers to the project website for improved visualization.
- Supplementary contents: The supplementary material covers temporal-consistency details for dynamic object segmentation, ablations, limitations, and additional qualitative results.These materials are organized in Sections A through D.
A. Dynamic Object Segmentation
Easi3R builds temporally consistent dynamic segmentation by clustering DUSt3R features across frames and fusing cluster-level dynamic attention scores.
- Temporal Feature Clustering: Cross-frame k-means clustering groups similar encoder tokens and maps cluster assignments back across frames for temporal feature aggregation.The concatenated feature tensor has dimension (T × h × w) × c, and k = 64 clusters are used in all experiments.
- Attention Fusion: Cluster-level dynamic scores average base attention values within each cluster before producing the fused dynamic attention map.The fused map assigns each pixel the dynamic score of its cluster and is then used to infer segmentation.
- Segmentation Pipeline: The resulting segmentation is used in a second inference pass and in global optimization.Per-frame segmentation is first generated using an automatic Otsu threshold before these later stages.
- Temporal Consistency: Feature clustering improves temporal consistency because DUSt3R encoder features remain consistent across frames.Figure 8 visualizes the dynamic attention map, cluster assignments, and cluster-fused map.
- Evaluation: Table 6 evaluates the dynamic object segmentation ablation on DAVIS.The supplied passage identifies the evaluation table but does not provide its numerical results.
B. Ablation Study
The ablations test temporal attention, decoder re-weighting, and segmentation-aware alignment, showing that these design choices affect segmentation and 4D reconstruction quality.
- Dynamic Object Segmentation: Disabling any temporal cross-attention map reduces dynamic object segmentation performance.The result indicates that all four aggregated temporal cross-attention maps contribute to segmentation quality.
- Dynamic Object Segmentation: Cross-frame feature clustering enhances dynamic segmentation by exploiting temporal consistency in DUSt3R encoder features.This supports the temporal fusion design used in the segmentation pipeline.
- Qualitative Analysis: Easi3R improves static reconstruction alignment in one viewpoint but still produces floaters near object boundaries in another.The limitation is illustrated using top- and bottom-row reconstructions from two viewpoints.
- 4D Reconstruction: Re-weighting only the reference-view decoder outperforms re-weighting both reference and source decoders.The reference view serves as the static standard, motivating asymmetric decoder treatment.
- 4D Reconstruction: Segmentation consistently improves 4D reconstruction when incorporated into global alignment.Table 7 evaluates camera pose estimation and point cloud reconstruction on DyCheck.
C. Limitations
Easi3R remains constrained by inaccurate backbone depth predictions, especially for per-view depth accuracy, although it improves global reconstruction metrics and pose alignment.
- Depth Accuracy: Easi3R can fail when the DUSt3R or MonST3R backbone produces inaccurate depth predictions.The method improves completeness and distance on global point clouds, but a gap remains in depth accuracy.
- Static Scenes: Figure 10 reports improved pose estimation in static scenes after reweighting low-confidence regions.The figure connects this behavior to the results in Table 9.
- Evaluation: Table 8 provides video-depth evaluation results using CUT3R evaluation results for the baselines.The supplied passage gives the evaluation setup but no numerical values.
- Scope Boundary: The method focuses on dynamic regions and global alignment rather than correcting depth predictions in static regions.Per-view depth correction is identified as future work.
D. Addtional Results
Additional evaluations cover longer sequences, camera pose, segmentation, static scenes, runtime, and qualitative disentanglement, extending the reported evidence across settings and baselines.
- Evaluation Protocols: Full-length sequences introduce more dynamic and challenging motion than prior short-clip evaluation settings.The study also reports MonST3R and CUT3R protocol evaluations in Tables 8 and 9.
- Static Scenes: Easi3R improves static-scene performance on ScanNet through attention reweighting.The result is reported in Table 9 and illustrated in Figure 10.
- Dynamic Segmentation: Easi3R achieves zero-shot state-of-the-art dynamic segmentation using only the image as input.The comparison is reported against 2D dynamic segmentation baselines in Table 10 on DAVIS.
- Ablation: Using SegAnyMo improves pose accuracy by 9.62% and depth accuracy by 4.11% on Sintel.These results come from an ablation of segmentation settings in Table 11.
- Runtime: Easi3R runs at 0.31 FPS versus MonST3R at 0.33 FPS for 512 × 144 images on an NVIDIA RTX 4090.The reported speeds indicate nearly identical runtime under this resolution and hardware setting.
- Qualitative Comparison: Qualitatively, MonST3R under-segments and DAS3R over-segments dynamic masks, while CUT3R can misalign static structures and produce ghosting.Easi3R is reported to achieve more accurate segmentation and camera pose estimation in these comparisons.