PaperlayerView exact revision on arXiv

Source-linked AI summary

Pose-Free Feed-Forward 3D Inpainting via Learnable Mask Attention and Support Token Refinement

Jingyi Pan, Dan Xu, Qiong Luo

arXiv:2610.11857v1cs.CV

TL;DR

Existing 3D scene inpainting typically requires calibrated camera poses, limiting use with casual, in-the-wild image collections. FreeInpaint adapts a 3D foundation model to inpaint unposed masked images, and experiments report strong performance without precomputed poses. Its support-token refinement depends on 2D diffusion priors, and very restricted views of complex 360-degree scenes remain challenging.

  • Problem

    Existing 3D inpainting approaches typically require accurately calibrated camera poses, adding preprocessing overhead and limiting applicability to casual, in-the-wild captures.

    Source: §1 · pp. 1–2

  • Method

    FreeInpaint adapts a 3D foundation model for joint pose estimation, geometry recovery, and inpainting from unposed masked images, using learnable mask attention and diffusion-generated support tokens.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 6

  • Results

    FreeInpaint achieves superior performance across evaluated metrics against feed-forward unposed baselines and runs in 0.4s via a single forward pass.

    Source: §4.3 · pp. 8–9

  • Takeaways & Limitations

    FreeInpaint performs 3D scene inpainting directly from unposed, masked collections without relying on precomputed camera poses.

    Source: §1 · p. 2

    Final 3D quality is bounded by the 2D diffusion prior, and scenes with only 1 or 2 restricted views of a complex 360-degree scene remain challenging.

    Source: §A.6 · p. 16

Abstract

from arXiv · show

3D scene inpainting aims to recover missing or occluded regions in edited 3D scenes, while ensuring geometric and textural consistency. Existing approaches, however, typically require accurately calibrated camera poses, which restricts their applicability in casual, in-the-wild scenarios and introduces additional preprocessing overhead. To overcome this limitation, we present FreeInpaint, a novel feed-forward framework that generates complete and 3D-consistent scenes directly from unposed multi-view images with masked regions. At its core, FreeInpaint extends a 3D foundation model to propagate masked regions from a reference view to other unposed views, bridging 3D reconstruction and scene inpainting while preserving the model's native ability to recover camera poses and scene geometry. Our method addresses two key challenges in adapting feed-forward 3D foundation models to masked inputs. First, masked regions can corrupt cross-view correspondence reasoning, degrading pose estimation and geometry recovery. To address this, we introduce a Learnable Mask Attention mechanism that preserves the spatial anchoring of reliable observations while allowing masked regions to progressively absorb useful context in deeper layers. Second, under severe occlusions, a single forward pass often lacks sufficient appearance evidence for high-fidelity completion. Therefore, we propose a Support Token Refinement strategy, which injects diffusion-generated support evidence as confidence-weighted auxiliary tokens to refine under-observed regions while preserving the original spatial anchor. Extensive experiments across diverse datasets demonstrate that FreeInpaint achieves superior inpainting quality, eliminating the reliance on pre-computed camera poses while keeping a fast inference speed. The project page is https://rorisis.github.io/FreeInpaint/.

1 Introduction

3D scene inpainting must preserve geometric and textural consistency across views, but existing approaches depend on calibrated camera poses and often brittle SfM preprocessing. FreeInpaint adapts a 3D foundation model for feed-forward inpainting from unposed, masked images using mask attention and support-token refinement.

1 Introduction

  • Existing 3D inpainting methods rely on precisely calibrated camera poses, often requiring time-consuming SfM preprocessing that can fail on sparse, untextured, or masked captures.Failed or inaccurate calibration can compromise downstream inpainting and distort geometry.Source: §1 · pp. 1–2
  • FreeInpaint generates complete, 3D-consistent scenes directly from unposed image collections with user-defined masked regions.It adapts a 3D foundation model while preserving its geometric priors.Source: §1 · p. 2
  • Masked regions can disrupt cross-view correspondences needed for pose and geometry estimation, while severe occlusions can leave a single pass without enough appearance cues.These problems can corrupt spatial anchoring or produce blurry textures.Source: §1 · p. 2
  • Learnable Mask Attention regulates masked-token participation across network depth, while Support Token Refinement injects confidence-weighted generated evidence into under-observed regions.The approach aims to preserve reference-driven structure and multi-view consistency while refining blind spots.Source: §1 · p. 2
  • Experiments on SPIn-NeRF, 360-USID, and LLFF report performance across forward-facing and 360-degree scenes without requiring camera poses.The contributions also identify diffusion-generated support evidence as auxiliary tokens for blind-spot inpainting.Source: §1 · p. 2

2 Related Work

Prior 3D inpainting spans optimization-based and reference-based approaches, while feed-forward 3D models recover scene structure from unposed images but assume full observation. FreeInpaint adapts those models for pose-free inpainting.

2 Related Work

  • 3D inpainting methods use 2D foundation-model semantics or diffusion priors for localized editing and iterative completion [31] [6] [43] [30] [49] [27] [7] [42].Reference-based approaches propagate texture and geometry from a reference image across viewpoints [26] [22] [21] [34] [29] [46].Source: §2 · p. 3
  • Most existing reference-based methods require computationally intensive per-scene optimization and accurate SfM camera poses, limiting generalization to casual, sparse inputs.FreeInpaint instead proposes a feed-forward, pose-free framework.Source: §2 · p. 3
  • Feed-forward reconstruction models predict 3D representations from images; DUSt3R processes unposed pairs [38], while VGGT jointly predicts poses, depth, and correspondences [36].Foundation models including DA3 extend this line toward high-fidelity novel-view synthesis [11] [18], but assume full observation and lack dense inpainting capacity for severe occlusions.Source: §2 · p. 3

3 Method

FreeInpaint adapts a 3D foundation model to reconstruct 3D scenes from masked, unposed views, estimating relative poses while inpainting masked geometry and texture. Its method combines learnable mask attention, inference-time support refinement, and fine-tuning for inpainting.

3.1 Preliminary

  • The framework builds on DA3’s transformer and specialized depth, camera, and Gaussian prediction heads, representing scenes with explicit anisotropic 3D Gaussians for novel-view rendering.DA3 uses learnable camera tokens and alternating within-view and cross-view attention for unposed inputs.Source: §3.1 · pp. 3–4

3.2 Learnable Mask Attention

  • Learnable Mask Attention initializes per-token attention bias from occlusion masks and updates it across layers, preserving reliable spatial anchors while allowing masked tokens to contribute as they gather context.The bias is conditioned on current source tokens and controls how strongly masked regions participate in cross-view reasoning at different depths.Source: §3.2 · pp. 4–5

3.3 Support Token Refinement

  • Support Token Refinement identifies under-observed blind spots, selects an anchor view, and injects diffusion-generated support tokens with confidence weights to refine those regions.The diffusion view uses the current rendered image as a spatial canvas and the reference image as a textural prior; low-opacity blind spots receive higher support confidence.Source: §3.3 · pp. 5–6

3.4 Optimization

  • Training uses random, brush, and depth-projected masks, teacher-guided depth distillation, and combined photometric and multi-view depth losses, with photometric supervision on held-out views.The frozen DA3 teacher supplies depth priors from unmasked views; the photometric loss combines MSE and LPIPS.Source: §3.4 · pp. 6–7

4 Experiments

The experiments evaluate FreeInpaint on multiple scene-inpainting benchmarks and analyze its implementation, comparative results, and component behavior. Results include fast inference, comparisons with prior methods, and analyses of view-count and reference-view sensitivity.

4.1 Datasets and Evaluation Metrics

  • Evaluation uses SPIn-NeRF, 360-USID, and LLFF; PSNR and LPIPS assess datasets with ground truth, while C-FID and C-KID assess LLFF.Metrics are computed within the object-mask bounding box; CLIPdir is additionally reported for text-driven object replacement.Source: §4.1 · pp. 7–8

4.2 Implementation Details

  • Training uses DL3DV-10K and two-stage fine-tuning; inference can trigger support-token refinement when masked-region visibility falls below 0.97.The refinement strategy allows a maximum of three total feed-forward passes.Source: §4.2 · p. 8

4.3 Main Results

  • FreeInpaint consistently outperforms prior optimization-based methods across most metrics and completes inference in 0.4s via a single forward pass.Optimization-based approaches require 30 minutes to 5 hours.Source: §4.3 · pp. 8–9
  • FreeInpaint achieves superior performance across all evaluated metrics compared with LAMA+DA3 and MVInpainter+DA3.LAMA+DA3 suffers multi-view inconsistency, while MVInpainter+DA3 exhibits pose misalignments.Source: §4.3 · p. 9
  • For object replacement, FreeInpaint consistently outperforms reference-free and reference-based methods and produces clean, high-fidelity textures.Previous methods show artifacts and blurriness in inpainted regions.Source: Table 3 · p. 8, §4.3 · p. 9, Figure 6 · p. 9

4.4 More Analysis

  • On 360-USID, fine-tuning alone leaves blurry regions; learnable mask attention improves metrics and sharpness, while support-token refinement further helps blind spots.The qualitative ablation attributes sharper structural detail to mask attention and higher-fidelity textures in occluded regions to token refinement.Source: Table 4 · p. 8, §4.4 · pp. 9–10, Figure 7 · p. 9
  • Learnable mask attention improves ATE, RPEt, and RPEr over fixed hard masking, while both masking strategies outperform having no mask attention.The comparison evaluates pose accuracy on 360-USID.Source: Table 5 · p. 9, §4.4 · p. 10
  • Reconstruction quality generally improves as input views increase from 4 to 32, while perceptual metrics remain stable with minor fluctuations.The view-count analysis uses SPIn-NeRF and 360-USID.Source: §4.4 · p. 10, Table 6 · p. 10

5 Conclusion

FreeInpaint adapts a 3D foundation model to perform pose estimation, geometry recovery, and inpainting from uncalibrated masked image collections. It combines learnable mask attention with inference-time support token refinement and achieves high-quality, multi-view-consistent inpainting with fast feed-forward inference.

  • FreeInpaint jointly estimates poses, recovers geometry, and inpaints uncalibrated masked image collections without pre-computed camera poses.Source: §5 · p. 10
  • Learnable Mask Attention adaptively regulates masked-token participation in cross-view reasoning to protect spatial anchoring.Source: §5 · p. 10
  • Inference-time Support Token Refinement injects diffusion-generated evidence as auxiliary tokens for progressive inpainting under severe occlusions.Source: §5 · p. 10
  • Experiments demonstrate high-quality, multi-view-consistent inpainting with fast feed-forward inference in unposed settings.Source: §5 · p. 10

A.1 Demo Videos

The project page includes demo videos of FreeInpaint results.

  • Demo videos on the project page illustrate the method’s multi-view consistency and geometric stability.Source: §A.1 · p. 14

A.2 Additional Dataset Details

The appendix adds IMFine for further ablations and qualitative evaluation.

  • IMFine is used for extended ablation studies and qualitative evaluation, with complex scenes and challenging view variations suited to assessing robustness.Source: §A.2 · p. 14

A.3 Additional Implementation Details

Learnable mask attention is applied to DA3’s cross-view attention blocks through independently instantiated lightweight networks.

  • Each DA3 cross-view attention block gets an independent lightweight network that predicts a learnable mask bias to suppress invalid masked information.Source: §A.3 · p. 14
  • The bias network uses 3 × 3 and depthwise 3 × 3 convolutions followed by a 1 × 1 projection, with GELU activations between layers.Source: §A.3 · p. 14
  • Zero-initializing the final projection starts the mask at the binary-mask prior before training adapts it; hidden dimension is 8, α is 1.5, and β is 10.Source: §A.3 · p. 14

A.4 Additional Details of Feed-Forward Baselines

The baselines combine pretrained image or multi-view inpainting with pose-free 3D reconstruction, while FreeInpaint adapts the 3D foundation model directly to masked observations.

A.4 Additional Details of Feed-Forward Baselines

  • LaMa+DA3 and MVInpainter+DA3 use official pretrained checkpoints without task-specific fine-tuning, and all components remain frozen.The baselines use officially recommended inference settings without test-set tuning.Source: §A.4 · p. 14
  • LaMa inpaints each masked view independently, whereas MVInpainter jointly processes masked multi-view inputs before DA3 reconstruction.The completed views are passed to pretrained DA3-Giant to estimate camera parameters and reconstruct 3D Gaussian scenes.Source: §A.4 · p. 14
  • All methods use identical input views, masked inputs, resolution, and pose-free evaluation, with no ground-truth or externally estimated poses as inputs.These baselines test whether cascaded 2D or multi-view inpainting plus generic 3D reconstruction suffices; FreeInpaint instead performs mask-aware cross-view reasoning.Source: §A.4 · p. 14

A.5 More Analysis

The analyses examine support-token refinement, fine-tuning, and mask-attention choices, then report results on view-count scalability and in-the-wild scenes. They also identify limits from 2D diffusion priors and very sparse views, and acknowledge risks of misleading visual content.

A.5.1 Discussion on Support Token Refinement

  • Combining IP-Adapter and SDEdit repairs geometric distortions while preserving texture identity without extra artifacts.IP-Adapter without SDEdit introduces structural artifacts, while SDEdit alone generates textures that do not match the reference; Figure 8 illustrates the comparison.Source: §A.5.1 · p. 14, Figure 8 · p. 15
  • Three iterations balance refinement quality and efficiency; higher visibility thresholds can over-refine reliable regions and increase the average to 4.33 iterations.Increasing from one to two or three iterations improves PSNR and LPIPS, but the gain from three to four is only +0.02 dB PSNR with linear computational overhead; τ = 0.97 is the selected threshold.Source: §A.5.1 · p. 15, Table 8 · p. 15
  • The 4-step LCM prior cuts diffusion time to 0.3s per refinement iteration versus 1.6s for the 50-step baseline, with better PSNR and LPIPS on tested scenes.Runtime includes two FreeInpaint forward passes.Source: §A.5.1 · p. 15, Table 9 · p. 16
  • Depth-guided self-distillation during fine-tuning constrains geometry and improves texture quality on SPIn-NeRF.Without it, inpainted regions show geometric artifacts such as holes or flattened surfaces.Source: §A.5.1 · p. 15, Figure 9 · p. 16
  • Among tested mask-attention strategies, FreeInpaint’s mask-initialized, feature-conditioned approach performs best in scene inpainting and camera pose estimation.MAT-style dynamic binary propagation and MLLAM-style layer-wise modulation both outperform the no-mask-attention baseline.Source: §A.5.1 · p. 16, Table 11 · p. 17

A.6 Limitations and Future Work

  • Quality is bounded by the 2D diffusion prior, and complex 360-degree scenes with only one or two restricted views remain challenging.The authors attribute the sparse-view difficulty to error accumulation during support token refinement; Figure 11 also describes occasional artifacts with highly sparse inputs and large masks.Source: §A.6 · p. 16, Figure 11 · p. 17

A.7 More Results

  • FreeInpaint outperforms InstaInpaint across both datasets and evaluated metrics, and benefits from denser unposed inputs while perceptual quality remains stable.On 10 GS25 and 15 Co3D scenes with eight input views, it also achieves the best results across all four reported no-reference metrics.Source: §A.7 · pp. 17–18, Table 12 · p. 17, Table 13 · p. 17

A.8 Impact Statement

  • The authors identify misleading or manipulated visual content as an ethical risk of 3D generative models.They state a commitment to responsible research and development.Source: §A.8 · p. 18
Loading 2610.11857v1…