PaperlayerView exact revision on arXiv

Source-linked AI summary

Fill My Mirror: Geometry-Constrained Mirror Inpainting

Ofek Basson, Shimon Vainer, Yacov Hel-Or, Ohad Fried

arXiv:2609.03740v1cs.CV

TL;DR

Geometrically consistent mirror inpainting is difficult because reflections depend on scene geometry and viewpoint, while existing methods often rely on hallucinated appearance priors. Fill My Mirror projects visible 3D scene content into the mirror and uses two-mask diffusion to complete and refine the remaining regions. Experiments on synthetic and real images report improved reflection consistency over relevant baselines, especially for geometry-constrained pixels.

  • Problem

    Mirror inpainting must generate reflections consistent with a fixed visible scene, but mirror appearance is determined by scene and viewing geometry rather than learned appearance priors alone.

  • Method

    The training-free method projects visible scene geometry into the mirror, then uses two-mask diffusion to refine projected content and complete occluded or uncertain regions.

  • Results

    Experiments on synthetic and real datasets show consistent improvements over inpainting, image-editing, and mirror-aware generation baselines, particularly on geometrically constrained mirror pixels.

  • Takeaways & Limitations

    Explicit scene geometry improves mirror reflection consistency, and the proposed correspondence-based Reflection Consistency Score evaluates reflection correctness on real images without ground-truth geometry.

  • Takeaways & Limitations

    Performance depends on estimated scene geometry; large structural inconsistencies cannot be fully resolved, and view-dependent effects or glass-window reflection-transmission mixtures are outside scope.

Abstract

from arXiv · show

Mirrors are common in real-world images, yet producing geometrically consistent reflections with generative models remains challenging. Unlike most objects, mirror appearance depends on scene geometry and viewpoint, making it hard to synthesize using learned appearance priors alone. We address this in the mirror inpainting setting, where the scene is fixed and only the mirror region is generated. Our key insight is that much mirror content is geometrically constrained by the visible scene and need not be hallucinated. We estimate scene geometry and project visible content into the mirror to recover reflection regions determined by geometry. A generative model then completes the mirror region via a two-mask diffusion strategy balancing geometric constraints with the model's learned priors, reducing projection artifacts and improving reflection consistency. The method is training-free and applicable to complex real-world scenes. We evaluate on MirrorBench-V2 (synthetic) and real images. Using standard and geometry-aware metrics, we show that explicitly using scene geometry improves consistency.

1. Introduction

Mirror inpainting requires reflections that remain geometrically consistent with a fixed visible scene, unlike unconstrained image generation. The method projects geometry-determined content into the mirror and uses diffusion to complete the remainder, improving reflection quality when geometry is explicitly incorporated.

  • Motivation: Existing inpainting methods can produce visually pleasing mirror content but often fail to maintain geometrically consistent reflections because they hallucinate missing regions from learned priors.Depth-conditioned methods use depth as an auxiliary signal, while the proposed approach treats projected geometry as an explicit constraint.
  • Motivation: Mirror appearance changes substantially under small viewpoint changes because it is determined by scene and viewing geometry rather than intrinsic surface appearance.A 5° orientation change strongly alters a mirror while changing a non-reflective brick wall only slightly.
  • Approach: The proposed training-free method deterministically projects visible scene geometry into the mirror, then applies two-mask diffusion to refine projected content and generate unconstrained regions.The two masks balance geometric constraints with the model’s learned priors while mitigating projection artifacts.
  • Evaluation: Evaluations on MirrorBench-V2 and real images use standard image metrics and geometry-aware reflection consistency assessment to test the method with and without ground-truth geometry.The benchmark provides ground-truth geometry, while real images do not.
  • Results: Explicitly incorporating scene geometry produces better mirror reflections according to the reported standard and geometry-aware evaluations.The paper frames this improvement as a consequence of using geometry to constrain reflection generation rather than hallucinating the whole mirror.

2. Related Work

Prior work models mirror geometry, injects geometric signals into diffusion, or trains reflection-specific inpainting systems, but these approaches generally do not directly project scene geometry into the mirror during inference. The paper extends explicit geometric reasoning to single-image mirror inpainting and introduces reflection-specific evaluation.

  • Mirror Geometry and Rendering: Mirror-aware reconstruction and rendering methods treat reflections as secondary viewpoints governed by mirror symmetry, addressing duplicated geometry and inconsistent multi-view predictions.Related approaches use virtual reflected cameras or separate reflected and transmitted scene components.
  • Geometry-Guided Diffusion: Diffusion editing methods inject depth maps, surface normals, or other spatial structure to improve structural alignment, enabled by advances in monocular geometry estimation.ControlNet and geometry-guided editing represent this broader conditioning direction.
  • Reflection Inpainting: Reflection-specific inpainting methods train on synthetic data and feed monocularly estimated depth to the generation network as an additional conditioning signal.The cited methods use depth-conditioned reflection synthesis with data augmentation or curriculum strategies.
  • Paper Positioning: Unlike these approaches, the proposed method is training-free, requires no mirror-specific examples, and imposes deterministic geometric projection before denoising.Diffusion primarily completes regions not constrained by reflection geometry.
  • Evaluation: Common image-level and perceptual metrics do not explicitly measure physical reflection consistency, while silhouette IoU can miss pose, scale, or viewpoint errors.The paper therefore focuses evaluation on mirror pixels whose appearance is determined by scene geometry and supports assessment without ground-truth depth.

3. Method

The method reconstructs visible reflection content by projecting estimated or ground-truth 3D scene geometry into the mirror, then completes unresolved regions with a two-mask diffusion process. Interpolated denoising predictions preserve geometric structure while allowing generative refinement to address projection errors and uncertainty.

  • 3.1. Projection of Content From Scene Onto the Mirror: The pipeline estimates 3D scene attributes and camera parameters, projects visible scene content into the mirror, and identifies reflection pixels that can be deterministically recovered.Occluded or uncertain regions remain unresolved for subsequent inpainting.
  • 3.2. Geometry-Constrained Inpainting via Two-Mask Decomposition: Direct projection can propagate occlusion-recovery failures and inaccurate geometry into the result, so the refinement mask allows diffusion to correct projection artifacts.The two-mask formulation addresses errors without retraining or modifying the underlying diffusion model.
  • 3.2. Geometry-Constrained Inpainting via Two-Mask Decomposition: The projected inpainting mask becomes a geometry constraint mask, while the original mirror mask becomes a generative refinement mask covering the full mirror.The two masks respectively preserve geometric evidence and permit unconstrained correction or completion.
  • 3.2. Geometry-Constrained Inpainting via Two-Mask Decomposition: At each denoising step, the method computes velocity predictions using the geometry constraint mask and full-mirror refinement mask, then interpolates them.The mixed prediction is normalized to match the ℓ2 norm of the geometry-guided prediction and is restricted to selected timesteps.
  • 3.2. Geometry-Constrained Inpainting via Two-Mask Decomposition: Early diffusion steps emphasize geometry-guided global reflection structure, whereas later steps use generative refinement to improve appearance details and complete uncertain regions.The transition sharpness is controlled by n, and mixing across all timesteps empirically degraded results.
  • Implementation Scope: The dual-mask approach requires a backbone whose transformer accepts an explicit mask, making it incompatible with latent-compositing inpainting inference that lacks mask conditioning.The stated boundary includes models such as FLUX.2 and Qwen-Image-Edit under their described inference procedures.

4. Experiments

The experiments evaluate geometry-constrained mirror inpainting on synthetic and real images using standard metrics and geometry-focused evaluation. Across these settings, explicit scene geometry improves reflection quality and consistency, while noise interpolation strengthens robustness to projection errors.

  • Real Images without Ground-Truth Geometry: Qualitative comparisons show that our method better preserves reflected object structure, placement, color, and orientation than the baselines.Examples include beds, headboards, closets, and doors in captured real images.
  • Synthetic Scenes with Ground-Truth Geometry: Our method improves all reported metrics over competing methods on MirrorBench-V2 across constrained-pixel and full-mirror evaluations.The comparison includes inference with and without ground-truth geometry.
  • Real Images without Ground-Truth Geometry: On real images without ground-truth depth, our method improves PSNR, SSIM, and LPIPS under both RCS-constrained and full-mirror masks.RCS isolates pixels whose reflections are geometrically constrained by visible scene content.
  • Real Images without Ground-Truth Geometry: RCS constructs an evaluation mask from MASt3R correspondences between the scene view and horizontally flipped mirror view, excluding reflections of unseen content.Matched mirror pixels are dilated and intersected with the mirror region before computing masked metrics.
  • Real Images without Ground-Truth Geometry: Removing noise interpolation degrades nearly all metrics, with statistically significant gains for the full method over the ablation and competing baselines.The improvements are significant across full-mirror and RCS-constrained metrics for the ablation comparison.
  • Real Images without Ground-Truth Geometry: As estimated geometry worsens, the performance gap between interpolation and no interpolation grows, with correlations of r=0.741 for PSNR and r=0.778 for SSIM.Interpolation is mildly harmful with exact geometry but increasingly corrects occlusions and projection errors as geometry degrades.

5. Limitations

The method depends on accurate scene geometry and is bounded by the mirror and appearance settings it models. Noise interpolation mitigates moderate projection errors, but several structural, optical, runtime, and scope limitations remain.

  • Geometry Dependence: Inaccurate estimated geometry can misalign projected reflections, and noise interpolation cannot fully resolve large structural inconsistencies.The framework can benefit from improved geometry estimators, but its quality remains tied to recovered scene geometry.
  • Geometry Dependence: As the projected region shrinks, the pipeline increasingly relies on the inpainting model’s generation prior and approaches behavior without projection.This reduces the influence of explicit geometric constraints.
  • Appearance and Scope: Direct color copying can render specular highlights incorrectly, while glass-window reflections that combine reflection and transmission are outside the method’s scope.These effects are not handled by the projection formulation.
  • Appearance and Scope: The formulation assumes one planar mirror and does not handle multiple mirrors, higher-order interreflection, or curved mirrors.Supporting these cases would require extending the geometric projection stage.
  • Efficiency: Runtime increases to 100.5s per image versus 47.4s for FLUX.1 Fill alone, a 2.1× increase at the same 24.5GB GPU memory.The additional cost comes from geometry estimation and projection.

6. Conclusion

Fill My Mirror is a training-free, geometry-aware mirror inpainting method that combines deterministic projection with generative completion. Experiments on synthetic and real datasets show consistent improvements, especially on geometrically constrained pixels, with robustness to imperfect geometry.

  • 6. Conclusion: The method decomposes mirror inpainting into deterministic geometric projection and generative completion using two-mask diffusion.The strategy balances physical reflection constraints with learned priors while resolving occluded or uncertain regions.
  • 6. Conclusion: Experiments show consistent improvements over inpainting, image-editing, and mirror-aware generation baselines, particularly on geometrically constrained mirror pixels.The evaluation covers both synthetic and real datasets.
  • 6. Conclusion: The proposed Reflection Consistency Score evaluates reflection correctness on real images without ground-truth geometry.It is a correspondence-based evaluation protocol introduced alongside the method.

A. Implementation Details

The implementation uses standardized 1024 × 1024 inference with 30 diffusion steps and provides interchangeable monocular geometry estimators. Geometry projection substantially increases runtime over FLUX.1 Fill while using the same peak GPU memory.

  • A. Implementation Details: All experiments use 1024 × 1024 inputs, 30 diffusion inference steps, resized outputs, and closely followed official baseline inference examples.Reported Qwen-Image-Edit results use the newer Qwen/Qwen-Image-Edit-2511 weights.
  • A. Implementation Details: MoGe-v2 and Depth Anything 3 are supported as drop-in monocular geometry estimators, while reported results use MoGe-v2.The estimators provide interchangeable geometry inputs within the pipeline.
  • A. Implementation Details: 100.5s per image versus 47.4s for FLUX.1 Fill alone, a 2.1× runtime increase at the same 24.5GB peak GPU memory.The geometric projection stage averages 38 seconds per real image on an RTX 6000 Ada.

C. Existing Mirror Datasets and Their Limitations for Inpainting

Existing real-world mirror datasets are poorly suited to quantitative inpainting evaluation because reflections may be ambiguous, mirrors may violate planar assumptions, and text prompts are absent. The paper therefore uses a captioned Mir­rorVerse subset and validates geometry-aware evaluation and diffusion parameters on synthetic data.

  • C. Existing Mirror Datasets and Their Limitations for Inpainting: Public mirror datasets contain ambiguous or invisible reflections, curved or partial mirrors, and no text captions, limiting reliable quantitative inpainting evaluation.These conditions make pixel-level ground truth unreliable and conflict with the planar-mirror assumption and prompt-conditioned methods.
  • C. Existing Mirror Datasets and Their Limitations for Inpainting: A MirrorVerse subset supplies generated ground-truth reflections and prompts, enabling comparison with MirrorVerse and FLUX.1 Fill on real-looking images.The qualitative comparison reports more geometrically consistent reflections for the proposed method.
  • D.1. Quantitative hyperparameter selection: The diffusion exponent n controls transition sharpness, while start timestep t′ controls when interpolation begins; earlier starts and sharper transitions provide more generative freedom but may weaken geometric constraints.This effect is stronger for n ≥5 and t′ ≥750, while differences are small for n ≥5 and t′ ≤625.
  • D.2. RCS mask selection: The selected RCS mask parameters are radius = 5 and iterations = 1, achieving mean precision 0.85, recall 0.90, and F0.5=0.85.Few iterations favor precision, whereas more iterations expand coverage at the cost of precision.

F. Seed Consistency

On real images, geometry-aware conditioning produces more stable outputs across random seeds in geometrically constrained regions. This advantage narrows over the full mirror mask because unconstrained pixels require hallucination and admit multiple plausible reflections.

  • F. Seed Consistency: The proposed method outperforms baselines and produces substantially more stable outputs across seeds in geometrically constrained regions.The result suggests that geometric conditioning anchors outputs to physical scene constraints rather than random generation.
  • F. Seed Consistency: The variance gap is smaller over the full mirror mask because unconstrained pixels permit multiple equally plausible reflections.All methods, including the proposed method, must hallucinate content in these regions, adding cross-seed variance.
  • F. Seed Consistency: The qualitative comparison orders columns as GT, MirrorVerse, FLUX.1 Fill, and Ours, with the proposed method showing more geometrically consistent reflections.The other methods exhibit incorrect orientation or content even on examples selected by MirrorVerse authors.

G. More Qualitative Results

Additional qualitative comparisons examine real captured images and parameter choices for constrained mirror inpainting. The evaluations use identical prompts and seeds where stated, while safe-pool configurations assess constrained-region fidelity.

  • Qualitative comparisons: Real-image comparisons use identical prompts and seeds across methods, with results reported in table G1.The comparison includes real captured images without ground-truth geometry.
  • Parameter selection: Safe-pool configurations in Figure D2 retain constrained-region PSNR within 0.5 dB of the best ground-truth masked PSNR.Gray configurations fall outside this threshold, while colored configurations remain inside it.
  • Parameter selection: Earlier interpolation and sharper mask transitions increase generative freedom but can reduce adherence to geometric constraints.

H. Robustness to Imperfect Geometry

Noise interpolation is mildly harmful with exact geometry but becomes increasingly beneficial as projected geometry degrades. The robustness analysis evaluates this trade-off using PSNR and SSIM gaps on MirrorBench-V2.

  • Interpolation under degradation: When geometry is exact, noise interpolation is mildly harmful because it departs from an already-correct projection.
  • Interpolation under degradation: As projected geometry becomes less reliable, interpolation increasingly recovers from occlusion inconsistencies.
  • Interpolation under degradation: The analysis measures PSNR and SSIM gaps between versions with and without interpolation as the depth-degradation factor λ changes.The evaluation averages results over 20 MirrorBench-V2 scenes.

I. Additional Robustness: Jitter-Tolerant PSNR

Additional robustness analyses test jitter-tolerant evaluation, diffusion parameters, mask dilation, seed consistency, imperfect geometry, and qualitative comparisons. Across the supplied evidence, the method remains highest under jitter PSNR on both datasets, while parameter choices trade geometric adherence against generative freedom.

  • Jitter-tolerant PSNR: Jitter PSNR can tolerate narrow local misalignment but does not require neighboring pixels to share a coherent displacement.Each pixel independently selects its best local offset, so the metric does not verify physically plausible registration.
  • Jitter-tolerant PSNR: Our method remains highest under jitter PSNR on both datasets, with Qwen-2511 and MirrorFusion 2.0 tied on real images.Jitter PSNR independently searches a ±3-pixel neighborhood for each constrained ground-truth pixel.
  • Diffusion parameter robustness: The selected diffusion configuration is n=13, t′=625; lower t′ or n=1 leaves smearing, whereas n=13 and t′=1000 produces incorrect shadow reflections and sofa orientation.
  • Diffusion parameter robustness: At t′=1000, n≥5 fails to preserve the sofa reflection, and n≥9 additionally shrinks the mirror region.
  • RCS mask dilation: For the selected radius r=5, one dilation iteration is labeled “ours”; fewer iterations leave masks sparse, while more iterations over-expand them.
  • RCS mask dilation: At radius 5 and one iteration, the RCS mask achieves mean precision 0.72 (σ=0.16) and recall 0.88 (σ=0.20) across MirrorBench-V2 scenes.
Loading 2609.03740v1…