PaperlayerView exact revision on arXiv

Source-linked AI summary

MirrorWorld: Taming Video Diffusion Models for Mirror Reflection Generation

Youjun Zhao, Alex Warren, Gary K. L. Tam, Rynson W. H. Lau

arXiv:2608.07463v1cs.CVcs.LG

TL;DR

Mirror reflection generation requires scene-consistent content and spatial arrangements, which existing video diffusion models do not specifically model. MirrorWorld addresses this with reflection-aware video inpainting using semantic relation distillation and geometric transformation alignment, improving reconstruction quality over representative image-based and video inpainting methods.

  • Problem

    Existing video diffusion models lack explicit modeling of scene-to-mirror relationships needed for reflection-consistent video generation.

  • Method

    MirrorWorld uses reflection-aware video inpainting with Semantic Relation Distillation for reflected content and Geometric Transformation Alignment for its spatial arrangement.

  • Results

    MirrorWorld improves reflection reconstruction and video-level quality over representative image-based reflection generation and video inpainting methods on a unified benchmark.

  • Takeaways & Limitations

    The unified benchmark supports systematic evaluation of reflection reconstruction across diverse scenes, object configurations, camera motions, and mirror appearances.

  • Takeaways & Limitations

    When reflected content lies entirely outside the camera’s view, MirrorWorld cannot determine its exact identity or spatial configuration from observed evidence.

Abstract

from arXiv · show

Recent advances in video diffusion models (VDMs) have enabled high-fidelity video synthesis. However, generating mirror reflections remains challenging because the content within a mirror must remain consistent with the surrounding scene. Existing VDMs are not specifically designed to model scene-to-mirror relationships, which can lead to reflections with incorrect content or inconsistent spatial arrangements. We observe that mirror reflection generation involves two complementary challenges: determining what scene content should be reflected and how the reflected content should be spatially arranged within the mirror region. Motivated by this observation, we propose MirrorWorld, a reflection-aware video inpainting framework that models scene-to-mirror relationships during generation. Specifically, we introduce Semantic Relation Distillation (SRD), which transfers relational information from a frozen visual foundation model to encourage semantic associations between visible scene content and mirror regions. We further propose Geometric Transformation Alignment (GTA), which learns a transformation that guides the spatial arrangement of reflected content. The two components play complementary roles, with SRD modeling what should be reflected and GTA modeling how it should be arranged. To facilitate research on this problem, we construct a benchmark for video mirror reflection generation by repurposing four existing video mirror datasets into a unified reflection reconstruction task. Experimental results show that MirrorWorld achieves improved reflection reconstruction quality over representative image-based reflection generation methods and strong video inpainting baselines.

Introduction

MirrorWorld addresses video mirror reflection generation as a reflection-aware reconstruction problem, motivated by advanced VDMs producing realistic scenes but inconsistent reflections. It separates the task into determining what content to reflect and how to arrange it spatially, using SRD and GTA within a unified benchmark.

  • Motivation: Advanced VDMs can synthesize realistic videos, yet mirror reflections may contain incorrect objects, implausible spatial arrangements, and temporal inconsistencies.Veo 3.1 is cited as producing visually realistic scenes while failing to maintain consistency between reflected content and the visible environment.
  • Problem formulation: Video mirror reflection generation requires determining what scene content should be reflected and how it should be spatially arranged within mirror regions.Unlike conventional video inpainting, mirror content is constrained by reflection-relevant scene content and must remain consistently arranged in the mirror.
  • Method: MirrorWorld is a reflection-aware video inpainting framework that reconstructs masked mirror regions while preserving non-mirror content by aligning scene and mirror-region features.The framework explicitly models scene-to-mirror relationships during reflection reconstruction.
  • Method: Semantic Relation Distillation (SRD) transfers relational information from a frozen visual foundation model, while Geometric Transformation Alignment (GTA) learns spatial transformations for reflected content.SRD addresses what should be reflected, whereas GTA addresses how it should be arranged.
  • Evaluation: The authors repurpose four existing video mirror datasets into a unified reflection reconstruction benchmark and report improved quality over representative image-based methods and strong video inpainting baselines.The benchmark covers diverse scenes, object configurations, camera motions, and mirror appearances for systematic evaluation.

Related Work · Method

MirrorWorld extends video diffusion and inpainting for scene-level mirror reflection reconstruction. Its method combines semantic relation distillation for what appears in the mirror with geometric transformation alignment for how it is arranged, trained on a unified benchmark.

  • Video Diffusion Models: Video diffusion models jointly model spatial appearance and temporal dynamics, while later methods improve video quality and scalability with motion modules and diffusion transformers.
  • Video Inpainting Methods: Video inpainting reconstructs missing regions from visible frames and text prompts using temporal information and, increasingly, video diffusion priors.
  • Mirror Understanding and Reflection Generation: Prior mirror research emphasizes detection and segmentation, while reflection generation remains limited to image-based and object-centric settings rather than video scene-level reconstruction.
  • Overview: Given a masked video, mirror mask, and text prompt, MirrorWorld reconstructs reflection-consistent mirror content while preserving visible non-mirror regions.
  • Semantic Relation Distillation (SRD): Semantic Relation Distillation transfers relational knowledge from a frozen visual foundation model to establish semantic correspondences between mirror regions and visible reflection-relevant content.SRD aligns scene-to-mirror relations between reference and diffusion features, with gradients propagated only through the diffusion representation.
  • Geometric Transformation Alignment (GTA): Geometric Transformation Alignment learns a feature-space mapping from visible scene content to reflected regions, using source maps that exclude direct access to mirror-region target features.It aggregates a local temporal window of K = 2r+1 feature steps and predicts an affine transformation initialized from the identity mapping.
  • Training Objective: The final training objective combines the base diffusion generation loss with SRD and GTA, whose weights control the strength of the two auxiliary objectives.SRD learns scene content associated with the mirror, while GTA learns its spatial transformation within the reflection.
  • Benchmark: MirrorWorld builds a unified benchmark from four existing video mirror segmentation and detection datasets, covering diverse scenes, camera motions, and mirror appearances.Clipping yields 1,242 video clips: 1,142 for training and 100 for testing, with source-video-level splitting.

Experiments

MirrorWorld is evaluated against image-based reflection methods and video inpainting baselines using mirror-region reconstruction and video-level metrics. Results and ablations show that semantic and geometric supervision jointly improve reflection consistency, while performance depends on the transformation strategy, temporal window, and visibility of reflected content.

  • Baseline Comparison: MirrorWorld is compared with MirrorFusion, MirrorVerse, VideoPainter, and VACE, with all baselines fine-tuned on the same training set.The image-based methods are applied independently to each video frame, while VACE uses Wan2.1-14B as its base model.
  • Quantitative Evaluation: MirrorWorld achieves the strongest overall performance across mirror-region reconstruction and video-level evaluation.PSNR, SSIM, and LPIPS are computed within mirror regions, while FVD measures video-level generation quality.
  • Quantitative Evaluation: Compared with VACE, MirrorWorld better preserves pixel values, structural appearance, perceptual features, and similarity to real-video distributions.The results support adding reflection-specific semantic and geometric supervision beyond conventional video inpainting.
  • Qualitative Evaluation: Image-based methods produce unrelated or structurally inconsistent reflections that change across frames, while VideoPainter and VACE retain scene or local coherence but misalign reflected content and spatial boundaries.VACE duplicates visible objects and generates clothing reflections where the observed scene indicates none should appear; MirrorWorld avoids these errors.
  • Ablation Studies: Using SRD or GTA alone yields limited reconstruction improvements, whereas their complementary roles model reflected content identity and its geometric organization.Without SRD, incorrect objects appear inside the mirror; without GTA, a partially visible metal ring is copied into an incomplete reflection.
  • Ablation Studies: VideoMAEv2 outperforms DINOv3 on all three mirror-region reconstruction metrics, while DINOv3 obtains slightly lower FVD without improving reconstruction quality.The geometric transformation strategy also outperforms horizontal feature flipping, and GTA performs best with a local window of K = 5.

Conclusion

MirrorWorld is a reflection-aware video inpainting framework that models relationships between visible scene content and mirror regions. It separates reflection generation into deciding what content to reflect and how to spatially arrange it, using SRD for semantic relations and GTA for geometric alignment.

  • MirrorWorld is a reflection-aware video inpainting framework modeling relationships between visible scene content and mirror regions.
  • Reflection generation is decomposed into two complementary questions: what content should be reflected and how it should be spatially arranged.
  • SRD transfers semantic relations from a frozen visual foundation model to guide relationships between visible content and mirror regions.
  • GTA aligns visible scene features with mirror-region representations through learned geometric transformations.

Text Prompt

MirrorWorld uses one task-level text prompt for every video clip and baseline, requiring reflection content to be inferred from the visible scene and mirror mask. The prompt specifies natural completion, contextual consistency, correct structure, realistic details, and temporally coherent motion.

  • Text Prompt: A unified task-level prompt is applied to all video clips and baseline methods for consistent evaluation.This prevents prompt-design differences from affecting comparisons.
  • Text Prompt: The prompt avoids object-specific descriptions, so mirror content must be inferred from the visible scene and mirror mask.It does not provide explicit semantic cues about the reflected objects.
  • Text Prompt: The prompt asks for naturally completed mirror regions with reflections consistent with their surroundings, correct structure, and realistic details.These requirements define the desired spatial and visual properties of reconstructed reflections.
  • Text Prompt: The prompt additionally requires temporally coherent motion in the generated reflections.This specifies the temporal behavior expected during video generation.

Temporal Consistency Evaluation

Temporal consistency is evaluated with flow warping error Ewarp computed only within valid mirror pixels, where lower values indicate stronger motion-compensated consistency. MirrorWorld achieves the lowest error, while VACE shows that temporal stability alone does not guarantee correct reflection content or placement.

  • Metric: Ewarp measures residual differences between adjacent generated frames after flow-based warping, averaged over valid mirror pixels while excluding non-mirror regions and invalid correspondences.Lower Ewarp indicates stronger motion-compensated temporal consistency in generated reflections.
  • Results: 0.192 and 0.188 Ewarp: MirrorFusion and MirrorVerse produce the largest warping errors among the evaluated methods.Their frame-by-frame application is associated with substantial changes in generated objects, people, dark regions, and unrelated scene structures across frames.
  • Results: 0.025 Ewarp: MirrorWorld achieves the lowest flow warping error among the evaluated methods, indicating more temporally consistent reflected content after motion compensation.GTA uses local temporal context to provide stable geometric guidance across neighboring frames.
  • Results: 0.026 Ewarp: VACE also obtains a low warping error despite producing duplicated or incorrectly placed reflections.This shows that low temporal warping error can coexist with reflection content or spatial-placement errors.
  • Results: 0.076 Ewarp: VideoPainter performs better than the image-based methods but worse than VACE and MirrorWorld in temporal consistency.The comparison is reported for mirror regions in Table A.

Additional Ablation Study

The ablations show that intermediate auxiliary-loss weights and separate token usage improve mirror-region reconstruction. Reconstruction-focused metrics can favor different choices from FVD, which measures full-frame video distributions.

  • Loss-weight ablation: λSRD = 0.05 and λGTA = 0.01 produce the strongest mirror-region reconstruction across PSNR, SSIM, and LPIPS.Smaller or larger weights reduce reflected-content accuracy, indicating that intermediate strengths are preferable for both auxiliary objectives.
  • Metric and component ablation: Although GTA alone achieves the lowest FVD, combining SRD and GTA yields the strongest mirror-region reconstruction results.FVD evaluates complete-frame distributions, whereas PSNR, SSIM, and LPIPS target reconstructed mirror regions.
  • Token-sharing ablation: Sharing SRD’s sampled token locations with GTA slightly improves LPIPS and preserves SSIM but lowers PSNR and substantially increases FVD.The results suggest sparse semantic token pairs lack sufficient spatial coverage for geometric transformation learning, whereas GTA benefits from all tokens.
  • Token-sampling ablation: Random SRD token sampling outperforms Top-N sampling on all mirror-region reconstruction metrics, while Top-N achieves lower FVD.Random sampling uses up to 64 tokens per region, whereas Top-N uses N = 16; broader relation coverage better supports complex reflections.

Benchmark Statistics

The benchmark unifies four video mirror datasets into a reflection reconstruction resource of 1,242 clips and 45,373 frames, with source-level splits preventing leakage and original clips providing reflection targets.

  • Dataset composition: The benchmark integrates VMD-D, DVMD-D, MMD, and ZOOM into a unified reflection reconstruction setting, totaling 1,242 video clips and 45,373 frames.The videos span a wide range of spatial resolutions and form a larger, more diverse resource for training and system evaluation.
  • Dataset composition: Split assignments use original source-video identities, ensuring clips from one source video belong exclusively to either training or testing.This prevents content leakage from placing clips derived from the same video in different partitions.
  • Dataset composition: Original unmasked clips provide ground-truth reflections, while mirror regions are masked only in conditional inputs.The original clips remain the supervision and evaluation targets.

Additional Implementation Details

MirrorWorld preserves video aspect ratios, adapts a frozen backbone with rank-32 LoRA, and uses separate 256-dimensional projections for SRD and GTA. Baselines are fine-tuned under matched benchmark conditions while retaining their original architectures and protocols where possible.

  • MirrorWorld training: Training preserves each video’s original aspect ratio while limiting frame resolution to 399,360 pixels.The backbone uses rank-32 LoRA on transformer projections, while pretrained backbone parameters remain frozen.
  • MirrorWorld training: SRD and GTA operate on shared diffusion features that are separately projected to 256 dimensions for their objectives.This implementation provides distinct feature projections for the two training objectives.
  • Baseline evaluation: All baselines use officially released implementations, matched epoch counts, the same training split, and identical evaluation inputs and protocols.Original architectures and settings are preserved except for minimal modifications needed for the unified benchmark.
  • Baseline-specific settings: MirrorFusion and MirrorVerse process frames at 512 × 512 with Marigold-estimated depth, use learning rate 10^-5 and global batch size 16, and update different parameter subsets.MirrorFusion updates only BrushNet, whereas MirrorVerse jointly updates BrushNet and the U-Net.
  • Baseline-specific settings: VideoPainter fine-tunes only the context branch on clips up to 49 frames at 480×720, using 50 denoising steps and guidance scale 6.0 at inference.Its first frame is generated with FLUX-Fill from the masked input without ground-truth video access.
  • Baseline-specific settings: VACE fine-tunes Wan2.1-VACE-14B with rank-32 LoRA and learning rate 10^-4, using 50 denoising steps, guidance scale 5.0, and conditioning scale 1.0.Training uses AdamW.

Metric Definitions

The evaluation uses five metrics covering mirror-region reconstruction, overall video quality, and temporal consistency. PSNR, SSIM, LPIPS, and flow warping error are computed within mirror regions, while FVD uses complete frames with a fixed 10-frame temporal prefix.

  • Evaluation scope: Flow warping error Ewarp is evaluated within mirror regions, alongside PSNR, SSIM, and LPIPS in quantitative comparisons and ablations.These metrics are restricted to annotated mirror regions, whereas FVD is not.
  • Mirror-region metrics: PSNR measures pixel-level reconstruction accuracy using mean squared error computed only inside the mirror mask, with higher values indicating more accurate reflected content.PSNR is reported for mirror-region reconstruction.
  • Mirror-region metrics: SSIM evaluates structural similarity between generated and ground-truth reflections by averaging the standard full-frame SSIM map over the mirror mask.Local windows extending beyond the mirror boundary are excluded, and higher SSIM indicates better reflection-structure preservation.
  • Mirror-region metrics: LPIPS measures perceptual differences with deep visual features and averages spatial distances only within the resized mirror mask.The implementation uses an AlexNet backbone, and lower LPIPS indicates greater perceptual similarity to the ground truth.
  • Video-level metrics: FVD measures distributional discrepancy over complete spatial frames without mirror masks, using a fixed 10-frame prefix resized to 224×224 for every video.The fixed prefix standardizes temporal support despite benchmark videos ranging from 10 to 49 frames; FVD is computed from I3D features.

Flow Warping Error (Ewarp)

Ewarp measures temporal consistency after motion compensation by computing photometric warping error within valid mirror-region pixels. Lower Ewarp indicates more temporally consistent reflections.

  • Metric definition: Ewarp evaluates temporal consistency after motion compensation using photometric warping error.Optical flow is estimated between adjacent ground-truth frames with RAFT-Large and used to warp corresponding generated frames.
  • Metric computation: The photometric warping error is computed only over valid mirror-region pixels.Out-of-bound and forward–backward inconsistent locations are excluded.
  • Interpretation: Lower Ewarp indicates more temporally consistent reflections.

Additional Qualitative Results

Additional qualitative comparisons show that MirrorWorld produces semantically related reflections with stable spatial organization across frames, outperforming image-based methods and video inpainting baselines in representative examples. Its reconstructions remain consistent across changing viewpoints, mirror geometries, scene layouts, and foreground motion.

  • MirrorFusion and MirrorVerse generate unrelated content that changes substantially across frames, while VideoPainter improves temporal coherence but remains weakly related to the visible scene.
  • MirrorWorld preserves both the dispenser and electrical outlet reflections and maintains their spatial relationship across frames, unlike VACE, which misses the outlet reflection.
  • Across Figure E, MirrorWorld maintains more coherent reflected content and spatial organization than methods producing unrelated interiors or inconsistent structures near the mirror boundary.
  • MirrorWorld keeps doorways, wall boundaries, sofa, and carpet spatially consistent within reflections as viewpoints change across sampled frames.
  • MirrorWorld adapts reflections to polygonal and oval mirror boundaries while preserving room structure and following changes in a person’s pose and position.
Loading 2608.07463v1…