Source-linked AI summary
MEt3R: Measuring Multi-View Consistency in Generated Images
Mohammad Asim, Christopher Wewer, Thomas Wimmer, Bernt Schiele, Jan Eric Lenssen
TL;DR
Multi-view generators need consistency metrics that do not depend on paired ground truth, camera poses, or a specific scene, because existing measures can miss 3D inconsistencies. MEt3R uses DUSt3R-based reconstruction and DINO feature warping to measure pairwise consistency, and its evaluation finds that MV-LDM offers the best quality–consistency trade-off among evaluated methods.
Problem
Existing image-quality and reconstruction metrics are insufficient for evaluating 3D consistency in generated multi-view images, while reliable scene-independent evaluation is needed.
Method
MEt3R reconstructs dense point maps with DUSt3R, projects image features into a shared camera frame, and compares them with cosine similarity without requiring camera poses.
Results
MV-LDM achieves the best trade-off between 3D consistency and novel-view quality among evaluated methods, while DFM obtains better MEt3R consistency but blurrier renderings.
Takeaways & Limitations
MEt3R reliably compares multi-view generation methods and captures gradual consistency changes that existing metrics may miss.
Takeaways & Limitations
FVD requires collections of frames and is sensitive to sample size and blur, while MEt3R inherits DUSt3R’s pose-free stereo reconstruction design.
Abstract
from arXiv · showhide
We introduce MEt3R, a metric for multi-view consistency in generated images. Large-scale generative models for multi-view image generation are rapidly advancing the field of 3D inference from sparse observations. However, due to the nature of generative modeling, traditional reconstruction metrics are not suitable to measure the quality of generated outputs and metrics that are independent of the sampling procedure are desperately needed. In this work, we specifically address the aspect of consistency between generated multi-view images, which can be evaluated independently of the specific scene. Our approach uses DUSt3R to obtain dense 3D reconstructions from image pairs in a feed-forward manner, which are used to warp image contents from one view into the other. Then, feature maps of these images are compared to obtain a similarity score that is invariant to view-dependent effects. Using MEt3R, we evaluate the consistency of a large set of previous methods for novel view and video generation, including our open, multi-view latent diffusion model.
1. Introduction
Multi-view generative models need scene-independent measures of consistency because conventional image-quality and reconstruction metrics are inadequate without paired ground truth. MEt3R addresses this need with pose-free, gradual 3D-consistency evaluation and supports analysis of existing methods and MV-LDM.
- Generative models lack paired ground truth for individual samples, motivating metrics such as FID, KID, and CMMD for generated-image quality.
- Multi-view generators can scale from image and video data but have weak inductive bias toward 3D-consistent results needed for lifting images into 3D.
- MEt3R evaluates 3D consistency independently of scene and generating model, under changing lighting, without camera poses, differentiably, and on a gradual rather than binary scale.
- MEt3R uses DUSt3R reconstructions and feature comparisons to quantify consistency, while MV-LDM is introduced as an open multi-view latent diffusion model for evaluation.
- The contributions include a camera-pose-free consistency metric, analysis of multi-view and video-generation methods, and an open-source MV-LDM with the best quality-versus-consistency trade-off.
2. Related Work
Related work includes multi-view generation strategies and metrics for generated-image quality and 3D consistency. Existing approaches either do not measure 3D consistency or require costly reconstruction-based evaluation.
- Multi-view Generative Models: Multi-view generative models include pose-conditioned single-view methods, multiple-output-view methods, and methods using internal 3D representations to enforce consistency.
- Existing Metrics: Distribution-based metrics such as FID, KID, IS, and CMMD measure alignment with a target distribution but do not measure 3D consistency.
- Existing Metrics: A NeRF-based consistency metric trains on generated views and compares rendered novel views with held-out generated images.
- Existing Metrics: The NeRF-based approach requires many generated images, fails on sparsely observed scenes, is expensive, and is difficult to interpret because NeRF quality can affect dissimilarities.
3. MEt3R: Measuring Consistency
MEt3R measures pairwise multi-view consistency by reconstructing dense 3D point maps, warping image features into a shared camera frame, and comparing the resulting features. Its symmetric score is designed to assess consistency independently of image-quality or reconstruction targets.
- MEt3R is designed to penalize inconsistent image pairs without penalizing consistent pairs that differ from ground truth or a desired distribution.
- DUSt3R processes I1 and I2 into dense point maps, while DINO features upsampled by FeatUp are unprojected and rendered into I1’s camera frame.
- The projected feature maps are compared with pixel-wise cosine similarity, producing directional scores S(I1, I2) and S(I2, I1).
- MEt3R sums the two directional scores, is symmetric, lies in [0, 2], and can be approximated with one direction when runtime is constrained.
- DUSt3R regresses pixel-aligned 3D point clouds in I1’s camera space using shared features and cross-view attention, without requiring camera poses.
- Feature-space comparison reduces sensitivity to view-dependent effects such as lighting, while FeatUp preserves detailed structures through image-adaptive upsampling.
4. Multi-View Latent Diffusion Model
MV-LDM is an open-source multi-view latent diffusion model inspired by CAT3D and initialized from Stable Diffusion 2.1. It incorporates camera information and cross-view attention, then uses anchored generation to reduce autoregressive error accumulation.
- MV-LDM is an open-source model inspired by CAT3D and initialized with Stable Diffusion 2.1 because CAT3D is not publicly available.
- Architecture and Training: The model encodes images into Stable Diffusion’s latent space, concatenates ray maps for camera-pose information, and adds attention between views at each UNet block.
- Anchored Generation: Anchored generation first samples four widely distributed camera views conditioned on one input, then conditions remaining views on the nearest anchor and initial input.
- Anchored Generation: The anchoring strategy aims to prevent accumulating errors during autoregressive target-view generation.
5. Experiments
Experiments validate MEt3R against existing metrics and assess consistency across multi-view, video, and object-level generation. MEt3R distinguishes nuanced consistency differences, captures degradation and anchor-related artifacts, and reveals trade-offs between consistency and image quality.
- Experimental Setup: The experiments compare MEt3R with TSED, SED, FWS (PSNR), and FVD across multi-view, video, and object-level generation settings.The evaluation includes generated sequences, video models, and object-level 360° rotations.
- Validating MEt3R: MEt3R captures consistency differences that TSED and SED often miss, including increasing inconsistency for PhotoNVS, DFM, and MV-LDM.For GenWarp, MEt3R also captures decreasing consistency and periodic MV-LDM transition artifacts.
- Multi-View Generation: MV-LDM achieves the best trade-off between 3D consistency and novel-view quality, while DFM improves consistency but produces blurrier renderings.DFM’s stronger 3D consistency is reflected by better MEt3R scores, whereas its blurry outputs are penalized by FID.
- Metric Properties: MEt3R applies directly to generated videos without camera poses, unlike TSED and SED.This enables consistency measurement on video outputs in a manner similar to FWS (PSNR).
- Video Generation: SVD achieves the best MEt3R score among evaluated video models, followed by Ruyi-Mini-7B and I2VGen-XL.Ruyi-Mini-7B and I2VGen-XL generate larger camera motion at the expense of 3D consistency, while Ruyi-Mini-7B shows abrupt inconsistency spikes.
- Object-Level Generation: On object-level GSO evaluation, SyncDreamer achieves the best MEt3R score, followed by VideoMV and EpiDiff.MEt3R also captures slightly increasing inconsistency for EpiDiff as views move farther from the condition.
- Alternative Similarities: Feature-space similarity is preferred because it reduces sensitivity to view-dependent effects, while DINO features better separate consistent and inconsistent generations than DINOv2 or MaskCLIP.RGB metrics such as PSNR and SSIM are sensitive to blur, lighting, and reflections.
6. Conclusion
MEt3R measures 3D consistency in generated multi-view images by warping views into a shared space with DUSt3R and comparing DINO features. It enables comparison of multi-view generation methods without ground-truth camera poses, with MV-LDM achieving the best trade-off between novel-view quality and consistency.
- MEt3R warps images into a shared view using DUSt3R without requiring ground-truth camera poses.
- MEt3R computes DINO feature similarities to abstract from view-dependent effects.
- MEt3R effectively compares multi-view generation approaches, including the open-source MV-LDM.
- MV-LDM achieves the best trade-off between novel-view quality and consistency.
Supplementary Material
The supplementary materials provide detailed MV-LDM information and additional comparisons, metric discussion, baseline details, runtime statistics, and evaluation video.
- Section A provides detailed architectural, training, and sampling information for MV-LDM.
- The supplementary materials compare MEt3R with FWS variants and discuss the MEt3R metric.
- Additional sections describe multi-view generation baselines, runtime statistics, and a supplementary evaluation video.
A.1. Architecture.
MV-LDM uses a shared multi-view 2D UNet initialized from Stable Diffusion 2.1, with 3D self-attention connecting tokens across views.
- MV-LDM shares a 2D UNet across multiple input views and applies 3D self-attention at each UNet block.
- The UNet is initialized with Stable Diffusion 2.1 weights.
- Each token in one view attends to all tokens from the other views through the replaced attention layers.
- The multi-view UNet contains 1.1B parameters, while the VAE contains 83.7M parameters.
A.2. Training and Evaluation with MEt3R
MV-LDM is trained on RealEstate10K using sampled conditioning and target views, then evaluated with MEt3R across training progress, sampling strategies, and image resolutions.
- Dataset: RealEstate10K training samples two conditioning views separated by 50–180 frames and three target views around them.
- Training: MV-LDM uses DDPM training with one or two conditioning views and three or four target views, for 1.65M iterations.
- Training: Classifier-free guidance drops clean conditioning views with 10% probability and uses guidance scale 3 during sampling.
- Training evolution: MEt3R improves substantially during the initial 100k iterations and saturates near 1M iterations.
- Anchored vs. autoregressive sampling: Anchored sampling improves per-image-pair MEt3R by limiting error accumulation and reducing anchor-to-anchor transitions.
- Multiple scales: MEt3R is more robust to image-resolution changes than SED because it measures similarity in feature space.
- Anchored vs. autoregressive sampling: Autoregressive sampling produces periodic MEt3R spikes that grow as compounding errors accumulate.
- Multiple scales: Similar image resolutions are recommended across baselines for fair comparison.
B. Comparison of FWS Variants
The paper compares flow-warping score variants with MEt3R for evaluating consistency in multi-view and video generation. Feature- and perceptual-similarity metrics avoid some pixel-level sensitivities, but FWS (LPIPS) becomes unreliable at larger frame distances.
- FWS variants: FWS computes optical flow between image pairs before evaluating warped-image similarity with PSNR, SSIM, LPIPS, or RMSE.These variants use RAFT optical flow and standard image or perceptual metrics.
- FWS variants: PSNR, SSIM, and RMSE rank DFM above real video among multi-view methods because they are sensitive to blur, pixel perturbations, and noise.The reported ranking conflicts with the intended consistency assessment because DFM’s outputs can be blurry despite its 3D-consistency design.
- FWS variants: MEt3R is more robust to input-resolution changes than SED because it measures similarity in feature space rather than pixel space.The paper reports that SED variations are more than an order of magnitude larger in most cases.
- FWS variants: MEt3R and FWS (LPIPS) ignore blur, pixel-level perturbations, and noise by relying on feature or perceptual similarity.MEt3R operates in feature space, whereas FWS (LPIPS) uses perceptual similarity after flow warping.
- FWS variants: At larger frame distances, FWS (LPIPS) can score DFM, GenWarp, and MV-LDM better than real videos.Figure 14 specifically reports this failure for a frame distance of 10.
C. Additional MEt3R Architectural Details
The additional details describe how MEt3R projects DUSt3R reconstructions and normalizes feature comparisons over visible overlap. The implementation adjusts projection parameters and uses an overlap mask to avoid misleading consistency scores.
- Projection and masking: MEt3R uses DUSt3R point maps to project both images into a shared canonical view before comparing their features.The pipeline includes projection of both point maps to the first view and an overlap-mask computation.
- Projection and masking: The canonical point map is a confidence-weighted combination of the two DUSt3R point maps, from which coordinate maps and projection parameters are computed.The resulting fx, fy, cx, and cy form the projection matrix.
- Projection and masking: The overlap mask is a crucial normalization step that restricts MEt3R to pixels jointly covered by the two projected images.Rasterization assigns large negative background values, which are then used to construct the mask.
- Projection and masking: MEt3R automatically adjusts focal length according to the canonical point-map scale to maximize the projected evaluation area.This adjustment provides more pixels for feature-similarity evaluation than using a fixed focal length.
- Projection and masking: Without overlap-mask normalization, the lower bound shifts upward and DFM becomes worse than every other baseline, contradicting theoretical and visual consistency judgments.PhotoNVS becomes nearly similar to GenWarp under this altered calculation.
- Baseline details: GenWarp projects monocular-depth estimates into 3D and inpaints target views, generating one novel view at a time.It conditions each novel view on the fixed input because autoregressive sampling diverges quickly from accumulated errors.
- Baseline details: PhotoNVS uses cross-view attention and conditions autoregressively on the last generated frame to improve consistency across a sequence.It also generates a single view at a time from a conditioning image.
- Baseline details: DFM incorporates a neural radiance field and pixelNeRF so novel views are designed to be 3D consistent.Each extrapolated target view becomes additional conditioning for subsequent sampling steps.
E. Runtime
The runtime comparison covers generation of 80-frame sequences and evaluates the cost of MEt3R itself. MV-LDM and GenWarp are fastest among the baselines, while DFM and PhotoNVS are substantially slower.
- Runtime comparison: Runtime measurements compare methods generating 80-frame video sequences on an NVIDIA RTX4090 with 24GB VRAM.The comparison reports full-sequence sampling times.
- Runtime comparison: MV-LDM and GenWarp achieve the fastest sampling, followed by DFM and then PhotoNVS.This ordering is stated in the runtime comparison table.
- Runtime comparison: GenWarp is fastest because high-quality but inconsistent novel views can be produced with 20 DDIM steps.Its per-view generation strategy prioritizes speed despite the reported inconsistency.
- Runtime comparison: MV-LDM generates multiple views at once, uses 70 DDIM steps for good image quality, and remains only slightly slower than GenWarp.The passage links its multi-view generation strategy with improved 3D consistency.
- Runtime comparison: DFM and PhotoNVS are an order of magnitude slower because of volumetric NeRF rendering and many denoising steps, respectively.The stated causes differ by method: rendering for DFM and denoising for PhotoNVS.
- Runtime comparison: MEt3R evaluates one image pair in 95ms.This is the reported metric-evaluation runtime, separate from sequence-generation time.