PaperlayerView exact revision on arXiv

Source-linked AI summary

Texture Consolidation of Generated Multi-View Renderings

Hongyi Liu, Yifan Wang, Vladimir G. Kim, Misha Kazhdan

arXiv:2610.36098v1cs.GR

TL;DR

Generated multi-view renderings can be inconsistent when mapped onto geometry, complicating the creation of a coherent texture. The paper uses confidence-aware back-projection, surface registration, and gradient-domain stitching; it reports high-quality textures on sparse generated inputs, while differing layouts in repeating patterns can still cause localized blur.

  • Problem

    Generated views can be geometrically inconsistent and sparse, making it difficult to consolidate them into a coherent texture.

    Source: §1 · pp. 1–2

  • Method

    The method back-projects renderings into texture images and performs confidence-aware surface registration and gradient-domain stitching.

    Source: §1 · p. 2

  • Results

    Existing RGB-D registration methods degrade on sparse generated inputs, while the framework produces high-quality texture images in that setting.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method is agnostic to the rendering pipeline and serves as a post-processing module that complements prior multi-view generation methods.

    Source: §1 · p. 2

    Different layouts of repeating patterns can merge with localized blurring, even when the individual stripes remain sharp.

    Source: §8 · p. 12

Abstract

from arXiv · show

We present a system for consolidating a sparse set of multi-view renderings into a single coherent texture image, providing a framework for integrating generative models for image generation within the geometry-processing pipeline. Given a texture-atlased triangle mesh and a set of renderings with associated camera parameters, we generate the texture image by: (1) back-projecting the renderings onto the mesh to obtain texture images with normalized colors and confidences; (2) solving for flow-fields on the mesh that align the colors to each other, taking into account the associated confidences; and (3) fusing the registered colors into a single texture image using gradient-domain stitching, again incorporating confidences. Our approach complements existing techniques for multi-view image generation, leveraging mesh-based optical flow and texture-space stitching. In doing so it addresses the challenge that while modern image generation pipelines generate highly convincing renderings, they struggle to produce renderings that are consistent across views. Code is available at https://github.com/hliuw/TextureConsolidation.

1. Introduction

Generated multi-view renderings can contain geometric and photometric inconsistencies, making a coherent view-independent texture difficult to construct. The paper addresses this with confidence-aware surface registration and stitching, producing high-quality textures in the sparse-view setting.

1. Introduction

  • Generated multi-view renderings can be convincing individually yet inconsistent when mapped onto geometry, complicating integration into one view-independent model.Source: §1 · p. 1
  • Generated views can introduce additional inconsistency, making view augmentation a trade-off and motivating robust fusion from a small number of renderings.Each surface point is visible in only a few views, and generated views may be geometrically inconsistent.Source: §1 · p. 2
  • The method back-projects renderings to texture images, then performs confidence-aware registration and stitching in the surface domain.Quadrature-based finite-elements integration supports sub-triangle function evaluation and bridges texel- and mesh-resolution discretizations.Source: §1 · p. 2
  • Existing registration methods for RGB-D captures degrade on sparse generated inputs, while the proposed framework produces high-quality texture images in this setting.The method is agnostic to the rendering pipeline and complements prior multi-view generation methods.Source: §1 · p. 2

2. Related Work

Prior work generates textured 3D shapes through several representations and pipelines, while texture registration and blending address inconsistencies among views. This work focuses on registering and compositing generated renderings using surface-domain optical flow and stitching.

2. Related Work

  • Textured 3D shape methods use NeRF or Gaussian-Splat representations, or back-project generated renderings onto input geometry and aggregate colors with progressive inpainting.Some approaches use hybrid representations or extract meshes from NeRF or Gaussian-Splat methods.Source: §2 · p. 2
  • Multi-view texture inconsistencies arise from lighting, occlusions, and geometric distortions, while generated textures can add independently varying details.Source: §2 · p. 2
  • Prior registration methods correct misalignment through optical flow, pose optimization, learned tolerance, texture-detail selection, or exclusion and filling of misaligned regions.Optical-flow examples include Eisemann et al. [EDDM∗08], Casas et al. [CVCH14], and Dellepiane et al. [DMC∗12].Source: §2 · p. 2
  • View blending combines selection and stitching, with methods ranging from per-face source assignment to all-camera blending and generated-view selection.Generated-view approaches include SIFT-flow warping and joint labeling [ZSG∗24], and geodesic-aware neural fusion [GLAK25].Source: §2 · p. 2
  • This work registers textures with optical flow and composites them with gradient-domain stitching, performing both operations over the surface.Earlier methods often perform view-dependent processing followed by a final blending step.Source: §2 · pp. 2–3
  • Surface-domain optical flow uses tangent-vector-field smoothness and variational formulations to align details and reduce ghosting; gradient-domain stitching composites gradients for seamless results.Texture-domain stitching also addresses cross-chart discontinuities and non-uniform texture-mapping distortion [PKCH18].Source: §2 · p. 3

3. Overview

Given a texture-atlased mesh and sparse multi-view renderings, the pipeline produces a globally consistent texture through preprocessing, surface-domain registration, and confidence-aware gradient-domain stitching. Combining registration and stitching addresses both ghosting and color discontinuities.

3. Overview

  • The input is a texture-atlased manifold triangle mesh with sparse renderings and corresponding camera parameters, and the goal is a globally consistent texture image.Source: §3 · p. 3
  • Preprocessing back-projects each rendering onto the surface, assigns per-texel confidence from visibility, depth, and viewing angle, and normalizes colors.Source: §3 · p. 3
  • Registration computes a surface-domain optical flow field for each color and confidence image, then advects texel values along the fields.Source: §3 · p. 3
  • Stitching fuses the flow-aligned color and confidence images using confidence-aware gradient-domain stitching.Source: §3 · p. 3
  • Confidence-weighted blending alone leaves ghosting, optical flow aligns color features but leaves discontinuities, and stitching yields a seamless, ghost-free texture.Source: §3 · p. 3

4. Preliminaries

The preliminaries formulate gradient-domain processing and optical flow over a mesh using queryable functions and quadrature-based integration. Stitching fits composite gradients, while optical flow estimates and iteratively refines a tangent field that advects signals toward alignment.

4. Preliminaries

  • Queryability means mesh functions are bounded and evaluable almost anywhere, allowing them to be integrated over the mesh.Source: §4 · p. 3

4.1. Gradient-Domain Processing

  • Gradient-domain processing seeks a signal that fits both target values and target gradients; stitching emphasizes fitting the composite input gradients.In smoothing, the target gradient is zero, and the gradient-fitting weight controls the extent of smoothing.Source: §4.1 · p. 3
  • Quadrature-based integration lets the gradient-domain linear system be defined from queryable basis functions, derivatives, and constraints.Source: §4.1 · p. 3

4.2. Optical Flow

  • Optical flow computes a tangent vector field that advects a source signal along the mesh to align it with a target.Its energy measures the difference between the approximately advected source and target.Source: §4.2 · p. 3
  • Figure 2 traces back-projection, color normalization, confidence-aware flow registration, advection, and final gradient-domain stitching.Figure 3 shows that confidence blending ghosts, flow alignment leaves color discontinuities, and stitching produces a seamless texture.Source: Figure 2 · p. 4, Figure 3 · p. 4
  • The flow regularizer encourages smoother fields, and the optimization iterates by advecting the source with prior estimates and computing a correcting field.Source: §4.2 · p. 4
  • Hierarchical flow estimation aligns low frequencies first, then progressively addresses higher frequencies by reducing blur and smoothness regularization.Source: §4.2 · p. 4
  • Signal advection steps backward along the flow, using the hinge-map to flatten adjacent triangles when crossing an edge before sampling.Source: §4.2 · p. 4

5. Quadrature-based Integration

Quadrature-based integration bridges mesh-resolution flow fields with texel-resolution colors and confidences. It supports confidence-weighted integration of both scalar functions and scalar gradients.

5. Quadrature-based Integration

  • Quadrature-based integration mixes mesh- and texel-resolution representations while incorporating confidence into the discretization.Scalar functions use mesh-resolution hat and texel-resolution bases; vector fields use mesh-resolution basis functions.Source: §5 · p. 5

5.1. Integrating Scalar Functions

  • For scalar functions, quadrature approximates confidence-weighted integrals using queryable target and confidence functions.The approach also supports confidence-weighted mass and stiffness matrices when basis-function derivatives are queryable.Source: §5.1 · p. 5

5.2. Integrating Gradients of Scalar Functions

  • For scalar gradients, quadrature approximates the confidence-weighted pairing of a tangent vector-field basis function with a function’s gradient.This requires queryable function, gradient, and confidence values; similar integration defines mass and stiffness matrices when basis derivatives are queryable.Source: §5.2 · p. 5

6. Method

The method adapts optical-flow and gradient-domain energies to use confidence, then preprocesses renderings into texture color and confidence images. Confidence-weighted stitching lets the weights determine which values and gradients contribute across the domain.

6. Method

  • The gradient-domain energy includes a small regularization term to keep the system nonsingular where confidence is zero.The regularizer encourages smoothness only in regions without confidence.Source: §6 · p. 5
  • Confidence-weighted gradient-domain stitching lets per-target confidences control which values and gradients are active across the domain.This formulation avoids first compositing colors or gradients into a single input function or vector field.Source: §6 · p. 5
  • The method adapts optical-flow energies to confidence and defines gradient-domain energies from target values, gradients, and confidences.It also describes confidence-weighted blurring and normalized, rescaled weights whose sum indicates whether a point is covered by a target.Source: §6 · p. 5

6.1. Preprocessing

  • Preprocessing back-projects renderings into texture color and confidence images, then normalizes color to correct low-frequency variation.The method also adapts the construction of linear systems to incorporate confidence.Source: §6.1 · p. 6, §6 · p. 6

Back-Projection

Back-projection samples each rendering at the camera-projected location of mesh points represented by texture texels. Confidence combines view alignment and depth information to reduce the influence of silhouette-related misprojections.

Back-Projection

  • For each texel, back-projection finds its mesh point from the closest texture triangle and samples the rendering at that point’s camera-projected position.Barycentric coordinates determine the surface point; points outside a texture triangle are mapped to the closest mesh point.Source: Back-Projection · p. 6
  • Confidence combines surface-normal alignment and depth discontinuities to downweight colors that may land incorrectly near silhouettes.Depth confidence is reduced near discontinuities and set to zero for occluded texels; normal and depth confidences are multiplied, with low values quantized to zero.Source: Back-Projection · p. 6
  • The confidence scheme can also accommodate rendering-quality cues such as blurriness, though preprocessing mainly supplies color and confidence images for downstream processing.The described implementation uses normal/view-direction alignment and depth continuity.Source: Back-Projection · p. 6

Color Normalization

The pipeline normalizes back-projected colors before registration, incrementally aligns and merges texture signals, then stitches them into a seamless texture. It uses different discretizations for flow fields and texel-level colors and confidences.

Color Normalization

  • Normalization estimates local color mean and scale, reducing low-frequency variation that can destabilize optical flow.The passage describes overlap colors becoming more consistent after normalization.Source: Color Normalization · pp. 6–7

6.2. Registration

  • Instead of jointly optimizing all flow fields, successive registration selects an initial target and incrementally registers remaining signals to it.The target is evolved by blending advected source colors and confidences; the initial target and subsequent sources are selected by confidence overlap.Source: §6.2 · p. 7

6.3. Stitching

  • Gradient-domain stitching combines the advected color and confidence images into a seamless texture using the confidence-aware adaptation of classical stitching.Source: §6.3 · p. 7

6.4. Process Summary

  • The process back-projects and normalizes inputs, refines flow fields hierarchically, registers textures, and stitches the output.Flow fields are represented at mesh resolution while colors and confidences are represented at texel resolution; the stated quadrature counts are 12 for optical flow and 3 for stitching.Source: §6.4 · pp. 7–8

6.2. Registration

  • Each registration step computes a correcting flow between smoothed source and target colors, then advects and confidence-blends the source into the target.Hierarchical iterations refine previously estimated flow fields and reduce smoothness weights to register finer detail.Source: §6.4 · p. 7, §6.2 · p. 7

7. Evaluation

The evaluation spans generated renderings and RGB-D scans, comparing texture quality, robustness, and runtime. The method ranks best on LPIPS across generated datasets and handles sparse-view misalignment, while robustness and runtime vary with the distortion and input size.

7. Evaluation

  • Evaluation uses Instant3D renderings reconstructed with MeshLRM, SyncDreamer views paired with GSO meshes, and RGB-D scans; SyncDreamer also uses a leave-four-out test from 12 of 16 views.Instant3D uses nine views; the RGB-D experiments use Fountain keyframes and a subsample of Armchair scans.Source: §7 · p. 9, §7.2 · p. 10

7.1. Our Method: Generated Data

  • Seamless, ghost-free results require combining optical-flow registration with gradient-domain stitching; the latter also smoothly fills regions unseen by any camera.Additional results show alignment of photometric features and removal of discontinuities.Source: §7.1 · p. 10

7.2. Comparisons: Sparse-view Generated Data

  • MVS-tex shows patch-boundary seams, ColorMap and AdvIA degrade on sparse data, and Im2SurfTex produces blurry, ghosted textures; our method resolves misalignment and merges back-projections seamlessly.The method comparisons also report that MVS-tex tends toward higher contrast, AdvIA can shift colors, and Im2SurfTex can darken renderings.Source: §7.2 · p. 10, Figure 8 · p. 11
  • MeshLRM’s view-dependent NeRF reproduces input renderings but shows artifacts in frontal views absent from its input set.Examples include misaligned flag stripes and ghosting on the Pharaoh’s headpiece.Source: §7.2 · p. 10

7.3. Comparison: Dense-view RGB-D Data

  • On RGB-D scans, ColorMap and our method avoid registration artifacts; our method remains robust to motion blur despite not filtering blurry images through view selection.MVS-tex exhibits seams and ghosting, while AdvIA produces speckling.Source: §7.3 · p. 10

7.4. Robustness to Different Types of Distortions

  • As distortion severity grows, our method performs best on texture, mesh, and combined distortions; ColorMap leads on camera-pose distortion, while ours is most robust at the largest error after pose refinement.Every method except ColorMap improves with refined poses.Source: §7.4 · pp. 10–11

7.2. Comparisons: Sparse-view Generated Data

  • Our approach ranks best on LPIPS on every generated dataset; SSIM favors averaging-based methods and does not fully reflect the perceptual differences visible in Fig. 8.Scores are averaged over all objects and views.Source: §7.2 · p. 10, Table 2 · p. 11

7.5. Performance

  • On sparse input, runtime is comparable to state-of-the-art methods despite repeated optical-flow solves, but grows less favorably with input size because all back-projections contribute to each output texel.Runtime comparisons cover generated and real-world datasets.Source: §7.5 · p. 11, Table 3 · p. 12

8. Discussion, Limitations, and Future Work

The discussion examines cases where generated views disagree in repeating patterns, integration with camera-pose optimization, and failure modes including insufficient overlap and registration drift. It also outlines future work and describes RGB-D scan preparation.

8. Discussion, Limitations, and Future Work

  • Different stripe counts across views make perfect registration impossible, causing optical-flow twist and stitching blur on the Penguin’s back.Where stripe counts agree on the front, the approach aligns the stripes without becoming trapped in local minima.Source: §8 · p. 12
  • The Maneki-neko result keeps individual stripes sharp despite differing layouts, with only localized blurring where inconsistent layouts are merged.The blurring is attributed to merging the inconsistent layouts.Source: §8 · p. 12
  • Combining surface-domain registration and stitching with ColorMap camera-pose optimization produces more symmetric textures than either method alone.Source: §8 · p. 12
  • Registration struggles when views have little overlap or too little photometric detail to guide optical flow, producing ghosting around the paws in one example.Source: §8 · p. 12
  • Because the optical-flow energy is approximately invariant to applying the same deformation to all textures, registration can drift jointly from the textures’ natural locations.Regularization is the only stated bias toward smaller deformation; the example resolves wheel ghosting while shifting the undercarriage.Source: §8 · pp. 12–13
  • The authors propose adding blur estimates to confidence values, which could reduce runtime on denser multi-view data, and exploring learned confidences and simplified parameters.The RGB-D scan preparation removes background and spurious geometry, remeshes the surface, and texture-atlases it.Source: §8 · p. 13, Appendix A · p. 17

Appendix B: Implementation Details

The implementation details specify how depth confidence is computed, describe optical-flow parameter trade-offs and settings, and give gradient-stitching weights. The appendix also introduces the implementation and parameter-selection details for the reported results.

Appendix B: Implementation Details

  • The appendix presents implementation and parameter-selection details for the results in Section 7.Source: Appendix B · p. 17

Back-Projection

  • Depth confidence for visible texels falls with distance from depth-map discontinuities, with values below 0.15 quantized to zero.The procedure detects discontinuities using the normalized depth-map Laplacian and fixes the distance falloff radius at 20 pixels.Source: Back-Projection · p. 17

Optical Flow Registration

  • Four optical-flow parameters control hierarchy depth, flow smoothness, and local-mean and scale smoothing; too few levels oversmooth, while too many allow high-frequency deformation oscillations.The reported settings are L = 4 and βblur = 0.002 for most experiments, βreg = 0.2 for generated data and 0.1 for RGB-D scans, and ϵn = 0.001.Source: Optical Flow Registration · p. 17

Gradient-domain Stitching

  • Gradient stitching generally uses αgf = 0.01, with αreg = 10^-8 so smoothness regularization acts only where confidence is zero.The gradient-fitting weight required fine-tuning in a small number of cases.Source: Gradient-domain Stitching · p. 17

Baselines

The baseline setup uses publicly available implementations and adapts settings for the synthesized data and RGB-D scans. It also selects the best visual result among tested ColorMap variants and mesh resolutions.

Baselines

  • The evaluations use publicly available baseline implementations, and report the best visual result among three tested ColorMap optimization variants.The variants are alternating optimization, rigid-only optimization, and rigid followed by non-rigid optimization; alternating optimization was not optimal on synthesized data.Source: Baselines · p. 17
  • For synthesized data, AdvIA and Im2SurfTex use 256 × 256 texture resolutions to reduce speckling; both use 1024 × 1024 for RGB-D scans.Im2SurfTex uses its released implementation and pretrained weights, only the neural back-projection stage, and modifications for full extrinsics and lower memory use.Source: Baselines · p. 17
  • All methods are tested with zero to three subdivision passes, and the best results are presented; MVS-Tex and ColorMap use planar one-to-four subdivision, while AdvIA uses Loop subdivision.Source: Baselines · p. 17

Appendix C: Quantitative Comparisons

The appendix explains why LPIPS and SSIM can favor a ghosted blend, while ECD identifies a sharper result as more similar to the reference. It also notes that the S3 sharpness difference does not measure cross-view texture consistency.

Appendix C: Quantitative Comparisons

  • LPIPS and SSIM do not reliably penalize ghosting, whereas ECD identifies the sharper texture as more similar to the reference in the Figure 16 example.ECD compares local distributions of gradient orientations and magnitudes, making it more sensitive to blurring and ghosting; the blended result scores better on LPIPS and SSIM despite ghosting.Source: Appendix C · p. 17, Figure 16 · p. 18
  • S3 sharpness differences do not reflect cross-view texture consistency, and favor MVS-Tex because per-triangle view selection preserves input sharpness within each triangle.Table 4 reports differences in S3 scores, with lower values indicating closer sharpness to the input.Source: Appendix C · p. 18, Table 4 · p. 18
Loading 2610.36098v1…