PaperlayerView exact revision on arXiv

Source-linked AI summary

IntrinSync: Joint Intrinsic Decomposition and Reciprocal Rendering

Zheng Gu, Rui Huang, Xilu Zhang, Jingbo Zhang, Min Lu, Zhida Sun, Dani Lischinski, Daniel Cohen-Or, Hui Huang

arXiv:2610.11138v1cs.CV

TL;DR

Intrinsic properties are interdependent, but prior work has explored cross-channel interaction and inverse-forward reciprocity separately. IntrinSync combines joint intrinsic modeling with reciprocal rendering, achieving competitive or improved performance across three benchmarks while producing more coherent, physically consistent decompositions.

  • Problem

    Different combinations of surface color, illumination, geometry, and material can explain the same RGB appearance, while prior methods explore channel interaction and inverse-forward coupling separately.

    Source: §1 · p. 2

  • Method

    IntrinSync jointly denoises albedo, shading, normals, roughness, and metallic maps from RGB, and couples decomposition with rendering through dual cycle-consistent paths.

    Source: §1 · p. 2

  • Results

    Across HyperSim, InteriorVerse, and MatrixCity, IntrinSync achieves competitive or improved intrinsic-estimation and forward-rendering performance with more coherent, physically consistent decompositions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Intrinsic maps can be manipulated as explicit physical properties and rendered back into RGB while preserving scene content.

    Source: §1 · p. 2

    Decomposing one 1024×768 image into five channels takes about 90 seconds with 30 denoising steps, and scarce accurate real-scene annotations leave a sim-to-real gap for complex geometry and illumination.

    Source: §5 · p. 9

Abstract

from arXiv · show

Inverse rendering decomposes an image into intrinsic properties such as appearance, illumination, geometry, and material, yet these properties are inherently interdependent. A reliable decomposition should produce intrinsic maps that are not only individually plausible, but also mutually compatible in explaining the image. However, existing methods either model intrinsic channels in isolation or treat inverse and forward rendering as separate processes, leaving the interdependence underexploited. In this paper, we introduce IntrinSync, a unified framework that captures this interdependence through joint-channel modeling and reciprocal inverse-forward rendering. At the channel level, we jointly decompose an input RGB into albedo, shading, surface normal, roughness, and metallic maps through a 1-to-N mapping, enabling information exchange across channels throughout generation. At the process level, we establish inverse-forward reciprocity through a dual cycle-consistent objective that aligns corresponding predictions across a closed loop. Experiments on three datasets demonstrate that our method achieves competitive intrinsic estimation and forward rendering performance, improving coherence and physical consistency. Beyond decomposition, IntrinSync provides a physically grounded interface for image editing, allowing intrinsic properties to be explicitly manipulated and rendered back into RGB images.

1 INTRODUCTION

Intrinsic properties are interdependent, yet prior approaches explore cross-channel interaction and inverse-forward reciprocity separately. IntrinSync combines joint intrinsic estimation with reciprocal rendering and reports competitive or improved performance alongside more coherent decompositions and intrinsic-guided editing.

1 INTRODUCTION

  • Intrinsic decomposition is challenging because different combinations of surface color, illumination, geometry, and material can explain the same observed appearance.Prior work explores cross-channel interaction or inverse-forward loop closure separately, leaving their combined interdependence underexplored.Source: §1 · p. 2
  • IntrinSync jointly denoises albedo, shading, surface normal, roughness, and metallic maps from RGB, allowing channels to exchange information during generation.The same architecture synthesizes RGB from intrinsic maps through an N-to-1 mapping.Source: §1 · p. 2
  • A dual cycle-consistent objective aligns predictions across reciprocal RGB-to-intrinsics-to-RGB and intrinsics-to-RGB-to-intrinsics paths, in addition to direct supervision.The paper also curates indoor and outdoor photographs without ground-truth intrinsic annotations alongside synthetic training data.Source: §1 · p. 2
  • Across HyperSim, InteriorVerse, and MatrixCity, IntrinSync achieves competitive or improved intrinsic-estimation and forward-rendering performance while producing more coherent, physically consistent decompositions.The framework also supports editing intrinsic properties and rendering them back into RGB while preserving scene content.Source: §1 · p. 2

2 RELATED WORK

Prior work spans diffusion-based image decomposition and intrinsic estimation, but channel interaction and inverse-forward coupling have largely been treated separately. IntrinSync models both forms of interdependence within one reciprocal rendering framework.

2.1 IMAGE DIFFUSION MODELS

  • Unlike semantic or compositing-layer decomposition, IntrinSync targets physical scene properties; Qwen-Image-Layered jointly decomposes RGBA layers using inter-layer attention within an MMDiT.Figure 2 contrasts independent channel prediction, which can leak appearance, with joint 1-to-N decomposition that allows information exchange.Source: §2.1 · pp. 2–3, Figure 2 · p. 3

2.2 INTRINSIC DECOMPOSITION

  • Earlier intrinsic decomposition used shading and reflectance priors or extra observations, while later data-driven methods drew on densely annotated synthetic datasets and learned estimators.The cited datasets include Hypersim, InteriorVerse, and MatrixCity.Source: §2.2 · p. 3
  • Recent diffusion-based intrinsic methods include separate per-channel prediction and synthesis, bidirectional cycle losses, shared representations, and VLM-guided rewards.Related work also extends intrinsic properties to video.Source: §2.2 · p. 3
  • IntrinSync’s distinction is to combine joint denoising across intrinsic channels with reciprocal coupling of decomposition and rendering, adding consistency feedback to direct supervision.The two interactions operate at the channel and process levels.Source: §2.2 · p. 3

3 METHOD

IntrinSync jointly models intrinsic channels and couples inverse decomposition with forward rendering. Its dual cycle-consistent training aligns corresponding predictions across reciprocal paths.

3.1 PROBLEM FORMULATION

  • The task estimates spatially aligned albedo, shading, camera-space surface normals, roughness, and metallic maps from RGB, and synthesizes RGB from those maps.The goal is for the predicted maps to form a consistent explanation of image reconstruction.Source: §3.1 · p. 4

3.2 JOINT-CHANNEL INTRINSIC MODELING

  • The model jointly denoises all intrinsic channels from RGB, allowing each channel’s update to depend on the evolving features of the full intrinsic set.The decoded maps are albedo, shading, surface normals, roughness, and metallic.Source: §3.2 · p. 4
  • Forward rendering uses the same representation and backbone to generate RGB from intrinsic maps, forming reciprocal paths with decomposition.Unavailable conditioning channels are filled with zeros to support datasets with different annotations.Source: §3.2 · p. 5

3.3 DUAL CYCLE-CONSISTENT TRAINING

  • Direct supervision trains both RGB-to-intrinsics and intrinsics-to-RGB mappings, but does not by itself check either process’s output through the other.The supervision uses paired RGB images and intrinsic channels; the two processes are otherwise optimized separately.Source: §3.3 · p. 5
  • A vanilla cycle matches reconstructed endpoints, which can permit accurate reconstruction despite inaccurate intermediate intrinsic maps.The described reciprocal paths are I → X′ → I′ and X → I* → X*.Source: §3.3 · p. 5
  • The dual cycle aligns corresponding decomposition and rendering predictions across reciprocal paths, and training combines this objective with direct supervision.The alignment encourages consistent behavior after crossing from one process to the other.Source: §3.3 · p. 5

4 EXPERIMENTS

IntrinSync is evaluated on synthetic datasets across inverse rendering, forward rendering, and intrinsic-guided editing. The experiments report competitive decomposition and rendering, improvements from joint-channel modeling and dual-cycle supervision, and controllable editing through intrinsic maps.

4.1 IMPLEMENTATION

  • Training combines 20K Hypersim, 15K InteriorVerse, and 20K MatrixCity images with 45K real-world images from four datasets to improve in-the-wild generalization.The decomposition prompt specifies albedo, shading, camera-space normal, roughness, and metallic; Qwen-VL supplies RGB captions.Source: §4.1 · p. 6

4.2 EVALUATION SETUP

  • Evaluation uses Hypersim, InteriorVerse, and MatrixCity test subsets, with PSNR and LPIPS for intrinsic maps, RMSE and SSIM for albedo, angular-error measures for normals, and PSNR, LPIPS, and SSIM for rendering.Forward rendering uses ground-truth intrinsic maps, while RGB reconstruction renders predictions from estimated maps.Source: §4.2 · p. 6

4.4 QUANTITATIVE RESULTS

  • Across three benchmarks, IntrinSync improves albedo estimation consistently and achieves competitive normal, roughness, and metallic estimates, while also performing strongly in forward rendering.The reported results support mutually compatible intrinsic predictions for faithful reconstruction and neural rendering.Source: Table 1 · p. 7, §4.4 · p. 7

4.3 QUALITATIVE RESULTS

  • Qualitative comparisons show competitive decomposition across datasets, including recovered illumination in unlabeled dark floor regions and reconstruction of blurry metal connectors.These examples concern physical properties not finely annotated in the datasets.Source: §4.3 · p. 7

4.5 ABLATION STUDY

  • Joint prediction improves all six reported decomposition metrics over 1-to-1 decomposition, including albedo PSNR by 1.27 dB, irradiance PSNR by 0.91 dB, and normal mean angular error by 2.32°.The variants are compared on a Hypersim subset; Figure 6 also reports cleaner maps and lower training loss for joint decomposition.Source: §4.5 · p. 8, Table 2 · p. 8, Figure 6 · p. 8
  • The vanilla cycle improves normal estimation, perceptual quality, and RGB reconstruction but lowers albedo and irradiance PSNR; the dual cycle improves those PSNRs, normal accuracy, and forward-rendering performance.The authors attribute the vanilla-cycle trade-off to possible information leakage across intrinsic channels under input-reconstruction supervision.Source: Figure 7 · p. 8, §4.5 · p. 9

4.6 APPLICATIONS

  • Intrinsic-guided editing either manipulates decomposed maps before rendering or synthesizes images from selected intrinsic channels while text prompts specify unconstrained properties.The examples support explicit control over illumination and surface materials.Source: §4.6 · p. 9, Figure 8 · p. 9

5 CONCLUSION, LIMITATIONS AND FUTURE WORK

The conclusion presents IntrinSync as a unified framework that links intrinsic-channel modeling with reciprocal inverse and forward rendering. The authors identify inference cost, incomplete physical supervision, and a real-scene sim-to-real gap as limitations.

5 CONCLUSION

  • IntrinSync treats decomposition and rendering as reciprocal views of image formation, combining cross-channel and cross-process interaction to encourage accurate, consistent scene representations.The conclusion states that these interactions reduce feature leakage.Source: §5 · p. 9

5 LIMITATIONS AND FUTURE WORK

  • Decomposing a 1024×768 image into five channels takes approximately 90 seconds using 30 denoising steps.The authors identify inference expense as a current implementation limitation.Source: §5 · p. 9
  • Variation in channel availability and scene distributions limits physical supervision, while scarce accurate real-scene annotations leave a sim-to-real gap for complex geometry and illumination.Future work includes more efficient sampling and larger-scale training to improve efficiency and generalization.Source: §5 · p. 9

A IMPLEMENTATION DETAILS

Implementation uses a roughly 100K-image training set spanning synthetic and real-image datasets. Real-image intrinsic supervision uses pseudo labels, while training handles missing channels and weights losses differently by data type.

A IMPLEMENTATION DETAILS

  • The training set comprises about 100K images collected from three synthetic and four real-image datasets; because real-image datasets lack ground-truth intrinsic annotations, Ouroboros provides pseudo labels for supervision.Source: A IMPLEMENTATION DETAILS · p. 13
  • Missing intrinsic channels are represented by zero-valued latents and masked from decomposition loss; available conditions are independently dropped with probability 0.05 during composition.Direct decomposition and composition losses each have weight 1.0, while real-image decomposition loss has weight 0.5.Source: A IMPLEMENTATION DETAILS · p. 13
  • The two directions share a pretrained MMDiT backbone but use separate LoRA adapters; training runs for 35K iterations on five NVIDIA Pro6000 GPUs and takes about 72 hours.The LoRA rank and alpha are both 32.Source: A IMPLEMENTATION DETAILS · p. 13

B ADDITIONAL RESULTS

Additional results assess generalization and controllability across synthetic, real-world, and out-of-distribution images, and demonstrate intrinsic-guided editing.

B ADDITIONAL RESULTS

  • The appendix reports synthetic comparisons with ground truth, tests on indoor and outdoor real-world scenes and publicly sourced OOD photographs, and edits individual intrinsic channels while holding the others fixed.Source: B ADDITIONAL RESULTS · p. 13

B.1 COMPARISON ON SYNTHETIC IMAGES

On synthetic test sets from Hypersim, InteriorVerse, and MatrixCity, IntrinSync achieves the best decomposition quality among compared methods, with gains across several intrinsic components.

B.1 COMPARISON ON SYNTHETIC IMAGES

  • IntrinSync achieves the best decomposition quality across Hypersim, InteriorVerse, and MatrixCity, with intrinsic maps that most closely match ground truth in structure and appearance.Comparisons cover only components with available ground-truth annotations.Source: §B.1 · p. 13
  • Its albedo and shading better disentangle reflectance from illumination.Source: §B.1 · p. 13
  • Predicted normals preserve geometric details more effectively.Source: §B.1 · p. 13
  • Roughness and metallicity estimates are more accurate and spatially coherent.Source: §B.1 · p. 13

B.2 COMPARISON ON REAL IMAGES

On real indoor and outdoor images without ground-truth intrinsic annotations, IntrinSync is compared by visual plausibility, input consistency, and cross-intrinsic leakage. It produces more plausible outputs, better preserves image content and structure, and shows less leakage.

B.2 COMPARISON ON REAL IMAGES

  • Real-image comparisons use the same input images and text prompts, assessing visual plausibility, input consistency, and cross-intrinsic leakage because ground-truth intrinsic annotations are unavailable.Source: §B.2 · p. 13
  • Compared with existing methods, IntrinSync produces more visually plausible results while better preserving input-image content and structure.Source: §B.2 · pp. 13–14
  • Its predicted intrinsic components exhibit less cross-intrinsic leakage, indicating improved interdependence among intrinsic properties.Source: §B.2 · p. 14

B.3 INTRINSIC-GUIDED IMAGE EDITING

IntrinSync supports editing by decomposing an input into five intrinsic channels and independently changing albedo or shading while holding the other components and text condition fixed. The resulting changes are smooth and interpretable, with the other intrinsic properties and scene content largely preserved.

B.3 INTRINSIC-GUIDED IMAGE EDITING

  • For editing, the method decomposes an input into five intrinsic channels and varies albedo saturation or shading brightness independently, keeping the other components and text condition fixed.Albedo saturation factors are 1.0, 1.5, 2.0, and 3.0; shading brightness factors are 0.25, 0.5, 1.0, 1.5, and 2.0.Source: §B.3 · p. 14
  • Albedo edits smoothly change surface colorfulness while largely preserving illumination and scene structure; shading edits change perceived illumination without noticeable changes to reflectance or scene content.Source: §B.3 · p. 14
  • Additional examples show global albedo scaling and relighting rendered back into RGB images.Source: Figure 18 · p. 20, Figure 19 · p. 21

Related papers

Loading 2610.11138v1…