Source-linked AI summary
Generalization over Memorization: Generalization-Aware Diffusion Adaptation for Single-Image Multi-View Synthesis
Jie Li, Xingchen Zou, Yuxuan Liang
TL;DR
Single-image multi-view synthesis is difficult to evaluate when shared scenes let memorization resemble transferable view control. GoM addresses this with scene-disjoint, exposure-matched selection and targeted diffusion adaptation, and it won the challenge among 293 registered teams under a strict one-model, one-pass protocol.
Problem
Shared training and validation scenes can make memorization appear to be transferable view control in a 40-scene challenge requiring 26 views from one RGB input without explicit geometry or post-processing.
Method
GoM combines scene-disjoint validation, exposure-matched transfer selection, rank-32 LoRA adaptation of a 4B rectified-flow DiT, trajectory control, and VAE-decoder tuning.
Results
GoM ranked first among 293 registered teams and led SSIM, NRIQA, and the composite score on unseen Phase-B scenes.
Takeaways & Limitations
Under limited data, generalization-aware selection and targeted adaptation can be more effective than simply increasing model capacity.
Takeaways & Limitations
The study uses few paired transfer estimates, fixed discrete target views, and no evaluation of continuous trajectories, multi-view consistency, or cross-dataset generalization.
Abstract
from arXiv · showhide
We present the winning solution to the ACM Multimedia 2026 Grand Challenge on Single-Image Guided Multi-Angle Image Synthesis. It ranks first among 293 registered teams; 56 teams obtained at least one scored submission on the public Phase-A leaderboard. With only 40 training scenes, the challenge requires 26 target views from one RGB model and one forward pass per view; it prohibits explicit geometry, external rendering, chained generation, candidate selection, and post-processing. We identify a critical model-selection failure: shared training and validation scenes make memorization appear as transferable view control. We therefore introduce GoM. Short for Generalization over Memorization, the framework combines scene-disjoint validation, exposure-matched selection, and targeted diffusion adaptation. Its synthesis model adapts a 4B rectified-flow DiT using rank-32 LoRA, optimizer restarts, late-checkpoint averaging, and VAE decoder tuning. More than 300 offline experiments and 24 online submissions show that validation design and training-trajectory control can matter as much as architecture scale in small-data generative modeling.
1 Introduction
Single-image multi-view synthesis must infer unseen geometry and appearance while preserving scene identity, but shared-scene validation can make memorization resemble transferable view control. GoM addresses this through generalization-aware selection and targeted adaptation, achieving first place under the challenge protocol.
- Challenge setting: The challenge requires 26 target views from one 1280 × 720 RGB photograph using one model and one forward pass per view, without geometry, rendering, chaining, selection, or post-processing.Only 40 training scenes are available, and Phase B evaluates unseen scenes.
- Generalization failure: Shared training and validation scenes let view-level splits preserve scene identity, so models may improve through known-scene appearance recovery rather than transferable view control.The same container drops from 1.9386 in Phase A to 1.8512 in Phase B, mainly through structural and perceptual consistency.
- GoM: GoM uses scene-disjoint validation and exposure-matched transfer estimates to distinguish transferable gains from gains confined to overlapped scenes.Its synthesis layer combines rank-32 LoRA on a 4B rectified-flow DiT, optimizer restarts, checkpoint averaging, and VAE-decoder adaptation.
- Outcome: GoM won the challenge among 293 registered teams, with 56 teams obtaining scored Phase-A submissions, and led SSIM, NRIQA, and the composite score on unseen Phase-B scenes.The solution operated under the single-model, one-pass-per-view protocol without geometry or post-processing.
2 Related Work
Prior novel-view systems use camera conditioning, geometry, depth, warping, temporal priors, or joint reconstruction to improve view synthesis. GoM instead combines parameter-efficient diffusion adaptation, trajectory averaging, and separate decoder tuning for a small-data setting.
- Novel-view synthesis: Zero-1-to-3, ZeroNVS, SV3D, CAT3D, and MultiDiff use increasingly explicit camera, pose, depth, warping, temporal, or reconstruction priors for novel-view synthesis.These methods demonstrate strong geometric or temporal priors but rely on intermediates or multi-view structure that the challenge setting excludes.
- Efficient adaptation: LoRA provides low-rank adaptation, while stochastic weight averaging, model soups, snapshot ensembles, and SGDR combine or restart training trajectories to improve robustness.GoM studies these ideas where optimization gains can be confounded by scene memorization.
- Decoder bottlenecks: Because VAE decoding can limit pixel fidelity, GoM adapts the decoder separately from the DiT while keeping the encoder fixed.This separates generation and reconstruction bottlenecks rather than tuning only the denoiser.
3 Methodology
GoM separates model selection from synthesis adaptation: it estimates which improvements transfer across scenes, then adapts a geometry-free diffusion pipeline under strict compute and inference constraints. The method combines scene-disjoint evidence, controlled trajectory changes, LoRA, decoder tuning, and deterministic single-model inference.
- Overview: GoM has a model-selection layer that evaluates likely transfer and a synthesis layer that adapts a pretrained diffusion model under the challenge constraints.The two layers are designed so selection supplies evidence for synthesis decisions.
- Generalization-aware selection: The selection objective records paired candidate-minus-baseline improvements separately for overlapping-scene audit and scene-disjoint evaluation regimes.The audit is an overlapping-scene diagnostic, whereas the sister lineage provides scene-disjoint evidence.
- Generalization-aware selection: Candidates advance only when transfer-adjusted gains exceed 0.005, fall outside the training-variation band, and satisfy compute and component-quality constraints.Composite packages without an isolated transfer ratio require direct scene-disjoint evidence.
- Generalization-aware selection: GoM treats scene identity, rather than target view, as the unit of generalization and uses a six-scene audit plus a 34-scene sister lineage.The audit supports within-lineage deltas, while final B2 selection uses scenes excluded from both DiT and decoder training.
- Controlled comparisons: Paired offline and online comparisons prioritize high-transfer axes and treat composite or single-metric gains cautiously.Absolute validation scores are not compared across unrelated lineages, and repeated DDP variation of approximately ±0.006 defines indistinguishable gains.
- Synthesis model: The synthesis pipeline adapts a 4B rectified-flow DiT with LoRA and separately fine-tunes the VAE decoder, while view names specify targets without camera parameters.The encoder and text encoder remain frozen, and each requested view uses one deterministic encoder–DiT–decoder chain.
- Trajectory adaptation: Training uses 1,040 triplets from 40 scenes and rank-32 LoRA, with trajectory restarts and tensor-wise averaging of late checkpoints into one LoRA file.B1 averages epochs {130, 140, 150}; B2 selects {120, 130, 140, 150} through the scene-disjoint sweep.
- Decoder adaptation: Decoder adaptation targets the separate reconstruction bottleneck by tuning only the 49.6M-parameter decoder while keeping the encoder and DiT fixed.The staged procedure includes reconstruction-only adaptation and, for B2, high-frequency refinement on unlabeled natural images without challenge images or quality labels.
4 Experiments
Experiments evaluate GoM through decomposed scoring, scene-disjoint evidence, controlled interventions, and unseen-scene testing. The results show that structural generalization, decoder adaptation, and trajectory control matter more than overlapping-scene totals alone.
- Evaluation protocol: The official score combines normalized pixel fidelity, structural similarity, perceptual similarity, and no-reference image quality.PSNR, SSIM, and LPIPS compare against ground truth, while NRIQA is no-reference.
- Evaluation protocol: Scene-disjoint evaluation separates within-lineage audits, final recipe selection, sparse Phase-A transfer calibration, and unseen-scene Phase-B ranking.The early audit covers 156 targets but supports only paired comparisons because its LoRA saw all 40 scenes.
- Unseen-scene performance: B1 loses 0.0874 from overlapping Phase-A scenes to unseen Phase-B scenes, with SSIM and 1−LPIPS contributing 0.0786, or about 90%, of the gap.Normalized PSNR remains essentially unchanged, indicating that overlap primarily hides structural and perceptual failures.
- Unseen-scene performance: B2 raises the Phase-B total by 0.0116 and ranks first, with a 0.0124 combined PSNR/SSIM gain and a 0.0040 NRIQA gain relative to B1.A 0.0048 loss in 1−LPIPS offsets part of the improvement; GoM scores 1.8628 versus 1.8542 and 1.8488 for the other top-ranked teams.
- Controlled interventions: Late-checkpoint averaging raises the score from 1.9245 to 1.9314, while the four-checkpoint overlap result reaches only 1.9322, within the variation band.B2 is selected by its scene-disjoint structural profile rather than the largest overlapping-scene total.
- Controlled interventions: Decoder adaptation raises reconstruction fidelity from 32.4 to 36.4 dB and improves the scene-disjoint end-to-end score by 0.0114.The intervention adds exactly 0.0089 to both restarted and continuous LoRA lineages, supporting a decoder bottleneck distinct from DiT optimization.
- Failure analysis: The remaining error is upstream of reconstruction: adapted-VAE reconstruction reaches 0.9765 SSIM, whereas held-out end-to-end synthesis reaches 0.2668.Left- and bottom-facing targets obtain approximately 0.17–0.21 SSIM, compared with 0.29–0.31 for top-facing views.
- Negative results: More than 40 screened directions show that capacity increases, richer supervision, and metric-directed objectives do not substitute for transferable view control.GoM rejects overlapping-scene-only gains, unresolved ties, and metric-specific trade-offs, while retaining gains jointly supported by PSNR and SSIM.
5 Discussion
GoM makes generalization-aware model selection and targeted adaptation central to single-image novel-view synthesis. Its evidence combines scene-disjoint validation, controlled transfer calibration, targeted interventions, and explicit evaluation limits.
- Generalization-Aware Model Selection: Scene-overlapping validation can preserve scene identity and make memorization appear to be transferable view control.GoM defines generalization through scene identity and separates overlapping diagnostics from scene-disjoint evidence.
- Generalization-Aware Model Selection: Identity-disjoint selection should precede architecture and optimization comparisons when multiple outputs share an underlying scene identity.The paper frames validation as an interface between data design and optimization rather than a reporting choice.
- Targeted End-to-End Adaptation: GoM uses stage-specific adaptation: trajectory control targets generalization, while decoder tuning targets image formation without enlarging the backbone.The recipe combines targeted interventions while retaining a single end-to-end inference model.
- Controlled Evidence and Design Lessons: 24 online submissions, more than 300 offline experiments, and over 40 falsified directions support transfer calibration, replication, and falsification as complementary evidence.These controls turn the competition result into testable design claims rather than an isolated score.
- Controlled Evidence and Design Lessons: Full-reference metrics reward recorded completions even when hidden regions admit multiple plausible views, motivating componentwise and scene-level held-out reporting.The paper also recommends per-view distributions to expose evaluation mismatch.
- Scope and Limitations: The findings remain bounded by the challenge’s limited data, discrete target views, small audit, and lack of cross-dataset or continuous-trajectory evaluation.The authors call for larger scene-disjoint benchmarks and more general consistency tests.
6 Conclusion
GoM addresses transferable view control under limited data by combining scene-disjoint, exposure-matched selection with targeted diffusion adaptation. The authors report first place under a strict one-model, one-pass-per-view protocol and argue that generalization-aware selection can outperform simply increasing capacity.
- Conclusion: GoM ranked first among 293 registered teams under a strict one-model, one-pass-per-view protocol.The system operates without explicit geometry or test-time post-processing.
- Conclusion: GoM targets scene memorization through scene-disjoint, exposure-matched model selection and adapts generation with trajectory regularization and VAE decoder tuning.The approach uses a geometry-free diffusion model in the challenge’s 40-scene regime.
- Conclusion: Under limited data, reliable generalization-aware selection and targeted adaptation can be more effective than simply increasing model capacity.This is presented as a supported implication of the reported results.