PaperlayerView exact revision on arXiv

Source-linked AI summary

ReGain: Restoring Subject Fidelity in Personalization on Synthetic Images

Shubhang Bhatnagar, Ishan Bhatnagar, Viraj Shah, Narendra Ahuja

arXiv:2609.38680v1cs.CVcs.AIcs.LGeess.IV

TL;DR

DreamBooth personalization increasingly uses synthetic subject images, and ReGain corrects guidance inflation by attenuating guidance by frequency band and denoising step. It recovers 51–64% of the subject-fidelity gap on SD1.5 and improves subject fidelity on SDXL and SD3.5 while preserving prompt fidelity.

  • Problem

    DreamBooth personalization increasingly uses generative-model images, raising the question of how synthetic rather than real subject images affect outputs.

    Source: §1 · p. 1, §6 · p. 10

  • Method

    ReGain measures guidance inflation against the base model and attenuates guidance by frequency band and denoising step within the subject region, without retraining.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    51–64% of the subject-fidelity gap on SD1.5 is recovered by ReGain; it also improves subject fidelity on SDXL and SD3.5 while preserving prompt fidelity.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    ReGain needs no real photographs or retraining and preserves prompt fidelity while improving subject fidelity across SD1.5, SDXL, and SD3.5.

    Source: §1 · p. 2, §6 · p. 10

    Because ReGain uses base-model guidance as an imperfect reference, it cannot fully match Mreal; it also depends on a subject mask that may misidentify regions.

    Source: §5 · p. 10

Abstract

from arXiv · show

Text-to-image diffusion models are personalized to a subject by DreamBooth fine-tuning on a handful of its images. Increasingly, these images come from a diffusion model rather than a camera. We show that fine-tuning on such synthetic images degrades subject fidelity, producing oversaturated color and excess high-frequency detail. To isolate the cause, we fine-tune two models from the same base model with the same DreamBooth recipe, one on real photos of a subject and one on synthetic images of that subject generated by the first. We trace the degradation to classifier-free guidance (CFG). For the model personalized on synthetic images, the angle between the conditional and unconditional noise predictions, and with it the norm of their difference, is much larger than for the model personalized on real photos. This inflation grows toward high frequencies and also appears at other prompts semantically close to the subject, such as its class noun, but not at unrelated ones. We propose ReGain, a training-free correction applied at sampling time that measures how much each frequency band of the guidance is inflated relative to the base model and scales that band down accordingly. ReGain needs no real photos. On Stable Diffusion v1.5, ReGain closes 51-64% of the subject-fidelity gap to the model personalized on real photos, as measured by DINO, DINOv2 and CLIP-I. It also improves subject fidelity on SDXL and SD 3.5 and preserves text alignment on all three backbones.

1 INTRODUCTION

Personalization increasingly uses model-generated subject images, but synthetic-image fine-tuning produces oversaturated, high-frequency-heavy outputs and inflates CFG guidance. ReGain measures that inflation against the base model and corrects guidance by frequency band and denoising step.

1 INTRODUCTION

  • DreamBooth personalization binds a subject to a unique identifier using a handful of images, and those images increasingly come from generative models rather than cameras.They may be edited variants of a real subject.Source: §1 · p. 1
  • With the base model and DreamBooth recipe held fixed, synthetic-image fine-tuning produces oversaturated, high-frequency-heavy images and inflates the CFG guidance norm throughout denoising.The guidance gap widens as sampling proceeds.Source: §1 · p. 1, Figure 1 · p. 2
  • The inflation reflects a larger angle between conditional and unconditional predictions, appears on semantically related prompts, and is strongest in high-frequency bands with band-specific time profiles.Unrelated prompts show no inflation, and lowering guidance uniformly across bands and steps cannot undo the structured excess.Source: §1 · pp. 1–2
  • ReGain is a training-free sampling-time correction that measures per-band, per-step inflation against the public base model and reduces guidance accordingly where subject tokens govern prediction.It uses plain CFG elsewhere and does not require real photos.Source: §1 · p. 2
  • ReGain improves subject fidelity across four evaluation settings; on SD1.5, it closes 51–64% of the gap to a model personalized on real photos while preserving prompt fidelity.The evaluation covers DreamBooth and DreamBooth-LoRA on SD1.5 and DreamBooth-LoRA on SDXL and SD 3.5.Source: §1 · p. 2

2 RELATED WORK

Prior work studies subject personalization, synthetic-data training, and adjustments to classifier-free guidance. ReGain addresses a distinct case: guidance-term inflation after a single round of subject personalization on synthetic images.

Subject personalization

  • Personalization methods bind subjects to identifiers through full-weight, selected-layer, or low-rank-adapter fine-tuning, while tuning-free encoder or adapter methods reproduce fine identity detail less faithfully.Fine-tuning remains the choice when fidelity matters.Source: §2 · p. 2

Training on synthetic data

  • Synthetic-data research examines diversity and fidelity loss across model generations and remedies such as real-data mixing or sampling guidance, whereas this work studies one DreamBooth fine-tuning round.It identifies miscalibration of the CFG term itself, rather than the guidance scale, as the relevant change.Source: §2 · p. 3

Guidance analyses and fixes

  • Prior guidance methods adjust when, where, or how strongly CFG is applied, including by frequency band and for personalized models, but do not address inflation of the guidance term from synthetic-image personalization.ReGain corrects that inflation before other guidance adjustments are applied.Source: §2 · p. 3

3 METHOD

The section isolates how synthetic training images alter classifier-free guidance and finds that the resulting inflation varies with frequency, time, and prompt semantics. It introduces ReGain, which uses the base model to estimate and correct that inflation during sampling.

3.1 SETUP AND NOTATION

  • The controlled comparison holds base weights and DreamBooth recipe fixed while contrasting Mreal trained on real photos with Msyn trained on Mreal-generated subject images.Mreal is used for analysis and evaluation; practical settings provide Msyn, its training images, and the base model.Source: §3.1 · p. 3

3.2 WHY FINE-TUNING ON SYNTHETIC IMAGES INFLATES GUIDANCE

  • Inflation follows semantic proximity—nearly full for the class noun, about half for cat, and little to none for distant prompts—and persists on shared noisy training-image inputs.Lower-weight synthetic training and DreamBooth prior-preservation loss do not remove it.Source: §3.2 · p. 4
  • Inflation grows from low to high frequency bands, peaks earliest in high bands, and follows a distinct time profile in each band.Consequently, a single guidance weight cannot describe the inflation.Source: Figure 3 · p. 4, §3.2 · p. 5

3.3 REGAIN: RESTORING THE GUIDANCE FROM THE BASE MODEL

  • ReGain calibrates band- and step-specific guidance gains against the base model, compresses them into a schedule, then rescales guidance by band group inside the subject mask.It uses forward-noised training images for calibration and leaves guidance outside the mask unchanged.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

The experiments test ReGain across DreamBooth settings and backbones, comparing subject and prompt fidelity with artifact statistics and guidance ablations. ReGain recovers subject-fidelity loss while largely preserving prompt fidelity, and targeted corrections outperform subject-agnostic alternatives in the tested setting.

4.1 SETUP

  • The evaluation covers 30 DreamBooth subjects and four settings: DreamBooth and DreamBooth-LoRA on SD1.5, plus DreamBooth-LoRA on SDXL and SD3.5.Each setting compares synthetic-image fine-tuning with plain CFG against ReGain at the same seed, prompt, sampler, and guidance weight, using the real-photo model as reference.Source: §4.1 · p. 6

4.2 MAIN COMPARISON

  • ReGain recovers more than half of the subject-fidelity gap on SD1.5 and between a fifth and a half on SDXL and SD3.5, while CLIP-T changes by at most 0.004 in every setting.None of the three guidance baselines closes the gap; Figure 7 shows the qualitative behavior across DreamBooth-LoRA, SDXL, and SD3.5.Source: §4.2 · pp. 7–8, Table 1 · p. 8
  • On shared prompts and seeds, ReGain brings synthetic-model subject color and texture toward the real-model reference while retaining scene details and the requested attribute change.Figure 6 compares three models on the same subject, prompt, and seed in each row.Source: Figure 6 · p. 7, §4.2 · p. 8

4.3 OVER-GUIDANCE ARTIFACT METRICS

  • Under plain CFG, the synthetic model is 18% over-saturated, 22% over-contrasted, and has 13% excess high-band power; ReGain moves all three toward the real-model values without overshooting.It removes most saturation excess, about half the contrast excess, and a third of the high-band excess inside the subject region.Source: Table 2 · p. 8, §4.3 · p. 8

4.4 ANALYSIS AND ABLATIONS

  • Removing the subject mask reduces CLIP-T by 0.008 to 0.282, below plain CFG’s 0.289; ReGain takes 1.1× plain CFG’s sampling time on SD1.5.The mask lets ReGain correct the subject while leaving the background under plain CFG; calibration takes 12 minutes per subject on one RTX A4000.Source: Table 3 · p. 9, §4.4 · p. 10
  • Before correction, guidance inflation rises from low- to high-frequency bands; after the gain schedule at τ = 0.05, it is within 10% of 1 on average.Table 4 reports calibration-state inflation by band group and compression tolerance.Source: Table 4 · p. 9, §4.4 · p. 9
  • Lowering guidance weight weakens scene details and neither tested weight recovers DINO, while APG over-corrects saturation and lowers CLIP-T.ReGain alone moves all three reported metrics toward the real-model values without overshooting.Source: §4.4 · p. 9, Table 3 · p. 9
  • Coarser compression tolerances leave more residual inflation, especially in the high-frequency group, and τ = 0.05 is the coarsest tested tolerance that keeps inflation within 10%.The tested tolerances are 0.02, 0.05, 0.1, and 0.2.Source: §4.4 · p. 9, Table 4 · p. 9

5 LIMITATIONS

ReGain has practical limits: its base-model reference is imperfect, its subject mask can misplace attenuation, and its measurements do not locate the inflation inside the network.

5 LIMITATIONS

  • The base model is an imperfect reference, so sampling-time correction cannot fully match the model personalized on real photos.A training-time correction during personalization on synthetic images could close this remaining gap.Source: §5 · p. 10
  • Mask errors can attenuate guidance in the wrong region, while tracing the inflation inside the network is left for future work.The concern applies when segmentation misses part of the subject or spills onto the background; measurements are limited to output noise predictions.Source: §5 · p. 10

6 CONCLUSION

ReGain attenuates inflated guidance by frequency band and denoising step, recovering part of the subject-fidelity gap while preserving prompt fidelity. The conclusion also records the study’s data, AI-use, and reproducibility practices.

6 CONCLUSION

  • ReGain recovers 51–64% of the subject-fidelity gap on SD1.5, improves subject fidelity on SDXL and SD3.5, and preserves prompt fidelity.It measures inflation against the base model before personalization and attenuates guidance per frequency band and step inside the subject region, without real photos or retraining.Source: §6 · p. 10
  • The authors used a generative AI coding assistant for measurement and figure-generation code and manuscript drafting and editing, then reviewed and cross-checked its outputs.The authors state that they specified, executed, and verified the experimental designs, measurements, and claims, and take responsibility for the final content.Source: AI USE STATEMENT · p. 10
  • The study uses public DreamBooth subject images, reports no human-subject data or personally identifiable information, and follows the dataset and model licenses..Source: ETHICS STATEMENT · p. 11
  • Stable Diffusion v1.5 and the DreamBooth dataset are public, and Appendix C provides implementation details including the fine-tuning recipe, sampler settings, guidance weights, and seeds.The conclusion identifies these materials and settings as the basis for reproducibility.Source: REPRODUCIBILITY STATEMENT · p. 11

A SYNTHETIC TRAINING IMAGES AS THE SOURCE OF THE ∆INFLATION

The appendix tests whether factors besides synthetic-image origin explain the guidance inflation. The training sets are comparably diverse, and the tested alternatives do not account for the inflation.

A SYNTHETIC TRAINING IMAGES AS THE SOURCE OF THE ∆INFLATION

  • The appendix reports that neither lower diversity, inherited sampling guidance weight, nor omitting prior-preservation loss accounts for the inflation.It therefore attributes the inflation to the synthetic origin of the training images.Source: A SYNTHETIC TRAINING IMAGES AS THE… · p. 15

A.1 DIVERSITY OF THE TRAINING IMAGES

  • Mean pairwise CLIP ViT-B/32 diversity was 0.142 ± 0.050 for real photos and 0.140 ± 0.052 for generated images, indicating comparable diversity.The comparison averages one minus cosine similarity over image pairs and reports mean and standard deviation across 30 subjects.Source: §A.1 · p. 15

A.2 GUIDANCE WEIGHT OF THE TRAINING IMAGES

The appendix tests whether training-image guidance weight or prior-preservation loss removes the inflation, then checks its prevalence across subjects. Inflation persists across the tested settings and subjects.

A.2 GUIDANCE WEIGHT OF THE TRAINING IMAGES

  • Across training-image guidance weights 7.5, 5, and 3, inflation remained large and increased from low- to high-frequency bands; lowering weight from 7.5 to 3 reduced it by about 12%.Most of the reduction was already reached at weight 5; Table 5 reports inflation by band group, with 7.5 as the paper’s standard setting.Source: §A.2 · p. 15, Table 5 · p. 16

A.3 PRIOR-PRESERVATION LOSS

  • With prior-preservation loss, inflation against the base model was 7.38, 7.35, and 9.47 in low-, mid-, and high-k groups, and 8.41 overall.The loss therefore did not remove inflation, which remained largest in the high-k group.Source: §A.3 · p. 16

A.4 THE ∆INFLATION ON ALL SUBJECTS

  • In 29 of 30 subjects, Msyn’s ∥∆∥ exceeded Mreal’s, by 48 ± 22%; θ also rose by 48 ± 22%, while δ changed by −0.1 ± 0.1% and r by 0.6 ± 2.2%.The exception, rc car, showed no excess in either measure; Figure 9 plots per-step ratios of θ, δ, and r over five seeds.Source: Figure 9 · p. 16, §A.4 · p. 17

B SUBJECT MASK

The subject mask is built from saliency maps and assigns pixels through a parameter-free competition between subject and scene tokens. Calibration uses an auxiliary “in a scene” prompt when no scene content is present.

B SUBJECT MASK

  • On Stable Diffusion v1.5 and SDXL, saliency maps come from refined conditional cross-attention; on SD 3.5, they come from joint attention.The SD 3.5 procedure omits the T5 encoder for this evaluation and adds one conditional evaluation per sampling step.Source: B SUBJECT MASK · p. 17
  • The mask compares subject-token saliency against scene-token saliency at each pixel, assigning pixels claimed by neither set to the subject.The subject set contains identifier sub-word tokens and the class noun; the scene set contains scene-content words.Source: B SUBJECT MASK · p. 17
  • Figure 10 shows the mask at steps 0, 10, 20, and 40 of a 50-step trajectory, plus the final-sample mask.The masks are drawn on the clean image predicted at each step.Source: Figure 10 · p. 17
  • When calibration prompts lack scene content, an auxiliary evaluation adds “in a scene”; its masks agree with caption-based masks at mean IoU 0.86.The auxiliary phrase fills the scene-token set without using training-image captions.Source: B SUBJECT MASK · p. 18

C IMPLEMENTATION DETAILS

Implementation uses matched real-photo and synthetic-image fine-tuning procedures, with evaluation and sampling settings specified for each backbone. The evaluation prompts follow the official DreamBooth lists.

C IMPLEMENTATION DETAILS

  • Within each setting, Mreal and Msyn share fine-tuning procedures and hyperparameters, differing only in training images; the synthetic set contains five images generated by Mreal.The identifier is “monadikos,” and the fine-tuning prompt is “a photo of monadikos ⟨class⟩.”Source: C IMPLEMENTATION DETAILS · p. 18
  • Text encoders are frozen and no prior-preservation loss is used; evaluation uses DDIM except SD 3.5, which uses its flow-matching Euler scheduler.LoRA adapters use scaling factor α equal to rank and are merged into base-model weights before sampling.Source: Table 6 · p. 18
  • Three guidance baselines are sampled from the same Msyn checkpoints under their official settings, including S-CFG, CFG++, and FDG.Their specified settings include w = 7.5 for S-CFG and FDG, and λ = 0.6 for CFG++.Source: C IMPLEMENTATION DETAILS · p. 18
  • Figure 11 displays compressed gain schedules for pink sunglasses, cat2, and backpack, with band groups as rows and sampling steps as columns.Color and labels encode the gain.Source: Figure 11 · p. 19

C.1 EVALUATION PROMPTS

  • Evaluation uses official DreamBooth prompts verbatim, with 25 prompts each for object subjects and live subjects.The lists include scene recontextualizations and property modifications; live-subject prompts also include accessorization prompts.Source: §C.1 · p. 19

D OVER-GUIDANCE ARTIFACT METRICS

The section defines image statistics for saturation, contrast, and high-frequency power, and specifies the image region used for comparison. The high-band measure excludes zero-frequency power to focus on spectral shape.

D OVER-GUIDANCE ARTIFACT METRICS

  • The high-band fraction is Fourier power beyond a radial cutoff at one quarter of the Nyquist frequency, with the zero-frequency term removed.Saturation is mean HSV S-channel value, and root-mean-square contrast is the standard deviation of grayscale intensity.Source: D OVER-GUIDANCE ARTIFACT METRICS · p. 19
  • Leaving the zero-frequency term in the denominator would make the high-band statistic track contrast as well as spectral shape.The mean/DC term dominates total power and scales with brightness and variance.Source: D OVER-GUIDANCE ARTIFACT METRICS · pp. 19–20
  • The Table 2 measurement region is the same Msyn-steered mask for all three methods, covering 43% of the evaluation frame.High-band fraction is measured on the mask’s bounding box because it requires a rectangular grid.Source: D OVER-GUIDANCE ARTIFACT METRICS · p. 20

E ADDITIONAL QUALITATIVE RESULTS

Additional qualitative figures extend comparisons to DreamBooth-LoRA, SDXL, further subject-prompt combinations, and subject-agnostic guidance alternatives. The figures show reference photos, prompts, and outputs under the listed methods.

E ADDITIONAL QUALITATIVE RESULTS

Related papers

Loading 2609.38680v1…