PaperlayerView exact revision on arXiv

Source-linked AI summary

Training-Free Inpainting Across Domains with a Frozen Text-to-Image Diffusion Model

Zhenhuan Wang, Fengyi Yuan

arXiv:2609.00862v1cs.CV

TL;DR

A frozen generic text-to-image model lacks the inpainting-specific resources used by modern inpainters, yet the paper tests whether test-time latent control can support conditional inpainting without adding those resources. Step-PI combines boundary–interior feedback, persistent state, and a predefined release schedule; it transfers across three evaluated domains and improves all 15 dataset–metric cells in matched additions. It offers a complementary route to inpainting-trained systems, which retain the absolute metric leads.

  • Problem

    Modern diffusion inpainters use inpainting-specific weights or conditioning interfaces, while generic text-to-image models lack those resources.

    Source: §1 · p. 1

  • Method

    Step-PI controls deterministic sampling of a frozen vanilla SD1.5 model with boundary–interior feedback, discounted persistent state, and a predefined release schedule.

    Source: §3.2 · p. 3

  • Results

    Step-PI transfers unchanged from CelebA-HQ pilots to AFHQ and Places2, and the state-only and release-only additions each improve all 15 dataset–metric cells.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Test-time latent control provides a complementary route for repurposing a frozen generic text-to-image model for cross-domain inpainting.

    Source: §6 · p. 8

    The bounded per-step state characterization establishes neither trajectory-level stability nor task benefit.

    Source: §4.3 · p. 4

Abstract

from arXiv · show

We show that a frozen generic text-to-image diffusion model can perform conditional inpainting across three evaluated natural-image domains with one fixed controller configuration, without inpainting-specific weight training, dataset-specific weight adaptation, or learned inpainting-specific conditioning channels. Step-PI augments known-region projection with boundary-interior latent feedback, persistent PI state, and a predefined four-field release schedule that modulates controller signals along the reverse trajectory. Developed only on Main35-disjoint CelebA-HQ pilots, the controller transfers unchanged to AFHQ and Places2. Across two field-identical comparisons on the same 3,500 cases, adding persistent state and replacing uniform release with the predefined schedule each improve all 15 dataset-metric cells; 95% bootstrap intervals exclude zero for all five metrics in both comparisons. In descriptive native-route comparisons, Step-PI leads LanPaint and PILOT (the closest evaluated training-free baselines using vanilla SD1.5) on all five equal-dataset macro metrics. Inpainting-trained systems retain the absolute metric leads but rely on substantial inpainting-specific offline optimization. Our method provides a complementary approach for repurposing a frozen generic text-to-image model for cross-domain inpainting through test-time latent control.

1. Introduction

The paper presents Step-PI as a way to repurpose a frozen generic text-to-image model for conditional inpainting without inpainting-specific training or learned conditioning channels. It motivates boundary–interior feedback, persistent state, and scheduled control, then reports transfer and matched-comparison evidence across three domains.

1. Introduction

  • Known-region projection and final RGB compositing preserve observed content but do not constrain the generated region, motivating distinct boundary and interior corrections over the reverse trajectory.The useful correction balance changes along that trajectory.Source: §1 · p. 1
  • Step-PI combines boundary–interior current feedback, discounted trajectory memory, and a predefined release schedule, with the controller configuration fixed across all three domains.The release schedule modulates already-computed signals; text conditions are fixed within controller comparisons.Source: §1 · pp. 1–2
  • The paper evaluates persistent state and the predefined release schedule through field-identical comparisons that isolate each addition.Parameters were developed on Main35-disjoint CelebA-HQ pilots, with transfer to AFHQ and Places2 unchanged.Source: §1 · p. 2
  • Across three evidence layers, Step-PI transfers from CelebA-HQ pilots to AFHQ and Places2 unchanged, improves all 15 dataset–metric cells in two matched comparisons, and leads two vanilla-SD1.5 training-free baselines on five macro metrics.Inpainting-trained systems retain absolute metric leads; Step-PI avoids their inpainting-specific offline optimization but shifts substantial computation to inference.Source: §1 · p. 2

2. Related Work

Related work distinguishes systems with learned inpainting capability from training-free methods that keep deployed weights fixed. The paper positions Step-PI among the latter while using inpainting-trained systems as quality references.

2.1. Trained Inpainting Methods

  • Inpainting-trained systems acquire capability through learned weights or conditioning interfaces, so the paper treats them as quality references rather than matched interventions.The cited examples include SD-Inpaint, BrushNet, and PixelHacker [31].Source: §2.1 · p. 2

2.2. Training-Free Inpainting Methods

  • Training-free methods keep deployed weights fixed but differ in foundation, sampler, and conditioning interface, limiting component-level causal attribution across methods.LanPaint and PILOT [19] are the closest evaluated vanilla-SD1.5 baselines, although their native routes differ.Source: §2.2 · p. 2
  • Step-PI uses task-specific boundary–interior feedback on generic SD1.5, while its field-identical ladder isolates persistent memory and the joint effect of its release schedule.LanPaint and PILOT provide same-foundation external positioning, not field-identical component comparisons.Source: §2.2 · p. 2

3. Preliminaries: Sequential-Control View

The paper uses deterministic DDIM sampling as a controlled reverse-trajectory substrate and frames PI-inspired control as current feedback plus historical state. Its finite-horizon interface separates when feedback is computed from when control is applied.

3.1. Latent Diffusion and DDIM Sampling

  • Deterministic DDIM reuses a pretrained noise predictor for a non-Markovian reverse trajectory and adds no per-step noise, supporting controlled feedback comparisons.The paper does not claim superiority over stochastic DDPM.Source: §3.1 · p. 2

3.2. PI-Inspired Stateful Feedback

  • The controller borrows PI’s current-feedback/history decomposition, using normalized boundary–interior directions and clipped, exponentially discounted histories rather than textbook raw-error integration.Semantics come from the frozen text-conditioned transition; the controller regularizes spatial and contextual consistency rather than directly optimizing text alignment.Source: §3.2 · p. 3

3.3. Finite-Horizon Control Lens

  • Feedback is formed from the current clean estimate, while the bounded action is applied to the post-transition latent before the next projection.This order does not identify current-latent feedback with a next-state objective gradient or assume cross-timestep gradient transport.Source: §3.3 · p. 3

4. Methodology

Step-PI is a closed-loop latent controller for frozen text-to-image diffusion, combining boundary–interior feedback, persistent state, and a predefined release schedule. The method fixes its configuration across domains and uses matched controls to separate controller additions.

4. Methodology

  • Step-PI repurposes frozen text-to-image diffusion for an observed image, unknown-region mask, and fixed text condition, without weight updates or learned inpainting-specific conditioning channels.Image and mask determine projection and feedback; text enters the frozen transition.Source: §4 · p. 3, Figure 2 · p. 3

4.1. Problem Setup and Closed-Loop Interface

  • The closed-loop setup projects known context before a frozen transition, then computes feedback from the projected latent and applies a masked action to the post-transition latent.Final RGB compositing, rather than latent projection alone, restores known pixels exactly.Source: §4.1 · p. 3

4.2. Boundary–Interior Feedback

  • Boundary and interior objectives yield separate normalized feedback directions: one targets local stitching, while the other encourages broader contextual and structural coherence.The frozen text-conditioned transition supplies semantics; the controller regularizes spatial and contextual consistency rather than directly optimizing text alignment.Source: §4.2 · pp. 3–4

4.3. Persistent PI-Structured State

  • PI and Step-PI carry separate zero-initialized boundary and interior states, retaining discounted directional history alongside current feedback.The state update is a per-step bounded compromise; it does not establish trajectory-level contraction, stability, or task benefit.Source: §4.3 · p. 4

4.4. Predefined Release Schedule and Matched Controls

  • Step-PI’s predefined four-field schedule modulates controller signals along a fixed 50-step DDIM trajectory, and PI differs by using uniform release.The schedule was selected on Main35-disjoint CelebA-HQ pilots and frozen before evaluation; its timing rationale is not an optimality claim.Source: §4.4 · pp. 4–5

5. Experiments

The experiments evaluate Step-PI with a frozen SD1.5 backbone under fixed configurations, testing transfer, controller components, external comparisons, and inference cost. Step-PI leads the matched internal variants across three datasets, while comparisons with external methods remain subject to their differing native routes and resource regimes.

5.1. Evaluation and Transfer Protocol

  • The 3,500-case Main35 evaluation holds images, masks, prompts, and seeds fixed within comparisons; configurations developed on CelebA-HQ pilots transfer unchanged to AFHQ and Places2.A fixed prompt-presence intervention on 175 cases favors Correct over Empty, with a 95% CI excluding zero.Source: §5.1 · p. 5, §5.2 · p. 6

5.2. Three-Domain Frozen-Backbone Inpainting

  • Step-PI is best in all 15 dataset–metric cells across AFHQ, CelebA-HQ, and Places2 among four matched variants, improving monotonically over DDIM-Proj, P-Guidance, and PI.Table 1 reports the Main35 internal results; AFHQ and Places2 are transfer-only targets evaluated without retuning.Source: §5.2 · p. 6, Table 1 · p. 7

5.3. Matched Mechanism Evidence

  • In field-identical Main35 comparisons, adding persistent state and then the release schedule improves all 15 aggregates, with bootstrap intervals excluding zero across all five metrics for each comparison.The schedule’s five-field sensitivity probe finds that uniformizing any field worsens all five metrics; five-seed audits also support robustness of the state and schedule effects.Source: §5.3 · p. 6, Figure 4 · p. 7

5.4. External Positioning and Computational Cost

  • Step-PI leads LanPaint and PILOT on all five equal-dataset macro metrics, while trained systems retain absolute metric leads and Step-PI costs 30.48 s/image.External comparisons use native routes and are not compute-normalized; trained references rely on offline specialization, while Step-PI uses two feedback gradients per step.Source: §5.4 · pp. 7–8, Table 2 · p. 8

6. Conclusion

Step-PI repurposes frozen SD1.5 for training-free inpainting across three evaluated domains, transferring unchanged from CelebA-HQ development to AFHQ and Places2. Matched component controls improve all 15 Main35 dataset–metric cells, while inpainting-trained systems retain absolute metric leads.

6. Conclusion

  • Step-PI transfers unchanged from disjoint CelebA-HQ development to AFHQ and Places2, with matched state-only and release-only controls improving all 15 Main35 dataset–metric cells.The controller combines boundary–interior feedback, persistent state, and a predefined release schedule without weight updates or learned inpainting-specific conditioning channels.Source: §6 · p. 8
  • Descriptively, Step-PI leads LanPaint and PILOT on all five equal-dataset macro metrics and approaches selected inpainting-trained references on some metrics.Inpainting-trained systems retain the absolute metric leads but rely on substantial inpainting-specific offline optimization.Source: §6 · p. 8
  • The conclusion positions test-time latent control as a complementary route for repurposing a frozen generic text-to-image prior, despite substantial inference cost.Source: §6 · p. 8

Supplementary Material

The supplement follows the main paper’s evidence chain, organizing implementation, evaluation, mechanism tests, qualitative evidence, and computational positioning across Appendices A–E.

Supplementary Material

  • Appendix A gives implementation details, effective controller equations, the predefined release schedule, and persistent-state characterization.Source: Supplementary Material · p. 11
  • Appendix B records shared evaluation and reproduction protocols, while Appendix C pairs mechanism and text-only protocols with five-metric and prompt-alignment evidence.Source: Supplementary Material · p. 11
  • Appendix D provides qualitative evidence and stress-test diagnostics, and Appendix E reports computational costs, external positioning, and claim boundaries.Source: Supplementary Material · p. 11

A. Method Implementation Details and State Characterization

The appendices specify the controller’s effective implementation and characterize its state and safeguards, then document the shared evaluation stack and fixed inputs used in the studies.

A.1. Leakage-Free Projection and Frozen Transition

  • Visible pixels follow a fixed forward-noise trajectory during sampling, with latent projection reimposing known-region context and final RGB compositing restoring known pixels exactly.The context fill excludes hidden RGB values, and hidden-region reference pixels are not used.Source: §A.1 · p. 12

A.2. Boundary–Interior Feedback Objectives

  • Boundary objectives include seam pairing, support-masked variation, and RGB-gradient matching; the variation term also includes differences across the inner-ring support edge.The interior objective uses radii 16, 32, and 64; objective directions are normalized at the current projected latent before actuation at the post-transition latent port.Source: §A.2 · pp. 12–13

A.3. Effective Persistent State and Bounded Action

  • PI and Step-PI update both states at every reverse step, retaining EMA-like, exponentially discounted histories of normalized feedback directions.The PI-structured label refers to current-versus-persistent-history feedback, not integration of raw stationary error; release affects actuation but does not skip feedback or state updates.Source: §A.3 · pp. 13–14

A.4. Predefined Release Schedule

  • Step-PI applies four hand-specified, frozen release fields along the evaluated trajectory; schedule changes act on controller signals before shared safeguards.The complete-schedule comparison does not isolate temporal variation from its overall actuation profile or establish schedule optimality.Source: §A.4 · p. 14

A.5. Effective Persistent-State Recurrence

  • The effective recurrence yields a per-step bounded compromise toward the current weighted direction, but does not establish trajectory-level stability, task benefit, or descent.Under the evaluated configuration, three retained safeguards reduce to the identity, while boundary norm projection, CapNorm, and the final elementwise action clamp may remain nonidentity.Source: §A.5 · p. 15

B. Shared Evaluation and Reproduction Protocol

  • The shared protocol fixes configurations, case lists, seeds, model versions, and evaluation procedures; the stated reproduction package includes implementation and analysis materials.The supplement separates shared evaluation settings from the method specification and places study-specific protocols alongside their results.Source: B. Shared Evaluation and Reproduction… · p. 15

B.1. Frozen Inputs, Text Conditions, and Internal Inference Stack

  • Internal comparisons use fixed 512×512 images and pre-evaluation text conditions, with the same frozen SD1.5 stack and 50-step deterministic DDIM route.Prompts support a claim of prompt-presence responsiveness, not fine-grained masked-content control, instruction following, or paraphrase robustness; deterministic DDIM does not guarantee bitwise-identical GPU arithmetic.Source: §B.1 · pp. 15–16

B.2. Metrics, Estimands, and Conditional Statistical Scope

The section defines the five metric probes, equal-dataset aggregation, and conditional scope of the paired analyses. It also bounds transfer claims to the evaluated SD1.5 configuration and domains.

B.2. Metrics, Estimands, and Conditional Statistical Scope

  • Masked LPIPS evaluates the full-frame composite after exact known-region compositing, while Boundary L1 and Boundary LPIPS probe narrow and wider seam neighborhoods, respectively.The five metrics cover reconstruction, local seam behavior, and a no-reference quality proxy, but do not establish human preference or fine-grained semantic compliance.Source: §B.2 · p. 16
  • Main35 averages cases within protocols, protocols equally within datasets, and the three datasets equally; its intervals are conditional on fixed protocols, evaluated domains, and per-case seeds.Separate five-seed audits do not estimate uncertainty over unseen datasets, mask families, backbones, or samplers.Source: §B.2 · p. 17

B.3. Configuration Development and Transfer Scope

  • The reported transfer reuses CelebA-HQ-developed settings unchanged on AFHQ and Places2 within one fixed SD1.5, VAE, resolution, and DDIM stack.Transfer across backbones, VAEs, resolutions, samplers, or step counts was not evaluated; configuration locking also does not establish hyperparameter optimality or insensitivity to the pilot set.Source: §B.3 · p. 17

B.4. External Method Attributes and Native Routes

This block documents the frozen native routes used for external comparisons and reports matched evidence on persistent state and release scheduling. It also tests mechanism robustness across inference initializations on the fixed MS175 cohort.

B.4. External Method Attributes and Native Routes

  • LanPaint and PILOT use frozen vanilla SD1.5 weights on distinct native routes; their reported configurations are fixed, reproducible operating points.The comparison does not rank alternative LanPaint configurations, and Step-PI, LanPaint, and PILOT were not given a common tuning budget.Source: §B.4 · p. 17, Table 4 · p. 18, Table 5 · p. 18

C. Matched Mechanism, Robustness, and Text-Condition Evidence

  • The appendix extends matched Main35 evidence with five-metric results, five-seed mechanism studies, release interventions, and a matched prompt-presence test.The analyses use frozen inputs and the common evaluation protocol specified in Appendix B.Source: C. Matched Mechanism, Robustness, and… · p. 18

C.1. Matched Main35 State and Joint-Release Effects

  • All five direction-normalized mean effects favor PI, with 95% intervals excluding zero in the Main35 matched comparison.Pooled per-case win rates range from 57.03% to 67.51%, so improvement is broad but not universal.Source: Table 6 · p. 18, §C.1 · p. 18

C.2. Five-Seed Matched Mechanism Studies on MS175

  • Across five paired inference-seed sweeps on MS175, persistent-state effects favor PI on all five metrics, with every 95% interval excluding zero.For joint release, all 25 seed–metric and all 15 dataset–metric point estimates favor Step-PI; 14 of 15 dataset–metric intervals exclude zero, with Places2 CLIP-Q the exception.Source: §C.2 · p. 19, Table 9 · p. 20, Table 11 · p. 20

C.3. Five-Metric Conditional Sensitivity of the Predefined Release Schedule

Conditional interventions compare the full predefined release schedule with versions in which one field is uniformized. A separate matched text-only intervention tests whether the user-provided condition affects outputs, while qualitative examples provide visual context.

C.3. Five-Metric Conditional Sensitivity of the Predefined Release Schedule

  • These schedule interventions measure conditional sensitivity, not whether the four-field schedule is minimal or globally optimal.The fields are not an independent additive decomposition, including because qk and rI(tk) are serial interior scalings.Source: §C.3 · p. 20
  • All 20 one-field contrasts favor full Step-PI, with intervals excluding zero and 175/0/0 paired case wins/ties/losses.Each intervention uniformizes one field while the other three retain their predefined trajectories.Source: §C.3 · p. 20, Table 12 · p. 21

C.4. Prompt-Presence Responsiveness Under a Text-Only Intervention

  • The Correct condition yields higher paired Prompt-CLIP alignment than Empty in the equal-dataset aggregate, with positive effects in all three domains and all four 95% intervals excluding zero.Prompt-CLIP measures supplied-text alignment, not the no-reference Main35 CLIP-Q metric.Source: §C.4 · p. 21, Table 13 · p. 21

D. Qualitative Evidence and Stress-Test Diagnostics

  • The qualitative section presents construction-ladder inspection, strict-random prompt examples, native-route comparisons, and cross-domain stress-test diagnostics as visual counterparts to quantitative evidence.Its grids retain the same source image and mask across methods, with case identifiers and captions delimiting the intended reading.Source: D. Qualitative Evidence and… · p. 22

D.1. Random Construction-Ladder Inspection

The appendix uses randomly selected examples to make the construction ladder, prompt-presence cases, native-route systems, and cross-domain stress cases inspectable. These plates are qualitative companions to matched quantitative analyses, not substitutes for them.

D.1. Random Construction-Ladder Inspection

  • The three-case construction ladder traces DDIM-Proj → P-Guidance → PI → Step-PI under the same frozen backbone, without method-specific case reselection.The matched analyses, rather than a presumed visual monotonic ordering, support conclusions about persistent state and predefined release.Source: Figure 6 · p. 22, §D.1 · p. 22

D.2. Strict-Random Prompt-Presence Examples

  • Six strict-random cases show Original, Masked, Correct, and Empty columns, with only the positive text condition differing between Correct and Empty.Original is evaluation-only reader context, unavailable during inference; the aggregate and per-dataset estimates are reported separately in Appendix C.Source: Figure 7 · p. 23

D.3. Random Native-Route System Comparisons

  • Random native-route plates compare methods on shared row-specific images and masks, but provide descriptive complete-system inspection rather than intervention-matched attribution or population-frequency estimates.The held-out Original is reference context, not necessarily the unique valid completion.Source: §D.3 · p. 24, Figure 8 · p. 24, Figure 9 · p. 25

D.4. Cross-Domain Stress-Test Diagnostics

  • The separate six-case stress-test plate retains the complete active comparison lane to inspect challenging structural, semantic, and blending behaviors, not their prevalence.The cases include disconnected or thin masks, central occlusions, and long or repeated structures; these are observed case attributes, not experimentally isolated causes.Source: Figure 11 · p. 27, §D.4 · p. 27

E.1. 35-Case Runtime, Memory, and Structural Compute Audit

The audit measures complete-route cost on 35 protocol-selected cases using one A100, while separately comparing matched feedback-enabled routes and describing external quality positioning. Its scope is limited to the evaluated configuration and reproduction protocol.

E.1. 35-Case Runtime, Memory, and Structural Compute Audit

  • Step-PI requires a median 30.48 s/image [30.47, 30.50] and 15.55 GiB peak memory; versus DDIM-Proj, its overhead is +27.41 s/image [27.39, 27.44] and runtime ratio is 9.94× [9.93, 9.95].The feedback-enabled routes add 100 autograd calls to the same 50 UNet calls; the audit does not attribute wall-clock time to an individual operation.Source: §E.1 · p. 28, Table 14 · p. 28

E.2. Descriptive Native-Route Quality Positioning

  • Step-PI leads LanPaint and PILOT on all five equal-dataset macro metrics, although its point estimates are not better in every dataset–metric cell.It has better point estimates than LanPaint in 13 of 15 cells and than PILOT in 12 of 15; different native routes and compute budgets make this descriptive system-level positioning.Source: §E.2 · p. 29, Table 16 · p. 29

E.3. Reproduction Scope and Numerical Execution

  • The transfer claim is limited to one frozen vanilla-SD1.5 stack: 512×512 inputs, the original VAE, and 50-step DDIM, with AFHQ and Places2 reached without retuning.Protocol-level reproduction means recovering statistical conclusions, not byte-identical outputs across independent GPU executions.Source: §E.3 · pp. 29–30

Related papers

Loading 2609.00862v1…