PaperlayerView exact revision on arXiv

Source-linked AI summary

ProgressNet: Sketching and Prompting with a Frozen Text-to-Image Model

Arkaprabha Basu, Chaitat Utintu, Yi-Zhe Song

arXiv:2610.03512v1cs.CV

TL;DR

Image generators typically process a finished sketch in one pass, while interactive editors do not provide fast stroke-level control. ProgressNet lets a frozen text-to-image model follow additions, erasures, and prompt revisions, with reported fidelity and coherence across progressive sketch completion.

  • Problem

    Existing sketch-conditioned generators treat each sketch as a finished input, while interactive editing systems lack a fast pathway for stroke changes and erasures.

    Source: §1 · p. 2

  • Method

    ProgressNet uses a frozen text-to-image model with Previous-Concept Memory, Layer-Selective K/V Injection, and Banded Adaptive Control for progressive drawing and prompt revision.

    Source: §1 · p. 2

  • Results

    On FS-COCO, FLUX+ControlNet FID doubles between 10% and 100% completion while ProgressNet’s moves by under one point; it also leads or ties in DINOv2 at nearly all completion levels and has the highest mean PFC-DINO/PFC-LPIPS on FS-COCO and Sketchy.

    Source: §1 · p. 3, §5.1 · pp. 6–7

  • Takeaways & Limitations

    ProgressNet supports progressive sketch-and-text generation across three sketch domains, and users prefer it over five competitors, most widely on erasure.

    Source: §1 · p. 3

    ProgressNet is primarily evaluated with sketch-based control; extending its mechanisms to depth and pose remains unexplored.

    Source: §6 · p. 9

Abstract

from arXiv · show

Humans draw progressively: a few strokes, a look at the result, a stroke erased, a prompt revised. Image generators do not work this way. They typically take a finished sketch and produce the image in a single pass, so every edit starts the picture again, and the models that do keep state across turns are driven by text, cannot take a stroke, and are too slow to draw with. We present ProgressNet, a training-free framework that lets a frozen text-to-image model follow a drawing session as it unfolds: strokes are added and erased, the prompt is revised, and the image keeps up at about a second per turn. It needs no new parameters because the frozen model already has what a progressive generator needs, a pathway through which the previous turn can be remembered, layers that can carry appearance forward without freezing structure, and an internal signal of how far to trust an unfinished sketch; three inference-time mechanisms (Previous-Concept Memory, Layer-Selective K/V Injection and Banded Adaptive Control) use each in turn. As a sketch fills in, every existing method degrades, the FID of the FLUX+ControlNet baseline doubling between 10% and 100% completion on FS-COCO, while ProgressNet's barely moves; it maintains strong fidelity and progressive coherence across three sketch domains and is preferred by users over five competitors, most widely on erasure.

1 INTRODUCTION

ProgressNet targets drawing sessions in which strokes and prompts evolve, unlike generators that process a finished sketch once or cannot support fast stroke-level interaction. It uses three inference-time mechanisms to preserve prior appearance while adapting to edits, with strong reported progressive results.

1 INTRODUCTION

  • Existing sketch-conditioned systems regenerate statelessly, while interactive editors use text-driven turns or latent optimization that is too slow for drawing.ProgressNet instead builds on a frozen flow-matching backbone with ControlNet and few-step inference.Source: §1 · p. 2
  • Previous-Concept Memory feeds back the prior output and prepares it for additions, erasures, or prompt updates; LS-KVI carries appearance through less vital layers while leaving structure responsive.Banded Adaptive Control adjusts memory strength across layer bands and solver steps using changes in cached ControlNet memory features.Source: §1 · p. 2
  • Pilot studies show temporal drift, partial-stroke artifacts, and identity shifts after stroke removal for existing methods, while ProgressNet remains stable across turns.Figure 2 also contrasts raw-stroke-edge imprinting by FLUX+ControlNet with ProgressNet’s smoother adaptation and temporal coherence.Source: Figure 2 · p. 2, Figure 3 · p. 3
  • On FS-COCO, FLUX+ControlNet FID doubles from 10% to 100% sketch completion while ProgressNet’s moves by under one point.Every baseline degrades as sketches fill in; users prefer ProgressNet over five competitors, most widely on erasure.Source: §1 · p. 3

2 RELATED WORK

Prior controllable-generation work spans several ways to steer images, but the cited approaches operate single-shot rather than tracking changing sketch–text conditions across turns. ProgressNet addresses that setting with temporal memory, selective feature reuse, and adaptive control.

2 RELATED WORK

  • Controllable image generation includes residual conditioning, multi-condition control, lightweight adapters, layout grounding, and training-free feature steering.These approaches have also been brought to DiT and rectified-flow backbones.Source: §2 · p. 3
  • Inversion-based editing, attention injection, and few-step distillation improve editability and latency, bringing sketch interaction near real time.Other work mitigates text–sketch conflict through cross-attention re-weighting or control-residual rescaling.Source: §2 · p. 3
  • The cited methods operate single-shot: none tracks sketch–text conflict across turns or preserves prior content as conditions change.ProgressNet targets this non-stationary setting through training-free temporal memory, selective feature reuse, and ControlNet-driven adaptive control.Source: §2 · p. 3

3 PRELIMINARIES

ProgressNet builds on flow-matching backbones with Union ControlNet, whose attention and residual pathways support feature reuse and spatial conditioning. The framework addresses progressive-state failures with operation-specific memory preparation and adaptive control.

3 PRELIMINARIES

  • ProgressNet uses a flow-matching backbone with Union ControlNet, exposing joint-attention K/V features for LS-KVI and residual conditioning for spatial inputs.Control strength λ is critical under few-step inference because over- or under-conditioning cannot be corrected across many denoising steps.Source: §3 · p. 3
  • Pilot studies identify appearance drift, fixed-strength control that may over-imprint partial strokes or under-enforce them, and persistence of erased content.ProgressNet addresses these failures with explicit concept memory, selective temporal feature reuse, and adaptive control modulation.Source: §3 · p. 3
  • The overview prepares prior output by operation, conditions Union ControlNet on memory and the current sketch, and reuses prior-turn K/V features at six low-vitality FLUX layers.Removal white-fills the erased region; a prompt update gray-fills the uncommitted region.Source: Figure 4 · p. 4

4 PROPOSED METHODOLOGY

ProgressNet generates each turn from the current sketch, prompt, and previous output while keeping pretrained weights frozen. It combines operation-aware memory, selective temporal feature reuse, and adaptive control.

4.1 PREVIOUS-CONCEPT MEMORY (PCM)

  • PCM turns the previous output into operation-aware RGB memory: additions retain it, erasures white-fill erased regions, and prompt updates keep committed content while relaxing uncommitted regions.The sketch determines what changes, while PCM determines what remains, without an extra learned module.Source: §4.1 · p. 4

4.2 LAYER-SELECTIVE K/V INJECTION (LS-KVI)

  • LS-KVI leaves vital layers without direct temporal K/V injection and reuses features at selected lower-vitality layers, preserving appearance while structure follows the sketch.Layer vitality is measured by bypassing each layer; lower scores indicate smaller output deviations in the progressive setting.Source: Figure 5 · p. 5, §4.2 · p. 5

4.3 BANDED ADAPTIVE CONTROL (BAC)

  • BAC modulates memory conditioning by band and solver step, using the latest complete transition as a proxy for recent feature redistribution.Greater recent redistribution indicates a less stable appearance state and retains stronger memory conditioning; stable representations permit attenuation.Source: §4.3 · pp. 5–6

5 EXPERIMENTS

ProgressNet is evaluated across three sketch domains for generation quality, progressive stability, qualitative editing, user preference, and component contributions. It maintains strong semantic fidelity and progressive coherence, while user studies and ablations examine its interaction quality and mechanisms.

5 EXPERIMENTS

  • Evaluation uses FS-COCO, Photo-Sketching, and Sketchy, with FLUX.1-schnell and Union ControlNet at 1024×1024 resolution and two denoising steps.The reported setup includes quality measurements at completion stages and stability across representative transitions.Source: §5 · p. 6

5.1 QUANTITATIVE RESULTS

  • ProgressNet has the highest mean PFC-DINO/PFC-LPIPS on FS-COCO and Sketchy, and the highest final-transition scores across all three sketch domains.Across completion levels and domains, DINOv2 is highest or tied-highest at nearly all levels, with competitive CLIP/T5 alignment.Source: §5.1 · pp. 6–7, Table 2 · p. 7, Table 1 · p. 7

5.2 QUALITATIVE RESULTS

  • ProgressNet preserves appearance and scene context during progressive sketching, remains coherent after stroke removal, and supports localized edits while maintaining global consistency.The qualitative comparisons include progressive giraffe and zebra sketches, fine-grained edits, and open-ended creative generation.Source: §5.2 · pp. 7–8, Figure 7 · p. 8, Figure 8 · p. 8

5.3 USER STUDY

  • ProgressNet is consistently preferred over all baselines across preference, temporal coherency, generation quality, erasure, and prompt editing, with its largest share on erasure.StableFlow is the strongest competitor on coherency and fine-drawing.Source: §5.3 · p. 9, Table 3 · p. 9

5.4 ABLATION STUDIES

  • PCM supplies the dominant improvement in reference and joint fidelity; at full completion, FID-I decreases from 67.9 to 36.6 and DINOv2 increases from 0.361 to 0.414.PCM establishes the appearance memory on which the remaining components are evaluated.Source: §5.4 · p. 9, Table 4 · p. 9
  • Adding LS-KVI to PCM maintains comparable DINOv2 and text alignment, with a trade-off in FID-I and FID-C.The illustrated non-vital-layer configuration better preserves appearance than the vital-layer configuration; clearing stale K/V enables stronger recovery after stroke removal.Source: §5.4 · p. 9, Table 4 · p. 9
  • Adding BAC improves full-completion FID-C from 0.162 to 0.143 and CLIP/T5 from 0.300/0.653 to 0.303/0.655.BAC also reduces visible sketch-edge artifacts under conflicting prompts or imperfect strokes by adapting appearance-memory conditioning strength.Source: §5.4 · p. 9, Figure 9 · p. 9, Table 4 · p. 9

6 CONCLUSION AND LIMITATIONS

The conclusion presents ProgressNet as a training-free framework for stabilizing progressive sketch-and-prompt interactions. It identifies sketch-based evaluation and predefined BAC controls as boundaries, with broader modality support and automatic control-sensitive region selection left for future work.

6 CONCLUSION AND LIMITATIONS

  • ProgressNet combines PCM and LS-KVI for temporal consistency with BAC, which uses retrospective memory-feature divergence to adapt memory.The framework improves fidelity and alignment across completion levels with near-real-time performance.Source: §6 · p. 9
  • The evaluation is primarily sketch-based; extending memory and adaptive control to modalities such as depth and pose remains unexplored.Source: §6 · p. 9
  • BAC uses predefined layer bands and modulation bounds; automatically identifying control-sensitive regions for different conditioning modalities remains future work.Source: §6 · p. 9

A.1 DATASET AND PROGRESSIVE EVALUATION PROTOCOL

The protocol evaluates fixed, shared progressive sketch inputs across three datasets and completion stages. Its metrics summarize quality-adjusted temporal coherence and whether generation quality remains strong and stable across completion levels.

A.1 DATASET AND PROGRESSIVE EVALUATION PROTOCOL

  • The evaluation contains 11,750 samples from FS-COCO, Photo-Sketching, and Sketchy, with identical ordered sketch sequences and text conditions supplied to every method.Progressive states are evaluated at 10, 20, 30, 50, 70, 80, and 100% completion.Source: §A.1 · p. 15, Table 5 · p. 15

A.2 PROGRESSIVE EVALUATION METRICS

  • PFC evaluates temporal coherence jointly with distributional quality, using representative transitions associated with FID-I anchors at 10%, 50%, and 100% completion.PFC-DINO and PFC-LPIPS are higher-is-better quality-adjusted summaries rather than pure temporal-coherence measures.Source: §A.2 · p. 15
  • Cross-completion robustness measures whether standard generation metrics remain strong and stable across the 10%, 50%, and 100% anchors; scores are normalized to the FLUX baseline, where 100 denotes that baseline.Values above 100 indicate stronger quality-adjusted cross-completion robustness.Source: §A.2 · p. 16

A.3 EXTENDED QUANTITATIVE RESULTS

The extended FS-COCO sweep shows how distributional quality and text alignment evolve across sketch completion levels, with ProgressNet matching or leading from 20% onward on several reported measures.

A.3 EXTENDED QUANTITATIVE RESULTS

  • Tables 6 and 7 extend the method comparison and component ablation to intermediate levels, checking whether gains at 10%, 50%, and 100% hold across the sweep.Table 6 covers seven completion levels; Table 7 extends the PCM, LS-KVI, and BAC ablation.Source: §A.3 · p. 16, Table 6 · p. 17, Table 7 · p. 17
  • From 20% completion onward, ProgressNet matches or leads on FID-I and FID-C, while its FID-I is 36.5 versus 32.9 for the Flux Baseline at 10%.At 10%, sketches are highly sparse and provide limited structural constraint.Source: §A.3 · p. 16
  • From 20% completion onward, ProgressNet consistently leads or matches on reported text-alignment metrics, after StableFlow narrowly leads T5-Cos at 10%.At 50%, T5-Cos is tied at 0.656; at full completion ProgressNet leads 0.655 to 0.654, and it leads at every intermediate level.Source: §A.3 · p. 17

A.4 INTERPRETING THE ABLATION PROGRESSION

The full PCM + LS-KVI + BAC system maintains strong FID-I while improving FID-C and text alignment at later completion stages.

A.4 INTERPRETING THE ABLATION PROGRESSION

  • From 30% completion onward on FS-COCO, the full system obtains the strongest FID-C and remains at or above other ablations on CLIPScore and T5-Cos.Its FID-C values from 30% to 100% are 0.144, 0.141, 0.142, 0.141, and 0.143.Source: §A.4 · p. 17
  • Across later completion stages, the full system maintains strong FID-I while improving FID-C and textual alignment.The reported interpretation combines distributional quality and text-alignment metrics rather than treating temporal similarity alone as sufficient.Source: §A.4 · p. 17
  • The reported metrics cover distributional quality in Inception and CLIP image-feature spaces, paired visual-semantic fidelity, and text alignment.FID-I and FID-C measure distributional quality; DINOv2 measures paired visual-semantic fidelity, while CLIPScore and T5-Cos assess text alignment.Source: §A.4 · p. 17

A.5 INFERENCE AND REPRODUCIBILITY DETAILS

The appendix records the live inference configuration and sketch preprocessing, then details how Previous-Concept Memory builds and applies its commitment mask.

A.5 INFERENCE AND REPRODUCIBILITY DETAILS

  • Live progressive generation uses two inference steps in bfloat16 on one NVIDIA H200 GPU, with FLUX.1-schnell and Union ControlNet at 1024 × 1024 resolution.Structural and memory-conditioning scales are fixed at λs = 0.5 and λm = 0.65 unless otherwise stated.Source: §A.5 · p. 17
  • Sketch preprocessing converts the line drawing to a thresholded, inverted binary structural map and resizes it to the ControlNet input resolution.The grayscale threshold is 200, with nearest-neighbour resizing to 1024 × 1024.Source: §A.5 · p. 18
  • Across four FS-COCO examples, non-vital-layer K/V injection preserves appearance, lighting, and identity better through progressive completions than vital-layer injection.The figure contrasts the two approaches at 20%, 50%, 80%, and 100% completion.Source: Figure 10 · p. 18

B.1 PREVIOUS-CONCEPT MEMORY IMPLEMENTATION

  • PCM derives a commitment mask from the processed sketch: committed regions retain prior output, while prompt updates relax uncommitted areas and erasures withdraw evidence through neutral white filling.Added strokes use the previous output unchanged as memory.Source: §B.1 · p. 18
  • The soft commitment mask takes the pixelwise maximum of stroke-proximity and enclosed-region supports, with fixed construction parameters across reported experiments.The proximity support is dilated by 24 pixels and smoothed with a 41 × 41 kernel; enclosed regions use morphological closing, flood filling, and 21 × 21 smoothing.Source: §B.1 · p. 19

B.2 CHOICE OF VITAL AND NON-VITAL LAYERS

The appendix reports how layer choice affects appearance and erasure recovery, and validates banded adaptive control across solver steps, layers, and backbones.

B.2 CHOICE OF VITAL AND NON-VITAL LAYERS

  • Non-vital-layer injection preserves lighting, identity, and color while allowing structural adaptation; the six lowest-vitality FLUX layers are selected for LS-KVI.The selected layers are {5, 6, 7, 10, 13, 14}, with αkv = 0.3; the subset is fixed across reported datasets.Source: §B.2 · p. 19
  • Invalidating stale LS-KVI state during a controlled add–erase cycle reduces return LPIPS by 30.9% and improves recovery in 93 of 100 sequences.It also reduces erased-region error in 91 of 100 cases; the paired Wilcoxon p-value is 7.1×10^-15.Source: §B.2 · pp. 19–20

C EMPIRICAL VALIDATION OF BANDED ADAPTIVE CONTROL

  • Across progressive stages, measured control-induced perturbation profiles vary with backbone depth and solver step, motivating BAC’s band- and step-specific design.In standard Diffusers FLUX, ControlNet residual samples are distributed across backbone blocks rather than injected only into blocks ℓ0-4.Source: Figure 11 · p. 20, §C.1 · p. 20

C.2 CROSS-BACKBONE VALIDATION ON SD3.5 LARGE TURBO

  • On SD3.5, late-layer redistribution remains stronger despite its different MMDiT architecture and separate Canny and blur ControlNets.The inferred band boundary differs by modality: Canny separates at layer 11/19, while blur-memory separates at layer 5/19.Source: §C.2 · p. 20

C.1 LAYER- AND STEP-DEPENDENT CONTROL RESPONSES

  • On the out-of-distribution Pixelor sketch, channel-wise JSD yields smoother modulation than cosine similarity, while KL and standard JSD show stronger distortions at high completion.Without BAC, visible sketch-edge artifacts remain against a solid background.Source: §C.1 · p. 20, Figure 12 · p. 21

C.2 CROSS-BACKBONE VALIDATION ON SD3.5 LARGE TURBO

  • Across four sketch domains, band asymmetry persists at evaluated transitions, step asymmetry remains with content-dependent magnitude, and transition ranking is preserved.The tables report late/early JSD ratios, step-1/step-0 amplification, and transition rankings, respectively.Source: Table 11 · p. 22, Table 12 · p. 22, Table 13 · p. 22

C.3 CROSS-DATASET BAC VALIDATION

The supplied sections test whether BAC's signature persists across datasets, detail user-study protocols and related work, bound the Nano Banana comparison, and add qualitative examples.

C.3 CROSS-DATASET BAC VALIDATION

  • Stronger late-band redistribution and solver-step asymmetry persist beyond one sketch domain and backbone.The BAC diagnostic covers four sketch domains totaling 52,368 samples.Source: §C.3 · p. 22

D USER STUDY DETAILS

  • Study 1 compares six methods across 10 samples on preference, coherence and quality; Study 2 assesses erasure, editing, progression, fine-grained drawing and erasure, and content preservation.Both studies were double-blind, with hidden method identities and randomized presentation order.Source: D USER STUDY DETAILS · pp. 22–23

E EXTENDED RELATED WORK

  • Prior control work often assumes sufficiently specified conditions, editing commonly treats each edit as a fresh problem, and low latency alone does not ensure temporal consistency.The related-work discussion distinguishes progression-centric state from generation-centric conditioning.Source: E EXTENDED RELATED WORK · p. 23
  • LS-KVI leaves structurally important layers free to adapt, while BAC derives step- and band-specific control modulation from cached features across turns.The discussion contrasts this approach with preserving all influential layers too strongly and with fixed global control scales.Source: E EXTENDED RELATED WORK · pp. 23–24

F NANO BANANA PRO

  • Nano Banana Pro is a qualitative reference only; its proprietary interface prevents the controlled inference protocol required for quantitative tables.The comparison illustrates interactive sketch adherence, not a controlled benchmark ranking.Source: F NANO BANANA PRO · p. 24

G ADDITIONAL QUALITATIVE EXAMPLES

Related papers

Loading 2610.03512v1…