ToDoPro: Faster and Better Reference-Based Generation: Early Token Reduction via Downsampled Self-Probing

1 INTRODUCTION

• Multiple reference images expand token sequences, increasing latency and peak VRAM through O(N^2) attention and potentially degrading quality through interference and dilution.Existing in-model reduction occurs after the full sequence enters the model, so it does not reduce peak VRAM at the earliest attention layers.Source: §1 · p. 2
• ToDoPro performs a few denoising steps on spatially 2× downsampled references, using the DiT itself to probe token importance before main denoising.High-importance blocks retain all four tokens, low-importance blocks are discarded, and mid-importance blocks are replaced by a downsampled token.Source: §1 · pp. 2–3
• Early reduction applies computational savings across every timestep and layer, mitigates peak VRAM, and complements in-model acceleration without external models or added parameters.The probing design uses the generator itself to align importance estimates with its demands.Source: §1 · p. 3
• Discarding low-importance tokens mitigates attention interference and dilution, allowing the model to focus on prompts, generative tokens, and informative reference tokens.The authors report that this improves overall generation quality.Source: §1 · p. 3
• Across three base models and two benchmarks, ToDoPro delivers up to 3× mean speedups, cuts peak activation VRAM by up to 50%, and consistently improves generation quality.The introduction also reports that filtering low-importance tokens mitigates attention interference and dilution.Source: §1 · p. 3

2.1 REFERENCE-BASED IMAGE GENERATION AND EDITING

• Task-specific reference-generation methods use control encoders, adapters, or condition selection, while general-purpose DiT models concatenate references, text, and generative tokens to support varied tasks and inputs.The task-specific methods rely on tailored architectures or input assumptions; generality in joint-token models comes with rapidly growing token counts and quadratic self-attention costs.Source: §2.1 · p. 3

2.2 TOKEN REDUCTION FOR VISION MODELS

• Vision-model token reduction includes in-model pruning or merging and pre-model compression, including training-free similarity merging and attention-based or external-model importance estimates.Similarity-based merging avoids extra trainable modules but has bounded performance, while precise region-importance methods can require external models.Source: §2.2 · p. 3

2.3 EFFICIENT IMAGE AND VIDEO GENERATION

• Diffusion acceleration includes cross-step feature reuse, distillation, pruning, quantization, and in-model token reduction; efficient attention reduces attention computation but not Feed-Forward Network computation.These approaches address different parts of the generation pipeline.Source: §2.3 · p. 4

2.4 EFFICIENT REFERENCE-BASED GENERATION AND EDITING

• Reference-generation accelerators include methods limited to single-image editing, retrained edge-map pruning, ControlNet reduction, and pruning inside main denoising; early reference-token reduction remained absent.ToDoPro shortens the input sequence before the main DiT run, making it complementary to in-model techniques.Source: §2.4 · p. 4

3.1 PRELIMINARIES AND NOTATIONS

• Reference-based diffusion models jointly process generative, prompt, and reference tokens in an L-layer DiT; attention-pruning rules from VLMs do not transfer trivially to this generation task.Reference tokens must continuously provide fine-grained, low-level spatial details for generation.Source: §3.1 · p. 4, §3.2 · p. 4

3.2.1 REFERENCE UTILIZATION SCORE

• The reference utilization score measures attention from a query set to a reference token, and generative plus prompt tokens yield the best overall alignment with oracle importance.The oracle is defined using the gradient magnitude of final decoded pixels with respect to each reference token.Source: §3.2.1 · p. 5

3.2.2 LAYER-WISE ANALYSIS OF UTILIZATION SCORE

• Attention differs across layers: middle layers concentrate on prompt-requested salient regions, while final layers largely shift toward refining generative tokens.Middle layers correlate better with oracle importance than early and late layers, but single-layer scores remain susceptible to anomalous token spikes.Source: §3.2.2 · pp. 5–6
• Averaging reference utilization scores across layers outperforms any single layer because stochastic attention spikes vary across layers.The authors average all layers to support generalization across base models; middle 56% or 80% averaging also achieves excellent alignment.Source: §3.2.2 · p. 6

3.3 EARLY TOKEN REDUCTION VIA DOWNSAMPLED SELF-PROBING

• Downsampled self-probing precedes main denoising, reducing unimportant reference tokens before they enter the DiT.The separate probe estimates importance using a lightweight denoising loop.Source: §3.3 · p. 6

3.3.1 EXTRACTING IMPORTANCE VIA SPATIAL-TEMPORAL DOWNSAMPLED SELF-PROBING

• The probe uses spatially 2× downsampled images and a temporally sparse denoising schedule to approximate full-resolution reference utilization scores.The probe shares the formal denoising configuration apart from its spatially and temporally downsampled inputs and loop.Source: §3.3.1 · pp. 6–7

3.3.2 REFERENCE TOKEN REDUCTION

• High-importance 2 × 2 blocks retain all four full-resolution tokens, low-importance blocks are discarded, and mid-importance blocks are replaced by one downsampled token.Substitution preserves broader context at one-quarter the token cost of the original block.Source: §3.3.2 · p. 7

4.2 MAIN RESULTS

• Ours(fast) lowers peak VRAM while matching or surpassing baseline quality; Ours(slow) further improves all quality metrics at a smaller speedup.The comparison spans three base models and two benchmarks; Table 1 reports fine-grained Flux.2-Klein-4B scores.Source: §4.2 · p. 8, Table 1 · p. 8, Figure 6 · p. 8

4.3 ORTHOGONALITY WITH IN-MODEL METHODS

• Across in-model acceleration methods, integrating Ours(fast) yields compounded speedups.The comparison includes TaylorSeer, RepreShift, and TransPrune.Source: §4.3 · p. 9, Table 2 · p. 9

4.4 ABLATION STUDY

• Ablations show that using only downsampled references, removing the Low-Res Tier, or random reduction degrades quality.The controlled comparison uses the same reduction ratio on MultiBanana with Flux.2-Klein-4B.Source: §4.4 · p. 9, Table 3 · p. 9

4.5.1 EFFECT OF TOKEN REDUCTION RATIO

• Across a broad 30%–94% reduction window, filtering distractors improves quality, whereas reduction above 95% degrades it by irreversibly losing informative tokens.The optimal window is described as a wide plateau rather than a narrow range.Source: §4.5.1 · pp. 9–10, Figure 8 · p. 9

4.5.2 EFFECTIVE ATTENTION RATIO

• Token reduction raises Effective Attention Ratio (EAR), and averaging utilization scores across layers raises it slightly further than using the best-aligned single layer.The comparison evaluates meaningful reference regions on the MultiBanana object-composition task.Source: §4.5.2 · p. 10, Table 4 · p. 10
• All-layer averaging is comparable to averaging the top 20% of layers and avoids model-specific layer selection.The paper adopts all-layer averaging as its default for generalization across base models.Source: §4.5.2 · p. 10

5 CONCLUSIONS

• ToDoPro is a training-free early token-reduction framework for reference-based diffusion models.It uses the model itself to extract token importance and applies tri-tier reduction before main denoising.Source: §5 · p. 10
• The framework reduces latency and mitigates peak VRAM while compounding with in-model acceleration techniques.These are reported outcomes of the method in the conclusion.Source: §5 · p. 10
• The experiments report improved generation quality alongside efficiency gains.The conclusion summarizes experiments as showing improved generation quality.Source: §5 · p. 10

A.1 THEORETICAL COMPLEXITY ANALYSIS

• The FLOP analysis counts attention, Q/K/V/O projections, and FFN operations, with baseline cost TL C(N) for T denoising steps and L layers.It assumes an FFN intermediate width of 4d and defines the attention contribution as 4N^2d and the projection-plus-FFN contribution as 24Nd^2.Source: §A.1 · p. 16
• During probing, spatial 2× downsampling reduces image-token counts to one quarter, while the denoising schedule is downsampled by factor D.The formal-stage sequence length depends on the retained-reference-token reduction ratio.Source: §A.1 · p. 16
• Break-even ratios are approximately 9.9% for D = 2 and 3.9% for D = 5 in the representative two-reference setting.The calculation uses N = 12,388, N′ = 3,172, and 4,096 generative tokens plus 8,192 reference tokens; the authors say the thresholds are almost always met in practice.Source: §A.1 · p. 17

A.2 ORTHOGONALITY WITH MOST EXISTING ACCELERATION TECHNIQUES

• Early reduction shortens the reference sequence before main denoising, so existing in-model methods can operate on it without changes to internal model structures or operations.The paper calls this “early” rather than “pre-model” reduction because its lightweight self-probing stage has already run the DiT.Source: §A.2 · p. 17
• The method targets spatial reference-token redundancy, complementing methods that address temporal redundancy or prune generative tokens.The authors state that it can be combined with existing acceleration techniques for compounded efficiency improvement.Source: §A.2 · p. 17
• Reference tokens can be reduced once before the main loop or timestep-mapped; the paper defaults to timestep-mapped reduction for a slight performance gain.The one-time option averages reference utilization scores across probing steps.Source: §A.2 · p. 17

A.4 DISCUSSIONS WITH BROADER WORKS

• The tri-tier strategy resembles dynamic-resolution encoding by retaining full-resolution tokens for critical details and substituting downsampled tokens for mid-importance regions.Token importance and the multi-resolution layout are determined by the DiT through training-free downsampled self-probing.Source: §A.4 · p. 17

A.3 RESULTS UNDER VARYING NUMBERS OF REFERENCE IMAGES

• Figure 9 reports score and latency for Ours (slow) on MACRO with 2–10 reference images.The passage describing the figure identifies Flux.2-Klein as the base model.Source: §A.3 · p. 17, Figure 9 · p. 18

A.4 DISCUSSIONS WITH BROADER WORKS

• The method uses a model-level coarse-to-fine process: a coarse probe run determines reference-token sparsity for the subsequent fine run.The passage distinguishes this from approaches that apply coarse-to-fine structures at the attention level.Source: §A.4 · p. 18

B.1 MODELS AND INFERENCE

• Experiments use Flux.2-Klein-4B, Qwen-Image-Edit-2511, and HiDream-O1-Image, with 4, 40, and 50 inference steps, respectively.The first two are latent-space DiTs; HiDream-O1-Image is a pixel-level UiT that processes raw pixels without a VAE. Runs use one NVIDIA H800 GPU.Source: §B.1 · p. 18
• For MultiBanana, references retain native resolution unless their area exceeds 4096 × 4096; MACRO follows its official settings, and all runs use seed 42.The last reference's size sets the target generation size for MultiBanana.Source: §B.1 · p. 18
• Preserved full-resolution tokens retain their original 2D RoPE coordinates, while replacement tokens receive the spatial center of their corresponding 2 × 2 block.Source: §B.1 · p. 18

B.2.1 RATIO-BASED DECISION (DEFAULT).

• The default ratio-based policy assigns the top p1 fraction of scores to the Full-Res Tier, the next p2 fraction to the Low-Res Tier, and discards the rest.Ours(fast) uses p1 = 15% and p2 = 30%; Ours(slow) uses p1 = 30% and p2 = 60%, with a minimum per-reference retention guarantee.Source: §B.2.1 · p. 19
• The default timestep-aware shift uses 0.8p1 Full-Res and 1.2p2 Low-Res ratios early, linearly returning to base ratios during later refinement.The authors report that this adjustment slightly improves generation quality.Source: §B.2.1 · p. 19

B.2.2 THRESHOLD-BASED DECISION.

• Threshold-based selection normalizes scores by the number of downsampled reference tokens, allowing fixed thresholds to allocate more tokens to complex scenes and prune simple references more aggressively.Source: §B.2.2 · p. 19

B.3 DATASETS AND EVALUATION

• MultiBanana contains 3K+ samples with up to eight references and evaluates multiple editing and composition tasks across four difficulty axes.The paper averages task scores across five VLM-judged dimensions: Text-Instruction Alignment, Reference Consistency, Background-Subject Match, Physical Realism, and Visual Quality.Source: §B.3 · p. 19
• MultiBanana scores average two decoding configurations, while local-attribute-transfer samples with IDs ≤200 are excluded because of a confirmed prompt-reference mismatch bug.The maintainers recommended excluding those samples.Source: §B.3 · p. 20
• MACRO covers four tasks, each with 1,000 samples and 1–10 references; the reported quality measure is the average overall score across all samples.Source: §B.3 · p. 20
• Qwen3-VL serves as the VLM-as-judge for both benchmarks, deployed with LMDeploy on an NVIDIA H800.Source: §B.3 · p. 20
• Efficiency reporting includes end-to-end latency, peak activation VRAM across samples, and speedup over the corresponding base model.Latency includes the VAE and probing denoising stage but excludes disk I/O.Source: §B.3 · p. 20

B.4 DETAILS OF COMPARED METHODS

• ToMeSD is applied to reference-image tokens before they enter the DiT.Source: §B.4 · p. 20
• VisPruner scores visual tokens with encoder self-attention, selects important tokens, and fills the remaining budget through cosine-similarity deduplication for diversity.The scoring source varies by base model: the method reuses built-in vision encoders when available and otherwise uses CLIP for Flux.2 Klein.Source: §B.4 · p. 20

B.5 EFFECTIVE ATTENTION RATIO (EAR)

• For EAR, the object-composition task supplies salient-object ground truth via BiRefNet; the metric measures the share of generative- and prompt-token attention landing there across timesteps and layers.BiRefNet's soft saliency map is resized to the reference latent grid.Source: §B.5 · pp. 20–21
• Table 4 contrasts no reduction with utilization scores from a best-aligned single layer and the top 20% best-correlated middle layers.The reduction budget is held constant across the compared approaches.Source: §B.5 · p. 21

• The paper focuses on efficiency losses from token growth when multiple reference images are used.Source: C LIMITATIONS · p. 21
• Pure instruction-based single-image editing has relatively lower acceleration than the multi-reference setting.Source: C LIMITATIONS · p. 21
• Empirical validation is limited to multi-reference image generation, while transfer to ViTs, VLMs, and long-video generation is proposed only in principle.Source: C LIMITATIONS · p. 21