PaperlayerView exact revision on arXiv

Source-linked AI summary

APEX: Asynchronous Prefix Denoising for Extreme Compression

David Dinkevich, Nisan Chiprut, Yoav HaCohen, Dani Lischinski

arXiv:2610.12843v1cs.CVcs.LG

TL;DR

At fixed spatial compression, adding channels improves reconstruction but makes latent generation harder, complicating extreme spatial compression for cheaper latent diffusion. APEX pairs a resolution-aware channel hierarchy with asynchronous denoising and outperforms compared methods across evaluated channel counts and compute budgets.

  • Problem

    At fixed spatial compression, adding channels improves reconstruction but makes latent generation harder, complicating extreme spatial compression for cheaper latent diffusion.

    Source: §1 · p. 1

  • Method

    APEX couples resolution-aware channel prefixes with asynchronous denoising at staggered local times in a fixed-token latent, without an additional explicit information bottleneck.

    Source: §1 · p. 2, §3.1 · p. 4, §4.3 · p. 8

  • Results

    On ImageNet-512 at 64× and 128× compression, APEX outperforms compared methods at every evaluated channel count and beats a 256-token baseline across evaluated guided-inference budgets at matched training compute.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The results make extreme compression competitive under constrained compute while maintaining comparable reconstruction quality without restricting latent information through an additional explicit bottleneck.

    Source: §1 · p. 2, §5 · p. 9

    APEX requires choosing channel boundaries, per-level target resolutions, the stagger gap, and the global-clock distribution; optimal choices may vary across settings.

    Source: §5 · p. 9

Abstract

from arXiv · show

Fewer latent tokens promise cheaper diffusion. Yet aggressive latent compression creates a reconstruction-generation dilemma: adding channels improves reconstruction but makes generation harder. We introduce APEX, short for Asynchronous Prefix denoising for EXtreme compression, which pairs a hierarchical latent space with asynchronous denoising over channel groups. Our autoencoder orders channel prefixes from structure to detail, while one shared denoiser generates them at staggered local times without adding tokens. On ImageNet-512, APEX outperforms existing methods at every evaluated channel count under 64x and 128x compression. At matched training and guided sampling compute, it also outperforms a 256-token model at every evaluated budget, with larger gains at lower compute. Code is available at https://daviddinkevich.github.io/APEX.

1 INTRODUCTION

Extreme spatial compression reduces diffusion tokens but creates a reconstruction–generation dilemma: more channels preserve detail while making generation harder. APEX organizes channel prefixes from structure to detail and denoises them asynchronously, outperforming compared methods across evaluated compression settings and compute budgets.

1 INTRODUCTION

  • Spatial compression shortens the token sequence and can reduce denoiser cost, while shrinking the grid typically requires more channels to preserve reconstruction quality.The paper motivates this trade as a way to permit more optimization and sampling under a fixed compute budget.Source: §1 · p. 1
  • At fixed spatial compression, adding channels improves reconstruction but makes the latent harder to generate, leaving standard latent diffusion to trade detail against generation quality.At 128× compression, a 512² image has only a 4 × 4 grid of 16 tokens.Source: §1 · p. 1
  • APEX orders channel prefixes from structure to detail and uses one shared Transformer to denoise added channel groups at staggered local times, without increasing token count.The autoencoder supervises prefixes at progressively higher resolutions, so later refinements use cleaner preceding context.Source: §1 · p. 2, Figure 1 · p. 2
  • At 64× and 128× compression, APEX outperforms compared methods across evaluated channel counts; at matched training compute, its 64-token model beats a 256-token baseline at every evaluated guided inference budget.At f128c512, APEX reaches 8.98 guided gFID from 16 tokens with SiT-XL, while every compared method remains in the double digits.Source: §1 · p. 2

2 RELATED WORK

Related work reduces latent-diffusion difficulty through information control, pretrained semantic structure, hierarchical representations, and frequency-based methods. APEX instead combines cumulative channel prefixes of one reconstruction latent with resolution-matched supervision and asynchronous generation.

2 RELATED WORK

  • DC-AE 1.5 structures channel prefixes but trains diffusion on randomly sampled prefixes at a shared timestep, while FreqWarm uses a frequency curriculum; APEX pairs resolution-matched levels with staggered local times.The cited passage identifies DC-AE 1.5 and FreqWarm as the closest direct baselines and reports that the components do not transfer successfully in simple combinations (Table 2).Source: §2 · p. 3
  • Other methods control latent information or import pretrained semantic structure, whereas APEX uses no external semantic representation or explicit latent prior and retains image-specific detail in later levels.SFD and Learning When to Denoise pair a DINOv2-derived semantic latent with a separate texture latent and denoise semantics first (Pan et al., 2025; Qian & Cheng, 2026; Oquab et al., 2024).Source: §2 · p. 3
  • Unlike frequency-based and other coarse-to-fine approaches, APEX uses cumulative channel prefixes of one jointly decoded latent on a fixed spatial grid, with each prefix supervised at a corresponding resolution.The passage contrasts this with Denoising-VAE’s progressively low-pass levels and WaiT’s pixel-space wavelet-detail generation.Source: §2 · p. 3

3 METHOD

APEX pairs cumulative channel levels ordered from structure to detail with a shared Transformer that denoises groups at staggered local times. Resolution-aware supervision shapes the hierarchy, while asynchronous channel generation avoids adding spatial tokens.

3.1 RESOLUTION-AWARE AUTOENCODER TRAINING

  • Cumulative channel prefixes define levels, with progressively higher target resolutions as channel capacity increases.At larger widths, intermediate levels keep refinement groups from becoming excessively large; additional levels are supervised at full resolution.Source: §3.1 · pp. 3–4
  • Resizing each level’s reconstruction and target to its assigned resolution encourages early levels to preserve coherent layout and shape, leaving later levels to restore detail.When every level is supervised at full resolution, early reconstructions can preserve sharp edges without recovering surrounding structure.Source: §3.1 · p. 4

3.2 STAGGERED CHANNEL DIFFUSION

  • A global clock maps to ordered local denoising windows, so earlier levels are never noisier than the refinement levels they condition.Unstarted groups are zeroed, denoising groups are noised at local times, and completed groups provide clean context.Source: §3.2 · p. 5
  • Training weights per-level velocity errors equally across denoising groups, while the clock sampler gives extra probability to the interval where L0 is active.Weights are normalized across denoising levels so wider refinement groups do not dominate solely because they contain more channels.Source: §3.2 · p. 5
  • Across increasing latent widths, APEX gFID degrades more slowly than DC-AE and DC-AE 1.5 at f128, with matched denoiser-training FLOPs and total NFE.Figure 5 compares 64-token f64 latents and 16-token f128 latents.Source: Figure 5 · p. 6
  • At inference, one shared network evaluation updates all denoising groups per step, and each newly activated group starts from Gaussian noise.Under the equal-gap schedule, each step has the same cost as a flat model; cumulative prefixes are decoded as levels reach t = 0.Source: §3.2 · pp. 5–6

4 EXPERIMENTS

Experiments on ImageNet-512 evaluate APEX at 64× and 128× spatial compression through channel-count comparisons, compute-constrained sampling, component ablations, and schedule studies. APEX degrades more slowly as latents widen, beats f32c32 at matched guided inference budgets, and depends on coupling its hierarchy with asynchronous denoising.

4.1 GENERATION ACROSS CHANNEL COUNTS

  • At f64, APEX gFID is 2.85/3.15/4.9 at c128/256/512 versus DC-AE’s 3.22/9.19/26.1, showing slower degradation as widths grow.At f128 c512/c1024, APEX is 8.98/18.77, versus DC-AE 14.44/89.47 and DC-AE 1.5 11.86/51.47; Spectrum Matching and FreqWarm reach 73.71/79.51 at c1024.Source: §4.1 · p. 7

4.2 APEX MAKES HIGH COMPRESSION ADVANTAGEOUS UNDER CONSTRAINED COMPUTE

  • At 1×, APEX gFID is 2.74 with 250 NFEs versus 2.95 for f32c32 with 64; it beats f32c32 at all matched budgets, with larger margins at lower compute.At 1000 NFEs (4×), APEX reaches gFID 2.67.Source: §4.2 · pp. 7–8, Figure 6 · p. 8

4.3 THE HIERARCHY AND DENOISING PROCESS MUST WORK TOGETHER

  • gFID falls from 17.83 to 8.12 when matched diffusion is added; removing matched-resolution targets yields 8.77, while the DC-AE 1.5 construction reaches 52.89.APEX prefixes are coherent and usable at lower resolutions, whereas DC-AE 1.5 prefixes mix coarse structure with isolated high-frequency detail.Source: §4.3 · p. 8, Figure 7 · p. 8

4.4 DESIGN CHOICES

  • The stagger-gap trade-off is optimized near g = 0.25: larger gaps give later levels cleaner context but lengthen the trajectory and reduce overlap under equal-gap scheduling.At g = 0, the process reduces to synchronous diffusion.Source: §4.4 · p. 9, Figure 8 · p. 9
  • Increasing the training probability assigned to L0 improves gFID, with pL0 = 1 best in the three-level study; main models use 0.9 to preserve tail coverage in deeper hierarchies.L0 carries unconditional generation, while later levels perform conditional refinements.Source: §4.4 · p. 9
  • With the same 250-NFE budget, gFID is 19.83, 9.04, and 8.12 after L0, L1, and L2 complete, respectively, showing that all three levels contribute.L2 contributes despite receiving no direct supervision over the latter half of its trajectory.Source: §4.4 · p. 9, Figure 8 · p. 9

5 CONCLUSION AND LIMITATIONS

APEX combines resolution-aware channel prefixes with asynchronous denoising, and is reported to make extreme compression competitive under constrained compute. Its configuration still requires choices whose optima may vary across settings.

5 CONCLUSION AND LIMITATIONS

  • At matched training compute, APEX outperforms a 256-token baseline across evaluated guided inference budgets, with larger gains at lower compute.The conclusion describes this as making extreme compression competitive under constrained compute.Source: §5 · p. 9
  • APEX still requires choosing channel boundaries, per-level target resolutions, the stagger gap, and the global-clock distribution.The optimal choices may vary across settings; Section 4.4 provides practical guidance, and joint learning is proposed as a promising direction.Source: §5 · p. 9
  • The appendices include training details, configurations, evaluation protocols, FLOP accounting, pseudocode, and figure-reproduction instructions.Appendix F records the protocol and data required to reproduce each main-text figure; code is available online.Source: REPRODUCIBILITY · p. 10

A.1 GENERATED PREFIXES

  • Figure 9 shows generated samples as successive prefixes reach t = 0, with rows representing samples and columns showing the decoded image at each level.The illustrated four-level configuration is f64c8–32–128–256.Source: §A.1 · p. 14, Figure 9 · p. 14

A.2 REFINEMENT DIFFERENCES

The appendix visualizes successive prefix decodes and the differences introduced by later levels, using a shared display scale for the two examples.

A.2 REFINEMENT DIFFERENCES

  • The two Figure 3 examples show successive prefix decodes and their differences; x0, x1, and x2 decode 8, 32, and 128 channels, respectively.The supplied passage identifies Sample 1 but gives no further sample-specific result.Source: §A.2 · p. 15
  • Figure 10 computes differences per channel on full-resolution decodes and displays them on a shared gray-centered scale of ±0.5 in [0, 1] RGB.The scale clips 1.3% of pixels, and zero is gray.Source: Figure 10 · p. 15
  • Both samples use the same difference scale, so displayed changes can be compared across the examples.The scale is shared across the two samples.Source: Figure 10 · p. 15

B.1 COMPUTE-MATCHED TRAINING

The appendix specifies compute-matched training comparisons and the model and optimization setup used for ImageNet-512 experiments.

B.1 COMPUTE-MATCHED TRAINING

  • Comparisons match cumulative denoiser-training FLOPs, adjusting training steps for model cost and global batch size rather than matching updates or images seen.The matching applies when spatial compression, channel count, or denoiser size changes.Source: §B.1 · p. 15
  • The main SiT-XL/1 schedules correspond to approximately 1.8 × 10^20 denoiser-training FLOPs, and Table 3 gives their schedule details.Figure 5 uses f64 and f128 settings; Figure 6 uses f32 and f64 settings.Source: §B.1 · p. 15, Table 3 · p. 15
  • Training cost is calculated arithmetically from denoiser forward-pass cost, global batch size, and update count, with a multiply-add counted as two FLOPs.A factor of three approximates forward and backward passes; autoencoder training is outside this budget.Source: §B.1 · p. 15

B.2 MODELS AND OPTIMIZATION

  • Experiments use ImageNet-1K at 512 × 512, with class-conditional SiT-XL/1 for main experiments and SiT-B/1 for component and schedule ablations.Training uses AdamW at a constant learning rate of 10^-4 and EMA decay 0.9999; baselines share the DC-AE and SiT framework and, within comparisons, the backbone, optimizer, compute, and evaluation protocol.Source: §B.2 · p. 16

B.3 SAMPLING AND EVALUATION

The appendix defines sampling and evaluation protocols and reports an APEX comparison with the standard f8c4 SD-VAE at substantially lower denoiser-training compute.

B.3 SAMPLING AND EVALUATION

  • Generation is evaluated with 50,000 class-conditional samples using the ADM protocol, reporting gFID, sFID, and Inception Score.Classifier-free guidance uses scale 1.5 unless specified otherwise; reconstruction rFID and PSNR use ImageNet validation data and the complete latent.Source: §B.3 · p. 16
  • For N sampling steps, inference costs NFforward without guidance and 2NFforward with guidance, excluding the autoencoder decoder.At the stated costs, one f32 evaluation is approximately four times an f64 evaluation; Figure 6's 1× budget pairs 64 f32 evaluations with 250 f64 evaluations.Source: §B.3 · p. 16

B.4 COMPARISON TO F8C4 SD-VAE

  • The published SiT-XL/2 ImageNet-512 f8c4 result reaches gFID 2.62 with CFG 1.5 and 250 Euler–Maruyama steps.It follows 3,000,000 steps at batch size 256 and corresponds to 1.21 × 10^21 denoiser-training FLOPs.Source: §B.4 · p. 16
  • At 1,000,000 steps and batch size 2,048, APEX reaches gFID 2.62 using 1.78 × 10^20 FLOPs, or 15% of the f8c4 training compute.The standard f8c4 SD-VAE result is gFID 2.62 after 3,000,000 steps at batch size 256, using 1.21 × 10^21 denoiser-training FLOPs; Table 4 covers training compute to match that published gFID.Source: §B.4 · p. 16, Table 4 · p. 16

C AUTOENCODER TRAINING

APEX trains one autoencoder to make cumulative channel prefixes represent progressively finer detail, supervising each prefix at its assigned resolution. Its staggered clock gives channel levels offset denoising windows, with earlier levels becoming clean context for later ones.

C AUTOENCODER TRAINING

  • At each step, a uniformly sampled cumulative prefix is retained, later channels are zeroed, and reconstruction and target are resized to that level’s resolution before loss computation.The encoder and decoder are shared across levels; zeroed channels are not supervised.Source: C AUTOENCODER TRAINING · p. 17
  • Prefixes of 8, 32, and 128 channels add 8, 24, and 96 channels on the same grid; roughly 4× wider prefixes correspond to 2× larger target sides at lower widths.Intermediate levels at larger widths avoid excessively large refinement groups; the proportionality is a guideline.Source: C AUTOENCODER TRAINING · p. 17, Table 5 · p. 17
  • The f64c128 and f128c512 autoencoders are trained to match the rFID of their corresponding official DC-AE models.Source: C AUTOENCODER TRAINING · p. 17

D.1 THE STAGGERED CLOCK

  • Each level’s local time decreases from 1 (noise) to 0 (clean), with level ℓ starting at τ = ℓ·g and finishing at τ = 1 + ℓ·g.Earlier levels are therefore cleaner than later levels they condition on.Source: §D.1 · p. 17
  • With g = 0.25, the three denoising windows are (0, 1), (0.25, 1.25), and (0.5, 1.5), so later levels begin before earlier ones finish.Before a level starts, its input is zero; after its local time reaches zero, it stays fixed as context.Source: §D.1 · p. 18, Figure 11 · p. 18

D.2 TRAINING-CLOCK SAMPLING

Training-clock sampling allocates time across the staggered trajectories, while diffusion training computes loss only on active channel groups. At inference, one shared prediction updates active groups as their local times change.

D.2 TRAINING-CLOCK SAMPLING

  • The clock sampler assigns probability pL0 to (0, 1), where L0 is still denoising, and the remainder to (1, T), sampling uniformly within each interval.Uniform sampling over the full clock corresponds to pL0 = 1/T; when g = 0, all levels share one timestep.Source: §D.2 · p. 18
  • Most training examples are drawn while L0 is active, and refinement levels also train during overlapping portions of their trajectories; inference traverses their complete trajectories.For three levels with g = 0.25 and pL0 = 1, L1 trains at local times (0.25, 1) and L2 at (0.5, 1).Source: §D.2 · p. 18

D.3 DIFFUSION TRAINING

  • Unstarted groups are zero, active groups are noised, and completed groups provide clean context; squared error is averaged within active groups and then equally across them.The target velocity is noise minus the clean latent; unstarted and completed groups do not contribute to loss.Source: §D.3 · pp. 18–19

D.4 DIFFUSION INFERENCE

  • At inference, each group starts from Gaussian noise when its window opens, and one shared prediction updates groups according to their local-time changes.Unstarted and completed groups remain fixed; the time grid includes activation points ℓ·g.Source: §D.4 · p. 19

E.1 GENERATION ACROSS CHANNEL COUNTS

E.2 SCHEDULE, COMPONENT, AND HIERARCHY ABLATIONS

The ablation materials describe evaluations of stagger gap, training-clock allocation, and completed prefixes, with reported protocols and table scopes but no outcome values in these passages.

E.2 SCHEDULE, COMPONENT, AND HIERARCHY ABLATIONS

  • The gap ablation compares unguided gFID under matched training FLOPs using SiT-B/1 at f64c128; Table 8 reports Figure 8(a)'s stagger-gap results, with † extending the main plot.Source: §E.2 · p. 20, Table 8 · p. 20
  • The training-allocation comparison is at g = 0.25; pL0 = 2/3 denotes uniform clock sampling.Table 9 is captioned as Figure 8(b)’s training-allocation results.Source: §E.2 · p. 20, Table 9 · p. 20
  • The completed-prefix comparison stops the same pL0 = 1 model at 1, 1.25, or 1.5, redistributing all 250 NFEs over the retained interval for one, two, or three levels.Table 10 gives the Figure 8(c) values for completed prefixes of the pL0 = 1 model.Source: §E.2 · p. 21, Table 10 · p. 21

F FIGURE REPRODUCTION

The reproduction section documents the procedures and settings behind the paper’s figures, including benchmark comparisons, sampling sweeps, and staged generations. It also specifies which visuals are schematic rather than measured.

F FIGURE REPRODUCTION

  • Figure 2 schematically depicts a 512-channel latent on a 4 × 4 grid of 16 tokens, with channels ordered from structure to detail and later levels delayed by gap g.Its reconstruction–generation frontier is illustrative, not measured; Figure 5 supplies the quantitative comparison.Source: F FIGURE REPRODUCTION · p. 21
  • Figure 1’s structure-heavy clock example assigns probability 0.9 to (0, 1) and 0.1 to (1, 1.5), plotting densities 0.9 and 0.2, respectively.The example does not require training another model.Source: F FIGURE REPRODUCTION · p. 22
  • The Figure 5 comparison evaluates five methods across f64c128, f64c256, f64c512, f128c512, and f128c1024 at approximately matched cumulative training FLOPs.The f64 panel uses 64 tokens and the f128 panel uses 16; Tables 6 and 7 provide numerical values.Source: F FIGURE REPRODUCTION · p. 22
  • For the inference-compute plot, f32 evaluations cost approximately four f64 evaluations, so the 1× budget pairs 64 f32 NFEs with 250 f64 NFEs.At each budget, CFG is swept independently and the best evaluated gFID is reported.Source: F FIGURE REPRODUCTION · p. 22
  • At the 1× inference-compute budget, APEX and f32 select guidance scale 1.4, while DC-AE f64 selects 1.5.The comparison plots guided gFID against guidance scale; these are the selected scales.Source: F FIGURE REPRODUCTION · p. 22
  • The prefix visualization retains cumulative channels, zeros later channels, and decodes at full resolution; it compares DC-AE 1.5 prefixes with APEX’s resolution-aware hierarchy.The same image region is shown at each level to compare detail in early prefixes.Source: F FIGURE REPRODUCTION · pp. 22–23
  • The pL0 = 1 model’s completed-prefix gFIDs are 19.83, 9.04, and 8.12 for 8, 32, and 128 channels, respectively, using 250 NFEs in each condition.All conditions use the same model; no additional diffusion models are trained.Source: F FIGURE REPRODUCTION · p. 23

F.12 ADDITIONAL VISUALIZATIONS AND COMPARISONS

  • Figure 9 decodes completed prefixes from the same generation, while Figure 10 subtracts consecutive full-resolution prefix decodes using a shared display scale.Figure 9 examples are generations, not reconstructions of real images.Source: §F.12 · pp. 23–24

Related papers

Loading 2610.12843v1…