PaperlayerView exact revision on arXiv

Source-linked AI summary

How Do DiTs Denoise? An Aggregator-Predictor View of Attention and MLPs

Gyeonghun Kang, William Xingxu Chen, Xiang Cheng

arXiv:2610.29854v1cs.CVstat.ML

TL;DR

The paper asks how DiT components divide image representation and context-dependent reuse for generalization. It develops an aggregator–predictor account of local denoising, with theoretical, architectural, and empirical analyses. The findings support a component-level account, while the experiments’ scope and transplantation results limit how broadly or uniquely the roles can be inferred.

  • Problem

    How DiT components divide denoising computation for context-dependent generalization remains unclear, despite accounts based on representations and spatial locality.

    Source: §1 · p. 1

  • Method

    The paper frames DiTs as local denoisers decomposed into context aggregation and patch prediction, and analyzes the corresponding roles of attention and MLPs.

    Source: §3.1 · p. 5, §4.1 · p. 6

  • Results

    With sufficiently informative aggregation, patchwise linear denoisers exactly recover the empirical local denoiser, while experiments report noise-dependent contexts, competitive shared-memory predictors, and component-compatibility patterns.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The findings offer a statistical and architectural perspective on DiT denoising and motivate further work on interpretable predictors and targeted model adaptation.

    Source: §6 · p. 10

    Experiments use modest-sized unconditional DiTs on 64 × 64 images and a limited dataset collection; the shared-memory comparison has one seed per configuration, and transplants do not uniquely identify functional roles.

    Source: §6 · p. 10

Abstract

from arXiv · show

How do diffusion transformers (DiTs) represent image content and learn rules for reusing it to generalize? We develop an aggregator--predictor view of DiTs as input-adaptive local denoisers, in which context aggregation and patch prediction provide functional roles for interpreting attention and MLPs. We first establish that, for Gaussian Markov random field (GMRF) priors, the global Bayes denoiser can be approximated using only a subset of context patches, with error decaying exponentially in context radius. We further show that patchwise linear denoisers, often invoked to explain diffusion generalization, recover the empirical local denoiser with sufficiently informative aggregation. Within this view, AdaLN-modulated attention naturally parameterizes an input- and noise-dependent aggregator, while the MLP admits a learned key--value representation for tokenwise patch prediction. Our experiments support this view. DiTs trained on real images exhibit noise-dependent denoising contexts resembling those in the analytic GMRF setting, while attention mass exhibits a qualitatively corresponding broadening pattern. In a controlled CelebA64 study, shared-memory patch predictors used in place of DiT MLPs achieve competitive denoising and generative performance. Finally, cross-dataset component transplantation reveals compatibility patterns consistent with the proposed roles of QK in spatial weighting, VO in feature transport, and MLPs in patch prediction. Together, these results provide a component-level account of DiT denoising and motivate architecture-specific fine-tuning and controlled intervention.

1 Introduction

The paper asks how DiT components divide the work of representing image content and applying context-dependent rules for generalization. It proposes an aggregator–predictor view, establishes a statistical basis for local denoising, and presents empirical evidence consistent with the proposed component roles.

1 Introduction

  • Existing accounts leave open how DiT components divide denoising computation, including how attention, MLPs, and AdaLN support context-dependent generalization.Prior analyses address whole-image tokens or spatial locality, while attention can also select input-dependent, spatially non-contiguous context.Source: §1 · p. 1
  • The aggregator–predictor view casts DiTs as input-adaptive local denoisers, with AdaLN-modulated attention aggregating context and MLPs predicting patches from contextualized tokens.At the branch level, QK parameterizes spatial compatibility weights and VO transforms features; roles may interact in a multilayer residual stack.Source: §1 · p. 2
  • With sufficiently informative aggregation, a patchwise linear denoiser exactly recovers the empirical local denoiser.For GMRF priors, the global Bayes denoiser can also be approximated by a local denoiser with error decaying exponentially in context radius.Source: §1 · p. 2
  • Across real-image datasets, denoising contexts broaden with noise, attention mass shows a related qualitative trend, and shared-memory patch predictors retain competitive denoising and generation performance in a controlled CelebA64 study.These observations are presented as evidence consistent with the proposed component roles.Source: §1 · pp. 2–3
  • Cross-domain component transplantation reveals systematic compatibility patterns associated with source–target spatial similarity, consistent with—but not uniquely identifying—the proposed component roles.For face-to-face fine-tuning, QK-only replacement yields source-like faces, while MLP replacement produces visual changes; coherent target-like structure in tested CelebA→STL-10 hybrids appears primarily after joint replacement.Source: Figure 1 · p. 2, §1 · p. 3

2 Background and Problem Setup

The setup defines denoising as estimating clean patches from Gaussian-corrupted observations and distinguishes conditioning on all noisy patches from patchwise retrieval. It frames the central question as when local retrieval approximates the population denoiser and how a DiT could implement it.

2 Background and Problem Setup

  • The noisy observation is Xt = atX0 + btε, with normalized observation Yt = Xt/at = X0 + σtε and effective noise σt = bt/at.The setup includes VE and VP diffusion and linear Gaussian interpolation paths used in flow matching.Source: §2 · p. 3
  • The population global denoiser predicts each clean patch conditioned on all noisy patches, whereas the empirical global denoiser is a weighted sum of whole training images.Shared mixture weights keep the full-image prediction in the convex hull of training images, restricting independent recombination of local content.Source: §2 · p. 3
  • The setup motivates patchwise linear denoising because patch-specific retrieval weights can combine content from different training images; it asks when this estimator approximates the population denoiser and how a neural architecture can implement it.The PLD depends on context patches and is linear in their aggregation.Source: §2 · p. 3

3 Local denoiser can approximate global denoiser

The paper establishes a statistical basis for local denoising under GMRF priors and then frames local denoisers as context aggregation followed by patch prediction. It connects patchwise linear denoising to the empirical local conditional mean when aggregation preserves the information needed for prediction.

3 Local denoiser can approximate global denoiser

  • For GMRF priors, a radius-r local denoiser approximates the global denoiser with error decaying exponentially in r.The result follows from graph sparsity and analyticity of the inverse function; the radius-r context is defined by graph distance and need not be spatially contiguous.Source: §3 · p. 4
  • The certified radius needed for a given error tolerance is nondecreasing with noise over a bounded range, while exact radii can be substantially smaller.At extreme noise, the posterior mean approaches the zero prior mean and the absolute bound vanishes.Source: §3 · p. 4

3.1 Aggregator–predictor decomposition of local denoiser

  • An aggregator–predictor decomposition first maps local context patches to a statistic, then predicts the clean patch from that statistic.A predictor can recover the local conditional mean when aggregation preserves the information needed to determine it.Source: §3.1 · p. 5
  • With sufficient aggregation, a patchwise linear denoiser exactly recovers the empirical local denoiser.A compressive linear aggregator can suffice when the training-context differences lie in a lower-dimensional affine subspace.Source: §3.1 · p. 5
  • The PLD connection identifies retrieval as an estimator of the empirical counterpart of the population local denoiser, but the required low-dimensional-subspace condition is restrictive and motivates a nonlinear DiT aggregator.The decomposition is generally nonunique, allowing the aggregator to discard details unnecessary for the local conditional mean.Source: §3.1 · p. 5

4 DiTs as Adaptive Local Denoisers

The section frames DiT denoising as context aggregation followed by patch prediction: attention supplies an input-adaptive context, and the MLP predicts from the contextualized token. These roles are exact for isolated branches but can interact in a residual stack.

4 DiTs as Adaptive Local Denoisers

  • The single-layer view decomposes denoising into an attention aggregator followed by a tokenwise MLP predictor, while residual-stack interactions can prevent functional separation.VO and MLP updates can alter later QK routing, and attention can affect later MLP updates.Source: §4 · p. 6, §4.2 · p. 6

4.1 Attention as an input-adaptive aggregator

  • AdaLN-modulated attention parameterizes an input- and noise-dependent aggregator whose QK scores set relative spatial weights and whose VO operators transform and mix selected features.Softmax can suppress low-scoring patches, yielding an approximately local context when attention concentrates on a suitable set.Source: §4.1 · p. 6

4.2 MLP and shared-memory patch predictors

  • LPLD, SCA, and SMLP progressively connect explicit patch retrieval to an ordinary MLP by learning memory, sharing it across patches, and replacing softmax with unnormalized, potentially signed GELU coefficients.The shared bank can be used across layers, freeing parameter budget for predictor heads.Source: §4.2 · p. 7
  • The MLP admits a key–value memory interpretation: AdaLN modulates its query to retrieve values from learned memory, producing a tokenwise feature update from the contextualized patch.The interpretation assigns memory a patch-prediction role, not necessarily a named concept per memory entry.Source: §4.2 · pp. 6–7

5 Experiments

Experiments find noise-dependent broadening in real-image denoising contexts and competitive performance from shared-memory predictors in a controlled CelebA64 study. Component transplants show compatibility patterns associated with spatial similarity and component coadaptation.

5.1 Real-image denoising echoes GMRF locality

  • Real-image sensitivity contexts and attention contexts generally broaden with noise, though sensitivity and attention mass measure different quantities and need not identify identical context sets.Sensitivity can be concentrated at low noise and broader, sometimes disconnected, at higher noise.Source: §5.1 · p. 8, Figure 3 · p. 8

5.2 Shared-memory predictors can replace MLPs in a controlled setting

  • At epoch 400, shared SMLP with H = 8 achieves competitive denoising and lower FID than the ordinary MLP, while patch-shared SCA and SMLP outperform position-specific LPLD at comparable parameter budgets.The comparison uses a common backbone and one training seed per configuration for unconditional 64 × 64 face generation.Source: Figure 4 · p. 8, §5.2 · p. 9

5.3 Component transplants support the aggregator–predictor view

  • Source QK compatibility is higher for spatially similar datasets, while VO reversion penalties increase with joint non-additivity; source VO restoration can yield fragmented textures in natural-image-to-face transfers.The transplant study jointly fine-tunes QK, VO, and MLP while leaving AdaLN and other parameters fixed.Source: §5.3 · pp. 9–10, Figure 5 · p. 9
  • Natural-image source QK retains partial compatibility on face targets, while joint adaptation produces clearer results in the reported transfer examples.The reported endpoint-agreement pattern is asymmetric between STL-10-to-CelebA and the reverse direction.Source: Figure 5 · p. 9, §5.3 · p. 10

6 Conclusion and limitations

The paper presents an aggregator–predictor account of DiT denoising, with statistical and architectural support and complementary experimental evidence. Its experiments are limited in scale and scope, and the authors identify broader model and adaptation settings as next steps.

6 Conclusion and limitations

  • The conclusion assigns QK to spatial context weighting, VO to feature transport, and MLPs to contextualized patch prediction, alongside a statistical account of local denoising.The experiments provide complementary evidence through noise-dependent contexts, shared-memory predictors, and cross-domain compatibility patterns.Source: §6 · p. 10
  • Experiments use modest-sized unconditional DiTs on 64 × 64 images from a limited set of face and natural-image datasets, so broader model scales and settings remain untested.The shared-memory comparison uses CelebA64 and one training seed per configuration; the transplant study fixes AdaLN and global parameters and does not uniquely identify functional roles.Source: §6 · p. 10

AI use statement

  • The authors report using generative AI tools for literature review, code and experiment assistance, proof cleanup, manuscript feedback, and text polishing, with results independently reviewed and verified.The authors state that they made the final scientific and methodological decisions and take responsibility for the paper.Source: AI use statement · p. 10

Reproducibility statement

  • The paper states that assumptions and proofs, experimental protocols, and metric definitions are documented in appendices, with code and reproduction instructions provided.The stated materials include numerical summaries used to render reported plots and tables.Source: Reproducibility statement · p. 10

A Related Literature

The related literature covers representations and compositional reuse, denoising interpretations of architectures, learned MLP memories, and component interventions. The paper positions its account as connecting attention, MLPs, and AdaLN through adaptive context aggregation and patch prediction.

A Related Literature

  • Prior work studies diffusion representations, alignment with pretrained features, and compositional generation, including unseen combinations and limits in learning finer dependencies.The paper frames reuse as selecting context patches that provide evidence for predicting a clean patch from a new noisy image.Source: A Related Literature · p. 16
  • Prior work connects generalization to local denoisers, spatial locality, and balanced representations; the paper identifies a statistical target and conditions for local denoising and patchwise retrieval.Theorem 1 bounds the discrepancy between population local and global denoisers under a GMRF prior, while Theorem 3 concerns exact recovery of an empirical local conditional mean under sufficient aggregation, not a finite-sample population-risk bound.Source: A Related Literature · pp. 16–18
  • Existing denoising accounts analyze U-Net belief propagation, attention-based score denoising, local attention, and AdaLN-related feature modulation; this paper connects attention, MLPs, and AdaLN via aggregation and prediction.The proposed context can vary with input and noise, including non-contiguous patterns.Source: A Related Literature · p. 17
  • Prior MLP-as-memory work motivates editing learned associations, while this paper assigns the DiT MLP a memory role specifically in patch prediction from aggregated context.The interpretation allows representations to be distributed across memory entries rather than tied to named concepts.Source: A Related Literature · p. 17
  • Prior diffusion editing and localization methods intervene on parameters, attention, or activations; the paper’s transplants instead examine compatibility among jointly adapted QK, VO, and MLP components with AdaLN fixed.The transplant outcomes measure reuse and coadaptation under that intervention protocol.Source: A Related Literature · pp. 17–18

B Proofs

The appendix sets notation for block-partitioned vectors and graph neighborhoods, then introduces Chebyshev polynomials for constructing polynomial approximations with exponentially decaying error.

B.1 Notation

  • Graph balls B_r(m) contain patches within graph distance r of patch m, and x_Br(m) collects the corresponding blocks.The appendix also defines Euclidean and operator norms, spectral quantities, and Gaussian-distribution notation.Source: §B.1 · p. 18

B.2.1 Chebyshev polynomials and the reciprocal expansion

  • A degree-at-most-r polynomial can approximate the reciprocal function with error that decays exponentially in r.The passage identifies the Chebyshev convergence factor, though its formula is not fully present here.Source: §B.2.1 · p. 19

C.1 Full statement and proof of Theorem 1

For GMRF priors, the global Bayes denoiser admits an exponentially accurate radius-limited approximation; the appendix also establishes noise-dependent certified locality and conditions for recovering the empirical local denoiser from an aggregate.

C.1 Full statement and proof of Theorem 1

  • Theorem 1 bounds the error of a radius-r local denoiser for a GMRF prior by a term decaying exponentially in r.The rate depends on the spectral conditioning of I + σ_t^2 Q; worse conditioning slows the certified decay, while q− = q+ makes radius zero exact.Source: §C.1 · pp. 19–20

C.2 Full statement and proof of Corollary 2

  • The certified radius needed to meet a fixed error tolerance is nondecreasing with noise level over the stated bounded range.At zero noise, radius zero is exact.Source: §C.2 · p. 22

C.3 Proof of Theorem 3

  • The local conditional mean can be recovered from an aggregate when it preserves the relevant conditional mean, even if it does not preserve the full context.The stated row-space condition is sufficient, not necessary, and strict compression is unavailable when its required dimension equals the full context dimension.Source: §C.3 · pp. 22–23

D From the aggregator–predictor construction to implementation

The implementation links attention to input- and time-conditioned spatial aggregation and the MLP to tokenwise prediction, while distinguishing exact single-branch roles from their interactions in residual stacks.

D.1 Patch, position, and time conditioning

  • Time conditioning enters through AdaLN parameters, while each initial token combines a projected noisy patch with its position embedding.The derivation also permits a known scalar preconditioning of the noisy input before patch projection.Source: §D.1 · p. 24

D.2 Spatial weights and shared channel operators

  • At fixed time, QK determines relative spatial weights, while VO transforms and mixes the weighted patch features.The full attention branch need not be a convex combination because its channel operators are unconstrained.Source: Figure 6 · p. 24, §D.2 · p. 25

D.3 Soft locality of Attention Aggregator

  • Unmasked attention can be soft-local through suppression of low-scoring patches, but remains globally supported without explicit masking.Masking to a chosen context makes the branch local; its expected squared approximation error depends on discarded attention mass and feature and operator magnitudes.Source: §D.3 · pp. 25–26

D.4 General predictor parameterizations

  • The predictor parameterization treats an MLP as a learned key–value memory shared across patches within a layer, with independently learned memories across layers.The parameter comparison excludes the spatial-attention backbone and AdaLN parameters common to the compared models.Source: §D.4 · p. 27, Table 1 · p. 27

D.4.1 Learned PLD (LPLD)

  • LPLD retrieves from a learned spatial memory bank using a query from the conditioned layer input and values projected from raw memory patches.A learned compatibility map accommodates a memory width different from the token width.Source: §D.4.1 · pp. 27–28

D.4.2 Shared cross-attention (SCA)

  • SCA uses a shared library across layers, while its projections and query and output maps remain layer-specific.Its shared memory entries are reused across layers and patch positions.Source: §D.4.2 · p. 28

D.4.3 Shared MLP (SMLP)

  • SMLP shares a library like SCA but uses unnormalized GELU retrieval, evaluating every predictor head for every token without an expert router.The shared representations are reused across depth while retaining the MLP’s unnormalized GELU expansion.Source: §D.4.3 · p. 28

D.5 From a single layer to a residual stack

  • In a residual stack, aggregation updates a token’s state before prediction adds a feature update, but branch roles need not remain separable.Later VO or predictor updates can alter QK routing, and attention can affect later MLP updates.Source: §D.5 · p. 29

E Experimental details

The experimental protocol uses unconditional 64 × 64 RGB images and includes a synthetic GMRF study to compare exact and certified locality radii with a separate sensitivity-based measure of statistical dependence.

E Experimental details

  • The real-image experiments use unconditional 64 × 64 RGB images, while the synthetic GMRF experiment uses an analytically specified prior without training a DiT.The dataset table reports prepared-image counts and held-out evaluation splits.Source: E Experimental details · pp. 29–30, Table 2 · p. 29

E.1.1 Synthetic Gaussian MRF prior.

  • The synthetic prior uses a 16 × 16 graph of scalar patches with 384 sampled edges and positive edge weights, and evaluates 43 noise levels.A positive diagonal shift ensures the precision matrix is positive definite.Source: §E.1.1 · p. 30

E.1.2 Exact and certified radii for absolute locality error

  • The experiment compares the minimum radius meeting an absolute locality-error tolerance with the radius certified by Corollary 2.The certified radius is common across target patches, and the main figure uses tolerance ϵ = 10−3.Source: §E.1.2 · pp. 30–31

E.1.3 Sensitivity radius as a measure of statistical dependence

  • The sensitivity radius measures the fraction of a target patch’s total squared denoiser sensitivity attributable to input patches, rather than an absolute locality-error tolerance.The study evaluates q ∈ {0.90, 0.95}; these radii need not agree with exact or certified radii.Source: §E.1.3 · p. 31

E.1.4 Interpretation of Fig. 2

In the GMRF example, certified local-context radii grow with noise over a bounded interval, while sensitivity can remain broad even when absolute locality error is small. For real-image DiTs, the appendix estimates denoising context from input-output Jacobians and compares it with attention mass.

E.1.4 Interpretation of Fig. 2

  • A radius-3 context can contain 12 or 50 graph nodes and need not be spatially contiguous; locality follows the prior’s dependence graph, not image-grid proximity.Each graph-distance ball remains connected in the dependence graph.Source: §E.1.4 · p. 31
  • The certified radius is nondecreasing only over the specified noise interval; its maximum is 16 hops, versus 4 hops for each exact radius in this realization.Beyond the interval, both radii eventually decrease as posterior means approach the zero prior mean; this does not imply better reconstruction.Source: §E.1.4 · p. 32
  • Sensitivity radius can stay broad as absolute locality error shrinks: at extreme noise, relative influence reflects prior covariances even as absolute Jacobian entries vanish.The paper cautions that this synthetic result does not establish monotonic sensitivity-radius growth or that outside observations can be removed without affecting denoising accuracy.Source: §E.1.4 · p. 32

E.2 Experiment 1-2: Denoising context in DiTs trained on real-image datasets

  • For real-image datasets, trained-denoiser Jacobians provide an estimate of denoising-relevant dependence when the population prior and exact local conditional means are unknown.This interpretation applies to the extent that learned denoisers approximate population denoisers.Source: §E.1.4 · p. 32, §E.2 · p. 32

E.2.1 Sensitivity context set based on input-output Jacobian

  • Sensitivity weights assign each input patch its share of a target patch’s normalized squared Jacobian energy; radius and count measure spatial extent and the fewest patches reaching coverage q.The real-image analysis uses Chebyshev grid neighborhoods for radius and permits arbitrary-shaped context sets for the count.Source: §E.2.1 · p. 33

E.2.2 Attention context set based on attention mass

  • Attention context uses averaged softmax mass rather than output sensitivity, so matching coverage thresholds do not make the two measures equivalent.Attention mass is averaged across heads and layers; it is not weighted by value/output projections or residual gates.Source: §E.2.2 · p. 34

E.2.3 Interpretation of Fig. 3

On real-image datasets, estimated denoising sensitivity broadens with noise, and attention shows a related but distinct trend. The appendix also motivates and specifies a CelebA predictor comparison that replaces only the patch predictor while keeping the common backbone architecture.

E.2.3 Interpretation of Fig. 3

  • Sensitivity is concentrated at low noise and generally broadens as noise grows; attention mass also spreads more broadly, though its low-noise mass is already more diffuse.Figures 7 and 8 corroborate the overall broadening across additional datasets and at 90% and 95% coverage.Source: §E.2.3 · pp. 34, 36, Figure 7 · p. 35, Figure 8 · p. 35
  • Sensitivity-bearing context can be irregular or disconnected, with higher-noise sets spreading over more of the image; radius and patch count capture spatial extent and number of influential patches, respectively.These are complementary descriptions of context, not interchangeable measurements.Source: §E.2.3 · p. 34
  • Attention weights and input-output Jacobians measure different quantities, so their shared broadening is qualitative agreement rather than equality of context sets.The paper interprets the trend as consistent with attention adapting aggregation to denoising-relevant dependence.Source: §E.2.3 · p. 36

E.3.1 Motivation

  • The predictor experiment replaces only the patch predictor while keeping the surrounding DiT components common, to compare shared-memory alternatives with an ordinary MLP.The proposed comparison varies patch-specific versus shared storage, softmax versus unnormalized GELU retrieval, and shared versus layerwise memory.Source: §E.3.1 · p. 36

E.3.2 Settings

  • The CelebA configurations share the patch embedding, attention, AdaLN conditioning, residual structure, and output head; evaluation reports epoch-400 denoising and FID metrics.The common backbone does not mean attention weights are frozen or shared between runs.Source: §E.3.2 · pp. 36–37
  • The 256-width LPLD run developed generation artifacts and degraded denoising, so displayed comparisons use a separate recovery from its epoch-225 checkpoint.The recovery reaches cumulative epoch 400 with FID 29.197; its trajectory is not spliced onto the original run.Source: §E.3.2 · p. 37

E.3.3 Results and interpretation

In the CelebA64 comparison, shared-memory patch predictors are competitive with position-specific memory and ordinary MLPs, with results favoring GELU retrieval and multiple heads. The transfer-study setup compares dataset spatial similarity and hybrid-component agreement to probe compatibility among jointly adapted components.

E.3.3 Results and interpretation

  • At the largest H = 8 budget, SMLP reaches denoising AUC 0.012777 and FID 15.767, versus 0.013087 and 19.848 for LPLD; SCA is also competitive.Sharing prototypes across positions permits a 1,024-entry bank rather than 32 at a similar parameter count; SMLP also beats SCA across tested settings, consistent with more flexible unnormalized GELU coefficients.Source: §E.3.3 · pp. 37–38
  • For H = 8, multiple heads improve both held-out denoising measures over one head at every tested width in the shared-memory families, and improve SCA and SMLP FID at every width.For SCA, FID falls from 22.508 to 17.145 and for SMLP from 19.556 to 15.767 as H increases from one to eight at d′h = 1024.Source: §E.3.3 · p. 38
  • With H = 8, shared-memory SMLP has epoch-400 FIDs of 15.017, 16.652, and 15.767 at d′h = 256, 512, and 1024, compared with 19.550, 17.574, and 16.060 for the respective MLP controls.At larger budgets SMLP has slightly higher denoising AUC despite lower FID; the experiment is unconditional CelebA64 face generation with modest models and one seed per configuration, so it does not establish that all DiT MLPs should share memory.Source: §E.3.3 · p. 38

E.4 Experiment 3: Component compatibility in cross-dataset adaptation

  • The transfer study jointly adapts QK, VO, and MLP, then swaps source and adapted weights across eight hybrids to assess compatibility with the joint endpoint.It covers twelve directions and holds AdaLN and global parameters fixed; joint-transplant contrasts do not determine what independently fine-tuned components could learn or uniquely assign roles in a residual stack.Source: §E.4 · p. 39

E.4.1 Spatial MMD for dataset similarity

  • Spatial MMD2 compares distributions of fixed-grid patch self-similarity descriptors, with lower values indicating more similar descriptor distributions.Descriptors retain the spatial positions of pairs of patches with similar features; Figure 10 uses 1,000 unpaired images per dataset pair and disjoint 500-image halves on the diagonal.Source: §E.4.1 · p. 40, Figure 10 · p. 40

E.4.2 Endpoint agreement and component contrasts

  • Endpoint agreement compares each hybrid’s generated image with the jointly adapted output using global-feature agreement (EA) and spatial self-similarity agreement (SA).Both scores set the source to 0 and the joint endpoint to 1; component reversion penalties and interaction contrasts quantify dependence among hybrid states.Source: §E.4.2 · p. 41

E.4.3 Results and interpretation

Across the tested transfers, spatial similarity is associated with source-QK compatibility, while VO dependence tracks joint non-additivity. Source QK also retains partial compatibility on some face targets, and joint fine-tuning is generally comparable to or better than scratch training under the tested update budget.

E.4.3 Results and interpretation

  • Spatial MMD2 correlates with the QK reversion difference at ρEA = 0.739 and ρSA = 0.844; joint non-additivity, minimum reversion penalty, and three-way interaction also increase with spatial distance.The correlations for those latter quantities are 0.634/0.655, 0.739/0.844, and 0.641/0.774 for EA/SA, respectively; the authors characterize the patterns as descriptive given dependent directions and shared cube terms.Source: §E.4.3 · p. 41, Table 5 · p. 42, Figure 12 · p. 43
  • VO reversion difference correlates with joint non-additivity at ρEA = 0.832 and ρSA = 0.930, exceeding the corresponding QK- or MLP-reversion associations in both scores.Reverting VO also leaves fragmented or distorted outputs in the reported transfers compared with coherent joint endpoints.Source: §E.4.3 · p. 41, Figure 12 · p. 43, Table 6 · p. 43
  • Source QK retains partial endpoint agreement on face targets: for STL-10→CelebA, EA011 and SA011 are 0.267 and 0.340, versus 0.143 and 0.021 for CelebA→STL-10.ImageNet64→CelebA also retains partial agreement (0.387, 0.444); joint adaptation generally yields clearer endpoints.Source: §E.4.3 · pp. 41, 43, Figure 11 · p. 42

E.4.4 Additional generated-image cubes

  • Figures 13–15 extend the component-state image comparisons with additional latents across targets.The supplied captions identify CelebA, STL-10, and ImageNet64 as sources; they do not state additional outcomes.Source: §E.4.4 · p. 43, Figure 13 · p. 44, Figure 14 · p. 45

E.4.5 Training trajectories and restricted fine-tuning

  • Joint fine-tuning is comparable to or better than scratch training in held-out denoising and usually in FID-50k; QK-only and QK+VO-only fine-tuning generally lag on larger shifts under the tested budget.STL-10→CelebA is an exception for FID at the selected joint endpoint.Source: §E.4.5 · p. 47, Figure 16 · p. 47, Figure 17 · p. 48

Related papers

Loading 2610.29854v1…