PaperlayerView exact revision on arXiv

Source-linked AI summary

The Composition Gap in Dataset Distillation

Guang Li, Takahiro Ogawa, Miki Haseyama

arXiv:2609.34343v1cs.LG

TL;DR

The paper asks whether separately distilled datasets can be combined while preserving training fidelity, a question that individual evaluations do not answer. It derives composition error laws and analyzes learned sets, finding that residuals can dominate structural discrepancy and that joint distillation can retain an accuracy advantage even when that structural term is zero.

  • Problem

    It is unclear whether the union of independently distilled datasets reproduces training on the union of real data, since evaluating each distillate separately does not answer the composition question.

    Source: §1 · p. 1

  • Method

    The paper analyzes training maps, derives an exact quadratic composition law, and decomposes learned-set union error into local endpoint discrepancy and aggregate source residual.

    Source: §1 · pp. 1–2

  • Results

    Under endpoint matching, the aggregate source residual exceeds the structural term by more than an order of magnitude, while joint distillation retains an accuracy advantage for same-dataset distillates where the structural term is zero.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Training fidelity and downstream accuracy are distinct requirements, so evaluating each distillate on its own establishes neither for their combination.

    Source: §1 · pp. 1–2

    The exact results assume fixed-step gradient descent on quadratic objectives and apply to networks only locally; adaptive optimizers and long horizons are outside their scope.

    Source: §5 · p. 6, §7 · p. 9

Abstract

from arXiv · show

Dataset distillation compresses a training set into a small synthetic set, usually evaluated one at a time. In federated and data-governance settings, several parties distill their own data and a user trains on their union. We ask whether the union of separately distilled sets reproduces training on the union of the real data composability and show that it can fail even when every source is distilled exactly and the total budget admits an exact joint distillate. Compressing a training trajectory into fewer steps transforms the source statistics nonlinearly, so averaging compressed sources differs from compressing their average. For quadratic objectives we derive the exact composition error for two-to-one step compression in terms of the source-Hessian variance and the linear terms of the losses, and on a smooth network at small step sizes this prediction captures the local endpoint discrepancy in magnitude and direction. For learned synthetic sets, however, the composed error decomposes exactly into this local discrepancy and an aggregate source residual. Under endpoint matching the residual exceeds the structural term by more than an order of magnitude, and under distribution matching the two terms partly cancel. Joint distillation also retains an accuracy advantage when both sets are distilled from the same dataset, where the local discrepancy is exactly zero. Training fidelity and downstream accuracy are therefore distinct requirements, neither established by evaluating each set on its own.

1 INTRODUCTION

The paper asks whether independently distilled datasets can be combined without losing training fidelity, and shows that exact source-wise fidelity does not guarantee composability, even when an equal-budget exact joint distillate exists. It traces the discrepancy to nonlinear compression and finds that learned-set residuals and accuracy differences also matter beyond the structural curvature effect.

1 INTRODUCTION

  • Exact source-wise fidelity does not guarantee composability, even when the total budget admits an exact joint distillate.Evaluating each independently distilled set alone does not establish whether their union reproduces training on the union of real data.Source: §1 · p. 1
  • Compression and aggregation can disagree because compressing each source before averaging differs from compressing the averaged source statistics.In quadratic regression, sufficient statistics combine linearly, but matching two gradient steps with one synthetic step preserves a nonlinear transformation of source curvature.Source: §1 · p. 1, Figure 1 · p. 2
  • For two-to-one compression of quadratic objectives, the exact endpoint discrepancy scales with squared step size and depends on source-Hessian variance and an affine term coupling Hessians with linear loss coefficients.The variance captures differences in both eigenvalues and eigenvectors.Source: §1 · p. 1
  • With distribution matching on CIFAR-10, SVHN, and CIFAR-100, the union’s excess endpoint error is near zero under IID partitioning and positive under label skew.The supplied result continues with the finding that joint-distillation accuracy gains persist at every skew level.Source: §1 · pp. 1–2
  • The paper also proposes a network-computable local statistic, Pcomp, based on source gradients and Hessian-vector products.Its quadratic composition law includes a zero-error condition, extensions to other compression ratios, and a certificate for approximate distillation.Source: §1 · p. 2
  • The composition error of learned synthetic sets decomposes into the local endpoint discrepancy and an aggregate source residual, which dominates by more than an order of magnitude under endpoint matching.Under distribution matching, the residual is anti-aligned with the local discrepancy under label skew and partly cancels it.Source: §1 · p. 2
  • Joint distillation retains an accuracy advantage even when both distillates come from the same dataset, where the structural discrepancy is exactly zero.Reweighting constituents removes at most about half of the squared composed error and produces no detectable accuracy change.Source: §1 · p. 2

2 RELATED WORK

Prior dataset-distillation work studies several construction methods and what a distillate preserves, while federated work trains on synthetic data shared by clients. This paper analyzes post-hoc composition under finite-step training and characterizes failure even with exact constituents and an equal-budget joint solution.

2 RELATED WORK

  • Dataset distillation has been studied through unrolled training [Wang et al., 2018], gradient matching [Zhao et al., 2021; Zhao & Bilen, 2021], distribution matching, kernel-ridge solutions, and trajectory matching [Cazenavette et al., 2022; Li et al., 2024a].Theory has characterized preserved solutions, losses, and feature-correlation spectra.Source: §2 · p. 2
  • Federated dataset distillation trains on synthetic data transmitted by clients [Zhou et al., 2020; Goetz & Tewari, 2020; Hu et al., 2022; Xiong et al., 2023; Liu et al., 2023a; Wang et al., 2024; Holland et al., 2024; Arazzi et al., 2025; Shi et al., 2026].The closest analysis is local-update drift in federated optimization [Li et al., 2020b], whose endpoint-averaging identity is recovered by the one-step case.Source: §2 · p. 2
  • This work analyzes post-hoc composition under finite-step training and shows failure can persist with exact constituents and an equal-budget joint solution.It characterizes the discrepancy for noncommuting source Hessians and general compression ratios.Source: §2 · pp. 2–3

3 SETUP

The setup treats distillation as matching training endpoints under a specified update budget and initialization distribution, then defines composition error relative to training on a weighted union. It also establishes that universal composability requires an affine compression map.

3 SETUP

  • A dataset’s training map is induced by K full-batch gradient-descent steps of size η, and a distillate is assessed by whether its prescribed synthetic updates reproduce the real-source endpoint.The endpoint error is defined over initializations drawn from P0.Source: §3 · p. 3
  • The weighted union D1 ⊕α D2 uses α as the loss weight, not the share of stored synthetic samples.The setup distinguishes constituent error, independent composition error, oracle joint error, and composition gap.Source: §3 · p. 3
  • The theoretical composition gap is nonnegative because the composed set is an admissible joint candidate, but the empirical RMS excess can be negative.The empirical quantity replaces the joint infimum with one joint distillation run and compares RMS errors rather than squared errors.Source: §3 · p. 3
  • The theoretical results assume fixed-step gradient descent on symmetric quadratic losses with ηL < 1; other assumptions supply realizability, a common optimum, or slowly varying network Hessians where needed.The two-to-one identity requires symmetry, while arbitrary-ratio and lower-bound results require the positive-branch condition.Source: §3 · p. 3
  • For quadratic compression, the finite-step Hessian transformation is nonlinear when the compression ratio exceeds one, explaining why averaging after compression can differ from compressing after averaging.The displayed Figure 2 illustrates curvature disagreement and rotated eigenbases as sources of nonzero discrepancy.Source: §3 · p. 3, Figure 2 · p. 4
  • Universal composition requires affinity: a continuous source-wise compression map composes for every weighted collection exactly when it is affine.The training-endpoint equivalence additionally requires injectivity on the relevant convex hull of compressed statistics.Source: §3 · pp. 3–4

4 THE COMPOSITION LAW

The composition law first shows that individually exact distillates can fail to compose, then gives an exact matrix discrepancy for two-to-one quadratic compression. Curvature variance governs the leading discrepancy across compression ratios, and a bound can certify detectable error for approximate constituents under stated conditions.

4.1 ONE DIMENSION

  • For two distinct one-dimensional source curvatures with a shared optimum, individually exact distillates can compose to a curvature below the exact joint value, despite an equal-budget joint distillate.With a1 = 1, a2 = 3, η = 0.1, and p = 2, the source synthetic curvatures 1.9 and 5.1 average to 3.5, while the joint value is 3.6; the shortfall is ηα(1 − α)(a1 − a2)^2.Source: §4.1 · p. 4

4.2 EXACT LAW FOR TWO-TO-ONE COMPRESSION

  • For two-to-one compression, the exact matrix law applies without a commutativity assumption, with discrepancy determined by source-Hessian variance and an affine covariance between Hessians and linear coefficients.Under a common optimum and full initialization coverage, the discrepancy vanishes if and only if all sources have the same Hessian.Source: §4.2 · p. 5

4.3 OTHER COMPRESSION RATIOS

  • For any compression ratio p = T/K > 1, curvature variance governs the leading operator term, and the endpoint discrepancy has a lower bound without requiring commuting Hessians.The bound is unconditional for p = 2.Source: §4.3 · p. 6

4.4 APPROXIMATE DISTILLATES

  • For approximate distillates, if the structural gap exceeds the aggregate source-error bound, the composed union misses the real-union endpoint by at least their difference.The quantities can be estimated separately, but on a network without an established common optimum the inequality is a diagnostic rather than a certificate; constituent errors alone cannot upper-bound composition error.Source: §4.4 · p. 6

5 FROM QUADRATIC MODELS TO NETWORKS

The network analysis uses a frozen quadratic as a local proxy and bounds the nonlinear composition error by the synthetic- and real-training remainders. A gradient- and curvature-based statistic can approximate the observed error when those remainders and the affine-statistic error are small.

5 FROM QUADRATIC MODELS TO NETWORKS

  • The frozen quadratic uses the neural loss Hessian at a reference point, and its synthetic-set proxy is realizable only when assumption (A2) holds.The nonlinear RMS composition error differs from the quadratic counterpart by at most the sum of the synthetic and real training remainders.Source: §5 · p. 6
  • Pcomp estimates the local structural term at a shared parameter without a common optimum or explicit Hessians, using source gradients and Hessian-vector products.For ideal two-to-one quadratic compression, Pcomp equals Eind; each rcomp evaluation costs one Hessian-vector product per source.Source: §5 · p. 6
  • Pcomp approximates observed RMS error when the affine-statistic error and both nonlinear training remainders are small relative to that error.The source-matching errors and their contribution are measured in the experiments.Source: §5 · p. 7

6 EXPERIMENTAL RESULTS

Experiments find composition error under label skew and show that learned-set residuals can dominate or partly cancel the local structural discrepancy. Joint distillation retains an accuracy advantage even when both sets use the same dataset.

6 EXPERIMENTAL RESULTS

  • Once each quadratic source is matched, learned distillates reproduce the exact composition law; below the dimension-sized budget, composed error exceeds the closed form by 4.8 to 11.7 times.For twenty random source pairs in d = 12, learned Gram matrices recover the transformed statistics to machine precision once the budget reaches the dimension.Source: §6 · p. 7
  • At ηref, the SiLU network’s quadratic prediction matches the local discrepancy’s direction with cosine 1.000 and a 2 to 4 percent relative residual.The prediction degrades as step size grows, overshooting the local discrepancy by 1.9 times at 16ηref.Source: §6 · p. 7
  • At Dir-0.1, endpoint-matching source residuals are 24, 13, and 13 times Cstruct at one, five, and ten images per class, respectively.The residual, rather than the structural term, sets the composed error in these experiments.Source: §6 · p. 7
  • 7.3–7.5 × 10−3 endpoint error for joint distillation contrasts with 9.85 × 10−3 for the separately distilled union at Dir-0.1; excess error is positive under skew.The excess is near zero under IID partitioning and is an order of magnitude larger on every label-skewed partition; SVHN and CIFAR-100 replicate the pattern.Source: Figure 4 · p. 8, §6 · pp. 8–9
  • Under label skew, the source residual and local discrepancy are anti-aligned, so the observed error is smaller than the residual; their cosine averages −0.35 to −0.48.The quadratic prediction matches the local discrepancy in magnitude but is nearly orthogonal to the observed error at ηL ≈ 1.Source: Figure 3 · p. 8, §6 · p. 9
  • 5.3 to 5.9 points of accuracy gain for joint distillation occur at every heterogeneity level, and the no-split control still gains 6.6 ± 0.4 points.In the no-split control, both independent distillates are learned from the full training set and the local discrepancy is exactly zero.Source: Figure 4 · p. 8, §6 · p. 9

7 CONCLUSION AND LIMITATIONS

The conclusion distinguishes source-wise fidelity from composability and downstream accuracy, while bounding the theory’s scope. Related work and framework sections contrast procedure compression with mergeable summaries and define weighted unions and finite-sample realizability.

7 CONCLUSION AND LIMITATIONS

  • Exact results apply to fixed-step gradient descent on quadratic objectives and only locally to networks; adaptive optimizers and long horizons are outside their scope.The network experiments have not reached the regime where the structural term dominates, and whether a distillation method can reach it remains open.Source: §7 · p. 9

A EXTENDED RELATED WORK

  • Mergeable summaries and composable coresets guarantee union behavior by construction, whereas standard source-wise distillation compresses a training procedure rather than a statistic.Prior theory of distillation often studies converged solutions; this work instead fixes an optimizer and finite horizon and asks whether compression commutes with source mixing.Source: A EXTENDED RELATED WORK · p. 15

B FORMAL FRAMEWORK AND FINITE-SAMPLE REALIZABILITY

  • The framework distinguishes weighted endpoint and excess-error quantities from structural curvature terms, constituent-error bounds, and nonlinear remainders.Its notation includes curvature variance and affine conflict, plus bounds on weighted constituent RMS error.Source: B FORMAL FRAMEWORK AND FINITE-SAMPLE… · p. 15

B.1 WEIGHTED UNIONS AND TRAINING MAPS

  • A weighted union combines source losses according to their weights, separating intended source mass from stored-set cardinality; naive concatenation uses sample counts as weights.Endpoint comparisons use full-batch gradient descent at a fixed learning rate and the same initialization distribution.Source: §B.1 · p. 15

B.3 FINITE-SAMPLE REALIZABILITY

  • For p = T/K > 1 and 0 ⪯ηH ≺I, transformed quadratic statistics preserve the source’s range and rank and can reproduce T real steps in K synthetic steps.For positive-mass sources, the exact transformed real-union objective needs at most the sum of the source ranks in examples.Source: §B.3 · p. 16

B.2 PROOF OF THE COMPOSABILITY–AFFINITY PRINCIPLE

The proof characterizes source mixing that commutes with compression as an affine map. It derives this from midpoint composition, with injectivity needed when the identity is specified through a downstream endpoint map.

B.2 PROOF OF THE COMPOSABILITY–AFFINITY PRINCIPLE

  • Repeated midpoint composition establishes the mixing identity for dyadic weights, and continuity extends it to every weight, making C affine.The proof uses this implication to show that composition-preserving C must be affine.Source: §B.2 · p. 16
  • When composition is specified through an endpoint map E, injectivity on the relevant convex hull converts endpoint midpoint equality back into the statistic-level identity.For overparameterized networks, the statistic-to-trajectory map is generally not injective, so this equivalence is an algebraic property of the quadratic model, not a network impossibility result.Source: §B.2 · p. 16

C ONE-DIMENSIONAL EXACT SEPARATION

Separately exact distillates need not compose: finite-step compression makes the union’s endpoint differ from that of a joint distillate. The section gives exact quadratic examples and identifies when the gap vanishes.

C ONE-DIMENSIONAL EXACT SEPARATION

  • For distinct scalar curvatures, exact constituents still yield a positive composition gap under the correct union mass, while one exact joint example realizes zero error.The gap follows because the average of the compressed curvatures lies below the compression of their average.Source: C ONE-DIMENSIONAL EXACT SEPARATION · p. 17

C.1 QUADRATIC ENDPOINT MATCHING (E5)

  • At n_syn ≥ d, endpoint matching recovers the transformed Gram statistic to machine precision and the composed defect equals the closed form; below d, approximation error dominates.The closed form is η^2/4 (H1 − H2)^2; budget-limited errors can also cancel the structural term in other settings.Source: §C.1 · p. 19, Table 2 · p. 19

D.1 TWO-SOURCE IDENTITY

  • For one synthetic step, exact constituents make the composition defect coincide with the drift between averaging local updates and training on pooled data.The text identifies this K = 1 case with the parameter-averaging drift in federated optimization.Source: §D.1 · p. 19

D.2 MULTI-SOURCE IDENTITY

  • The multi-source curvature-variance term is positive semidefinite and vanishes exactly when all positive-mass source Hessians are equal.The identity requires symmetric Hessians; positive semidefiniteness is separately used for finite-dataset realizability and the positive contraction branch.Source: §D.2 · p. 20

D.3 CURVATURE AND AFFINE STATISTICS

The section characterizes the two-to-one endpoint defect through curvature variance and an affine conflict, then extends the identity to momentum and expected minibatch training. For broader compression ratios, it gives a noncommutative decomposition and a finite-step gap that vanishes in the flow limit.

D.3 CURVATURE AND AFFINE STATISTICS

  • The two-to-one composition defect separates a curvature-variance term from an affine-conflict term, with zero error characterized by both conditions.With full-rank initialization coverage and a common optimum, exact composition requires all positive-mass source Hessians to be equal.Source: §D.3 · p. 20

D.4 MOMENTUM AND MINIBATCH STOCHASTICITY

  • The two-to-one discrepancy is unchanged under fixed-coefficient heavy-ball momentum and, in expectation, under independent unbiased minibatches.Adaptive preconditioners such as Adam are not covered.Source: §D.4 · p. 21

E GENERAL FINITE-STEP JENSEN GAP

  • For arbitrary real compression ratios with a shared optimum, the endpoint defect has an exact decomposition without requiring source Hessians to commute.The sufficient-separation bound can become uninformative even though the exact composition gap remains nonnegative.Source: E GENERAL FINITE-STEP JENSEN GAP · pp. 23–24
  • For bounded symmetric Hessians with a shared optimum, the finite-step composition gap is O(η^2) at fixed K and O(η) when Kη → τ/p, vanishing as η → 0.This is the flow-limit boundary of the finite-step Jensen mechanism.Source: E GENERAL FINITE-STEP JENSEN GAP · p. 24

E.3 DIFFERENT OPTIMA VIA AFFINE AUGMENTATION

The section extends composition identities to sources with different optima and gives tools for interpreting endpoint and statistic errors. It also states the assumptions and sufficient condition needed to infer a nonzero composed error.

E.3 DIFFERENT OPTIMA VIA AFFINE AUGMENTATION

  • For arbitrary real p > 1, exact affine compression has an identity for different source optima without requiring commuting Hessians.The identity compares compressing each source before weighted mixing with applying the same principal power after mixing.Source: §E.3 · p. 25

F.1 DIFFERENCE-OF-POWERS BOUND

  • For square matrices, the difference-of-powers bound ∥X^K − Y^K∥₂ ≤ Kρ^(K−1)∥X − Y∥₂ holds without commutativity.This bound supports stability arguments for the associated K-step maps.Source: §F.1 · p. 27

F.2 PROOF OF AFFINE-STATISTIC STABILITY

  • Affine-statistic perturbations bound the excess RMS error of the actual composed map over the structural gap.The stated bound assumes ideal and actual one-step affine contractions have operator norm at most one and bounds the affine perturbation.Source: §F.2 · p. 27

F.3 FROM ENDPOINT ERROR TO STATISTIC ERROR

  • Endpoint fidelity controls curvature statistics only with full-rank initialization coverage and a fixed positive dynamics branch.The stated stability result also assumes the ideal and actual statistics share a fixed point.Source: §F.3 · p. 28

F.4 PROOF OF THE SUFFICIENT DETECTABILITY CONDITION

  • When the structural term exceeds the aggregate source residual, their difference lower-bounds the composed error; failure of this sufficient condition does not imply non-detectability.On a network without an established common optimum, the inequality is a diagnostic rather than a certificate.Source: §F.4 · p. 28

F.5 WHY INDIVIDUAL ERRORS ALONE ARE INSUFFICIENT

Individual endpoint errors alone cannot control composition error: even exact scalar constituents can have unbounded composition error as their optima separate.

F.5 WHY INDIVIDUAL ERRORS ALONE ARE INSUFFICIENT

  • No function of constituent endpoint errors alone can upper-bound every composition error while mapping zero constituent errors to zero.The result rules out a universal composition guarantee based only on source-wise endpoint fidelity.Source: §F.5 · p. 28
  • Exact scalar constituents can have unbounded composition error as their optima separate.The construction uses η = 1/2, T = 2, K = 1, curvatures 1/2 and 3/2, and equal masses.Source: §F.5 · pp. 28–29
  • In the construction, the error includes a positive structural term at a = 0 and a term that grows without bound with separated optima, despite zero constituent errors.For general initialization mean µ0, the expression includes σ0²/256 and (µ0/16 + a/8)².Source: §F.5 · p. 29

G LOCAL NONLINEAR REMAINDER

Locally, nonlinear-versus-quadratic endpoint gaps are bounded by remainders, yielding a finite-step detectability window. Exact scalar retiming across sources or modes requires specific calibration conditions.

G LOCAL NONLINEAR REMAINDER

  • Under Hessian-Lipschitz curvature, a trajectory staying in a radius-r ball with ∥I − ηH∥2 ≤ ρ has nonlinear RMS-gap deviation bounded by δR + δS.This bounds the difference between the nonlinear and frozen-quadratic RMS gaps.Source: G LOCAL NONLINEAR REMAINDER · p. 29
  • With gradients bounded by G, K-step displacement is O(η), making the remainder O(η³); for K=1, δS vanishes and uncertainty reduces to ∆aff + δR.The local-ball condition can be checked retrospectively or enforced through step-budget and initialization-radius restrictions; trajectories crossing curvature regimes are not covered.Source: G LOCAL NONLINEAR REMAINDER · pp. 29–30
  • c1/(2√(c2c3)) is the maximum of Dloc(η)=c1η/(c2+c3η²); a certified interval requires the stated discriminant condition and stability constraints.The window compares a c1η² structural signal against source error bounded by c2η and remainders bounded by c3η³.Source: G LOCAL NONLINEAR REMAINDER · pp. 29–30
  • The structural term grows as η², while constituent error grows as η and nonlinear error as η³; the mechanism vanishes in the flow limit and may be masked by constituent error.The analysis is local and requires a slowly varying Hessian.Source: Figure 8 · p. 30, G LOCAL NONLINEAR REMAINDER · p. 30

H.1 MODE-SPECIFIC RETIMING

  • For commuting Hessians, one scalar retiming repairs all task-relevant modes only when their mode-specific K*j values coincide; discrete GD additionally requires an integer or approximation.The exact scalar calibration described for the flow limit does not generally compensate for curvature scaling across every mode in discrete time.Source: §H.1 · pp. 30–31, H CALIBRATION AND COMPONENTS THAT… · p. 30

H CALIBRATION AND COMPONENTS THAT SCALAR RETIMING CANNOT REPAIR

  • For uniform source clocks, choosing βi=(αi/ci)/Z and τ=TZ makes the matrix exponentials agree; with linearly independent Hi, exactness requires βici/αi constant across sources.Here Z=Σi αi/ci.Source: §H.1 · p. 31

H.3 REPAIRABILITY TAXONOMY

This section characterizes repairability of composition errors, distinguishing scalar-calibratable defects from mode-dependent or higher-rank repair and covering rank budgets, observable gaps, network validation, and estimation safeguards. It also examines how composed endpoint error and downstream accuracy can respond differently, with source residuals dominating or offsetting local structural discrepancy and joint distillation retaining an accuracy advantage even when that discrepancy is zero.

H.3 REPAIRABILITY TAXONOMY

  • The repairability taxonomy separates scalar calibration from mode-dependent or higher-rank repair; a nonzero mean affine discrepancy alone does not imply non-repairability.Only semantic-mass and clock interventions were tested empirically; other taxonomy entries are theoretical. The rank-budget proof gives a lower bound for rank-r synthetic Hessians and extends its argument to finite-step dynamics.Source: §H.3 · pp. 31–32
  • For partitions a and b, proxy separation greater than ζa + ζb certifies the corresponding observed ordering.The certificate accounts for affine and nonlinear deviations from the frozen-model reference.Source: §H.3 · p. 32

I HVP ESTIMATOR AND CONCENTRATION; I.1 HVP-ONLY ACTION; I.3 RADEMACHER PROBES

  • The curvature-only trace estimator is unbiased with finite-probe concentration guarantees, and raw Rademacher probes are exact when B = 0.The HVP-only action uses 3m HVP calls per probe for m sources and requires no explicit Hessian.Source: §I.1 · p. 33, §I.3 · p. 33

I.4 IMPLEMENTATION SAFEGUARDS

  • Correct estimation requires Hessians at a shared parameter from complete empirical-mean source losses, detached first HVPs, complete-partition source masses, and separate calibration and hold-out probes.Raw probes estimate tr(V^2H), whereas probes divided by d estimate the trace divided by d.Source: §I.4 · p. 34

J.2 VERIFICATION OF THE LOCAL PREDICTION ON THE NETWORK (E1)

  • At ηref, SiLU and tanh calibration ratios were 0.996/0.969 and 0.992/0.991 (IID/Dir-0.1), while ReLU’s IID ratio of 14.9 ± 1.7 was not meaningful near-zero Cstruct.For smooth activations, squared discrepancy scaled approximately as η^4 inside ηL < 1; the primary response was RMS endpoint-map distance, with test accuracy assessed separately. ReLU is a boundary case because activation crossings are not captured by the frozen-Hessian prediction.Source: §J.1 · p. 34, §J.2 · pp. 35–36, Figure 9 · p. 36

J.3 ENDPOINT-MATCHING DISTILLATES ON THE NETWORK (E10)

  • At ηref, endpoint-matched source residuals were 23.6, 13.3, and 12.7 times Cstruct across the three budgets, while joint distillation had lower error in every cell.The local prediction matched the discrepancy in magnitude and direction at ηref, with cosine 1.000 for SiLU and 0.999 for tanh and relative residual 0.02–0.04 on every partition. The source-error inequality held for none of 96 initializations at ηref; larger initialization banks and longer training did not remove the source error.Source: §J.3 · p. 37

J.4 INDEPENDENT VERSUS JOINT DISTILLATION (E2)

  • The official-schedule evidence uses distribution matching; gradient-based condensation is only a short-schedule control because its full-data run reached 41.4% at IPC 10, below the published 44.9%.The retained short-schedule DC results have endpoint-level behavior matching DM, with a positive gap at Dir-0.1.Source: §J.4 · pp. 37–38
  • Per-set scalar calibration fits each synthetic set’s displacement to its own real two-step endpoint, so reported calibrated endpoint errors differ from fixed-step endpoint matching.The fitted multiplier is ci = ⟨di, δi⟩/∥di∥2; the composed union is the mass-weighted sum of calibrated source displacements.Source: §J.4 · p. 39

J.7 DOWNSTREAM ACCURACY AND THE NO-SPLIT CONTROL (E7)

  • Joint distillation gains +5.3, +5.7, +5.9, and +5.3 accuracy points across IID through Dir-0.1; the no-split control gains +6.55 ± 0.42 despite zero local discrepancy.Warm-starting from the no-split union recovers +0.44 ± 0.47 of the 6.15-point gap after 2,000 iterations, less than the +1.83 ± 0.65 gain from a cold start.Source: Table 7 · p. 40, §J.7 · pp. 43–44, Table 11 · p. 44

J.5 EXACT DECOMPOSITION OF THE OBSERVED DISCREPANCY (E9)

  • At skewed CIFAR-10 levels, the local discrepancy and source residual are anti-aligned, while the structural prediction explains a negative fraction of composed-error squared magnitude.At IID, the local discrepancy is 1.6 × 10−5 and the composed error approximately equals the source residual; without scalar calibration, scale mismatch dominates, but excess remains positive at skewed levels and indistinguishable from zero at IID.Source: §J.5 · pp. 41–42, Table 8 · p. 42

J.6 POST-HOC CLASS REWEIGHTING (E12)

  • Held-out class reweighting reduces independent-union squared error by 15–25% at skewed levels, but excess endpoint error remains positive under every tested weighting.The oracle removes 45–53% on CIFAR-10 (41% on SVHN); held-out reweighting leaves Dir-0.1 accuracy changes undetectable for both union and joint sets.Source: §J.6 · pp. 42–43, Table 10 · p. 43

J.8 CLASS-ALLOCATION-MATCHED JOINT SET (E11)

  • Matching class allocation leaves excess endpoint error positive in all nine cells, while reducing the accuracy gain on two Dir-0.1 partitions.The gain falls from 4.5 to 1.0 points on partition 0 and from 5.4 to 3.4 on partition 1; allocation therefore explains part, but not all, of the accuracy difference.Source: §J.8 · p. 44, Table 12 · p. 45

J.9 IMAGE BUDGET AND THE LOCAL REFERENCE

  • As source image budget rises from IPC 1 to 50, composed error approaches the local reference, while excess error and accuracy gain follow different budget patterns.At IPC 50, Cind/Cloc is 1.02 ± 0.08 and the crossing-or-plateau outcome remains undetermined; excess is about 1.7× its IPC-10 value at IPC 1, while accuracy gain is near zero at IPC 1, about five points at IPC 5–10, and about two points at IPC 20–50.Source: Figure 16 · p. 45, §J.9 · pp. 45–46, Table 13 · p. 46

J.10 FURTHER DATASETS (SVHN AND CIFAR-100)

  • Across SVHN and CIFAR-100, the endpoint pattern replicates, but the accuracy response differs: skew increases the gain on SVHN, while CIFAR-100’s Dir-0.1 gain is smaller than IID.The paired SVHN increase is +2.24 ± 1.21; CIFAR-100’s paired difference is −1.10 ± 1.00 and unresolved across three partitions.Source: §J.10 · p. 46

J.11 CROSS-ARCHITECTURE EVALUATION (E14)

Across the appendix diagnostics, joint-set accuracy gains persist on a second architecture, while feature-space and assumption checks qualify their interpretation. Preliminary tests show that better constituents do not always yield a better union.

J.11 CROSS-ARCHITECTURE EVALUATION (E14)

  • On CIFAR-10, ResNet-18 joint sets beat unions on every partition by +1.8 to +3.4 points, versus +5.3 to +5.9 on ConvNet.The gain’s dependence on skew differs by evaluation network; the ResNet-18 gain also persists without source heterogeneity.Source: §J.11 · pp. 46–47, Table 15 · p. 47

J.12 FEATURE-SPACE DIAGNOSTICS OF INDEPENDENTLY AND JOINTLY DISTILLED SETS (E8)

  • Under IID, independent sets have a cross/within distance ratio below one, while label skew yields a ratio above one; joint sets occupy more directions per class at matched budgets.These are descriptive statistics in the chosen frozen embedding, not direct measures of real-data coverage.Source: §J.12 · pp. 47–48, Table 16 · p. 48

J.13 ASSUMPTION DIAGNOSTICS (E3)

  • At the distillation rate, ηL ≈1 and the Hessian changes by 14–29% within two steps, so two diagnostics sit near assumption boundaries.ReLU has the largest and most variable Hessian change despite zero activation second derivative almost everywhere; the change is not attributable to activation-boundary crossings alone.Source: Table 17 · p. 48, §J.13 · pp. 48–49

J.14 EARLIER SHORT-SCHEDULE AND PRELIMINARY DIAGNOSTICS

  • For random-projection feature skew, the observed composed error is 13.4 times the local reference and has cosine 0.011 with the quadratic prediction.All eight random-projection gaps are negative; this was an earlier short-schedule diagnostic.Source: Table 18 · p. 49, §J.14 · p. 49
  • At Dir-1.0, stored semantic mass lowers individual error by 60–71% while composed error is unchanged; at Dir-0.1, some improvement survives union.These results use earlier short-schedule or single-condition preliminary diagnostics.Source: §J.14 · p. 49, Figure 17 · p. 50
  • In one Dir-0.1 condition, local reference, quadratic prediction, excess endpoint error, and accuracy gain all grow from two to five sources, but source count is confounded with total budget.The five-source observation does not imply that source-Hessian variance generally grows with source count.Source: Table 19 · p. 50, §J.14 · p. 50

K LIMITATIONS AND FUTURE WORK

The theory and evidence have defined scope boundaries, and the paper identifies open questions about extending the model, reducing source error, and explaining joint distillation’s accuracy advantage.

K LIMITATIONS AND FUTURE WORK

  • Exact results cover full-batch, fixed-step gradient descent on quadratic objectives; network applicability is local and assumes a slowly varying Hessian.The deep-network remainder constant cannot be estimated reliably, so Pcomp calibration is checked empirically rather than certified.Source: K LIMITATIONS AND FUTURE WORK · pp. 50–51
  • The main evidence centers on official-schedule two-source partitions and replications, while one distillation method and two evaluation networks limit the tested scope.Several controls also use few units.Source: K LIMITATIONS AND FUTURE WORK · p. 51
  • Open questions include longer horizons and other optimizers, whether source error can enter the law’s regime, and what produces joint distillation’s accuracy advantage.Future directions include ImageNet-scale endpoint tests, composition-aware objectives, budget-matched source-count studies, and coordination without pooling data.Source: K LIMITATIONS AND FUTURE WORK · p. 51

Related papers

Loading 2609.34343v1…