FuseReg: Regularizing Layer Fusion Mitigates the Reconstruction-Generation Gap in Representation Autoencoders

Motivation

• Expanding the fused subset from k=7 to k=23 raises reconstruction PSNR from 22.58 to 27.04 dB but also raises unguided gFID from 1.65 to 3.01.The passage attributes the trade-off to shallow, pixel-aligned layers helping reconstruction while deeper semantic layers yield lower gFID in this setting.Source: §1 · p. 2

Approach

• FuseReg trains the decoder and generator on randomly sampled layer subsets, enabling one decoder to support full, sparse, and single-layer fusions without retraining.The generator receives noisy subset fusions and learns to predict the full-layer representation.Source: §1 · p. 2

Theoretical basis

• FuseReg preserves the full-layer mean while explicitly penalizing cross-layer disagreement, separating randomized fusion from fixed layer selection.The paper also establishes a second-order separation from deterministic global fusions, including heuristic selection and learned global gates.Source: §1 · p. 2

Results

• Decoder replacement reduces unguided gFID from 3.01 to 2.21 at k=23, while joint regularization reduces DiT-Base gFID from 13.96 to 9.93.The single decoder also reconstructs from full, sparse, and single-layer fusions; gains extend to other pretrained encoder families.Source: §1 · p. 2

Representation-based generative methods

• RAEs use frozen visual encoders as generative-model latents, while later work redesigns latent spaces or modifies tokenizer and generator objectives for representation alignment.The cited examples include extensions to video generation (Guo et al., 2026).Source: §2 · pp. 2–3

Learned multi-layer fusion

• RAEv2 uses a fixed sum of encoder layers, while DRoRAE and attentive multi-layer probing learn deterministic fusion weights across depths (Singh et al., 2026).These methods seek a preferred layer composition rather than robustness across configurations.Source: §2 · p. 3

Stochastic training methods

• Dropout, nested dropout, LayerDrop, and Matryoshka learning vary paths or subsets during training; FuseReg applies this strategy to frozen encoder layers in an RAE.Its stated aim is robustness across fusion configurations rather than dependence on one fixed fusion.Source: §2 · p. 3

3 FUSEREG: RANDOMIZED LAYER FUSION AS REGULARIZATION

• FuseReg targets dependence on a fixed fusion by constructing a mean-preserving distribution over layer subsets for the two prediction stages of an RAE.The section frames fixed subsets and learned global gates as shared fixed-composition alternatives.Source: §3 · p. 3, §3.1 · p. 3

3.1 FROM A FIXED FUSION TO A DISTRIBUTION OVER LAYER SUBSETS

• Fixed subsets and learned global gates expose downstream models to one layer composition, whereas FuseReg samples subsets during training.In the experiments described, the learned gate concentrates on a single-layer shortcut.Source: §3.1 · p. 3

3.2 TRAINING ON NORMALIZED RANDOM-SUBSET MEANS

• FuseReg samples a separate layer-dropout mask for each example and normalizes each nonempty subset, preserving the deployment latent in expectation.Changing the drop rate varies disagreement around the full-layer mean without shifting that mean in expectation.Source: §3.2 · pp. 3–4

3.3 TWO PREDICTION STAGES, TWO REGULARIZATION RATES

• The decoder reconstructs pixels from subset fusions, while the DiT predicts the full-layer target from noisy subset latents; their drop rates are chosen separately.Setting pdit=0 recovers the standard full-fusion objective; positive pdit trains the generator to map multiple subset fusions to the same full-layer target.Source: §3.3 · p. 4

4 WHY FUSEREG TARGETS LAYER DISAGREEMENT

• Normalized subset sampling leaves the all-layer latent unchanged in expectation and varies it along directions where encoder layers disagree.The paper uses homogeneous linear predictors and squared loss as an analytically tractable surrogate for studying decoder and latent-prediction training signals.Source: §4 · p. 4, §4.1 · p. 5

4.1 WHAT SUBSET SAMPLING CHANGES

• The sampling-induced covariance averaged over inputs is cK(p)Γ, so the drop rate controls variation in disagreement components while shared components remain unchanged.The sampling identity is exact and does not require a linear downstream model.Source: §4.1 · p. 5

4.2 WHY THE TWO STAGES NEED DIFFERENT RATES

• Under the linear squared-loss analysis, subset sampling penalizes predictions that change with layer disagreement, and separate drop rates tune this cost for pixel reconstruction and full-layer latent prediction.The result characterizes the induced training signal rather than replacing the nonlinear experiments.Source: §4.2 · p. 5

4.3 WHY A FIXED FUSION IS NOT EQUIVALENT

• Random subsets preserve average fusion weights while varying every layer-contrast direction, so no deterministic input-independent fusion matches both weight moments.The comparison is limited to input-independent fusion rules; the paper states that the second moment of subset sampling has rank K, versus at most 1 for fixed weights.Source: §4.3 · pp. 5–6

5.1 RECONSTRUCTION ROBUSTNESS ACROSS SUBSETS

• One FuseReg decoder achieves 23.77, 27.52, and 25.13 dB across k=7, k=23, and ℓ11, while fixed-fusion RAEv2 decoders lose substantial PSNR off their training fusion.The FuseReg decoder also has rFID below 0.6 across these fusions.Source: §5.1 · p. 6, Table 1 · p. 6

5.2 GENERATION GAINS FROM DECODER REPLACEMENT

• 3.01→2.21 gFID at native k=23 and 27.73→1.92 at shifted k=7 when the plain RAEv2 decoder is replaced by FuseReg.At native k=23, guided gFID returns to 1.25 at pdec=0.95 after temporarily worsening; under shifted k=7, guided gFID improves from 16.35 to 1.42.Source: §5.2 · p. 7, Figure 3 · p. 7

5.3 GENERATION GAINS FROM JOINT REGULARIZATION

• 9.93 gFID is reached on DiT-Base with joint FuseReg, versus 13.96 for the reproduced baseline; the joint gain exceeds the additive gains from either stage alone.Decoder-only and generator-only settings reach 12.96 and 12.09 gFID, respectively.Source: §5.3 · p. 7, Table 2 · p. 8
• On DiT-XL, generator-only regularization does not improve gFID, while joint regularization yields further gFID gains; IS responds differently, improving as either rate increases.The preferred generator rate also varies by metric and scale.Source: §5.3 · p. 7, Table 2 · p. 8
• At pdit=0, increasing decoder drop rate can initially worsen gFID before higher rates outperform baseline; pdit≥0.5 stabilizes the sweep so nearly all nonzero decoder rates beat baseline.The passage attributes the trend to drop rate amplifying layer-disagreement variance rather than destroying signal.Source: §5.3 · p. 7
• FuseReg’s benefits also extend to other pretrained encoder families.Source: §5.3 · p. 7

• FuseReg reconstructs effectively from individual layers and small subsets, with flatter leave-one-out sensitivity and diminishing marginal gains after fewer fused layers.Because the encoder is frozen, these changes reflect the decoder’s ability to recover information already present across depths.Source: §6 · p. 8, Figure 4 · p. 9
• The shared disagreement covariance acts on different prediction problems: pixel reconstruction across alternative fusions for the decoder, and noisy-trajectory prediction of a full-layer target for the DiT.The DiT’s regularization strength is time-dependent.Source: §6 · pp. 8–9
• Joint regularization gives the best measured DiT-Base result, but DiT-XL favors pdit=0 for lowest gFID and pdit=0.9 for highest IS.The robust decoder benefit persists across these scales, while the preferred generator rate changes.Source: §6 · p. 8
• Internal guidance combines the full DiT output with an intermediate REPA prediction, and guidance strength amplifies changes in their relative responses to layer subsets.Aligned responses can partially cancel in the guided combination, whereas opposing responses reinforce one another.Source: §6 · pp. 8–9

• The empirical scope is ImageNet-256; higher resolutions, other data domains, and longer training schedules remain unevaluated.The objective decompositions assume homogeneous linear predictors and squared loss; extending the characterization to mixed reconstruction losses and the complete guided sampling trajectory remains open.Source: §7 · p. 9

8 CONCLUSION

• FuseReg trains downstream models across sampled layer compositions, making information across encoder depths more accessible while keeping the pretrained encoder fixed.The conclusion reports a single decoder supporting full, sparse, and single-layer fusions, with decoder and generator regularization strengths chosen separately.Source: §8 · p. 9

A.1 REPRESENTATION AUTOENCODERS AND TOKENIZER DESIGN

• RAEv2’s fixed-k fusion defines a reconstruction–generation Pareto frontier because one latent serves a pixel decoder and a generator with different preferences.FuseReg takes this fixed-fusion trade-off as its starting point.Source: §A.1 · p. 14

A.2 REPRESENTATION ALIGNMENT AND GUIDANCE

• Work on representation alignment finds that alignment benefits depend on architecture and the space in which alignment is applied.The cited work includes intermediate DiT alignment, tokenizer-level alignment, and cases where direct pixel-space alignment can fail.Source: §A.2 · p. 14

A.3 LAYER-FUSION LEARNING AND STOCHASTIC CONFIGURATION

• Learned fusion methods converge on one combination rule per model or task, whereas FuseReg trains across random layer subsets.The authors report that a globally learned gate collapses toward the shallowest, pixel-aligned layer under pixel-reconstruction loss.Source: §A.3 · p. 15
• Unlike stochastic configuration methods aimed at deployment efficiency, FuseReg places every nonempty subset of frozen encoder layers in training support.The paper relates this approach to dropout, LayerDrop, slimmable networks, and Matryoshka Representation Learning.Source: §A.3 · p. 15

B EXPERIMENTAL SETUP AND EVALUATION

• The experiments evaluate decoder fusion robustness first and then the full generation pipeline using frozen DINOv3-L features and ImageNet at 256×256 resolution.The main reconstruction fusions are k=23, k=7, and single layer ℓ11; evaluation uses PSNR, SSIM, rFID, gFID, and IS.Source: B EXPERIMENTAL SETUP AND EVALUATION · p. 15, §B.1 · p. 15

B.2 TRAINING BUDGETS AND REGULARIZATION GRIDS

• Decoder and generator FuseReg rates are varied independently, and generator studies evaluate each trained DiT across the corresponding decoder-rate grid.Decoder sweeps train for 16 epochs, while the DiT grids use 40 epochs; controls are matched within each model scale.Source: §B.2 · pp. 15–16, Table B.1 · p. 16

B.3 BASELINES AND CONTROLLED COMPARISONS

• The decoder-swap study holds the generator and sampled latents fixed, while joint grids vary both training rates.Table 1 uses official RAEv2 fixed-fusion checkpoints; rate-sweep baselines are reproduced and comparisons are made within each table.Source: §B.3 · p. 16

C SUPPLEMENTARY RECONSTRUCTION RESULTS

• The primary and supplementary reconstruction results use a frozen, pretrained DINOv3-L encoder.The primary comparisons distinguish full-fusion fidelity from robustness to reduced layer inputs.Source: C SUPPLEMENTARY RECONSTRUCTION RESULTS · p. 16, Table C.1 · p. 16

C.1 DECODER-RATE SELECTION ACROSS LAYER FUSIONS

• At pdec=0.95, one decoder reaches 23.77, 27.52, and 25.13 dB on full, k=7, and ℓ11 fusions, respectively.The nearby pdec=0.9 setting retains slightly higher full-fusion PSNR, illustrating rate selection for the intended mix of inputs.Source: §C.1 · p. 16, Table C.1 · p. 16

C.2 QUALITATIVE RECONSTRUCTION UNDER CHANGED FUSIONS

• On k=7 inputs, FuseReg reaches 19.6–22.4 dB versus 18.9–21.9 dB for the matched RAEv2 decoder; on ℓ11 inputs it reaches 20.2–23.3 dB versus 14.5–17.2 dB for fixed-fusion baselines.For k=7, the RAEv2 k=23 decoder collapses to low-contrast outputs at 10.8–14.4 dB; FuseReg supports both changed fusions without retraining.Source: §C.2 · pp. 16–17, Figure C.1 · p. 17

C.3 LAYER RELIANCE, SUBSET VALUE, AND MARGINAL GAINS

• FuseReg reconstructs better from small layer subsets and approaches full-fusion performance with fewer layers, with diminishing gains also visible in MSE.Its broadly preserved Shapley ordering indicates that improved subset robustness coexists with unequal layer contributions.Source: §C.3 · p. 18, Figure C.2 · p. 18

D.1 DECODER REPLACEMENT WITH FIXED GENERATED LATENTS

• With the generator and latents fixed, decoder replacement lowers unguided gFID from 3.01 to 2.21 at k=23 and from 27.73 to 1.92 at k=7.At k=7, guided gFID falls from 16.35 to 1.42; the k=23 guided gFID remains 1.25.Source: §D.1 · p. 18, Table D.1 · p. 19

D.2 COMPLEMENTARITY OF DECODER AND GENERATOR REGULARIZATION

• On DiT-Base, joint decoder and generator regularization reaches gFID 9.93 from a 13.96 baseline, 1.16 beyond the additive extrapolation of their individual gains.This four-setting comparison is at the selected configuration and is not a statistical interaction test.Source: §D.2 · p. 19

D.3 RATE SELECTION UNDER INTERNAL GUIDANCE

• Under internal guidance, the lowest DiT-XL gFID is 1.50 at (pdec, pdit) = (0.9, 0), versus 1.57 at (0, 0), while the highest IS remains at (0, 0).The guidance setting changes which rate pair is preferred by each metric.Source: §D.3 · p. 19, Table D.2 · p. 19

E GENERALIZING TO OTHER PRETRAINED ENCODER FAMILIES

• FuseReg’s reconstruction robustness and generation gains extend beyond DINOv3-L to frozen SigLIP2-L and EUPE-B encoders.The evaluation uses ImageNet-256, with decoder training for 16 epochs and DiT-Base training for 40 epochs.Source: E GENERALIZING TO OTHER PRETRAINED… · p. 20

E.1 RECONSTRUCTION ROBUSTNESS ACROSS LAYER SUBSETS

• Across SigLIP2-L and EUPE-B, FuseReg substantially improves reconstruction from sparse and single-layer inputs while preserving full-fusion quality at moderate decoder rates.The preferred rate varies with encoder, input fusion, and metric.Source: §E.1 · p. 20, Table E.1 · p. 20

E.2 GENERATION GAINS FROM JOINT REGULARIZATION

• Joint FuseReg reduces gFID from 13.77 to 8.26 for SigLIP2-L and from 12.17 to 8.00 for EUPE-B.In both grids, the best gFID and IS occur with a strongly regularized decoder, while the preferred generator rate depends on encoder and metric.Source: §E.2 · p. 21, Table E.2 · p. 21

F THEORETICAL ANALYSIS AND PROOFS

• The theory analyzes size-normalized nonempty random-subset aggregation, with exact sampling, mean, and covariance properties extending beyond linear consumers.Objective decompositions and closed-form optimizers require homogeneous linear predictors and squared loss, so they provide mechanism-level intuition rather than quantitative predictions for nonlinear models.Source: F THEORETICAL ANALYSIS AND PROOFS · p. 22

F.1 MEAN PRESERVATION AND DISAGREEMENT COVARIANCE

• Increasing the layer drop rate raises covariance along layer-disagreement directions while leaving the expected latent equal to the all-layer mean.The covariance coefficient is bounded in [0, 1] and nondecreasing with drop rate.Source: §F.1 · p. 23

F.2 STAGE-SPECIFIC OBJECTIVES AND OPTIMAL LINEAR PREDICTORS

• Increasing drop rate suppresses a linear predictor’s gain along shared directions where layers disagree, while leaving consensus directions unaffected.For the decoder, the closed form suppresses directions with large disagreement covariance; the DiT receives a time-weighted disagreement penalty while predicting the full-layer consensus from noisy subset inputs.Source: §F.2 · pp. 23–25

F.3 COVERAGE OF NONEMPTY SUBSETS

• FuseReg’s objective includes every nonempty fusion, and a small average loss bounds each subset’s loss more tightly when that subset is sampled more often.The bound applies to nonnegative reconstruction loss and weighted squared DiT training loss conditioned on a subset.Source: §F.3 · p. 25
• A decoder can have zero loss on the all-layer fusion yet exceed any chosen loss threshold on a Hamming-adjacent singleton fusion.Two layers can cancel noise in the full fusion, hiding arbitrarily large errors when one layer is removed.Source: §F.3 · p. 25

F.4 SEPARATION FROM DETERMINISTIC GLOBAL FUSION

• Normalized random-subset weights have a full-rank second-moment matrix, separating their linear objective from any deterministic global fusion.For a homogeneous linear consumer with squared error, the objective depends on the first two weight moments; the rate relaxation used by FuseReg is a structured slice, not an exact parameterization of every subset pair.Source: §F.4 · p. 26, §F.5 · p. 27

F.5 PER-LAYER RATES CONTINUOUSLY RELAX SUBSET SELECTION

• The per-layer-rate relaxation recovers every nonempty deterministic subset at its corresponding nonzero vertex and is analytic in the interior under the stated invertibility conditions.FuseReg uses equal per-layer rates at each stage, yielding a structured two-dimensional slice of the product relaxation.Source: §F.5 · p. 27

F.6 FROM EXACT IDENTITIES TO NONLINEAR-MODEL INTUITION

• The exact subset identities and full-support bound do not require linear consumers, but the linear-predictor propositions assume homogeneous linear predictors and squared loss.For nonlinear consumers, latent mean preservation does not imply prediction mean preservation or an exact disagreement penalty; the analysis does not predict the two-rate contrast’s sign or establish super-additivity.Source: §F.6 · p. 27