PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Delta Learning

Yifan Zhang, Yifeng Liu, Mengdi Wang, Quanquan Gu

arXiv:2601.00417v4cs.LGcs.AIcs.CLcs.CV

TL;DR

Standard residual addition leaves reading, comparison, and replacement implicit even though an expressive branch can represent them. Deep Delta Learning makes these operations an explicit target-seeking residual update, and reported pretraining runs show improved validation loss and one-shot average accuracy, with expanded-state gains carrying additional costs.

  • Problem

    Standard residual addition leaves reading, comparison, cancellation, and writing implicit inside an unconstrained branch, although it can represent replacement in principle.

    Source: §1 · p. 1

  • Method

    Deep Delta Learning uses a gated rank-1 update that reads the residual along a learned direction, compares it with a target, and writes the discrepancy back along that direction.

    Source: §1 · pp. 1–2

  • Results

    DDL lowers validation loss and raises one-shot average accuracy in the reported scalar and expanded-state runs, with larger gains in expanded variants.

    Source: §1 · p. 3

  • Takeaways & Limitations

    DDL provides a precise residual interface and a promising quality–efficiency tradeoff within the scope of the reported experiments.

    Source: §1 · p. 3, §7 · p. 13

    The experiments do not establish an iso-FLOPs or iso-wall-clock advantage, and expanded-state gains cannot be assigned uniquely to the read-and-erase term without a matched control.

    Source: §2.2 · p. 5, §5 · p. 12

Abstract

from arXiv · show

Transformer residual streams evolve through additive updates. Although a sufficiently expressive residual block can represent content replacement, standard architectures do not parameterize reading, comparison, and replacement as an explicit residual operation. We introduce Deep Delta Learning (DDL), a structured residual update that preserves the identity path while enabling target-seeking edits to the residual state. Each layer reads the current state along a learned direction, compares the resulting readout with a learned target, and writes back a gated rank-1 correction along the same direction. Closing the gate recovers the identity map, while fully opening it exactly overwrites the selected residual readout. We instantiate DDL with both scalar and expanded residual states. The expanded formulation provides multiple persistent value channels while keeping attention and MLP computation at the original model width, thereby separating residual-state capacity from backbone compute width. Controlled LLM pretraining experiments show that DDL improves language-modeling quality and average one-shot downstream performance over additive residual baselines in the reported runs, while introducing explicit memory and throughput tradeoffs. These results suggest that depth-wise delta-rule updates provide a useful inductive bias for managing Transformer residual streams.

1 Introduction

Deep Delta Learning makes residual reading, comparison, and writing explicit as a target-seeking update, while retaining the identity path. Reported pretraining runs show quality gains over additive residual baselines, with expanded-state gains accompanied by architectural and evaluation tradeoffs.

1 Introduction

  • Standard residual addition can represent replacement, but leaves reading, comparison, cancellation, and writing implicit in an unconstrained branch.Source: §1 · p. 1
  • DDL parameterizes a gated rank-1 read-compare-write correction: it reads along a learned direction, compares with a target, and writes the discrepancy back along that direction.Closing the gate recovers the identity map; βl = 1 exactly matches the selected readout to the target.Source: §1 · pp. 1–3
  • The selected-coordinate error contracts for 0 < βl < 2, is removed at βl = 1, and receives an over-relaxed correction when βl > 1.The analysis concerns local shortcut geometry and does not claim semantic interpretability of the learned direction.Source: §1 · p. 3
  • Expanded residual states add persistent value channels while attention and MLP computation remain at width d.A compressor maps the expanded state to width d before standard attention or MLP processing; expansion adds memory, bandwidth, and compression costs.Source: Figure 1 · p. 2, §1 · p. 3
  • DDL lowers validation loss and raises one-shot average accuracy in the reported scalar and expanded-state runs, with larger gains in the expanded variants.The experiments use equal-token training; expanded variants also change residual capacity, input expansion, and compression.Source: §1 · p. 3
  • The reported experiments train decoder-only models on FineWeb-Edu for 49.15B tokens at approximately GPT-2 small and medium parameter budgets, but use one run per configuration and are not iso-FLOPs comparisons.The reported evidence therefore does not isolate all expanded-state gains to the erase term.Source: §1 · p. 3

2 Deep Delta Learning

Deep Delta Learning structures the residual correction as a target-seeking rank-1 update while preserving an identity path. The section extends this update to expanded residual states and analyzes its conditioned shortcut geometry.

2 Deep Delta Learning

  • DDL parameterizes the residual correction as a target-seeking rank-1 update, while a sufficiently expressive conventional residual branch can represent the same correction.The selected read, target, direction, and step size are explicit quantities.Source: §2 · p. 4

2.1 Delta Residual Rewrite

  • The shared gate synchronizes erasure and writing: β_l = 0 preserves the state, β_l = 1 matches the selected readout to its target, and 0 < β_l < 2 contracts its discrepancy.Separate erase and write gates would permit a more general affine update but would not enforce synchronized target matching.Source: §2.1 · pp. 4–5

2.2 Expanded Residual State

  • Expanded residual states are compressed to width d for standard sublayer processing, then updated persistently; this increases state capacity without widening attention or MLP activations.The design also increases activation traffic and compression and rewriting costs; without a matched expanded-state additive control, gains cannot be assigned uniquely to the erase term.Source: §2.2 · p. 5

2.3 Spectral Analysis

  • For the frozen shortcut, eigenvalue 1 holds on k⊥ and 1 − β_l on k; the same operator acts on each value column, leaving k⊥ unchanged.This spectrum is conditioned on one layer, token, and residual state, not the full nonlinear block.Source: §2.3 · pp. 5–6
  • Locally, β_l near 0 approaches a skip, β_l = 1 exactly matches the target, and β_l > 1 crosses it; the learned direction need not represent a human-readable feature.As β_l approaches 2, only the direct shortcut approaches a Householder reflector.Source: §2.3 · p. 6

3 DDL Transformer

The Transformer implementation preserves width-d attention and MLP computation while changing the residual interface. It supports scalar and expanded residual states, with additional costs for expanded-state variants.

3 DDL Transformer

  • DDL is evaluated in decoder-only Transformers with pre-norm RMSNorm, RoPE multi-head attention, and SwiGLU MLPs, while preserving width-d attention and MLP computation.The ordinary attention or MLP sublayer supplies the edit direction, while lightweight branches produce the target and gate.Source: §3 · p. 6

3.1 Scalar residual state: dv = 1

  • The scalar-state setting avoids expanded residual capacity and compression, but still adds direction normalization and lightweight value and gate branches.The reported unit-direction analysis is accurate when the proposed direction norm is much larger than ε_k.Source: §3.1 · p. 7

3.2 Expanded residual state: dv > 1

  • Expanded-state DDL evaluates dv = 4, initializing value channels with embedding convolution by default and compressing them before standard width-d sublayer processing.The protocol compresses the state, processes it with attention or an MLP, then rewrites the persistent state.Source: §3.2 · p. 7
  • Expanded-state updates and compression add per-token work and activation traffic; DDL-TC also needs a short per-layer history during autoregressive generation.DDL-CC mixes only the current token's channels and avoids that additional token-history cache; the variants lack a matched expanded-state additive control.Source: §3.2 · pp. 7–8

4 Experiments

Across two equal-token scales, scalar and expanded-state DDL have lower final validation-loss point estimates than the baseline, and one-shot averages rise; expanded-state results carry throughput and memory costs.

4.1 Experimental settings

  • Configurations train on FineWeb-Edu for 49.15B tokens at approximately 124M and 353M parameters, with one run per configuration, making the comparisons equal-token rather than iso-FLOPs.The reported point estimates are not means over seeds.Source: §4 · p. 8, §4.1 · pp. 8–9

4.3 Expanded-state implementation variants

  • The dv=4 DDL-TC mixes over sequence positions and DDL-CC over value channels; DDL-CC is the default for measured quality–efficiency compromise, and w/o EC ablates input expansion.Bare DDL refers to DDL-CC.Source: §4 · p. 8, §4.3 · p. 9

4.2 Experimental results

  • One-shot average accuracy rises by 0.17 and 0.73 points for scalar DDL and 0.91 and 1.18 points for the best expanded-state variants at the two scales, while zero-shot averages are mixed.Downstream evaluations are treated as secondary evidence, not as uniform task-level dominance.Source: Table 1 · p. 8, §4.2 · p. 9, Table 2 · p. 10
  • Expanded-state best validation loss is 2.8299 (a reduction of 0.0244 from baseline) at small scale and 2.5758 (a reduction of 0.0295) at medium scale; scalar DDL reduces validation loss by 0.0061 and 0.0014, respectively.These are single-run point estimates; scalar differences cannot be distinguished from training-seed variation in the reported design.Source: §4.2 · p. 9, Table 3 · p. 11
  • Expanded-state models trade lower validation loss for lower throughput and higher memory use; hardware-specific measurements do not replace total-FLOP or iso-FLOPs comparisons.The reported throughput and peak-memory measurements are from Tables 4 and 5.Source: §4.2 · p. 9, Table 4 · p. 11, Table 5 · p. 11

5 Scope of the Empirical Evidence

The empirical evidence is bounded by single-run training, unresolved attribution in expanded-state models, missing compute-matched comparisons, and operator-level rather than semantic interpretability.

Training variance

  • Each configuration has one training run, leaving scalar improvements without per-seed variance and not statistically resolved.Scalar DDL is numerically better than the baseline in final validation loss and one-shot average at both scales, but the margins are small.Source: §5 · p. 11

Attribution in the expanded state

  • Expanded-state results support the combined architecture, but do not identify delta rewriting alone as the cause of its gains.The dv = 4 models jointly change residual capacity, compression, and sometimes input expansion; the no-EC ablations do not isolate the read-and-erase term.Source: §5 · pp. 11–12

Compute matching

  • Equal-token training and measured throughput or memory do not establish an iso-FLOPs or iso-wall-clock advantage.The study lacks FLOPs-per-token, total-pretraining-FLOPs, and end-to-end GPU-hours measurements; the medium-scale cost table also lacks scalar DDL.Source: §5 · p. 12

Operator-level rather than semantic interpretability

  • Interpretability is limited to the conditioned operator, not semantic meaning in the model’s internal representations.The experiments do not audit learned gate regimes or discrepancy reduction, or show that k_l is human-readable.Source: §5 · p. 12

6 Related Work

DDL relates to residual pathways, depth-wise delta-rule updates, expanded residual states, and low-rank transformations, while distinguishing its explicit residual correction from those neighboring ideas.

Residual and gated pathways

  • DDL’s distinction from residual and gated pathways is an explicit normalized rank-1 correction with a target readout, shared erase/write gate, and identity limit.Conventional residual blocks can represent replacement; DDL does not claim otherwise.Source: §6 · p. 12

Delta rules and memory updates

  • The delta rule is established in sequence-memory models, whereas DDL applies its erase/write pattern over network depth as the residual interface between Transformer sublayers.Its contribution is the depth-wise architectural use and analysis of the rule, not invention of the delta rule.Source: §6 · p. 12

Expanded residual states and compression

  • For dv > 1, DDL separates persistent-state capacity from expensive sublayer width, relating it to expanded or projected residual streams.DDL-CC uses standard learned channel mixing as its compressor, treated as an implementation choice rather than a novel operation.Source: §6 · p. 12

Orthogonal and low-rank transformations

  • DDL imposes no global orthogonality; only its frozen direct shortcut approaches a Householder reflector as β approaches 2.The complete layer is input-dependent and affine after conditioning.Source: §6 · pp. 12–13

7 Conclusion

DDL makes target-seeking rank-1 residual corrections explicit while retaining an identity limit and exact target matching at β = 1. Reported equal-token runs show lower validation-loss point estimates than baseline, but leave statistical robustness and mechanism attribution unresolved.

7 Conclusion

  • DDL parameterizes residual correction as a target-seeking rank-1 read-compare-write update, although additive residual branches can represent the same map.The design makes the selected readout, target, discrepancy, and step size explicit, with an identity limit and exact local target matching at β = 1.Source: §7 · p. 13
  • At approximately 124M and 353M parameters, all reported DDL configurations give lower final validation-loss point estimates than baseline under equal-token training.Scalar gains are small; expanded-state gains are larger but jointly reflect extra residual capacity, compression, and the DDL update.Source: §7 · p. 13
  • The results motivate further study, since they do not resolve complete mechanism attribution or statistical robustness.The conclusion identifies matched expanded-state write-only controls, multi-seed evaluation, compute-matched curves, and an audit of gate and discrepancy use as needed evidence.Source: §7 · p. 13

A Implementation and Parameterization Details

The appendix distinguishes the concrete parameterization used in reported models from the abstract generator functions and specifies compression and EC naming conventions.

A Implementation and Parameterization Details

A.1 Reported Direction, Value, and Gate Generators

The reported generators use the standard sublayer output to set the edit direction, a normalized context to produce the target, and a token-wise scalar gate to control the update.

A.1 Reported Direction, Value, and Gate Generators

  • The standard attention or MLP output determines the edit direction; no independent MLP predicts that direction from pooled residual statistics.A small epsilon guard is used for precision-friendly normalization.Source: §A.1 · p. 17
  • The target comes from the same normalized context as the backbone, as a scalar for dv = 1 or a value-channel vector for dv = 4.The value branch adds no attention or MLP block.Source: §A.1 · p. 17
  • The token-wise scalar gate is computed in fp32, with its output bias initialized from the configured starting value β0.The same gate configuration is used across reported variants unless stated otherwise.Source: §A.1 · p. 17

A.2 Expanded-state Transformer Implementation Details

Expanded-state DDL uses multiple value channels and different compression axes, while relating its depth-wise residual update algebraically to DeltaNet’s time-wise delta-rule update. The appendix also identifies the model-size hyperparameter table.

A.2 Expanded-state Transformer Implementation Details

  • Expanded-state DDL represents each residual as (B, T, d, dv), uses embedding convolution by default, and compresses along tokens in TC or value channels in CC.The EC-enabled CC variant is the default because it provides the best reported overall quality–cost tradeoff.Source: §A.2 · pp. 18–19
  • Autoregressive generation with token-axis convolution requires retaining the last k −1 expanded residual states per layer, in addition to the attention KV cache.Source: §A.2 · p. 18

B Relation to DeltaNets and Householder Products

  • DDL and DeltaNet share a rank-1 shortcut operator, but apply the delta-rule update over network depth and time, respectively.The paper limits its novelty claim to a depth-wise Transformer residual interface and its analysis and instantiation.Source: B Relation to DeltaNets and… · p. 19

C Hyper-parameter Settings

D Additional Results

The supplementary section reports zero-shot evaluations for small and medium models. Its task-level estimates are mixed across implementations and are presented as supplementary rather than uniform evidence of downstream improvement.

D Additional Results

Related papers

Loading 2601.00417v4…