Source-linked AI summary
Who invented deep residual learning?
Juergen Schmidhuber
TL;DR
Through 2014, backpropagation-based FNNs had remained rather shallow, with at most 20-30 layers or so despite massive help through fast GPU-based hardware.[MLP1-3] The paper traces residual connections from Hochreiter’s recurrent networks through LSTM and Highway Networks to ResNet. The account identifies Sepp Hochreiter’s 1991 thesis as introducing recurrent residual connections and describes ResNet as a feedforward, open-gated Highway Net variant.
Problem
Through 2014, backpropagation-based FNNs had remained rather shallow, with at most 20-30 layers or so despite massive help through fast GPU-based hardware.[MLP1-3]
Source: May 2015: deep Highway Net—gated… · p. 3Method
The paper traces residual connections from Hochreiter’s recurrent networks through LSTM and Highway Networks to ResNet.
Source: 1991: recurrent residual connections… · p. 1, 2005: unfolding LSTM—from recurrent to… · p. 3, May 2015: deep Highway Net—gated… · p. 3Results
The account identifies Sepp Hochreiter’s 1991 thesis as introducing recurrent residual connections and describes ResNet as a feedforward, open-gated Highway Net variant.
Source: 1991: recurrent residual connections… · p. 1, Dec 2015: ResNet—like open-gated… · p. 4Takeaways & Limitations
LSTM brought essentially unlimited depth to supervised RNNs and Highway Networks brought it to FNNs; the account dates the shared residual-connection principle to 1991.
Source: Dec 2015: ResNet—like open-gated… · p. 5Near-1.0 residual weights are acceptable in many applications, but a weight of 0.9 leaves about 0.0027% of an error signal after 100 steps.
Source: 1991: recurrent residual connections… · p. 2
Abstract
from arXiv · showhide
Modern AI is based on deep artificial neural networks (NNs). As of 2025, the most cited scientific article of the 21st century is an NN paper on deep residual learning with residual connections. Who invented this? We present a timeline of the evolution of deep residual learning.
1991: recurrent residual connections solve the vanishing gradient problem
Sepp Hochreiter introduced recurrent residual connections in a June 1991 diploma thesis, which also identified and analyzed vanishing and exploding gradients [DLP][DLH]. The unit-weight identity connection was derived to address that problem.
1991: recurrent residual connections solve the vanishing gradient problem
- Hochreiter introduced recurrent residual connections in his June 1991 diploma thesis, which also identified and analyzed vanishing and exploding gradients [DLP][DLH].The thesis was supervised by Jürgen Schmidhuber.Source: 1991: recurrent residual connections… · p. 1
- A unit with identity activation connects to itself with weight 1.0, adding its current input to its previous activation at each time step.This makes the unit an incremental integrator.Source: 1991: recurrent residual connections… · p. 2
- With derivatives and weights fixed at 1.0, error signals can backpropagate through these units for millions of steps without vanishing or exploding.The invariant connections transport error signals to typically nonlinear adaptive parts, where they can support weight changes.Source: 1991: recurrent residual connections… · p. 2
- A weight of 0.99 retains about 37% of an error signal after 100 steps, whereas a weight of 0.9 retains about 0.0027%; near-1.0 connections are acceptable in many applications.Only weight 1.0 neutralizes vanishing or exploding gradients.Source: 1991: recurrent residual connections… · p. 2
1997 LSTM: plain recurrent residual connections (weight 1.0)
Recurrent residual connections with weight 1.0 are a defining feature of LSTM [LSTM0,1a]. Its core units, called constant error carrousels, support long time lags between inputs and target outputs.
1997 LSTM: plain recurrent residual connections (weight 1.0)
- LSTM core units with weight-1.0 residual connections were called constant error carrousels (CECs).Recurrent residual connections are a defining feature of LSTM [LSTM0,1a].Source: 1997 LSTM: plain recurrent residual… · p. 2
- CECs let LSTM deal with time lags of hundreds or thousands of steps between inputs and target outputs.This became essential for processing speech and language [DLH].Source: 1997 LSTM: plain recurrent residual… · p. 2
- The 1997 LSTM journal paper became the most cited AI paper of the 20th century [MOST].The passage identifies recurrent residual connections as a defining LSTM feature [LSTM0,1a].Source: 1997 LSTM: plain recurrent residual… · p. 2
1999 LSTM: gated recurrent residual connections (gates initially open: 1.0)
The 1999 LSTM variant known as vanilla LSTM [LSTM2,2a] used adaptive multiplicative gates to modulate residual connections. Its forget gates began open at 1.0, and could later close to reset the network.
1999 LSTM: gated recurrent residual connections (gates initially open: 1.0)
- The 1999 vanilla LSTM used adaptive multiplicative gates to modulate residual connections and learn when to reset [LSTM2,2a].Its forget gates were initialized at 1.0, starting with plain residual connections.Source: 1999 LSTM: gated recurrent residual… · p. 2
- The forget gates began open at 1.0, so the LSTM started with plain residual connections [LSTM2,2a].The network could learn to close them later, temporarily shutting down constant error flow and reintroducing vanishing gradients in a controlled way.Source: 1999 LSTM: gated recurrent residual… · p. 2
- Gate closure could temporarily interrupt constant error flow, for example to focus on new tasks.The passage describes this as a controlled return of the vanishing-gradient problem.Source: 1999 LSTM: gated recurrent residual… · p. 2
2005: unfolding LSTM—from recurrent to feedforward residual NNs
The account traces feedforward residual depth from unfolded LSTMs to Highway Networks and then ResNets. Highway Networks adapted gated LSTM residuals for very deep feedforward networks; ResNet is described as their permanently open-gated form.
2005: unfolding LSTM—from recurrent to feedforward residual NNs
- BPTT unfolds an LSTM into a feedforward network with one virtual layer per observed time step, where recurrent residual connections become feedforward residual connections.These deep unfolded networks help explain LSTM’s ability to handle lags of hundreds or thousands of time steps; residual paths retain their role whether weights are shared across time or not.Source: 2005: unfolding LSTM—from recurrent to… · p. 3
May 2015: deep Highway Net—gated feedforward residual connections
- May 2015 Highway Networks adapted the 1999 LSTM gated-residual principle [LSTM2a][LSTM2] to create the first working gradient-trained feedforward nets with hundreds of layers, over ten times prior depth.Highway layers combine a gated carry g(x)x with a transformed input; gates typically start open and can close selectively if that improves performance.Source: May 2015: deep Highway Net—gated… · pp. 3–4
Dec 2015: ResNet—like open-gated Highway Net (or unfolded 1997 LSTM)
- Setting Highway gates permanently to 1.0 yields ResNet, published seven months later; Highway Nets performed roughly as well on ImageNet, while ResNet won the 2015 competition.The account characterizes ResNet as a feedforward version of the 1997 LSTM [LSTM1], and Highway Net as a gated feedforward version of the 1999 LSTM [LSTM2].Source: Dec 2015: ResNet—like open-gated… · pp. 4–5
- The account disputes ResNet’s “concurrent” label, noting that it followed Highway Net by seven months, and says the ResNet paper obscures their open-gated relationship.It also says the ResNet paper does not credit Sepp Hochreiter with first identifying vanishing and exploding gradients and deriving residual connections in 1991.Source: Dec 2015: ResNet—like open-gated… · pp. 4–5
- Highway Networks demonstrated gradient training of very deep feedforward nets, a principle the account says is relevant to Transformers; it traces the residual principle back to 1991.The account describes LSTM residual connections as central to both deep recurrent and feedforward networks [DLH][MIR].Source: Dec 2015: ResNet—like open-gated… · p. 5