PaperlayerView exact revision on arXiv

Source-linked AI summary

Gate Initialisation versus Recurrent Dropout and Zoneout in Learning Long Dependencies: A Controlled Study of LSTM and GRU Networks on Synthetic Tasks

Oleksandr Kholodniak, Oleksandr Prokhorov

arXiv:2610.29034v1cs.LGcs.NE

TL;DR

Long dependencies make recurrent-network gradients decay with lag, and it is unclear whether early gradient propagation helps predict which runs will learn. The study crossed gate initialisation with recurrent regularisers and measured learning and input sensitivity; chrono initialisation produced the largest learning contrast, while early sensitivity added little condition-level predictive value.

  • Problem

    Long-dependency gradients can decay with lag, and whether early gradient reach predicts learning matters because training is slow and outcomes vary across seeds.

    Source: §1 · pp. 1–2

  • Method

    The study compared conventional and chrono gate initialisation with recurrent regularisers in LSTM and GRU networks on adding and delayed-recall tasks, recording input sensitivity across lags.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    At tested rates and lengths, gate initialisation produced larger learning contrasts than regularisers; chrono retained its advantage with horizons about four times too short or too long, and early sensitivity added little condition-level prediction.

    Source: §1 · p. 2, §5 · p. 12

  • Takeaways & Limitations

    The results support chrono initialisation, even with a rough horizon, on synthetic long-dependency tasks.

    Source: §6 · p. 13

    Because both synthetic tasks use fresh or exhaustive training data, the study measures optimisation effects rather than generalisation from a finite training set.

    Source: §5 · p. 12

Abstract

from arXiv · show

Dropout and zoneout are routinely added to long short-term memory (LSTM) and gated recurrent unit (GRU) networks, but, to our knowledge, their placements have not been crossed with gate initialisation in a factorial design, and it is not known whether early gradient propagation tells which runs will learn long dependencies. We trained one-layer LSTMs and GRUs on the adding problem and on delayed recall in a factorial grid of 720 runs, crossing conventional and chrono gate initialisation with up to six recurrent regulariser placements at one rate each, with pre-specified follow-ups, and recorded how sensitive the final hidden state was to the input at every lag. Because both tasks supply fresh or exhaustive training data, the study measures the regularisers' effects on optimisation and does not assess generalisation from a finite training set. At lengths chosen to make conventional initialisation difficult, all 360 chrono-initialised runs reached the validation criterion within 5000 updates, as did 119 of 120 unregularised runs with a horizon about four times too short or too long, whereas conventionally initialised networks learned slowly or did not reach the criterion within the budget at the longer lengths. Under conventional initialisation, dropout generally delayed learning and zoneout helped only in some conditions. The early input sensitivity added little to predicting which conditions learn once the early validation loss was modelled per task, architecture and initialisation. Within conditions, a flatter early decay went with the seeds that learned, but in a pre-specified test the association was modest (concordance 0.61), condition-dependent and not robust to resampling seeds. The results support chrono initialisation, even with a rough horizon, on such tasks.

1 Introduction

The introduction frames long-dependency learning as a problem of decaying gradients and asks how recurrent regulariser placements compare with gate initialisation, and whether early gradient reach predicts learning. Across the tested tasks and settings, initialisation produced the larger learning contrast, while early sensitivity added little predictive value beyond early validation loss.

1 Introduction

  • In recurrent networks trained with BPTT, inputs seen T steps before the loss receive direct credit only through gradients carried back over those steps; contracting Jacobians make the gradient decay exponentially with lag.Source: §1 · p. 1
  • The study crosses gate initialisation with recurrent regulariser placements in one-layer LSTMs and GRUs on two long-dependency tasks, then measures input sensitivity by lag.The grid uses five seeds and includes pre-specified follow-ups.Source: §1 · p. 2
  • Gate initialisation produced larger learning contrasts than any tested regulariser, and chrono’s advantage persisted with horizons about four times too short or too long.Regularisers still changed attainment and learning time in some conditions.Source: §1 · p. 2
  • Zoneout flattened the input-sensitivity slope after 100 updates, but a similar flattening appeared in evaluation mode before training.The comparison concerns changes in slope across regulariser placements (Table 3).Source: §1 · p. 2
  • Early input sensitivity added little to predicting which conditions learn after accounting for early validation loss, while its seed-level association was modest and not robust to resampling seeds.The pre-specified test supported the association when resampling conditions, not seeds (Figure 3).Source: §1 · p. 2
  • The authors release the code, per-run results, and gradient curves for every run.Figure 1 summarises the main grid.Source: §1 · p. 2

2 Related work

Related work contrasts recurrent dropout placements and zoneout, gate-bias strategies for long memory, and gradient-propagation analyses. The paper’s study measures gradient reach as an observable and tests whether it predicts training outcomes.

Regularising the recurrence

  • Prior recurrent-dropout methods differ in where masks act: on non-recurrent connections, the cell state, gate inputs, or candidate update.The cited work motivates placements partly by whether stored state is erased [23].Source: §2 · p. 2

Related work

  • Figure 1 cells report criterion-reaching runs and restricted mean updates; compare chrono with conventional initialisation across architectures and lengths.The caption reports that chrono reached the criterion in every main-grid run, while conventional initialisation often did not at longer lengths.Source: Figure 1 · p. 3

Regularising the recurrence

  • Zoneout retains dropped units at their previous values; among these works, only Krueger et al. (2017) reported a gradient measurement, on one task.Their measurement was a normalised snapshot of cell-state gradient norms.Source: §2 · p. 4

Gating and initialisation

  • Chrono initialisation links gate biases to task-required timescales, while other approaches alter recurrent matrices or use recurrences designed for long sequences.The study focuses on classic nonlinear gated cells, where gate biases play this role.Source: §2 · p. 4

Gradient propagation and trainability

  • Prior work analyses gradient decay, propagation timescales, and gradient-norm estimates, whereas this study treats gradient measurements as observables and tests their predictive value.The cited applications include choosing truncation lengths and assembling mini-batches.Source: §2 · p. 4

3 Methods

The study compares one-layer LSTMs and GRUs across gate initialisations and recurrent regularisers on two long-dependency tasks, measuring training outcomes and early input sensitivity. Pre-specified follow-ups and prediction analyses test how these factors relate to learning.

3.1 Networks

  • Networks used conventional or chrono gate biases; the GRU update gate starts at 0.5 conventionally, while chrono biases assign units timescales drawn up to Tmax.The main grid set Tmax to the sequence length; follow-ups used horizons T/4 and 4T.Source: §3.1 · p. 4
  • The study compared six recurrent-regulariser placements for LSTMs and four for GRUs, including recurrent and variational dropout, candidate-update dropout, and zoneout.Dropout rates were 0.25 and zoneout rates 0.15; at evaluation, dropout was off and zoneout used its expectation.Source: §3.1 · p. 5

3.2 Tasks

  • The adding task used marked values whose sum was predicted at the final step, while delayed recall required reproducing one of eight symbols after intervening zero inputs.Adding used fresh sequences and recall validated on all eight possible sequences, so neither task assessed generalisation from a finite training set.Source: §3.2 · p. 5

3.3 Training and endpoint

  • Runs used Adam with learning rate 10^-3, batches of 32, and jointly clipped recurrent and output gradients; training stopped at the validation criterion or budget.The criterion was validation mean squared error below 0.01 for adding or recall of all eight symbols; the main budget was 5,000 updates.Source: §3.3 · p. 5

3.4 Gradient probes

  • At scheduled updates, probes measured gradients of final-hidden-state projections with respect to inputs at each lag, summarising sensitivity by a log-slope over the more recent half of the sequence.The slope describes relative attenuation in that window, not sensitivity to the earliest inputs; evaluation-mode probes disable dropout but include expected zoneout.Source: §3.4 · pp. 5–6

3.5 Experimental design

  • The factorial design crossed task, sequence length, architecture, gate initialisation and regulariser across 720 runs, with five seeds and three pre-specified follow-ups.The follow-ups added seeds, extended the budget to 20,000 updates, and tested mismatched chrono horizons.Source: §3.5 · p. 6, Table 1 · p. 6

3.6 Statistical analysis

  • The study compared a gradient-free prediction baseline with a model adding early input-sensitivity slope, using grouped out-of-sample folds and resampling to assess prediction.A within-condition concordance analysis ranked successful versus unsuccessful runs; a confirmatory test was specified before follow-up runs.Source: §3.6 · pp. 6–7

4 Results

Chrono initialisation produced much stronger learning outcomes than conventional initialisation, even with substantially mismatched horizons. Regulariser effects varied by condition, while early input sensitivity added little to cross-condition prediction of learning.

4.1 In this grid, initialisation decided whether and how fast the tasks were learned

  • All 360 chrono-initialised runs reached the criterion within 5000 updates; 120 of 360 conventionally initialised runs did, and 119 of 120 unregularised runs with fourfold horizon mismatch succeeded.With a fourfold budget, 121 of 160 runs in the shorter conventional conditions reached the criterion within 20,000 updates.Source: §4.1 · p. 7

4.2 Regularisers changed learning speed, and conventional initialisation exposed the difference

  • Under conventional initialisation, dropout generally delayed learning and sometimes prevented criterion attainment, while zoneout helped in some conditions but was not uniformly beneficial.With chrono initialisation every regulariser reached the criterion in every run, though learning times still differed and contrasts were tested at one rate per family.Source: §4.2 · p. 8

4.3 Initialisation set the decay of input sensitivity more than the regularisers did

  • After 100 updates, conventional networks showed steep input-sensitivity decay, whereas chrono networks were nearly flat; regulariser effects were smaller and placement-dependent.Zoneout flattened decay when applied to the memory-carrying path, but a similar effect was already present before training in evaluation mode.Source: §4.3 · pp. 8–10, Figure 2 · p. 9, Table 3 · p. 10

4.4 The early slope added little to predicting which conditions learned

  • Adding early input-sensitivity slope to a stratified predictor reduced log loss only from 0.162 to 0.154, with an interval including zero, so it added little beyond early validation loss and condition.The larger reduction against the additive baseline was mostly attributable to structure that baseline could not express.Source: §4.4 · p. 10, Table 4 · p. 10

4.5 Within some conditions, flatter early decay went with the seeds that learned

  • In confirmatory runs, early slope showed a modest, condition-dependent association with which seeds learned, but its concordance of 0.613 was not robust to resampling seeds.The resampled-seed and crossed condition-and-seed intervals included 0.5; the association varied across task and architecture conditions.Source: §4.5 · pp. 11–12, Figure 3 · p. 11

5 Discussion

Gate initialisation produced larger learning contrasts than the tested recurrent regularisers, while the early gradient-slope measure added little to prediction beyond early validation loss. The conclusions are bounded by the synthetic-task design, selected settings, and limits of the sensitivity measure.

5 Discussion

  • With matched chrono initialisation, every run reached the criterion and regularisers changed only learning time; a horizon about four times too short or long still worked in 119 of 120 unregularised runs.Conventional initialisation showed sensitivity decay by orders of magnitude, and regulariser choice could determine whether a run learned within the budget.Source: §5 · p. 12
  • Zoneout on the memory-carrying state flattened input sensitivity, but its learning benefits varied; dropout generally slowed learning and often steepened decay after training began.Variational dropout flattened decay at training-mode initialisation but steepened it most after training began; the proposed fixed-mask compensation mechanism was not tested.Source: §5 · p. 12
  • Early validation loss predicted learning nearly as well without gradient slope, whose within-condition association was modest, condition-dependent, and not robust to resampling seeds.The within-condition findings concern conventional initialisation; chrono runs had no outcome variation to predict.Source: §5 · p. 12
  • The results support chrono initialisation for one-layer gated networks carrying information over many steps, but do not assess regularisation's generalisation benefits on finite training data.The study also excludes non-recurrent input/output dropout and did not evaluate an early-stopping policy based on either predictor.Source: §5 · p. 12
  • The initialisation contrast reflects task lengths chosen so most conventional conditions missed the criterion at the longest length within 5000 updates; the study also used one-layer, 64-unit networks and one optimiser and learning rate.Other limitations include one rate per regulariser family and two synthetic tasks with fresh or exhaustive training data.Source: §5 · p. 12
  • The measured sensitivity concerns projections of the final hidden state, not the LSTM cell-state Jacobian, and one exponential fits chrono curves poorly.The endpoint was first validation success; 19 of 262 adding runs did not have that success confirmed on the test set, and the confirmatory seed analysis depended on treatment of shared seeds.Source: §5 · p. 13

6 Conclusion

The study compared gate initialisation with recurrent dropout and zoneout in one-layer LSTM and GRU networks on two synthetic long-dependency tasks. Initialisation produced the largest differences in sensitivity and learning, while the early decay measure added little predictive value beyond early validation loss.

6 Conclusion

  • The comparison used one-layer LSTM and GRU networks on two synthetic long-dependency tasks with fresh or exhaustive training data and one rate per regulariser.Source: §6 · p. 13
  • Gate initialisation produced the largest differences in both input-sensitivity reach and whether and how fast networks learned.Dropout generally slowed learning, while zoneout helped in some settings and not others when initialisation left sensitivity decaying fast.Source: §6 · p. 13
  • Chrono initialisation retained its advantage with a horizon about four times too short or too long.Source: §6 · p. 13
  • After 100 updates, decay added little to predicting which conditions learn once early validation loss was modelled by task, architecture, and initialisation.Within conditions, flatter early decay was associated with learning seeds, but the prespecified association was modest, condition-dependent, and not robust to resampling seeds.Source: §6 · p. 13

Code and data availability

The experiment code, analysis, per-run results, and archived gradient curves are available online.

Code and data availability

  • Experiment code, analysis, and per-run results are available at the project repository, and every run's gradient curves are archived at Zenodo.The networks were trained with rust-lstm 0.10.0.Source: Code and data availability · p. 13

Related papers

Loading 2610.29034v1…