PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Distant Gradients Need Not Be Reliable: reliability-weighted credit assignment for long-horizon autoregressive forecasting

Junhao Zhao, David Michael Simberg, Jacob Kang, Colin Connor Kurniawan, Nan Xu

arXiv:2609.12890v3cs.LGcs.AI

TL;DR

Long-rollout BPTT can amplify distant gradients without making their learning signal reliable, because predictable signal and unpredictable innovation are transported together. Internal-DW reliability-weights residual backward routes while preserving the rollout and horizon losses; it improves forecasting on history-dominated, weak-drive testbeds, with benefits bounded by history availability and sampler fit.

  • Problem

    In long-rollout forecasting, distant BPTT gradients can gain leverage without gaining reliable learning signal because Jacobian products amplify predictable credit and unpredictable innovation together.

    Source: §1 · p. 2, §3.1 · p. 3

  • Method

    Internal-DW applies bounded Wiener reliability weights to residual backward routes while preserving the forward rollout and all horizon losses.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    Internal-DW reduces relative L2 by 5.17%–13.76% versus full BPTT across four history-dominated, weak-drive testbeds, and benefits diminish or reverse when history is limited or the sampler misses driven variation.

    Source: §1 · p. 2, §5.2 · p. 8

  • Takeaways & Limitations

    Long-horizon supervision can be retained without trusting every backward contribution equally, but Internal-DW is not a universal stabilizer.

    Source: §6 · p. 10

    The gains depend on noise-sampler accuracy, and the local route-risk objective does not guarantee a better training trajectory.

    Source: §6 · p. 9

Abstract

from arXiv · show

In autoregressive forecasting, long prediction rollouts provide distant supervision, but backpropagation through time (BPTT) carries gradients from those losses through many autoregressive steps. Repeated Jacobian products can make distant gradients dominate the update while amplifying predictable signal and unpredictable innovation together; a large distant gradient therefore need not carry reliable learning signal. Motivated by this, we introduce Internal Dual-Wiener routing (Internal-DW), a backward-only intervention that preserves the full forward rollout and all step losses while reliability-weighting internal gradient routes. At each residual block, we derive bounded Wiener gains for the identity and nonlinear routes that balance preserving predictable learning signal against suppressing unpredictable variation, and estimate them from route-level gradient statistics and an explicit noise model. In a controlled system with known gradient signal-to-noise ratio (SNR), we show that distant gradients can grow even as their SNR falls, and that Internal-DW reduces error in recovering predictable gradient signals and improves forecasting. On four history-dominated, weak-drive testbeds, Internal-DW reduces forecast error by 5.2%-13.8% relative to full BPTT, outperforms gradient clipping and Jacobian regularization on three testbeds, with similar performance on shear flow, and outperforms validation-selected truncated BPTT (TBPTT) on three. It also extends or preserves the fitted optimal training-horizon range across these four testbeds. Across benchmarks, the current Internal-DW estimator has a clear applicability boundary: its benefit diminishes or reverses when usable history is limited or when the selected sampler fails to represent dominant drive-dependent variation. The results show that retaining long-horizon supervision does not require trusting every backward contribution equally.

1 INTRODUCTION

Long-rollout BPTT can give distant gradients high leverage without making them reliable, because Jacobian products amplify predictable signal and unpredictable innovation together. Internal-DW preserves the forward rollout and horizon losses while weighting backward routes by estimated reliability; its benefit depends on usable history and the noise sampler.

1 INTRODUCTION

  • A distant gradient can dominate the summed update even when much of its magnitude is unreliable innovation, so magnitude reflects backward sensitivity rather than useful credit.The same Jacobian products amplify predictable learning signal and unpredictable innovation.Source: §1 · p. 2
  • Clipping and adaptive optimizers rescale gradients only after forecast-step contributions are combined, while Jacobian regularization changes learned dynamics and shorter rollouts remove distant supervision.The paper motivates a backward-only intervention that preserves the complete rollout, horizon losses, and forward dynamics.Source: §1 · p. 2, Figure 1 · p. 2
  • Internal-DW weights identity and nonlinear residual routes using a bounded Wiener solve that trades distortion of predictable signal against retained unpredictable variation.The noise estimate comes from either centered training residuals or domain-supported structure.Source: §1 · p. 2
  • In a known-SNR controlled system, delayed gradients grow as their SNR falls, while Internal-DW reduces route-gradient risk and achieves the lowest forecast error among tested methods.Across four history-dominated, weak-drive testbeds, it improves forecasting and extends or preserves the useful training-horizon range.Source: §1 · p. 2
  • The current estimator is best suited to credit propagated through state history; separating that credit from newly injected drive remains an important extension.Source: §1 · p. 3

2 RELATED WORK

Prior approaches intervene on rollout distributions, forward dynamics, temporal supervision, or delayed backpropagation. Internal-DW instead estimates the reliability of internal transported gradients before route contributions are aggregated, while leaving the forward rollout and horizon losses unchanged.

2 RELATED WORK

  • Existing approaches alter rollout distributions, constrain forward dynamics, modify temporal supervision, or limit and damp delayed backpropagation.Examples include scheduled sampling, Jacobian penalties, STLW, HERO, COLA, TBPTT, and stop-gradient methods.Source: §2 · p. 3
  • Internal-DW differs by estimating reliability of transported internal VJPs before route contributions are aggregated, without changing the forward rollout, horizon losses, or learned dynamics.Source: §2 · p. 3

3 RELIABLE GRADIENT TRANSPORT IN LONG-HORIZON ROLLOUT

The framework separates delayed-gradient leverage from reliability and derives bounded reliability-weighted residual routing as a minimum-risk trade-off. Its estimator requires noise-model assumptions because open-route gradients alone cannot identify predictable signal separately from innovation.

3 RELIABLE GRADIENT TRANSPORT IN LONG-HORIZON ROLLOUT

  • Theoretical results formalize the leverage–reliability mismatch, derive minimum-risk bounded linear weighting of residual routes, and show that identifying the weights requires an explicit noise model.Source: §3 · p. 3

3.1 LARGE DELAYED GRADIENTS NEED NOT BE RELIABLE

  • The same Jacobian products transport predictable conditional-mean credit and unpredictable innovation, so amplification can increase a contribution’s leverage without making it informative.The proposed backward-only weighting minimizes mean-squared error relative to the predictable delayed contribution before step contributions are aggregated.Source: §3.1 · p. 3

3.2 INTERNAL-DW: OPTIMAL RELIABILITY-WEIGHTED GRADIENT TRANSPORT

  • Internal-DW reweights identity and nonlinear routes in residual blocks while leaving forward values unchanged, choosing bounded gains to preserve predictable credit while suppressing transported innovation.Route moments are estimated from fully open total-moment and noise probes, then substituted into the routing objective.Source: §3.2 · pp. 4–5

3.3 NOISE IDENTIFIABILITY AND SAMPLER ASSUMPTIONS

  • The open-route law does not identify the predictable-signal and noise split; DW-Generic uses centered residuals as a noise proxy, while DW-Prior uses domain-supported predictors or covariance models.DW-Generic can misclassify systematic learnable error as noise, and sampler choice is fixed using validation loss before test evaluation.Source: §3.3 · p. 5

3.4 SCOPE AND BOUNDARIES OF INTERNAL-DW

  • When history supplies little reliable delayed signal, Internal-DW has little credit to preserve; sampler mismatch can instead inflate estimated noise and close routes that carry useful drive-dependent information.More data cannot remove the mismatch component when the sampler model is misspecified.Source: §3.4 · p. 6

4 EXPERIMENTAL SETUP

The experiments use a known-SNR controlled system and application testbeds spanning different predictive regimes. Held-out probes classify those regimes, and methods are compared under shared settings with validation-based selection before test evaluation.

4 EXPERIMENTAL SETUP

  • The study uses one stationary, undriven linear-Gaussian AR process with known SNR and eight application testbeds spanning delayed dynamics, driven control, neural recordings, temporal forecasting, and gridded fields.Source: §4 · p. 6
  • Held-out probes distinguish history-dominated weak-drive, drive-dominated weak-history, weak-drive-and-history, and a driven Mackey–Glass intervention; these groupings describe probe-recovered structure rather than intrinsic dataset properties.The probes predict the clearest Internal-DW benefit in the history-dominated, weak-drive regime.Source: §4 · p. 6, Figure 3 · p. 6
  • Comparisons include full BPTT, clipping, Jacobian regularization, and, on four history-dominated testbeds, TBPTT and a tied static gain, with variants and checkpoints selected by validation loss before testing.Methods otherwise use the same experimental settings within each dataset.Source: §4 · pp. 6–7

5 RESULTS

The results test whether distant gradients remain reliable and whether reliability weighting helps forecasting across regimes and training horizons. Internal-DW improves forecasting in history-dominated, weak-drive testbeds, while its benefit is bounded by the predictive regime and sampler.

5.1 MECHANISM AND ESTIMATOR RESPONSE

  • At grad = 24, accumulated innovation outweighs recovered signal; Internal-DW lowers normalized route risk from 1.00 to 0.52, while shifting the gains raises it to 0.65.An oracle lowers route risk further to 0.35, and Internal-DW has the lowest forecast error among tested methods.Source: §5.1 · p. 7, Figure 4 · p. 7
  • On application data, gradients often grow as held-out utility fluctuates or turns negative, while both route gains decrease as controlled noise increases.ETTm1 and WeatherBench-2 are complementary cases where growing gradients remain mostly useful; native-data gradient SNR and oracle gains are unavailable.Source: §5.1 · pp. 7–8, Figure 5 · p. 8

5.2 FORECASTING EFFECTS ACROSS REGIMES AND HORIZONS

  • At identified regime boundaries, Internal-DW offers no consistent benefit; it can worsen forecasting when the sampler treats predictable drive-dependent variation as noise.NARMA is 7.87% worse, and adding strong observed drive to Mackey–Glass reverses a 5.17% improvement into a 7.71% degradation relative to full BPTT.Source: §5.2 · p. 8
  • 5.17%–13.76% lower relative L2 than full BPTT across four history-dominated, weak-drive testbeds; Internal-DW beats TBPTT on three and Clip and JReg on three.Performance is similar to Clip and JReg on shear flow; Internal-DW also beats Static on ETTm1, ETTm2, and shear flow.Source: §5.2 · p. 8, Figure 6 · p. 9
  • Internal-DW shifts the fitted minimum toward larger training horizons and lowers fitted minimum error across all four testbeds.The useful training-horizon range extends or is preserved; not every horizon pointwise improves.Source: Figure 7 · p. 9, §5.2 · p. 9

6 DISCUSSION AND CONCLUSION

Internal-DW reliability-weights backward routes while retaining the rollout and its horizon losses, but its benefit depends on reliable delayed credit and an appropriate noise sampler. The discussion situates this approach among methods that alter gradient assignment or propagation, and describes regimes in which recurrent sensitivities accumulate.

6 DISCUSSION AND CONCLUSION

  • Internal-DW's gains depend on the selected noise sampler, and independently solving each horizon and layer ignores interactions after route contributions are combined.The local quadratic route-risk objective does not guarantee a better training trajectory.Source: §6 · p. 9
  • Internal-DW retains per-horizon targets, the forward rollout, and all horizon losses while weighting transported identity and nonlinear route VJPs; it is not a universal stabilizer.Benefits are primarily reported in history-dominated, weak-drive regimes, and performance can be neutral or harmful when usable history is weak or the sampler mismatches driven information.Source: §6 · pp. 9–10

A EXTENDED RELATED WORK

  • Related methods alter delayed-return assignment, gradient computation, or backward messages; Internal-DW instead estimates reliability of transported routes within residual merges.Unlike TBPTT or ARTBP, it retains the rollout VJPs and observed per-horizon targets.Source: A EXTENDED RELATED WORK · p. 17

B WHY DELAYED GRADIENTS ACCUMULATE

  • Recurrent sensitivity can plateau under contraction, accumulate under unit-modulus modes, or grow exponentially when an excited unstable mode is present.Time-varying Jacobians, correlated injections, and cancellation can change these rates; the regimes are sufficient conditions, not claims about every nonlinear autoregressive model.Source: B WHY DELAYED GRADIENTS ACCUMULATE · p. 18

C INTERNAL-DW: THEORY AND IMPLEMENTATION

  • The section presents an optimality result and a training-and-calibration procedure that alternates ordinary training with fully open calibration probes updating route moments and gains.Further subsections specify routing placement, probe construction, the constrained solve, and calibration cost.Source: C INTERNAL-DW: THEORY AND IMPLEMENTATION · p. 18

C.1 PROOF OF THEOREM 1

The section frames Internal-DW as constrained route-risk minimization and describes where its gains are applied, how route moments are calibrated, and how noise samplers feed the solve.

C.1 PROOF OF THEOREM 1

  • The constrained Wiener objective has a unique solution; for one route, the unconstrained gain is P/(P + R).For two routes, the constrained solution is the T-metric projection of the unconstrained minimizer onto [0, 1]^2.Source: §C.1 · p. 19

C.2 WHERE THE ROUTE WEIGHTS ARE APPLIED

  • Internal-DW weights identity and nonlinear backward routes at internal residual merges while leaving forward values unchanged.The recurrent-state path uses the nonlinear coefficient; the outer autoregressive update remains fully open.Source: §C.2 · p. 19

C.3 HOW THE ROUTE MOMENTS ARE CALIBRATED

  • Fully open total-moment and noise probes estimate route moments, but transfer from probe loss to task loss is an estimator approximation, not a guarantee.The resulting Wiener objective is a local MMSE criterion under the selected sampler, not a claim of global optimality for nonlinear training.Source: §C.3 · p. 20

C.4 HOW THE TWO ROUTE WEIGHTS ARE SOLVED

  • The two-route constrained solve checks the feasible unconstrained optimum, each edge optimum, and all four corners, then selects minimum quadratic risk.It does not simply clip the unconstrained solution because route correlations can make coordinatewise clipping suboptimal.Source: §C.4 · p. 20

C.5 WHEN CALIBRATION RUNS AND WHAT IT COSTS

  • Calibration minibatches add two VJP traversals; soft gains do not prune the graph, so the implementation adds computation but does not reduce activation memory.The task backward uses previously committed gains, and new moments and gains are committed afterward.Source: §C.5 · p. 20

D HOW THE NOISE SAMPLERS ARE CONSTRUCTED

D.1 GENERIC SAMPLER: RESAMPLING FORECAST RESIDUALS

The section constructs noise trajectories by resampling centered forecast residuals or predicting reproducible structure before treating held-out residuals as noise. The spatial sampler instead matches residual scale and spatial power, and each sampler's distribution enters the shared Internal-DW calibration pipeline.

D.1 GENERIC SAMPLER: RESAMPLING FORECAST RESIDUALS

  • The generic sampler centers stored forecast residuals by horizon, then randomly signs each whole trajectory to create zero-mean noise while preserving joint error patterns.Any predictable error remaining in centered residuals is counted as noise.Source: §D.1 · pp. 21–22

D.2 PRIOR SAMPLERS: PREDICTING REPRODUCIBLE STRUCTURE FIRST

  • Prior samplers cross-fit predictors of reproducible structure, then treat held-out prediction residual trajectories as noise and randomly sign them.Using the same sampled trajectory and sign across horizons preserves empirical coordinate and cross-horizon covariance.Source: §D.2 · p. 22

D.2.1 IEEG: PREDICTING SHARED TEMPORAL FACTORS

  • The iEEG sampler predicts shared temporal factors from past activity and treats the remaining held-out variation as noise.Its identifying assumption is that reproducible activity is concentrated in a small set of cross-electrode factors with predictable temporal evolution.Source: §D.2.1 · p. 23

D.2.2 MOVIE FMRI: PREDICTING SHARED AND SUBJECT-SPECIFIC RESPONSES

  • The movie-fMRI sampler predicts stimulus-shared responses and temporally predictable subject deviations, treating remaining held-out variation as noise.Its diagonal AR model for predictable subject deviation is an additional identifying assumption.Source: §D.2.2 · pp. 23–24

D.3 SPATIAL FIELDS: MATCHING RESIDUAL SPATIAL STRUCTURE

  • The spatial sampler matches residual scale and per-horizon, per-channel spatial power, but not cross-channel or cross-horizon covariance.Its identifying assumption is approximate translation stationarity on the regular grid.Source: §D.3 · pp. 24–25

D.4 HOW SAMPLER OUTPUTS ENTER INTERNAL-DW

  • Every sampler supplies a K-step output-noise trajectory to qnoise; Internal-DW then uses the shared route probe, PSD projection, and constrained Wiener solve.Blocked out-of-fold prediction can reject a sampler assumption when residuals remain predictable, but cannot prove residuals are physically irreducible noise.Source: §D.4 · p. 25

E.1 PROOF OF ROUTE-LAW NON-IDENTIFIABILITY

The appendix establishes that open-route observations cannot identify signal and innovation separately, so the estimated gains depend on sampler assumptions. It then states consistency conditions, analyzes sampler error, and details controlled known-SNR risk and forecasting protocols.

E.1 PROOF OF ROUTE-LAW NON-IDENTIFIABILITY

  • Distinct signal–noise decompositions can produce the same complete open-route distribution but different Wiener gains, so the split is not identifiable from that distribution alone.The appendix notes that additional samples cannot resolve a population-law ambiguity; identification requires added structure or an explicit noise model.Source: §E.1 · p. 25

E.2 WHEN A FIXED SAMPLER RECOVERS THE INTENDED NOISE

  • A fixed sampler’s residual covariance and transported route-noise moment converge under stationarity, ergodicity, cross-fitting, boundedness, and covariance-consistency assumptions.If the predictor class contains the conditional mean, its residual is the conditional innovation; otherwise it also includes predictable structure the class cannot represent.Source: §E.2 · p. 26

E.3 HOW SAMPLER ERROR CHANGES THE ESTIMATED GAINS

  • Sampler mismatch creates systematic moment error that more data cannot remove, while finite probes add estimation error to the gain.The resulting moment error affects local route risk through gain bias and variance, without implying an end-to-end forecasting guarantee.Source: §E.2 · p. 26, §E.3 · p. 27

F.1 SEPARATING CONDITIONAL GRADIENT SIGNAL AND NOISE

  • The controlled study uses an oracle to separate conditional-mean gradient signal from innovation noise while holding forecasting parameters fixed for its initial panels.The simulator supplies the conditional future, allowing future-noise realizations to vary from the same observed state.Source: F CONTROLLED KNOWN-SNR STUDY · p. 27, §F.1 · p. 27

F.2 RISK OF INCLUDING MORE FORECAST STEPS

  • Full BPTT has zero missing-signal bias at the full-window endpoint, but need not minimize total risk because innovation risk remains.The controlled known-SNR analysis reports finite Monte Carlo closure error of at most 0.082 of full BPTT risk across four splits.Source: §F.2 · p. 27
  • The risk decomposition separates missing-signal bias from innovation risk, with the cross term vanishing by conditional centering.Full BPTT is the endpoint with no missing-signal bias, but need not have minimum total risk.Source: §F.2 · p. 27

F.3 LOCAL ROUTE RISK AND FORECASTING PERFORMANCE

  • The forecasting protocol compares separately trained methods using matched seeds, validation-selected checkpoints, and dense autoregressive test rollouts without state resets or ground-truth refreshes.The reported scores average relative L2 error across forecast steps, origins, and trajectories.Source: §F.3 · p. 28

G DATASETS AND TRAINING DETAILS

  • The appendix records dataset and configuration details behind the selected-estimator comparison.The supplied passage points to appendices covering preprocessing, splits, hyperparameters, protocols, and seed-level results.Source: G DATASETS AND TRAINING DETAILS · p. 28

G.1 DATASETS

The appendix describes the processed datasets and matched training procedures, then defines predictor-based probes for history and drive value. In a controlled Mackey–Glass sweep, the probes respond to increasing observed drive while continuing to detect added history value.

G.1 DATASETS

  • Dataset-size counts refer to trajectories, subjects, or sequence segments, and the stated state shape excludes time and external-drive features.The appendix also describes dataset-specific preprocessing, splits, and input construction.Source: §G.1 · pp. 28–30, Table 2 · p. 29

G.2 IMPLEMENTATION DETAILS FOR TRAINING

  • Matched full-BPTT and Internal-DW runs share splits, initialization, forward model, loss, optimizer update, validation rule, and evaluation code; Internal-DW changes the backward operator and adds calibration probes.For iEEG, models are trained separately by participant, so the evaluation is within-participant rather than generalization to unseen participants.Source: §G.2 · p. 31

H.2 INTERPRETING THE REGIME SCORES

  • The probe scores are predictor-dependent diagnostics, not precise measurements of datasets’ intrinsic properties.For autonomous testbeds, the drive-only predictor is the training mean, so ΨD equals zero by construction.Source: §H.1 · p. 32, §H.2 · p. 33

H.1 PROBE CONSTRUCTION AND REGIME SCORES

  • The probes use linear ridge regression and nonlinear nearest-neighbor prediction, with history-augmented features compared against drive-only prediction and a shifted-history null.Their scores indicate detected predictive value under the chosen probes; negative raw scores are interpreted as no detected value, not negative information.Source: §H.1 · pp. 32–33, §H.2 · p. 33

H.3 VALIDATION OF THE PREDICTIVE-REGIME PROBES

  • As drive scale λD rises from .03 to .34, measured drive-only gain ΨD increases from approximately .03 to .82 while ∆ΨH decreases but remains positive.This is the reported direction of response for the predictive-regime probes; Figure 8 plots the probe coordinates across drive settings.Source: Figure 8 · p. 34, §H.3 · p. 34

I CONTROLLED TEST OF STRONG OBSERVED DRIVE

A controlled Mackey–Glass comparison retains delay dynamics while strengthening observed AR(1) drive. Internal-DW improves the autonomous condition but performs worse after drive is strengthened.

I CONTROLLED TEST OF STRONG OBSERVED DRIVE

  • The driven variant adds an observed unit-variance AR(1) forcing process with coefficient .9 and scale .08 while retaining the autonomous delay parameters.The same matched training protocol is used in both conditions.Source: I CONTROLLED TEST OF STRONG OBSERVED… · pp. 34–35
  • The driven variant retains substantial long-horizon history value while also providing strong observed drive, making it a stress test of the estimator’s drive-related boundary.The comparison indicates that substantial history alone does not guarantee improvement.Source: I CONTROLLED TEST OF STRONG OBSERVED… · p. 35
  • Internal-DW improves the autonomous Mackey–Glass condition but is worse after observed drive is strengthened.Table 8 reports mean ± standard deviation over three matched seeds; relative L2 is normalized within each condition, so comparisons are within rows.Source: Table 8 · p. 35, I CONTROLLED TEST OF STRONG OBSERVED… · p. 35

J HOW FORECASTING ERROR IS EVALUATED

Forecasting error is evaluated from autoregressive rollouts at selected test origins, with a single headline relative-L2 score. The accompanying gradient-utility protocol compares per-step training gradients with a held-out full-horizon gradient.

J HOW FORECASTING ERROR IS EVALUATED

  • The headline forecasting score averages relative L2 over 1:Heval after continuous rollouts from up to 64 evenly spaced origins per test unit, weighting units equally.Trained-window, post-window, and terminal summaries are not reported as separate performance criteria.Source: J HOW FORECASTING ERROR IS EVALUATED · p. 35

K HOW PER-HORIZON GRADIENT UTILITY IS MEASURED

  • Per-horizon gradient utility U(k) measures alignment with a disjoint test example’s full-horizon BPTT gradient; U(k) below zero means the infinitesimal training update raises held-out loss.The displayed percentage is the fraction of forecast steps whose median utility is negative.Source: K HOW PER-HORIZON GRADIENT UTILITY IS… · p. 35
  • The dense-horizon relative-L2 table reports mean ± sample standard deviation over three matched seeds, with lower values better.The supplied caption identifies these values as underlying Figure 6.Source: Table 9 · p. 36

L DENSE-HORIZON RELATIVE-L2 RESULTS

Internal-DW has a negative mean paired difference on all four history-dominated, weak-drive testbeds. The paired intervals exclude zero for MG, while all three evaluated seed differences are negative for ETTm1, ETTm2, and shear flow.

L DENSE-HORIZON RELATIVE-L2 RESULTS

  • Paired differences compare Internal-DW with full BPTT across three matched seeds, with iEEG seed-level errors averaging the same 16 participants equally.The reported intervals are pointwise 95% paired Student-t intervals.Source: L DENSE-HORIZON RELATIVE-L2 RESULTS · p. 36
  • −0.05446 (MG), −0.06357 (ETTm1), −0.11948 (ETTm2), and −0.01863 (shear flow) are the mean paired differences, all favoring Internal-DW.The pointwise 95% interval lies entirely below zero for MG; the other three intervals include zero, although all three seed-level differences are negative for each.Source: L DENSE-HORIZON RELATIVE-L2 RESULTS · p. 36
  • The mean paired differences are +0.06843 for NARMA-5, +0.06212 for movie fMRI, and +0.00057 for WeatherBench-2.The table defines negative differences as favoring Internal-DW; lower relative L2 is better, and values are intended for within-dataset rather than across-dataset comparison.Source: L DENSE-HORIZON RELATIVE-L2 RESULTS · p. 36

M END-TO-END TRAINING TIME

Internal-DW increases end-to-end training time, while its fitted optimal training horizon is at least as long as Full BPTT’s in every dataset–seed comparison.

M END-TO-END TRAINING TIME

  • 20.3–41.7% is the mean paired increase in epoch time across the eight testbeds for Internal-DW over Full BPTT.The end-to-end measurements include online covariance probes, routewise gain solves, and gain application; the forward model is unchanged and there is no inference-time computation.Source: M END-TO-END TRAINING TIME · p. 37, Table 10 · p. 37

N PER-SEED FITTED CURVES

  • The Internal-DW fitted optimum is at or to the right of Full BPTT’s in all 12 dataset–seed comparisons, strictly in 11.ETTm1 seed 0 has the same boundary optimum for both methods, so the shift is not produced by averaging across seeds.Source: N PER-SEED FITTED CURVES · p. 37, Figure 9 · p. 38

Related papers

Loading 2609.12890v3…