PaperlayerView exact revision on arXiv

Source-linked AI summary

Context-dependent time-series prediction via HyperReservoirs

Kohei Tsuchiyama, Takatomo Mihana, Ryoichi Horisaki, André Röhm

arXiv:2609.34847v1cs.LGnlin.AOnlin.CD

TL;DR

When similar recurrent representations must support different context-dependent predictions, it matters where context enters the computation. The paper proposes a HyperReservoir that uses a context pathway to modulate a shared reservoir’s readout, and it achieves the lowest mean test cNMSE in all three tasks. The results suggest this approach is especially useful when recurrent features can remain shared while predictive mappings change.

  • Problem

    Different contexts can require different responses from similar or shared recurrent representations, including when attractor geometry is the same.

    Source: §I · pp. 1–2

  • Method

    The HyperReservoir uses a common main reservoir for observed dynamics and a smaller context reservoir to modulate the readout, with fixed recurrent subsystems and linear-regression training.

    Source: §I · p. 2

  • Results

    The augmented HyperReservoir achieved the lowest mean test cNMSE in all three tasks, with a modest advantage on the first and larger advantages on the latter two.

    Source: §VI · p. 10

  • Takeaways & Limitations

    Temporal-scale multifunctionality can involve different computations on a common state-space structure, not only multiple attractors.

    Source: §VI · p. 10

    Evaluation supplies observed states at every step, uses explicit context, and tests a finite discrete set of contexts; autonomous generation, unknown or switching context, and unseen-regime generalization remain unaddressed.

    Source: §VI · p. 11

Abstract

from arXiv · show

Time series prediction is a common application of reservoir computing. When the training and testing time series data contains multiple dynamical regimes, because an underlying parameter is changing, or the data in fact consists of multiple distinct systems, simple application of the reservoir computing principle produces high prediction errors. Here, we propose a HyperReservoir as an extended model of reservoir computing especially designed for such cases. The HyperReservoir combines a main reservoir with a smaller context reservoir, where the latter modulates the output weights of the former. This structure resembles the hypernetworks from deep neural network literature. However, in contrast, HyperReservoirs retain the simple training via linear regression of standard reservoir computing. We compare the proposed architecture with a conventional ESN, in which context acts at the input, and a full-matrix Conceptor, in which context modulates the reservoir state space. We evaluate all three models on time-series prediction tasks based on Lorenz and R\"ossler systems, including for varying bifurcation parameters and time sampling scales. We find that the HyperReservoir achieves the lowest mean test error in all three tasks, and particularly outperforms conceptors on data that is sampled from the same attractor but at different time scales.

I. INTRODUCTION

Reservoir computing reuses fixed recurrent dynamics, but adapting a shared representation to context-dependent outputs remains a challenge. The paper compares input-, state-, and readout-level contextual adaptation and reports lower mean error for the HyperReservoir than for a context-input ESN, especially in related and shared-attractor regimes.

I. INTRODUCTION

  • Reservoir computing trains a simple output transformation over fixed high-dimensional recurrent dynamics, but that fixed substrate limits adaptation to changing conditions.The approach is attractive when the underlying dynamics are difficult or costly to modify.Source: §I · p. 1
  • When contexts share attractor geometry, similar current states can require different future predictions, making context-dependent mappings important even when state-space regions overlap.Temporal scaling changes the rate of evolution without changing the underlying continuous-time orbit geometry.Source: §I · pp. 1–2
  • The proposed HyperReservoir retains a common main reservoir and uses a smaller context reservoir to modulate its readout.Its recurrent subsystems remain fixed, and the complete readout is fitted by linear regression.Source: §I · p. 2

II. CONTEXTUAL ADAPTATION IN RESERVOIR COMPUTING

  • The compared architectures place context at different stages: the ESN supplies it as input, Conceptors filter reservoir states, and the HyperReservoir changes the readout.In the context-input ESN, the recurrent operator and readout remain shared; in the full-matrix Conceptor, a context-specific operator filters the provisional state.Source: §I · p. 2, A. Context-input echo state network · p. 2, Figure 1 · p. 3

I. INTRODUCTION

  • The HyperReservoir achieves lower mean prediction error than the context-input ESN across the three settings, with larger differences for related and shared-attractor regimes.The results support context-dependent decoding when recurrent features can remain substantially shared while the required predictive mapping changes with context.Source: §I · p. 2

C. Proposed HyperReservoir for context-dependent readout modulation

The HyperReservoir keeps the main recurrent dynamics fixed and lets supplied context alter how their shared representation is decoded. A bilinear readout couples the observation and context pathways, yielding context-dependent corrections to a shared baseline mapping.

C. Proposed HyperReservoir for context-dependent readout modulation

  • The HyperReservoir processes observations and context in separate fixed recurrent subsystems, then couples their representations through a bilinear readout.The main reservoir represents observed dynamics; the smaller context reservoir represents the supplied context.Source: C. Proposed HyperReservoir for… · p. 4
  • The effective readout is an affine family: a shared baseline mapping is combined with a context-dependent correction.This structure allows shared predictive features to remain in the baseline while the context-dependent part of the mapping changes.Source: C. Proposed HyperReservoir for… · p. 4
  • The context-dependent parameterization resembles the Hypernetwork viewpoint, but confines context dependence to the readout and fits coefficients in one ridge-regression problem.Neither recurrent subsystem is trained by backpropagation.Source: C. Proposed HyperReservoir for… · p. 4
  • Unlike independently fitted readouts for each regime, the construction couples effective readouts through shared and context-dependent terms.The bilinear features are cross-interactions between separately driven context and main-reservoir states, rather than a generic nonlinear feature expansion.Source: C. Proposed HyperReservoir for… · pp. 4–5

III. NUMERICAL EXPERIMENTS

The experiments compare the three architectures on future-state prediction settings using Lorenz and Rössler dynamics. The first task pairs systems with clearly distinct attractor geometries and supplies system identity as context.

III. NUMERICAL EXPERIMENTS

  • The experiments span geometrically distinct attractors, related Rössler regimes, and a shared attractor traversed at different temporal scales.All three architectures receive the same trajectories, data partitions, prediction targets, and evaluation metric.Source: §III · p. 5
  • In every experiment, context is explicitly supplied, fixed within each trial, and available at every time step.The task settings range from distinct systems to regimes sharing attractor geometry.Source: §III · p. 5

A. Geometrically distinct Lorenz and R¨ossler dynamics

B. Related R¨ossler regimes

The Rössler experiments include known related regimes and a shared attractor traversed at different temporal scales. The protocol uses observed-state one-step prediction and selects reservoir allocations and Conceptor apertures using validation data.

B. Related R¨ossler regimes

  • The related-regime task evaluates two Rössler regimes included in training, validation, and test, rather than interpolation to an unseen cR value.The regimes belong to the same Rössler family.Source: B. Related R¨ossler regimes · p. 5

C. Same attractor under different temporal scales

  • The temporal-scale task uses the same Rössler system in both contexts; scaling its full vector field preserves orbit geometry but changes traversal rate and future predictions.The same current state can therefore require different future-state predictions over a fixed interval.Source: C. Same attractor under different… · p. 5

D. Prediction protocol

  • Evaluation measures one-step prediction along observed trajectories, with observed dynamical states supplied at every step rather than feeding predictions back recursively.Trajectories are integrated with fourth-order Runge–Kutta and sampled every 0.05 continuous-time units.Source: D. Prediction protocol · pp. 5–6

E. Recurrent-state allocation and parameter selection

  • The selected (main, context) dimensions were (110, 10) for Lorenz–Rössler and (115, 5) for each Rössler task; Conceptor apertures were 4, 1, and 1, respectively.Selections used validation data rather than test data.Source: E. Recurrent-state allocation and… · p. 7
  • Matching recurrent-state counts does not equalize fitted-coefficient counts, storage requirements, or arithmetic cost.These differences are considered separately.Source: E. Recurrent-state allocation and… · p. 7

F. Evaluation metric

The section defines cNMSE for masked post-washout prediction and reports that HyperReservoir consistently outperforms the two alternatives, with larger advantages on related or shared-attractor settings.

F. Evaluation metric

  • cNMSE quantifies component-normalized prediction error over valid post-washout targets, and lower values indicate more accurate prediction.The denominator uses ε = 10^-4 to prevent numerical instability when target energy is very small.Source: F. Evaluation metric · p. 7

G. Results

  • Below 4.5×10^-2 mean cNMSE, all three models perform on the Lorenz–Rössler task, while HyperReservoir has the lowest error in each of three reservoir realizations.Source: G. Results · p. 7
  • On the Rössler regimes with cR ∈ {3.5, 5.7}, HyperReservoir has the lowest error, while Conceptor error is approximately one order of magnitude above the context-input ESN.Source: G. Results · p. 7
  • On the same Rössler attractor at different temporal scales, HyperReservoir has the lowest test cNMSE and a larger separation from both reference models than on the Lorenz–Rössler task.Source: G. Results · p. 7
  • Across all three tasks, HyperReservoir has a consistent advantage over the context-input ESN and Conceptor, with a weaker advantage when target dynamics are geometrically distinct.The reported pattern is consistent with context-dependent decoding being especially useful when the main reservoir representation can remain shared while predictive mappings change.Source: G. Results · p. 7

IV. CONTEXTUAL SEPARATION ON A SHARED ATTRACTOR

This section probes shared-attractor prediction through local errors, context replacement, and Conceptors' state-space alignment; HyperReservoir's low errors coincide with strong sensitivity to supplied context.

IV. CONTEXTUAL SEPARATION ON A SHARED ATTRACTOR

A. Finite-time prediction on a shared attractor

  • For representative slow and fast trajectories, HyperReservoir has smaller local errors than the context-input ESN and full-matrix Conceptor, particularly in the fast regime.The comparison uses the same representative reservoir realization and evaluation window across models; the panels use different vertical scales.Source: A. Finite-time prediction on a shared… · p. 8

B. Correct- and wrong-context prediction

  • HyperReservoir has the smallest correct-to-wrong context error ratio, while the ESN response is intermediate and the Conceptor ratio remains closer to unity.Its prediction error therefore increases most strongly when supplied context is replaced, accompanying its low prediction error in this setting.Source: B. Correct- and wrong-context prediction · p. 8

C. Similarity of the slow and fast Conceptors

  • Slow and fast Conceptors remain substantially aligned across reservoir realizations, indicating that their temporal distinction is only weakly expressed in the state-space representation used here.They are not identical; the analysis compares Conceptors learned for the two regimes using a validation-selected aperture.Source: C. Similarity of the slow and fast… · pp. 8–9

V. STRUCTURE OF THE CONTEXT-DEPENDENT READOUT

The augmented HyperReservoir has the lowest mean test cNMSE across the three tasks, and its combined additive and bilinear readout outperforms either feature construction alone. The dimension sweep shows that more context states or fitted coefficients do not guarantee better prediction.

V. STRUCTURE OF THE CONTEXT-DEPENDENT READOUT

  • The Augmented readout has the lowest mean test cNMSE in all three settings, outperforming Concat and Strict, especially for related Rössler.Concat uses additive access to both reservoir states; Strict uses only the bilinear interaction, while Augmented combines both.Source: §V · p. 10, Figure 5 · p. 11
  • With N + M fixed, increasing M reallocates states from the main reservoir and can reduce accuracy, even as fitted coefficients increase monotonically.Validation selects relatively small context reservoirs; prediction accuracy does not improve monotonically with context dimension or readout size.Source: §V · p. 10

VI. CONCLUSION

  • The augmented construction combines a shared baseline mapping with a context-dependent correction, retaining linear-regression training for contextual flexibility.The main reservoir represents observed dynamics, and the context pathway modulates how that representation is decoded.Source: §VI · pp. 10, 12
  • The augmented HyperReservoir achieves the lowest mean test cNMSE in all three tasks against the context-input ESN and full-matrix Conceptor.Its advantage is modest for geometrically distinct systems and larger for related and shared-attractor regimes.Source: §VI · p. 10
  • In the temporal-scale task, slow and fast regimes share continuous-time orbit geometry but have different finite-time flow maps, so multifunctionality need not require multiple attractors.This is complementary to multistability, where a trained system supports multiple coexisting attractors.Source: §VI · p. 10
  • The conclusions concern short-horizon prediction with observed states and explicit context supplied; generalization to unseen regimes remains untested.Inference of unknown or switching context is also not addressed.Source: §VI · p. 11

AUTHOR DECLARATIONS

The declarations report no conflicts, and the supplemental material records shared numerical settings, trajectory generation, and reservoir initialization procedures.

AUTHOR DECLARATIONS

  • The authors report no conflicts of interest and list Kohei Tsuchiyama as leading conceptualization, methodology, analysis, investigation, visualization, and the original draft.Takatomo Mihana, Ryoichi Horisaki, and André Röh m are listed in supporting roles, with review and editing shared by the authors.Source: AUTHOR DECLARATIONS · p. 12

S1. NUMERICAL SETTINGS AND DATA GENERATION

S1.1. Trajectory generation

  • Trajectories use fourth-order Runge–Kutta integration at 0.005, are sampled every 0.05 time units, and contain 1000 recorded samples.No observation or process noise is added, and state components are trimmed to [−80, 80] during generation.Source: S1.1. Trajectory generation · p. 14

S1.2. Reservoir initialization

  • Reservoir recurrent matrices are initialized with Gaussian entries, then rescaled to spectral radius 0.9; input weights use a Gaussian scale based on σin and the available channels.Bias terms in both reservoir-state updates are zero.Source: S1.2. Reservoir initialization · p. 14

S1.3. Context signals

The supplemental methods specify how contexts are supplied, how data are normalized, and how the readouts and Conceptors are fitted.

S1.3. Context signals

  • Task context identifies the active system or Rössler parameter regime, and HyperReservoir context channels are supplied only to the context reservoir.For the Lorenz–Rössler task, context is a two-dimensional one-hot vector identifying the active system.Source: S1.3. Context signals · p. 14

S1.4. Normalization

  • Normalization statistics are estimated from training data and reused unchanged for validation and test; Lorenz and Rössler trajectories share combined statistics rather than system-specific scaling.The shared statistics avoid introducing system identity through context-dependent preprocessing.Source: S1.4. Normalization · p. 14

S1.5. Readout fitting

  • Output coefficients are fitted by ridge regression after washout, with an unregularized bias and a fixed ridge coefficient λ = 10−4.The ridge coefficient is not selected from validation data.Source: S1.5. Readout fitting · p. 15

S1.6. Conceptor estimation

  • Full-matrix Conceptors are estimated from post-washout training states for each context, and neither validation nor test states enter their estimation.The readout is trained from the Conceptor-filtered training states.Source: S1.6. Conceptor estimation · p. 15

S1.7. Validation-based architecture selection

Architecture and hyperparameter selection uses validation cNMSE, while test performance is evaluated after selection and summarized across reservoir realizations.

S1.7. Validation-based architecture selection

  • Numerical architecture and hyperparameter choices use validation cNMSE, not test performance, and a single task-level configuration is selected rather than one per test-performing seed.The same selected Conceptor aperture is used across reservoir realizations and related analyses for that task.Source: S1.7. Validation-based architecture… · p. 16

S1.8. Evaluation-mask details

  • Evaluation excludes the first 20 samples for washout and the final H samples without valid future-state targets; reported mean and sample standard deviation are descriptive summaries across only three realizations.Individual seed values are shown because the number of realizations is small.Source: S1.8. Evaluation-mask details · p. 16

S2. MAIN-COMPARISON RESULTS

  • Table S3 reports seedwise test cNMSE after validation-based architecture selection.Its final column summarizes the three reservoir realizations with an arithmetic mean and sample standard deviation.Source: S2. MAIN-COMPARISON RESULTS · p. 16

S3. CONTEXT-RESERVOIR DIMENSION SWEEP

The supplemental analyses compare context-reservoir sizes and readout variants, then examine same-attractor predictions and counterfactual context use.

S3. CONTEXT-RESERVOIR DIMENSION SWEEP

  • Under the fixed recurrent-state budget, increasing context dimension does not monotonically improve test accuracy; validation selects M = 10 for Lorenz–Rössler and M = 5 for both Rössler tasks.Larger M also reduces the main-reservoir dimension and changes bilinear-readout size; Table S4 summarizes test cNMSE across the dimension sweep.Source: S3. CONTEXT-RESERVOIR DIMENSION SWEEP · p. 16, Table S4 · p. 17

S5. DETAILED SAME-ATTRACTOR ANALYSES

  • In same-attractor plots, raw predictions overlap closely, so residuals expose local model differences; independent vertical scaling affects visualization only, not the full-test cNMSE comparisons.Slow and fast panels use the same 80-sample display window.Source: S5. DETAILED SAME-ATTRACTOR ANALYSES · p. 17

S4. READOUT-COMPARISON RESULTS

  • The augmented construction has the lowest mean cNMSE in all three settings, with the readout variants compared using identical reservoir realizations, data partitions, and fitting conventions.Table S5 reports mean test cNMSE and sample standard deviation across three random reservoir realizations.Source: S4. READOUT-COMPARISON RESULTS · p. 17, Table S5 · p. 18

S5.1. Wrong-context intervention

  • Replacing supplied context while holding the observed trajectory fixed increases errors by several orders for the ESN and HyperReservoir, most clearly for the HyperReservoir, but produces a smaller separation for the Conceptor.The intervention recomputes each model's forward pass under the counterfactual context.Source: S5.1. Wrong-context intervention · p. 18

S5.2. Full-matrix Conceptor comparison

The slow and fast Conceptor spectra largely overlap, while the section also reports readout and storage counts and analyzes how context dimension changes model size and computational costs.

S6. FITTED READOUT DIMENSIONS AND STORED CONTEXTUAL QUANTITIES

  • 363 fitted coefficients serve the conventional ESN and shared Conceptor readout (F = 120); the two-context Conceptor stores 28,800 matrix entries, and matched state budgets do not equalize these quantities.Source: S6. FITTED READOUT DIMENSIONS AND… · p. 18

S5.2. Full-matrix Conceptor comparison

S6.1. Dependence on context dimension

S7. SOFTWARE ENVIRONMENT

  • Final experiments ran in Python 3.10.14 and PyTorch 2.7.1 on macOS 14.4 (arm64), using an Apple M3 with 8 CPU cores and 16 GB memory.Source: S7. SOFTWARE ENVIRONMENT · p. 20
Loading 2609.34847v1…