PaperlayerView exact revision on arXiv

Source-linked AI summary

Bifurcations from Weight Space: JEPAWG on Parametric Dynamical Systems

Tobias Göbel, Miranda C. N. Cheng

arXiv:2610.30847v1cs.LGcs.AInlin.CD

TL;DR

Weight-space methods lack ground truth for what populations of trained models should encode, limiting systematic validation. This paper builds four parameter-indexed dynamical-system model zoos and analyses them with JEPAWG, finding competitive held-out forecasting and a weight-response signal for local and global bifurcations.

  • Problem

    Weight-space methods lack ground truth for what populations of trained models should encode, making systematic validation difficult.

    Source: §1 · pp. 1–2

  • Method

    The paper builds four parameter-indexed model zoos and uses JEPAWG to decode weights at unseen parameters and probe their parameter derivative.

    Source: §1 · p. 2

  • Results

    JEPAWG is competitive with an end-to-end hypernetwork on held-out interpolation parameters, while its weight response detects Hopf and global bifurcations and resolves a period-doubling cascade.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The decoded-weight response can reveal dynamical information, including global bifurcations, that is unavailable from Jacobian computation.

    Source: §1 · p. 2

    Detection is limited to transitions reflected in training trajectories, and transitions closer than the effective parameter-grid resolution cannot be distinguished.

    Source: §4.3 · p. 7

Abstract

from arXiv · show

Trained network weights are increasingly treated as data, but there is rarely ground truth about what a population of models should encode, which hampers systematic validation of weight-space methods. This makes parametric dynamical systems a natural test-bed: the dynamics change qualitatively at submanifolds of parameter space that are, in well-studied families, known analytically or numerically. Fine-tuning a forecasting network at each point of a parameter grid yields a model zoo indexed by a physical parameter, against which weight-space diagnostics can be checked. We build such model zoos for four dynamical systems and analyse them with JEPAWG, a recently proposed joint-embedding hypernetwork built around a learned parameter--weight representation. It was first introduced for neural samplers for lattice field theories, where its latent geometry located the phase transition and estimated the 2D Ising critical exponent. Applied to dynamical systems, JEPAWG is competitive on forecasting with a hypernetwork trained directly for that task, despite being supervised on the parameter--weight correspondence rather than on forecasts. Beyond forecasting, the norm of the parameter-derivative of the decoded weights locates Hopf points and curves, tracks a period-doubling cascade closely enough to give a qualitative estimate of the Feigenbaum constant, and detects a global bifurcation, which has no eigenvalue signature and is therefore invisible to linearisation. As such, JEPAWG provides interpretable insight into the dynamics that is not available from other hypernetworks. Together, these results indicate that JEPAWG is a workable instrument for reading physical structure off a population of trained networks.

1 Introduction

The paper uses parametrised model zoos with known bifurcation structure to test whether JEPAWG can both generate useful forecasters and reveal physical structure from trained weights. JEPAWG is competitive on held-out forecasting and its decoded-weight response detects local and global bifurcations.

Contributions

  • JEPAWG is competitive with an end-to-end hypernetwork on held-out interpolation parameters despite being trained on parameter–weight correspondence rather than forecasting loss.The weight response also localises Hopf points and curves, resolves a period-doubling cascade, and registers a global bifurcation.Source: §1 · p. 2
  • Four dynamical-system model zoos provide known ground truth for validating weight-space diagnostics, including Hopf loci and period-doubling cascades.The systems are the Brusselator, Lorenz system, Rössler oscillator, and Hénon map; the cascades are expected to approach the Feigenbaum constant δ ≈4.6692.Source: §1 · p. 2
  • For Lorenz at σ = 10 and β = 8/3, the response peaks at ρ = 24.51 ± 0.23, inside the bistable window before the subcritical Hopf point ρH ≈24.74.It also tracks the Lorenz Hopf locus as both parameters vary, locates the Brusselator supercritical Hopf curve, and resolves the first five Hénon period doublings.Source: §1 · p. 2
  • The same response detects the Lorenz Bykov T-point, a global bifurcation at which every fixed point retains its stability.Unlike local bifurcations, global bifurcations have no eigenvalue signature, so this detection reads information unavailable to a Jacobian computation.Source: §1 · p. 2
  • The decoded map suppresses per-run training noise and retains accurate forecasts longer than fine-tuned weights on all four systems.The JEPA-style latent is organised by dynamical regime, has intrinsic dimension close to dim p, and is insensitive to parameters absent from the dynamics.Source: §1 · p. 2

2 Related Work

Related work studies model populations, hypernetworks, and learned detection of transitions. This paper distinguishes its approach by using a physically indexed population of trained weights as a diagnostic for known bifurcation structure.

Trained weights as data

  • Weight-space research includes curated model zoos, direct structure-learning with permutation-aware encoders and neural functionals [6] [7], and unstructured public repositories [8], targeting property prediction across heterogeneous populations.This work instead studies a single population indexed by a physical parameter, where known bifurcation structure provides ground truth for weight-space diagnostics.Source: §2 · p. 2

Hypernetworks for dynamical systems

  • PHLieNet [9] trains parameter-to-weight maps through forecasting loss, while CoDA [10] and TiDHy condition on variables inferred from trajectories rather than directly interpretable physical parameters.In this work the parameter-to-weight map also serves as a diagnostic.Source: §2 · p. 3

Transitions read off learned models

  • Earlier weight-based transition studies include networks whose weights organise around the Ising critical temperature, adiabatically fine-tuned neural quantum states [15], and JEPAWG for lattice field theory [4].The present work transfers JEPAWG to dynamical systems with known bifurcation structure.Source: §2 · p. 3
  • Prior data-driven bifurcation methods target single equilibria, whereas differentiating weights of a model of the full dynamics can also address global bifurcations.The approach requires no order parameter, linearisation, or governing equations.Source: §2 · p. 3

3 Background

The background defines bifurcations as parameter-dependent qualitative changes in dynamics and motivates reading them from rapid changes in trained weights. It then contrasts forecasting hypernetworks with JEPAWG, which learns a parameter–weight representation and decodes weights at unseen parameters.

3 Background

  • A parametric dynamical system has state x and control parameters p; continuous-time dynamics can be integrated into trajectories, while discrete maps advance through a separate state-update rule.Attractors include fixed points, limit cycles, and strange attractors; bifurcations reorganise dynamics as parameters vary.Source: §3 · p. 3
  • Local Hopf and period-doubling bifurcations appear in a fixed point's or orbit's linearisation, but global bifurcations have no such local eigenvalue criterion.Global changes involve saddle invariant manifolds and can occur while every fixed point retains its stability.Source: §3 · p. 4
  • The proposed probe trains a forecasting network at each parameter and treats rapid weight change as a signal of dynamical reorganisation.The rationale is that weights are fitted to trajectories on the attractor, so a changed attractor can drive a large parameter sensitivity.Source: §3 · p. 4

3.1 Forecasting Hypernetworks

  • PHLieNet maps parameters through learned interpolated embeddings to forecaster weights and is trained end-to-end using forecasting loss.Its weight-space geometry is unconstrained by that loss, and its anchor-based interpolation saturates outside the training range.Source: §3.1 · p. 5
  • JEPAWG treats per-parameter trained weights and parameters as two views, predicts the weight embedding from the parameter embedding, then separately trains a decoder to produce weights at new parameters.The parameter encoder, predictor, and decoder compose the inference path; the resulting latent geometry can be probed for changes such as bifurcations.Source: §3.1 · p. 5

4 Methods

The methods build a parameter-indexed zoo by fine-tuning a shared forecasting model, fit JEPAWG to the resulting weights, and evaluate both forecasts and weight sensitivity. The sensitivity probe is limited by what the training trajectories and parameter grid capture.

4 Methods

  • The pipeline builds model zoos, fits JEPAWG to the fine-tuned weights, and evaluates decoded weights both as forecasters and as a probe through their parameter derivative.It compares forecasts at unseen parameters and checks response peaks against known bifurcations.Source: §4 · p. 5

4.1 Data Generation

  • A shared conditional forecaster initializes each per-parameter fine-tune, reducing the risk that symmetry-related weight solutions obscure dynamical differences.The common initialization acts as an approximate gauge fixing because independently trained neighboring networks could land in different symmetry copies.Source: §4.1 · pp. 5–6
  • At each sampled parameter, trajectories from random initial conditions train a fine-tuned forecaster initialized from the conditional base, producing the zoo of target weights.Training uses per-system trajectory-generation choices and standardises states using pooled training trajectories.Source: §4.1 · p. 6

4.2 JEPAWG Training

  • JEPAWG is fitted to parameter–weight pairs using MLP encoders, a latent predictor, and a decoder trained in two stages.The latent dimension is 16, deliberately larger than the parameter manifold dimension, and five independent seeds are trained per configuration.Source: §4.2 · p. 6

4.3 Analysis

  • Evaluation decodes weights at unseen parameters, rolls forecasts forward, and measures normalized error and time-to-threshold against test trajectories.Forecasting is evaluated both inside and outside the training range; results are averaged across test parameters and reported over five seeds.Source: §4.3 · pp. 6–7
  • The probe measures how quickly decoded weights change with parameters, expecting peaks near bifurcations because attractor reorganisation changes the trajectory distributions that constrain trained weights.Decoded-weight derivatives suppress per-parameter training noise, while detection depends on the training trajectories and grid resolution; transitions closer than the effective parameter spacing cannot be distinguished.Source: §4.3 · p. 7

5 Experiments

Experiments use four parametric dynamical systems with known bifurcation structure to test JEPAWG as both an unseen-parameter forecaster and a probe of regime changes.

5 Experiments

  • Four families with independently known bifurcation structure form the benchmark; a conditional forecaster is fine-tuned across parameter values, then decoded weights are evaluated and probed through G(p).The four systems are Lorenz, Rössler, Brusselator, and Hénon.Source: §5 · pp. 7–8

5.1 Forecasting

  • On interpolation, JEPAWG ties PHLieNet on three systems, is within two time units on Brusselator, and beats AE and VAE on all four; extrapolation performance depends on regime.For moving cycles, JEPAWG leads on both measures; for Lorenz's stable fixed point, PHLieNet preserves the fixed point more reliably.Source: §5.1 · pp. 8–9, Table 2 · p. 10

5.2.1 Lorenz

  • Lorenz response peaks near the subcritical Hopf (24.51 ± 0.23 versus ρH ≈ 24.74), tracks its curve for σ < 13, and rings the Bykov T-point despite unchanged fixed-point stability.The T-point response captures a global bifurcation that equilibrium linearisation does not register.Source: §5.2.1 · pp. 10–11

5.2.2 Brusselator

  • Brusselator response traces the supercritical Hopf curve, while its latent sheet separates regimes across an internal boundary stretched near the curve.The curve is b = 1 + a2.Source: §5.2.2 · p. 11, Figure 4 · p. 12

5.2.3 Hénon

  • For Hénon, G(u) peaks at each of the first five period doublings within |∆u| ≤ 0.04, and estimated ratios qualitatively recover Feigenbaum scaling.The estimated ratios (4.97, 4.15, 4.49) are comparable to reference ratios (4.81, 4.49, 4.65).Source: §5.2.3 · pp. 12–13

6 Conclusion

JEPAWG serves both as a forecaster at unseen parameters and as a probe of the dynamics encoded by trained model zoos. Its bifurcation results are promising for guiding simulation in less-understood systems, within the limits of the training data and parameter range.

6 Conclusion

  • Across four systems, JEPAWG is competitive with an end-to-end baseline on held-out interpolation, while extrapolation performance depends on the dynamics.It gives better forecasts where Rössler limit cycles continue changing, whereas PHLieNet preserves fixed-point dynamics more reliably; both methods lose forecasts within a few Lyapunov times in chaotic regimes.Source: §6 · p. 13
  • The decoded-weight derivative recovers Lorenz and Brusselator Hopf structure, detects the Lorenz Bykov T-point, and localises the first five Hénon period doublings within |∆u| ≤0.04.The period-doubling locations yield a qualitative measurement of the Feigenbaum constant; the Bykov T-point is invisible to linearisation.Source: §6 · p. 13
  • JEPAWG’s decoded map can guide where to concentrate simulation or further training in systems whose dynamics are not fully understood.This prospect follows its validation against known bifurcation structure; the authors suggest it could avoid sweeping parameter ranges at uniform cost.Source: §6 · p. 13
  • The probe detects only transitions represented in training trajectories, and its resolution and peak locations depend on parameter spacing, map smoothness, and parameter coordinates.Outside the training range, decoded weights are unconstrained; in chaotic regimes, forecast errors are amplified and all tested methods lose forecasts within a few Lyapunov times.Source: §6 · p. 13

A Architectural Details

  • The forecasting target is a causal temporal CNN, while the embedding baselines use 16-dimensional latents and post-hoc parameter-to-latent regressors.The target network is TCNN-CD; the baselines are PCA, AE, and VAE, paired with an identical MLP regressor because they lack a parameter-to-latent map.Source: A Architectural Details · p. 16

B Intrinsic Dimension and Ablation

The ablations show that JEPAWG’s latent and decoded weights preserve useful structure in the trained model zoo: they track parameter dimensions, support forecasting, and ignore parameters unrelated to the dynamics. Comparisons with alternative embeddings and an extrapolation example qualify where these benefits appear.

B Intrinsic Dimension and Ablation

  • JEPAWG’s latent dimension is closest to the control dimension in 15 of 16 estimates, while its estimates on the two-parameter grids level off at 2.3 for Brusselator and 2.2 for Hénon.The raw weight grids have substantially higher estimated dimensions; the authors attribute the closer alignment to regularisation from the covariance-free VICReg objective.Source: B Intrinsic Dimension and Ablation · p. 16, Table 4 · p. 17
  • Decoded weights outperform raw weights in TtT on all four systems; the decoder has the lowest one-step error on three continuous systems and recovers the Rössler horizon lost by per-parameter fine-tunes.On Hénon, fine-tuning gives no improvement over the base network.Source: B Intrinsic Dimension and Ablation · p. 17, Table 5 · p. 17
  • A direct parameter-to-weight regressor produces a noisier Lorenz response than JEPAWG, with peaks elsewhere that make the Hopf point unreliable to locate.A peak remains visible at the Hopf point, but comparable or larger peaks appear elsewhere.Source: B Intrinsic Dimension and Ablation · p. 18, Figure 6 · p. 18
  • With frozen encoders and predictor, the Brusselator control achieves comparable reconstruction error and response correlation, but its random-feature geometry cannot distinguish parameters by their dynamical influence.Its response has Pearson r = 0.96 with JEPAWG’s, while JEPAWG’s learned latent can drop parameters that do not affect the dynamics.Source: B Intrinsic Dimension and Ablation · p. 18, Figure 7 · p. 18
  • Appending dynamics-irrelevant random parameters leaves JEPAWG’s Lorenz Hopf peak and forecast horizon essentially unchanged, while the random-feature control’s peak and horizon degrade as their number grows.At k = 10, JEPAWG’s interpolation horizon is 20.1 ± 0.2 time units versus 20.2 ± 0.2 at k = 0; the control drops from 20.2 ± 0.2 to 18.5 ± 0.5.Source: B Intrinsic Dimension and Ablation · p. 18, Figure 8 · p. 19

C Extrapolation per Regime

  • At the furthest extrapolation parameters, JEPAWG-generated trajectories follow the true cycle more closely in limit-cycle regimes, whereas PHLieNet trajectories tend to diverge.The figure compares predicted and true trajectories at the furthest extrapolation parameter in each regime.Source: Figure 9 · p. 19, C Extrapolation per Regime · p. 19

D Additional Results per Dynamical System

At held-out Lorenz parameters within the training range, JEPAWG forecasts follow the true trajectories with reasonable precision across the illustrated dynamical regimes.

D.1 Lorenz

  • At held-out ρ values within the training range, JEPAWG forecasts follow the true Lorenz trajectories with reasonable precision.The figure shows a stable fixed point, a slowly decaying spiral just below the subcritical Hopf point, and a chaotic attractor.Source: Figure 10 · p. 20
  • In the stable fixed-point regime, the forecast shows a spiral trajectory into the fixed point.The figure compares the truth as a solid line and JEPAWG as a dashed line.Source: Figure 10 · p. 20
  • Just below the subcritical Hopf point, the illustrated Lorenz trajectory is a slowly decaying spiral.This held-out parameter lies within the training range.Source: Figure 10 · p. 20

D.2 Rössler

For the Rössler system, JEPAWG’s weight-space response has peaks aligned with features of the bifurcation diagram, though finite parameter resolution limits visibility near chaos.

D.2 Rössler

  • JEPAWG’s response G(c) shows distinct peaks at Rössler period doublings and transitions into chaos.The plotted range is c ∈[3, 9].Source: §D.2 · p. 20, Figure 11 · p. 20
  • The response peaks mark the 2→4 doubling at c=3.83 and the entry and exit boundaries of periodic windows.The figure compares the response with a ground-truth diagram of successive local maxima of x(t).Source: Figure 11 · p. 20
  • Finite parameter resolution limits visibility of the period doublings closer to chaos.The figure shows per-seed response curves and their seed mean at M=1000.Source: Figure 11 · p. 20

D.3 Hénon

On the Hénon map, JEPAWG’s weight-space response tracks interior period-doubling structure on a two-parameter grid and signals early doublings in the cascade, with weaker resolution at later splittings.

D.3 Hénon

  • Across the Hénon u-cascade, G(u) peaks at each of the first five doublings, shows a step at a6, and has no feature at a7 where seeds begin to disagree.The plotted cascade uses 6000 parameter values and 512 attractor iterates after a 2×10^5-iterate transient.Source: Figure 12 · p. 21
  • The last two orbit-point splittings are narrower than 10^-3 in x and are not resolved at the plotted scale.The cascade diagram marks a1 exactly and locates a2 through a7 by bisection on the attractor period.Source: Figure 12 · p. 21
  • On the (a, b) grid, the ridge of the weight-space response follows the 2→4 and 4→8 doubling curves and broadens over the chaotic band.The response is compared with the ground-truth orbit-period map.Source: §D.3 · p. 21, Figure 13 · p. 21

E Compute Resources

Constructing each weight grid is the dominant computational cost, while JEPAWG training and evaluation take substantially less time.

E Compute Resources

  • 2–6 GPU-hours per grid are dominated by conditional pre-training and M per-parameter fine-tunes.The cost varies with trajectory length and grid size.Source: E Compute Resources · p. 22
  • ∼40 seconds are needed to train one JEPAWG seed on a finished grid, making the five-seed embedding stage negligible relative to grid construction.Source: E Compute Resources · p. 22
  • 5–20 GPU-minutes per system are added for evaluation.Source: E Compute Resources · p. 22

F Experimental Details

Experiments use scrambled Sobol parameter draws and specified training and test protocols. The protocol also defines the reported Lyapunov range and the interpolation and extrapolation test settings.

F Experimental Details

  • M parameter values are drawn from scrambled Sobol sequences, with stratified per-band draws for the Brusselator b sweep and Hénon u-cascade, and joint draws for two-parameter families.The sequence uses seed 0; Table 6 also reports integration transients, recorded training/test lengths, initial conditions, G-sweep evaluation steps, and mean parameter spacing.Source: Table 6 · p. 22
  • max is the largest Lyapunov exponent over the training range, per time unit or per iteration for maps; ≤0 marks ranges without chaotic parameters.Input noise is 5% of the standardised feature scale.Source: F Experimental Details · p. 22
  • Interpolation test parameters are Sobol draws from the training range, except Lorenz-ρ, which uses 10 fixed values.Fixed parameters are Lorenz σ=10 and β=8/3, Rössler a=b=0.2, Brusselator a=1.25, and Hénon b=0.3.Source: Table 7 · p. 22

Related papers

Loading 2610.30847v1…