PaperlayerView exact revision on arXiv

Source-linked AI summary

LLN: Learnable Lens Networks for Parameter-Efficient Long-Horizon Dynamical Prediction

Binbin Yong, Zhao Su, Lan Guo, Haoran Li, Jun Shen, Qingguo Zhou

arXiv:2609.34493v1cs.LG

TL;DR

Residual addition offers a gradient shortcut but leaves feature geometry across depth largely unconstrained. LLN replaces direct feature-space residual accumulation with optical transport in position-angle phase space. It is globally invertible and volume-preserving internally, and experiments report its clearest long-horizon gains on dissipative spatial systems with fewer parameters than selected same-depth comparators.

  • Problem

    Residual additions provide a gradient pathway but do not explicitly regulate representation geometry across depth, while repeated one-step prediction can accumulate errors over long horizons.

    Source: §1 · pp. 1–2

  • Method

    LLN replaces feature-space residual accumulation with repeated free propagation and learnable refraction over coupled position-angle states.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    LLN's clearest gains occur on dissipative spatial systems; for Burgers dynamics, rollout errors are 25.6% and 27.9% of the selected comparator, using 5.5% and 5.9% of its parameters on the two resolutions.

    Source: §5.2 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    LLN offers a parameter-efficient phase-space alternative with analyzable latent geometry, near-neutral initialization gradients across ten systems, and inspectable trajectories on two moons.

    Source: §6 · p. 9

    Evaluation is limited to simulated systems, fixed-depth MLP-style one-step models, and a descriptive two-moons case study; geometric preservation does not guarantee favorable singular values, rollout accuracy, Lyapunov stability, or end-to-end sensitivity.

    Source: LIMITATIONS AND FUTURE WORK · p. 9

Abstract

from arXiv · show

Explicit residual connections of the form (x+f(x)), often combined with normalization layers, have become a standard strategy for training very deep neural networks. However, residual addition primarily provides an algebraic shortcut for gradient propagation, while leaving the evolution of feature geometry across layers largely unconstrained. We introduce Learnable Lens Networks (LLN), a physics-inspired architecture that replaces direct feature-space residual accumulation with learnable optical transport in an augmented position-angle phase space. Each layer alternates between free propagation, which provides an implicit transport path, and a learnable lens field that performs nonlinear trajectory transformation and focusing. Theoretically, we establish that LLN transport is globally invertible and volume-preserving for any differentiable lens field, with the implemented coordinate-wise Gaussian transport further satisfying symplecticity. Importantly, these structural constraints do not limit expressivity: with unrestricted embeddings and readouts, LLN retain universal approximation of continuous end-to-end maps. Experiments across diverse dynamical systems demonstrate that LLN improves long-horizon prediction while using substantially fewer parameters than same-depth comparators. Further analysis reveals stable depth-wise gradient transport and interpretable learned dynamics under the coupled propagation and refraction design.

1 INTRODUCTION

Deep residual networks need to manage signal and gradient behavior across repeated layers, but additive feature-space updates do not explicitly regulate representation geometry. LLN replaces direct residual accumulation with phase-space optical transport and is evaluated for prediction, gradient transport, and learned trajectories.

1 INTRODUCTION

  • Residual shortcuts aid information and gradient propagation, but additive updates leave feature evolution across depth unregulated when stabilization mechanisms are unavailable.Repeated additions may change feature scale and geometry even when gradients remain measurable.Source: §1 · p. 1
  • Recursive one-step prediction can accumulate representation and transition inaccuracies, making long-horizon dynamics a test of stable transformations beyond single-step prediction.Related representation-learning approaches impose structure on latent evolution for multistep prediction Lusch et al. (2018).Source: §1 · p. 2
  • LLN replaces direct feature-space residual accumulation with learned optical transport over coupled position-angle states.Each layer combines free propagation with learned refraction.Source: §1 · pp. 1–2
  • LLN transport is globally invertible and volume-preserving for differentiable lens fields, while its unrestricted embedding and readout retain universal approximation of continuous end-to-end maps.The implemented Gaussian transport is symplectic.Source: §1 · p. 2
  • The empirical studies assess long-horizon rollout accuracy, matched gradient transport, and interpretable low-dimensional optical trajectories, with controlled ablations examining the transport mechanism.Together, they address prediction, parameter efficiency, depth-wise signal propagation, and phase-space interpretability.Source: §1 · p. 2

2 RELATED WORK

Related work addresses deep-network stabilization, structured dynamical evolution, and learned operators or bases. LLN differs by applying structured position-angle transport within a compact fixed-resolution autoregressive predictor.

2 RELATED WORK

  • Deep-network stabilization methods regulate activation scale, residual magnitude, initialization, or depth-dependent tuning; LLN instead replaces explicit feature-space skip addition with position-angle transport.The cited approaches include normalization, initialization, residual-scaling, and normalization-alternative designs Ioffe & Szegedy (2015); Ba et al. (2016); Zhang et al. (2019); Bachlechner et al. (2021).Source: §2 · p. 2
  • Dynamical and structure-preserving networks encode evolution or geometric biases, whereas LLN constrains repeated latent transport while leaving embedding and readout unrestricted.This permits dissipative observed-state maps despite internal transport constraints.Source: §2 · p. 2
  • Neural operators learn families of differential-equation solution maps, while LLN is evaluated as a compact fixed-resolution autoregressive predictor rather than a resolution-independent operator kernel.The cited operator literature emphasizes discretization transfer, PDE constraints, and simulation acceleration Li et al. (2021; 2024); Azizzadenesheli et al. (2024).Source: §2 · pp. 2–3
  • Adaptive-basis models use learnable basis expansions, and LLN uses Gaussian basis learning to refract an angle variable inside a coupled transport map.The cited examples include radial-basis-function networks and Kolmogorov-Arnold Networks Park & Sandberg (1991); Liu et al. (2025); Rigas et al. (2026); Li et al. (2025).Source: §2 · p. 3

3 LEARNABLE LENS NETWORKS

LLN represents hidden states in position-angle phase space and composes free propagation with learnable refraction. Its coupled states give depth a trajectory-based transport interpretation rather than direct feature-space residual addition.

3 LEARNABLE LENS NETWORKS

  • LLN separates feature position y and angle θ, with free propagation and learnable refraction repeated across depth in a phase-space map whose layer Jacobian has determinant one.The architecture contrasts with same-feature-space residual addition, which can accumulate forward variance when normalization is removed.Source: Figure 1 · p. 3

3.1 PHASE-SPACE REPRESENTATION

  • The initial position y0 represents the current feature position, while bounded θ0 is an auxiliary directional state that controls transport through depth.Bounding θ0 prevents excessively large initial displacements and sets a natural scale when normalization is absent.Source: §3.1 · p. 3

3.2 OPTICAL TRANSPORT LAYER

  • Each optical layer first propagates position by the current angle and then refracts the angle at the propagated position using a learnable lens field.The field uses coordinate-wise Gaussian basis functions with learnable coefficients, fixed control points, and locality controlled by σ.Source: §3.2 · p. 4
  • The complete one-step map embeds the input, repeats transport layers, and reads out the final position; setting L = T/D keeps cumulative free-propagation distance fixed as depth changes.The terminal refraction updates the auxiliary angle but does not change the one-step prediction, which depends only on the final position.Source: §3.2 · p. 4

3.3 STRUCTURED RESIDUAL TRANSPORT

  • Position stores the current representation while angle determines its next displacement, so information travels through depth as a trajectory transformed by learned refraction.This separates transport from nonlinear transformation instead of directly overwriting the representation.Source: §3.3 · p. 4

4 GEOMETRIC ANALYSIS OF LLN DYNAMICS

Each LLN transport layer is globally invertible and preserves volume in augmented phase space for any continuously differentiable lens field. These internal constraints coexist with flexible end-to-end prediction through unrestricted embedding and readout.

4.1 PHASE-SPACE JACOBIAN

  • For any continuously differentiable lens field, an LLN layer is a globally invertible diffeomorphism with determinant-one Jacobian, preserving augmented phase-space volume.The inverse is explicit and globally defined whenever the lens field is continuously differentiable.Source: §4.1 · p. 5
  • Volume preservation constrains internal transport but leaves the encoder and readout flexible enough to represent dissipative and other non-volume-preserving target maps.The determinant statement applies to the repeated latent transport stack, not the end-to-end predictor.Source: §4.1 · p. 5
  • A unit determinant constrains the product of layer singular values, but individual singular values may still evolve differently across depth, motivating gradient and rollout diagnostics.The implemented symplectic layer has reciprocal singular-value pairing.Source: §4.1 · p. 5
  • LLN uses phase-space geometry as an architectural bias for stable deep transformations and long-horizon prediction, not as an objective for density modeling.This distinguishes its purpose from normalizing flows.Source: §4.1 · p. 5

5 EXPERIMENTS

The experiments evaluate LLN in long-horizon rollouts, matched gradient-transport tests, and optical trajectory case studies. LLN’s strongest rollout advantages appear on dissipative spatial systems, while its gradients remain near-neutral at initialization and its learned trajectories can be inspected layer by layer.

5.1 EXPERIMENTAL SETUP

  • The study trains one-step models on simulated dynamical systems and evaluates normalized MSE over recursive rollouts, comparing LLN with five non-LLN architectures across ten seeds.The rollout comparison includes depth sweeps; the long-horizon setup evaluates recursively generated states.Source: §5.1 · p. 6, Figure 2 · p. 6

5.2 EXPERIMENT 1: LONG-HORIZON ROLLOUT PERFORMANCE

  • LLN lowers rollout error with substantially fewer parameters on dissipative spatial systems; its advantage narrows on mechanical systems, and it slightly trails the selected comparator on Lorenz-28.The depth sweep also reports the lowest mean error on both advection-diffusion settings at every evaluated depth and on reaction-diffusion at D ≥20.Source: Table 1 · p. 7, §5.2 · pp. 7–8

5.3 EXPERIMENT 2: MATCHED DEPTH-WISE GRADIENT TRANSPORT

  • At initialization, LLN’s mean log10(g1/g40) ranges from 0.000 to 0.008, while plain MLP attenuates first-layer gradients by about 15 orders of magnitude and unscaled ResMLP amplifies them by more than three.The LLN values correspond to g1/g40 between 1.00 and 1.02; Figure 3 shows position-channel gradients near the final-layer reference through depth on five representative systems.Source: Figure 3 · p. 7, §5.3 · pp. 8–9, Table 2 · p. 8

5.4 EXPERIMENT 3: INTERPRETING LLN THROUGH OPTICAL TRAJECTORIES

  • In a frozen two-moons case study, the PCA basis explains 97.0% of LLN position-state variance versus 72.7% of MLP hidden-state variance, and the displayed LLN error lies at the 55th percentile.Across seven LLN errors, the median maximum turn is 16.96° versus 15.84° for 393 correct samples; the figures trace position paths and layerwise refraction.Source: Figure 4 · p. 8, §5.4 · p. 9, Figure 5 · p. 9

6 DISCUSSION AND CONCLUSION

LLN is presented as a parameter-efficient phase-space alternative whose clearest gains occur on evaluated dissipative spatial dynamics. The paper also identifies limits on its evaluation scope and outlines additional theory and reproducibility materials.

6 DISCUSSION AND CONCLUSION

  • LLN’s clearest gains are on evaluated dissipative spatial dynamics, alongside parameter efficiency, gradient neutrality, and inspectable position-angle trajectories.The conclusion also reports the strongest multi-trajectory gains on five systems and a best full structure in nine of ten controlled ablations.Source: §6 · p. 9
  • Evaluation is limited to simulated systems, fixed-depth MLP-style one-step models, and a descriptive two-moons case study.The authors note that volume preservation and symplecticity do not guarantee favorable singular-value behavior, rollout accuracy, Lyapunov stability, or end-to-end sensitivity.Source: LIMITATIONS AND FUTURE WORK · p. 9
  • Sections 3–5 specify the model, theory, protocols, metrics, baselines, depth, and seed averaging; the supplement is to include code, seeds, raw tables, and figure scripts.Appendices A–B provide proofs, counts, ablations, configurations, and extended depth results.Source: REPRODUCIBILITY STATEMENT · p. 10

A ADDITIONAL THEORY AND EXPRESSIVITY

  • Appendix A completes the theory with symplecticity, reciprocal singular-value structure, and universal approximation, alongside exact counts; Appendices B and C add extended empirical evidence and trajectory cases.The appendix also clarifies that latent symplecticity does not make observed dynamics Hamiltonian.Source: A ADDITIONAL THEORY AND EXPRESSIVITY · p. 13

A.1 SYMPLECTIC STRUCTURE OF THE IMPLEMENTED TRANSPORT

The implemented coordinate-wise Gaussian transport is symplectic, with reciprocal singular-value pairs, but this structure is not a conditioning guarantee. The appendix also quantifies parameter and arithmetic scaling while distinguishing operation counts from platform-dependent latency.

A.1 SYMPLECTIC STRUCTURE OF THE IMPLEMENTED TRANSPORT

  • A symmetric lens Jacobian makes each implemented Gaussian transport layer, its finite compositions, and their explicit inverses symplectic.The coordinate-wise Gaussian lens has a diagonal, hence symmetric, Jacobian.Source: §A.1 · p. 13
  • Symplectic transport has reciprocal singular-value pairs, but may still have simultaneously very large and very small singular values.The result applies to the square latent map; non-square embedding and position-only readout do not make the observed predictor symplectic or volume-preserving.Source: §A.1 · p. 13

A.2 PARAMETER AND OPERATION SCALING

  • At h = 128 and K = 12, the depth-dominant parameter ratio is 9.3%.The depth-dependent learnable component scales as O(DhK) for LLN versus O(Dh2) for a dense MLP.Source: §A.2 · pp. 13–14
  • The operation audit counts each multiply-add as two FLOPs and reports exponential and hyperbolic-tangent evaluations separately.LLN evaluates DhK exponentials and h hyperbolic tangents, unlike the plain MLP, which evaluates no transcendental functions.Source: §A.2 · p. 14
  • For h = 128 and K = 12, the reported count ratio is 75/258 ≈ 29.1%; these counts establish parameter and arithmetic scaling, not proportional wall-clock latency reduction.Wall-clock latency also depends on kernel fusion, memory access, and transcendental throughput on the execution platform.Source: §A.2 · pp. 13–14

A.3 UNIVERSAL APPROXIMATION

A shallow LLN can approximate any continuous map on a compact domain when its hidden dimension is sufficiently large. This is an existential expressivity result, not a guarantee about approximation rate, conditioning, or optimization.

A.3 UNIVERSAL APPROXIMATION

  • With sufficiently large hidden dimension, a shallow LLN with two optical layers and a linear readout can approximate any continuous map on compact K to arbitrary accuracy.The construction uses a restricted subset of LLN parameters; the practical architecture includes it as a special case.Source: §A.3 · pp. 14–15
  • The proof realizes finite Gaussian ridge expansions using paired coordinates, a single active Gaussian lens term, zero initial angles, and a zero second lens.A linear readout recovers the expansion, which uniformly approximates the target map.Source: §A.3 · pp. 14–15
  • The theorem establishes density in C(K, Rq), but gives no finite-width approximation rate, conditioning guarantee, or optimization guarantee.Unrestricted, generally non-square embeddings and readouts are consistent with latent volume preservation.Source: §A.3 · p. 15

B.1 EXPERIMENT PROTOCOLS AND SELECTED CONFIGURATIONS

The experiments generate trajectories from specified dynamical systems rather than downloading benchmark data, and use different recorded protocols for selected comparisons. The appendix indexes protocols and deployments so tables can focus on their diagnostic results.

B.1 EXPERIMENT PROTOCOLS AND SELECTED CONFIGURATIONS

  • Trajectories come from source-code system dynamics and an RK4 integrator; spatial-system suffixes denote discretization points, while Lorenz suffixes denote parameter settings.SpringChain-n has n masses and 2n state variables; Lorenz systems always have three state variables.Source: §B.1 · p. 15
  • A fixed lens scale multiplies the learned lens correction; setting it to zero disables refraction while retaining the parameter tensors.The scale is normally 1 and adds no learnable parameters; Appendix Table 3 indexes protocols, while Table 4 lists selected LLN deployments.Source: §B.1 · p. 15, Table 3 · p. 16
  • For dynamics rollouts, unless otherwise stated, evaluation uses 10 random seeds, 3000 training pairs, 700 test pairs, and 500-step free-running evaluation; the original-grid recipe uses six Adam epochs.The original-grid LLN learning rate is 10−3; the tuned non-LLN recipe uses learning rate 3×10−4, weight decay 10−6, and gradient clipping 1.0. Lorenz-28 completion uses 20 AdamW epochs with learning rate 5×10−4, weight decay 10−5, gradient clipping 1.0, and cosine decay.Source: §B.1 · p. 15

B.2 FROZEN MULTI-TRAJECTORY CONFIRMATION

Frozen multi-trajectory evaluation supports initial-condition generalization on a subset of systems, not a blanket all-system advantage. Supplementary rollout curves show LLN below the selected baseline at every displayed step on three dissipative PDE cases, with mixed behavior on Lorenz-24.

B.2 FROZEN MULTI-TRAJECTORY CONFIRMATION

  • The confirmation freezes all ten selected D = 40 deployments and evaluates four perturbed training trajectories against four disjoint perturbed test trajectories.Geometric means aggregate trajectory windows after averaging log ratios within each training seed; intervals resample six confirmatory seeds as clusters.Source: §B.2 · pp. 16–17
  • Five of ten systems have seed-clustered ratio intervals entirely below one, while Lorenz-24, Lorenz-28, and SpringChain-32 favor the comparator.AdvecDiff-32 and ReactionDiff-32 have LLN-favorable geometric means but intervals crossing one; the confirmation does not support an all-system claim.Source: §B.2 · p. 17, Table 5 · p. 17

B.3 SUPPLEMENTARY TEMPORAL ROLLOUT CURVES

  • LLN stays below the selected baseline at every displayed step on ReactionDiff-48, AdvecDiff-48, and Burgers-48; Lorenz-24 shows mixed chaotic-flow behavior.Figure 6 plots geometric-mean normalized state MSE over seven confirmation seeds, with 95% seed-bootstrap intervals; lower is better.Source: §B.3 · p. 17, Figure 6 · p. 18

B.4 CONTROLLED STRUCTURE ABLATION

Controlled ablations favor the complete LLN on most systems, while identifying SpringChain-32 as a boundary case. The additional diagnostics report a stronger LLN margin at larger depths on dissipative PDEs and consolidate two-moons error and turn analyses.

B.4 CONTROLLED STRUCTURE ABLATION

  • The complete LLN is best on nine of ten systems; removing refraction raises error everywhere, zeroing the initial angle worsens nine systems, and removing propagation severely degrades Lorenz and PDE cases.SpringChain-32 is the exception: variants are close, and no-propagation is slightly lower; spring-chain results remain near the full model when propagation is removed.Source: §B.4 · p. 18, D System LLN MLP Res Res-S Res-BN LS… · p. 19

B.5 ALL-DEPTH AND EXTENDED-BASELINE DIAGNOSTICS

  • The fixed-depth sweep reports LLN-to-comparator MSE, parameter, and matched arithmetic-FLOP ratios, and states that LLN's margin strengthens at larger depths on dissipative PDEs.The FLOP audit matches system dimension, depth, and width; it reports LLN exponential and hyperbolic-tangent evaluations separately from simple arithmetic FLOPs.Source: §B.5 · p. 19, Table 7 · p. 19

C ADDITIONAL OPTICAL CASE STUDIES

Related papers

Loading 2609.34493v1…