PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Transferable Policies from Action-free Time Series Through Dynamical Embeddings

Niklas Emonds, Georgia Koppe

arXiv:2610.03065v1cs.LG

TL;DR

Learning control from action-free recordings is difficult because intervention effects are unobserved and policies can exploit errors in reconstructed dynamics. The paper combines hierarchical dynamical reconstruction with shared, embedding-conditioned policies trained in simulation under an explicit intervention model. These policies transfer more reliably than independently trained policies on Lorenz-63 and double-pendulum systems, while the approach's generality depends on assumptions about system variation and action effects.

  • Problem

    Action-free recordings leave control-action effects unobserved, while policies trained in reconstructed simulators may exploit model errors.

    Source: §1 · pp. 1–2

  • Method

    The framework combines hierarchical dynamical reconstruction with shared policy and value networks conditioned on low-dimensional embeddings and trained on simulated transitions under an explicit latent intervention model.

    Source: §1 · p. 2

  • Results

    Hierarchical policies transferred more reliably to the Lorenz-63 and double-pendulum systems than independently trained policies.

    Source: §5 · p. 13

  • Takeaways & Limitations

    Piecewise-linear dynamics and decoder-based constraints support mechanistic analysis of control and interventions across distinct intervention and target modalities.

    Source: §1 · p. 2, §5 · p. 12

    The approach assumes subject-specific dynamics lie in linear subspaces and that factors distinguishing dynamics also determine appropriate control.

    Source: §5 · p. 13

Abstract

from arXiv · show

Learning control from action-free recordings is challenging because intervention effects are unobserved and policies may exploit errors in reconstructed dynamics. We present a hierarchical model-based reinforcement learning framework that uses shared structure across related systems to learn system-specific control policies from action-free recordings. A hierarchical dynamical system reconstruction model captures shared dynamics and individual variation through low-dimensional embeddings. These embeddings are then reused to parameterize shared policy and value networks, linking differences in reconstructed dynamics to differences in control. Policies are trained entirely via simulation under an explicit intervention model with additive latent perturbations. Piecewise-linear recurrent neural networks enable mechanistic analyses of the controlled dynamics, while decoder-based constraints make the immediate effects of interventions interpretable in observation space and permit interventions on one modality while protecting another from direct manipulation. On Lorenz-63 and double-pendulum systems, hierarchical policies improve transfer over independently trained policies. On Lorenz-63, they also achieve a higher mean reward than repeated planning with the same reconstructed models, perform comparably to methods trained with controlled interactions, and generalize to systems absent from policy training after embedding inference alone. Applications to neural-behavioral recordings demonstrate suppression of predicted movement under constrained neural perturbations. Together, these findings show how shared dynamical representations support transferable control and mechanistic hypothesis generation from action-free recordings.

1 Introduction

The paper addresses learning control from recordings without observed control actions, where reconstructed dynamics may be exploited and actuator effects must be assumed. It combines hierarchical dynamical reconstruction with simulated policy learning, using shared embeddings to connect system variation to control.

1 Introduction

  • Action-free recordings contain no responses to the considered control actions, leaving intervention effects unobserved and actuator coupling to prior knowledge or an explicit model.Policies may exploit reconstruction errors, especially when actions move systems into regions poorly constrained by the recordings.Source: §1 · pp. 1–2
  • The framework reuses low-dimensional reconstruction embeddings to condition shared policy and value networks trained with SAC on simulated transitions.The shared parameterization provides an alternative constraint when no empirical action distribution is available for policy regularization.Source: §1 · p. 2
  • The intervention model adds actions as latent-state perturbations before learned transitions, while linear decoders expose their immediate observation-space effects and support modality-specific constraints.For example, interventions can alter decoded neural activity while preserving immediate behavioral output.Source: §1 · p. 2
  • Piecewise-linear ALRNN dynamics provide analytic access to fixed points and stability properties for examining how interventions shift control mechanisms.These analyses support hypotheses about which observed-variable changes achieve control and through which dynamics.Source: §1 · p. 2
  • On Lorenz-63 and double-pendulum systems, hierarchical policies transfer to the underlying systems better than independently trained SAC controllers.The experiments also include embedding-based adaptation to unseen Lorenz-63 systems and in-silico behavioral control in models fitted to neural–behavioral recordings.Source: §1 · p. 3

2 Related Work

The related work spans model-predictive control, model-based and offline reinforcement learning, dynamical-system reconstruction, and neural control. The paper positions hierarchical parameter sharing as a way to connect reconstruction embeddings to control when action data are absent.

Model-predictive control

  • Classical MPC repeatedly optimizes finite-horizon actions, while learned extensions such as CEM-MPC and TD-MPC retain planning in learned models or latent spaces.The paper's CEM-MPC baseline uses the same reconstructed ALRNN but replans at every time step.Source: §2 · p. 3

Model-based and offline RL

  • Offline MBRL methods can regularize against empirical state-action distributions, but action-free recordings provide no observed action distribution, motivating hierarchical sharing as an implicit regularizer.Here, hierarchy means shared and subject-specific parameter components rather than temporal scales or option-based control.Source: §2 · p. 3

Reconstruction of families of dynamical systems

  • Hierarchical DSR models system families with shared parameters and low-dimensional embeddings, and this work conditions policy and value functions on the same embeddings.DSR targets attractor geometry and long-run temporal structure from time series.Source: §2 · p. 3

Control in neuroscience and cross-modal interventions

  • Neural-control approaches use simulated environments or observational transition models when interventional data are constrained, with action effects supplied by a simulator or assumed perturbation model.The neural experiments restrict actions to neural decoder directions so predicted behavioral changes pass through learned latent dynamics.Source: §2 · p. 3

3 Methods

The methods reconstruct related systems from action-free recordings, then use those reconstructions as simulated environments for shared control. Decoder constraints make intervention effects interpretable, while piecewise-linear dynamics support analysis of how actions alter the system.

3.1 Preliminaries and problem setting

  • The control environment combines a learned transition from action-free recordings with specified latent perturbations and a reward based on reaching target states.The reward also regularizes control effort.Source: §3.1 · p. 4, §3.3 · p. 5

3.2 Learning the dynamics from recordings

  • Hierarchical ALRNN reconstructions learn latent dynamics and observation decoders while sharing parameters across systems through low-dimensional embeddings.Generated trajectories are assessed against observed state distributions and temporal structure, as well as prediction accuracy.Source: §3.2 · p. 4

3.3 Turning the reconstruction into a control environment / world model

  • Actions perturb decoded latent dimensions before the learned transition, and decoder null-space projections can prevent direct changes to specified outputs.The approach’s validity depends on reconstruction accuracy and whether the prescribed perturbation represents the intervention of interest.Source: §3.3 · p. 5

3.4 Learning a policy shared across systems

  • SAC policies and value networks reuse reconstruction embeddings to share parameters across systems, restricting specialization; unseen systems can be added by fitting a new embedding.For a new system, the embedding is fit from action-free data and inserted into the fixed control networks without policy optimization.Source: §3.4 · p. 6, Figure 1 · p. 7

3.5 Understanding how actions modify the dynamics

  • Piecewise-linear latent dynamics divide state space into regions with computable fixed points and stability, letting analyses track how actions change regions and shift fixed points.The effective local transition matrix depends on the active region; fixed points may be virtual if they or their approach cross into another region.Source: §3.5 · p. 6

4 Experiments

Experiments test control transfer on Lorenz-63 and double-pendulum systems, reuse on unseen systems, and in-silico movement suppression from neural and behavioral recordings.

4.1 Evaluation and comparison methods

  • Synthetic benchmarks were evaluated on both learned and true systems against independent policies, same-reconstruction online planning, and methods trained with controlled interactions.Methods shared reward, action space, horizon, and initial-state distribution within each benchmark; reward scales were compared only within tasks.Source: §4.1 · pp. 7–8

4.2 Transfer of control in families of benchmark dynamical systems

  • On true Lorenz-63, hierarchical SAC scored −0.040 ± 0.004 versus independent SAC’s −0.171 ± 0.017, and was near CEM-MPC (−0.0442±8·10−5) without action training or online planning.Independent policies scored higher in their training world models, while TD-MPC and DreamerV3 used controlled interactions.Source: Table 1 · p. 9, §4.2 · p. 9
  • On double-pendulum systems, hierarchical SAC outperformed independent SAC (−9.74±2.42 vs. −16.82±7.78) and stabilized true systems in 49% of trials.CEM-MPC achieved higher rewards; DreamerV3 outperformed hierarchical SAC, whereas TD-MPC performed worse. Embeddings also reflected mass and length ratios despite nonlinear equations of motion.Source: Table 1 · p. 9, §4.2 · p. 10, Figure 3 · p. 10
  • On unseen Lorenz systems, fitted embeddings improved true-system reward to −0.036±0.002 from −0.046±0.005 with training embeddings; unseen-pendulum performance did not significantly improve over nearest-training embeddings.For unseen Lorenz systems, only subject vectors were fitted while shared dynamics parameters remained fixed.Source: §4.2 · pp. 9–10
  • Under Lorenz control, 90% of time steps fell in at most four regions whose local dynamics favored convergence, with fixed points shifting toward the target.The concentration was not imposed by the reward structure; occupied regions also showed predominantly stable eigenvalues.Source: Figure 2 · p. 8, §4.2 · p. 9

4.3 Control in models of neural and behavioral recordings

  • Across LINK sessions, the shared controller suppressed predicted finger movement (median reward −0.090±0.002), but control failed when eight channels were excluded.The 96 channels lay in a 10-dimensional decoded subspace, with two directions reserved for protected finger outputs.Source: Figure 4 · p. 11, §4.3 · p. 11
  • In delayed-reaching recordings, neural actions suppressed predicted hand movement despite the go cue, while predicted eye trajectories also changed and neural dynamics remained physiologically plausible.The hand was the control target; the policy needed only sparse actions after the goal cue.Source: Figure 5 · p. 12, §4.3 · p. 12

5 Discussion

Hierarchical policies transfer more reliably than independently trained policies across Lorenz-63 and double-pendulum systems, and unseen-system results suggest embeddings capture dynamics relevant to control. The discussion connects shared representations to control mechanisms while noting limits on generality and physiological validation.

5 Discussion

  • Across Lorenz-63 and double-pendulum benchmarks, hierarchical policies transferred more reliably than independently trained policies, despite the latter performing as well or better in simulation.The authors suggest parameter sharing reduces overfitting to subject-specific world-model inaccuracies.Source: §5 · p. 13
  • Because action-free recordings provide no empirical action distribution, reduced policy capacity may regularize against specialization to individual world-model inaccuracies.The authors present this as an additional role for parameter sharing beyond data efficiency and generalization.Source: §5 · p. 13
  • On unseen Lorenz systems, embedding inference supported generalization; on unseen double pendulums, the shared policy maintained performance, but fitted embeddings did not outperform nearest-training-system embeddings.Repeated online planning partly improved performance.Source: §5 · p. 13
  • Controlled trajectories concentrate in a subset of previously visited linear regions whose fixed points lie near the goal and whose local dynamics are attractive.The policy and value function remain fixed while the subject-specific representation is adapted, providing a common coordinate system for reconstruction and control.Source: §5 · p. 13
  • In multimodal neural-behavioral simulations, decoder-null-space actions modify predicted hand trajectories while minimally shifting neural summary statistics, but do not validate physiological control.The constraint prevents direct behavioral manipulation, so changes arise through learned neural-behavioral couplings.Source: §5 · p. 13
  • Generality may be limited by linear-subspace parameterization, the assumption that reconstruction embeddings determine suitable control, the need for system-family coverage, and unknown physical intervention coupling.The intervention model represents actions as additive latent perturbations.Source: §5 · p. 13

A.1 Lorenz-63

  • The Lorenz-63 appendix contrasts an example with actions restricted to the third state dimension, control-related linear-region and fixed-point analyses, and trajectories for all 64 systems.The figures identify the single-dimension example as having limited controllability and depict the target state in the trajectory panels.Source: Figure 7 · p. 18, Figure 8 · p. 19, Figure 9 · p. 20
  • −0.036 ± 0.002 reward for fine-tuned vectors on unseen subjects compared with −0.046 ± 0.005 for unchanged nearest-training-subject vectors; the PCA projection relates embeddings to Lorenz parameter regimes.Figure 10 shows fine-tuned vectors inserted into the fixed SAC model controlling unseen systems without further optimization; Figure 6 encodes σ by color and β and ρ by vertical and horizontal separation.Source: Figure 6 · p. 18, Figure 10 · p. 21

A.2 Macaque delayed center-out reach data

The supplied appendix materials show control-related analyses of macaque neural data and trajectories for LINK subjects, alongside DSR training details and Lorenz-63 data-generation procedures.

A.2 Macaque delayed center-out reach data

  • During control of multimodal neural data, occupied linear regions admit non-virtual fixed points close to the target, indicating that the policy steers into regions whose linear dynamics favor reaching the goal.Figure 11 reports region occupancy and each occupied region’s minimal distance from the target to a non-virtual fixed point.Source: Figure 11 · p. 22

A.3 LINK Data

  • Figure 12 presents trajectories for all LINK data subjects without reporting a quantitative comparison or outcome.The caption identifies the subject coverage but does not specify an encoding or trajectory result.Source: Figure 12 · p. 22

B DSR Training Protocol

  • DSR is trained by minimizing trajectory mean-squared error, with generalized teacher forcing, subject-specific variance normalization, and additional latent/observation-space losses and invertibility penalties for non-identity observation models.The protocol also applies L2 regularization to selected model parameters.Source: B DSR Training Protocol · p. 23

B.1 Lorenz-63 data

  • Lorenz-63 data use 64 parameter configurations, 5,000 training and 100,000 test steps per subject, and training noise with variance equal to 5% of subject-specific data variance.The system’s regimes vary with parameters and may include stable fixed points, periodic orbits, and aperiodic chaotic dynamics.Source: §B.1 · p. 23

B.2 Lorenz-63 training and evaluation

The Lorenz-63 world model lifts three-dimensional observations into a larger latent state, with ReLU units defining piecewise-linear regions. Reported test-set metrics quantify geometric and temporal agreement, and a figure shows ground-truth and reconstructed trajectories.

B.2 Lorenz-63 training and evaluation

  • The model lifts three-dimensional observations to latent dimension L = 26, with ReLU applied to P = 12 units, yielding at most 4096 linear regions.The encoder and decoder are identity mappings.Source: §B.2 · p. 23
  • World-model validity is assessed through geometric agreement using DKL and temporal agreement using dimension-averaged Hellinger distance DH between power spectra.The passage describes estimating these measures using discrete binning and the Fast Fourier Transform, respectively.Source: §B.2 · p. 23
  • Test-set medians were DKL = 1.8 and DH = 0.15 across subjects; for 64 fine-tune subjects, DH = 0.26 and DKL = 5.3.Figure 13 depicts ground-truth test trajectories in blue and reconstructions in orange.Source: §B.2 · p. 23, Figure 13 · p. 24

B.3 Double-pendulum data

The double-pendulum data section specifies the observed state representation, energy quantities, and training and fine-tuning parameter configurations with trajectory sampling details.

B.3 Double-pendulum data

  • The six-dimensional observations encode each pendulum angle with sine and cosine and include both angular velocities.The passage defines arm lengths, masses, and gravitational acceleration for the equations of motion.Source: §B.3 · p. 25
  • The reported kinetic and potential energy expressions describe quantities associated with the double-pendulum data generation.The displayed energy expression is provided in the passage.Source: §B.3 · p. 25
  • Training uses 16 configurations of arm lengths and masses, with 50 time series of 200 steps per configuration; fine-tuning uses a separate set of parameter configurations.Initial angles and velocities are sampled uniformly from [0, 2π] and [−5, 5], respectively; test trajectories contain 10,000 steps per subject and training energy.Source: §B.3 · p. 25

B.4 Double-pendulum training and evaluation

The appendix evaluates model reconstruction and training data across double-pendulum and neural-behavioral recordings, reporting test errors and describing the control and data settings.

B.4 Double-pendulum training and evaluation

  • Double-pendulum actions affected only angular velocities, and stabilization required low final target distances and angular-velocity variability.The thresholds were below 0.5 rad for angles, 5 rad/s for angular velocities, and 5 rad/s standard deviation over the final 100 steps.Source: §B.4 · p. 25
  • The double-pendulum world model had median DKL = 6.4 and DH = 0.22, with prediction quality limited and energy fluctuations increasing over time.The reconstruction used data alone, without physics-informed inductive biases.Source: §B.4 · p. 25, Figure 14 · p. 26

B.5 LINK data

  • LINK recordings came from one nonhuman primate performing two-degree-of-freedom finger movements, with neural and kinematic signals filtered, normalized, and segmented into 200-step sequences.The dataset contains 312 sessions collected over about 3.5 years.Source: §B.5 · p. 27

B.6 LINK training

  • For LINK, neural and kinematic signals were modeled with Gaussian decoders, and the target position served as external input to the latent ALRNN.Each subject had about 100 training and validation time series and 25 test series on average; a causal convolutional encoder inferred latent-state mean and variance.Source: §B.6 · p. 27
  • Test-set median mean squared error was 0.81 for neural data and 0.09 for kinematic data across LINK subjects.The hierarchical approach treated 12 randomly selected sessions as subjects.Source: §B.6 · p. 27

B.7 Macaque delayed center-out reach data

  • In Pei et al. (2021), a macaque controlled a cursor to reach targets in maze configurations; recordings included 182 neurons in primary motor and dorsal premotor cortex at 1000 Hz, plus hand and eye position.Only mazes without barriers were considered.Source: §B.7 · p. 27

B.8 Macaque delayed center-out reach training

The delayed center-out reach model is evaluated on held-out trials using neural spike-rate and position reconstruction errors, with distinct decoders for these observation types.

B.8 Macaque delayed center-out reach training

  • The model used a causal convolutional encoder, Gaussian decoders for hand and eye positions, and a zero-inflated negative binomial decoder for spike counts.The hand-position decoder alone served as the goal decoder.Source: §B.8 · p. 27
  • Spike rates had variance-normalized mean squared error of 0.17 ± 0.06 across test trials, while position variables had mean squared error of 0.04 ± 0.07.The figure shows empirical and reconstructed spike rates for a representative trial.Source: §B.8 · p. 27, Figure 15 · p. 28
  • The evaluation held out 50 of 437 trials for testing, with the remaining 387 used for training and validation.Source: §B.8 · p. 27

C SAC implementation

The SAC implementation trains hierarchical actor and critic networks from simulated world-model trajectories, constrains direct effects on protected outputs, and uses measurement-aware batching for reconstruction and control.

C SAC implementation

  • The actor and twin Q-networks are ReLU feedforward networks with hierarchical weights and biases, and critic targets use Polyak-smoothed target networks.Because rewards are non-positive, target values are clipped at a maximum of 0.Source: C SAC implementation · p. 28
  • The replay buffer contains only world-model-generated trajectories, with Gaussian latent-state noise added at each step and no RL gradients propagated into the world model.At evaluation, expected policy and decoder values are used.Source: C SAC implementation · p. 28
  • Latent actions are projected onto the null space of protected decoder weights so they cannot directly change protected outputs such as goal decoders.The projection is formed using singular value decomposition after tanh-squashing actions.Source: C SAC implementation · p. 28

D Batching protocol

  • Measurement-aware batches contain subject-specific subbatches; DSR losses are averaged across subbatches before a joint optimizer step.SAC alternates actor-phase batch collection with critic-phase replay updates to Q-networks and policy.Source: D Batching protocol · pp. 28–29

E Baseline implementation

The baseline comparison uses shared evaluation conditions while contrasting planning in reconstructed dynamics with methods trained using controlled interactions; hyperparameter settings and evaluation runs are also specified.

E Baseline implementation

  • All methods used the same reward, initial-state distribution, 500-step evaluation horizon, and projected action space for each subject.Only projected y and z components were applied in Lorenz-63, and only projected angular-velocity components in the double pendulum.Source: E Baseline implementation · p. 29
  • CEM-MPC planned by simulating candidate action sequences in each pretrained subject-specific ALRNN and applying only the first selected action, without controller training.Source: E Baseline implementation · p. 29
  • TD-MPC and DreamerV3 learned from controlled true-environment interactions, whereas CEM-MPC used only pretrained world models for planning.TD-MPC optimized short-horizon sequences with an MPPI-style planner; DreamerV3 used imagined trajectories and actor-critic learning.Source: E Baseline implementation · p. 29

F Hyperparameters

  • Evaluation used at least six SAC runs with different seeds, and Table 2 reports hyperparameters selected or optimized through Bayesian optimization.Group-level parameters used Xavier initialization.Source: F Hyperparameters · p. 29, Table 2 · p. 30

Related papers

Loading 2610.03065v1…