PaperlayerView exact revision on arXiv

Source-linked AI summary

A neural-astrocyte architecture implements a hybrid automaton for evidence accumulation

Giacomo Vedovati, Ilya E. Monosov, Thomas J. Papouin, ShiNung Ching

arXiv:2609.16217v1q-bio.NCcs.NE

TL;DR

The paper asks how astrocytes may help neural networks infer latent context in reinforcement-learning environments. It models a slow astrocytic network coupled to fast neuronal dynamics and finds that their interaction implements evidence accumulation and context switching as a hybrid automaton. The authors conclude that this architecture offers a computational framework for contextual modulation by astrocytes.

Abstract

from arXiv · show

Astrocytes are non-neuronal glial cells that are receiving widespread attention due to their emerging role in neural computation. In this paper, we propose and study dynamical mechanisms by which astrocytes may augment the ability of neural networks to infer context in reinforcement learning (RL) settings. We construct a biologically inspired, two-level dynamical neural-astrocyte network with distinct spatial and temporal organization. We train this model on a hierarchical multi-context task that requires the agent to infer changes in latent task rules based on derived rewards. We find that in this setting, astrocytes enable evidence accumulation of changes in context and subsequent context-specific modulation of neural dynamics. We show that these functions are implemented via two dynamical mechanisms: (i) reward-induced bifurcations that relocate an asymptotically stable attractor into different, context-specific regions of state space, and (ii) the relative shallowness of these attractors, mediated by the entropy of the environment, giving rise to behavioral stickiness. Together, these mechanisms amount to a hybrid automaton, in which uncertainty accumulates until, eventually, the neural dynamics are switched to a new context. This model provides a neuro-dynamic schema, compatible with neural-astrocyte biology and prior empirical observations, for how astrocytes may integrate information from the periphery and drive contextual changes in neural circuits.

I. INTRODUCTION

The introduction frames astrocytes as slow, spatially extended modulators that may help neural networks infer latent context from reinforcement-learning feedback. The paper models this role with a fast-slow neural-astrocyte architecture and characterizes its context-switching dynamics.

A. Neural-Astrocyte interactions and computation

  • Astrocytes’ second-to-minute dynamics complement millisecond neuronal activity, while their signaling and gap-junction networks can integrate and spatially distribute modulation.The cited background describes sensing local activity, regulating synaptic efficacy and ionic conditions, and influencing neuronal dynamics across microcircuits.Source: A. Neural-Astrocyte interactions and… · p. 1

B. Astrocytic contextual guidance and reinforcement learning

  • Astrocytes may support inference of latent rules by accumulating evidence from behavioral outcomes, including futility feedback, before context-specific modulation.The paper develops a network model to study this proposed role in reinforcement-learning environments.Source: B. Astrocytic contextual guidance and… · p. 1

C. Fast-slow neural-astrocyte network models

  • The model couples a slow astrocytic network that integrates contextual inputs with a fast neuronal recurrent circuit whose activity it modulates.The architecture is probed on a task where latent context must be inferred from action-outcome observations.Source: C. Fast-slow neural-astrocyte network… · pp. 1–2
  • The paper characterizes how astrocytes accumulate evidence about latent environmental parameters and modulate neural dynamics, yielding a hybrid-automaton mechanism with a sticky win-stay, lose-switch policy.Figure 1 distinguishes explicit tasks, where external stimuli specify rules, from implicit tasks, where rules must be inferred from action-outcome histories.Source: C. Fast-slow neural-astrocyte network… · p. 2, Figure 1 · p. 2

II. MODEL FORMULATION

The model separates trial-wise astrocytic updates from fast within-trial neuronal dynamics. Astrocytic context signals modulate the neuronal network’s effective connectivity, with a low-rank structure intended to represent localized astrocytic tiling.

A. Overview

  • At each trial onset, the slow A-RNN updates from the previous trial’s action and reward, then holds its astrocytic action constant during the fast N-RNN evolution.The two subnetworks operate at separated timescales, with τh ≫ τx.Source: A. Overview · p. 2
  • The task output is linearly decoded from the N-RNN state.The equations are introduced as coupled A-RNN and N-RNN dynamics, followed by the output-decoding relation.Source: A. Overview · p. 2
  • The selected discrete action is represented as a one-hot vector πk+1, with its single active index matching action gk+1.The vector is the contextual signal emitted by the astrocytic network.Source: A. Overview · p. 2
  • The effective recurrent connectivity is defined using a Hadamard-product formulation, with the modulatory term depending on the active contextual state πk.The cited formulation is attributed to [23].Source: A. Overview · p. 2
  • The all-ones matrix has dimensions matching the neuronal recurrent matrix Jxx.This dimensional relation is stated for the effective-connectivity definition.Source: A. Overview · p. 2
  • The astrocytic modulation term is low-rank and is intended to capture localized gating of neuronal ensembles rather than uniform global control.The authors describe this as astrocytic tiling, with references,.Source: A. Overview · p. 2

B. Task and environment set-up

The task couples a delayed match-to-sample problem to a stochastic bandit over latent contexts. Because reward outcomes are noisy, a zero reward alone cannot establish whether the context changed.

B. Task and environment set-up

  • The N-RNN must retain a point-on-a-circle stimulus across a random delay and reach its corresponding target, with the correct mapping determined by one of four latent contexts.Successful low-level execution requires context inference, and the astrocytic network receives the resulting rewards.Source: B. Task and environment set-up · p. 2
  • Context remains active for a random block of 3–25 trials, after which the environment selects a new context uniformly from the available contexts.The block duration is sampled from L ∼ U(3, 25).Source: B. Task and environment set-up · p. 3
  • A zero reward is ambiguous because it can reflect either a context shift or probabilistic reward omission under the correct context.The correct context receives reward with probability p⋆, while any incorrect context receives it with probability (1 − p⋆)/(M − 1).Source: B. Task and environment set-up · p. 3
  • The astrocytic network acts as an M-arm bandit, selecting a context and using the ensuing stochastic reward to guide future emissions.The paper considers four reward-entropy regimes, from H = 0.627 to H = 1.838.Source: B. Task and environment set-up · p. 3

C. Model training

The model trains the neuronal and astrocytic subnetworks with different objectives suited to their roles. It autonomously infers context from action-reward history and delivers that context by changing neuronal connectivity rather than appending it to the task input.

C. Model training

  • Training decouples the subnetworks: supervised mean-squared-error learning trains the N-RNN to reach the target, while A2C+GAE trains the A-RNN from sparse one-bit rewards.The astrocytic policy processes action-outcome observations across multiple trials and yields a sticky win-stay/lose-switch scheme.Source: C. Model training · p. 3
  • Figure 2 depicts the N-RNN’s delayed match-to-sample trial, with stimulus presentation, a variable memory delay, and a response window.Its second panel represents the four-context mapping.Source: Figure 2 · p. 3
  • A curriculum temporarily raises correct-context rewards to 10 for 500 trials, then anneals the reward scale toward 1 over 250 trials.During warm-up, the A-RNN’s continuous state dynamics guide task execution and context inference.Source: C. Model training · p. 3

D. Relationship to Meta-RL and Hypernetworks

  • Unlike meta-RL agents that adapt their policy through interaction, these trained models embed a fixed policy that responds to environmental inputs.The paper focuses on how the hierarchy’s architectural constraints enable that policy to emerge.Source: D. Relationship to Meta-RL and… · p. 3
  • The A-RNN infers latent context shifts from its own action-reward history, without an oracle-provided context or task-boundary hidden-state resets.It evolves continuously across trial blocks, allowing long-term history to shape its state trajectory.Source: D. Relationship to Meta-RL and… · p. 3
  • Rather than concatenating inferred context onto task inputs, the A-RNN reconfigures the N-RNN’s connectivity on each trial.The paper describes this dynamic parameter selection as akin to hypernetworks [40].Source: D. Relationship to Meta-RL and… · p. 3

III. RESULTS

Context-switching latency depends on environmental entropy: agents switch after more unrewarded trials as entropy rises. Their action transitions retain a directional cycling structure, though noise and high entropy increase behavioral variability.

A. Networks learn an automaton-like sticky policy

  • As environmental entropy rises, context switches follow more unrewarded trials, with greater average stickiness and temporal variance; internal noise broadens switching-latency distributions.At low entropy, switching is almost deterministic.Source: Figure 3 · p. 4, A. Networks learn an automaton-like… · p. 4
  • The latency pattern fits an entropy-sensitive policy: low entropy favors switching after the first reward-prediction error, whereas high entropy favors waiting through successive errors.The analysis measures cumulative reward-prediction errors before an action change.Source: A. Networks learn an automaton-like… · p. 4
  • Across entropy and noise conditions, action switches retain a dominant directional cycling structure, though its probabilities fade under maximum entropy and heavy internal perturbation.The transition matrices summarize source-to-target action probabilities across 100 independently trained models.Source: Figure 4 · p. 4, A. Networks learn an automaton-like… · p. 4
  • Increasing internal noise raises both stickiness and cycling dispersion, while structural variation persists even without internal noise, particularly at elevated entropy.The noisefree variation indicates that internal noise is not the only source of behavioral variability.Source: A. Networks learn an automaton-like… · p. 5

B. Rule switches are mediated through action-reward bifurcations

The network’s action-reward pair changes its dynamics across action-level sets, which are defined by the decoded action. Rewarded and unrewarded outcomes lead trajectories toward different attractors, with unrewarded outcomes eventually prompting an action switch.

B. Rule switches are mediated through action-reward bifurcations

  • Action and reward act as bifurcation parameters, altering the vector field so each pair is associated with a distinct limit-cycle attractor within an action-level set.The action-level sets themselves are independent of the emitted action or derived reward.Source: B. Rule switches are mediated through… · p. 5

1) Dynamics and contextual inference:

  • When an emitted action is rewarded, trajectories converge to a rewarded attractor; when unrewarded, they eventually approach an attractor in another action-level set and switch actions.The illustrated unrewarded traversal takes three steps, while its duration varies with environmental entropy.Source: 1) Dynamics and contextual inference · p. 5

2) Vector field geometry and stickiness:

Reward omission reorganizes the vector field, and entropy shapes how quickly trajectories leave the current action level set. Lattice-coupled astrocytes retain performance comparable to dense connectivity across a broad sparsity range.

2) Vector field geometry and stickiness:

C. Astrocytic tiling and lattice topology are sufficient for network-wide modulation

  • Lattice-coupled models train at a comparable rate and reach similar asymptotic regret to fully connected models, with average reward remaining high up to 90% sparsity.The comparison is shown in Fig. 10; connectivity that becomes too sparse limits reliable network communication.Source: Figure 10 · p. 7, C. Astrocytic tiling and lattice… · p. 7
  • The model assumes non-overlapping tiled astrocyte domains, with a 1:3 astrocyte-to-neuron ratio, and tests whether local coupling constrains its dynamics.Each astrocyte interacts only with its k nearest neighbors in the lattice topology.Source: C. Astrocytic tiling and lattice… · p. 7

D. Hierarchical timescale separation advantages trainability

The paper tests whether slow astrocytic timescales aid trainability relative to conventional recurrent networks. The monolithic network struggles to learn the task, while the proposed hierarchy separates slow evidence integration from fast execution.

D. Hierarchical timescale separation advantages trainability

  • The comparison includes a supervised RNN given ground-truth context as an oracle ceiling and a same-size policy-gradient RNN with task and context readouts.Source: D. Hierarchical timescale separation… · p. 8
  • The monolithic RNN struggles to learn the task, whereas the neural-astrocyte model explicitly decouples slow context evidence integration from fast task execution.The neural-astrocyte model is, in principle, a special case of the monolithic architecture.Source: D. Hierarchical timescale separation… · p. 8
  • The proposed architecture assigns continuous evidence integration to a slow A-RNN, which provides a stable modulatory signal to the fast N-RNN.The authors attribute the monolithic model's difficulty to the challenge of preserving slow reward histories while generating fast task dynamics.Source: D. Hierarchical timescale separation… · p. 8

IV. CONCLUSIONS

  • The paper frames slow astrocytic integration and rapid neuronal execution as an integrative, flexible computational substrate for contextual switching.Source: §IV · p. 8

A. Bifurcations and a hybrid-automaton mechanism

Reward prediction errors drive bifurcations that accumulate evidence of contextual incongruence and switch neural dynamics to a new context. The astrocytic network also retains broad modulation under substantial connection sparsity, while evaluation remains within one parameterized task family.

A. Bifurcations and a hybrid-automaton mechanism

  • A context switch requires two bifurcations: reward omission annihilates the active attractor, then crossing into a new decision region reconfigures the vector field around that context.Under reward, the selected context has a stable attracting limit cycle; omission instead yields low-velocity dynamics.Source: A. Bifurcations and a hybrid-automaton… · p. 8
  • Astrocytic dynamics integrate successive reward prediction errors until a level-set traversal switches neural dynamics, separating latent-state inference from immediate action execution.The mechanism generalizes integrate-to-bound models by changing the location of multidimensional bounds in state space.Source: A. Bifurcations and a hybrid-automaton… · p. 8

B. Topological robustness of the astrocytic network

  • Astrocytic modulation remains efficacious at approximately 90% connection sparsity, suggesting tolerance to substantial heterogeneity and local connection loss.The modeled local, diffusive communication and spatial tiling are proposed as an efficient way to achieve broad neuronal modulation.Source: B. Topological robustness of the… · p. 8

C. Limitations and Future Directions

  • The context-invariant block lengths are sampled from ∼U(3, 25) trials, and evaluation uses task instances drawn from the same parameterized family used for optimization.The slow A-RNN infers latent rule shifts within this task space; generalization across structurally distinct regimes may require alternative meta-learning procedures.Source: C. Limitations and Future Directions · p. 8

APPENDIX

The appendix describes a spatial working-memory task with four contextual input-output mappings, along with model initialization and training procedures. Training combines supervised learning for the fast network with reinforcement learning and a staged reward curriculum for the slow network.

A. Task setup

  • Trials present a point on the unit circle, require retention through a variable delay, and cue a response after fixation and target presentation.The stimulus is a 3D vector: two components encode location and the third encodes the response trigger.Source: A. Task setup · p. 10
  • Four contexts map the target to the desired output through identity, a 180° shift, a nonlinear function, or a piecewise quadrant projection.The nonlinear mapping is f(x, y) = (x log(1 + y), y log(1 + x)).Source: A. Task setup · p. 10

B. Parameter Initialization Specifications

C. Training Paradigm and Curriculum Schedule

  • The N-RNN is trained by supervised learning and the A-RNN by trial-based reinforcement learning, with a curriculum used to stabilize value estimates and accelerate convergence.The schedule augments correct-choice rewards, anneals them, then removes augmentation for sparse feedback.Source: C. Training Paradigm and Curriculum… · pp. 10–11
  • After augmentation and annealing, the A-RNN is evaluated using unaugmented sparse feedback with rewards in {0, 1}.The three phases span trials 1–500, 501–750, and 751 onward, respectively.Source: C. Training Paradigm and Curriculum… · pp. 10–11

D. A-RNN Recurrent Mask and Sparse Topology

The A-RNN uses a fixed binary mask to constrain recurrent weights to a spatially local, sparse topology. Only selected pathways receive gradient updates during training.

D. A-RNN Recurrent Mask and Sparse Topology

  • A static binary mask constrains A-RNN recurrent connections to local nearest-neighbor pathways, with a selected fraction defining the active edge set.The topology remains fixed across training and evaluation, and gradients update only active pathways.Source: D. A-RNN Recurrent Mask and Sparse… · p. 11
Loading 2609.16217v1…