Source-linked AI summary
Free-Energy-Gated Plasticity for Real-Time Online Motor Learning in Physical Human-Robot Interaction
Hiroki Sawada, Jun Tani
TL;DR
Fully online embodied learning must acquire new behaviors while preserving previously learned dynamics during ongoing interaction. The paper extends PV-RNN with FEGP, which regulates synaptic plasticity using variational free energy. In physical human-robot interaction, FEGP supported acquisition of three motor patterns and improved repertoire coverage and retention, with benefits depending on the timing of plasticity relative to model-environment mismatch.
Problem
Fully online embodied learning requires synaptic adaptation without offline training, replay, task-boundary signals, or interference with previously learned dynamics.
Method
The paper extends PV-RNN with Free-Energy-Gated Plasticity, scaling synaptic updates as a sigmoidal function of variational free energy.
Results
FEGP enabled three motor patterns to be acquired from scratch and substantially improved repertoire coverage and retention; matched-rate and temporally disrupted controls did not reproduce these improvements.
Takeaways & Limitations
The results indicate that when plasticity is expressed relative to model-environment mismatch matters more than its average magnitude or gain-value distribution.
Takeaways & Limitations
Experiments covered only three periodic movement patterns, and larger repertoires, richer behaviors, and longer interactions remain to be tested.
Abstract
from arXiv · showhide
Fully online embodied learning requires synaptic adaptation to acquire new behaviors while preserving previously learned dynamics during ongoing interaction. We extend the Predictive-Coding-inspired Variational Recurrent Neural Network (PV-RNN) to continuously adapt its synaptic weights and propose Free-Energy-Gated Plasticity (FEGP), which regulates the effective learning rate according to variational free energy. In real-time physical human-robot interaction, a randomly initialized network acquired three cyclic motor patterns without offline pretraining, replay, or task-boundary signals, with all three patterns emerging in autonomous rollouts. Controlled experiments over ten randomized teaching streams and five network initializations per stream showed that FEGP substantially improved repertoire coverage and retention of previously acquired patterns after they left the recent observation window. Neither a constant learning rate matched to the gate's time-averaged effective rate nor replay of the same gain values with disrupted temporal organization reproduced these improvements. These results indicate that the temporal allocation of plasticity relative to model-environment mismatch, rather than simply its average magnitude or distribution, is critical for maintaining previously acquired behaviors during continued online learning.
1 Introduction
Fully online embodied learning requires regulating synaptic adaptation so new motor dynamics can be acquired without interfering with previously learned behaviors. The paper proposes FEGP, which uses variational free energy to regulate plasticity during continuous physical interaction.
- Setting and challenge: Fully online learning updates weights at every sensory sample without offline training, task boundaries, or separate inference and learning phases.Each update must finish before the next sensory sample is processed.
- Setting and challenge: The central challenge is balancing acquisition of new behavioral dynamics with retention of previously learned dynamics during ongoing interaction.Embodied weight changes alter subsequent actions and observations, coupling learning and behavior.
- Setting and challenge: Recent-observation windows can bias continuous adaptation toward recent behavior, causing interference with previously acquired dynamics.The model cannot optimize over the entire interaction history within each real-time update interval.
- Proposed mechanism: FEGP scales synaptic updates as a sigmoidal function of variational free energy, increasing plasticity under model-environment mismatch and suppressing it as predictions improve.The gate uses a signal already computed within predictive-coding inference.
- Contribution: The study evaluates fully online embodied learning without task boundaries, replay, or offline consolidation, and reports improved repertoire coverage and retention with FEGP.Controlled comparisons attribute the effect to the temporal allocation of plasticity relative to model-environment mismatch.
2 Method
The method extends PV-RNN into a fully online loop in which inference, synaptic adaptation, generation, and physical interaction recur together. FEGP regulates synaptic plasticity from variational free energy, increasing adaptation under mismatch and suppressing it when recent dynamics are well explained.
- Fully Online Learning: The model removes offline pretraining: random weights support repeated online posterior inference, synaptic adaptation, and future generation during interaction.The finite sliding window contains only recent history, not completed demonstrations or long-term replay memory.
- Fully Online Learning: Each update incorporates the newest observation, optimizes posterior states, adapts synaptic parameters, and generates a future trajectory for motor control.Saved trajectories are segmented and classified only after interaction; evaluation provides no feedback to online learning.
- Naive Online Weight Adaptation: Naive online adaptation applies the same base learning rate to gradients from the recent window, allowing new patterns to interfere with earlier dynamics.It lacks a state-dependent mechanism that reduces plasticity once recent observations become predictable.
- Free-Energy-Gated Plasticity: FEGP scales the effective learning rate with a plasticity gain driven by variational free energy, treating free energy as an internally generated model-environment mismatch signal.Plasticity increases when mismatch is large and is suppressed as existing dynamics better explain recent interaction.
- Free-Energy-Gated Plasticity: The gate uses the logarithm of window-averaged free energy, combining prediction divergence with a meta-prior-weighted complexity term.The accuracy term averages prediction divergence across the current window, 14 sensory dimensions, and 10 softmax units per dimension.
- Free-Energy-Gated Plasticity: A sigmoid threshold gate approaches the base learning rate above threshold and progressively suppresses adaptation below it; hysteresis uses separate thresholds to reduce rapid switching.The single-threshold formulation is recovered when λlow = λhigh = λ.
3 Experiments
Two experiments evaluated online synaptic adaptation with randomly initialized PV-RNNs: controlled prerecorded teaching streams and live physical human-robot interaction. Performance was assessed with a fixed trajectory evaluator measuring repertoire coverage, retention, shape distance, and unidentified segments.
- Experimental design: Two experiments tested online synaptic adaptation using prerecorded kinesthetic streams and real-time physical human-robot interaction.Both used randomly initialized PV-RNNs without offline pretraining.
- Experimental design: Ten cycles were recorded for each of three periodic two-arm movement patterns, A, B, and C.The recorded cycles supported the prerecorded teaching streams used in Experiment 1.
- Experimental design: Experiment 1 used ten 6,000-update teaching streams that switched movement patterns probabilistically after completed cycles.A switch occurred with probability 0.2, with the next pattern selected uniformly from the two alternatives.
- Evaluation pipeline: The fixed evaluation pipeline segmented generated trajectories and classified cycles as A, B, C, or Unknown using random-forest boundaries and multivariate DTW.Classification used a Sakoe-Chiba band of 20 steps, with acceptance based on DTW distance and duration criteria.
- Evaluation pipeline: At τ = 0.90, the evaluator rejected 85.7% of control trajectories while rejecting none of the recorded cycles.Nearly all accepted controls were time-reversed trajectories, indicating limited sensitivity to movement direction.
- Evaluation metrics: Repertoire coverage measured the presence of all three instructed classes in 3,000-step rollouts, while retention measured class reappearance after 500 updates absent from the observation window.Shape distance quantified similarity to demonstrated paths, and unidentified-segment ratio quantified Unknown segments in Experiment 2.
- Controlled analysis: Experiment 1 compared FEGP with controls matching its average effective learning rate or gain-value distribution across 50 sessions per condition.The selected FEGP thresholds were (λlow, λhigh) = (−9, −7).
4 Results and Discussion
FEGP enabled online acquisition and retention of multiple motor patterns during continued interaction, with benefits tied to when plasticity was allocated relative to model-environment mismatch. The hysteretic gate supported autonomous emergence of all three patterns, while controls showed that average learning rate or gain distribution alone was insufficient.
- Operating Regime and Hysteretic Gate Dynamics: Hysteresis reduced average regime transitions from 718 to 54 and intermediate-gain updates from 19.3% to 4.7%.
- What Gating Contributes: Nested Controls: FEGP increased repertoire coverage from 18.9% to 45.9% and retention from 0.0% to 23.1% versus constant plasticity.
- What Gating Contributes: Nested Controls: The matched-rate control achieved only 12.5% coverage and 0.2% retention, so matching FEGP’s average effective learning rate did not reproduce its benefit.
- What Gating Contributes: Nested Controls: Permuting, reversing, or shifting FEGP gains reduced coverage to 21.3%, 11.4%, and 17.1%, respectively, with retention of 0.5%, 0.8%, and 2.2%.
- Retention of Patterns Absent from the Observation Window: FEGP preserved patterns after they left the observation window, whereas constant-rate controls lost them almost immediately.FEGP retention ranged from 23.6% to 32.7% across longer absence intervals, while constant plasticity was 0.0% thereafter.
5 Conclusion
The study demonstrates fully online acquisition of multiple motor patterns and finds that FEGP improves repertoire coverage and retention during continued learning. Its benefits depend on the temporal organization of plasticity relative to model-environment mismatch, not merely on average plasticity.
- Starting from random weights without offline pretraining or task boundaries, the PV-RNN acquired three instructed movement patterns during continuous physical interaction.All three patterns eventually appeared in autonomous generation.
- FEGP substantially improved both repertoire coverage and retention of previously acquired motor patterns.
- A constant learning rate matched to FEGP’s time-averaged effective rate produced substantially lower coverage and almost no retention.
- Permuting, time-reversing, or temporally shifting the same gain values failed to reproduce FEGP’s performance.These controls indicate that temporal organization, rather than gain magnitude or distribution alone, matters.
- FEGP preserved previously acquired patterns after they left the recent observation window, while constant-plasticity controls lost them almost immediately.Patterns remained available under FEGP after several hundred model updates of absence.
- The improvement reflected broader repertoire maintenance rather than generally more accurate individual generated movements.Constant plasticity reproduced demonstrated trajectories somewhat more closely, whereas FEGP maintained greater breadth and retention.
- The comparison with continuously accumulating Synaptic Intelligence was based on one network initialization and is therefore only a representative baseline.
- The experiments covered only three periodic patterns, and larger repertoires, richer behaviors, longer interactions, and broader parameter transfer remain untested.Thresholds and evaluation parameters were selected empirically, and DTW has limited sensitivity to movement direction.
S1 Validation of the Evaluation Pipeline
The evaluation pipeline was validated against known cycle boundaries, recorded patterns, control trajectories, and held-out demonstrations. Results support accurate segmentation and classification, while exposing limited sensitivity to reversed movement direction.
- Boundary detection: 459 ground-truth boundaries from four representative streams were used to evaluate boundary detection across matching tolerances and detection thresholds.
- Boundary detection: Removing concatenation discontinuities, smoothing joins, or time-warping cycles by ±10% left boundary detection unchanged.
- Segment classification: Given ground-truth boundaries, all 459 cycles were assigned to the correct pattern and none was rejected; end-to-end accuracy on detector-produced segments was 0.998.
- Segment classification: The shape term achieved an area under the ROC curve of 0.988, with positive median distance 0.359 and negative median distance 2.362.
- Segment classification: Time reversal remained the one control category the rule could not reliably reject because the pose representation is insensitive to movement direction.Duration filtering was retained because DTW can accept a correct path executed at implausible tempo.
- Demonstration-level hold-out: A demonstration-level hold-out used demonstrations 1–5 for fitting and demonstrations 6–10 for testing, avoiding overlap between reference construction and evaluation.
S2 Synaptic Intelligence Update Rule
The study uses a continuously accumulating Synaptic Intelligence variant because task boundaries are unavailable. Its update rules track parameter importance online and add a penalty relative to a fixed initialization vector.
- A continuously accumulating SI variant was used because the boundary-free setting provides no point for consolidating parameter importance.
- The importance estimate is recomputed after each synaptic update rather than consolidated only at task boundaries.
- The update equations use elementwise multiplication and division in the importance calculation.
- A single reference parameter vector θ* is fixed at initialization, and an importance-weighted penalty is added to the objective.
- All SI conditions used γ = 1 − 10^-6 and ξ = 10^-6.
S3 Additional Figures
The additional-figure section compares late-stage weight adaptation and free energy across values of λ for four representative teaching streams. Larger λ suppresses adaptation while increasing residual free energy.
- The sweep reports final-2,000-update average weight-update magnitude and late-stage log-scaled free energy for streams Seq1–4.
- Increasing λ progressively suppresses weight adaptation while residual free energy increases.