PaperlayerView exact revision on arXiv

Source-linked AI summary

Intrinsic Goals for Autonomous Agents: Hierarchical Exploration Captures Future Volitional Action in Drosophila

Declan Kutscher, Reece Keller, Alyn Kirsch, Felix Pei, Aran Nayebi

arXiv:2610.34638v1q-bio.NCcs.AIcs.LGcs.RO

TL;DR

The paper asks how intrinsic motivation can generate animal-like autonomy and tests 20 intrinsic objectives in embodied fly agents against fly behavior and neural activity. DIAYN provides the strongest joint account of the signatures examined, including prospective activity and variation in turning strategies.

  • Problem

    Intrinsic objectives are commonly assessed using artificial-agent criteria, leaving uncertain whether they reproduce biological behavior and neural representations of autonomous action.

    Source: §1 · p. 1

  • Method

    The authors train embodied fly agents with 20 intrinsic objectives over hierarchical motor primitives and compare frozen policies with flies in a virtual tethered-ball assay.

    Source: §1 · p. 3, §3.2 · p. 7

  • Results

    DIAYN provides the strongest joint account of the behavioral and neural signatures examined, including individual turning differences and prospective activity.

    Source: §1 · p. 3, §5 · p. 9, §6 · p. 10

  • Takeaways & Limitations

    Behavioral diversity and prospective internal activity may reflect a shared computational objective, so both are useful constraints when evaluating accounts of autonomy.

    Source: §5 · p. 9, §6 · p. 10

    The predefined locomotor primitives and low-level controllers constrain learned behavior primarily to 2-D locomotion.

    Source: LIMITATIONS AND FUTURE WORK · p. 10

Abstract

from arXiv · show

Animals have a remarkable capacity for autonomy---driven by intrinsic motivation---without external sensory cues or extrinsic objectives, yet the computational mechanisms behind this ability largely remain a mystery. We combine whole-brain recordings in the fruit fly (Drosophila melanogaster), which reveal neural representations of spontaneous exploratory decisions more than 10 seconds in advance, with a neuromechanical digital twin to test whether optimizing intrinsic goals yields realistically embodied artificial agents that quantitatively match fly behavior and internal representations of future volitional action during exploration. Our hierarchical policy selects from a discrete library of low-level motor primitives, consistent with ethological theories of behavioral syllables. We train agents without external reward using 20 intrinsic motivation algorithms spanning curiosity, entropy, and competency methods. The agents are evaluated in a virtual tethered-ball assay modeled on the fly experiment, comparing spontaneous locomotor statistics and internal representations and decoding future turn direction from kinematics and neural activity on the same timescales observed in flies. DIAYN, a hierarchical competency algorithm learning discrete policies that maximize behavioral entropy and the separability of induced sensory observations, achieves the strongest joint behavioral and neural prospective coding performance relative to flies. Its learned policies closely capture individual variability in fly exploration strategies, suggesting that this diversity serves to maximize state-space coverage at the population level. DIAYN's latent activity also closely captures the sum-difference rule for interhemispheric encoding of kinematics in biological flies. These results establish hierarchical, intrinsically motivated agents as a powerful framework for reverse-engineering animal autonomy.

1 INTRODUCTION

The paper asks whether intrinsic objectives can explain animal autonomy and produce agents whose spontaneous behavior and internal representations resemble those of flies. It tests this with embodied agents, hierarchical motor primitives, and comparisons to fly behavior and neural recordings.

1 INTRODUCTION

  • Intrinsic objectives are usually evaluated on artificial-agent criteria, leaving open whether they reproduce biological behavior or internal representations.The authors argue that animal behavior and neural activity provide an independent constraint on candidate objectives.Source: §1 · p. 2
  • Fly recordings show that IPS activity predicts future turn direction 15 to 8 seconds before a turn, providing a neural benchmark for models.The paper uses both behavioral dynamics and future-action representations as comparison targets.Source: §1 · p. 2
  • A hierarchical policy selects behavioral primitives implemented by low-level controllers, allowing intrinsic objectives to operate over meaningful actions rather than joint commands.The design is motivated by behavioral syllables organized across timescales.Source: §1 · pp. 2–3
  • The study trains embodied fly agents with 20 intrinsic-drive methods and evaluates their behavior and internal activity against flies in a virtual tethered-ball assay.All methods share embodiment, architecture, training environment, and budget; evaluation tests future-turn prediction and kinematic encoding.Source: §1 · p. 3
  • DIAYN provides the strongest joint account among tested objectives, matching multiple behavioral and neural signatures while exposing limits of average statistics alone.The authors report that objectives matching mean turning bias or predictable behavior can still miss prospective internal signals.Source: §1 · p. 3

2 BACKGROUND AND RELATED WORK

The background motivates modeling spontaneous behavior at multiple timescales and evaluating intrinsic objectives against both behavioral and neural signatures. It situates the approach among curiosity, entropy, and competency methods and embodied computational models.

2.1 INTRINSICALLY GENERATED AND HIERARCHICALLY STRUCTURED ANIMAL BEHAVIOR

  • Behavioral syllables form reusable movement patterns organized across timescales, while fly recordings provide behavioral and neural signatures of future action.IPS activity predicts turn direction about 15 to 8 seconds before turns, motivating hierarchical policies over higher-level locomotor primitives.Source: §2.1 · pp. 3–4

2.2 COMPUTATIONAL THEORIES OF INTRINSIC MOTIVATION

  • Intrinsic-motivation methods are grouped as curiosity, entropy, or competency objectives, respectively emphasizing prediction, state-distribution coverage, or controllability and skill identifiability.The taxonomy is presented as an organizational device, not a strict division of mutually exclusive principles.Source: §2.2 · pp. 4–5

2.3 ARTIFICIAL NEURAL NETWORKS AS HYPOTHESES OF BIOLOGICAL COMPUTATION

  • The paper extends embodied computational modeling by training agents through intrinsic objectives and expanding hierarchical control beyond two coarse behavioral states to a larger primitive repertoire.The agents are compared with biological behavior and neural dynamics at test time.Source: §2.3 · pp. 5–6

3 METHODS

The methods combine a biomechanical fly model with hierarchical primitive control and intrinsic-reward training. Frozen policies are then tested in a virtual tethered-ball assay matched to the biological experiment.

3.1 HIERARCHICAL AGENT ARCHITECTURE AND BIOMECHANICAL MODEL

  • The embodied agent receives visual, olfactory, and proprioceptive inputs and selects among seven behavioral primitives every 100 ms, with fixed low-level controllers executing them.Locomotor primitives drive a CPG, while grooming uses replayed kinematics; the architecture and joint-level mechanics are shared across objectives.Source: §3.1 · p. 6

3.2 INTRINSIC DRIVE METHODS AND TRAINING

  • Twenty intrinsic-drive policies are trained with intrinsic reward alone under matched architecture, environment, hyperparameters, and a 50-million-decision budget.DIAYN maximizes skill-state distinguishability and action entropy within each skill, with actions corresponding to behavioral primitives.Source: §3.2 · p. 7

3.3 TETHERED-BALL EVALUATION

  • Frozen policies undergo nine 30-minute rollouts in a virtual tethered-ball assay, with vision and olfaction set to zero to match the biological experiment.Skills are sampled at episode onset and resampled at their method-specific horizon; recurrent hidden state is retained during evaluation.Source: §3.3 · p. 8

4 RESULTS

The paper compares future-turn prediction from agent activity and behavior with flies, then assesses how closely intrinsic objectives reproduce fly-like behavioral diversity and neural organization. DIAYN best matches flies jointly on neural and behavioral predictability, while its skills span fly-like turning variability and its internal activity recovers a related sum-difference structure.

Neural and Behavioral Decoding

  • Future turns were decoded from internal activity in the −15 to −8 s window using elastic-net logistic regression with five-fold stratified cross-validation.Fly-tuned decoder hyperparameters were fixed for agents; behavior-only decoding used mean angular and forward velocity, and PCA tested sum-difference organization.Source: §4 · p. 8
  • Fly IPS neural prospective coding was 66.5 ± 1.4%; DIAYN reached 82.1 ± 0.9% and NGU 65.4 ± 1.8%.The decoder predicts turn direction from neural activity in the −15 to −8 s before-turn window.Source: §4 · p. 8
  • Behavior-only prospective coding was 60.5 ± 0.9% for flies, versus 92.2 ± 5.9% for MAX, 74.7 ± 1.4% for DIAYN, 72.4 ± 1.2% for γ-progress, and 66.3 ± 9.4% for NGU.Nine of 20 methods produced no turns or turns in only one direction, preventing stratified-fold accuracy calculation and mismatching fly behavior.Source: §4 · p. 8

Performance Ratio

  • DIAYN and NGU alone match or exceed flies on both neural and behavioral predictability, and only DIAYN matches the fly neural-to-behavioral predictability ratio.The comparison distinguishes internally predictive representations from behavior that is simply predictable from past behavior.Source: §4 · p. 8

Turning Bias

  • DIAYN’s learned skills span a wider range of turning biases and more fly-like trajectory coverage than the narrower CSF and Random-DIAYN distributions.CSF and Random-DIAYN used the same one-skill-per-rollout schedule, indicating the wider spread was associated with the learned objective rather than sampling schedule.Source: §4 · p. 9

Internal Representations

  • DIAYN’s component difference correlated with angular velocity (r = 0.153) and sum with forward velocity (r = 0.413), near fly IPS values of r = 0.191 and 0.429.This sum-difference organization was absent in Random-DIAYN, whose best pair reached r = 0.021 and 0.113; Figure 5 depicts the comparison.Source: §4 · p. 9, Figure 5 · p. 9

5 DISCUSSION

The discussion proposes that behavioral diversity and prospective internal activity may reflect a shared computational objective. In this hierarchical implementation, DIAYN organizes selection among seven motor primitives, and its persistent skills provide a candidate account of both population-level diversity and future-action representations.

5 DISCUSSION

  • Behavioral diversity and prospective internal activity may reflect a shared computational objective, which DIAYN’s joint correspondence motivates as an account of autonomy.Matching average turning bias or producing highly predictable behavior alone need not reproduce predictive neural representation.Source: §5 · p. 9
  • DIAYN selects among seven motor primitives, rewarding skills that induce distinguishable sensory states while encouraging high-entropy action selection within each skill.Its objective organizes behavior above the level of individual motor commands.Source: §5 · p. 9
  • DIAYN’s learned skill repertoire captures wide individual variation in fly exploration strategies; the authors propose that population diversity may maximize state-space coverage.They hypothesize that relatively fixed individual strategies distribute the strategies needed for survival across individuals.Source: §5 · p. 10
  • Trajectory-level optimization may sustain skill-dependent motor choices that preserve sensory-state distinctions, allowing recurrent activity to encode an exploratory strategy and predict later turns.The proposed account links behavioral diversity and future-action representations under one intrinsic objective.Source: §5 · p. 10

6 CONCLUSION

The framework combines an embodied digital twin, hierarchical behavioral primitives, and joint comparison with fly behavior and neural recordings. Its scope is bounded by predefined locomotor primitives and a focus on IPS-related activity, with broader behavior and whole-brain dynamics left for future work.

6 CONCLUSION

  • Across 20 intrinsic objectives, DIAYN best accounts jointly for individual turn differences, prospective neural activity, and interhemispheric sum-difference encoding of kinematics.These properties emerge from an intrinsic goal without external rewards or training on fly data.Source: §6 · p. 10
  • Predefined locomotor primitives and low-level controllers constrain learned behavior primarily to 2-D locomotion.The authors suggest learning primitives jointly with high-level selection or extending them to flight.Source: LIMITATIONS AND FUTURE WORK · p. 10
  • A whole-brain account of locomotion remains future work; the study focuses on IPS and uses a simple training environment to reproduce key neural responses.The authors note that whole-brain modeling would require an ethologically relevant environment with variable terrain, manipulation opportunities, and resources.Source: LIMITATIONS AND FUTURE WORK · p. 10

A.1 SENSORY OBSERVATIONS

  • The proprioceptive state includes 45 joint angles and velocities, body-root height, body-frame gravity, and root linear and angular velocities.The 45 degrees of freedom comprise seven per leg and three neck joints.Source: §A.1 · p. 18
  • Vision is supplied as a 2 × 2 × 32 × 32 tensor formed by resampling the eye inputs; 623 of 721 ommatidia are represented.The eye cameras have a 157° field of view, and visual input is not normalized.Source: §A.1 · p. 18

A.2 HIERARCHICAL LOCOMOTOR CONTROL

  • Every 0.1 seconds, the policy selects one of seven primitives; six control left- and right-leg descending signals, while the seventh triggers grooming replay.Locomotor primitives drive a six-oscillator CPG; grooming replays a recorded clip while holding the body root fixed.Source: §A.2 · pp. 18–19

A.3 POLICY ARCHITECTURE

The policy combines multimodal features with recurrent actor and critic LSTMs, then selects among seven behavioral primitives. Skill-conditioned methods append the skill to the LSTM inputs, while the intrinsic drive module receives features without the skill.

A.3 POLICY ARCHITECTURE

  • A shared feature extractor produces a 310-dimensional base vector that feeds separate 256-unit actor and critic LSTMs and the intrinsic drive module.Visual, odor, and proprioceptive representations contribute 128, 18, and 164 dimensions, respectively.Source: §A.3 · p. 19
  • The actor outputs a categorical distribution over seven primitives after separate MLP heads follow the actor and critic LSTMs.The visual CNN shares weights across the two eyes; separate MLPs embed odor and proprioception.Source: §A.3 · p. 19
  • For skill-conditioned methods, the skill is appended to both LSTM inputs, but the IDM receives only the base features to prevent trivial skill-discriminator solutions.A skill of dimension dz adds 2,048 dz parameters to the 1,407,576-parameter policy without a skill input.Source: §A.3 · p. 19

A.4 INTRINSIC DRIVE METHODS

The appendix describes a shared intrinsic-drive pipeline and implementations spanning curiosity, entropy, and competency objectives. Methods differ in the signals they reward, including prediction error, novelty, empowerment, and skill identifiability.

A.4 INTRINSIC DRIVE METHODS

  • All 20 methods share a pipeline in which the IDM receives policy features without gradient flow into the feature extractor, and intrinsic reward alone supplies PPO’s reward.IDM parameters update after each rollout; rewards are computed online at each decision using parameters from the previous rollout.Source: §A.4 · pp. 19–20
  • Curiosity methods reward prediction error, prediction-error reduction, novelty, or model disagreement, with implementations including ICM, RND, NGU, γ-progress, and MAX.The listed implementations also include RE3 and APT; the appendix describes modifications from original publications where identified.Source: CURIOSITY · pp. 20–21
  • Entropy and competency methods use learned representations or skills, including neighbor-based novelty, contrastive objectives, empowerment variants, and skill-conditioned rewards.DIAYN rewards skill identifiability from features; potential empowerment rewards estimated channel capacity across the seven actions.Source: CURIOSITY · p. 21, ENTROPY · p. 21
  • DIAYN samples one-hot skills uniformly once per episode and rewards a discriminator’s ability to infer the skill from the agent’s features.The implementation uses 50 skills; its only action-entropy term is PPO’s entropy bonus.Source: COMPETENCY · p. 21
  • Other competency methods vary skill duration and inference: VIC and VALOR infer skills from segments, while DADS and rDADS use skill-conditioned transition models.CSF rewards feature changes aligned with sampled skills, and empowerment variants estimate action–state information or capacity.Source: COMPETENCY · pp. 21–22

A.5 TRAINING PROTOCOL

Policies train in procedurally randomized FlyGym arenas and are evaluated under a common decision budget. Each method is trained once, and recorded compute times are lower bounds on full job time.

A.5 TRAINING PROTOCOL

  • Training environments vary visual, olfactory, and proprioceptive inputs using three odor sources and 12 to 50 obstacles, with episodes lasting 5 seconds.Episodes may terminate early for physics errors, non-finite states, falls, or leaving the arena bounds.Source: §A.5 · pp. 22–23
  • Each method receives 50 million policy decisions across 2,048 parallel environments, using intrinsic reward alone and a single training seed.The end-of-budget checkpoint is evaluated without selecting checkpoints by evaluation metrics.Source: §A.5 · p. 23
  • Recorded training time ranges from 108 to 175 hours per method and is a lower bound on full job time.Runs used one GPU and 64 CPU cores; throughput varied from roughly 80 to 129 policy decisions per second.Source: §A.5 · p. 23, Table 4 · p. 23

A.6 TETHERED-BALL EVALUATION

The evaluation recreates the fly’s tethered-ball assay in simulation and measures locomotor behavior and agent representations under matched rollout conditions. Controls include random-initialized policies and an odor-seeking extrinsic-reward policy.

A.6 TETHERED-BALL EVALUATION

  • The simulated assay fixes the fly’s body root above a 9 mm ball and reads forward and angular velocities from the ball.Only the tibiae and tarsi contact the ball; visual and olfactory inputs are set to zero to match darkness and control for odor.Source: §A.6 · p. 23
  • Each policy runs nine sampled 30-minute rollouts, with skill resampling schedules varying by method.DIAYN and CSF hold one sampled skill for each rollout; VIC and VALOR resample every 5 seconds, while DADS and rDADS resample every 10 seconds.Source: §A.6 · pp. 23–24
  • Neural activations are recorded each decision and averaged into consecutive 0.532-second bins to match the fly imaging rate.Recorded signals include the input embedding, actor and critic LSTM states, and policy and value MLP representations.Source: §A.6 · p. 24

A.7 CONTROLS

  • Random-initialized controls retain the corresponding skill input when applicable and are evaluated like trained policies.The random-nonskill control comes from a method without a skill input.Source: §A.7 · p. 24
  • The extrinsic-reward baseline uses the same architecture and PPO settings but is trained on an odor-seeking task.Its final checkpoint was evaluated after 10 million decision steps, when task success rate had saturated.Source: §A.7 · p. 24

A.8 BIOLOGICAL DATA

The biological comparison uses whole-brain recordings from nine flies and defines turn events from ball yaw rate. Behavioral and neural features are then used to decode turn direction in before-turn and during-turn windows.

A.8 BIOLOGICAL DATA

  • The biological dataset contains processed GCaMP6f recordings from nine flies, with brain regions defined using the JFRC2010 template warped to JRC2018F.The public release lacks the identified pre-motor supercluster, so the analysis uses the anatomically defined whole IPS in each hemisphere.Source: §A.8 · pp. 24–25

A.9 TURN DETECTION AND BEHAVIORAL STATISTICS

  • Turns are local maxima of smoothed absolute yaw rate above 200°/s, and turning bias is log(NR/NL) when both turn directions occur.The dataset yields 3,786 turns (1,362 left, 2,424 right); the stationary subset contains 27 left and 27 right turns.Source: §A.9 · p. 25, §A.8 · p. 25

A.10 DECODING

  • Decoding uses features averaged from −15 to −8 seconds before turns or −0.4 to +0.4 seconds during turns.Agent features include candidate internal network layers and raw proprioception; fly features concatenate left- and right-IPS voxel activity, and behavioral features are mean angular and forward velocity.Source: §A.10 · p. 25
  • A logistic-regression decoder uses nested cross-validation on fly IPS data to select shared hyperparameters, then stratified cross-validation to estimate balanced accuracy.The selected configuration is C = 0.1 and ℓ1 ratio 0.75; the candidate agent layer with the highest before-turn prospective coding is also used for the during-turn window.Source: §A.10 · pp. 25–26

A.11 REPRESENTATIONAL ANALYSIS

The representational analysis asks whether pairs of model activity components relate to behavior like the fly’s bilateral signal, while explicitly treating this as a functional rather than anatomical comparison.

A.11 REPRESENTATIONAL ANALYSIS

  • Because the policy lacks bilaterally matched anatomical superclusters, its activity cannot be compared through the fly’s spatial pairing analysis.Instead, the analysis compares functional relationships between model activity axes and behavior, not anatomical correspondence.Source: §A.11 · p. 26
  • A 10-component PCA is fit to actor-LSTM activity pooled across nine DIAYN or Random-DIAYN rollouts, with components scaled to unit variance.Angular and forward velocities are interpolated to activation timestamps for the behavioral comparison.Source: §A.11 · p. 26
  • The selected component pairs are PCs 2 and 8 for DIAYN and PCs 5 and 10 for Random-DIAYN.Pairs are selected by maximizing the functional score and oriented so their difference increases with angular velocity.Source: §A.11 · p. 26

B EXTENDED RESULTS

The extended results section points readers to per-method turn statistics and prospective-coding results, alongside a layer-by-layer breakdown of early-window decoding.

B EXTENDED RESULTS

  • Table 7 reports turn statistics and prospective coding for all 20 intrinsic-drive methods, controls, and flies.The caption states that accuracy is balanced accuracy and notes that some turning biases are undefined when rollouts lack turns in both directions.Source: B EXTENDED RESULTS · p. 26, Table 7 · p. 27
  • Table 8 reports early-window balanced decoding accuracy for candidate network feature sets and a raw-observation control.Its caption distinguishes the selected internal representation from raw proprioceptive observations and marks cases where the raw-observation control scores higher.Source: B EXTENDED RESULTS · p. 26, Table 8 · p. 27

Related papers

Loading 2610.34638v1…