PaperlayerView exact revision on arXiv

Source-linked AI summary

tinyDSM: A Framework for Skill Modeling and Development for Resource-Constrained Millirobots

Markus D. Kobelrausch, Michael Miedler, Axel Jantsch

arXiv:2608.17596v1cs.ROcs.AI

TL;DR

Tiny millirobots must learn complex, robust behaviors despite severe hardware and resource constraints. tinyDSM combines intrinsic motivation, fitness assessment, and structured knowledge to support autonomous skill development, enabling a 36 cm^3 millirobot running in 9 kB to progress from atomic motions to complex geometric behaviors within 15 minutes.

  • Problem

    Tiny robots need to learn complex, robust behaviors despite constraints on size, sensors, actuators, and computation.

  • Method

    tinyDSM combines intrinsic motivation, fitness-based evaluation, curriculum learning, and hierarchical knowledge-graph reasoning for open-ended skill development under resource constraints.

  • Results

    A 36 cm^3 millirobot progressed from atomic motion patterns to complex geometric behaviors within 15 minutes using only 9 kB of memory.

  • Takeaways & Limitations

    The results support lifelong, self-directed skill development in resource-constrained millirobots, including adaptation to added weight and retention of previously acquired skills.

  • Takeaways & Limitations

    Future work is needed to evaluate richer sensing, higher-level planning, lightweight neural networks, and more complex scenarios.

Abstract

from arXiv · show

In this study, we investigate developmental mechanisms that enable small, resource-constrained systems such as cm-sized millirobots to autonomously explore, learn, and adapt their capabilities throughout their lifespan. Reinforcement learning algorithms guide the agent's skill acquisition and adaptation through the interplay of our proposed tinyDSM, which integrates intrinsic motivation and fitness-based assessment. We strive for minimal, hard-wired skills while encouraging the open-ended development of new skills. A key emphasis in our approach is to encode minimal a-priori general knowledge, which serves as a foundational starting point for the system as it further learns system-specific dependencies from the initial knowledge provided. Thus, by design, our approach attempts to cover very generic application domains. The methodology is based on (a) developmental mechanism with intrinsic motivation, and (b) a cognitive architecture (knowledge, reasoning, learning), while (c) utilizing minimal resources. It uses a hierarchical knowledge graph and kinematic reasoners to model and evaluate simple and advanced motion related skills. In our experiments, we use a resource-constrained millirobot with a volume of 36 cm^3 with a Raspberry Pi Pico 32-bit microcontroller (RP2040) that integrates all described features and capabilities except the camera system in 9 kB. Starting with learning the most elementary motor skills the millirobot autonomously progresses from simple linear and angular movements to complex geometric patterns within 15 minutes. To complement the physical experiments, we perform a simulation-based analysis that enables systematic comparisons across learning algorithms and intrinsic motivation parameters.

I. INTRODUCTION · II. BACKGROUND

tinyDSM targets lifelong skill development in resource-constrained millirobots by combining intrinsic motivation, cognitive architecture, and minimal prior knowledge. It builds on developmental robotics, structured knowledge modeling, intrinsic motivation, and TinyML while extending intrinsically motivated learning with novelty and difficulty factors.

  • I. INTRODUCTION: Tiny robot learning deploys machine learning on low-cost autonomous robots constrained by size, weight, area, power, sensors, actuators, and computation.These robots weigh under 500 g and can operate in small spaces.
  • I. INTRODUCTION: The work combines developmental robotics with resource-constrained tiny robot learning through a cognitive architecture limited to knowledge, reasoning, and learning.Its goal is development-oriented competency acquisition and open-ended learning of new skills and knowledge.
  • I. INTRODUCTION: tinyDSM starts from minimal general knowledge, allowing the robot to learn system-specific actuator effects and environmental dependencies rather than relying on extensive prior assumptions.The intended flexibility includes adaptation to inaccurate sensors and actuators, aging, and wear-out effects.
  • I. INTRODUCTION: Intrinsic motivation drives autonomous skill development, while a fitness reasoner evaluates learning progress and skill effectiveness to guide continued improvement and adaptation.The robot’s changing knowledge graph influences its pursuit of specific skills.
  • I. INTRODUCTION: The framework develops motion skills hierarchically, discovering angular and linear movement patterns before learning specific distances, angles, and more complex behaviors.The experiments examine patterns produced through intrinsic motivation without a predetermined goal.
  • I. INTRODUCTION: tinyDSM comprises a developmental mechanism with intrinsic motivation, a cognitive architecture for knowledge, reasoning, and learning, and minimal resource utilization.These three elements define the framework’s principal design features.
  • II. BACKGROUND: The background connects tinyDSM to developmental robotics, cognitive architectures, ontologies, intrinsic motivation, and resource-efficient machine learning for lifelong autonomous adaptation.Related systems include layered cognitive architectures and semantic knowledge models for reasoning about tasks.
  • II. BACKGROUND: Compared with prior intrinsically motivated and curriculum-based approaches, tinyDSM additionally incorporates novelty and difficulty factors into intrinsic motivation.The cited prior work selects goals through learning progress and self-organizes goal complexity.

III. DEVELOPMENTAL SKILL METHOD … 2) Fitness:

tinyDSM models robot skills through a sensor-actuator space, fitness assessment, and skill specifications linking desired sensor transformations to execution routines. Fitness compares observed quantities with targets and permissible deviations, producing a competence score that guides learning and determines when a skill is learned.

  • 1) SAS:: The sensor-actuator space SAS = (S, C) comprises sensor readings and actuator commands indexed by identifiers and time.Sensor readings map to interface value vectors, while actuator commands specify parameters passed to actuator interfaces.
  • 1) SAS:: For a two-motor robot, each motor command uses activation time, rotational force, and force duration.The commands are represented as cm1(t, f, τ) and cm2(t, f, τ).
  • 2) Fitness:: Fitness assesses skill quality using a skill-dependent set of quantities, including sensor readings or outputs of other skills.The framework permits fitness functions based on n quantities, with n ∈N and n > 0.
  • 2) Fitness:: Fitness compares a target vector with an observation vector within permissible deviations defined by a range vector.The error vector is calculated from the target, range, and observed quantities.
  • 2) Fitness:: The resulting scalar fitness value f lies in [0, 1] and represents the quality of an executed activity.The scalar value is used by learning routines to direct learning and track the agent’s competence.
  • 2) Fitness:: A skill is learned when f(.) ≥fthreshold, with the required threshold typically around 0.95.The threshold can vary across skills or conditions.
  • A. Definition of SAS, Fitness and Skills: A skill consists of a specification, a learning routine, and a state during execution.The skill specification relates the action space to the sensor space.
  • A. Definition of SAS, Fitness and Skills: The skill specification is a 2-tuple DS = (T, F) containing a desired sensor-reading transformation T and the fitness F.T(S0, t0, S′, t′) denotes the desired transformation on sensor readings.

3) Skill: · B. Knowledge Graph · C. Intrinsic Motivation

tinyDSM models skills as action sequences whose fitness is optimized by learning routines, while a lightweight knowledge graph supports dependency-aware skill development and intrinsic motivation drives autonomous, competence-based exploration. The framework is designed to extend across physical actions under severe memory and runtime constraints.

  • 3) Skill:: A learning routine modifies the action sequence to maximize fitness, with reinforcement learning offered as a typical example.The routine’s objective is fitness optimization.
  • 3) Skill:: A skill state comprises an action sequence A and current fitness f̄, with learning modifying A to maximize fitness.Initially, A may be empty; applying the skill executes A.
  • 3) Skill:: The skill formulation targets physical actions, including non-motion behaviors such as regulating sensed brightness from an LED actuator.The concept is intended to extend to all kinds of physical actions.
  • B. Knowledge Graph: The cognitive system is founded on a semantically featured knowledge graph that contextualizes components and dependencies while remaining flexibly extendable under limited resources.The representation is intentionally simple so the robot can address skills and problems as they arise.
  • B. Knowledge Graph: The generic knowledge graph infers required sensorimotor abilities for particular skills and higher-order skills that become available as the robot evolves.Semantic entities represent skills, sensors, actuators, and the sensorimotor system.
  • C. Intrinsic Motivation: Intrinsic motivation generates autonomous objectives because skills appear interesting through inner drives and curiosity rather than strict externally given sequencing.The approach most closely follows competence-based intrinsic-motivation models.
  • C. Intrinsic Motivation: Novelty encourages exploration of unfamiliar skills, while progress from fitness assessment promotes continued improvement and prevents stagnation.Together, these factors help ensure skills in the knowledge graph are not permanently ignored.
  • C. Intrinsic Motivation: Intrinsic motivation is implemented with minimal historical data to address the millirobot’s memory and runtime constraints.The formulation is detailed in Appendix A.

D. Developmental Process · IV. EXPERIMENTAL SETUP · A. SAS

tinyDSM enables resource-constrained millirobots to develop hierarchical motion skills from minimal general knowledge through intrinsic motivation, fitness assessment, and lightweight learning. The experimental implementation uses modular embedded software, a two-motor robot with pose sensing, and abstract sensor-actuator interfaces for generic method development.

  • D. Developmental Process: tinyDSM minimizes hard-wired skills while using minimal general knowledge as a foundation for open-ended skill development and learning system-specific dependencies.It is designed for low-energy microcontrollers with only a few hundred kilobytes of RAM.
  • D. Developmental Process: The agent queries its knowledge graph to generate a masterable Skill Pool MO, selects skills through intrinsic motivation, and uses sensor-based fitness reasoning to guide learning.An incomplete sensor-actuator set can leave the skill pool empty, such as when the millirobot has no motors.
  • D. Developmental Process: Learned skills can unlock more complex skills, knowledge-graph extensions can reveal new skills, and physical modifications can expand the developmental space.The curriculum factor directs the agent toward valuable search spaces to improve learning efficiency and accelerate progress.
  • D. Developmental Process: The skill optimizer uses small models and a minimalist reward design to optimize selected skill policies under strict resource constraints.This addresses the difficulty of generalizable rewards and hyperparameter optimization in system-specific applications.
  • IV. EXPERIMENTAL SETUP: The C++ framework combines memory management, communication, skill, scheduler, learner, and agent modules with optimized embedded libraries for flexible low-resource experiments.The framework supports integrating different learning algorithms and implementing various skills.
  • IV. EXPERIMENTAL SETUP: The experiments model five hierarchical motion-related skills, grouping ATOMIC MOVE as foundational and MOVE as specialized, with MOVE SQUARE demonstrating geometric-pattern following.The proposed hierarchy requires mastering less complex skills before tackling advanced ones.
  • A. SAS: The millirobot’s SAS controls two motors through time, force, and duration parameters and reads pose as sp(t) → [x, y, ψ].These interfaces abstract robot-specific sensor readings and actuator commands so higher-level generic methods need not depend on specific hardware implementations.

B. Kinematic Reasoner · C. Skills

The kinematic reasoner infers linear and angular motion from changes in position and orientation, while the skill hierarchy develops these primitives into distance-, angle-, and pattern-specific behaviors. Skills are evaluated through sensor-based fitness and motor-command mappings, culminating in a predefined square-tracing task.

  • B. Kinematic Reasoner: The inverse kinematic reasoner uses position changes over time and wheel-force effects to determine the millirobot’s spatial movement.It connects kinematic coordinates with dynamics acting through the wheels.
  • B. Kinematic Reasoner: The reasoner infers linear and angular motion from horizontal and vertical displacement and orientation change wrapped to [−180◦, 180◦].Linear motion uses d = √(∆x^2 + ∆y^2), while pure angular motion has nonzero ∆ψ and d = 0.
  • B. Kinematic Reasoner: This two-dimensional motion knowledge generalizes to moving objects and supports logical inference and decision-making from structured or incomplete information.The kinematic reasoner contributes to the system’s minimal general knowledge base.
  • C. Skills: The skill hierarchy progresses from ATOMIC MOVE primitives to range-specific MOVE behaviors and the compound MOVE SQUARE pattern.MOVE SQUARE combines lower-level movements to outline a rectangle.
  • C. Skills: ATOMIC MOVE [LINEAR] drives straight with minimal rotation, whereas ATOMIC MOVE [ANGULAR] rotates in place with minimal ∆x and ∆y.Both skills use sensor readings, fitness specifications, and motor-command sequences with constant command durations.
  • C. Skills: MOVE [LINEAR] builds on ATOMIC MOVE [LINEAR] to stop at a target distance by learning the duration mapping τ(dl).The kinematic reasoner computes observations, while the learner reuses the primitive’s straight-line force mapping.
  • C. Skills: MOVE [ANGUALAR] similarly builds on ATOMIC MOVE [ANGULAR] to learn the duration mapping τ(da) for a specified angular distance.The actuator-force mapping is inherited from the atomic angular skill.
  • C. Skills: MOVE SQUARE has predefined actions and evaluates learned skills through a geometric sequence of linear and angular movements.The pattern uses 50 mm linear displacements and 90◦ rotations repeated to trace a rectangle.

D. Parameterization · E. Simulation Based Analysis

The study fixes skill and intrinsic-motivation settings, then compares learning algorithms and intrinsic-motivation configurations in a repeatable physics-based simulation. Evaluation combines mean skill fitness with metrics of skill-selection balance and neglect, summarized by an IM Score.

  • D. Parameterization: All skills use fthreshold = 0.95, with intrinsic-motivation settings specified in Table II.These settings define the parameterization used throughout the experiments.
  • E. Simulation Based Analysis: The Python pygame environment is physics-based and mirrors the real millirobot’s SAS for transferability between simulation and physical experiments.The controlled simulation provides a repeatable setting for studying developmental dynamics.
  • E. Simulation Based Analysis: Three learners are compared: Simulated Annealing, Q-learning, and uniform random action selection as a lower-bound reference.Simulated Annealing is also used on the real robot, while Q-learning uses a Q-table.
  • E. Simulation Based Analysis: All learners share identical intrinsic motivation, fitness threshold, and knowledge-graph structure across independent runs with different random seeds.This controls the comparison across learning algorithms and configurations.
  • E. Simulation Based Analysis: Mean skill fitness averages fitness across five motion-related skills, providing a compact continuous measure of general developmental behavioural abilities.The metric is intended to be both compact and sensitive to developmental performance.
  • E. Simulation Based Analysis: Selection entropy Nmax measures how evenly skills are selected, while maximum neglect H measures how long any skill remains unselected.Both metrics use per-skill selection statistics and recency counters.
  • E. Simulation Based Analysis: Six intrinsic-motivation parameterizations bias scheduling toward exploration, exploitation, novelty, earlier mastery, or stronger post-threshold convergence.The configurations are defined in Table III through high explore, high exploit, high Nlimit, lower fthr, and high postslope settings.
  • E. Simulation Based Analysis: The IM Score combines learning performance with scheduling stability using time-averaged fitness, logarithmic maximum neglect, and selection entropy.The score uses λ = 0.25 to penalize long-term skill neglect and µ = 0.05 to reward exploration diversity.

V. EXPERIMENTAL RESULTS · A. Development

The experiments show that tinyDSM enables efficient, adaptive skill development in a resource-constrained millirobot. In V-A, the robot progresses from atomic motions to increasingly complex skills, learning basic movements in approximately 4.5 minutes.

  • V. EXPERIMENTAL RESULTS: V. EXPERIMENTAL RESULTS: The experiments examine skill development, adaptation to environmental changes, and fitness evaluation in separate stages.V-A analyzes intrinsic motivation and its influence on actions; V-B studies responses to changes such as added weight; V-C discusses fitness evaluation.
  • A. Development: A. Development: The millirobot starts without developed skills, queries its knowledge graph, and initially forms a skill pool from available sensor-actuator dependencies.Around 100 ms, only oAMA and oAML are available because they depend only on the sensor-actuator set.
  • A. Development: A. Development: Novel atomic motions sharply increase motivation, prompting random selection between oAMA and oAML while angular motion initially improves faster.Both skills have equal motivation at discovery, but oAML shows faster fitness progress during the initial exploration.
  • A. Development: A. Development: Motivation alternates between skills as fitness develops, although other trials sometimes show latching on one skill with substantially faster fitness growth.Between segments (A) and (B), both skills show similar fitness growth; latching instead concentrates motivation on one skill for an extended period.
  • A. Development: A. Development: After around 1.8 min and ∼50 physical interactions, oAMA reaches fthreshold, experience becomes 1, and the more complex oMA is discovered.oMA depends on oAMA and therefore requires the previously developed skill for its activities.
  • A. Development: A. Development: Reaching fthreshold for oMA and oAML raises experience to 2 and 3, respectively, and leads to discovery of oML despite delayed progress for the harder oAML.The agent briefly focuses on angular motions before reaching the oAML threshold in the next interaction, then shifts toward the newly discovered oML.
  • A. Development: A. Development: Discovery of oMS enables square-path driving by sequencing previously learned skills, and the robot focuses on navigating squares while retaining some atomic movements.The more complex behavior is treated as offering greater utility and adaptability, but simpler moves are not entirely ignored.
  • A. Development: A. Development: High-quality squares raise experience to 5, while fitness monitoring allows immediate recognition of environmental changes and motivates later adaptation analysis.Because oMS has no learner, it cannot be improved directly; environmental-change adaptation is addressed in experiment V-B.

B. Adaption · C. Fitness Assessment

tinyDSM enables the millirobot to detect fitness losses after physical changes, increase motivation, and relearn affected skills. Its hierarchical fitness assessment tracks variance and propagates lower-level errors into complex movements, supporting targeted correction and adaptation.

  • B. Adaption: Weight change sharply reduced oMA and oML fitness, while atomic skills oAMA and oAML remained unaffected because only command duration τ changed.The ratios between f1 and f2 were unchanged, preserving the atomic motion skills.
  • B. Adaption: During the ∼9 min weight interval, decreased fitness sharply increased intrinsic motivation, driving the agent to relearn its impaired skills.The increase was primarily caused by the progress factor.
  • B. Adaption: 380 g added after roughly 6.5 min invalidated previously learned movement parameters because the millirobot reacted differently to identical motor commands.The weight change marked the start of the adaptation interval.
  • B. Adaption: The agent successfully relearned its skills, shown by increasing fitness scores and improved experience after the physical change.tinyDSM continuously monitors actions and detects deviations through fitness computations, enabling adaptation to unseen conditions.
  • C. Fitness Assessment: 0.7 < f < 0.95 movements showed high variance and large target deviations, whereas f ≥0.95 movements were closer to targets with substantially reduced variance.Some variance remained at high fitness because the millirobot’s physical properties were far from ideal.
  • C. Fitness Assessment: Gearbox stalling during braking sometimes extended movement, causing the robot to overshoot targets and produce fluctuating fitness values.The resulting square trajectories often remained open, missed targets, and deviated from a perfect square.
  • C. Fitness Assessment: Fitness calculation scales to complex skills: lower-level errors propagate to higher levels, allowing targeted correction and relearning only of affected skills.The hierarchical structure was demonstrated with the square skill, whose trajectory comprised 8 sequential movements.

D. Resources … 2) Intrinsic Motivation Dynamics:

tinyDSM uses predictable, memory-efficient resource management while supporting rapid skill acquisition and balancing exploration against exploitation. Across learning algorithms and intrinsic-motivation settings, Q-learning and simulated annealing outperform random policies, while the baseline motivation configuration provides the most stable development schedule.

  • D. Resources: Memory allocation occurs only during startup and skill creation, making normal-cycle consumption predictable and reducing bookkeeping and out-of-memory risk.
  • D. Resources: 52% of total memory is consumed by learners, while the five tested skills require 488 B, SAS uses 720 B, and the scheduler requires 840 B.
  • D. Resources: 96 instances of the linear move skill would fit within the RP2040’s 250 kB available memory after reserving 16 kB for the Pico SDK.
  • 1) Skill Acquisition Across Learning Algorithms:: Q-learning and simulated annealing rapidly acquire the full skill set with near-optimal performance, whereas the random policy fails to develop meaningful behaviour.
  • 1) Skill Acquisition Across Learning Algorithms:: Q-learning converges fastest, followed closely by simulated annealing, while random policies stagnate below the meaningful skill-acquisition threshold because higher-order move skills are bottlenecks.
  • 2) Intrinsic Motivation Dynamics:: The baseline configuration combines high selection entropy (≈0.83) with low maximum neglect (≈1.37), indicating balanced and stable development.
  • 2) Intrinsic Motivation Dynamics:: High exploration increases selection entropy but causes extreme maximum neglect, whereas high exploitation reduces neglect while slightly lowering coverage.
  • 2) Intrinsic Motivation Dynamics:: The baseline achieves the highest IM rating, while lower fthr, high postslope, and high explore respectively increase neglect, reduce scheduling stability, or starve previously learned skills.

VI. CONCLUSION

tinyDSM enables resource-constrained millirobots to autonomously develop open-ended skills from minimal prior knowledge by combining intrinsic motivation, fitness-based evaluation, and structured knowledge representation. Experiments demonstrate rapid motion-skill development, adaptation, and retention within severe hardware and memory constraints, while simulations show sensitivity to motivation scheduling and parameterization.

  • Contribution: tinyDSM enables autonomous, open-ended skill development in resource-constrained millirobots from minimal prior knowledge.The method combines intrinsic motivation, fitness-based evaluation, and structured knowledge representation.
  • Experimental results: 15 minutes is sufficient for a 36 cm3 millirobot running on an RP2040 with 9 kB of memory to progress from atomic motions to complex geometric behaviours.Curriculum-based learning balances novelty, progress, and difficulty through intrinsic motivation.
  • Simulation analysis: Simulation results show that developmental dynamics and skill-acquisition stability are highly sensitive to motivation scheduling and parameterization.The simulation enabled systematic evaluation across learning algorithms and intrinsic motivation configurations.
  • Experimental results: The agent developed core motion skills, adapted to added weight, and maintained previously acquired performance through continuous fitness-based evaluation.This supports lifelong learning while the system pursues new competencies.
  • Future work: Future work will investigate richer sensing, higher-level planning, lightweight neural networks for on-device learning, and more complex scenarios.These directions extend self-directed lifelong learning under limited resources and dynamic environments.

APPENDIX · A. Intrinsic Motivation

The appendix defines intrinsic motivation for skill exploration using novelty, progress, and difficulty-related factors. At each time step, the agent selects a skill from the available pool by maximizing this motivation.

  • A. Intrinsic Motivation: Intrinsic motivation m(o, t) is defined as the product of three factors for pursuing a masterable skill o ∈ M_O.The factors are expressed through novelty n(o, t), progress p(o, t), and difficulty d(o, t).
  • A. Intrinsic Motivation: Novelty n(o, t), progress p(o, t), and difficulty d(o, t) are abbreviated to make the intrinsic-motivation formulation compact.The intrinsic-motivation function is denoted m(o, t).
  • A. Intrinsic Motivation: Novelty is updated by decay after skill execution and by growth toward Nlimit otherwise.The update uses β as the decay rate, γ as the growth rate, Ninit as the initial novelty, and Nlimit as the maximum novelty value.
  • A. Intrinsic Motivation: Progress is scaled and offset differently below and above the fitness threshold fthreshold.pscale and poffset are the scaling and offset parameters for progress.
  • A. Intrinsic Motivation: Prerequisite-skill difficulty depends on the fitness f(opk, t) of each prerequisite skill opk at time t.The prerequisite set is defined from the fitness of prerequisite skills.
  • A. Intrinsic Motivation: At every discrete time step t, the agent chooses the next skill by maximizing the motivation function.The maximization is performed over the available skill pool M_o.

B. Experience · C. Intrinsic Motivation Health Metrics

The framework tracks whether skills are learned and aggregates this experience to measure overall agent progress. It also evaluates intrinsic-motivation health through maximum neglect and selection entropy, capturing skill-sampling coverage and long-term neglect.

  • B. Experience: Experience E(o, t) indicates whether skill o ∈Mo has been learned at time t.
  • B. Experience: Total accumulated experience aggregates the experience of all skills at time t.
  • B. Experience: The accumulated-experience metric reflects the agent’s overall progress.
  • C. Intrinsic Motivation Health Metrics: Maximum neglect measures the worst-case number of steps since any skill was last selected.For skill k, Rk(t) denotes the number of steps since its last selection.
  • C. Intrinsic Motivation Health Metrics: Selection entropy uses cumulative skill-selection counts Ck(t) to determine each skill’s selection probability.
  • C. Intrinsic Motivation Health Metrics: Normalized Shannon entropy quantifies the distribution of skill selection.
  • C. Intrinsic Motivation Health Metrics: H(t) ∈[0, 1] measures skill-sampling coverage, while Nmax(t) captures long-term skill neglect.
Loading 2608.17596v1…