Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots

Adrien Baranes; Pierre-Yves Oudeyer

Source-linked AI summary

Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots

Adrien Baranes, Pierre-Yves Oudeyer

arXiv:1301.4862v1Jan 21, 2013cs.LGcs.AIcs.CVcs.NEcs.RO

TL;DR

The paper addresses how robots can learn inverse models for high-dimensional redundant sensorimotor spaces when task reachability and body mappings are initially unknown. SAGG-RIAC actively explores parameterized task-space goals using competence progress, then learns goal-directed policies through regression and optimization. Across robotic setups, task-space exploration efficiently learns varied reachable tasks, progressively focuses on more complex goals, and discovers reachability limits.

Problem
Robots need inverse models for generating policies that achieve desired effects, but many body and environment mappings cannot be analytically modeled and high-dimensional exploration is sample-limited.
Method
SAGG-RIAC actively self-generates parameterized task-space goals according to competence progress and triggers low-level goal-directed policy learning using learned correspondences and regression.
Results
Across redundant robotic setups, task-space exploration efficiently learns inverse models and varied reachable tasks while competence-based selection progressively focuses exploration on tasks of increasing learnability complexity.
Takeaways & Limitations
The architecture lets robots learn inverse models while actively discovering reachable and unreachable task-space regions, rather than spending exploration on impossible goals.
Takeaways & Limitations
SAGG-RIAC is robust in task spaces up to 100 times larger than the reachable space but has difficulty in still larger spaces, where human definition of exploration bounds may remain necessary.

Abstract

from arXiv · show

We introduce the Self-Adaptive Goal Generation - Robust Intelligent Adaptive Curiosity (SAGG-RIAC) architecture as an intrinsi- cally motivated goal exploration mechanism which allows active learning of inverse models in high-dimensional redundant robots. This allows a robot to efficiently and actively learn distributions of parameterized motor skills/policies that solve a corresponding distribution of parameterized tasks/goals. The architecture makes the robot sample actively novel parameterized tasks in the task space, based on a measure of competence progress, each of which triggers low-level goal-directed learning of the motor policy pa- rameters that allow to solve it. For both learning and generalization, the system leverages regression techniques which allow to infer the motor policy parameters corresponding to a given novel parameterized task, and based on the previously learnt correspondences between policy and task parameters. We present experiments with high-dimensional continuous sensorimotor spaces in three different robotic setups: 1) learning the inverse kinematics in a highly-redundant robotic arm, 2) learning omnidirectional locomotion with motor primitives in a quadruped robot, 3) an arm learning to control a fishing rod with a flexible wire. We show that 1) exploration in the task space can be a lot faster than exploration in the actuator space for learning inverse models in redundant robots; 2) selecting goals maximizing competence progress creates developmental trajectories driving the robot to progressively focus on tasks of increasing complexity and is statistically significantly more efficient than selecting tasks randomly, as well as more efficient than different standard active motor babbling methods; 3) this architecture allows the robot to actively discover which parts of its task space it can learn to reach and which part it cannot.

1 Motor Learning and Exploration of Forward and Inverse Models

The paper frames inverse-model learning as constrained active exploration in high-dimensional redundant sensorimotor spaces. SAGG-RIAC explores parameterized task goals, uses competence progress to select them, and learns corresponding policies through regression.

Inverse models map desired effects or tasks to action policies, but many relevant mappings cannot be analytically specified because bodies, materials, and objects are difficult to model.
Unconstrained random exploration is inefficient because lifetime training examples are limited relative to the size and complexity of high-dimensional spaces.
Competence progress measures how attempts at similar goals improve the robot’s capability, providing the architecture’s criterion for interestingness.
Task-space exploration leverages lower task-space dimensionality and redundancy, learning only policy subregions sufficient to achieve reachable effects rather than a complete forward model.
SAGG-RIAC actively samples novel parameterized tasks, triggering low-level goal-directed learning of policy parameters that achieve them.
The architecture targets broad task learning, progressively increasing learnability complexity, unknown reachable subparts, and inverse models across redundant robot body schemas.

2 Competence Based Intrinsic Motivation: The Self-Adaptive Goal Generation RIAC Architecture

SAGG-RIAC organizes active learning into high-level goal selection and low-level goal-directed action learning. Its competence-based exploration adapts to reachable structure while regression and optimization support policy learning.

The higher level self-generates and selects parameterized goals according to competence-based interest, while the lower level explores actions to reach them.
SAGG-RIAC represents a field of parameterized tasks in which action sequences or motor synergies transition the robot from starting contexts toward desired outcomes.
Once a goal is chosen, the system attempts to reach it from the current state, treating every state as a potential starting state.
Regression techniques infer motor-policy parameters from previously learned correspondences, while goal-directed optimization improves policies for the selected goal.
Unreachable goals are handled with a timeout, stopping low-level learning after a maximum number of actions.
Rest-position and preparatory-action heuristics reduce initiation-state variation and overly complex actuator configurations, increasing competence feedback and reducing bootstrap effort.

3 Experimental Setup 1: Learning Inverse Kinematics with a Redundant Arm

The first experimental setup studies online inverse-kinematics learning in a redundant arm, where multiple actuator configurations can produce the same task-space outcome.

The experiment examines a robotic arm learning its forward and inverse kinematics under discrete-time position and velocity control.
Learning inverse kinematics supports adaptation to geometry changes, calibration errors, and cases where no accurate kinematic model is available.
The manipulator state is represented by joint positions α ∈ R^n, while end-effector position and orientation form the task vector y ∈ R^m.
For redundant manipulators with n > m, inverse solutions are generally non-unique, requiring learning algorithms to select particular actuator configurations.
The study uses local, incremental non-parametric regression, including an Approximate Nearest Neighbors method based on k-means tree splitting.

3.3 Robotic Setup

The arm experiment models joint angles as the state space and end-effector positions as the task space, updating an inverse-kinematics database after each micro-action.

Joint angles α ∈ R^n define the context/state space, while end-effector position y ∈ R^m defines the Euclidean task space.
Each executed micro-action θ = ∆α produces stored measurements of α, ∆α, and ∆x, building a database of discovered action-effect changes.

3.4 Evaluation of Competence

Competence measures quantify how effectively the robot reaches a goal relative to its starting distance, while handling near-start and movement-away cases explicitly.

Competence is defined as the negative Euclidean distance between the goal and final reached position, normalized by starting distance.This normalization gives comparable competence for goals at different distances from the start.
Goals closer than εC receive competence 0 because they are too near the starting position.
Competence is set to −1 when the end-effector finishes farther from the goal than from its starting position.

3.5 Addition of subgoals

The system adds intermediate competence evaluations along goal-reaching pathways to obtain denser feedback without biasing exploration.

Because reaching goals may require multiple micro-actions, competence measurements can otherwise be slow to collect.
For each generated goal, the method inserts l subgoals along the straight pathway from ystart to yg before attempting the terminal goal.The subgoals are defined as yi = (i/l) × (yg − ystart).
Each experimented end-effector position can also be treated as a maximally competent reached goal, helping distinguish covered and novel regions.

3.6 Active Goal Directed Exploration and Learning

Active goal-directed exploration alternates between model-based movements toward selected goals and local random exploration when prediction errors become too large.

The reaching phase computes a displacement toward the current goal and uses the estimated Jacobian pseudoinverse to select the micro-action.
Exploration is triggered when the discrepancy between predicted and actual displacement exceeds εmax.
Local random actions improve the regression model relating action changes to end-effector displacement around the current actuator position.
A micro-action counter resets for each goal and enforces a reaching timeout, with later experiments allowing up to 1.5 times the start-to-goal distance.

3.7 Qualitative Results for a 15 DOF Simulated Arm

In the 15-DOF arm experiment, competence and goal-generation patterns progressively expand from simple reachable regions toward more complex reachable areas while separating reachable from unreachable space.

Experimental setup: The simulated arm has 15 DOF, a 32-dimensional sensorimotor problem, and a two-dimensional task space bounded by [0; 150] × [−150; 150].
Competence evolution: After 30000 micro-actions, competence is generally high inside and near the reachable space and low in the remaining task space.
Competence evolution: Competence initially covers areas near the reachable-space limits, then progressively expands as learning supports more complex joint movements.
Global exploration: Goal generation first concentrates near the rest position, then increases its exploration radius and shifts toward additional reachable regions near the arm basis.
Robustness: In the high-volume task space, the algorithm focuses generation within reachable subspaces of progressively increasing reachability complexity.
Scope: The qualitative experiment used a convex reachable region without obstacles, while obstacle handling depends on the low-level optimization and action representation.

3.8 Quantitative Results for Experiments with Task Spaces of Different Sizes

These experiments compare task-space goal exploration with actuator-space exploration across task spaces ranging from mostly reachable to much larger than the reachable region. SAGG-RIAC improves learning efficiency and generalization in reachable spaces, remains robust up to roughly 100 times the reachable-space size, but struggles in still larger unprepared spaces.

Task-space exploration learned inverse kinematics more efficiently than random or RIAC-like actuator-space exploration.
3.8.1 Exploration in the Reachable Space: SAGG-RIAC improved both learning speed and final generalization over SAGG-RANDOM when most candidate goals were reachable.It focused sampling on areas providing more informative data instead of repeatedly selecting already well-reached regions.
3.8.2 Exploration in Large Task Spaces: In a large task space where most goals were unreachable, SAGG-RIAC was evaluated for its ability to discriminate and focus exploration on reachable regions.The experiment used Y = [0; 500] × [−500; 500] and a blocking-criteria timeout.
3.8.3 Robustness in Very Large Task Spaces: SAGG-RIAC remained robust in spaces up to 100 times larger than the reachable space.When the task space became still larger, it became less efficient than ACTUATOR-RANDOM; in Y = [0; 500] × [−500; 500], it surpassed that baseline only after 10000 micro-actions.
3.8.3 Robustness in Very Large Task Spaces: The results indicate that autonomous exploration in very large unprepared spaces still benefits from approximate externally defined task-space limits.The authors identify human supervision or complementary maturational constraints as necessary for spaces beyond the algorithm’s effective range.

3.9 Quantitative Results for Experiments

Experiments across 7, 15, and 30 DOF arms and different morphologies test whether SAGG-RIAC remains effective when reachable goals occupy only a small part of the task space. It generally outperformed actuator-space methods, with the strongest improvements in the higher-DOF systems.

The task space covered less than 1/9 of the reachable-goal choice space, requiring the robot to discover which regions were reachable.The experiments used Y = [0; 150] × [−150; 150] for arms with total length 50 units.
The experiments covered 7, 15, and 30 DOF arms with equal-length or decreasing-length limbs, including highly redundant 62-dimensional sensorimotor problems.
3.9.1 Quantitative Results: For 7 DOF, ACTUATOR-RANDOM eventually exceeded SAGG-RANDOM, while adding RIAC produced the lowest reaching errors.SAGG-RANDOM focused heavily outside the reachable area, whereas SAGG-RIAC improved learning capabilities.
3.9.1 Quantitative Results: For 15 and 30 DOF, both SAGG methods outperformed actuator methods, and SAGG-RIAC significantly improved over every other algorithm.For 15 DOF, the end-of-experiment significance level was p = 0.002 after 200000 micro-actions.
Across morphologies, SAGG-RIAC showed high efficiency and robustness compared with approaches exploring in actuator space.Random goal exploration also remained effective when the task space was more than 9 times larger than the reachable subspace.

3.10 Qualitative Results for a Real 8 DOF Arm

A real 8 DOF arm with noisy motors was used to test whether SAGG-RIAC could organize exploration outside simulation. The system discovered reachable limits, continued exploring within them, and increased competence in already visited regions.

The real setup used an 8 DOF position-controlled arm with motors averaging 20% movement noise.The camera-defined task space covered more than three times the arm’s reachable region.
After goal 57, the algorithm discovered the reachable-area limits and drove exploration inside them through the remainder of the experiment.The observation came from histograms over sliding windows during 10000 micro-actions.
Exploration progressively moved toward more central reachable positions as generalization increased competence in previously visited areas.The authors attribute this behavior to the high level of motor-control noise and improved generalization capacity.
The experiment supports SAGG-RIAC’s ability to guide learning in noisy real robotic setups with few iterations while controlling exploration complexity.

4 Experimental Setup 2: Learning Omnidirectional Quadruped Locomotion with Motor Synergies

The quadruped experiment uses parameterized motor synergies to learn an inverse model from 24 control parameters to a three-dimensional locomotion task space. SAGG-RIAC focuses exploration on reachable task-space regions and learns reaching performance faster than actuator-based or random alternatives.

Experimental setup: Motor synergies reduce complex movement control to tuning higher-level parameters while retaining a control space of several dozen dimensions.The experiment uses parameterized sinusoids and low-level feedback control to generate action sequences.
Experimental setup: The 12-DOF quadruped uses 24 continuous synergy parameters: 12 amplitudes and 12 phases controlling sinusoidal joint commands.Each resulting behavior is mapped to the robot’s position and orientation in (u, v, φ).
Measure of competence: Competence measures normalized Euclidean reaching error over the goal, robot position, and orientation, with all three rescaled dimensions equally weighted.The robot resets to a fixed origin before each reaching attempt.
Qualitative results: SAGG-RIAC explores a reachable area approximately [−10; 10] × [−10; 10] × [−π; π] inside the larger task space [−45; 45] × [−45; 45] × [−2π; 2π].The reachable distribution is asymmetric along v because of the robot’s decentered head weight.
Qualitative results: SAGG-RIAC expands exploration to nearly twice the u coverage and up to three times the v coverage of previous algorithms, reaching v = 7.5 versus 2.5.The method drives learning toward reachable regions that chance-based exploration rarely discovers.
Quantitative results: After 1000 iterations, SAGG-RIAC shows higher efficiency than the other three methods, while SAGG-RANDOM becomes significantly more efficient than actuator methods by 10000 iterations.The authors attribute rapid error reduction to creating many small regions that focus exploration on reachable areas and their surroundings.
Conclusion of results: The quadruped results show that task-space exploration efficiently learns inverse models and reaches regions that are difficult to discover through motor-synergy sampling.SAGG-RIAC also appears to improve asymptotic performance relative to the alternatives.

5 Experimental Setup 3: Learning to Control a Fishing Rod with Motor Synergies

The fishing-rod experiment tests inverse-model learning for a flexible wire controlled by a 4-DOF arm. SAGG-RIAC explores the complex, asymmetric reachable float positions more efficiently than actuator-random exploration, with a temporary error increase when newly discovered redundancies require disambiguation.

Experimental setup: A 4-DOF arm controls a flexible fishing rod through four velocity-valued motor-synergy parameters, targeting the float’s two-dimensional water-contact position.The forward model records correspondences from (v1, v2, v3, v4) to (x, y).
Qualitative results: After 10000 successful water-contact trials, ACTUATOR-RANDOM and SAGG-RIAC produce different histograms of float positions on the water surface.The central point represents the arm’s base.
Qualitative results: The reached-position distribution is asymmetric because the robot geometry, starting configuration, and motor-primitives structure make the reachable space complex and asymmetric.This asymmetry is especially visible under uniformly sampled actuator parameters.
Quantitative results: After 1000 successful trials, SAGG-RIAC achieves significantly more efficient reaching results than ACTUATOR-RANDOM across ten random-seed experiments.After 6000 trials, its reaching error increases slightly as newly discovered motor synergies are explored.
Quantitative results: New motor synergies temporarily reduce inverse-model generalization by creating redundant solutions for already mastered goals until local models become disambiguated.The interference occurs while the newly discovered parameters are being explored sufficiently.

6 Conclusion and Future Work

SAGG-RIAC efficiently learns inverse models through intrinsically motivated goal exploration, leveraging task-space redundancy and competence progress. The paper also identifies scope boundaries and future extensions involving task-space dimensionality and multiple task spaces.

Conclusion: SAGG-RIAC uses active goal generation in parameterized task space to learn inverse models while exploiting redundancy in sensorimotor mappings.The approach aims to learn many effects without learning numerous redundant action policies for the same effect.
Conclusion: Competence-progress-driven exploration focuses learning on tasks of increasing complexity while helping the robot discover reachability limits and avoid impossible tasks.The resulting developmental trajectories progressively concentrate exploration where learning improves.
Future Work: The experiments assumed that a low-dimensional task space was provided in advance, limiting direct application to developmental settings without predefined task spaces.The authors identify autonomous discovery of suitable task spaces as an additional required capability.
Future Work: Higher task-space dimensionality would make competence-progress evaluation suffer from the curse of dimensionality.The paper therefore proposes mechanisms to find low-dimensional task spaces autonomously.
Future Work: Future extensions include actively exploring multiple task spaces to support learning fields of skills that may differ in kind.The paper also suggests higher-level stochastic generation and selection of task spaces through global competence progress.
Future Work: SAGG-RIAC must initially use random exploration to distinguish unreachable areas and areas of differing interest.This initialization becomes problematic when reachable regions are much smaller than, or embedded in, a high-dimensional task space.

7 Biographies

The supplied section combines author biographies with algorithmic material describing SAGG-RIAC’s spaces, models, goal generation, and goal-directed exploration loop.

Biographies: Adrien Baranes received M.S. and Ph.D. degrees in artificial intelligence and robotics and studied developmental mechanisms for constraining robot exploration.His doctoral work concerned mechanisms enabling robots to progressively learn large quantities of knowledge and know-how.
Biographies: Pierre-Yves Oudeyer is an Inria researcher leading the FLOWERS team and working in developmental and social robotics.The passage also notes his publications, patents, lectures, prizes, and ERC Starting Grant.
Architecture: SAGG-RIAC represents state/context, parameterized policies, parameterized tasks, forward and inverse regression models, and regions with interest measures.Its inputs include competence thresholds, a resting state, a starting position, and an experiment budget.
Architecture: The architecture resets the system periodically, self-generates a task-space goal, optionally computes subgoals, and executes policies inferred for each goal.After each execution it records the achieved effect, updates the regression models, computes competence, and updates task-space regions.
Exploration: Low-level goal-directed exploration optimizes policy parameters toward selected goals under a budget of physical experiments.The implementation supports evolving-context and fixed-or-resettable-context variants, with local exploration around candidate policies.
Exploration: UpdateRegions assigns each competence measure to its goal-containing region, recomputes regional interest over a time window, and may split regions.This procedure operationalizes competence-progress-based selection in the task space.