PaperlayerView exact revision on arXiv

Source-linked AI summary

Bootstrapping Intrinsically Motivated Learning with Human Demonstrations

Sao Mai Nguyen, Adrien Baranes, Pierre-Yves Oudeyer

arXiv:1112.1937v1cs.LGcs.AIcs.RO

TL;DR

Intrinsic-motivation methods face limits in high-dimensional and unbounded spaces, while socially guided learning can constrain learner autonomy. SGIM-D combines demonstrations with autonomous exploration; the reported fishing experiment finds improved precision and exploration of otherwise overlooked subspaces.

  • Problem

    Intrinsic-motivation methods can identify learnable zones inefficiently in high dimensions and cannot explore every locality in an unbounded space, while social guidance can constrain autonomy.

    Source: B. Intrinsically Motivated Exploration · p. 1, A. Socially Guided Exploration · p. 1

  • Method

    SGIM-D combines intrinsic exploration with demonstrations of means and goals, placing demonstrations among unreached goals at a regular pace.

    Source: C. Combining Internally Guided… · p. 2, 4) The nearest among the unreached goals · p. 4

  • Results

    Demonstrations improve inverse-model precision over autonomous exploration or demonstration-only learning, and SGIM-D explores a small subspace ignored by random exploration and SAGG-RIAC.

    Source: 2) Performance of SGIM · p. 7

  • Takeaways & Limitations

    SGIM-D uses demonstrations to explore unknown subspaces and autonomous SAGG-RIAC exploration to improve performance and precision across a wide range of tasks.

    Source: §VI · p. 7

    The teaching parameters, including demonstration timing, were not optimized, and imitation-to-emulation switching was fixed rather than autonomous.

    Source: §VI · p. 8

Abstract

from arXiv · show

This paper studies the coupling of internally guided learning and social interaction, and more specifically the improvement owing to demonstrations of the learning by intrinsic motivation. We present Socially Guided Intrinsic Motivation by Demonstration (SGIM-D), an algorithm for learning in continuous, unbounded and non-preset environments. After introducing social learning and intrinsic motivation, we describe the design of our algorithm, before showing through a fishing experiment that SGIM-D efficiently combines the advantages of social learning and intrinsic motivation to gain a wide repertoire while being specialised in specific subspaces.

I. APPROACHES FOR ADAPTIVE PERSONAL ROBOTS

Adaptive robots must learn new skills as environments and users change, but open-ended learning makes deciding what to explore crucial. This section motivates combining social guidance with intrinsic motivation to address limits of each in continuous, unbounded, non-preset environments.

I. APPROACHES FOR ADAPTIVE PERSONAL ROBOTS

  • Personal robots must adapt to changing environments and user needs rather than rely only on actions preprogrammed by engineers.Source: §I · p. 1

A. Socially Guided Exploration

  • Prior socially guided systems can depend heavily on human input, while greater learner autonomy can address constraints such as patience, ambiguous input, and correspondence problems –.Source: A. Socially Guided Exploration · p. 1

B. Intrinsically Motivated Exploration

  • Intrinsic-motivation methods face inefficient identification of learnable zones in high-dimensional spaces and cannot sample every locality in an unbounded space within a lifetime.Source: B. Intrinsically Motivated Exploration · p. 1

C. Combining Internally Guided Exploration and Socially Guided Exploration

  • Social guidance can introduce intrinsically motivating activities for later autonomous exploration, while self-exploration tends toward broader repertoires and teacher guidance toward faster learning of fewer tasks.Source: C. Combining Internally Guided… · p. 2
  • The paper introduces SGIM to merge socially guided exploration and intrinsic motivation in unbounded, non-preset, continuous environments.Source: C. Combining Internally Guided… · p. 2

II. INTRINSIC MOTIVATIONS : THE SAGG-RIAC ALGORITHM

SGIM uses SAGG-RIAC as its intrinsic-motivation component to learn skills by selecting goals and attempting to reach them. Its architecture separates longer-term goal choice from shorter-term goal-directed learning.

II. INTRINSIC MOTIVATIONS : THE SAGG-RIAC ALGORITHM

  • SAGG-RIAC supplies competence-based intrinsic motivation and is selected for learning a wide range of skills in high-dimensional spaces, including easy and unlearnable subparts.Its goal-directedness supports bidirectional merging with social methods, and detecting unreachable spaces suits unbounded spaces.Source: §II · p. 2

A. Formalisation of the Problem

  • The system learns forward and inverse models of a mapping from context and action to new states, treating the resulting task-space state as a goal reached by means a.Actions are parameterised dynamic motor primitives: temporally extended policies controlled through a few parameters.Source: A. Formalisation of the Problem · p. 2

B. Global Architecture of SAGG-RIAC

  • SAGG-RIAC separates longer-timescale goal selection from shorter-timescale attempts to reach the selected goals.The higher level sets goals based on previous achievement; the lower level attempts them.Source: B. Global Architecture of SAGG-RIAC · p. 2

C. Lower Time Scale: Active Goal Directed Exploration and Learning

  • During goal-directed exploration, the system computes forward and/or inverse models for later goals and selects new actions using local measures of learned-model quality.Source: C. Lower Time Scale: Active Goal… · p. 2

D. Higher Time Scale: Goal Self-Generation and Self-Selection

The higher-level process uses competence feedback and changes in local competence to guide goal self-generation and selection. Competence assesses reaching success under constraints, while interest estimates recent progress within regions of the task space.

D. Higher Time Scale: Goal Self-Generation and Self-Selection

1) Competence for a Reaching Attempt:

  • Competence γ_yg is based on similarity between the reached state and goal under constraints; a value near zero indicates competent reaching.The goal counts as reached when similarity exceeds tolerance ε_sim.Source: 1) Competence for a Reaching Attempt · p. 3

2) Definition of Interest:

  • Each region contains attempted goals ordered by trial time, and its interest estimates local competence progress over the ζ most recent attempts.Source: 2) Definition of Interest · p. 3

3) Goal Self-Generation Using the Measure of Interest:

Goal generation partitions task space by interest and selects where to explore, while the social-interaction analysis frames teacher and learner behavior as responses to one another. The stated setup assumes transparent communication and an omniscient teacher.

3) Goal Self-Generation Using the Measure of Interest:

  • Goal self-generation splits task space to distinguish regions by interest, then selects a region and generates goals using one of three sampling modes.Splits occur after a maximum number of attempts and maximize the difference in interest between resulting subspaces.Source: 3) Goal Self-Generation Using the… · p. 3
  • The modes sample a random goal in an interest-weighted region, a random goal in the whole space, or a goal near a low-competence attempt in an interest-weighted region.Source: 3) Goal Self-Generation Using the… · p. 3

III. ANALYSIS OF SOCIAL INTERACTION MODES

  • Social interaction is framed as a transformation of robot and teacher models, with teacher behavior responding to visible state and robot behavior responding to teacher guidance.The formulation assumes noiseless access to the robot’s visible state and a teacher with a perfect model.Source: §III · p. 3

A. Role of the Teacher

The section compares teacher interactions by what they provide and argues that demonstrations combining a means and a goal give learners both example actions and goals. It distinguishes this from feedback, goal-only, and context-setting guidance.

1) The teacher provides high-level evaluation, feedback, or labels to a machine learner:

  • High-level feedback estimates the distance between the robot's visible state and a desired visible state, but the authors favor more informational guidance than cheering alone.Source: 1) The teacher provides high-level… · p. 3

2) The teacher shows how to reach the goal that the robot aims at:

  • Showing how to reach a self-chosen goal lets the robot learn and replicate a new way to reach that goal; this is imitation in a restricted sense.The restricted definition means copying specific motor patterns.Source: 2) The teacher shows how to reach the… · p. 3

3) The teacher shows a context (new initial conditions):

4) The teacher demonstrates goals:

  • Showing only a goal lets the robot learn from environmental results and changes, then attempt to reproduce them; this is emulation.The passage contrasts emulation with imitation, including children's tendency to copy action sequences through over-imitation.Source: 4) The teacher demonstrates goals · p. 4

5) The teacher shows both a means and a goal:

  • Combining a demonstrated means and goal gives the learner action and goal information, supporting both imitation and emulation.The authors describe this as the most complete approach because it influences the learner from both the action and goal points of view.Source: 5) The teacher shows both a means and… · p. 4

B. Timing of the Social Interaction

The section considers when interaction should occur and which demonstrations to give. It contrasts beginning-only, regular, and progress-triggered interaction, then notes that random examples amount to random exploration.

1) In the very beginning:

2) At a regular pace:

  • Regular interaction provides continuous teacher contact and is presented as the best timing for quantitatively assessing learning improvement.Source: 2) At a regular pace · p. 4

3) When the robot stops making progress:

  • Progress-triggered help may use the teacher efficiently, but requires deciding when the robot is stuck and assumes the teacher monitors its state; the experiment instead uses constant-frequency random demonstrations.Source: 3) When the robot stops making progress · p. 4

1) One sample among a set of completely random examples:

2) One random among the unreached goals:

The authors choose regular demonstrations of means and goals sampled from unreached goals, while SGIM combines social interaction with intrinsic motivation in a two-level learning structure.

2) One random among the unreached goals:

4) The nearest among the unreached goals:

IV. SGIM ALGORITHM

  • SGIM combines SAGG-RIAC intrinsic motivation with social interaction to learn an inverse model in continuous, unbounded, non-preset settings, using two levels of learning.Source: §IV · p. 4, Figure 1 · p. 5

A. Higher level of Learning

  • At the higher level, goal generation and interest computation work with social interaction, which registers demonstrations for emulation and sends demonstrated actions to imitation.Source: A. Higher level of Learning · p. 4

B. Lower Level of Learning

The lower level varies demonstrated actions to explore their locality, within a fishing experiment that tests inverse-model learning in a continuous, unbounded environment. The experiment also defines a normalized competence measure for reaching goals.

B. Lower Level of Learning

  • The lower-level imitation module repeats a demonstrated action with variations, while the overall structure can accommodate other social-interaction modes by changing its relevant modules.The modules communicate bilaterally, allowing intrinsic-motivation feedback to reach the teacher, including in an active-learning setup where the learner requests demonstrations.Source: B. Lower Level of Learning · p. 5

V. FISHING EXPERIMENT

  • The fishing experiment tests learning an inverse model for a six-degree-of-freedom arm using a rod to place a hook at a desired position in a continuous, unbounded task.The hook's landing position is two-dimensional, and each joint command is a four-scalar Bezier curve, yielding 24 action parameters.Source: A. Experimental Setup · p. 5, §V · p. 5

A. Experimental Setup

  • After every P movements, the teacher interrupts exploration with an example; SGIM-D records it and imitates it with a random parameter variation smaller than epsilon before autonomous exploration resumes.Source: A. Experimental Setup · p. 5

B. Measure of Competence

  • Competence is linked to goal-to-outcome distance and normalized by the origin-to-goal distance; the distance is rescaled to [0,1] so dimensions contribute equally.Source: B. Measure of Competence · p. 5
  • The experiment estimates an inverse model mapping desired task-space positions to movement parameters, using an optimization mechanism divided into two regimes.Source: B. Measure of Competence · p. 6

1) Exploitation Regime:

The exploitation regime selects and interpolates locally reliable movement data, then refines the action toward the goal. Exploration uses random movement parameters, with the choice between regimes guided by distance to the nearest reached position.

1) Exploitation Regime:

  • Local exploitation selects neighbors by distance and variance, interpolates their actions with distance-weighted Gaussian coefficients, then refines the action using Nelder-Mead.The reliability weight uses α = 0.5 in the experiment.Source: 1) Exploitation Regime · p. 6

2) Exploration Regime:

  • Exploration samples random movement parameters; the probability of exploration is proportional to distance from the goal to the nearest reached position.The complementary probability selects exploitation.Source: 2) Exploration Regime · p. 6

C. Simulations

The simulation defines benchmark and teaching sets within the reachable task space, then compares learning algorithms over repeated 5,000-movement runs. Performance is assessed against benchmark points, alongside histograms of explored positions.

C. Simulations

  • The simulation uses a 129-point benchmark sampled across the apparent reachable space and 27 demonstrations selected for replicability.The reachable space was estimated from about 70,000 points; the benchmark uses a 26 × 16 grid.Source: C. Simulations · p. 6

D. Experimental results

  • Each algorithm runs for 5,000 movements; SGIM-D receives one demonstration every 150 movements, while demonstrations-only learning varies the latest example.The benchmark is assessed every 250 movements, with values averaged over benchmark points and runs.Source: • SAGG-RIAC · p. 6, D. Experimental results · p. 6

• SAGG-RIAC

  • Histograms track cumulative fishing-rod positions in the 2D goal space over time, with rows for random input, SAGG-RIAC, and SGIM-D.Mean distance to benchmark points across runs is plotted to assess performance.Source: • SAGG-RIAC · p. 6, Figure 4 · p. 7, Figure 5 · p. 7

1) SAGG-RIAC compared to random exploration:

SAGG-RIAC broadens exploration and improves precision over random movement, while SGIM-D improves precision and explores a subspace missed by the autonomous baselines. The conclusion notes that validation was simulation-only and relied on strong assumptions about the teacher.

1) SAGG-RIAC compared to random exploration:

  • SAGG-RIAC explores a wider and more uniform task space than random movement, and reaches benchmark points more precisely.Its exploration changes over time as the system finds and focuses on new interesting subspaces.Source: 1) SAGG-RIAC compared to random… · p. 7

2) Performance of SGIM:

  • SGIM-D improves inverse-model precision over autonomous exploration and demonstrations alone, while exploring a small subspace missed by random exploration and SAGG-RIAC.The demonstrations occurred once every 150 movements; the teacher's example triggered interest in the subspace, which remained a focus while competence improved.Source: 2) Performance of SGIM · p. 7

VI. CONCLUSION AND DISCUSSION

  • The study concludes SGIM-D combines demonstrations with autonomous exploration in a continuous 24-dimensional action space, but evaluates only simulations with a teacher assumed to share the robot's motion rules.Real human demonstrations remain to be tested, including correspondence and teacher-bias issues.Source: §VI · p. 7
  • Teaching parameters, including demonstration timing, were not optimized, and the authors identify further study of interaction settings as future work.The experiment imitates for a fixed period before using new data from the goal point of view.Source: §VI · pp. 7–8

Related papers

Loading 1112.1937v1…