PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-timescale Nexting in a Reinforcement Learning Robot

Joseph Modayil, Adam White, Richard S. Sutton

arXiv:1112.1133v3cs.LGcs.RO

TL;DR

Prior TD prediction work mostly used simulation, table lookup, and few predictions, while robotic nexting requires many personal predictions across timescales. The approach uses linear function approximation to form each prediction from a feature-vector representation of the world's state. Online learning of thousands of parallel robot predictions was practical and accurate, with predictive accuracy competitive with offline mean-squared-error optimization.

  • Problem

    Prior TD prediction work mostly used simulation, table lookup, and few predictions, while robotic nexting requires many personal predictions across timescales.

    Source: §1 · p. 2

  • Method

    The approach uses linear function approximation to form each prediction from a feature-vector representation of the world's state.

    Source: §2 · p. 3

  • Results

    Online learning of thousands of parallel robot predictions was practical and accurate, with predictive accuracy competitive with offline mean-squared-error optimization.

    Source: §5 · p. 8

  • Takeaways & Limitations

    A single feature representation and learning-parameter set sufficed for diverse predictions, with the most dramatic gains achieved after 30 minutes.

    Source: §4 · p. 8, §7 · p. 10

Abstract

from arXiv · show

The term "nexting" has been used by psychologists to refer to the propensity of people and many other animals to continually predict what will happen next in an immediate, local, and personal sense. The ability to "next" constitutes a basic kind of awareness and knowledge of one's environment. In this paper we present results with a robot that learns to next in real time, predicting thousands of features of the world's state, including all sensory inputs, at timescales from 0.1 to 8 seconds. This was achieved by treating each state feature as a reward-like target and applying temporal-difference methods to learn a corresponding value function with a discount rate corresponding to the timescale. We show that two thousand predictions, each dependent on six thousand state features, can be learned and updated online at better than 10Hz on a laptop computer, using the standard TD(lambda) algorithm with linear function approximation. We show that this approach is efficient enough to be practical, with most of the learning complete within 30 minutes. We also show that a single tile-coded feature representation suffices to accurately predict many different signals at a significant range of timescales. Finally, we show that the accuracy of our learned predictions compares favorably with the optimal off-line solution.

1 Multi-timescale Nexting

Nexting is the continual prediction of an individual’s immediate sensory signals across many parallel predictions and timescales. The paper motivates learning this ability in robots and distinguishes its approach from prior forward models and TD-based work.

1 Multi-timescale Nexting

  • Nexting means continually predicting what will happen next in an individual's immediate, personal sensory experience.Examples include anticipating the next note in a melody or changes in one's trajectory while moving.Source: §1 · p. 1
  • People and animals appear to make many small predictions simultaneously at multiple timescales, from letters to words to sentences.Animal conditioning experiments also established learned predictions over short timescales, including predictions not initially expressed in behavior.Source: §1 · p. 2
  • The paper frames nexting as basic knowledge of how the world works in interaction with the body, making robotic nexting a natural AI goal.It characterizes nexting as a limited forward model of world dynamics.Source: §1 · p. 2
  • Prior myopic forward models predict a few selected state variables only infinitesimally or one step ahead, unlike nexting.These models use differential equations or state-transition probabilities.Source: §1 · p. 2
  • The paper follows prior work using TD methods for long-term predictions, but that work mostly used simulation, table lookup, and few predictions.Prior robot work demonstrated real-time TD prediction but not many predictions in real time or a single shared feature representation.Source: §1 · pp. 2–3

2 Nexting as Multiple Value Functions

The paper formulates nexting as learning many value functions, each predicting a target signal over a discount-defined timescale. Linear TD(λ) learns these predictions from a shared feature representation, with an offline least-squares solution serving as a benchmark.

2 Nexting as Multiple Value Functions

  • Each prediction targets a raw sensor signal or state-feature component, and its discount rate sets the timescale of the discounted future target sum.The prediction is intended to approximate the corresponding return.Source: §2 · p. 3
  • All predictions are formed as inner products of the state feature vector and their corresponding weight vectors.The paper uses linear function approximation for this shared state representation.Source: §2 · p. 3
  • Linear TD(λ) learns the prediction weight vectors, treating each target signal as reward-like.The approach extends TD prediction to many target signals and multiple timescales.Source: §2 · p. 3
  • With λ = 1 and decreasing step sizes, TD(λ) asymptotically minimizes mean squared error to the return, while smaller λ values are commonly used for faster learning.The λ = 1 case provides a theoretical touchstone.Source: §2 · p. 4
  • An offline least-squares solution supplies a performance standard, with residual error indicating when improved features rather than more experience are needed.Its upper limit on prediction quality is determined by the feature representation, and the analysis can be used with physical-world experience.Source: §2 · p. 4

3 Experimental Setup

The experiments use the Critterbot, a mobile robot with diverse internal and environmental sensors, to gather data while following a wall in a pen. Its sensory readings are encoded with tile-coded features, and learning runs online on a connected laptop.

3 Experimental Setup

  • The Critterbot provides 53 normalized sensor readings spanning motor state, ambient light, obstacle distance, acceleration, rotation, and magnetic field.It is a sensor-rich mobile platform with holonomic motion from three omni-wheels.Source: §3 · p. 4
  • The binary state representation uses 6,065 tile-coded features, with exactly 457 active at each step.Tile coding combines overlapping sensor tilings and selected sensor pairs; the features provide no history or sensor-value averaging.Source: §3 · pp. 4–5
  • The robot followed a stochastic wall-following policy in a roughly two-meter-square pen, completing a loop about every 40 seconds.It selected a random action with probability 0.05 and gathered sensor data approximately every 100 ms.Source: §3 · p. 5
  • The policy, tile coding, and TD(λ) ran in Java on a laptop linked to the robot, while data were logged for 120,000 steps.The logged run lasted 3 hours and 20 minutes.Source: §3 · pp. 5–6

4 Results

The robot learned 2,160 predictions across four timescales, with each update completed within its 100 ms duty cycle. Its predictions anticipated light-sensor changes and approached offline-optimal predictions; learning improved rapidly with one shared feature representation and parameter set.

4 Results

  • 55 ms per cycle let the robot update 2,160 predictions within its 100 ms duty cycle, using 400 MB of memory.The targets covered sensor readings and selected feature components at timescales of approximately 0.1, 0.5, 2, and 8 seconds.Source: §4 · p. 6
  • The TD(λ)-learned light-sensor prediction anticipated both rises and falls, closely matching the return and offline-optimal prediction despite noise.Figure 2 compares ideal predictions with learned predictions after 2.5 hours of experience; the 8-second learned trace behaves similarly to the return and offline solution.Source: Figure 2 · p. 6, §4 · p. 7
  • Averaging 100 aligned circuits showed that TD(λ) and offline predictions closely matched the return while anticipating both sensor transitions.The cycles were aligned on the onset of sensor saturation.Source: Figure 3 · p. 7, §4 · p. 7
  • The learning curves show decreasing error across a sample of 2,160 predictions, with the TD(λ) solution approaching offline-optimal performance despite a cooling-stop disruption.TD(λ) performed similarly to TD(1) and slightly outperformed TD(0); the bias-only representation performed poorly.Source: §4 · p. 7, Figure 4 · p. 8
  • The most dramatic learning gains occurred within 30 minutes, using the same parameters and feature representation across predictions.Predictions continued improving over time, without convergence problems reported for the shared setup.Source: §4 · p. 8

5 Discussion

The results support practical, accurate online learning of thousands of robot predictions in parallel. The same features work across tasks, with predictive accuracy approaching an offline-optimal solution after limited robot experience.

5 Discussion

  • Online learning of thousands of predictions in parallel was practical and accurate, and its predictive accuracy could approach offline mean-squared-error optimization.The reported results used one feature representation for all predictions, without tuning algorithm parameters, and required only a few hours of robot experience.Source: §5 · p. 8
  • The approach's parallel scalability contrasts with the predominantly sequential approaches common in robot learning.The authors describe this scalability of knowledge acquisition as substantially novel.Source: §5 · p. 8
  • The experiments show that the same features can be used across tasks to anticipate events before they occur, approaching offline-optimal accuracy with limited robot experience.The discussion notes that reinforcement-learning theory offers little guidance on feature choice, parameter selection across tasks, or experience needed to approach convergence.Source: §5 · pp. 8–9

6 More General Nexting

The section extends discounted nexting to broader predictions, including predictions with state-varying discounting. In one example, the robot predicts power use until a light-sensor event or a two-second horizon.

6 More General Nexting

  • Nexting can also use time-varying γ or λ and off-policy prediction, beyond the exponentially discounted predictions emphasized in the paper.The example varies γ with the agent’s state.Source: §6 · p. 9
  • Figure 5 shows the robot learned to predict power use before light saturation or spontaneous termination.The target is the sum of instantaneous power consumption across the three wheels.Source: Figure 5 · p. 9, §6 · p. 9
  • The power prediction sums indexed motor-voltage–current products and applies throttling gamma when the light sensor is saturated, while the cited passage provides no numerical gamma setting.Source: §6 · p. 9

7 Conclusions

The paper demonstrates real-time, multi-timescale nexting on a physical robot, with thousands of predictions learned in parallel. It also reports that one feature representation and one set of learning parameters suffice for many diverse predictions.

7 Conclusions

  • Thousands of anticipatory predictions at multiple timescales can be learned in parallel on a physical robot in real time.The approach uses a large feature representation with an online learning algorithm.Source: §7 · p. 9
  • A single feature representation and a single set of learning parameters suffice for learning many diverse predictions.Source: §7 · p. 10
  • A laptop can support real-time predictions about the robot’s sensors at multiple timescales.Source: §7 · p. 10
Loading 1112.1133v3…