PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning: A Survey

L. P. Kaelbling, M. L. Littman, A. W. Moore

arXiv:cs/9605103v1cs.AI

TL;DR

Reinforcement-learning agents must address challenges such as temporal credit assignment while learning from interaction. This paper surveys the field’s historical basis and selected current work, reporting that reaching the final goal can speed learning significantly.

  • Problem

    Temporal credit assignment is identified as the biggest problem facing a reinforcement-learning agent.

    Source: p. 15

  • Method

    The paper surveys reinforcement learning’s historical basis and selected current work.

    Source: p. 1

  • Results

    The survey reports that reaching the final goal can speed learning significantly.

    Source: . Conclusions · p. 39

  • Takeaways & Limitations

    The survey provides an overview of reinforcement learning’s historical foundations and current research.

    Source: p. 1

    The survey notes that operation in non-stationary environments may fall outside a potentially disappointing assumption.

    Source: p. 3

Abstract

from arXiv · show

This paper surveys the field of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the field and a broad selection of current work are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but differs considerably in the details and in the use of the word ``reinforcement.'' The paper discusses central issues of reinforcement learning, including trading off exploration and exploitation, establishing the foundations of the field via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state. It concludes with a survey of some implemented systems and an assessment of the practical utility of current methods for reinforcement learning.

. Conclusions

Reinforcement-learning techniques work well on many small problems but rarely scale to larger ones, so complex tasks require incorporating useful bias rather than relying on tabula rasa learning. Promising forms of bias include shaping, local reinforcement signals, imitation, problem decomposition, and reflexes, although substantial work remains on approximating, decomposing, and incorporating bias.

Scaling limitations

  • Few reinforcement-learning techniques scale well beyond small problems because solving arbitrary problems in the general case is very difficult.The conclusion attributes this limitation to problem difficulty rather than poor invention of learning techniques.Source: . Conclusions · p. 39

Bias for complex problems

  • Complex reinforcement-learning problems require useful bias, supplied by human programmers or teachers, instead of purely tabula rasa learning.The paper concludes that appropriate bias can eventually make complex problems solvable.Source: . Conclusions · pp. 39–40

Bias mechanisms

  • Shaping progressively presents a learner with simple problems before exposing it to more complex problems, while local reinforcement signals may reward steps up a gradient when one can be computed.Source: . Conclusions · p. 39

Reflexes

  • Reflexes can keep agents alive and directed toward goals while learning, and can later be overridden by more detailed learned knowledge.Without prior guidance, agents may wander randomly or be eliminated before reaching interesting parts of the state space.Source: . Conclusions · pp. 39–40

Open questions

  • Important open questions remain about methods for approximating, decomposing, and incorporating bias into reinforcement-learning problems.The conclusion states that much work remains despite the prospect of solving complex problems with appropriate bias.Source: . Conclusions · p. 40

Ac kno wledgemen ts

The authors thank Marco Dorigo, anonymous reviewers, and colleagues in the reinforcement-learning community for feedback, contributions, and explanations that improved the paper.

  • The authors credit Marco Dorigo, three anonymous reviewers, and reinforcement-learning colleagues with comments, contributions, and explanations that helped improve the paper.Source: Ac kno wledgemen ts · p. 40
  • Leslie Pack Kaelbling, Michael Littman, and Andrew Moore received partial support from NSF grants, Bellcore, an NSF Research Initiation Award, and IBM Corporation.Source: Ac kno wledgemen ts · p. 40
Loading cs/9605103v1…