Source-linked AI summary
Towards Deep Symbolic Reinforcement Learning
Marta Garnelo, Kai Arulkumaran, Murray Shanahan
TL;DR
Contemporary DRL often needs large datasets, offers limited abstract reasoning and transfer, and remains opaque to humans. The paper proposes an end-to-end architecture combining a neural back end with a symbolic front end, then evaluates a prototype on simple game variants. The prototype learns effectively, supports symbolic and superficial transfer capabilities, and dramatically outperforms DQN on the most difficult stochastic variant, while retaining important limitations.
Problem
Contemporary DRL requires very large training sets, has limited abstract reasoning and transfer capabilities, and is largely opaque to humans.
Method
The paper combines a neural back end that converts raw perceptual data into symbolic representations with a symbolic front end that selects actions.
Results
The prototype learns effectively across the game variants, dramatically outperforms DQN on the most difficult stochastic variant, and requires no back-end retraining across new game variants.
Takeaways & Limitations
Symbolic rules provide a humanly comprehensible chain of justifications for the prototype’s action choices and illustrate the architecture’s potential for transfer learning.
Takeaways & Limitations
The prototype cannot learn a globally optimal policy for these games, and its locality assumption can produce locally optimal policies inferior to the global optimum.
Abstract
from arXiv · showhide
Deep reinforcement learning (DRL) brings the power of deep neural networks to bear on the generic task of trial-and-error learning, and its effectiveness has been convincingly demonstrated on tasks such as Atari video games and the game of Go. However, contemporary DRL systems inherit a number of shortcomings from the current generation of deep learning techniques. For example, they require very large datasets to work effectively, entailing that they are slow to learn even when such datasets are available. Moreover, they lack the ability to reason on an abstract level, which makes it difficult to implement high-level cognitive functions such as transfer learning, analogical reasoning, and hypothesis-based reasoning. Finally, their operation is largely opaque to humans, rendering them unsuitable for domains in which verifiability is important. In this paper, we propose an end-to-end reinforcement learning architecture comprising a neural back end and a symbolic front end with the potential to overcome each of these shortcomings. As proof-of-concept, we present a preliminary implementation of the architecture and apply it to several variants of a simple video game. We show that the resulting system -- though just a prototype -- learns effectively, and, by acquiring a set of symbolic rules that are easily comprehensible to humans, dramatically outperforms a conventional, fully neural DRL system on a stochastic variant of the game.
1 Introduction
The paper proposes an end-to-end hybrid architecture that combines neural learning from raw data with symbolic representations and reasoning to address major shortcomings of contemporary DRL. Its proof-of-concept targets conceptual abstraction, compositional structure, common-sense priors, causal reasoning, and more interpretable decisions.
- Motivation: Contemporary DRL requires very large training sets, learns slowly, and has limited abstraction, transfer, causal reasoning, and transparency.These shortcomings motivate combining neural and symbolic methods.
- Architecture: The proposed architecture combines a neural back end that maps raw perceptual data to symbolic representations with a symbolic front end that selects reward-maximising actions.Both components are shaped by machine learning, enabling end-to-end learning with minimal environmental assumptions.
- Conceptual abstraction: Symbolic representations support higher-level similarity judgments between states, facilitating data-efficient learning and transfer learning beyond low-level statistical generalisation.The paper contrasts this with conventional DRL systems that gradually build a statistical picture of the state space.
- Compositional structure: Compositional representations can be combined and recombined in open-ended ways, supporting abstract concepts and general-purpose reasoning across tasks and domains.The paper identifies first-order logic as a classical foundation for this representational medium.
- Causal reasoning: The architecture aims to discover causal structure and encode it as symbolic rules, enabling abstract analogy, planning, lookahead, and offline exploration.These capabilities are intended to increase the benefit of past experience and support action selection.
- Prototype scope: The prototype learns symbolic representations of game-state sequences involving objects, types, locations, and interactions, but its neural back end and symbolic reasoning remain restricted.The authors describe it as barely scratching the surface of the larger proposal.
2 Experimental setup
The prototype is evaluated on four variants of a simple grid-like game in which an agent moves toward objects whose shapes determine positive or negative rewards. The variants progressively introduce multiple object types and randomized object positions.
- Game task: The agent uses up, down, left, or right actions to collect or avoid objects, receiving rewards determined by their shapes.Crosses (‘x’) yield positive reward, while circles (‘o’) yield negative reward.
- Variant 1: Variant 1 contains only negative-reward circles arranged in a fixed grid, with each encounter scoring -1.The player starts in the middle of the board.
- Variant 2: Variant 2 retains the fixed layout while adding positive-reward crosses alongside negative-reward circles.Circles give -1 points and crosses give 1 point.
- Variant 3: Variant 3 retains only negative-reward objects but randomizes their positions for every new game.This increases the difficulty of learning.
- Variant 4: Variant 4 combines randomized object positions with the two object types from Variant 2.It therefore combines the task’s positional randomness and mixed rewards.
3 Methods
The method builds a symbolic game-state representation through low-level symbol generation, object tracking, interaction extraction, and reinforcement learning. It combines learned visual features with persistence, transition, neighborhood, and locality heuristics.
- The pipeline comprises low-level symbol generation, representation building, and reinforcement learning.
- Low-level symbol generation: A convolutional autoencoder trained on 5000 randomly generated images supplies unsupervised object symbols from salient feature activations.Objects are assigned types by comparing activation spectra; unmatched symbols can create new types.
- Representation building: Object tracking combines spatial proximity, learned type transitions, and neighborhood similarity to assign persistent identities across frames.The transition matrix models type changes, while neighborhood currently uses differences in neighbor counts.
- Representation building: Tracking adds unique object identifiers, types, and positions to the symbolic game-state representation.
- Reinforcement learning: The locality heuristic bounds the interaction state space and enables faster training, but can yield a locally optimal policy below the global optimum.The benchmark also assumes the directly controlled moving object is the plus sign, a restriction requiring automatic identification in more general settings.
- Representation building: Frame differences and relative object interactions produce a concise spatio-temporal representation used as input to reinforcement learning.The representation captures object locations, types, and changes in interactions and relative positions.
4 Results
The prototype learned effectively across four game variants, with scores stabilizing after early training and object-level evaluation revealing learning beyond aggregate reward. It achieved about 70% positive-object collection in two mixed-object variants and outperformed DQN in random and transfer settings, though DQN reached 100% in the simpler grid setting.
- Learning across game variants: Scores increased within the first few hundred epochs and then remained approximately constant across all four games.Agents were trained for up to 1000 epochs and evaluated periodically over 200 time steps on 10 games.
- Learning across game variants: Average score alone incompletely characterizes learning when positive and negative objects can cancel each other out.Object-level percentages distinguish selective positive-object collection from behavior that produces the same net score through mixed collections.
- Learning across game variants: Approximately 70% of collected objects were positive after training in both two-object-type game variants.The initial proportion was about 50%, and the percentage increased during training.
- Comparison to DQN: In the grid environment, DQN reached 100% positive-object collection after a few hundred epochs, while the symbolic agent reached 70%.The grid geometry allowed DQN to learn a diagonal route that collected positive objects and avoided negative ones.
- Transfer learning: After grid-only training, the symbolic agent rapidly reached approximately 70% on random games, while DQN remained at chance after 1000 epochs.The authors hypothesize that DQN might eventually learn the random game when trained on it, but did not run the experiment long enough to confirm this.
5 Discussion
The prototype combines symbolic representations with neural learning to achieve effective performance, transfer across game variants, and humanly comprehensible decisions, while motivating extensions for richer environments and reasoning.
- 5 Discussion: The prototype learns effectively across game variants but cannot learn a globally optimal policy.On the random-placement game, it acquired an effective policy in 200 epochs, whereas DQN remained at chance after 1000 epochs.
- 5 Discussion: The system transfers its learned ontology to new game variants without retraining the neural back end, but its current transfer remains superficial.The architecture is intended to support stronger analogy-based transfer, which the present system barely exploits.
- 5 Discussion: Symbolic inference provides a humanly comprehensible chain of justifications for action choices through the Q functions and object relations.This supports greater transparency in the prototype's decisions.
- 5 Discussion: More realistic images will require a more sophisticated neural back end capable of unsupervised learning of disentangled, compositional representations.The present benchmark uses simpler images than richer environments would require.
- 5 Discussion: Proposed symbolic extensions include inductive logic programming, formal analogical reasoning, and planning based on learned causal structure.These additions target stronger generalisation, similarity detection, and offline discovery of action sequences leading to rewards.
- 5 Discussion: The symbolic components might themselves be implemented with neural networks, while directly implemented symbolic reasoning remains a promising research direction.The authors leave open neurally based implementations of symbolic reasoning functions.