Source-linked AI summary
Neural Programmer-Interpreters
Scott Reed, Nando de Freitas
TL;DR
The paper addresses how machines can learn, compose, and conditionally execute programs across diverse environments from limited supervision. NPI combines a shared recurrent core with persistent program memory and domain-specific encoders, and demonstrates multi-task learning, strong generalization, and continued program acquisition. The main scope boundary is that the 3D canonicalization model receives pixels and target pose but not query-frame poses.
Problem
Teaching machines to learn and compose programs across varied tasks remains a central AI challenge, motivating learning from fewer examples with richer compositional supervision.
Method
NPI uses a shared recurrent core, persistent key-value program memory, domain-specific encoders, and supervised execution traces to compose and execute programs.
Results
A single NPI learned 21 programs across addition, sorting, trajectory planning, and image-based tasks, with strong sorting generalization beyond the training range.
Takeaways & Limitations
Program memory and compositional representations support multi-environment execution, reuse of learned programs, and continued learning without forgetting previously learned programs.
Takeaways & Limitations
For 3D canonicalization, NPI receives only rendered pixels and the target pose, not the poses of query frames.
Abstract
from arXiv · showhide
We propose the neural programmer-interpreter (NPI): a recurrent and compositional neural network that learns to represent and execute programs. NPI has three learnable components: a task-agnostic recurrent core, a persistent key-value program memory, and domain-specific encoders that enable a single NPI to operate in multiple perceptually diverse environments with distinct affordances. By learning to compose lower-level programs to express higher-level programs, NPI reduces sample complexity and increases generalization ability compared to sequence-to-sequence LSTMs. The program memory allows efficient learning of additional tasks by building on existing programs. NPI can also harness the environment (e.g. a scratch pad with read-write pointers) to cache intermediate results of computation, lessening the long-term memory burden on recurrent hidden units. In this work we train the NPI with fully-supervised execution traces; each program has example sequences of calls to the immediate subprograms conditioned on the input. Rather than training on a huge number of relatively weak labels, NPI learns from a small number of rich examples. We demonstrate the capability of our model to learn several types of compositional programs: addition, sorting, and canonicalizing 3D models. Furthermore, a single NPI learns to execute these programs and all 21 associated subprograms.
1 INTRODUCTION
NPI is a compositional architecture for representing and executing programs across diverse environments. It learns from supervised execution traces, generalizes strongly, and supports continual program learning through a shared core and program memory.
- Motivation: The architecture targets program learning across motor behaviors, image transformations, reinforcement-learning policies, classical algorithms, and symbolic relations.The introduction presents teaching, composing, and conditionally executing programs as central AI challenges.
- Architecture: NPI combines an LSTM-based core, learnable key-value program memory, and environment-specific representations to execute programs across tasks.The core selects subprograms, arguments, and termination conditions from program embeddings, caller arguments, and environment features.
- Multi-task learning: A single shared-parameter NPI learned 21 programs, including addition, sorting, and trajectory planning from image pixels.Different environments can use specialized perception modules and actuators while sharing the same core model.
- Training: Training uses curriculum learning and rich execution traces that specify input-conditioned calls to immediate subprograms.This supervision provides compositional information with fewer labeled examples than approaches based on many weaker labels.
- Generalization: NPI generalizes from sorting sequences of up to twenty numbers to much longer sequences, whereas sequence-to-sequence LSTMs show weak generalization.The paper frames this as an advantage of compositional model building and training.
- Continual learning: A trained NPI can interpret existing programs and generate program embeddings for new tasks that are added to its learned library.This supports continued program acquisition using fixed core parameters.
2 RELATED WORK
NPI builds on prior work in programmable and recurrent neural systems, hierarchical learning, and program induction. Its distinguishing emphasis is explicit compositional structure implemented through program memory and reusable subprograms.
- Programmable networks: Earlier work explored dynamically programmable networks and meta-learning systems in which one network generates context-dependent changes for another.The paper places these ideas in a longer history spanning Sigma-Pi units and related neural architectures.
- Hierarchical learning: Related research studied hierarchical reinforcement learning, imitation learning, apprenticeship learning, and elicitation of options through human interaction.The authors describe these directions as promising but lacking significant impact.
- Recurrent models: Neural Turing machines, Pointer Networks, and other recurrent models extended neural networks beyond simple sequence prediction.These systems addressed tasks including copying, sorting, associative recall, and combinatorial optimization.
- Program induction: Program-induction research has used genetic programming, program embeddings, character-level LSTMs, and recurrent architectures to infer or execute programs.The cited approaches vary in whether they operate from input-output examples, program symbols, or program text.
- Distinction: NPI differs from contemporaneous recurrent approaches by explicitly incorporating compositional structure through program memory and subprogram combination.This design allows the model to learn new programs by combining previously learned subprograms.
3 MODEL
NPI uses a shared LSTM core, persistent program memory, and domain-specific encoders to route and execute compositional programs across varied environments. Its inference produces program calls, arguments, and termination decisions while environment actions update the state.
- Inference: The NPI core routes between programs using an LSTM conditioned on state observations and previous hidden states, with a learned termination signal returning control to callers.Program invocation uses a call stack, and termination occurs when the end probability exceeds 0.5.
- Inference: Domain-specific encoders convert varied observations and arguments into fixed-length state vectors, allowing one modality-agnostic core to operate across environments.The environments include arrays, scratch pads, and CAD renderers, with encoders potentially shared across tasks.
- Inference: At each step, the model emits an end probability, a key for the next program, and output arguments that become the next program’s inputs.The state is encoded, processed by the LSTM, and decoded into these three outputs.
- Inference: Program keys and embeddings are stored in persistent key-value memory, which can grow by adding rows for newly learned programs.At test time, the predicted key is compared with stored keys to retrieve the next program embedding.
- Inference: Execution alternates between recursively running subprograms and applying environment actions, whose arguments specify operations such as pointer movement or writing values.The shared ACT program performs low-level actions, while the environment transition updates the next state.
- Training: Training directly maximizes the likelihood of supervised execution traces, factorizing each step into predictions for the next program, arguments, and halt decision.Variable-length traces are modeled with the chain rule and optimized by gradient ascent.
4 EXPERIMENTS
The experiments evaluate NPI on addition, bubblesort, and 3D-car canonicalization, including sample efficiency, length generalization, and learning new programs with a fixed core.
- Tasks: NPI is evaluated on addition, bubblesort sorting, and 3D-car pose canonicalization across scratch-pad and image environments.The model uses task-specific encoders and environment interfaces for these settings.
- Sample complexity and generalization: 1,216 bubblesort traces train NPI to sort arrays up to size 60, while sequence-to-sequence LSTMs fail to generalize beyond length 25.At size 60 and beyond, NPI errors involve pointer sweeps taking the wrong number of steps, suggesting counting as a limiting factor.
- Sample complexity and generalization: For length-20 sorting, NPI starts learning with 2 examples and sorts almost perfectly with 8, whereas sequence-to-sequence models require 64 to start and over 250 to sort well.The comparison varies the number of training examples while holding array length fixed at 20.
- Canonicalizing 3D models: NPI canonicalizes cars from multiple starting poses and generalizes to appearances not encountered during training.The target pose is frontal with 15° elevation, using image renderings and a convolutional encoder.
- Learning new programs with a fixed core: Adding the MAX program to a multitask NPI updates only two new program embeddings and preserves performance on addition, sorting, and canonicalization.MAX first calls BUBBLESORT and then RJMP to reach the maximum element in the sorted array.
5 CONCLUSION
The NPI learns across dissimilar environments while retaining strong sorting generalization and adding programs without forgetting prior tasks.
- Table 1 reports per-sequence accuracy, including performance after adding max-finding subprograms and results for unseen versus seen car models.
- NPI learns programs in environments with different affordances, including addition, sorting, and 3D car canonicalization.
- NPI exhibits very strong sorting generalization compared with sequence-to-sequence LSTMs.
- A fixed NPI core can learn new programs without forgetting programs learned previously.
6 APPENDIX
The appendix documents NPI execution traces and compares its addition generalization with sequence-to-sequence variants across increasingly long problems.
- 6 APPENDIX: The appendix presents the learned programs for addition, sorting, and 3D car canonicalization, with ACT behavior depending on environment and arguments.
- 6.2 GENERATED EXECUTION TRACE OF BUBBLESORT: BUBBLESORT uses two pointers for adjacent comparisons and swaps, while a third pointer counts BUBBLE calls until execution halts.RESET returns the swapping pointers to the array beginning and advances the counting pointer.
- 6.2 GENERATED EXECUTION TRACE OF BUBBLESORT: Figure 8 traces BUBBLESORT calls while sorting the array [9,2,5].
- 6.3 ADDITIONAL EXPERIMENT ON ADDITION GENERALIZATION: The addition comparison evaluates NPI against s2s-stacked and s2s-easy generalization to longer sequences.The Seq2Seq variants use reversed and stacked input digits or an easier sequence arrangement.
- 6.3 ADDITIONAL EXPERIMENT ON ADDITION GENERALIZATION: 100% accuracy was achieved by NPI on addition lengths up to 3000 after training on 32 examples covering lengths 1,...,20.s2s-easy generalized to just over length 2000, whereas s2s-stacked barely generalized beyond length 5.