PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning with Latent Language

Jacob Andreas, Dan Klein, Sergey Levine

arXiv:1711.00482v1cs.CLcs.NE

TL;DR

Few-shot learning needs useful structure to learn from sparse examples, and the paper asks whether language can provide it. The authors pretrain an interpreter of descriptions and search descriptions to learn new concepts; they report stronger performance than direct approaches across several task types.

  • Problem

    With only a few examples, expressive models risk overfitting, creating a need for extra structure in few-shot learning.

    Source: §2 · p. 2

  • Method

    The method learns a language interpretation model during pretraining, then searches natural-language descriptions to fit new tasks.

    Source: §1 · p. 1

  • Results

    The approach outperforms strong baselines on classification, structured prediction, and reinforcement learning tasks.

    Source: §8 · p. 9

  • Takeaways & Limitations

    Language can improve multitask learning even when the task has no language data at training or test time, and its linguistic bottleneck can support compositional generalization.

    Source: §8 · p. 9

    Some state-of-the-art classification meta-learning approaches are not well-defined for possibly overlapping evaluation classes with only positive examples; the authors provide a robust implementation intended to be as comparable as possible.

    Source: §4 · p. 5

Abstract

from arXiv · show

The named concepts and compositional operators present in natural language provide a rich source of information about the kinds of abstractions humans use to navigate the world. Can this linguistic background knowledge improve the generality and efficiency of learned classifiers and control policies? This paper aims to show that using the space of natural language strings as a parameter space is an effective way to capture natural task structure. In a pretraining phase, we learn a language interpretation model that transforms inputs (e.g. images) into outputs (e.g. labels) given natural language descriptions. To learn a new concept (e.g. a classifier), we search directly in the space of descriptions to minimize the interpreter's loss on training examples. Crucially, our models do not require language data to learn these concepts: language is used only in pretraining to impose structure on subsequent learning. Results on image classification, text editing, and reinforcement learning show that, in all settings, models with a linguistic parameterization outperform those without.

1 Introduction

The paper asks whether language can scaffold the reusable abstractions needed for efficient learning. It uses natural-language descriptions as a parameter space for learning concepts and reports greater accuracy and interpretability than direct few-shot approaches.

1 Introduction

  • The paper is motivated by the possibility that linguistic structure reflects useful abstractions, while reusable abstract structure remains difficult to discover automatically.Source: §1 · p. 1
  • A pretrained language interpretation model maps descriptions to predictors, and new concepts are learned by searching descriptions to minimize training loss.The selected description can then classify new images.Source: §1 · p. 1, Figure 1 · p. 1
  • The learning setup uses natural-language annotations on other tasks to support learning and generalization from a small number of examples on the target task, which has no such annotations.Source: Figure 2 · p. 2
  • Language hypotheses can ease discovery of compositional concepts, reduce overfitting to few examples, and make inferred patterns easier to understand.Source: §1 · p. 2
  • The approach outperforms direct real-valued few-shot methods and methods using language only as extra supervision, while inferred descriptions can aid interpretation and debugging.The authors also report better efficiency and exploration across varied settings.Source: §1 · p. 2

2 Background

Few-shot learning has little labeled data, making expressive models prone to overfitting; existing approaches supply extra structure through constrained hypothesis spaces or additional training data. The paper reviews these alternatives as context for its approach.

2 Background

  • A conventional classifier can use a richly parameterized model indexed by shared weights to map images to labels.Source: §2 · p. 2
  • With only about five or ten input-label pairs, a model expressive enough to capture the task may overfit, so few-shot learning needs extra structure.Source: §2 · p. 2
  • Program synthesis uses discrete formal program descriptions and syntax as a strong prior, but requires hand-engineered primitives and can be difficult to apply to computer vision.Source: §2 · p. 2
  • Multitask learning imports structure from other learning problems rather than defining it manually.Source: §2 · p. 2
  • Meta-learning approaches such as Schmidhuber (1987), Santoro et al. (2016), and Vinyals et al. (2016) use the same data but combine inner adaptation and prediction in one learned model.Source: §2 · p. 3

3 Learning with Language

Learning with latent language uses natural language as an intermediate task representation that is compact and expressive yet learned from data. A language interpretation model is pretrained, then task-specific descriptions are searched to fit training examples.

3 Learning with Language

  • Learning with latent language (L3) uses natural language as an intermediate task representation, combining compact expressivity with learning from data rather than domain engineering.Source: §3 · p. 3
  • During language learning, the model learns to interpret descriptions as functions from inputs to outputs, using natural-language descriptions paired with task data.The shared real-valued parameters are learned during this phase; task learning then optimizes over strings.Source: §3 · p. 3
  • Because strings are discrete and the interpretation model is a black box, the method learns an effective optimization procedure for its language parameters.Source: §3 · pp. 3–4
  • A proposal distribution approximates descriptions given task data; the method samples candidate descriptions, selects the one with lowest loss under the interpretation model, and predicts with it.Source: §3 · p. 4
  • The generic procedure equips related learning problems with a natural-language hypothesis space and is applied to classification, sequence prediction, and reinforcement learning.Source: §3 · p. 4

4 Few-shot Classification

This section evaluates L3 on few-shot visual concept matching: learners see positive examples of a concept and decide whether a held-out image matches it.

4 Few-shot Classification

  • Learners see four positive visual examples and decide whether a fifth, held-out image matches their concept.Source: Figure 3 · p. 4, §4 · p. 4
  • Validation and test tasks split between previously seen concepts with new images and wholly novel concepts, with unrelated distractor shapes in the images.Source: §4 · p. 5
  • Some meta-learning approaches are not well-defined for possibly overlapping evaluation classes when only positive examples are provided.Source: §4 · p. 5
  • L3 consistently improves over multitask, meta-learning, and Meta+Joint baselines for both previously seen and novel concepts.Source: Table 1 · p. 5, §4 · p. 5
  • Predictions can be accurate despite rarely matching ground-truth descriptions, while recovering high-level shapes and spatial relations.Source: Figure 4 · p. 5, §4 · p. 5

5 Programming by Demonstration

The section tests language-based learning on string transformations, using a new dataset with natural-language explanations and regular-expression annotations. L3 leads the compared systems, and further analyses examine sampling and description type.

5 Programming by Demonstration

  • Each task provides five example string transformations, and the learner applies an inferred rule to a sixth string.Source: §5 · p. 6, Figure 5 · p. 6
  • The authors create a dataset by applying sampled random regular transducers to dictionary words and collecting human explanations; it includes 3,000 language-learning tasks and 500 each for validation and testing.The explanations include both natural-language and structured annotations.Source: §5 · p. 6
  • L3 is the best-performing string-editing system, outperforming alternatives that learn directly from examples with or without descriptions.Table 2 reports the percentage of cases where the predicted string exactly matches the reference.Source: Table 2 · p. 6, §5 · p. 6
  • Multiple-sample decoding substantially improves performance, and the full model does better with inferred natural-language descriptions than with regular expressions or ground-truth natural language.Source: Table 3 · p. 6
  • In ordinary evaluation, natural-language learning beats regular-expression learning, and inferred descriptions outperform human-provided ground-truth natural-language descriptions; oracle regular expressions do best.With an oracle regular-expression evaluator, the learned inference procedure gets almost 90% of the way to the synthesis-based approach.Source: §5 · p. 7

6 Policy Search

In treasure hunting, agents must infer landmark-relative treasure rules while navigating environments with sparse, time-consuming feedback. L3 searches over language-structured policies and, after choosing a description, improves reward rapidly, outperforming the multitask baseline.

6 Policy Search

  • Treasure is buried at a fixed but unknown offset from an unknown landmark, and agents must navigate while avoiding water before digging.Language learning uses 250 tasks and evaluation uses 50 additional tasks; some relational concepts are reused, but the environments are distinct.Source: §6 · pp. 7, 9, Figure 7 · p. 7
  • Because agents receive observations only through interaction, the goal is fast adaptation; language-structured search is intended to avoid inefficient exploration among nonsensical random policies.The section motivates structured parameter search as a way to consider behaviors likely to correspond to reasonable policies.Source: §6 · p. 8
  • The interpretation model is a policy conditioned on the current state and a linguistic parameterization, while its proposal model generates descriptions without external direction.The policy uses a discrete action space, with action probabilities proportional to a bilinear function of encoded description and world state.Source: §6 · p. 8
  • After a description is chosen, L3 rapidly surpasses the multitask baseline and improves further with fine-tuning, though its initial policy is not perfect.Figure 8 shows learning curves averaged across evaluation environments after language learning; the max possible reward is 3 points.Source: Figure 8 · p. 8, §6 · p. 9
  • For each sampled description, L3 estimates induced-policy reward from multiple rollouts, selects the highest-scoring description, then fine-tunes its policy.DAgger trains from expert policies during language learning, and vanilla policy gradient adapts to individual environments during concept learning.Source: §6 · p. 8

7 Other Related Work

The related-work discussion distinguishes optimization over natural-language strings from prior uses of strings, annotations, and instructions in learning. It also situates the approach among work on language-guided policies and reinforcement learning.

7 Other Related Work

  • The authors describe their work as the first approach they know of to frame general learning as optimization over natural-language strings.Prior work includes string-valued latent variables for morphology and sentence compression, and natural-language annotations for logical concept descriptions or auxiliary losses.Source: §7 · p. 9
  • The paper distinguishes latent string optimization from using language annotations alongside training examples as supervision or an auxiliary loss.The discussion compares the latter use to the paper's Meta+Joint baseline.Source: §7 · p. 9
  • Related approaches use structured language-like annotations to improve generalizable policies [Denil et al., 2017].Other work supplies natural-language instructions at concept-learning time for high-level strategy, environment dynamics, or exploration [Narasimhan et al., 2017; Harrison et al., 2017].Source: §7 · p. 9

8 Conclusion

The paper presents learning with natural-language parameterizations and reports that it outperforms strong baselines across classification, structured prediction, and reinforcement learning. It concludes that linguistic structure supports compositional generalization and structured exploration, with potential utility beyond tasks that directly use language.

8 Conclusion

  • The approach uses natural-language-parameterized models and outperforms strong baselines on classification, structured prediction, and reinforcement learning tasks.The authors build representation and search models with standard neural encoder–decoder components.Source: §8 · p. 9
  • A linguistic bottleneck that explicitly expresses compositional structure can support stronger generalization to new concepts.The conclusion contrasts this with standard deep-learning architectures, which may recognize new instances of known concepts without generalizing as well to new ones.Source: §8 · p. 9
  • Latent linguistic parameterizations support exploration by restricting sampled behaviors to a class likely a priori to be goal-directed and interpretable.Source: §8 · p. 9
  • Language can help multitask learning even when language data is unavailable at training or test time, and is described as more expressive and easier to obtain than formal supervision.The authors present this as suggesting broader opportunities to use naturally occurring language data in machine learning.Source: §8 · p. 9

A Model and Training Details

The appendix lists implementation settings for recurrent components, classification, programming by demonstration, and policy search. Training uses task-specific optimizer, dimension, sampling, or rollout settings across these tasks.

A Model and Training Details

  • All models use gated recurrent units for their RNN encoders and decoders.The cited implementation reference is Cho et al. (2014).Source: A Model and Training Details · p. 13
  • Few-shot classification uses ADAM with step size 0.0001, batch size 100, and 512-dimensional hidden states, task parameters, and word vectors.Pretraining iterations are tuned on development-set concept-learning performance, and 10 hypotheses are sampled per evaluation task.Source: A Model and Training Details · p. 13
  • Programming by demonstration uses step size 0.001, 512-dimensional hidden states, 128-dimensional task parameters, 32-dimensional word vectors, and 100 sampled hypotheses.Source: A Model and Training Details · p. 13
  • Policy search uses DAgger for pretraining and vanilla policy gradient for concept learning, with 64-dimensional hidden states and task embeddings.Both algorithms use ADAM with step size 0.001 and batch size 5000; 100 hypotheses are sampled and evaluated using 1000 episodes before fine-tuning.Source: A Model and Training Details · p. 13

B Dataset Information

The appendix describes the datasets and data preparation used for ShapeWorld, regular-expression, and navigation tasks. Its ShapeWorld examples also show how an inferred description can differ from the stated description.

B Dataset Information

  • ShapeWorld scenes contain 4 or 5 entities, and descriptions express spatial relations between pairs using shape, color, or both.The dataset has 8 colors, 8 shapes, 30 vocabulary words, and 2643 distinct captions averaging 12.0 words.Source: B Dataset Information · p. 13
  • Regular-expression examples came from Mechanical Turk users who inferred a rule from five transformed strings before describing how to transform a sixth.Descriptions were preprocessed to remove punctuation and tokenize each character literal; the dataset has 1015 rules and 1986 descriptions.Source: B Dataset Information · p. 13
  • The navigation dataset adapts Janner et al. (2017) with related goal tasks, human-annotated language, and a special DIG action required to finish at a chosen goal cell.It contains 74 vocabulary items and 446 distinct hints.Source: B Dataset Information · p. 13
  • For the small regular-expression and navigation datasets, training augments descriptions by swapping templates that preserve high-level semantics across examples.The templates replace recognizable entity names with placeholders, then specialize them with other character literals or landmarks.Source: B Dataset Information · p. 13

C Examples: ShapeWorld

  • In a ShapeWorld example, the stated description is “a red ellipse is to the right of an ellipse,” while the inferred description specifies a red shape and red semicircle.Source: C Examples: ShapeWorld · p. 14

D Examples: Regular Expressions

The examples illustrate inferred string-editing rules and navigation instructions in natural language, including cases where descriptions express the same kind of transformation or goal in different words.

D Examples: Regular Expressions

E Examples: Navigation

  • A navigation example pairs the human description “move to the star” with the inferred description “reach the star cell.”Source: E Examples: Navigation · p. 16
  • Navigation examples phrase a goal as either “reach square one right of triangle” or “reach cell to the right of the triangle.”Source: E Examples: Navigation · p. 16
Loading 1711.00482v1…