PaperlayerView exact revision on arXiv

Source-linked AI summary

The Omniglot challenge: a 3-year progress report

Brenden M. Lake, Ruslan Salakhutdinov, Joshua B. Tenenbaum

arXiv:1902.03477v2cs.AIcs.CVcs.LG

TL;DR

People can learn concepts from very few examples and use them across tasks, motivating Omniglot as a test of learning beyond recognition. This paper reviews subsequent approaches and renews the challenge with minimal training splits; it concludes that progress remains uneven and no single model yet meets the full multi-task challenge.

  • Problem

    Human learners can acquire concepts from one or a few examples and use them beyond recognition, while many AI benchmarks provide far more training experience.

    Source: Introduction · p. 1

  • Method

    The paper reviews progress on Omniglot and re-releases its drawing data with two five-alphabet minimal splits to support research on human-like learning.

    Source: Introduction · p. 3, One-shot classification · p. 4

  • Results

    Approaches remain far from the full challenge of one model performing all five tasks; on the original one-shot task, BPL has 4.5% error versus 7.3% for the best neurally grounded architecture.

    Source: Discussion · p. 6

  • Takeaways & Limitations

    The authors identify adding compositionality and causality to neurally grounded architectures as a promising avenue toward the Omniglot challenge.

    Source: Discussion · p. 6

    Discriminative methods perform only the task they were trained for, so they are not a plausible account of human task-general conceptual representations on their own.

    Source: One-shot classification · p. 4

Abstract

from arXiv · show

Three years ago, we released the Omniglot dataset for one-shot learning, along with five challenge tasks and a computational model that addresses these tasks. The model was not meant to be the final word on Omniglot; we hoped that the community would build on our work and develop new approaches. In the time since, we have been pleased to see wide adoption of the dataset. There has been notable progress on one-shot classification, but researchers have adopted new splits and procedures that make the task easier. There has been less progress on the other four tasks. We conclude that recent approaches are still far from human-like concept learning on Omniglot, a challenge that requires performing many tasks with a single model.

Introduction

Omniglot was released as a testbed for human-like learning from very few examples, where concepts should support more than recognition. The challenge is one model performing five tasks at a human level, and the authors renew it with smaller training splits after finding uneven progress.

Introduction

  • Omniglot contains handwritten characters from 50 alphabets and was designed to study learning a new concept from a single example.The authors describe the characters as complex, high-dimensional concepts that remain tractable for studying human and machine learning.Source: Introduction · p. 1
  • Large AI datasets offer far more training experience than people need, while human learners apply concepts across tasks beyond recognition.The authors present Omniglot as a benchmark for studying learning from one or a handful of examples.Source: Introduction · p. 1
  • The challenge is one model performing five concept-learning tasks at a human level, rather than classification alone.Bayesian Program Learning was introduced to address the suite using compositionality, causality, and learning to learn.Source: Introduction · p. 1, Figure 1 · p. 2
  • The dataset was widely adopted, but progress on one-shot classification was hard to assess because newer splits and procedures made the task easier; the other four tasks drew less attention.No new algorithm had attempted all five tasks together.Source: Introduction · pp. 1–2
  • The authors re-release the drawing data in a new format and highlight two five-alphabet minimal splits intended to better represent prior writing and drawing experience.They hope the renewed challenge will encourage more human-like learning research across AI and cognitive science.Source: Introduction · pp. 2–3

One-shot classification

One-shot classification results depend strongly on the task split and the amount of background training, so scores from different setups are difficult to compare. On the original within-alphabet task, prototypical networks trail BPL, and the gap widens with minimal background training.

One-shot classification

  • Human participants achieved a 4.5% error rate on 20-way within-alphabet classification, an upper bound because they responded quickly and lacked performance incentives.BPL used 964 background characters from 30 alphabets and achieved 3.3% error; the ConvNet error rate was 13.5%.Source: One-shot classification · p. 3
  • 6.2% error with matching networks, 4.0% with prototypical networks, and 4.2% with MAML were reported on the between-alphabet task with class augmentation.These results are difficult to compare with the original, harder within-alphabet task because the setup adds background characters and augments classes four-fold.Source: One-shot classification · p. 3, Table 1 · p. 3
  • On two minimal splits containing only five background alphabets, BPL errors were 4.3% and 4.0%, versus 30.8% and 29.3% for the meta-learner; class augmentation brought network error to 6.0%.Source: One-shot classification · p. 4
  • ARC achieved 2.5% error between alphabets and 1.5% within alphabets, using class augmentation and random image deformations.The authors note that both augmentation types were needed for ARC training.Source: One-shot classification · p. 4
  • Human-like training remains a challenge: people typically experience only one or a few alphabets, whereas many approaches use larger background sets and class augmentation.The authors call for exploring settings with few examples per class and few background classes.Source: One-shot classification · pp. 3–4
  • Discriminative methods perform only their trained task, while human conceptual representations are more flexible and task-general.The authors therefore regard discriminative learning alone as an implausible account of human conceptual learning.Source: One-shot classification · p. 4

Generating new exemplars

BPL generates exemplars that approach human indistinguishability, while deep generative models show mismatches in variation. The paper points to causal and compositional structure, including learning from drawing demonstrations, as a route toward more human-like generation.

Generating new exemplars

  • 52% ID level on exemplar generation was near the 50% ideal, with only 3 of 48 judges reliably above chance.The visual Turing test compared human and machine drawings.Source: Generating new exemplars · p. 4
  • SG and VHE sometimes generate compelling examples but vary too much, while RCN shows too little variation relative to human examples.Deep generative samples can also have unarticulated strokes and have not been quantitatively compared with human-generated examples.Source: Figure 2 · p. 5, Generating new exemplars · p. 5
  • Current neural models train on images alone, whereas BPL also learns from drawing demonstrations and incorporates some causal and compositional components.The authors suggest training neural models on drawing demonstrations as one way to incorporate stronger structure.Source: Generating new exemplars · p. 5
  • BPL draws realistically enough to confuse most judges (59% ID), and the authors argue that genuine causal and compositional components are key to solving all five tasks.They re-release drawing demonstrations as a resource for training generative models with stronger compositionality and causality.Source: Generating new exemplars · pp. 5–6

Generating new concepts

BPL can generate plausible new characters from examples of an unfamiliar alphabet, while deep generative approaches have had limited success. Unconstrained sampling produces high-quality samples, but its creative novelty remains unevaluated.

Generating new concepts

  • 49% ID level in a visual Turing test meant judges had little success distinguishing BPL-generated characters from human-generated ones.BPL generates new characters after seeing a few examples from a novel alphabet.Source: Generating new concepts · p. 6
  • Deep generative models have had limited success generating new characters from an alphabet, often producing blurry, unarticulated examples.The authors describe this as a wide-open challenge for deep neural networks.Source: Generating new concepts · p. 6
  • Unconditional Omniglot sampling can produce high-quality samples, but these approaches have not been evaluated for creativity rather than copying training characters.The authors consider this task within reach of current neural networks; they identify architectures that perform all tasks together as the greater challenge.Source: Generating new concepts · p. 6

Discussion

The review finds progress on one-shot learning but concludes that recent approaches remain far from the full Omniglot challenge. It argues for deeper, more flexible representations and models that can perform many tasks together.

Discussion

  • Across tasks, BPL remains ahead on the cited one-shot classification comparisons: 4.5% vs 7.3% error on the original task and 4.2% vs 23.2% with minimal background training.The comparison is BPL versus the best neurally-grounded architecture on the original task, and BPL versus the best neural-network results with minimal background training.Source: Discussion · p. 6
  • BPL achieved 52% ID on exemplar generation (55% with minimal background training), 59% on parsing, 49% on generating concepts from a type, and 51% on unconstrained generation.The authors identify 50% ID as ideal performance on visual Turing tests.Source: Discussion · p. 6
  • Human concept learning combines broad expertise across concept types with deep representations usable in many ways, while algorithms often prioritize breadth over depth.The paper notes that matching networks, for example, covered three datasets but only one task: one-shot classification.Source: Discussion · p. 6
  • The authors call for human-level learning approaches that develop flexible representations and let one model perform a range of tasks together.They point to compositionality and causality as promising components for addressing the Omniglot challenge.Source: Discussion · p. 6
  • The five Omniglot tasks are only a starting point; the authors encourage additional tasks beyond classification, including filling occluded images and constructing characters from verbal descriptions.They suggest that new tasks can bring cognitive science and AI together in developing algorithms.Source: Discussion · pp. 6–7
Loading 1902.03477v2…