PaperlayerView exact revision on arXiv

Source-linked AI summary

Abstraction and Analogy-Making in Artificial Intelligence

Melanie Mitchell

arXiv:2102.10717v2cs.AI

TL;DR

Humanlike abstraction and analogy remain largely unsolved in AI, despite their importance for learning, reasoning, and adaptation. The paper reviews symbolic, deep-learning, and probabilistic-program approaches, then proposes challenge domains and robustness-oriented evaluations for quantifiable, generalizable progress.

  • Problem

    Humanlike abstraction and analogy remain largely absent from current AI, limiting robust transfer and the formation and use of abstract concepts.

  • Method

    The paper reviews selected symbolic, deep-learning, and probabilistic-program approaches and recommends idealized domains with evaluations spanning multiple dimensions of robustness.

  • Results

    The review concludes that current evaluations and systems do not yet establish generalizable abstraction and analogy abilities across diverse domains.

  • Takeaways & Limitations

    Progress requires challenge tasks and evaluation criteria that measure generality across domains and robustness to noise, concept variation, and task complexity.

  • Takeaways & Limitations

    Current methods can learn statistical shortcuts, are often difficult to interpret, and may require more examples than few-shot abstraction should require.

Abstract

from arXiv · show

Conceptual abstraction and analogy-making are key abilities underlying humans' abilities to learn, reason, and robustly adapt their knowledge to new domains. Despite of a long history of research on constructing AI systems with these abilities, no current AI system is anywhere close to a capability of forming humanlike abstractions or analogies. This paper reviews the advantages and limitations of several approaches toward this goal, including symbolic methods, deep learning, and probabilistic program induction. The paper concludes with several proposals for designing challenge tasks and evaluation measures in order to make quantifiable and generalizable progress in this area.

Introduction

Humanlike abstraction and analogy remain open AI problems despite major progress in other areas. The paper reviews prior work and recommends ways to make progress measurable and generalizable.

  • Current AI systems almost entirely lack humanlike concepts and abstractions despite dramatic progress in vision, language, and robotics.
  • Human abstraction supports rich object knowledge, recognition across contexts, and applying concepts to novel situations.
  • Analogy enables abstraction and contributes to concepts, reasoning, and adaptation to situations outside training regimes.
  • The paper reviews selected AI research on abstraction and analogy-making and recommends a path toward machines with these general abilities.
  • The paper surveys human intelligence, several AI approaches in idealized domains, and evaluation steps intended to support generalizable progress.

Abstraction and Analogy-Making in Intelligence

Analogy-making is a ubiquitous mode of human thought that supports abstraction, flexible categorization, and counterfactual inference. The paper reviews symbolic, neural, and probabilistic approaches because current AI lacks these abilities.

  • Analogy helps humans transfer meaning across perceptual and abstract situations, including scientific insights and metaphorical concepts.
  • Analogy-making is constant and lifelong, not a rare form of reasoning.
  • Analogy supports flexible categorization, concept formation, abstraction, and counterfactual inference.
  • Current AI lacks robust generalization, low-shot learning, and the ability to form and use abstract concepts.
  • The review covers structure-mapping, active-symbol, deep-neural, and probabilistic-program approaches without aiming to be exhaustive.

Symbolic Methods

Symbolic approaches represent inputs as structured logical statements and use those representations to support analogy-making. The paper presents early geometric, frame-based, and structure-mapping systems as examples.

  • Symbolic analogy systems typically represent inputs as structured logic statements with concepts expressed as words or phrases.
  • Early symbolic examples include Evans’s geometric-analogy solver, Winston’s story-analogy system, and Falkenhainer et al.’s Structure-Mapping Engine.
  • The section describes SME and the Active Symbol Architecture as symbolic or hybrid approaches to analogy-making.

The Structure-Mapping Engine

SME maps structured logical descriptions using relation-centered, systematic matching, and it contributed to strong RPM performance in a larger system. Its limitations concern representation rigidity and separating representation-building from mapping.

  • Structure-Mapping Engine: SME follows structure-mapping theory by prioritizing relational and higher-order matches over domain-specific content or superficial attributes.
  • Structure-Mapping Engine: SME receives logical propositions describing base and target situations and constructs mappings between them.
  • Structure-Mapping Engine: SME creates candidate pairings, forms consistent global matches, and scores them to select mappings.
  • Limitations: SME models mapping only, assuming situations are already represented logically and relying on other modules for representation, retrieval, and inference.
  • RPM application: In an RPM system, SME mapped row descriptions into higher-level patterns and selected the answer best completing the third row.
  • RPM application: The RPM system solved 56 out of 60 Standard Progressive Matrices problems.
  • Limitations: SME’s syntactic structure requires rigid categories, whereas real-world situations involve context-dependent and ambiguous object, attribute, and relation distinctions.
  • Limitations: The paper argues that representation-building and mapping should be intertwined because SME largely depends on another module to build representations first.

Active Symbol Architecture

The active symbol architecture models analogy-making as abstract perception, with representation-building and analogical mapping interleaved through probabilistic interactions. Copycat demonstrates this approach in letter-string problems but remains limited by manually specified knowledge and architecture.

  • Motivation: Copycat models analogy-making as abstract perception rather than as a process limited to letter strings.The letter-string domain serves as an illustrative setting for a domain-independent model of high-level perception and analogy.
  • Architecture: Copycat interleaves analogical mapping with representation-building through interacting concept-network, workspace, and perceptual-agent components.The concept network stores symbolic prior knowledge, while the workspace supports construction of representations and mappings between input situations.
  • Dynamics: Probabilistic perceptual agents explore and construct structures in a workspace whose temperature shifts processing from random exploration toward deterministic coherence.As the system approaches a coherent solution, temperature falls and agent actions become more deterministic.
  • Comparison: Copycat treats mapping and representation-building as inseparable, contrasting with the Structure-Mapping Engine’s separation of these processes.Its active-symbol architecture combines bottom-up activation, top-down influence, and probabilistic agent behavior.
  • Results: Copycat solved a wide selection of letter-string problems and inspired later approaches to analogy-making, music cognition, image recognition, and cognitive architectures.The system’s emergent dynamics also transition from parallel, random processing to focused, serial processing.
  • Limitations: Copycat’s manually constructed concept network, supplied source-target pair, and complicated parameter tuning limit its generality and concept-learning capabilities.Further research is needed on learning concepts, retrieving situations from memory, and simplifying the architecture and tuning.

Deep Learning Approaches

Deep learning approaches learn representations, abstraction, and analogy-making end-to-end from large datasets, but generated-task biases can permit shortcuts and make accuracy misleading. Evidence from PGM and RAVEN motivates more diverse datasets and evaluation measures beyond accuracy.

  • Approach: Deep learning encodes knowledge as high-dimensional vectors and learns representation, abstraction, and analogy-making through large training sets.Unlike symbolic approaches, these systems generally learn from pixels without relying on built-in knowledge.
  • Procedural datasets: PGM generated 1.2M training problems, 20K validation problems, and 200K test problems for training several deep learning methods, including WReN.Each problem samples relation, object, and attribute triples before rendering the result into pixels.
  • PGM results: 63% accuracy was achieved by WReN on the PGM test set, compared with 12.5% for random guessing.WReN achieved the best performance among the various networks examined in the reported experiments.
  • Dataset limitations: PGM’s large size and limited diversity may disguise model fitting as reasoning that fails to generalize to other scenarios.The concern is that the generation procedure restricts the kinds of problems that can be produced.
  • RAVEN results: A ResNet achieved about 90% accuracy on RAVEN using only the eight answer panels, exploiting a majority-attribute shortcut rather than the matrix.RAVEN’s seven incorrect answers were generated by changing one attribute of the correct answer.
  • RAVEN results: Some methods that performed well on RAVEN performed much more poorly on Impartial-RAVEN, whose answer sets were generated with a more complex sampling method.Impartial-RAVEN was proposed as an “unbiased answer set” and became a target for further accuracy comparisons.
  • Trade-offs: Deep learning avoids built-in knowledge and explicit separation between representation-building and mapping, but requires large datasets and offers limited transparency.Procedural generation can introduce biases that allow networks to perform well without the intended abstraction and analogy mechanisms.
  • Evaluation: Accuracy alone can mislead, motivating evaluation metrics that assess abstraction and analogy-making more directly.The paper argues that training extensively on examples is also questionable when RPMs are intended to measure general human intelligence.

Probabilistic Program Induction

Probabilistic program induction defines a space of possible programs and computes a task-dependent probability distribution over that space. Solving a task involves sampling programs guided by those probabilities.

  • Core approach: Probabilistic program induction defines a space of possible programs, often through a program grammar, and computes a probability distribution over that space for a task.The task is solved by sampling from the program space under the resulting probability distribution.

Recognizing and Generating Handwritten Characters

Lake et al.’s Omniglot system represents handwritten-character concepts as hierarchical generative programs built from primitive pen strokes and learned prior distributions. It matched or exceeded human one-shot classification performance and generated characters that judges typically could not distinguish from human-generated ones.

  • Task: Omniglot studies one-shot learning by requiring recognition and generation of handwritten characters from a single example.The task uses characters from 50 different writing systems.
  • Generative representation: A character concept is represented as a hierarchical program that samples parts, sub-parts, primitive strokes, sequences, and relations.The resulting program can be run with motor noise to render a character token.
  • Learning: The system learned a library of primitive pen strokes and probability distributions over their features from human-drawn characters spanning 30 alphabets.These learned priors were used in a Bayesian framework for one-shot classification and generation.
  • Classification: The Omniglot system matched or exceeded human performance on one-shot classification.For classification, it approximated P(I(t)|I(c)) for each candidate and selected the candidate with the highest probability.
  • Generation: Human judges were typically unable to distinguish the system’s one-shot generated characters from those generated by humans.The system searched for a character program and then ran that program to generate a new example.

Solving Bongard Problems

Bongard problems ask systems to identify abstract concepts distinguishing visual frames, but solving them remains difficult. Probabilistic rule induction addresses this by extracting visual structure and searching a human-designed rule space.

  • Bongard problems require identifying the concept distinguishing six frames on the left from six frames on the right.The concepts may be simple but abstractly represented in the figures.
  • No artificial vision system has come close to solving all 100 original Bongard problems decades after their introduction.
  • Depeweg et al.’s system maps raw pixels from 12 frames to a logic-like rule true on one side and false on the other.This differs from the original task, which requested English-language expressions contrasting the two sides.
  • Rules are generated by probabilistically expanding symbols in a human-designed grammar until all variables are expanded.The grammar defines the space of possible rules for the induction process.
  • Bayesian search ranks candidate rules using P(R|E, G), combining a grammar-based prior P(R|G) with the likelihood P(E|R).The likelihood is one only when the rule separates the two sides and the frames are informative about it.
  • The system extracts objects, attributes, and relationships using simple image processing and a small repertoire of visual types.The authors focus on shapes, properties, and relations appearing frequently in Bongard problems rather than building a general vision system.

Summary of Program Induction Approaches

Probabilistic program induction represents concept learning as program generation, offering flexible and interpretable abstractions. Its generality is limited by substantial human-supplied domain knowledge and probability distributions.

  • Program induction frames concept learning as generating a program, supporting flexible abstraction, reusability, modularity, and interpretability.
  • Bayesian inference combines prior knowledge and preferences with likelihoods while enabling powerful sampling methods.
  • Recent work combines program induction with neural networks, reinforcement learning, neuroscience-inspired models, and psychology-inspired methods.
  • General-purpose use requires substantial human-structured knowledge in program primitives and a domain-specific grammar.
  • These methods also require humans to define prior and likelihood distributions over possible programs, which is not always straightforward.

Abstraction and Reasoning Corpus

ARC is a manually designed benchmark for visual analogy and few-shot abstraction, using compact tasks intended to test core knowledge without language. Despite its promise, leading submissions achieved only about 20% top-3 accuracy and lacked likely generalizability.

  • ARC contains visual analogy tasks in which systems infer a grid transformation from demonstrations and apply it to a test grid.Each demonstration pairs an input grid with its transformed output, while the test requires an analogous transformation.
  • ARC tasks express abstract concepts through visual situations that can vary in unlimited ways.The domain combines features of letter-string analogies, Bongard problems, and Ravens Progressive Matrices.
  • Chollet manually designed 1,000 ARC tasks to compare AI systems and humans without language or other acquired human knowledge.
  • ARC tasks target innate core knowledge about objects, agents and goals, numerosity, and basic spatial-temporal concepts.
  • 400 ARC tasks were public and 600 were reserved for hidden evaluation.
  • About 20% top-3 accuracy was achieved on the hidden set, and no submission used an approach likely to be generalizable.The challenge therefore remained open.

Discussion: How to Make Progress in AI on Abstraction and Analogy

The paper reviews diverse approaches to abstraction and analogy, then recommends challenge domains and evaluation procedures aimed at quantifiable, generalizable progress. Its proposals emphasize idealized core knowledge, cross-domain transfer, limited training, generative tasks, changing hidden evaluations, and robustness.

  • Approaches: Symbolic, active-symbol, deep-learning, and probabilistic-program approaches each provide advantages but retain important limitations.Symbolic systems offer explicit representations and heuristics, while deep learning learns from raw data but needs extensive task-specific training and can exploit shortcuts; probabilistic program induction supports abstraction and interpretability but remains constrained.
  • Evaluation gap: Current evaluations make general progress difficult to assess because systems are usually developed and tested within particular domains.The paper identifies missing assessments of generality across diverse domains and robustness beyond accuracy on fixed test sets.
  • Challenge domains: Idealized, non-linguistic domains make assumed prior knowledge explicit and reduce risks of anthropomorphizing machine performance.The paper acknowledges that such domains may not fully capture the real-world phenomena being modeled.
  • Challenge domains: Challenge tasks should rely on core, non-linguistic knowledge so systems can be compared fairly with one another and with humans.Examples include objects and intuitive physics, agents and goal-directedness, numbers and elementary arithmetic, and spatial geometry.
  • Evaluation design: Systems should be evaluated across multiple diverse domains and should learn core concepts before adapting them across tasks without task-specific training.This is intended to reduce domain overfitting and test general abstraction abilities rather than memorization from extensive domain examples.
  • Evaluation design: Generative tasks, hidden human-curated changing evaluations, and robustness tests can reduce shortcut solutions and probe deeper general abilities.Recommended robustness dimensions include noise, irrelevant distractions, concept variation, and scaling to more complex examples.

Conclusion

The paper argues that humanlike abstraction and analogy-making remain largely open problems despite their importance for general and trustworthy AI. It reviews several approaches and calls for renewed attention to these foundational abilities.

  • Conclusion: Humanlike abstraction and analogy-making are presented as key abilities for learning from few examples, robust generalization, and adaptation across domains and modalities.The paper links these abilities to more general and trustworthy AI systems.
  • Conclusion: The reviewed approaches include symbolic and active-symbol systems, deep learning, and probabilistic program induction.The paper appraises advantages and limitations across these approaches.
  • Conclusion: The problem of constructing machines that form abstractions and analogies remains almost completely open.The paper frames this challenge as central to understanding the foundations of general intelligence.
Loading 2102.10717v2…