PaperlayerView exact revision on arXiv

Source-linked AI summary

The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision

Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, Jiajun Wu

arXiv:1904.12584v1cs.CVcs.AIcs.CLcs.LG

TL;DR

NS-CL addresses learning visual concepts, words, and sentence parsing without explicit supervision. It combines object-based perception, executable semantic programs, and neuro-symbolic execution, using curriculum learning to train them jointly. The model reports accurate CLEVR reasoning and generalization to new compositions, concepts, and program domains, while remaining limited to object properties and relations rather than general events or activities.

  • Problem

    Learning visual concepts, words, and semantic parsing jointly from natural supervision remains a central challenge for vision-language reasoning.

  • Method

    NS-CL learns object-based visual representations and executable language programs from paired images, questions, and answers, bridged by a quasi-symbolic executor and curriculum learning.

  • Results

    NS-CL achieves near-perfect classification of object properties, state-of-the-art CLEVR performance without program annotations, and generalizes to new visual compositions, concepts, and domain-specific languages.

  • Takeaways & Limitations

    The learned visual and language concepts support interpretable visual question answering, compositional generalization, and transfer to image-caption retrieval without extra fine-tuning.

  • Takeaways & Limitations

    The framework focuses on object properties and relations and leaves events, activities, and general VQA datasets for future work.

Abstract

from arXiv · show

We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; instead, our model learns by simply looking at images and reading paired questions and answers. Our model builds an object-based scene representation and translates sentences into executable, symbolic programs. To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation. Analogical to human concept learning, the perception module learns visual concepts based on the language description of the object being referred to. Meanwhile, the learned visual concepts facilitate learning new words and parsing new sentences. We use curriculum learning to guide the searching over the large compositional space of images and language. Extensive experiments demonstrate the accuracy and efficiency of our model on learning visual concepts, word representations, and semantic parsing of sentences. Further, our method allows easy generalization to new object attributes, compositions, language concepts, scenes and questions, and even new program domains. It also empowers applications including visual question answering and bidirectional image-text retrieval.

1 INTRODUCTION

NS-CL jointly learns visual concepts, words, and semantic parsing from images paired with questions and answers, without explicit annotations. Curriculum learning incrementally expands from object concepts to relational and compositional reasoning, supporting broad generalization.

  • NS-CL jointly learns visual perception, words, and semantic parsing from images and question-answer pairs.It uses a neural perception module, a visually grounded semantic parser, and a symbolic program executor.
  • The model learns from natural supervision without image annotations or sentence-level semantic-program labels.Training begins with short questions about simple scenes before progressing to more complex inputs.
  • Curriculum learning progresses from individual object concepts to relational concepts and then highly compositional questions.The model iteratively adapts to increasingly complex scenes and questions.
  • NS-CL achieves state-of-the-art CLEVR performance and generalizes to longer programs, new attribute compositions, novel visual concepts, and image-caption retrieval.The reported generalization includes scenes with more objects, CLEVR-CoGenT compositions, fast learning of a new color, and transfer without extra fine-tuning.

2 RELATED WORK

Related work spans joint vision-language learning, visual question answering, object-based scene representations, and semantic parsing. NS-CL combines interpretable object-based representations with executable programs while learning from natural supervision.

  • Prior work learns visual concepts from image captions, visually grounded question-answer pairs, dense scene descriptions, video captions, and video-text alignment.
  • Visual question answering requires understanding visual content and language, while CLEVR evaluates reasoning models using synthetic scenes and program-generated questions.
  • Existing visual reasoning systems use convolutional feature maps, attention representations, or object-based representations, with some requiring fully annotated scenes.
  • NS-CL learns object-based visual representations from questions and answers, unlike approaches requiring annotated visual attributes or scene supervision.
  • Explicit programs improve interpretability but commonly require extra supervision, whereas NS-CL parses questions into programs and executes them on object features.

3 NEURO-SYMBOLIC CONCEPT LEARNER

NS-CL bridges object-based visual perception and executable semantic parsing with a differentiable quasi-symbolic executor. Its curriculum and modular training jointly learn concepts and language from paired images, questions, and answers.

  • NS-CL constructs object-based scene representations, translates questions into executable programs, and uses symbolic reasoning to bridge visual and language learning.
  • Visual attributes act as neural operators that map object representations into attribute-specific embedding spaces, where concept vectors classify attributes through similarity.Relational concepts are classified similarly using representations formed from pairs of objects.
  • The perception module detects objects and extracts latent representations, while the semantic parser produces hierarchical programs in a VQA-specific DSL.The DSL contains composable operations such as filtering objects by concepts and querying attributes.
  • The executor applies deterministic functional modules to the scene representation and provides a transparent execution trace while remaining differentiable with respect to learned representations.
  • Intermediate object sets are represented as probabilistic attention masks, enabling differentiable execution over scene objects.
  • The visual module is optimized through executor-based gradients, while the semantic parser uses REINFORCE with correctness as its reward and off-policy search for variance reduction.
  • Curriculum training proceeds from object concepts to relational questions, complex questions with fixed perception, and joint fine-tuning.The authors report that this staged procedure is essential for learning NS-CL.

4 EXPERIMENTS

NS-CL learns interpretable visual concepts and performs visual reasoning from natural supervision, achieving strong accuracy, data efficiency, and generalization across visual and language settings. Its learned representations also transfer to image-text retrieval and new program domains.

  • Visual concept learning: NS-CL learns visual concepts with near-perfect classification accuracy of approximately 99% for object properties.Spatial-relation results are relatively lower because CLEVR does not have uniform distributions of spatial relations.
  • Visual concept learning: NS-CL outperforms IEP by 8% and attention-based baselines by more than 2% on diagnostic visual-concept evaluation.The evaluation compares object-based concept representations against convolutional and neural-attention baselines.
  • Data-efficient visual reasoning: Using only 10% of the training images, NS-CL achieves results comparable to baselines trained on the full dataset.The paper attributes this data efficiency to disentangling visual concept learning from programmed symbolic reasoning.
  • Data-efficient visual reasoning: NS-CL achieves state-of-the-art CLEVR performance without program annotations and comparable performance to TbD-Nets trained with 700K programs.NS-VQA performs better on CLEVR but requires annotated visual attributes and program traces, unlike NS-CL.
  • Generalization to new attributes and compositions: NS-CL reaches 93.9% QA accuracy on CLEVR-CoGenT Split B, outperforming IEP by 4.6% and TbD by 6.1%.The authors associate this generalization with the Color operator’s more efficient learning of new visual concepts.
  • Extending to other program domains: Without sentence-semantics annotations, NS-CL outperforms CNN-LSTM by 30% on full-dataset image-caption retrieval and transfers learned visual concepts to a new DSL.The visual scenes remain fixed while only semantic parsing is learned for the new domain.

5 DISCUSSION AND FUTURE WORK

NS-CL jointly learns visual concepts, words, and sentence parsing from images and paired questions and answers without explicit object class labels. The discussion identifies extensions toward realistic 3D scenes, complex natural language semantics, and interactive domains.

  • Discussion: NS-CL learns visual concepts, words, and semantic parsing from natural supervision consisting of images and paired questions and answers.The framework does not require explicit object class labels.
  • Discussion: The learned concepts support question answering and generalization to new visual compositions, visual concepts, and domain-specific languages.
  • Future Work: Realistic scenes remain a future direction because constructing 3D object-based representations requires further exploration.
  • Future Work: Integrating formal semantics into complex natural-language processing is identified as meaningful future work because NS-CL assumes a domain-specific language.
  • Future Work: Extending the framework to video understanding and robotic manipulation requires semantic representations for actions and interactions beyond static spatial relations.

A CLEVR DOMAIN-SPECIFIC LANGUAGE AND IMPLEMENTATIONS

This section describes the CLEVR domain-specific language and the modular pipeline that parses questions into executable programs and executes them over probabilistic object representations. It also details concept classification, recursive parsing, type handling, and differentiable execution.

  • CLEVR DSL: The CLEVR DSL defines available operations and their type system for visual question answering.The operations are summarized in Table 6 and the type system in Table 7.
  • Semantic Parsing: The semantic parser encodes questions, extracts concept words, and recursively generates hierarchical programs using operation and concept decoders.Output encoders create sub-states for non-concept operation inputs.
  • Semantic Parsing: The parser’s recursive procedure initializes an empty program, predicts its operation, decodes any required concept, and recursively parses each non-concept input.The algorithm is presented as a string-to-tree semantic parser.
  • Semantic Parsing: Concept words are extracted with hand-coded rules and represented by word embeddings for program parsing, while the visual module uses separate concept embeddings.The CLEVR concept vocabulary is assumed known; automatic discovery is left for future work.
  • Program Execution: Object and object-set variables are length-n vectors whose entries represent probabilities that scene objects are referred to or belong to a set.The Unique operation casts an object set to an object using a softmax over logits.
  • Program Execution: Neural attribute operators and concept embeddings classify object attributes, relational concepts, and pairwise attribute equivalence probabilistically.Color classification produces object vectors, relational classification produces n × n matrices, and same-attribute classification produces pairwise matrices.
  • Program Execution: Quasi-symbolic execution stores probabilities in log space to improve numerical stability.

D OPTIMIZATION OF THE SEMANTIC PARSER

NS-CL optimizes semantic parsing in a non-smooth program space through off-policy search, curriculum learning, and suppression of spurious programs. The curriculum progresses from object concepts to relations and full questions, while validation shows semantic parsing exceeds 99.9% QA accuracy.

  • Program search: Off-policy program search addresses optimization over the non-smooth space of valid CLEVR programs.The search computes gradients for the semantic parser parameters over candidate programs generated for each question.
  • Spurious program suppression: Spurious supervision arises because distinct programs can be intrinsically equivalent or incorrect programs can answer correctly on a specific scene.The paper distinguishes intrinsic ambiguity from extrinsic spuriousness, illustrating each with program pairs.
  • Spurious program suppression: REINFORCE-inspired weighting reduces updates to spurious programs as sampling probabilities weight the likelihood objective.The method uses a sampling importance term Pr[p] so programs receive less update during training when they become less probable.
  • Curriculum learning: Curriculum learning adds visual concepts and increasingly complex questions across three stages, from object concepts to relations and the full CLEVR question set.Lessons are selected by latent program depth, with maximum depth 9 on CLEVR.
  • Curriculum learning: Removing curriculum over scene size leaves perception near random-guess accuracy, while removing program-complexity curriculum prevents joint training from converging.These ablations establish that both curriculum dimensions are important for training.
  • Evaluation: The semantic parser achieves > 99.9% QA accuracy on the validation split when program quality is evaluated by execution against ground-truth objects.Invalid or ambiguous programs count as incorrect under this evaluation.

F.2 IMPACTS OF THE IMAGENET PRE-TRAINING.

ImageNet pre-training has limited impact on learned concept classification, with Shape the notable exception. Random initialization mainly reduces Shape accuracy on the validation set.

  • ImageNet pre-training is the only extra supervision used for the visual perception module.
  • Randomly initializing the perception module leaves learned-concept classification accuracies almost unchanged except for Shape.
  • Shape accuracy drops from 98.7 to 97.5 on the validation set without ImageNet pre-training.

F.3 DATA EFFICIENCY AND OBJECT-BASED REPRESENTATIONS

NS-CL combines object-based representations with symbolic execution and is more data-efficient than neural baselines on CLEVR. Its reported accuracy reaches 99.2% using the CLEVR training set, while mask-guided attention mainly accelerates convergence.

  • Object-based representations: NS-CL uses object features alongside scene context, whereas object-based TbD and MAC baselines replace their original image-level reasoning inputs.The baselines use fixed stacks of 12 object features, padding scenes with null objects when necessary.
  • Object-based representations: Object-based TbD and MAC perform worse than their original models, reflecting architectural limitations in relational reasoning and quantity handling.The cited analysis notes that attention mechanisms are order-invariant but unsuitable for counting objects.
  • Mask-guided baselines: Mask-guided attention does not improve overall performance but produces noticeably faster convergence during training.The masks provide prior object-location information to guide attention.
  • Data efficiency: NS-CL is more data-efficient because it learns visual concepts and question interpretation, while TbD and MAC additionally learn program execution.NS-CL executes symbolic programs over learned concepts rather than learning all execution behavior from data.
  • Data efficiency: 99.2%: NS-CL accuracy using the CLEVR training set after selecting hyperparameters on a 5% hold-out validation subset.The hold-out consists of 3,500 images.

G.1 MINECRAFT DATASET

The Minecraft testbed extends NS-CL to different visual contexts, object configurations, and reasoning operations. NS-CL learns from images and question-answer pairs and outperforms the strongly supervised NS-VQA baseline by 5% overall accuracy.

  • Dataset: Minecraft scenes contain 3 to 6 objects selected from 12 categories, with four facing directions on a 2D plane.The dataset includes 10,000 generated scenes.
  • New reasoning operations: The Minecraft domain adds FilterMost and BelongTo operations to the domain-specific language.FilterMost selects the most qualifying object from a set, while BelongTo tests whether an object belongs to a set.
  • Results: 5%: NS-CL outperforms NS-VQA in overall Minecraft accuracy despite NS-VQA receiving strong supervision for scene representation and program traces.The authors attribute NS-VQA’s lower performance to noisy bounding-box localization caused by object occlusion.
  • Results: The Minecraft results are reported as comparable to baselines trained with full program annotations.Table 10 summarizes this comparison.

G.2 VQS DATASET

On the VQS dataset, NS-CL learns from visually grounded questions and answers, using parsed programs and symbolic reasoning to achieve performance comparable to established VQA baselines while providing transparent error inspection.

  • Dataset: VQS contains visually grounded questions associated with multiple human-annotated image regions needed to answer them.The dataset is a subset of VQA 1.0.
  • Setup: The VQS setup trains on 63,509 images, validates on 5,000 additional training images, and selects answers from 18 candidates.Models are tested on the test split using a multiple-choice VQA formulation.
  • Method: NS-CL extracts latent programs and learnable concepts from natural-language questions using a pretrained syntactic dependency parser.Concept embeddings are initialized with bag-of-words representations over GloVe word embeddings.
  • Baselines: The comparison includes MLP and MAC, with both baselines scoring candidate answers for the VQS multiple-choice task.MLP uses image, question, and candidate-answer representations; MAC receives the question and answer together.
  • Results: NS-CL achieves comparable results with the MLP baseline and the MAC network designed for visual reasoning.Its symbolic reasoning also provides transparent reasoning over natural images and language and helps diagnose error sources.

H VISUALIZATION OF EXECUTION TRACES AND VISUAL CONCEPTS

NS-CL’s execution traces make symbolic reasoning inspectable across CLEVR, Minecraft, and VQS, while visualizations show learned concepts spanning categories, attributes, and relations.

  • Execution traces: NS-CL provides interpretable execution traces across CLEVR, Minecraft, and VQS, and can reject ambiguous or invalid programs with exceptions.For example, a color question is ambiguous with multiple cylinders and invalid when no cylinders exist.
  • Visual concepts: Concept visualizations on CLEVR and Minecraft cover object categories, attributes, and relations using validation or test examples retrieved by concept queries.A threshold filters the returned images and objects; quantitative CLEVR concept evaluations are referenced in Tables 2 and 5.
  • Execution traces: The execution traces expose specific failure sources, including material misclassification in CLEVR and missed-object detection in Minecraft.The Minecraft example concerns a pig hidden behind a large tree.
  • Execution traces: On VQS, execution traces show correct answers, object-counting behavior, and a case where the model answers correctly despite locating the wrong object.These traces illustrate both symbolic transparency and the possibility of localized reasoning errors.
  • Visual concepts: The CLEVR and Minecraft figures include qualitative visualizations of concepts learned by NS-CL.The figures separately present concepts learned on the two datasets.
Loading 1904.12584v1…