PaperlayerView exact revision on arXiv

Source-linked AI summary

CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressions

Runtao Liu, Chenxi Liu, Yutong Bai, Alan Yuille

arXiv:1901.00850v2cs.CVcs.CLcs.LG

TL;DR

Referring expression benchmarks can contain bias and generally expose only final predictions, limiting direct evaluation of intermediate reasoning. The paper introduces CLEVR-Ref+, a controlled synthetic diagnostic dataset with functional-program annotations, evaluates existing models, and proposes IEP-Ref. IEP-Ref outperforms competing methods and supports step-by-step mask visualizations and false-premise handling, while the dataset and model analyses identify important reasoning difficulties and scope boundaries.

  • Problem

    Existing referring-expression datasets can contain exploitable bias and generally lack ground truth for evaluating intermediate step-by-step reasoning.

  • Method

    The paper constructs CLEVR-Ref+ from controlled CLEVR scenes with sampling control and functional-program annotations, then proposes IEP-Ref, a neural module network for referring-expression segmentation.

  • Results

    IEP-Ref outperforms competing referring-expression methods on CLEVR-Ref+ and produces high-IoU intermediate masks that reveal its reasoning process.

  • Takeaways & Limitations

    Attaching IEP-Ref’s segmentation module to intermediate outputs provides a direct quantitative way to inspect neural-module reasoning, including behavior on false-premise expressions.

  • Takeaways & Limitations

    The paper reports inconsistent behavior for a module intended to carry over the previously referred object, which often resembles selecting its complement.

Abstract

from arXiv · show

Referring object detection and referring image segmentation are important tasks that require joint understanding of visual information and natural language. Yet there has been evidence that current benchmark datasets suffer from bias, and current state-of-the-art models cannot be easily evaluated on their intermediate reasoning process. To address these issues and complement similar efforts in visual question answering, we build CLEVR-Ref+, a synthetic diagnostic dataset for referring expression comprehension. The precise locations and attributes of the objects are readily available, and the referring expressions are automatically associated with functional programs. The synthetic nature allows control over dataset bias (through sampling strategy), and the modular programs enable intermediate reasoning ground truth without human annotators. In addition to evaluating several state-of-the-art models on CLEVR-Ref+, we also propose IEP-Ref, a module network approach that significantly outperforms other models on our dataset. In particular, we present two interesting and important findings using IEP-Ref: (1) the module trained to transform feature maps into segmentation masks can be attached to any intermediate module to reveal the entire reasoning process step-by-step; (2) even if all training data has at least one object referred, IEP-Ref can correctly predict no-foreground when presented with false-premise referring expressions. To the best of our knowledge, this is the first direct and quantitative proof that neural modules behave in the way they are intended.

1. Introduction

CLEVR-Ref+ is introduced as a synthetic diagnostic dataset designed to reduce dataset bias and expose step-by-step reasoning in referring expression models. The paper also evaluates existing systems and proposes IEP-Ref, whose intermediate segmentation outputs make its reasoning process directly inspectable.

  • Motivation: Real-world referring expression datasets exhibit exploitable biases and support evaluation mainly on final boxes or masks, not intermediate reasoning.Selecting salient foreground objects without using the expression can yield a much higher-than-random baseline.
  • CLEVR-Ref+: CLEVR-Ref+ uses synthetic scenes, controlled sampling, and automatically annotated reasoning programs to support bias control and step-by-step analysis.The dataset adapts CLEVR questions into referring expressions and changes outputs to referred-object bounding boxes or segmentation masks.
  • CLEVR-Ref+: The dataset extends referring-expression coverage with templates for expression types found in real-world datasets, including ordinal and relational forms.Examples include references such as “The second sphere from left” and expressions involving objects sharing attributes or spatial relations.
  • Evaluation: Existing state-of-the-art models are especially challenged by ordinality, motivating more detailed sub-category diagnosis on CLEVR-Ref+.The evaluation reports overall IoU and accuracy alongside finer-grained analysis.
  • IEP-Ref: IEP-Ref uses an explicit functional program and neural module network, and its segmentation module can visualize intermediate reasoning steps.The same module can be attached to intermediate outputs to produce human-interpretable masks rather than relying only on gradient norms or soft attention maps.

2. Related Works

Related work establishes referring expressions as a joint vision-language task with detection and segmentation variants, while highlighting dataset bias and the value of synthetic diagnostic benchmarks. CLEVR-Ref+ builds on CLEVR’s controlled compositional design while targeting referred-object outputs and reasoning diagnosis.

  • 2.1. Referring Expressions: Referring expressions require understanding an image and a sentence to identify specific objects, with prior work addressing detection and segmentation outputs.Detection systems return one bounding box, whereas segmentation systems return a mask.
  • 2.2. Dataset Bias and Diagnostic Datasets: Referring-expression datasets can contain distributional bias, allowing image-only strategies to outperform random and weakening the faithfulness of performance as a measure of understanding.Real-world images and human-annotated expressions make such bias difficult to control.
  • 2.2. Dataset Bias and Diagnostic Datasets: Synthetic diagnostic datasets use controlled scenes, limited attributes, generated templates, and uniform sampling to mitigate bias and reveal compositional understanding.CLEVR is presented as the model for this design, which CLEVR-Ref+ repurposes for referring expressions.
  • 2.2. Dataset Bias and Diagnostic Datasets: Prior CLEVR models often visualize reasoning poorly despite high VQA accuracy because their textual-answer training domain differs from attention-based diagnosis.Referring expressions make the output and diagnosed visual target more interchangeable.

3. The CLEVR-Ref+ Dataset

CLEVR-Ref+ repurposes CLEVR scenes and functional programs for referring expressions whose outputs are referred objects, with automatic intermediate and final annotations. The dataset adds missing expression types, controls generation to reduce degeneracy, and distinguishes multi-object from single-object references.

  • Dataset construction: CLEVR-Ref+ uses CLEVR scenes, converts questions into referring expressions, and changes outputs from textual answers to bounding boxes or segmentation masks.The dataset retains 70K training images and 15K validation and test images, with 10 referring expressions per image.
  • Dataset construction: Known 3D object locations and properties allow functional programs to identify referred objects and provide ground truth at intermediate reasoning steps.Objects are projected back to the image plane for automatically rendered bounding-box and segmentation-mask annotations.
  • Module additions: CLEVR-Ref+ adds ordinal and visibility expression support after RefCOCO+ analysis found each category in more than 10% of sentences.Examples include “The second woman from left” and “The barely seen backpack”; visibility is defined using foreground-mask coverage thresholds.
  • Module additions: An object is partially visible when foreground masks occupy more than 20% of its bounding-box area, fully visible when the value is exactly 0, and undescribed for intermediate cases.This operational definition avoids ambiguous visibility descriptions between 0 and 20%.
  • Generation procedure: Expressions are generated by sampling a family, text template, and functional-program values, then rejecting inappropriate or empty-reference outcomes.Sampling probabilities are adjusted across categories, and ordinal and visible attributes are not combined.
  • Generation procedure: The dataset avoids near-degenerate filtering by uniformly sampling possible survivor counts and then attribute combinations, while evaluating detection on the single-object subset.That subset contains 222,569 referring expressions, or 32% of the entire dataset.

4. Experiments

Experiments evaluate baseline referring models and IEP-Ref on CLEVR-Ref+, diagnosing attribute, spatial, topology, relation-type, and intermediate reasoning abilities. IEP-Ref performs strongly and supports step-by-step inspection, while ordinality, same-attribute relations, and tree structures remain difficult.

  • Models and evaluation: IEP-Ref uses a generated sequence of CNN-parameterized modules to execute referring expressions compositionally, ending with a segmentation module.Its architecture is adapted from IEP and dynamically constructs a neural network from the referring expression.
  • Overall evaluation: IEP-Ref significantly outperforms RMI on segmentation IoU, while MAttNet performs favorably against SLR on detection accuracy.Detection is evaluated by selecting the correct candidate bounding box; segmentation uses Intersection over Union.
  • Basic referring ability: Detection accuracy is higher for expressions containing color, shape, and visibility, while segmentation shows smaller include–exclude performance gaps.Figure 2 interprets high exclude and low include performance as evidence that a module is more challenging to learn.
  • Basic referring ability: Ordinality is the hardest attribute: RMI IoU falls from 0.91 without ordinality to 0.27 when ordinality is required.The authors suspect ordinality is difficult because it requires global context, unlike local attributes.
  • Spatial reasoning ability: Adding spatial reasoning causes a small performance drop, but models show little additional difference between one, two, and three spatial-relation steps.This suggests that once spatial reasoning is learned, repeated application creates little extra difficulty.
  • Spatial reasoning ability: Tree-structured expressions are generally harder than chain-structured expressions, and same-attribute relations perform much worse than spatial relations.Tree structures require two reasoning paths before merging, while same-attribute relations require global context.
  • Step-By-Step Inspection of Visual Reasoning: Intermediate IoU drops by 0.66 after Unique modules but increases after Same or Relate modules, with masks recovering after Unique.The authors conjecture that Unique acts as preprocessing for subsequent Same and Relate operations.
  • False-Premise Referring Expressions: IEP-Ref predicts no foreground for false-premise expressions despite training data containing referred objects.Among 10,000 generated zero-object expressions, it predicts more than one-quarter foreground pixels more than half the time and at most eight foreground pixels more than one-third of the time.

5. Conclusion

The paper introduces CLEVR-Ref+ and IEP-Ref to reduce dataset bias, expose visual reasoning steps, and evaluate referring expression models. IEP-Ref outperforms competing methods and supports quantitative analysis of intermediate reasoning and false-premise expressions.

  • Contributions: CLEVR-Ref+ minimizes dataset bias through a synthetic setup and provides readily available ground-truth visual reasoning processes.Its synthetic construction controls scenes and makes reasoning annotations available without human annotators.
  • Contributions: IEP-Ref uses a module network approach and outperforms competing referring expression methods by a large margin.
  • Intermediate reasoning: Attaching the Segment module to intermediate steps reveals IEP-Ref’s reasoning process and yields high intermediate-step IoU.This provides quantitative evidence that the neural modules perform their intended roles.
  • False premises: IEP-Ref correctly handles false-premise referring expressions by predicting no foreground when appropriate.
  • Future work: The authors identify transfer to real data as an important direction for future work.

Supplementary Material

The supplementary material provides implementation details, additional performance analyses, and qualitative examples for IEP-Ref and CLEVR-Ref+.

  • Supplementary contents: The supplement covers IEP-Ref architecture details, additional CLEVR-Ref+ model analyses, and qualitative referring-expression examples.Examples include referring expressions together with ground-truth boxes or masks.

A. Network Architectures in IEP-Ref

IEP-Ref organizes its modules into preprocessing, unary, binary, and postprocessing components, with Segment producing the final segmentation mask.

  • Preprocess: Preprocess maps the image to a feature tensor that feeds the Scene module.
  • Unary: Unary modules transform one feature tensor into another, including Scene, Filter X, Unique, Relate, and Same X.
  • Binary: Binary modules combine two feature tensors into one through the And and Or operations.
  • Postprocess: The Segment postprocess module transforms a 128-channel feature tensor into a 1-channel segmentation mask.
  • Architecture provenance: The Preprocess, Unary, and Binary architectures are inherited directly from IEP.

B. More Model Analysis on CLEVR-Ref+

Model performance decreases as scene complexity increases: all evaluated models perform worse when scenes contain more objects.

  • Scene complexity: All models drop in performance as the number of objects in a scene increases.The analysis suggests that models struggle to carry out referring-reasoning steps when scenes contain too many objects.

B.2. Schedule of Acquiring Reasoning Abilities

Performance increased steadily across referring-expression categories during training, without evidence of a staged acquisition schedule for different reasoning abilities. The authors suggest random sampling may explain this pattern.

  • B.2. Schedule of Acquiring Reasoning Abilities: Performance steadily increased across referring-expression categories throughout training.The authors examined performance every 1/6 of the total training iterations.
  • B.2. Schedule of Acquiring Reasoning Abilities: The network showed no apparent schedule for acquiring spatial, logical, or other reasoning abilities.
  • B.2. Schedule of Acquiring Reasoning Abilities: Random sampling during training may have contributed to the absence of a staged acquisition schedule.

B.3. Novel Compositions

The models were evaluated on CLEVR’s compositional generalization data, training on one subset of object combinations and testing on the same and an alternative subset. Detection models showed only a very small validation gap, suggesting strong compositional generalization.

  • B.3. Novel Compositions: Models trained on one subset of object combinations were tested on the same subset in valA and a different subset in valB.
  • B.3. Novel Compositions: Detection models showed a very small performance gap between valA and valB.The authors interpret this small gap as evidence that detection models learned compositionality and generalized well.

C. More Data Examples from CLEVR-Ref+

The supplementary material presents random CLEVR-Ref+ images, referring expressions, and ground truth across the dataset’s referring-expression categories. It includes both detection and segmentation ground truth examples.

  • C. More Data Examples from CLEVR-Ref+: The examples cover at least one instance from each referring-expression category listed in the main paper.
  • C. More Data Examples from CLEVR-Ref+: The supplementary examples include expressions involving spatial relations, ordinal references, shape matching, and chained references.Examples include objects left of or behind another object, seventh or second objects in an ordering, same-shape objects, and chained left-of relations.
  • C. More Data Examples from CLEVR-Ref+: Figure 12 presents referring object detection ground truth examples from CLEVR-Ref+.
  • C. More Data Examples from CLEVR-Ref+: Figure 13 presents referring image segmentation ground truth examples from CLEVR-Ref+.
Loading 1901.00850v2…