PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Naturalist: Generating Fine-Grained Image Comparisons

Maxwell Forbes, Christine Kaeser-Chen, Piyush Sharma, Serge Belongie

arXiv:1909.04101v3cs.CLcs.CV

TL;DR

Fine-grained distinctions between visually similar animals can be difficult for non-experts to identify and explain. This paper introduces a dataset of bird-image comparisons and a model that generates comparative descriptions from image pairs. The comparative module improves generations, and the authors suggest such explanations may help non-experts distinguish closely related instances.

  • Problem

    Visually similar species can be difficult for casual observers to label, and non-experts may need training to know what visual details to look for.

    Source: §1 · pp. 1–2

  • Method

    The paper collects paragraph-length, everyday-language comparisons between bird photographs and proposes Neural Naturalist, which combines image encodings with a comparative module to generate descriptions.

    Source: §1 · pp. 1–2

  • Results

    Adding a comparative module over the combined image representations yields better generations than passing those representations directly to the decoder.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors suggest comparative explanations from neural models may help non-experts distinguish closely related instances in fine-grained visual domains.

    Source: §1 · p. 2

    The model sometimes hallucinates visual differences, and the authors suspect learned feature associations can outweigh visual evidence.

    Source: Error Analysis · p. 9

Abstract

from arXiv · show

We introduce the new Birds-to-Words dataset of 41k sentences describing fine-grained differences between photographs of birds. The language collected is highly detailed, while remaining understandable to the everyday observer (e.g., "heart-shaped face," "squat body"). Paragraph-length descriptions naturally adapt to varying levels of taxonomic and visual distance---drawn from a novel stratified sampling approach---with the appropriate level of detail. We propose a new model called Neural Naturalist that uses a joint image encoding and comparative module to generate comparative language, and evaluate the results with humans who must use the descriptions to distinguish real images. Our results indicate promising potential for neural models to explain differences in visual embedding space using natural language, as well as a concrete path for machine learning to aid citizen scientists in their effort to preserve biodiversity.

1 Introduction

Fine-grained visual differences can be hard for non-experts to explain, while expert-written field guides are costly to produce. Birds-to-Words collects adaptive, everyday-language comparisons, and Neural Naturalist generates them from image pairs.

1 Introduction

  • The paper introduces Birds-to-Words, a dataset of paragraph-length comparisons between bird photographs, and Neural Naturalist, a model that generates comparisons from two images.The model combines image representations through a joint encoding and comparative module before decoding.Source: §1 · pp. 1–2
  • Annotators naturally describe familiar body parts and adjust detail to the visual and taxonomic distance between images.Descriptions can range from broad distinctions such as a red body to a tiny stripe above the eye.Source: §1 · p. 2
  • Non-experts can provide high-quality comparisons without domain expertise, addressing the expert knowledge needed to create field guides.Field guides are costly to create because writing them requires expertise in class-level distinctions.Source: §1 · pp. 1–2
  • The comparative module improves generated comparisons over using the combined image representations without that module.The model uses a CNN to embed images, then a joint encoding and comparative module before a Transformer decoder.Source: §1 · p. 2
  • Comparative descriptions could help non-experts distinguish closely related instances, including amateur naturalists identifying species.The paper frames its approach for pairs of in-domain images with detailed language that remains understandable to laypeople.Source: §1 · p. 2
  • Earlier work studied class-conditioned captions, comparative language, or image pairs with high pixel overlap; this work studies everyday comparisons without pixel overlap.Related approaches include Hendricks et al. (2016), Vedantam et al. (2017), and Jhamtani and Berg-Kirkpatrick (2018).Source: §1 · p. 2

2 Birds-to-Words Dataset

Birds-to-Words is built from bird-image pairs selected to span visual and taxonomic distances, with paragraph-length descriptions elicited from annotators. Its sampling and annotation design aims to yield fine-grained comparisons across a broad range of related instances.

2 Birds-to-Words Dataset

  • Pivot-branch sampling selects pairs that are distinguishable but closely related, combining random pivots with branches chosen for fine-grained comparisons and broad coverage.The sampling procedure aims for a middle ground between very different pairs and near-identical views.Source: §2 · pp. 2–3

2.1 Domain

  • The domain is restricted to Aves, providing 1.7 million research-grade images and taxonomic labels while keeping instances structurally comparable.The dataset reports 3,347 image pairs and 16,067 paragraphs.Source: §2.1 · p. 3

2.2 Pivot Images

  • 405 pivot images are selected from 9,000 species, with species sampled uniformly to reduce observation bias toward common birds.These photographs serve as the starting images for branching comparisons.Source: §2.2 · p. 3

2.3 Branching Images

  • Each pivot branches to 12 comparison images: two selected by visual similarity and ten sampled across five taxonomic ranks.The taxonomic samples include two images each from the same species, genus, family, order, and class; the procedure yields 4,860 raw pairs.Source: §2.3 · p. 4

2.4 Language Collection

  • Annotators write five paragraphs per pair, comparing animal appearance without naming species or describing backgrounds, scenery, or pose.Annotations are retained only when both images meet the minimum clarity-rating threshold.Source: §2.4 · p. 4
  • 3,347 image pairs yielded 16,067 paragraphs after pairs with insufficient image-clarity ratings were discarded.The dataset contains 40,969 sentences, averaging 2.6 sentences per paragraph.Source: §2.1 · p. 3, §2.4 · p. 4

3 Neural Naturalist Model

Neural Naturalist generates paragraph-length comparisons between two images, combining image embeddings through a joint encoding and comparative module before decoding language. Its encoder-decoder design supports descriptions of image differences at varying levels of detail.

3 Neural Naturalist Model

  • The model generates a paragraph comparing two input images by encoding both images jointly, transforming their representations comparatively, and decoding a description.The decoder attends to the comparative representation at every time step.Source: §3 · p. 4, §3.4 · p. 6

3.1 Image Embedding

  • Shared-weight CNNs extract local feature grids from each image and flatten them into matrices for subsequent processing.The considered CNN architectures are ResNet and Inception.Source: §3.1 · p. 5

3.2 Joint Encoding

  • The joint encoding combines both image embeddings, a mutated combination, or both, testing whether simple mutations effectively combine image representations.Mutations include addition, subtraction, elementwise maximum, and elementwise product.Source: §3.2 · p. 5

3.3 Comparative Module

  • The comparative module can pass the joint encoding through directly or apply an N-layer Transformer encoder whose self-attention guides comparisons across image representations.The encoder output C is built from the joint encoding J through its layers.Source: §3.3 · p. 6

3.4 Decoder

  • An N-layer Transformer decoder produces token distributions, with intermediary multi-headed attention accessing the encoder output C at every time step.Training text X is positionally encoded and masked in the decoder’s first multi-headed attention.Source: §3.4 · p. 6

4 Experiments

Human evaluation shows Neural Naturalist distinguishes visually distinct species but struggles with the most similar pairs; automatic metrics favor its generations on BLEU-4 and ROUGE-L, though CIDEr-D rewards repetitive outputs. The analysis also finds adaptive description granularity and coherent comparisons alongside hallucinated or misassigned features.

4 Experiments

  • The dataset is split 80/10/10 by pivot image, keeping pivot-image species unique across train, validation, and test.Source: §4 · p. 6

4.1 Baselines and Variants

  • Comparisons include frequency, text-only, nearest-neighbor, CNN-LSTM, and Neural Naturalist variants, including a model without a comparative module.Source: §4.1 · p. 6

4.2 Quantitative Results

  • Neural Naturalist achieves the highest BLEU-4 and ROUGE-L scores, while CIDEr-D favors repetitive descriptions that say the animals appear the same.At its highest CIDEr-D score, the model produced a variant of that description for 95% of paragraphs.Source: §4.2 · p. 6, Table 2 · p. 7
  • The best-performing configuration pairs elementwise multiplication with a 6-layer comparative Transformer, a 6-layer decoder, and beam search.Joint encoding and comparative-module choices need to be balanced: multiplication performs best with a deep comparative module, while subtraction works moderately well alone.Source: §4.2 · p. 7, Table 3 · p. 8

Positive Findings

  • Descriptions adapt their granularity to image differences, using fine-grained distinctions for similar birds and broader contrasts for dissimilar ones.Source: Positive Findings · p. 8
  • The model produces varied comparison structures and coherent multi-statement paragraphs that cover different features without repeating topics.Examples combine contrasts across neck, legs, beak, belly, and wings.Source: Positive Findings · pp. 8–9

4.2 Quantitative Results

  • The model succeeds on visually distinct species but struggles on highly similar comparisons, where humans retain a substantial advantage.The evaluation covered 120 test examples, and annotators scored descriptions by whether they correctly identified the two images.Source: §4.2 · p. 7, Table 4 · p. 9

Error Analysis

  • Common errors include hallucinating differences, overlooking obvious distinctions, and assigning features to the wrong animal.The authors suspect learned feature associations can outweigh visual evidence and note there is no explicit bookkeeping mechanism for feature references.Source: Error Analysis · p. 9

5 Related Work

Prior work explores class- and image-discriminative captions, change captioning, and two-image reasoning; this work focuses on generating comparative language for images without pixel overlap.

5 Related Work

  • Birds-to-Words compares two bird photos and uses everyday language without an anatomical guide, unlike CUB Captions and CUB-Justify, which annotate individual photos.The paper describes its language as more complex, with diverse comparative structures and implied semantics.Source: §5 · p. 9
  • The approach complements single-image and discriminative captioning by using comparative language to bridge varying granularities of visual diversity.The related work also distinguishes this generation task from NLVR2's natural-language reasoning task.Source: §5 · p. 9
  • Unlike change-captioning work on images with few changing pixels, this work compares images with no pixel overlap and uses stratified sampling to select pairs.The paper names Spot-the-diff and other recent change-captioning work as methodological neighbors.Source: §5 · p. 9

6 Conclusion

The conclusion presents Birds-to-Words and Neural Naturalist as tools for comparative explanations of fine-grained visual differences, with descriptions that are adaptive and understandable to everyday readers.

6 Conclusion

  • Birds-to-Words provides paragraph-length, adaptively detailed descriptions in everyday language, alongside Neural Naturalist for generating comparative explanations of fine-grained visual differences.The authors hope continued study will produce models that can aid humans in domains such as citizen science.Source: §6 · p. 9

A Algorithmic Approach to Dataset Construction

The dataset-construction approach balances coverage, relevance, comparability, and annotation efficiency under practical constraints of data quality and cost.

A Algorithmic Approach to Dataset Construction

A.1 Goals

  • The goals are broad domain coverage for generalization, useful descriptions given model capabilities, comparable image pairs, and the largest dataset feasible under a fixed labor budget.Pairs that are too dissimilar can prompt lengthy descriptions without direct contrasts; pairs too similar can leave annotators unable to see a difference.Source: §A.1 · p. 11

A.2 Pivot-Branch Sampling

Pivot-branch sampling checks pivot images individually, then draws comparison images using visual similarity and taxonomic distance to manage annotation cost while balancing relevance, coverage, and comparability.

A.2 Pivot-Branch Sampling

  • Pivot-branch sampling vets each pivot image individually and pairs it with k branch images, reducing the annotation-cost scale from 1/(1−p)^2 to 1/(1−p) when images have discard probability p.For p = 2/3, the passage gives a cost scale of 1.5 rather than 2.25 for independently drawn pairs.Source: §A.2 · p. 11

A.3 Designing spivot(D)

  • Sampling pivots uniformly over available classes uses labels as a coarse measure of diversity and helps address category-level dataset bias.Source: §A.3 · p. 12

A.4 Designing sbranch(D, ipivot, k)

  • Branch sampling divides k into visually similar and taxonomically related images, selecting nearest visual neighbors and taxonomic leaves grouped by exact levels of shared ancestry.Increasing the taxonomic level ℓ gives mutually exclusive sets farther from the pivot class; round-robin sampling can distribute the taxonomic budget across those sets.Source: §A.4 · p. 12

A.5 Analyzing sbranch(D, ipivot, k)

  • Visual and taxonomic sampling complement each other: the taxonomy supports controllable coverage while distance choices maintain comparability and bias sampling toward relevant pairs.Clamping taxonomic distance limits how far pairs diverge; distributing samples across progressively larger taxonomic subsets favors harder-to-distinguish pairs while retaining sparse coverage.Source: §A.5 · p. 12

B Details for Constructing Birds-to-Words Dataset

The dataset construction process filters for clear bird photographs, selects pivot images, then samples comparison images across taxonomic distances. These steps aim to provide usable images and varied comparisons.

B.1 Clarity

  • Images must meet four clarity criteria: show one animal, show the animal itself, be in focus, and make it sufficiently visible.The dataset applies stricter image requirements than iNaturalist research-grade observations.Source: §B.1 · p. 13

B.2 Pivot Images

  • The authors manually selected 405 pivot images after sampling species with at least four observations and reviewing four candidate images per species.Species were sampled from Aves, with look-ahead branch sampling to favor species that could yield sufficient taxonomic comparisons.Source: §B.2 · p. 13

B.3 Branching Images

  • For each pivot, the authors selected 10 taxonomic branches: two images from each of five disjoint tree levels above the pivot's class.The levels range from the same species to broader taxonomic groupings; visually similar branches are described separately in Section 2.3.Source: §B.3 · p. 13

B.4 Annotations

Annotators screened branching images for clarity, and the authors retained 3,347 image pairs after applying a clarity threshold. Annotator writing quality and adherence were also manually checked.

B.4 Annotations

  • 3,347 image pairs remained after requiring at least four positive clarity judgments, a retention rate of 68.9%.The initial annotation covered 4,860 pairs formed from 405 pivot images and 12 branching images per pivot.Source: §B.4 · p. 13
  • The authors reviewed five reference annotations from each annotator during a pilot and conducted random quality checks during data collection.They report that individually vetting writing quality and guideline adherence was vital for ensuring high data quality.Source: §B.4 · p. 13

C Model Details

The model uses a frozen, ImageNet-pretrained ResNet-101 and Transformer encoder and decoder components, with specified training and decoding settings. The paper also provides image attributions.

C Model Details

  • The image encoder is an ImageNet-pretrained ResNet-101 with fixed weights; the authors report better performance than with Inception-v4.The Transformer encoder and decoder each use six layers, hidden size 512, eight attention heads, and dot-product self-attention.Source: C Model Details · p. 13
  • Paragraphs are clipped at 64 tokens during training, a length chosen to cover 94% of paragraphs; preprocessing includes tokenization, lowercasing, and image-specific special tokens.Source: C Model Details · p. 14
  • Beam search with beam size 5 is used for reported results because it performed best among the tested decoding strategies.The decoding ablations report results for greedy decoding, multinomial sampling, and beam search separately.Source: C Model Details · p. 14
  • Training uses Adagrad for 700k steps, a .01 learning rate, batch size 2048, and gradient clipping at magnitude 5.The learning rate decays after 20k steps by a factor of 0.9.Source: C Model Details · p. 14

D Image Attributions

Related papers

Loading 1909.04101v3…