PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels

Ishan Misra, C. Lawrence Zitnick, Margaret Mitchell, Ross Girshick

arXiv:1512.06974v2cs.CV

TL;DR

Human-written image tags and captions are scalable but omit visual concepts according to structured reporting bias, making them unreliable as direct visual labels. The paper decouples visual presence from mention relevance and reports improved classification and captioning across datasets, including doubled baseline performance on Yahoo Flickr 100M.

  • Problem

    Human-centric tags and captions omit substantial visual information and require methods that can learn visual concepts without exhaustive clean annotations.

  • Method

    The method jointly models a latent visual presence label and a relevance label, using shared ConvNet representations to predict what is present separately from what is mentioned.

  • Results

    Improved performance appears across image classification and captioning tasks and datasets; on Yahoo Flickr 100M, the model doubles MILVC baseline performance and increases mAP by 5.5 points.

  • Takeaways & Limitations

    The model provides interpretable predictions of concepts present versus concepts worth mentioning, consistent with observations about omitted typical properties.

  • Takeaways & Limitations

    Future work includes adding constraints to latent variables and estimating their posteriors, such as with EM.

Abstract

from arXiv · show

When human annotators are given a choice about what to label in an image, they apply their own subjective judgments on what to ignore and what to mention. We refer to these noisy "human-centric" annotations as exhibiting human reporting bias. Examples of such annotations include image tags and keywords found on photo sharing sites, or in datasets containing image captions. In this paper, we use these noisy annotations for learning visually correct image classifiers. Such annotations do not use consistent vocabulary, and miss a significant amount of the information present in an image; however, we demonstrate that the noise in these annotations exhibits structure and can be modeled. We propose an algorithm to decouple the human reporting bias from the correct visually grounded labels. Our results are highly interpretable for reporting "what's in the image" versus "what's worth saying." We demonstrate the algorithm's efficacy along a variety of metrics and datasets, including MS COCO and Yahoo Flickr 100M. We show significant improvements over traditional algorithms for both image classification and image captioning, doubling the performance of existing methods in some cases.

1. Introduction

Visual concept recognition needs scalable supervision, but human-written tags and captions omit concepts according to reporting bias. The paper models this bias to learn visually grounded classifiers and reports improvements across datasets and tasks.

  • 1. Introduction: Visual concept recognition supports applications in science, medicine, and industry, but scaling to hundreds of thousands of concepts would require vastly more clean annotations.Existing datasets typically cover hundreds to thousands of typical concepts, while supervised methods require exhaustive, clean labels.
  • 1. Introduction: User-generated tags and captions provide scalable annotation sources, reaching hundreds of millions of photos and hundreds of thousands of distinct tags.These sources relax the requirement for pristinely labeled data.
  • 1. Introduction: Human-centric annotations omit minor or typical visual concepts because annotators focus on information they judge salient or important.Bicycles may be present but unmentioned, while typical attributes such as bananas being yellow may be omitted.
  • 1. Introduction: The proposed model separates visual presence from relevance, jointly learning whether a concept is present and whether it is worth mentioning.The two classifiers are trained as multiple heads branching from a shared ConvNet trunk.
  • 1. Introduction: The method improves visual-classifier performance on fully labeled and human-generated data, improves caption generation quality, and doubles baseline classification performance on Yahoo Flickr 100M.The results are also interpretable and align with observations that humans often omit typical attributes.

2. Related work

Prior work characterizes label noise and human reporting behavior, while this paper addresses structured, image-dependent noise without requiring clean bootstrap labels. Its approach is related to, but distinct from, existing noise-modeling and noisy-label methods.

  • 2. Related work: Label noise is commonly divided into random noise and statistically dependent noise, and real-world data may contain both types.The distinction concerns whether noise depends on the data.
  • 2. Related work: Human-centric annotations exhibit structured noise because some labels are preferentially omitted, including objects, scenes, and attributes.This literature studies missing content in image descriptions and other human-generated annotations.
  • 2. Related work: Many existing noise models assume noise is conditionally independent of the image, ignoring how image content affects human annotations.This limitation includes models of symmetric and asymmetric label noise.
  • 2. Related work: Unlike an image-conditional noise model, the proposed training algorithm does not require a small cleanly labeled set for parameter-estimation bootstrapping.It is specifically designed for noise in human-centric annotations.
  • 2. Related work: Bootstrapping, semi-supervised learning, and positive-unlabeled learning generally require clean oracle labels, whereas this approach trains directly on noisy labels.The authors position their method as a possible starting point for these approaches.
  • 2. Related work: Psycholinguistic research reports that people omit obvious or typical attributes when they are not needed for conversational relevance, unique identification, or distinguishability.The paper connects this separation between observation and mention to its modeling of human-centric annotations.

3. Our Approach

The approach separates whether a concept is visually present from whether it is relevant enough for a human to mention, learning both from human-centric labels. It marginalizes the hidden visual-presence variable to reproduce annotations while allowing visually grounded predictions that can recover unmentioned concepts.

  • Factor decoupling: Naively training on human-centric labels learns annotator reporting behavior rather than whether each concept is visually present.The resulting classifier can predict how a human would label an image even when the concept is present or absent.
  • Factor decoupling: The model factorizes each concept predictor into a visual presence classifier and a relevance classifier conditioned on visual presence.The visual classifier estimates presence, while the relevance classifier estimates whether a present concept is worth mentioning.
  • Factor decoupling: The human-centric prediction is obtained by marginalizing over the latent visual-presence label z_w, which is unavailable as a training label.Training supplies human-centric labels y_w, while z_w indicates whether concept w is visually present.
  • Factor decoupling: For an unmentioned but visually present concept, the relevance classifier can explain the missing mention while the visual classifier assigns high probability to presence.This mechanism lets the model match human-centric labels without suppressing visually grounded predictions.
  • Model learning and parameterization: The model is trained jointly with direct optimization, using a shared ConvNet trunk and separate output branches for the two distributions.The conditional distributions are modeled with a ConvNet, and the full model is optimized over all concepts as a multilabel classification problem.
  • Model learning and parameterization: The relevance distribution is parameterized as a valid joint distribution by applying softmax normalization to four image-dependent scores.The scores use linear models with weights and biases, while global image features from the shared trunk provide the image representation.
  • Model learning and parameterization: Figure 4 measures reporting bias as the probability that detected objects are absent from captions and breaks missed objects down by size.The black line represents non-mention probability, while color bars show the size distribution of missed objects; many categories are dominated by small or medium objects.

4. Experiments

The experiments evaluate the proposed model on MS COCO and a random Yahoo Flickr Creative Commons 100M subset, using captions or tags as human-centric annotations. MS COCO detection labels support dataset analysis and evaluation.

  • Datasets: The evaluation uses Microsoft COCO and a random subset of Yahoo Flickr Creative Commons 100M.YFCC100M supplies user-generated image tags, while MS COCO supplies image captions as human-centric annotations.
  • Datasets: YFCC100M user-generated image tags and MS COCO image captions provide the human-centric annotations for evaluation.The two datasets represent noisy annotations from tags and captions, respectively.
  • Datasets: MS COCO object detection labels are used to analyze the data and evaluate the learned classifiers.These labels provide a reference for verifying classifier accuracy.

4.1. Experiments on MS COCO 1k visual concepts

Experiments on MS COCO 1k visual concepts measure reporting bias, compare latent-factor models with baselines, and evaluate both human-centric and visually grounded predictions. The latent model improves caption-label prediction and supports evaluation against fully labeled detection annotations, while image-conditioned relevance is important.

  • Dataset and setup: The 1,000 MS COCO visual concepts are the most common words in captions, and training uses approximately 80k images with five reference captions per image.Evaluation uses a 20k-image test set with 1,000-dimensional binary caption-label vectors.
  • Human reporting bias: Caption and detection annotations disagree substantially: the object with the highest reporting bias is mentioned roughly half as often as it appears.The reporting-bias statistic captures visually present objects omitted from captions.
  • Models and baselines: The experiments compare latent variants of MILVC and vanilla classification against corresponding baselines, including matched-parameter Multiple-fc8 controls.The latent variants produce either grid-based or single-output visual-presence predictions while modeling human-centric labels through relevance.
  • Evaluation: The MS COCO caption-label evaluation reports mAP and PHR across all concepts and by part of speech, comparing visually grounded and human-centric predictions.PHR measures precision at an estimated human-recall level based on multiple image references.
  • Results: The latent variable model improves classification over all baseline networks and architectures by 3 to 4 points on both mAP and PHR.A matched-parameter Multiple-fc8 model does not improve, indicating that the gain is not simply due to adding parameters.
  • Visual classification: On 73 COCO detection categories, fully labeled evaluation tests the visual-presence predictions; using human-centric predictions slightly hurts visual prediction performance.The 73 categories are mapped from fine-grained concepts and synonyms at evaluation time.
  • Conditioning on image content: Conditioning the relevance distribution on the input image significantly improves mAP, whereas an unconditioned model achieves only minor gains.This comparison tests the paper’s claim that reporting noise statistically depends on image data.

4.2. Experiments on Flickr image tagging

The model is evaluated on noisy Flickr tags to learn visually grounded classifiers while also modeling human-centric predictions. On YFCC100M, it substantially outperforms the MILVC baseline.

  • Dataset and setup: The experiment uses approximately 89k YFCC100M images, split into 75k training and 14k test images, with the top 1000 WordNet tags as visual concepts.Images contain between 5 and 30 human-annotated tags, and training follows the MS COCO hyperparameters for four epochs.
  • Dataset and setup: Table 4 reports mAP for visually grounded (v) and human-centric (h) predictions, including comparisons with MILVC and a parameter-matched Multiple-fc8 model.The table groups results by part of speech, including nouns, verbs, adjectives, pronouns, and prepositions.
  • Results: Our model doubles MILVC’s performance and increases mAP by 5.5 points on the YFCC100M test set.The evaluation uses the same human-annotated tags for the compared models.

4.3. Interpretability of the noise model

The relevance classifier adjusts visual predictions according to what humans are likely to mention in a particular image. This separates visual presence from reporting behavior and exposes interpretable context- and attribute-dependent patterns.

  • Noise-model interpretation: The relevance classifier r models image-conditioned human labeling noise and can enhance or suppress each concept’s visual prediction.It operates alongside the visual presence prediction rather than replacing it.
  • Noise-model interpretation: Among images with high visual-presence confidence v ≥0.8, variation in h demonstrates that the model decouples human-centric predictions from visual presence.Representative images are selected across quantiles of h, which increases from left to right.
  • Interpretability examples: The learned predictions capture context-sensitive reporting, including mentioning a hat on a cat, omitting a pumpkin’s color, and mentioning pink sheep.The model also learns that attributes can help distinguish objects, such as an orange pillow.

4.4. Correcting error modes by decoupling

By modeling latent noise, the method produces cleaner visual-presence predictions than a baseline trained to follow human-centric labels. Its corrections address spelling, localization, and counting errors.

  • Error correction: The model corrects misspellings such as desert versus dessert and localizes objects correctly, including fridges and nets in context or out of context.These corrections arise in visual-presence predictions compared with MILVC.
  • Error correction: The model is better at counting examples such as zebras and bananas than the MILVC baseline.The figure specifically contrasts corrected visual predictions for singular and plural concepts.
  • Error correction: The model corrects baseline false positives where MILVC predicts h ≥0.75 but the model predicts v ≤0.3.It also corrects false negatives where MILVC predicts h ≤0.3 while the model predicts v ≥0.75.

4.5. Using word detections for caption generation

The paper uses predicted visual concepts as image features for an LSTM caption generator. The setup feeds 1000 concept probabilities into the LSTM as its first hidden input.

  • Caption-generation setup: A basic LSTM generates captions from image features consisting of 1000 visual concept probabilities.The image features are fed once to the LSTM as its first hidden input.
  • Caption-generation setup: The captioning model uses 1000 LSTM cells and learns 256-dimensional input-word embeddings.Its vocabulary contains words appearing at least five times in the input captions.
  • Caption-generation setup: The captioning experiment follows a standard LSTM-based generation setup to test whether learned visual predictions improve caption generation.The supplied passage describes the model configuration but does not report captioning metric values.

5. Discussion

The paper models reporting bias by separating what is visually present from what humans choose to mention. This produces interpretable predictions and improves performance across image classification and captioning, while motivating further work on latent-variable constraints and posterior estimation.

  • The algorithm introduces a latent variable separating concepts present in an image from concepts labeled in it.
  • The approach improves performance over previous work on multiple tasks, including image classification and image captioning.
  • The model is highly interpretable, capturing concepts that may be included or excluded according to context and dependencies across visual concepts.
  • The model jointly decouples human reporting from algorithmic visual understanding in an end-to-end network.
  • Future work may constrain latent variables, estimate their posteriors with methods such as EM, and improve explicit handling of noisy data.
Loading 1512.06974v2…