PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?

Patrick Schramowski, Christopher Tauchmann, Kristian Kersting

arXiv:2202.06675v2cs.AIcs.CVcs.CY

TL;DR

Large datasets contain potentially inappropriate content, but manually documenting it is difficult, especially for large image collections. The paper proposes Q16, which prompt-tunes pretrained CLIP to detect such content and uses generated captions to document it; experiments on ImageNet and OpenImages suggest that machines can identify a broader range than previous manual studies. Human oversight remains necessary because automation may tempt dataset creators to neglect manual validation.

  • Problem

    Large datasets raise serious concerns about inappropriate content, while documenting that content manually is difficult and labor-intensive.

  • Method

    Q16 prompt-tunes pretrained models, including CLIP, to identify inappropriate images and uses generated captions to assist dataset documentation.

  • Results

    Q16 identifies inappropriate content broader than previous manual studies in demonstrations on large-scale computer-vision datasets.

  • Takeaways & Limitations

    Machines can assist dataset creators in answering Datasheets Question 16 about inappropriate image content.

  • Takeaways & Limitations

    Human validation remains important because automated documentation may tempt dataset creators to neglect manual validation.

Abstract

from arXiv · show

Large datasets underlying much of current machine learning raise serious issues concerning inappropriate content such as offensive, insulting, threatening, or might otherwise cause anxiety. This calls for increased dataset documentation, e.g., using datasheets. They, among other topics, encourage to reflect on the composition of the datasets. So far, this documentation, however, is done manually and therefore can be tedious and error-prone, especially for large image datasets. Here we ask the arguably "circular" question of whether a machine can help us reflect on inappropriate content, answering Question 16 in Datasheets. To this end, we propose to use the information stored in pre-trained transformer models to assist us in the documentation process. Specifically, prompt-tuning based on a dataset of socio-moral values steers CLIP to identify potentially inappropriate content, therefore reducing human labor. We then document the inappropriate images found using word clouds, based on captions generated using a vision-language model. The documentations of two popular, large-scale computer vision datasets -- ImageNet and OpenImages -- produced this way suggest that machines can indeed help dataset creators to answer Question 16 on inappropriate image content.

1 INTRODUCTION

Large, undercurated datasets make manual documentation of inappropriate content difficult, motivating Q16, a semi-automatic approach that uses pretrained vision-language models. Applied to ImageNet and OpenImages, Q16 identifies a broader range of potentially inappropriate material than previous manual studies.

  • Motivation: Manual documentation becomes difficult or infeasible for very large datasets, as illustrated by checking 50K of 12M RedCaps images.The cited documentation found one image containing nudity in that sample.
  • Q16: Q16 is a semi-automatic method designed to document inappropriate image content and answer Question 16 in datasheets.Question 16 asks whether datasets contain content that may be offensive, insulting, threatening, or anxiety-inducing when viewed directly.
  • Method: Prompt-tuning steers pretrained CLIP toward identifying inappropriate images, while MAGMA generates captions for accessible documentation.The approach uses pretrained model information to reduce the labor involved in dataset documentation and curation.
  • Results: The approach is illustrated on the ImageNet-ILSVRC-2012 and OpenImages datasets, where it reveals additional inappropriate content not detected by previous documentation.The paper presents automated dataset documentation for both large-scale computer vision datasets.
  • Results: Q16 detects a larger and broader range of potential inappropriate images than previous approaches, including violence, misogyny, and other offensive material.The identified images include persons, objects, symbols, and text, and extend beyond nudity and misogynistic images.

2 BACKGROUND AND RELATED WORK

Background work established the capabilities of large pretrained and vision-language models, alongside concerns about biased or inappropriate training data. Related research addressed dataset documentation and problematic content extensively in language, while this work extends automatic detection toward computer vision.

  • Relation to prior work: The present study targets computer vision by using pretrained models’ retained knowledge and prompt steering to classify potential inappropriate image content.The work builds on evidence that pretrained models encode moral information and can be guided through prompts.
  • Dataset documentation: Dataset documentation aims to increase transparency and accountability while helping researchers and practitioners select appropriate datasets.Prior work proposed documenting dataset sources, content, creation processes, and other features.
  • Problematic content detection: Most automatic methods for abusive, offensive, hateful, toxic, stereotypical, or biased content have focused on text rather than images.The background surveys NLP methods and evaluation datasets for several forms of problematic language.
  • Computer vision: Prior computer-vision work examined unfair concepts, representation inequalities, privacy, pornography, misogyny, and offensive products in datasets such as ImageNet.These efforts included filtering concepts, detecting faces, classifying pornographic content, and manually surveying misogynistic images.

3 THE Q16 PIPELINE FOR DATASHEETS

The Q16 pipeline combines conservative machine detection of potentially inappropriate images with automatically generated descriptions to support human dataset documentation. It uses socio-moral data to steer CLIP and summarizes identified concepts through annotations and word clouds.

  • Detection: The pipeline first uses socio-moral data and prompt-tuned CLIP to detect potentially inappropriate images.The detection step is designed to identify all potentially inappropriate content and remains part of a human-interactive process.
  • Detection: The conservative detector can produce large candidate subsets, including approximately 40K images for ImageNet1k.
  • Documentation: The second step generates automatic image descriptions to help dataset creators describe and validate the identified content.
  • Documentation: Q16 documentation includes the ratio of identified images to total samples and a summary of identified concepts.Concepts are made accessible through word clouds based on dataset annotations and generated descriptions.
  • Scope: The study focuses on inappropriate images defined as defamatory, false, inaccurate, abusive, indecent, obscene, menacing, or otherwise offensive.This definition is broader than some prior work addressing only faces, pornography, or misogynistic images.
  • Scope: The operational notion of inappropriateness may primarily reflect selected-data generators, annotators, and retained pre-trained-model knowledge.The authors emphasize that social norms evolve and people have diverse sentiments about offensiveness.

3.3 Inappropriate content detection of Q16

The authors evaluate standard pre-trained vision models and CLIP-based models for identifying inappropriate content in SMID. CLIP’s prompt-based approach performs strongly with little tuning data, while zero-shot classification is better for negatives than positives.

  • Baselines: The experiments use 10-fold cross-validation and compare fine-tuned standard computer-vision models with CLIP-based approaches.
  • Baselines: ImageNet-based linear-probing models show low precision and recall, limiting classification of truly inappropriate and non-inappropriate images.Fine-tuning increases recall, but precision remains comparatively low; larger ImageNet21k pre-training gives inconclusive results.
  • CLIP approach: Positive and negative natural-language labels produce the best zero-shot performance among the tested prompt labels.
  • CLIP approach: CLIP compares image and prompt embeddings through cosine similarity rather than a separately trained linear classifier.The visual and text encoders produce representations for an image sample and a prompt, whose similarity determines the comparison.
  • CLIP approach: Zero-shot CLIP performs on par with fully tuned ImageNet-based models for linear probing, but classifies true negatives better than positives.This imbalance suggests that the initial prompts, especially the positive-class prompt, are not optimal.
  • Prompt-tuning method: The proposed prompt-tuning learns one final sentence embedding per class label while keeping the visual and text encoder parameters fixed.

3.4 Dataset documentation of Q16

The paper proposes a human-in-the-loop pipeline that uses prompt-tuned CLIP to pre-select potentially inappropriate images, then uses generated captions and word clouds to document them for Datasheet Question 16.

  • Detection and scope: Prompt-tuned CLIP can, in principle, extract possible inappropriate images for dataset documentation.The approach focuses on documenting final dataset content rather than curating or removing images.
  • Detection and scope: The model’s conservative pre-selection aims to identify all potentially inappropriate content, so the resulting subsets can remain large.The paper gives approximately 40K images for ImageNet1k as an example.
  • Documentation and caveats: The documentation should report dataset size and the ratio of potentially inappropriate images, while creators should inspect infrequent but potentially severe content.Word clouds emphasize frequent concepts, which can underrepresent rarer severe examples.
  • Human validation: Dataset creators remain responsible for manually validating and describing the identified subset in a human-in-the-loop process.Users select images according to their setting and requirements rather than being advised to remove specific images.
  • Automatic description: MAGMA generates image descriptions, which are summarized into word clouds using dataset annotations and generated textual descriptions.The generated descriptions can expose concepts not captured by dataset annotations alone.
  • Word-cloud documentation: Three word-cloud views use annotation frequencies, generated-caption frequencies, and chi-squared weighting to highlight concepts and differences from remaining images.The chi-squared weighting removes common descriptions and compares word or bi-gram frequencies between subsets.

4 ANSWERING DATASHEET QUESTION 16 FOR IMAGENET AND OPENIMAGES

The proposed pipeline is applied to ImageNet1k and OpenImagesV6 using CLIP-based detection, MAGMA captions, and word-cloud documentation. It identifies diverse potentially inappropriate objects, symbols, actions, and scenes, while supporting human review rather than automatic removal.

  • Setup: The authors document ImageNet1k and OpenImagesV6 using CLIP steered by SMID-based optimized prompts and MAGMA-generated captions.For each image, the implementation generated 10 captions using two top-k and temperature settings.
  • 4.1 ImageNet: 40,501 ImageNet1k images were identified, with gasmask (797), guillotine (783), and revolver (725) as the top-3 classes.The classifier also selected animal classes such as tick (554) and spider (397), whose appropriateness depends on the task.
  • 4.1 ImageNet: The ImageNet1k pre-selection covered injured, aggressive, caged, and dead animals, along with blood-tainted scenes and apparent murders.Examples include king penguin, pembroke, great pyrenees, squirrel monkey, tub, and screwdriver.
  • 4.1 ImageNet: The ImageNet1k documentation surfaced offensive symbols, extremist or insulting gestures, offensive text, weapons, harm, misogyny, and derogative nudity.The authors report examples including swastikas, Ku-Klux-Klan uniforms, middle-finger gestures, and harmful scenes involving people and animals.
  • Implications: The authors advocate human-in-the-loop use of large-scale models because the method identified content missed by a prior human image selection and can reduce curation errors.They recommend investigating the range of selected examples and documenting the dataset rather than automatically removing images.

5 SOCIETAL IMPACT AND LIMITATIONS

The paper frames automated inappropriate-content detection as support for dataset documentation, while emphasizing human oversight, evolving norms, cultural variation, model bias, and limited evaluation.

  • Large datasets contain potentially offensive, threatening, or anxiety-inducing content, motivating more systematic documentation of dataset composition.
  • Manual documentation is tedious and error-prone for large image datasets, so the proposed method automates detection and description to assist documentation.
  • Human validation remains necessary because automation could tempt dataset creators to neglect manual review; the authors therefore recommend a human-in-the-loop setting.
  • The approach uses binary classification and broad inappropriate-content categories, while finer-grained and multi-label concepts could improve transparency and documentation.
  • The method’s scope is constrained by possible CLIP biases, unevaluated caption quality, and reliance on SMID norms drawn mainly from participants in the United States and partly India.
  • Because social norms and offensiveness evolve and annotators may disagree, the data, system, and documentation require updating and uncertainty-aware review.

6 CONCLUSION

The conclusion presents Q16 as a tool using pre-trained model knowledge to assist curation of large image datasets. Across ImageNet-ILSVRC2012 and OpenImages, it identifies inappropriate content broader than previous manual studies, while leaving interpretability and finer-grained classification for future work.

  • Q16 uses implicit knowledge in large-scale pre-trained models to assist humans curating millions of images.
  • CLIP retains knowledge about potentially offensive material and requires only few shots to identify it automatically.
  • On ImageNet-ILSVRC2012 and OpenImages, Q16 identified inappropriate content broader than previous manual studies.
  • Future work includes multi-label and gradual inappropriateness classification across additional computer-vision and multimodal datasets.
  • Because the underlying models are black boxes, explainable-AI methods could help explain why images are identified and described as inappropriate.

A Details on the Socio-Moral Image Database

SMID supplies moral-sentiment labels and images used to steer Q16 toward inappropriate-content detection. The dataset’s ratings, collection process, participant composition, and visualization define the proxy’s scope.

  • SMID provides the moral-sentiment basis for steering Q16, while the pre-trained model’s retained knowledge defines inappropriate image content in this work.
  • SMID contains 2,941 images collected through a multi-step process involving 476 participants who each contributed images for moral concepts.
  • The dataset reflects participants mainly recruited from the United States and partly from India, so selected images and annotations depend on the people involved.
  • The collection used more than 50 concepts, including Harm, Inequality, Degradation, Discrimination, and Exploitation, with filtering for duplicates, corruption, size, licensing, and unsuitable imagery.
  • SMID’s moral ratings average diverse sentiments, with rating<2.5 treated as inappropriate and rating>3.5 as counterexamples.

B PCA visualization of embedding-space

PCA visualizations compare pre-trained embedding spaces before SMID tuning. CLIP separates inappropriate content from counterexamples along PC2, unlike the ImageNet1k-pre-trained ResNet50 visualization.

  • Figure 7b shows PCA of embedded representations from an ImageNet1k-pre-trained ResNet50 before SMID training, where reliable inappropriate-content inference is unclear.
  • Figure 7c shows PCA for CLIP’s ViT-B/16 model pre-trained on WebImageText using contrastive language-image pre-training.
  • CLIP’s Vision Transformer separates inappropriate content from counterexamples along PC2 without explicit training for that task.

C Illustration of Soft-prompt tuning

The study detects inappropriate content using CLIP’s implicit knowledge, steered through prompt-tuning based on selected data. Exemplary nearest-neighbor images illustrate the learned prompts’ ability to distinguish possible inappropriate content from positive scenes.

  • CLIP is steered to detect inappropriate content using implicit knowledge from selected data.The approach uses prompt-tuning to guide the pre-trained vision-language model toward this detection task.
  • Prompt-tuning is based on the SMID dataset and produces learned prompts with exemplary nearest image neighbors.Figure 8 presents the tuning process and the resulting image neighbors.
  • The nearest-neighbor examples contrast a possible inappropriate image with a positive scene.The right-side image portrays possible inappropriate content, while the left-side image serves as a counterexample.
Loading 2202.06675v2…