Source-linked AI summary
RedCaps: web-curated image-text data created by the people, for the people
Karan Desai, Gaurav Kaul, Zubin Aysola, Justin Johnson
TL;DR
Web image-text datasets often require complex filtering because their sources are noisy or treat text as secondary to images. RedCaps collects 12M Reddit image-text pairs from manually curated subreddits, whose structure supports composition control without instance-level labeling. Models trained on RedCaps produce captions preferred by humans and visual features that transfer across many downstream tasks, while the dataset remains subject to privacy and consent constraints.
Problem
Existing web image-text datasets require complex filtering to manage noise, while their sources often treat text as secondary to images.
Method
RedCaps collects 12M image-text pairs submitted by Reddit users across manually curated subreddits that provide coarse labels and composition control.
Results
RedCaps-trained captioning models are preferred by human evaluators over CC-3M-trained models, and their visual features outperform SBU and CC-3M features across downstream tasks.
Takeaways & Limitations
Reddit is presented as a public, scalable source of natural image-text data whose learned representations transfer to many downstream tasks, including zero-shot settings.
Takeaways & Limitations
RedCaps uses coarse face filtering and subreddit curation to limit people-related content, creating a pragmatic tradeoff that may suppress language from marginalized groups reclaiming slurs.
Abstract
from arXiv · showhide
Large datasets of paired images and text have become increasingly popular for learning generic representations for vision and vision-and-language tasks. Such datasets have been built by querying search engines or collecting HTML alt-text -- since web data is noisy, they require complex filtering pipelines to maintain quality. We explore alternate data sources to collect high quality data with minimal filtering. We introduce RedCaps -- a large-scale dataset of 12M image-text pairs collected from Reddit. Images and captions from Reddit depict and describe a wide variety of objects and scenes. We collect data from a manually curated set of subreddits, which give coarse image labels and allow us to steer the dataset composition without labeling individual instances. We show that captioning models trained on RedCaps produce rich and varied captions preferred by humans, and learn visual representations that transfer to many downstream tasks.
1 Introduction
RedCaps addresses the quality and filtering challenges of web-scale image-text datasets by collecting conversational image-caption pairs from Reddit’s human-curated communities. Its 12M pairs use subreddit structure for coarse labeling and composition control, and support strong captioning and transfer-learning results.
- 12M image-text pairs from 350 subreddits form RedCaps, covering everyday objects, scenes, hobbies, and pets.
- Existing web datasets rely on complex filtering to remove noisy or mismatched image-text pairs, reducing data efficiency.CC-3M collected 5B pairs but retained 3.3M after filtering.
- RedCaps uses Reddit because users create images and text together for human interaction, yielding conversational, humorous, emotional, and diverse captions.
- Manually curated subreddits provide coarse image labels and let researchers steer dataset composition without labeling individual instances.Reddit voting and moderation also provide organic quality control through downvotes and content removal.
- Human evaluators prefer captions generated by RedCaps-trained models over those from CC-3M-trained models.RedCaps-trained visual features also outperform SBU- and CC-3M-trained features across the reported downstream evaluation.
2 RedCaps: Collecting image-text pairs from Reddit
RedCaps collects image-text pairs from Reddit image posts using subreddit curation, API-based extraction, minimal caption cleaning, and targeted risk filtering. The resulting dataset balances scalable collection with explicit attention to privacy, harmful stereotypes, consent, and demographic bias.
- Data source and post structure: Reddit image posts provide images, captions, scores, authorship information, and timestamps for RedCaps extraction.The collection uses captions as textual data and excludes associated comment threads.
- Data collection pipeline: The pipeline parallelizes collection by subreddit and year, then performs subreddit selection, image-post filtering, and caption cleaning.Subreddit selection steers composition, while filtering restricts image hosts and cleaning applies lowercase conversion, character normalization, and bracketed-substring removal.
- Dataset scale and extensibility: 13.4M instances were collected from 350 subreddits before privacy and harmful-stereotype filtering produced the final 12M-instance dataset.The pipeline avoids webpage crawlers, search engines, and large indexed-webpage databases, and can be extended with additional subreddits and future years.
- Ethical considerations: RedCaps addresses privacy through subreddit curation, face-image removal, URL distribution, and user-requested instance removal.The authors note that coarse language filtering may suppress reclaimed slurs and that URL-based removals reduce exact reproducibility over time.
- Ethical considerations: Automatic filtering targets images containing faces or NSFW content and captions containing potentially derogatory language.Detectors filter approximately 1.4M instances, with precision estimated from 5K detected images and missed detections assessed through 50K random images.
- Ethical considerations: Reddit’s nonrepresentative demographics and English-language focus may bias the objects, places, and descriptions represented in RedCaps.The authors provide usage restrictions and do not recommend training models to identify people or make decisions affecting people.
3 RedCaps data analysis
RedCaps is a large, growing public image-text dataset whose Reddit-based organization yields broad visual coverage and unusually diverse, conversational language. Its captions are longer-tailed and linguistically richer than those in comparison datasets.
- Dataset composition: RedCaps spans 350 subreddits in a long-tail distribution covering general photography, food, pets, hobbies, and accessories.This distribution reflects visual concepts encountered in daily life without requiring a predefined object ontology.
- Caption statistics: RedCaps captions have a mode of 5 words and a heavier tail of captions ≥25 words than CC-3M and SBU.Unlike SBU, RedCaps captions are not filtered to contain particular prepositions or manually curated terms.
- Linguistic diversity: RedCaps has more than 4× CC-3M’s unique unigrams and more than 3× its unique bigrams and trigrams occurring at least 10 times.The comparison uses unigram, bigram, and trigram counts above the stated frequency threshold.
- Linguistic diversity: RedCaps uses conversational and community-specific trigrams, including “I don’t,” “one of my,” and “this is my.”These patterns contrast with SBU’s prepositional phrases and CC-3M’s potentially automated-sounding “may contain” constructions.
- Linguistic diversity: RedCaps contains broader part-of-speech diversity and geographically varied proper nouns, including cities, states, and countries.It has >2× more common nouns and >4× more proper nouns than SBU, plus >2× more adjectives and >1.5× more verbs than CC-3M.
4 Experiments
The experiments train VirTex-v2 models on image-text datasets and transfer their visual backbones to classification, segmentation, and captioning tasks. RedCaps improves transfer performance over SBU and CC-3M, while producing captions preferred by human evaluators and controllable by subreddit style.
- Experimental setup: VirTex-v2 models are trained with the same architecture on SBU, CC-3M, RedCaps, and a size-controlled 3.2M-instance RedCaps subset.The visual backbone is transferred to downstream tasks to compare data-source effects while controlling architecture and, in one comparison, dataset size.
- Transfer learning: RedCaps-trained models outperform SBU- and CC-3M-trained models by wide margins on six of seven zero-shot classification datasets.RedCaps-20 also outperforms CC-3M, so the reported advantage is not attributed solely to RedCaps’s larger size.
- Transfer learning: Linear-probe classification shows similar trends to zero-shot transfer.The evaluation freezes visual features and trains logistic-regression classifiers on the downstream datasets.
- Comparison with CLIP: RedCaps-trained transfer performance remains below CLIP’s reported Food-101 top-1 result, 54.6 versus 81.1 with ResNet-50.The authors note that CLIP uses different architecture, scale, training duration, and prompt ensembling, so the results are not directly comparable.
- Image captioning: Workers preferred captions from the RedCaps-trained model for 633/1000 test images and preferred generated captions over ground-truth captions for 416/1000 images.Each comparison used majority votes from three crowd workers on a held-out set of 1K images.
- Image captioning: Subreddit prompting produces captions with distinct linguistic structures, community jargon, or altered subject emphasis, but is often ignored for unrelated images.Examples include “itap of ...” for r/itookapicture and subject changes for r/earthporn or r/food.
5 Related work
RedCaps extends web-scale image-text dataset construction for vision-language pretraining beyond search results and HTML alt-text. It is positioned alongside datasets and methods that pursue scalable representation learning without expensive annotation.
- Image-text datasets: SBU and Conceptual Captions are prominent image-text datasets used to pretrain representations transferred to retrieval, captioning, and downstream vision-language tasks.Their applications include visual question answering, referring expressions, and visual reasoning.
- Visual representation learning: Recent work also learns visual features from COCO Captions, YFCC-100M, and much larger non-public datasets such as those used by CLIP and ALIGN.These approaches connect image-text data to generic visual representation learning beyond captioning and retrieval.
- Alternative web supervision: Scaling image-text learning without annotation costs is related to methods that learn from noisy non-text labels in datasets including WebVision, YFCC-100M, JFT-300M, and Instagram-3.5B.These efforts pursue large-scale visual learning through alternative web supervision.
6 Conclusion
RedCaps is presented as a large Reddit-derived image-caption dataset whose subreddit structure supports curated content and visual representation learning. The authors also identify safety and representational limitations that constrain its use.
- RedCaps contains images and captions collected from Reddit, with subreddit curation guiding content without labeling individual instances.The dataset focuses on animals, objects, scenery, and activities while minimizing the appearance of people.
- RedCaps supports visual representations that transfer to many downstream tasks, including zero-shot settings without task-specific training data.
- RedCaps captioning models generate high-quality text in multiple styles.
- RedCaps may contain a small number of NSFW images or harmful language despite subreddit curation and automated filtering.The authors also note that Reddit’s demographic biases may prevent equal representation of all groups.
- The authors caution against applications that make predictions about people because of these content and demographic limitations.
Appendix B User studies interface for caption evaluation
The user study evaluated caption quality through an Amazon Mechanical Turk guessing game comparing VirTex-v2 models trained on CC-3M and RedCaps. Workers judged which caption appeared human-written.
- The evaluation used an Amazon Mechanical Turk task framed as a guessing game about an AI bot impersonating humans.
- Workers received $0.3 for batches of five images, and the study collected choices for 1K images with three workers per image.
- The interface provided instructions and example questions for comparing caption predictions from CC-3M- and RedCaps-trained VirTex-v2 models.
Appendix C Qualitative examples: CC-3M vs RedCaps
The appendix presents qualitative caption comparisons, subreddit-conditioned generation examples, and a t-SNE visualization showing how subreddit structure organizes visual features. These materials illustrate both caption preference and controllable dataset composition.
- CC-3M vs RedCaps: RedCaps-trained VirTex-v2 predictions were preferred as human-written captions 63.6% of the time in the qualitative evaluation.The underlined caption was selected by at least two of three crowd workers.
- Subreddit-controlled caption style: Providing subreddit names as partial prompts enables caption generation conditioned on subreddit-specific styles.
- T-SNE visualization: Subreddits with similar topics form tight local clusters in the visual feature space, including dog and food-related groups.
- T-SNE visualization: The clustered feature space demonstrates that manually selecting subreddits can steer RedCaps’ distribution of visual concepts.
- Additional transfer results: Table 5 reports zero-shot top-5 accuracy and low-shot transfer results across six datasets, with RedCaps models performing best except on SUN397.
Appendix F Transfer learning experiments: additional details
The appendix details linear-probe and low-shot transfer procedures for evaluating RedCaps features. In low-shot experiments, RedCaps models performed best on all but one dataset.
- Linear probe image classification: Linear-probe classification used logistic regression with a 10% validation split and swept L2 regularization values from 10^-6 to 10^6.
- Low-shot classification: Low-shot classification evaluated six datasets using end-to-end fine-tuning on 1000 randomly sampled, class-balanced instances per dataset.
- Low-shot classification: RedCaps and RedCaps-20 models performed best on all but one low-shot dataset.
Appendix G Datasheet for RedCaps dataset
RedCaps is documented as a freely collected, non-fully representative sample of Reddit image posts with metadata, task-dependent targets, and known noise and availability constraints.
- 12,011,111 instances represent individual Reddit image posts in RedCaps.
- RedCaps samples 350 hand-picked subreddits and does not fully represent Reddit’s broader collection of subreddits and topics.The selection emphasizes high-quality photographs with descriptive captions while excluding many topic-focused communities.
- Each instance contains metadata including the image URL, author, raw and cleaned captions, subreddit, score, and creation time.
- RedCaps defines no universal instance label or target; users choose targets according to tasks such as captioning, classification, generation, or self-supervised learning.
- The dataset is noisy by design, may contain duplicate posts, and can lose images when Reddit users delete the underlying posts.Such redundancies are described as a very small fraction, while deletions reduce dataset size over time.
- RedCaps distributes image URLs rather than image files, relying on Reddit, Imgur, or Flickr servers for access.Reddit archives post metadata but not the media content itself, so users retain control over image accessibility.
Q16. Does the dataset contain data that, if viewed directly, might be offensive, insulting, threat-
RedCaps uses subreddit curation, content filters, and automated detectors to reduce offensive or harmful material, while acknowledging that large-scale verification is incomplete.
- RedCaps cannot verify all images and captions, so some offensive, threatening, insulting, or anxiety-inducing material may remain.
- The pipeline excludes NSFW, banned, quarantined, and generally non-offensive-violating subreddits and removes posts marked NSFW.
- Captions containing any of 400 potentially offensive words or phrases are removed.
- An off-the-shelf face detector removes images with potential human faces, and the dataset excludes subreddits primarily devoted to images of people.
- Manual inspection of 50K random images found 79 identifiable human faces and one image containing nudity.The authors estimate approximately 19K images with identifiable people across the full dataset.
- RedCaps includes Reddit usernames, and some captions or images may permit identification of people or reveal subpopulation information.
Q26. Over what timeframe was the data collected? Does this timeframe match the creation
RedCaps contains Reddit image posts uploaded from 2008–2020, collected through the Reddit API and periodically refreshed, with deletion and consent-related constraints.
- RedCaps covers image posts uploaded to Reddit between 2008–2020, with data collected in early 2021 and freshly re-collected in August 2021.
- The authors did not conduct a formal institutional-review-board process or a data-protection impact analysis.
- RedCaps users did not explicitly consent to neural-network training, although Reddit submission makes posts publicly accessible through the official API.
- Deleting an underlying Reddit post removes its URL-based image from RedCaps, and users can also request removal through a public opt-out form.
- The dataset has been used for image captioning and transferable visual representations for image classification, object detection, and instance segmentation.
Q41. What (other) tasks could the dataset be used for?
RedCaps can support additional vision-and-language uses, but its licensing, demographic composition, and design impose explicit boundaries on appropriate applications.
- RedCaps could support image or text retrieval and text-to-image synthesis.
- Reddit’s non-representative demographics and English-primary subreddit selection may bias the images, languages, and ideas present in RedCaps.
- The dataset is intended broadly for non-commercial academic research rather than commercial or for-profit use.
- RedCaps should not be used for identifying people’s facial, gender, age, or ethnicity features or for decisions affecting people.
- The dataset distributes annotations and URLs rather than images, and commercial redistribution of the images is prohibited.
- Annual updates are planned, while new versions remove opt-out instances and eventually deprecate older versions.