Source-linked AI summary
YFCC100M: The New Data in Multimedia Research
Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, Li-Jia Li
TL;DR
Multimedia research needs large, reusable, openly sharable datasets rather than collections built for isolated tasks. The paper presents YFCC100M, a 100-million-object Creative Commons multimedia dataset, and reports that it supports a broad ecosystem of challenges and benchmarks while retaining a limitation in annotations.
Problem
Existing multimedia datasets are often single-purpose, insufficiently reusable, and not freely shared, limiting reproducibility, transparency, and accountability.
Method
The paper constructs and releases YFCC100M with photos, videos, metadata, annotations, precomputed features, and an ecosystem of related challenges and benchmarks.
Results
YFCC100M comprises 100 million Creative Commons media objects and has already supported diverse challenges and benchmarks across multimedia research.
Takeaways & Limitations
The dataset provides a shared foundation for improving and validating research methods and for pursuing new multimedia research challenges.
Takeaways & Limitations
The collection currently has limited metadata and annotations and cannot provide every content type or annotation found in specialized datasets.
Abstract
from arXiv · showhide
We present the Yahoo Flickr Creative Commons 100 Million Dataset (YFCC100M), the largest public multimedia collection that has ever been released. The dataset contains a total of 100 million media objects, of which approximately 99.2 million are photos and 0.8 million are videos, all of which carry a Creative Commons license. Each media object in the dataset is represented by several pieces of metadata, e.g. Flickr identifier, owner name, camera, title, tags, geo, media source. The collection provides a comprehensive snapshot of how photos and videos were taken, described, and shared over the years, from the inception of Flickr in 2004 until early 2014. In this article we explain the rationale behind its creation, as well as the implications the dataset has for science, research, engineering, and development. We further present several new challenges in multimedia research that can now be expanded upon with our dataset.
1. SHARING DATASETS
Multimedia datasets support experiments, validation, analysis, and applications, but existing collections often lack multimodality, sharing, and reuse. Publicly sharable, legally usable datasets can improve replication, research growth, and community synergy.
- Datasets support experiments, hypothesis validation, design analysis, and application building, but many multimedia collections contain only one media type.
- Data-sharing discussions identify practical, legal, and incentive-related barriers, including collection costs and copyright concerns.
- Creative Commons licenses communicate how owners permit media use, including restrictions on commercial use, remixing, and derivatives.
- Publicly sharable and legally usable data supports research equality, replication efforts, growth, and synergy beyond one-off task-specific collections.
2. THE YFCC100M DATASET
YFCC100M is a 100-million-object public multimedia dataset of Creative Commons Flickr photos and videos, distributed with metadata and designed for continued expansion.
- 100 million media objects make YFCC100M the largest public multimedia collection released, comprising approximately 99.2 million photos and 0.8 million videos.
- All objects were uploaded to Flickr between 2004 and 2014 and published under commercial or non-commercial Creative Commons licenses.
- The initial distribution is a 12.5GB compressed metadata archive, with expansion packs, extracted features, and planned cloud access to media content.
2.1 Metadata
YFCC100M combines extensive per-item metadata with temporal and geographic coverage, including user and machine annotations, capture history, and camera information.
- Each item includes identifiers, creator, camera, capture and upload times, location when available, license, titles, descriptions, tags, and Flickr links.
- 68,552,616 photos and 418,507 videos have user tags, while 3,343,487 photos and 7,281 videos carry machine tags.
- Figure 1 compares monthly capture and upload counts, while Figure 2 depicts global coverage for a one-million-photo sample.
- Capture dates span beyond Flickr’s 2004–2014 upload period, but camera-clock errors and backdated or future timestamps limit temporal reliability.
- 48,366,323 photos and 103,506 videos have geographic coordinates, and the dataset spans 249 territories, including international waters and airspace.
2.2 Content
YFCC100M contains diverse real-world visual and audiovisual content, with concept classifiers, user annotations, and videos whose audio and motion broaden the dataset’s multimedia scope.
- The collection includes street life, daily activities, holidays, events, people, animals, objects, food, architecture, and scenery.
- Creative Commons licensing is represented across the dataset through By Attribution, No Derivatives, Share Alike, and Non-Commercial categories.
- 1,570 binary SVM classifiers were trained to detect visual concepts, each tuned to achieve at least 90% precision on held-out data.
- Videos add audio and motion beyond visual content; only 5% lack an audio track, while sampled audio commonly contains ambient noise, music, or multiple speakers.
2.3 Representativeness
The dataset was constructed with limited filtering, while deliberately oversampling videos and geotagged photos; its representativeness was assessed against a random Flickr sample from the same period.
- Filtering excluded only media marked as “screenshot” or “other” by Flickr users.
- Videos were oversampled because they represent a small share of Flickr uploads.
- Geotagged photos were oversampled to encourage spatiotemporal research.
- Approximately half the dataset consists of videos and geographically associated photos, with the remainder randomly selected from Flickr’s CC photo pool.
- Representativeness was evaluated by comparing content and metadata frequencies with an additional random sample of 100 million public Flickr photos and videos from the same period.
2.4 Features and Annotations
YFCC100M provides precomputed multimodal features to reduce the computational burden of working at 100-million-item scale, while annotations remain limited and incomplete.
- Computing features for 100 million media items is time-consuming and computationally expensive.
- Precomputed features allow benchmark participants to focus on the target task rather than data processing.
- Released and planned features span visual, aural, textual, and motion descriptors, including Gist, SIFT, MFCC, closed captions, dense trajectories, and shot boundaries.
- Bounding boxes, object and face segmentations, and image captions were not yet available, although generating them was on the roadmap.
2.5 Ecosystem
The dataset has already supported an ecosystem of multimedia challenges and benchmarks, including geographic-placement and event-detection tasks.
- YFCC100M has generated diverse challenges and benchmarks comparable to community uses of ImageNet, PASCAL, and TRECVID.
- The MediaEval Placing Task uses YFCC100M to benchmark algorithms that estimate where photos or videos were captured.
- YLI-MED was introduced to support research in multimedia event detection.
- Table 4 reports the top 25 of 1,570 visually detected concepts, counting photos and videos each time they contain a concept.
2.6 Strengths and Limitations
YFCC100M emphasizes scale, multimodality, metadata richness, legal openness, and reproducibility, but currently provides limited annotations and cannot replace specialized collections.
- Design: The dataset is designed to be comprehensive, representative, expansive, expandable, free, and legal to use.
- Volume: 100 million media objects make YFCC100M the largest public multimedia collection released at the time described.
- Modalities: Including both photos and videos makes the collection a multimodal multimedia dataset.
- Metadata: Metadata includes machine tags, geotags, timestamps, and cameras, while changing social metadata is omitted but obtainable through the Flickr API.
- Licensing: The dataset provides usage rules addressing licensing, attribution, and copyright.
- Annotations: The current collection has many media objects but limited metadata and annotations, including no guarantee of specialized content such as object segmentations or broadcast videos.
2.7 Guidelines and Recommendations
The dataset’s scale creates substantial resource demands, so researchers should use reproducible, principled subsets and share their selections or derived features.
- Researchers can still run experiments on subsets because individual studies rarely require all 100 million objects.Data requirements and evaluation needs differ across fields.
- Subset construction should avoid arbitrary selections and use principled approaches that can be accurately reproduced.The paper identifies faithful reproduction of evaluation subsets as an important challenge.
- Shared subsets can be distributed as object identifiers, while extracted visual and aural features can also be made available for download.The MediaEval Placing Task is given as an example of sharing training and test sets alongside derived features.
2.8 Future Directions
YFCC100M supports large-scale research across visual, spatial, temporal, social, and cultural questions, while exposing challenges in organizing data and correcting unreliable metadata.
- AI and Vision: Large-scale unsupervised, semi-supervised, and noisy-data learning can use YFCC100M to address new and existing multimedia challenges.The paper highlights semantic connections from visually co-occurring tags and an expansion pack of detected visual concepts.
- AI and Vision: Rich image diversity supports total scene understanding through object localization, interaction, face recognition, and image annotation.These tasks aim to describe what is happening in an image and why it was taken.
- Spatiotemporal Computing: About half the dataset is geotagged, providing a snapshot of photographic activity across space and time for location and event research.The paper identifies time and location as key components for understanding and summarizing events.
- Spatiotemporal Computing: Automatically creating albums from growing personal photo collections requires algorithms that use spatiotemporal context at capture and query time.The challenge concerns organizing media for computers, cloud storage, and mobile devices.
- Spatiotemporal Computing: Incorrect or outdated capture times, locations, and time zones reduce the accuracy of spatiotemporal analysis and motivate correction or resilience methods.GPS metadata may be unavailable or stale, and camera clocks are often misconfigured during travel.
- Digital Culture and Preservation: Creative Commons photostreams and archives capture visual information about culture, memory, storytelling, communication, and identity.Understanding these non-homogeneous digital archives remains a broad sensemaking challenge.
3. CONCLUSIONS
YFCC100M addresses limited reusability, sharing, reproducibility, and transparency in multimedia datasets by providing a large, openly licensed collection with usage guidance.
- 100 million Creative Commons media objects form a comprehensive, expandable, free, and legally usable multimedia collection.The dataset is intended to consolidate and supplant many existing collections.
- The dataset encourages research-method improvement and validation, reduces data-acquisition effort, and stimulates innovation and new data uses.The authors also provide licensing, attribution, copyright, compatibility, and reproducibility guidance.