Source-linked AI summary
Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, Abhinav Gupta
TL;DR
Computer vision needs diverse real-world examples of ordinary daily activities, but such scenes are underrepresented in Internet and movie videos. The paper introduces Hollywood in Homes, crowdsourcing scripts, home recordings, verification, and annotation to build Charades, a large and diverse household-activity dataset. Baseline evaluations show that the dataset provides challenging action-recognition conditions and object-related action confusions.
Problem
Ordinary daily activities are underrepresented in available video sources, limiting evidence about realistic dynamic scenes and human–object interactions.
Method
Hollywood in Homes crowdsources script generation, home video recording, verification, and annotation through Amazon Mechanical Turk.
Results
Charades provides diverse real-world household-activity data, while state-of-the-art baselines achieve 17.2% mAP for action classification and show confusion among actions sharing interacted objects.
Takeaways & Limitations
Charades offers training data and a benchmark for action recognition, video description, and modeling object–action interactions in realistic daily scenes.
Abstract
from arXiv · showhide
Computer vision has a great potential to help our daily lives by searching for lost keys, watering flowers or reminding us to take a pill. To succeed with such tasks, computer vision methods need to be trained from real and diverse examples of our daily dynamic scenes. While most of such scenes are not particularly exciting, they typically do not appear on YouTube, in movies or TV broadcasts. So how do we collect sufficiently many diverse but boring samples representing our lives? We propose a novel Hollywood in Homes approach to collect such data. Instead of shooting videos in the lab, we ensure diversity by distributing and crowdsourcing the whole process of video creation from script writing to video recording and annotation. Following this procedure we collect a new dataset, Charades, with hundreds of people recording videos in their own homes, acting out casual everyday activities. The dataset is composed of 9,848 annotated videos with an average length of 30 seconds, showing activities of 267 people from three continents. Each video is annotated by multiple free-text descriptions, action labels, action intervals and classes of interacted objects. In total, Charades provides 27,847 video descriptions, 66,500 temporally localized intervals for 157 action classes and 41,104 labels for 46 object classes. Using this rich data, we evaluate and provide baseline results for several tasks including action recognition and automatic description generation. We believe that the realism, diversity, and casual nature of this dataset will present unique challenges and new opportunities for computer vision community.
1 Introduction
Hollywood in Homes addresses the lack of diverse, ordinary daily-activity videos by crowdsourcing script writing, recording, and annotation in workers’ homes. The resulting Charades dataset targets realistic human–object interactions while retaining controlled vocabularies and broad diversity.
- Motivation: Internet and movie videos underrepresent boring everyday activities because entertainment-oriented content is more likely to be uploaded or broadcast.YouTube searches for common activities return highly biased samples, while movies and online datasets remain oriented toward exciting or professionally edited scenes.
- Approach: Hollywood in Homes distributes script generation, video direction and acting, and video verification across Amazon Mechanical Turk workers.The approach moves the filming process from controlled laboratories into the homes of hundreds of people.
- Approach: Crowdsourcing provides diversity, expands scenario coverage, and lets the authors control scene, object, action, and video composition.Workers can enact the same script differently, while a proposed vocabulary constrains the content being collected.
- Dataset: Charades is a crowdsourced large-scale dataset of everyday household activities recorded by 267 people across three continents.The dataset includes action classification, localization, and video descriptions, and the paper provides extensive baseline evaluations.
- Positioning: Charades complements existing datasets by combining diverse real-world recordings with controlled object and action vocabularies.The authors contrast it with uncontrolled YouTube collections, professionally edited videos, fixed-background datasets, and synthetic data.
2 Hollywood in Homes
Hollywood in Homes uses a three-stage crowdsourcing pipeline to generate scripts, record videos, verify them, and annotate their contents. The process balances open-ended human activity generation with controlled vocabularies and quality checks.
- Pipeline: The pipeline first generates scripts, then records workers acting them out, and finally verifies script–video correspondence before annotation.The stages mirror filming while distributing content creation and checking across workers.
- Script generation: Workers compose realistic home-activity scripts from one scene, five sampled objects, and five sampled actions, using two objects and two actions.This design controls vocabulary while allowing workers to contribute their own biases and creativity.
- Video recording: Workers record 30-second videos themselves to increase diversity in scenes, objects, clothing, and behavior.The recordings follow the crowdsourced scripts and take place in workers’ own environments.
- Video recording: Recruitment and retention bonuses reduced the video-collection base cost from up to $3 to about $1 per video.A $5 first-submission bonus increased new-worker rates by 211% while increasing total cost by 17%; performance bonuses increased retention by 34%.
- Verification and annotation: Other workers verify submissions and annotate descriptions, interacted objects, action labels, and temporal action intervals.The procedure includes exhaustive action annotation and labeling of each selected action’s start and end points.
- Action classes: Generated verb–proposition–noun triplets are grouped into 157 action classes for the dataset.Examples include pouring into a cup, running, and folding a towel.
3 Charades v1.0 Analysis
Charades combines a compact vocabulary with open-ended scripts to model common household activity sequences. Its controlled acquisition yields broad contextual coverage, with multiple relevant actions appearing in each video.
- Dataset composition: Charades combines 40 objects, 30 actions, and 15 scenes to provide substantial coverage of household activities.The relatively small vocabulary supports standard action-class benchmarking while open-ended writing expands combinations.
- Activity structure: 6.8 relevant actions occur in each video on average because the acquisition process is controlled rather than based on Internet search.The videos contain multiple actions that occur in varied orders and contexts, resembling everyday activity sequences.
4 Applications
Charades supports applications in action classification and sentence prediction using realistic home videos with multiple actions and annotations. Baselines reveal difficult fine-grained recognition and coherent but insufficiently relevant generated captions.
- Evaluation setup: 7,985 training and 1,863 test videos are split by worker, preventing the same worker from appearing in both sets.The split contains 49,809 training and 16,691 test annotated action intervals.
- Action Classification: Action classification evaluates multi-label prediction over 157 action classes using mean average precision (mAP).The test-set labels have 95.6% precision after verification and repeated annotation comparisons.
- Action Classification: 17.2% mAP is achieved by IDT features, outperforming the tested baselines, although performance is lower than on most available benchmarks.Some classes remain easier, including Washing a window at 62.1% AP, while class frequency does not fully explain performance.
- Action Classification: 38.9% mAP is obtained for actions without a specific object of interaction, while fine-grained actions involving the same object remain particularly difficult.The confusion matrix shows most errors among actions involving the same object or objects with similar functional properties.
- Sentence Prediction: Sentence prediction compares baselines on scripts and human descriptions using CIDEr, BLEU, ROUGE, and METEOR, with S2VT as the strongest baseline.Descriptions are more similar to other descriptions than to scripts, and CIDEr has the highest resolution and similarity with human judgment.
- Sentence Prediction: S2VT captions are generally coherent but struggle with relevance and tend to overfit frequent patterns such as drinking from a glass or cup.The generated examples include both low- and high-CIDEr captions, illustrating this gap between language fluency and relevance.
5 Conclusions
The paper presents Hollywood in Homes as a crowdsourced dataset-building approach and positions Charades as a resource for realistic activity understanding and future computer vision research.
- Charades provides 66,500 annotations of actions with unique realism as training data.
- The publicly available dataset and provided baselines enable benchmarking future algorithms.
- Charades contains intricate object-action relationships for developing computer vision techniques targeting these settings.
- Hollywood in Homes offers a framework for collecting novel datasets in new domains.
- Charades supplies realistic human-generated data with complex activity co-occurrences for understanding daily activities.