Source-linked AI summary
From Lifestyle Vlogs to Everyday Interactions
David F. Fouhey, Wei-cheng Kuo, Alexei A. Efros, Jitendra Malik
TL;DR
Everyday interactions are poorly represented by training data because they are rarely tagged and difficult to gather at scale. This paper mines Lifestyle Vlogs, annotates them, and benchmarks hand-focused tasks. The authors report that implicit gathering yields a large, diverse dataset and exposes biases in explicitly gathered data.
Problem
Training data for everyday interactions is scarce because direct searches for rarely tagged actions fail, while scripted collection is difficult to scale and diversify.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Abstract
from arXiv · showhide
A major stumbling block to progress in understanding basic human interactions, such as getting out of bed or opening a refrigerator, is lack of good training data. Most past efforts have gathered this data explicitly: starting with a laundry list of action labels, and then querying search engines for videos tagged with each label. In this work, we do the reverse and search implicitly: we start with a large collection of interaction-rich video data and then annotate and analyze it. We use Internet Lifestyle Vlogs as the source of surprisingly large and diverse interaction data. We show that by collecting the data first, we are able to achieve greater scale and far greater diversity in terms of actions and actors. Additionally, our data exposes biases built into common explicitly gathered data. We make sense of our data by analyzing the central component of interaction -- hands. We benchmark two tasks: identifying semantic object contact at the video level and non-semantic contact state at the frame level. We additionally demonstrate future prediction of hands.
1. Introduction
Prior interaction datasets either miss everyday actions that are rarely tagged or rely on scripted collection that is difficult to scale and diversify. The paper mines Lifestyle Vlogs for naturally occurring interactions, then analyzes the resulting data through hand-focused tasks.
1. Introduction
- Past explicit searches miss everyday interactions because they are rarely tagged, while collection-by-acting is difficult to scale and diversify.Explicit datasets also tend to contain atypical examples and gaps in categories and negatives.Source: §1 · p. 1
- Implicitly mining Lifestyle Vlogs yields VLOG, a larger and more diverse dataset of everyday interactions than prior efforts.The paper describes a 14-day collection of 114K videos from 10.7K uploaders.Source: Figure 1 · p. 1, §1 · p. 2
- The paper proposes implicit gathering: mine interaction examples from a broader video genre whose videos have different tags, then annotate and analyze them.Lifestyle Vlogs are the paper’s example of this broader data source.Source: §1 · pp. 1–2
- The authors benchmark hand contact state at the frame level and whether hands contact specified objects at the video level.They also provide scene categories and hand bounding boxes to explore the data.Source: §1 · p. 2
- The paper also demonstrates hand future prediction and examines biases in popular explicitly gathered datasets.These analyses use VLOG and its annotations to explore naturally occurring interactions.Source: §1 · p. 2
2. Related Work
The paper situates its work among efforts to study affordances and action recognition, where interaction-rich video data remains difficult to gather at scale with diversity. It distinguishes its implicit collection approach from direct search and scripted data collection.
2. Related Work
- Interaction-rich video at scale remains difficult to obtain while maintaining diversity and coverage, despite large-volume egocentric and life-logging efforts.The paper presents its dataset and benchmarks as contributions to studying humans in the wild.Source: §2 · p. 2
- Unlike action datasets that target non-everyday actions or gather everyday actions by having people act them out, this work mines everyday actions implicitly.The authors say this approach achieves greater scale and diversity while avoiding some sources of data bias.Source: §2 · pp. 2–3
- The gathering pipeline searches templated queries, screens videos and clips, and uses human cleanup as the data volume falls and purity rises.The figure depicts an example of the paper’s implicit mining process.Source: Figure 3 · p. 3
3. Collecting Interaction-Rich Episodes
The authors mine Lifestyle Vlogs for interaction-rich episodes, using a semi-automatic pipeline that progressively filters videos and clips before human review. The resulting dataset is intended to retain scale while preserving useful properties of manually gathered data.
3. Collecting Interaction-Rich Episodes
- Lifestyle Vlogs document purportedly ordinary daily life; although routines may be aspirational, interaction moments such as pouring coffee are described as realistic.Interaction clips are interspersed with camera pans and monologues.Source: §3 · p. 3
- The pipeline generates candidate videos, filters them with relevance feedback, segments them into clips, filters clips by motion and content, then uses annotators.It defines positive videos as third-person indoor interactions and excludes makeup-only and purchase-unboxing videos.Source: §3 · p. 3
- The authors use 823 templated queries across English and 13 European languages to retrieve 216K candidate videos, with 23% purity at video level.Query templates target themes or activities, such as daily routines and tidying a bedroom.Source: §3 · p. 3
- Thumbnail-based screening reduces the candidate pool to 20K videos after labeling 1.5K examples and training a linear SVM on Alexnet features.The initial pool is described as too large and noisy to download directly.Source: §3 · p. 3
- Camera-motion segmentation yields 572K static clips, and subsequent filtering and human review produce 114K clips with 82% precision for the automatic stages.The clips are screened for relevance, adult content, and whether people touch something with their hands.Source: §3 · p. 3
- The dataset comparison table highlights VLOG’s scale alongside high demonstrator diversity, high resolution, static cameras, and open-world-like data.The table legend identifies diversity, resolution, viewpoint, and implicit versus explicit gathering as comparison dimensions.Source: Table 1 · p. 4
4. The VLOG Dataset
VLOG combines substantial volume with unusual source diversity, and its collection-before-labeling design permits relabeling and use of videos for unsupervised learning. The authors also note a trade-off between crawling’s lack of control and resolution approaching that of in-lab efforts.
4. The VLOG Dataset
- Because VLOG was gathered before labels were defined, its videos can be relabeled and can serve as labels for unsupervised learning.The authors describe the annotations as a starting point for analyzing the data.Source: §4 · p. 4
- VLOG has ≈9.4× more sources than Something-Something and ≈40× more than Charades, with a more balanced uploader distribution and 58 sampled frames expected before seeing the same person.Its reported uploader Gini coefficient is 0.74, compared with 0.57 for Charades; the frame comparison is 2 for CAD-120, 3 for Watch-n-Patch, and 10 for Charades.Source: §4 · p. 4
- Crawling offers less control than direct gathering, but VLOG’s average resolution approaches that of in-lab efforts.The authors attribute this partly to content creators’ intrinsic motivation or advertising income.Source: §4 · p. 4
- HICO contains 59 refrigerator and 22 microwave interaction instances, while VLOG has far more despite not directly searching for either interaction.The comparison concerns image-based datasets that depend on people taking, uploading, and tagging interaction photos.Source: §4 · p. 4
5. Labels
VLOG labels capture both the long tail of everyday object interactions and hand contact across objects, using clip-level semantic labels and frame-level contact states. Additional annotations characterize objects, scenes, people, and hand locations.
5. Labels
- A 500-image sample suggests VLOG contains 346 entry-level object categories, before finer object or interaction distinctions.The estimate uses standard species-richness techniques.Source: §5 · p. 4
5.1. Hand/Semantic Object Annotations
- VLOG averages 5× as many examples across 15 overlapping object categories as the largest prior dataset, though it has fewer door examples than Charades.The comparison also notes major category gaps in prior datasets and limited diversity in some of their examples.Source: §5.1 · p. 5
5.2. Hand Contact State Annotations
- Frame-level hand-contact labels describe whether visible hands touch something other than the person’s body or clothing, regardless of object category.The annotations cover a random 219K-frame subset and define six hand states within eight total categories.Source: §5 · pp. 4–5, §5.2 · p. 5
5.3. Additional Data Annotations
- A 20K-image annotation subset adds object presence, five scene classes, proxemic distance, human visibility configurations, and hand locations.The five scene classes cover 76% of the data, the common visibility configurations capture 92%, and hand boxes are annotated on 5K images.Source: §5.3 · pp. 5–6
6. Exploring Biases of Explicit Data
Off-the-shelf scene classification and object detection perform worse on VLOG than on their original datasets. The authors connect these failures to biases in explicitly gathered data, including mismatched scene imagery and missing negatives.
Scene Classification
- 43% top-5 accuracy on VLOG contrasts with 85% on Places365, while a no-fine-tuning linear SVM reaches 70% top-1 accuracy on VLOG.The authors report human consensus on the frames and attribute the scene-classification gap to domain shift between tagged scene images and images from those scenes.Source: §6 · p. 6
- Scene-classification accuracy falls to 66.5% at social distance, 48.2% at personal distance, and 28.2% at intimate distance.The authors describe the source images as sterile scenes, such as uncluttered kitchens and made beds, unlike views of everyday activity.Source: §6 · p. 6
Object detection
- 44% precision on VLOG contrasts with 96% on COCO for refrigerator detection at the stated operating point, with similar recall.The authors hypothesize that missing negatives in explicitly gathered data contribute to false positives, including detections based on blobby textures and shelf-like patterns.Source: §6 · p. 6, Figure 7 · p. 6
7. Benchmarking Interactions
This section benchmarks existing methods on semantic object contact, hand contact state, and hand detection, with additional analysis of whether learned features represent hands.
7.1. Hand/Semantic Object Interaction
- 78% of the best method’s performance comes from a single frame; FT I3D-K drops from 39.7% to 27.0% at social distance.Fine-tuning improves semantic-contact results, with larger gains for I3D than ResNet-50.Source: §7.1 · p. 7
7.2. Hand Contact State
- Across present and 6–60-frame hand-state prediction, models beat the 20% training-mode baseline, and pseudo-labels consistently boost accuracy.The most common confusions involve counting visible hands.Source: §7.2 · p. 7, Table 3 · p. 7
- Hand-box IoU reaches 31.3/40.5/47.8 with 10/100/3000 labeled images, versus 17.3/33.5/41.5 for the ImageNet-pretrained model.This feature-decoding result suggests the contact-task network has learned about hands.Source: §7.2 · pp. 7–8
7.3. Hand Detection
- VLOG-trained hand detection reaches 56.3 AP on a third dataset, versus 31.4 AP for EgoHands training.On VLOG, EgoHands-trained detection gets 29.5 AP, compared with 67.6 AP when trained on VLOG.Source: §7.3 · p. 8
8. Exploring A Large Set of Hands in Contact
The paper demonstrates predicting future hand locations from large-scale web video, with outputs showing plausible motion patterns, including hands holding cellphones.
8. Exploring A Large Set of Hands in Contact
- The authors identify future prediction, intuitive physics, imitation learning, and grasp analysis as potential applications of VLOG.Future hand-location prediction is their concrete demonstration.Source: §8 · p. 8
- The model predicts future hand locations from an image by combining image features with a feature map for the requested time horizon.It is trained for δ = 2,…,32 using segmentation predictions on 156K frames with significant change; training requires stationary-camera video.Source: §8 · p. 8
- Predictions show plausible hand dynamics: hands often continue in likely motion directions, while people holding cellphones rarely put them down.The paper presents predictions on held-out data at varying timescales.Source: §8 · p. 8
9. Discussion
The discussion argues that implicit collection can uncover poorly documented everyday interactions, while acknowledging that understanding them remains an open challenge.
9. Discussion
- The authors conclude that explicitly asking for target actions can be counterproductive, and that VLOG data offers a path toward understanding everyday interactions.They note that computer vision can automate much of the collection process, but that substantial work remains.Source: §9 · p. 8
A. Acquisition Pipeline Details
- The acquisition pipeline finds video candidates using thumbnail features, then detects episode boundaries with SIFT homography fitting and RANSAC.It first tests for identity transformations to save computation and scans forward at 30, 60, and 90 frames to check matches.Source: A. Acquisition Pipeline Details · p. 10
- Checking the 10 middle frames after coarse shot detection helps avoid premature cuts in aggressively trimmed videos.For N frames and k shots, the procedure uses N/10 + 10*(k − 1) SIFT feature extractions rather than N.Source: A. Acquisition Pipeline Details · p. 10
B. Additional Experimental Details, Statistics
The supplement points to Figure 9 for additional statistics describing VLOG’s composition.
B.1. Additional statistics
- Figure 9 reports VLOG’s sources and distributions of scene depth and class.Source: §B.1 · p. 10
- Figure 9 includes a cumulative distribution of uploaders and video lengths, with average video length of approximately 10 seconds.Source: §B.1 · p. 10
- Figure 9 also reports the visibility of various body types in VLOG.Source: §B.1 · p. 10
B.2. Entry-level categories in our sample
The appendix reports touched-object category counts from a 500-image sample and describes the hand-prediction network architecture.
B.2. Entry-level categories in our sample
- In the 500-image sample, bottle (36) and cellphone (31) were the most frequent listed touched-object categories, while many categories appeared only once.The object names are post-canonicalization; some broad labels may stand for more specific objects, and the categories do not include verbs.Source: §B.2 · pp. 10–11
- Object labels were canonicalized and omit verbs; broad labels such as “bottle” may represent more specific objects such as shampoo.Source: §B.2 · p. 10
B.3. Hand prediction
- Hand prediction fuses 512-channel image features with a 64D encoding of time offset, then applies three convolutions, 8× bilinear upsampling, and cross-entropy training.The network is ζ(cat(φ(I), ψ(δ))); C(k, s) denotes a convolution of k kernels of size s×s, R is ReLU, and BN is Batchnorm.Source: §B.3 · p. 11