Source-linked AI summary
Rekall: Specifying Video Events using Compositions of Spatiotemporal Labels
Daniel Y. Fu, Will Crichton, James Hong, Xinwei Yao, Haotian Zhang, Anh Truong, Avanika Narayan, Maneesh Agrawala, Christopher Ré, Kayvon Fatahalian
TL;DR
Domain-specific video events may lack pre-trained detectors, while training new models can be costly and labor-intensive. REKALL composes existing model outputs as spatiotemporal-label queries to specify new events. Across four application domains, expert-authored queries were more accurate than learned approaches on average and also supported video retrieval for human-in-the-loop analysis.
Problem
Pre-trained detectors may not exist for domain-specific video events, and training new models can be costly and labor-intensive.
Source: §1 · p. 1Method
Results
Across four application domains, REKALL queries were 6.5 F1 points more accurate on average and up to 26.1 F1 points more accurate for one task than learned approaches.
Source: §1 · pp. 1–2Takeaways & Limitations
REKALL queries also served as a video retrieval component for human-in-the-loop exploratory analysis.
Source: §1 · p. 2Queries require manual parameter tuning, depend on lower-level labels that may fail in context, and some events are better addressed by learned approaches.
Source: §8 · p. 11
Abstract
from arXiv · showhide
Many real-world video analysis applications require the ability to identify domain-specific events in video, such as interviews and commercials in TV news broadcasts, or action sequences in film. Unfortunately, pre-trained models to detect all the events of interest in video may not exist, and training new models from scratch can be costly and labor-intensive. In this paper, we explore the utility of specifying new events in video in a more traditional manner: by writing queries that compose outputs of existing, pre-trained models. To write these queries, we have developed Rekall, a library that exposes a data model and programming model for compositional video event specification. Rekall represents video annotations from different sources (object detectors, transcripts, etc.) as spatiotemporal labels associated with continuous volumes of spacetime in a video, and provides operators for composing labels into queries that model new video events. We demonstrate the use of Rekall in analyzing video from cable TV news broadcasts, films, static-camera vehicular video streams, and commercial autonomous vehicle logs. In these efforts, domain experts were able to quickly (in a few hours to a day) author queries that enabled the accurate detection of new events (on par with, and in some cases much more accurate than, learned approaches) and to rapidly retrieve video clips for human-in-the-loop tasks such as video content curation and training data curation. Finally, in a user study, novice users of Rekall were able to author queries to retrieve new events in video given just one hour of query development time.
1. INTRODUCTION
The paper addresses the cost and scarcity of models for domain-specific video events by proposing REKALL, a library for composing existing model outputs into event queries. Across four application domains, expert-authored queries matched or exceeded learned approaches, and the queries also supported retrieval and novice-authored event detection.
1. INTRODUCTION
- Domain-specific event models may be unavailable, while training new ones can require costly labeling, computation, and human expertise.Analysts may have only a small number of labeled examples, if any.Source: §1 · p. 1
- REKALL lets analysts specify new video events by composing outputs of existing pre-trained models, avoiding additional model training while allowing iterative query refinement.Its data and programming models adapt ideas from multimedia databases and complex event processing to video.Source: §1 · p. 1
- Across four application domains, REKALL queries were 6.5 F1 points more accurate on average and up to 26.1 F1 points more accurate for one task than learned approaches.The queries also served as a video retrieval component for human-in-the-loop exploratory analysis.Source: §1 · pp. 1–2
- The evaluation spans cable TV news, Hollywood films, static-camera vehicular streams, and autonomous vehicle logs.In the user study, novices had one hour to develop a query; after one-line expert tuning, queries achieved average precision above 94.Source: §1 · pp. 1–2
2. AN ANALYSIS EXAMPLE
The running example asks how to identify interviews with a political candidate using face detections and time-aligned transcripts. REKALL queries compose these annotations into event patterns, which analysts can refine after inspecting results.
2. AN ANALYSIS EXAMPLE
- The example assumes no pre-trained interview detector, but assumes access to face identification and time-aligned transcripts.The motivating task is tabulating time spent interviewing a political candidate in TV news.Source: §2 · p. 2
- An interview detector can seek a temporal pattern of the candidate and host appearing together, interleaved with candidate headshots.Refinements may allow brief off-screen intervals or align sequence parts with the candidate’s name or phrases such as “welcome.”Source: §2 · p. 2
- Query inputs can differ in modality and sampling rate, contain annotation noise, and require operators for temporal and spatial relationships.Figure 2 depicts nested labels for face-on-screen segments and frame-level detections, alongside caption and word labels.Source: §2 · p. 2, Figure 2 · p. 3
3. SPATIOTEMPORAL LABELS
REKALL represents multimodal video annotations as spatiotemporal labels, enabling queries to combine inputs sampled at different rates. Labels can be nested to represent higher-level segments containing lower-level annotations.
3. SPATIOTEMPORAL LABELS
- A spatiotemporal label associates an annotation with a continuous, axis-aligned interval of space and time in a video, and may include metadata.A face detection at 2:10 in a 30 fps video spans its bounding box and the interval from 2:10.0 to 2:10.333.Source: §3 · p. 3
- Labels can represent different annotation sources and rates: face detections span a frame, while transcript word labels span each utterance and occupy the full frame space.The examples use a 3D (X,Y,T) video domain; REKALL also supports additional spatial dimensions, such as 4D LIDAR point-cloud video.Source: §3 · p. 3
- A label’s metadata can contain other labels, allowing continuous face detections to form an on-screen segment and a phrase label to contain its constituent words.Queries can build further hierarchies, such as organizing frames into shots and shots into scenes or news segments.Source: §3 · p. 3
4. COMPOSING LABELS
REKALL queries compose existing labels into new labels for video events, using operations on label sets to express temporal and spatial relationships.
4. COMPOSING LABELS
- REKALL queries compose existing labels into new labels that correspond to event instances in video.Queries produce and consume label sets, each representing occurrences of an event.Source: §4 · p. 3, §4.1 · p. 3
4.1 Label Sets
- Label-set operations include map, filter, and group_by for manipulating labels, selecting subsets, and organizing nested labels.group_by can group face detections from the same frame into nested labels.Source: §4.1 · p. 3
4.2 Coalesce: Recursive Label Merging
- coalesce recursively merges nearby labels until its predicate permits no further merges, building higher-level labels from fine-grained inputs.It can merge labels within a specified temporal or spatial proximity and can smooth noise such as detector flicker.Source: §4.2 · p. 4
4.3 Joins
- Joins combine label streams using spatiotemporal predicates, while minus removes matched intervals from the first label set.Examples include finding times when Sanders and Tapper are both on screen, or subtracting intervals for one person from another’s.Source: §4.3 · p. 5
5. APPLICATIONS
REKALL queries support video analysis across news, films, fixed-camera parking feeds, and autonomous vehicle logs. The examples show how queries address domain-specific tasks, support clip retrieval, and help identify errors in model outputs.
5. APPLICATIONS
- REKALL has been used for video analysis across TV news, Hollywood films, fixed-camera vehicular feeds, and autonomous vehicle logs.Queries support both large-collection analysis and retrieval of clips for human-in-the-loop tasks.Source: §5 · p. 5
5.1 Analysis of Cable TV News Media
- News analysis queries detect commercials and interviews with specified political candidates, using broadcast-specific signals to identify commercials.In practice, the interview query can be adapted to any known host with a one-line change.Source: §5.1 · p. 6
5.2 Film Cinematography Studies
- Film queries identify shot boundaries, classify shot scale, detect conversations, and retrieve cinematography idioms for film analysis and clip curation.The 589-film study also used queries to retrieve idioms for a Star Wars trailer workflow that took less than four hours after queries were written.Source: §5.2 · p. 6
5.3 Static-Camera Vehicular Video Streams
- PARKING detects free spaces by subtracting detected vehicles from parking-spot intervals and requiring a continuous four-minute car-free period.The query assumes all spaces are occupied at the start; this assumption can be relaxed by identifying areas occupied for a significant period.Source: Figure 7 · p. 7, §5.3 · pp. 7–8
5.4 Mining Autonomous Vehicle Logs
- Autonomous vehicle queries find rapid traffic-light changes and nearby vehicle turns, directing human labelers to potentially important clips.Labelers can annotate ground truth, identify detector failures for an active-learning loop, or validate turn-signal and vehicle prediction and control systems.Source: §5.4 · p. 8
5.5 Model Debugging
- REKALL queries expose errors in pretrained models, including missed face detections and false positives from a gender classifier.In the gender-classifier example, most retrieved scenes were false positives, leading the team to replace the classifier with a better model.Source: §5.5 · p. 8
6. EVALUATION
REKALL queries performed on par with or better than learned baselines across most evaluated tasks, while supporting rapid query execution and authoring by domain experts and novices. In the novice study, queries exceeded 94 AP after expert modifications to address object-detector errors.
6.1 Query Accuracy
- Across six tasks, REKALL queries were on par with or more accurate than learned baselines in five, averaging 6.5 more F1 points and reaching a 26.1-point gain in one task.The comparisons use held-out test data; learned baselines were trained on development-set labels.Source: §6 · p. 8, §6.1 · p. 8, Table 2 · p. 9
6.1.1 Classification Tasks
- REKALL scored higher than the best learned model on four of five classification tasks, averaging 6.5 more F1 points; its largest margin was 26.1 points on SHOT SCALE.The DeepSBD shot detector outperformed REKALL on SHOT DETECT, while a weakly supervised model later matched DeepSBD at 91.3 F1 using 636× less ground-truth data.Source: §6.1.1 · p. 9, Table 2 · p. 9
6.1.2 Object Detection Tasks
- Both REKALL and the learned baseline exceeded 98.0 AP on PARKING, which evaluates detection and localization of empty parking spots.The learned baseline was a fine-tuned Faster R-CNN model.Source: §6.1.2 · p. 9, Table 2 · p. 9
6.1.3 Query Performance
- All final queries ran in under 30 seconds on development sets, enabling iterative development despite minimal implementation optimization.Labels were precomputed; the authors identify initial model evaluation as the primary computational cost.Source: §6.1.3 · p. 10
6.2 Usability
- Domain experts with no prior REKALL experience solved four representative tasks, with programming efforts ranging from an afternoon to two days.Their work included handling noisy source annotations and misaligned transcripts.Source: §6.2 · p. 10
- After a one-hour tutorial and one hour of query development, users 1–7 achieved above 94 AP after expert modifications for detector confusion and, for one user, parking-spot construction.The authors also report that success was strongly correlated with self-rated functional-programming familiarity; three users who iterated with coalesce improved AP by an average of 11.5 points.Source: §6.2 · p. 10, Table 3 · p. 10
7. RELATED WORK
REKALL adapts spatiotemporal composition to modern video annotations from machine-learning models. The related work distinguishes this programmatic approach from language-distribution-limited retrieval and connects it to event processing, few-shot learning, and weak supervision.
Multimedia Database Systems
- REKALL updates multimedia-database approaches to video querying by composing the richer annotations produced by modern machine-learning techniques.Earlier systems supported spatiotemporal joins and repeated-pattern queries, while REKALL uses modern primitive annotations and real-world tasks.Source: §7 · p. 10
Domain-Specific Video Retrieval Systems
- Domain-specific retrieval systems can complement REKALL: their events can be modeled as labels, then combined into broader patterns.The paper names SceneSkim, RoughCut, and Chalkboarding as related systems.Source: §7 · pp. 10–11
Complex Event Processing and Diverse Analytics Systems
- REKALL uses continuous intervals to combine video annotations sampled at different rates, unlike point-event systems such as Apache Flink and SiddiQL.The paper says Trill has interval-based operations and anti-semi joins, while an equivalent of coalesce does not appear directly.Source: §7 · p. 11
Few-Shot and Zero-Shot Learning
- Programmatic composition can serve as few-shot or zero-shot event detection and can also supply curated examples to bootstrap few-shot learning.When no examples are available, REKALL retrieval with human-in-the-loop curation can source an initial small labeled set.Source: §7 · p. 11
Weak Supervision Systems
- REKALL queries can provide richer video heuristics for weak supervision, including a SHOT DETECT model matching a method trained with 636× more ground-truth data.The paper notes that further work may be needed to adapt weak-supervision techniques to video.Source: §7 · p. 11
Video Retrieval Through Natural Language Interfaces
- Unlike natural-language localization limited to descriptions represented in training and test distributions, programmatic composition lets analysts encode complex concepts from domain knowledge.The paper gives “Jake Tapper interviews Bernie Sanders” as an example that may fail without suitable training coverage.Source: §7 · p. 11
8. DISCUSSION
REKALL offers a programmatic alternative to end-to-end learning for rapidly specifying video events from existing labels. The authors also identify limits in compositional queries and suggest roles for them within broader machine-learning pipelines.
8. DISCUSSION
- Heuristic composition lets analysts rapidly specify events, reuse existing labels, express domain knowledge, and debug failure modes without large-scale annotation and model-training costs.The authors contrast this with end-to-end learning from raw data.Source: §8 · p. 11
- Compositional queries require manual threshold tuning and depend on lower-level labels that may be absent or fail; some events are better addressed by learned approaches.The authors report that face and pose detector failures under embrace occlusions prevented a reliable kissing-scene detector.Source: §8 · p. 11
- The authors propose using compositional systems to surface training examples, search for anomalous model outputs, and provide weak supervision within machine-learning pipelines.They hope future systems can move more seamlessly between traditional query programs and learned models.Source: §8 · p. 11
10. APPENDIX
The appendix documents supplementary retrieval examples and query designs for commercials, shot transitions, conversations, and film idioms. It also details how REKALL queries were adapted into labeling functions for weakly supervised shot detection.
10.1 Supplemental Videos
- Supplementary videos demonstrate REKALL retrievals including film idioms, interviews, reaction shots, action sequences, dialogue phrases, and character arrangements.The examples include a Star Wars trailer assembled from mined clips and supercuts from several films.Source: §10.1 · p. 14
10.2.1 Commercial Detection
- The COMMERCIAL query uses black-frame sequences, speaker markers, and mixed-case or missing transcripts as dataset-specific cues, with coalesce joining temporally separated labels.Figure 9 presents the query.Source: §10.2.1 · p. 14, Figure 9 · p. 14
10.2.2 Shot Transition Detection
- The SHOT DETECT query flags abrupt histogram changes and removes candidates when face counts and positions match across a transition.Figure 10 shows the query; its output can be converted into contiguous shot labels for later queries.Source: §10.2.2 · p. 14, Figure 10 · p. 15
10.2.3 Conversation Detection
- The CONVERSATION query uses repeated appearances of the same people across neighboring shots, clustering faces by embeddings and coalescing shot-pair sequences.It avoids reliance on unreliable captions and off-the-shelf identity recognition; resulting sequences must span at least three shots.Source: §10.2.3 · pp. 14–15, Figure 11 · p. 15
10.2.4 Film Idiom Mining
- Film-idiom mining covered six trailer patterns, including establishing shots, ponderous statements, action sequences, dark tidings, and hero shots.The listed hero-shot query uses a manually set brightness threshold; the appendix also describes hopeful shots of main characters.Source: §10.2.4 · p. 16
10.3 Weak Supervision
- For SHOT DETECT weak supervision, the authors split one query into five labeling functions using RGB, HSV, and optical-flow histograms plus two face-based rules.Snorkel estimated labeling-function accuracies from agreements and disagreements, then generated probabilistic labels for training a detector that reached 91.3 F1 on the held-out test set.Source: §10.3 · p. 16