PaperlayerView exact revision on arXiv

Source-linked AI summary

MovieGraphs: Towards Understanding Human-Centric Situations from Videos

Paul Vicol, Makarand Tapaswi, Lluis Castrejon, Sanja Fidler

arXiv:1712.06761v2cs.CV

TL;DR

Socially intelligent agents need to understand emotions, motivations, relationships, and other factors shaping human behavior, but existing video understanding often represents such information implicitly. MovieGraphs introduces a graph-annotated movie-clip dataset and evaluates graph-based retrieval, interaction ordering, and reason prediction; the authors report rich representations that support clip localization and semantically meaningful retrieval.

  • Problem

    Socially intelligent agents need to read emotions, motivations, relationships, and other factors affecting behavior in human social situations.

  • Method

    MovieGraphs annotates 7,637 movie clips with graphs encoding characters, interactions, relationships, attributes, and inferred reasons, with visual and temporal grounding.

  • Results

    The graphs contain sufficient information to localize video clips and retrieve semantically meaningful situations, while supporting interaction ordering and reason prediction.

  • Takeaways & Limitations

    MovieGraphs provides a benchmark for analyzing inferred properties of human-centric situations through structured graph representations.

Abstract

from arXiv · show

There is growing interest in artificial intelligence to build socially intelligent robots. This requires machines to have the ability to "read" people's emotions, motivations, and other factors that affect behavior. Towards this goal, we introduce a novel dataset called MovieGraphs which provides detailed, graph-based annotations of social situations depicted in movie clips. Each graph consists of several types of nodes, to capture who is present in the clip, their emotional and physical attributes, their relationships (i.e., parent/child), and the interactions between them. Most interactions are associated with topics that provide additional details, and reasons that give motivations for actions. In addition, most interactions and many attributes are grounded in the video with time stamps. We provide a thorough analysis of our dataset, showing interesting common-sense correlations between different social aspects of scenes, as well as across scenes over time. We propose a method for querying videos and text with graphs, and show that: 1) our graphs contain rich and sufficient information to summarize and localize each scene; and 2) subgraphs allow us to describe situations at an abstract level and retrieve multiple semantically relevant situations. We also propose methods for interaction understanding via ordering, and reason understanding. MovieGraphs is the first benchmark to focus on inferred properties of human-centric situations, and opens up an exciting avenue towards socially-intelligent AI agents.

1. Introduction

MovieGraphs addresses the challenge of understanding human-centric social situations by representing movie clips with structured, grounded graphs. The dataset supports analysis of social patterns and graph-based retrieval and understanding tasks.

  • Dataset and motivation: The graphs model social behavior in context, including interpersonal relationships, locations, situations, emotions, and motivations.Graphs are more structured than natural language and can ground information in video.
  • Dataset and motivation: MovieGraphs contains 7,637 movie clips annotated with graphs representing characters, interactions, relationships, attributes, and inferred reasons.Each clip also has situation and scene labels plus a natural-language description.
  • Dataset and motivation: Annotations are grounded to video through character face tracks and time intervals for most interactions.This links graph entities and events to their visual and temporal evidence.
  • Tasks and analysis: The dataset supports common-sense correlation analysis, graph-based video retrieval, interaction ordering, and reason prediction.The authors report that graphs contain sufficient information to localize clips, while subgraphs retrieve semantically meaningful situations.
  • Relation to prior work: MovieGraphs extends video understanding beyond action classification and implicit representations by focusing on inferred human behavior in videos and dialogue.Its graph representation is designed to make social information more interpretable and queryable than typical internal representations.

4. Situation Understanding Tasks

MovieGraphs represents human-centric situations as structured graphs and evaluates them through retrieval, interaction ordering, and reason prediction. The retrieval model aligns graph characters with video tracks and combines visual and dialog evidence, while the other tasks use social context to predict plausible sequences or motivations.

  • Task overview: The paper uses graph-based video retrieval, interaction sorting, and reason prediction to evaluate different aspects of situation understanding.Graphs encode characters, attributes, relationships, interactions, topics, and reasons.
  • Graph-based retrieval: Graph queries represent character, attribute, relationship, interaction, topic, and reason nodes connected by directed edges.Scene and situation labels are treated as disconnected graph nodes, while other nodes generally connect to characters.
  • Graph-based retrieval: Retrieval scores graph components against aligned face and person tracks, global scene features, and dialog to identify semantically matching clips.The model learns separate potentials for visual graph evidence and dialog, then combines their scores through late fusion.
  • Graph-based retrieval: The retrieval model embeds graph labels with GloVe and learns shared and type-specific linear mappings for comparing query nodes with observations.Visual potentials score attributes, scene and situation labels, and pairwise relationships or interactions.
  • Learning and inference: Training uses ground-truth graph–clip alignments with max-margin ranking and negatives from other movies, the same movie, or incorrect within-clip alignments.Inference exhaustively searches clips and character alignments for the highest-scoring match.
  • Interaction ordering and reason prediction: Interaction ordering trains an attention-based GRU to select pairwise interactions sequentially using situation, scene, relationship, and attribute context.Reason prediction encodes scene, situation, interaction, and relationship context into a 100-dimensional vector and generates a reason word by word.

5. Experimental Results

Experiments evaluate graph quality, retrieval across descriptions, dialog, and visual clips, interaction ordering, and reason prediction. Graphs support meaningful retrieval and ordering, while visual localization remains sensitive to person identification.

  • Face Clustering and Person Identification: 75.8% weighted clustering purity and 43.7% track-level identification accuracy were obtained, compared with 13.2% chance identification.The evaluation uses clips spanning many decades and retains background characters and false-positive tracks.
  • Description Retrieval: Median rank was consistently 1 for graph-to-description retrieval with TF·IDF, max-sum, and max-sum · idf models.Descriptive topics, reasons, and character names may help localize scenes.
  • Dialog Retrieval: Dialog retrieval was harder: GloVe models reached med.-R 10 versus 6 for TF·IDF.The authors attribute this difference to similar embeddings across several word classes.
  • Movie Clip Retrieval: Scene-only retrieval improved median rank from 764 for random retrieval to 141.5, while scene-plus-situation retrieval reached 140.Directly mapping high-level situations to visual cues remains challenging.
  • Movie Clip Retrieval: Predicted clustering achieved median rank 19 with characters, but predicted clustering plus identification worsened it to 69; ground-truth clustering and identification reached 17.Person identification difficulty limits visual localization.
  • Ablations and Qualitative Results: Including interactions, topics, and relationships improved rank from 84 to 59, while ground-truth clusters and identification improved median rank from 59 to 10.Late fusion of video and dialog cues further increased performance, and small identity-agnostic subgraphs retrieved meaningful clips.
  • Interaction Ordering: The ordering model is evaluated with complete-sequence accuracy and longest common subsequence length, including plausible partial-order successes in a failure case.Qualitative examples show correctly sorted sequences and a failure whose interactions 2–5 remain correctly ordered.
  • Reason Prediction: Among 72 subgraphs with clear annotator agreement, 11 predicted reasons were rated very relevant and 10 semi-relevant.Ten AMT workers scored 100 subgraphs as very relevant, semi-relevant, or not relevant.

6. Conclusion

The paper introduces MovieGraphs as a graph-based benchmark for understanding human-centric situations in videos. Its tasks cover retrieval, interaction ordering, and reason prediction, with models showing both successes and challenges.

  • Dataset and Tasks: MovieGraphs represents everyday social situations with annotations of interactions, emotions, and motivations inferred from visual cues and dialog.The dataset supports three benchmark tasks: graph-based video retrieval, interaction ordering, and reason prediction.
  • Models and Evaluation: The proposed models address graph-based video retrieval, interaction ordering, and reason prediction, revealing successes and challenges across situation understanding.The conclusion frames these tasks as benchmarks for human-centric video understanding.

Supplementary Material

The supplementary material expands the paper’s dataset, models, results, annotations, and analyses. It includes ablations, qualitative examples, annotation details, distributions, and character-emotion visualizations.

  • Additional Details: Supplementary sections provide additional details about the dataset, models, and results.They also include a rooted graph of situations and examples of interaction annotations.
  • Retrieval Ablation: The material reports a TF·IDF retrieval ablation analyzing the discriminative power of each node type.It covers both graph-to-description and graph-to-dialog retrieval.
  • Models and Annotation: Further sections detail interaction ordering, reason prediction, person detection, clustering, identification, and the annotation interface.These sections add models, qualitative results, and data-collection information.
  • Dataset Statistics: Supplementary analyses include distributions of characters, interactions, and relationships per clip, plus the top-20 relationships and scenes.These provide descriptive statistics about the dataset’s graph contents.
  • Examples and Visualizations: Additional examples and visualizations show emotional profiles and timelines of main characters across several movies.They also visualize emotions of characters on both sides of interactions.

A. TF·IDF Ablative Study

The TF·IDF ablation evaluates which graph node types best discriminate clips for description and dialog retrieval. Character, topic, and reason nodes are the strongest localizing signals in both settings.

  • Ablation Setup: The ablation restricts retrieval to selected node types to measure their relative discriminative power.Test-set results use recall @ {1, 5, 10} and median rank.
  • Findings: Character, topic, and reason nodes are the most discriminative for both graph-to-description and graph-to-dialog retrieval.These node types correspond to who appears and the details and motivations of interactions.

B. Interaction Ordering Details

The interaction-ordering model uses an attention-based GRU to select interactions sequentially from an input set, conditioning on graph context to produce plausible social-event orders.

  • Interaction Sequence Encoding: Each training example contains scene and situation labels, character relationships and attributes, and a sequence of timestamped interactions with topics.Interactions are ordered by start time, then end time to break ties.
  • Context Encoding: The model encodes characters, relationships, scene, situation, and attributes into a global context vector passed to the decoder at every step.Character identities are represented through attributes, while graph components are encoded through linear layers.
  • Attention-Based Decoder RNN: A single-layer GRU uses the current input element, an attention-derived local context, and the global context to select interactions sequentially.The decoder attends to input elements and receives their concatenated representations at each time step.
  • Decoding and Training: At test time, the highest-scoring unselected interaction is chosen at each step, while training uses cross-entropy with teacher forcing half the time.Scores are inner products between the decoder output and interaction representations; selected inputs are masked afterward.
  • Results: Qualitative results show exact matches in some examples and plausible but non-identical orders in others, including failure cases with long correct subsequences.The evaluation reports complete-sequence accuracy and longest common subsequence length.

C. Reason Prediction Details

The reason-prediction model encodes interaction context with graph features and generates reasons word by word using a GRU decoder; qualitative outputs are often plausible despite disagreements with ground truth.

  • Model Overview: The model predicts reasons by encoding the interaction’s surrounding context and generating a natural-language explanation with a decoder RNN.The context includes characters, their attributes, the interaction, relationship, scene, situation, and optionally a topic.
  • Context Encoding: Character identities are treated as irrelevant, while age and gender attributes are retained because other attributes had little influence.Characters are represented by a weighted combination of attribute embeddings.
  • Context Encoding: The context combines encoded character, interaction, relationship, scene, situation, and optional topic information into a final vector.The model uses learned attention weights and separate doer and receiver parameters for interaction and relationship encoding.
  • Decoder RNN: A 100-dimensional GRU decoder receives the context and previous word at each time step, using temperature sampling to generate varied reasons.The decoder is trained end-to-end and evaluated on the test set.
  • Results: Qualitative evaluation includes Not relevant, Semi-relevant, and Very relevant examples, with some non-ground-truth predictions judged plausible.The paper notes that plausible alternatives illustrate the difficulty of predicting the exact annotated reason.

D. Person Detection, Clustering, and Identification

The person-processing pipeline detects and tracks faces, clusters tracks into character identities, and assigns cast identities, enabling graph characters to be localized in video.

  • Face Detection and Tracking: Face tracks are extracted from detections, linked across frames by IoU-based matching, and filtered to remove short tracks.The tracker allows gaps of up to five frames and discards tracks shorter than ten frames.
  • Face Features: Face features reduce VGG-Face representations from 4096 to 128 dimensions, normalize them, and train with triplet loss using a margin of 0.2.Training combines face tracks and single-face IMDb cast images.
  • Face Track Clustering: Within each movie scene, hierarchical agglomerative clustering groups face tracks using pairwise face-feature distances and a threshold of 0.75.The threshold was selected to minimize the operator clicks index on validation data.
  • Evaluation: Clustering is evaluated with cluster purity, weighted cluster purity, and operator clicks index, which estimates annotation cost from errors and cluster count.The metrics assess identity consistency and the effort required to correct clusters.

E. Annotation Interface

MovieGraphs annotations are collected through a web interface and a sequential workflow that combines labels, descriptions, graphs, and movie context for each coherent social clip.

  • Annotation Interface: For each clip, annotators provide a scene label, situation label, natural-language description, and graph through a custom web interface.The interface also supports clip selection and character selection from an IMDb-derived cast list.
  • Workflow: Annotators watch movies from start to finish and annotate clips in order so each graph can use context inferred from earlier events.The workflow supports understanding reasons behind characters’ actions.
  • Annotation Quality: Annotators are trained on shared clips and monitored to improve agreement about the level of graph detail.Uniformity is identified as a central challenge when using freelance annotators.
  • Clip Construction: Movies are automatically divided into scenes and manually grouped into clips representing coherent social situations.Clip lengths range from around 20 seconds to 2 minutes, averaging 44 seconds.

F. MovieGraphs Examples and Statistics

The MovieGraphs examples illustrate how annotated clips track story progression and provide structured statistics about social situations, relationships, attributes, and interaction structure.

  • Nine graphs sampled from “Jerry Maguire” illustrate how scenes, situations, and graph annotations progress through a movie.
  • The dataset reports distributions for the top 20 relationships and scenes, attribute types, and the numbers of characters, interactions, and relationships per clip.
  • Figure 11 shows example annotations across “Jerry Maguire,” including scenes, situations, graphs, and natural-language descriptions.
  • Figures 12–16 summarize dataset composition through relationship, scene, attribute, character-count, interaction-count, and relationship-count distributions.

G. More Examples

The examples show how MovieGraphs supports analysis of emotions across characters, interactions, relationships, and evolving situations. They also illustrate temporal situation transitions and the role of dialogue and visual cues in interaction annotation.

  • Emotional profiles: Emotional profiles distinguish the dominant emotions of characters across movies such as “Four Weddings and a Funeral” and “As Good as It Gets.”
  • Emotion timelines: Emotional timelines connect characters’ changing emotions with situations and relationships across movies.
  • Interaction emotions: Interaction roles correspond to different emotional distributions for the person acting and the person receiving the action.
  • Relationship emotions: Relationships likewise associate distinct emotional patterns with roles such as grandparent and grandchild, mistress and lover, or nanny and child.
  • Situation transitions: Possible situation transitions form sequences such as date, intimacy, talking about the past, and argument.
  • Interaction annotation: Most interaction examples rely mainly on dialogue, while at least one is inferred solely from visual cues.
Loading 1712.06761v2…