Source-linked AI summary
Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, Sanja Fidler
TL;DR
The paper addresses how to ground books’ fine-grained visual and high-level semantic descriptions in movie content. It aligns movie shots and dialogue with book sentences using neural text and video embeddings plus contextual modeling, and reports good quantitative alignment performance with applications including retrieval and story-like captioning. The authors note that alignment remains challenging because of data scale, writing variation, and differences between films and their source books.
Problem
Books provide rich visual and high-level semantic descriptions, but lack associated visual information for grounding those descriptions beyond simplistic captions.
Method
The paper aligns movie shots and dialogue with book sentences using book-trained sentence embeddings, video-text embeddings, and a context-aware CNN alignment model.
Results
The model demonstrates good quantitative movie/book alignment performance, retrieves the correct book for each test movie, and supports diverse qualitative applications.
Takeaways & Limitations
The alignment connects movies and books for cross-modal browsing, book retrieval, and story-like image captioning.
Takeaways & Limitations
Alignment is challenging because of large data scale, varied writing styles, difficult visual matches, and differences in how closely movies follow their books.
Abstract
from arXiv · showhide
Books are a rich source of both fine-grained information, how a character, an object or a scene looks like, as well as high-level semantics, what someone is thinking, feeling and how these states evolve through a story. This paper aims to align books to their movie releases in order to provide rich descriptive explanations for visual content that go semantically far beyond the captions available in current datasets. To align movies and books we exploit a neural sentence embedding that is trained in an unsupervised way from a large corpus of books, as well as a video-text neural embedding for computing similarities between movie clips and sentences in the book. We propose a context-aware CNN to combine information from multiple sources. We demonstrate good quantitative performance for movie/book alignment and show several qualitative examples that showcase the diversity of tasks our model can be used for.
1. Introduction
The paper motivates aligning books with their movie releases because books provide rich visual and mental-state descriptions while movies capture visual settings. It introduces multimodal, context-aware alignment to connect movie shots and dialogue with book sentences for story-like visual explanations and related applications.
- Books combine fine-grained visual details with high-level descriptions of characters’ thoughts, feelings, and evolving states.
- Movies and books are complementary: books describe character intentions and mental states, while movies capture visual aspects of settings.
- The paper formulates movie/book alignment as matching movie shots and subtitle sentences to corresponding book sentences.
- The approach combines book-trained sentence embeddings, video-text embeddings for movie clips, and contextual information in a final alignment model.The model uses a context-aware CNN and an energy-minimization formulation that encourages similar timelines.
- The alignment supports browsing between movies and books, book retrieval, and story-like captioning of images.
2. Related Work
Prior work addressed vision-language captioning, alignment, and movie-to-text matching, but this paper targets finer-grained movie/book alignment. It distinguishes its approach through neural embeddings and sentence- or paragraph-level correspondences.
- Vision-language research has studied captioning, alignment, question answering, textual visual-model learning, and semantic visual search.
- Earlier movie-to-text systems aligned movies with scripts using subtitles or visual and audio features, supporting weak labels for person naming.
- Unlike synopsis-to-shot alignment, this work handles verbose books that may diverge from their movie adaptations and uses learned neural embeddings instead of hand-designed similarities.
- Compared with chapter-level scene alignment, this work operates at the sentence or paragraph level and uses a dataset with 2,070 shot-to-sentence alignments.
- The Movie Description dataset provides short clips with DVS sentences, whereas this effort seeks longer, richer, higher-level video descriptions from books.
3. The MovieBook and BookCorpus Datasets
The paper introduces MovieBook and BookCorpus datasets for aligning movies with their source books and training sentence embeddings. The data span substantial variation in scale, writing style, descriptiveness, and alignment difficulty.
- The MovieBook Dataset contains 11 movies, their source books, timestamped subtitles, and book sentences, paragraphs, and chapters.
- Annotators labeled 2,070 visual, dialogue, or audio correspondences across 11 movie/book pairs after 90 hours of work.
- The dataset varies widely in book length, with 638 to 15,498 sentences per book despite similar movie durations.
- Movie/book alignment is difficult because movies average 1,800 shots and books average 7,750 sentences, while visual descriptions may be brief, verbose, or obscured in surrounding text.
- BookCorpus contains 11,038 web books longer than 20K words across 16 genres for training the sentence similarity model.
4. Aligning Books and Movies
The alignment approach combines sentence-level book representations, visual-semantic movie-clip representations, and contextual similarity modeling to match movie content with book text.
- Alignment framework: The system aligns movie shots and subtitle sentences with corresponding book sentences using visual and dialog information.It computes similarities between shots or subtitle sentences and book sentences.
- Sentence embedding: The sentence embedding encodes a sentence and reconstructs its preceding and following sentences from contiguous book text.This unsupervised objective produces representations intended to capture syntactic and semantic similarity.
- Sentence embedding: The embedding is trained on millions of sentences from books, and nearest-neighbor examples from held-out books are largely syntactically and semantically similar.The learning signal depends on contiguous sentences, making books a natural training corpus.
- Sentence embedding: GRU-based recurrent encoder-decoder networks represent sentences while using update and reset gates to control information flow.The final encoder hidden state serves as the whole-sentence representation.
- Visual-semantic embedding: The visual-semantic model learns an embedding between movie clips and descriptive video-service descriptions using mean-pooled frame features.Training uses 94 movies and 54,000 described clips, with GoogLeNet and hybrid-CNN frame features.
- Visual-semantic embedding: Clip and sentence representations are scored by the inner product of unit-normalized vectors, equivalent to cosine similarity, and trained with a pairwise ranking loss.The clip vector is linearly mapped into the visual-semantic embedding space before scoring.
5. Experimental Evaluation
The evaluation measures movie/book alignment on 11 pairs using recall, precision, and average precision, then examines model components, qualitative matches, book retrieval, and story-like descriptions.
- Experimental Setup: The model is trained on Gone Girl and tested on the remaining 10 movies from an 11-pair movie/book dataset.The video-text and sentence models process 1,440 movies and 870 books per day, respectively.
- Evaluation Protocol: Recall is the primary alignment metric because the ground-truth correspondences are incomplete; matches within 3 book paragraphs and 5 subtitle sentences count as recalled.Average precision is also reported from recall and precision measured across multiple alignment thresholds.
- Quantitative Results: Adding CNN layers produces the largest improvement, increasing recall by 14% and doubling AP; the BOOK embedding adds 4% recall, video-text embedding adds 2%, and CRF adds 2%.The comparisons use feature ablations, different CNN depths, and the full CRF model.
- Quantitative Results: The contextual CNN outperforms the linear SVM baseline by 30% in recall and doubles AP.The SVM combines similarity measures without the CNN’s contextual modeling, while both are used within the CRF alignment framework.
- Qualitative Results: Qualitative alignments retrieve semantically meaningful book passages despite substantial movie-book dialogue differences and verbose textual descriptions.The visualizations show the matched paragraph together with one preceding and one following paragraph.
- Book Retrieval: The model retrieves the correct book for every movie when ranking each movie against all 10 test books by CRF alignment similarity.Similarities are scaled relative to the highest-scoring book, which is assigned 100.
6. Conclusion
The paper introduces movie-book alignment as a new problem and combines multiple similarity signals with contextual modeling to address it. It reports results on a new alignment dataset and quantitative evidence of the approach’s potential.
- The paper explores aligning books with their movie releases as a new research problem.
- The approach computes similarities between movie shots, dialogs, and book sentences using learned sentence and video-text representations.
- A context-aware alignment model combines available similarity information to infer correspondences.
Appendix
The appendix contains additional qualitative results that extend the paper’s empirical illustrations.
- The appendix provides more qualitative results.
- These additional results supplement the paper’s main evaluations with qualitative evidence.
- The appendix extends the presentation beyond the results shown in the main paper.
A. Qualitative Movie-Book Alignment Results
Qualitative examples show the model aligning movie shots with nearby book paragraphs and using dialog to support visual grounding. The examples also illustrate semantic relevance across image-text and movie-book settings.
- Movie-Book Alignment: For selected movie shots, the model retrieves a corresponding book paragraph together with its immediate textual context.The examples present a central shot frame and overlapping subtitle sentences alongside the inferred book passage.
- Image-Text Retrieval: A CoCo image caption can retrieve a semantically relevant passage from a large book corpus, with surrounding sentences included for context.The retrieved passage consists of the best-matched sentence plus two sentences before and after it.
- Movie-Book Alignment: Predicted alignments are visualized against ground-truth visual and dialog matches, distinguishing predicted links from annotated correspondences.The figure uses blue for visual matches, magenta for dialog matches, and yellow for predicted alignments.
- Movie-Book Alignment: Movie dialogs that closely follow the book help align and ground visual information when visual evidence is weaker.Text surrounding the dialog describes the scene, connecting the subtitles to book descriptions and video content.
B. Borrowing “Lines” from Other Books
The borrowing experiments test whether movie shots can retrieve plausible paragraphs from books unrelated to the movie. Increasing the candidate corpus from 10 to 200 books produces increasingly relevant stories.
- 10-Book Experiment: The 10-book experiment matches movie clips with paragraphs from books other than the corresponding movie’s book.It excludes the training movie and the corresponding book, leaving nine alternative books for matching.
- 10-Book Experiment: In the small candidate set, top-scoring matches may be relatively low-similarity or not very meaningful.The experiment uses contextual-CNN similarity without enforcing a global alignment over the full book.
- 200-Book Experiment: With 200 randomly selected BookCorpus books, the model retrieves increasingly relevant stories from the larger candidate pool.The qualitative results include a failed example despite the overall improvement in relevance.
C. The CoCoBook
This section presents additional CoCo image-captioning results that use passages from books. The examples span visual descriptions, actions, settings, and characters’ emotional states.
- The section reports additional results for captioning CoCo images with passages from books.
- Several passages convey characters’ internal states or reactions, including fear, anger, romantic anticipation, and distress.
- The examples also describe dynamic actions and events, including shooting, flying, splashing, and preparing food.
- Book passages provide fine-grained visual descriptions of characters, clothing, objects, and spaces.