Source-linked AI summary
The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives
Mohit Iyyer, Varun Manjunatha, Anupam Guha, Yogarshi Vyas, Jordan Boyd-Graber, Hal Daumé III, Larry Davis
TL;DR
Comics require readers to infer unseen actions between panels and combine artwork with dialogue, a form of narrative understanding that remains difficult to test in computers. This paper builds COMICS and introduces three context-based cloze tasks; models underperform human baselines, and the analysis finds that words and images work together in most panels.
Problem
The paper asks whether computers can understand comic narratives that depend on inferring unseen actions across panels and connecting dialogue with artwork.
Source: §1 · pp. 1–2Method
The authors construct COMICS and introduce three cloze-style tasks that test narrative and character predictions from preceding panels.
Source: §2 · p. 2, §1 · p. 2, §5.1 · p. 6Results
Takeaways & Limitations
The findings confirm the importance of combining text and image and show that context is crucial for predicting narrative or character-centric panel aspects.
Source: §8 · p. 8Models need to become better at leveraging context and commonsense knowledge to approach human performance.
Source: §8 · p. 8
Abstract
from arXiv · showhide
Visual narrative is often a combination of explicit information and judicious omissions, relying on the viewer to supply missing details. In comics, most movements in time and space are hidden in the "gutters" between panels. To follow the story, readers logically connect panels together by inferring unseen actions through a process called "closure". While computers can now describe what is explicitly depicted in natural images, in this paper we examine whether they can understand the closure-driven narratives conveyed by stylized artwork and dialogue in comic book panels. We construct a dataset, COMICS, that consists of over 1.2 million panels (120 GB) paired with automatic textbox transcriptions. An in-depth analysis of COMICS demonstrates that neither text nor image alone can tell a comic book story, so a computer must understand both modalities to keep up with the plot. We introduce three cloze-style tasks that ask models to predict narrative and character-centric aspects of a panel given n preceding panels as context. Various deep neural architectures underperform human baselines on these tasks, suggesting that COMICS contains fundamental challenges for both vision and language.
1. Introduction
Comics tell stories through omissions between panels, requiring readers to infer unseen actions and connect text with artwork. This paper tests whether computers can perform that closure by building COMICS and evaluating models on closure-dependent tasks.
1. Introduction
- Closure combines understanding individual panels with connective inferences across panels to form coherent storylines.The gaps between panels can conceal actions and conversations that readers infer from dialogue, narration, and artwork.Source: §1 · p. 1
- COMICS contains about 1.2 million panels from almost 4,000 Golden Age comic books, whose artistic styles vary widely.Cartoonish lions in humorous comics contrast with more photorealistic lions in adventure comics.Source: §1 · pp. 1–2, Figure 2 · p. 2
- Comics weave speech balloons, thought clouds, and narration boxes with artwork, so neither text nor image alone often tells the whole story.Textboxes are identified and transcribed with OCR; dialogue order and visual context both matter to understanding scenes.Source: §1 · p. 2
- All evaluated neural architectures perform significantly worse than humans on three cloze-style tasks designed to test narrative and character understanding.The tasks require models to predict narrative or character-centric aspects using preceding panels as context.Source: §1 · p. 2
2. Creating a dataset of comic books
COMICS addresses the limited size and visual diversity of existing comics datasets by collecting public-domain Golden Age comics and extracting panels, textboxes, and OCR transcriptions. Its construction pipeline uses learned detection models because older scans often violate assumptions made by existing segmentation methods.
2. Creating a dataset of comic books
- Existing comics datasets are too small for data-hungry narrative models and lack diversity in visual style and genres.COMICS is built by downloading public-domain comics, segmenting pages into panels, extracting textbox locations, and transcribing text with OCR.Source: §2 · p. 2
2.1. Where do our comics come from?
- The dataset draws on the 4,000 highest-rated Digital Comics Museum books, whose scans vary in resolution and quality.Golden Age comics include self-contained stories and genres such as crime, romance, humor, and horror.Source: §2.1 · p. 2
2.2. Breaking comics into their basic elements
- Because existing textbox detectors also failed, the authors train Faster R-CNN on 1,500 annotated panels and order detected textboxes with Morton order.The pipeline extracts textbox locations and contents because both matter for character and narrative understanding.Source: §2.2 · pp. 2–3
- Non-standard layouts, dirty gutters, and textboxes spanning panels made existing panel segmentation methods unsuitable for these scans.The authors annotate 500 pages to train Faster R-CNN for panel detection, then order panels using Morton order.Source: §2.2 · pp. 2–3
2.3. OCR
- Google Cloud Vision OCR outperformed the other tested systems on comic text, despite occasional difficulty with short words and punctuation.The authors post-process systematic spelling errors and remove full-page advertisements using a classifier.Source: §2.3 · p. 3
3. Data Analysis
An analysis of annotated panel pairs finds that comics usually depend on both words and images, while most transitions require readers to infer changes across panels. Transition categories are diverse and harder to annotate consistently than intrapanel relationships.
3. Data Analysis
- Understanding most panel transitions requires complex inferences, often involving common sense, changes in time or space, or newly introduced characters.The authors conclude that narrative-flow models must tie multimodal inputs together through closure.Source: §3 · p. 3
- 92.1% of annotated panels are interdependent, with words and pictures conveying ideas neither could convey alone.Word-specific panels account for 4.4%, picture-specific for 2.8%, and parallel for 0.6%.Source: §3 · p. 3
- 34.6% of transitions are action-to-action, 32.7% subject-to-subject, 17.7% continued conversation, 13.8% scene-to-scene, and 0.4% moment-to-moment.Moment-to-moment transitions require the least closure because little time or space changes between panels.Source: §3 · pp. 3–4, Figure 3 · p. 4
- Annotators agreed on 96% of intrapanel labels (Cohen’s κ = 0.657), but only 68% of interpanel labels (Cohen’s κ = 0.605).Interpanel labels are harder to agree on because transitions are diverse and multiple types can occur in one pair.Source: §3 · p. 4
4. Tasks that test closure
Three cloze-style tasks test whether models can use preceding panels to predict text, images, or character dialogue order. Their candidate designs aim to assess closure rather than text-generation quality or easy stylistic shortcuts.
4. Tasks that test closure
- The three tasks test narrative and character understanding by asking models to predict aspects of a panel from preceding panels.Models rank candidates in a cloze framework, which avoids evaluating the quality of generated dialogue.Source: §4 · p. 4
- Text cloze selects a textbox using context panels and the current panel image, with textbox regions blacked out to prevent direct OCR-like matching.The current panel is restricted to one textbox so that most predictive information comes from earlier panels.Source: §4 · pp. 4–5, Figure 4 · p. 5
- Visual cloze selects the next panel image without giving the model text from that final panel.Character coherence reorders two dialogue boxes, requiring matching dialogue to textboxes and potentially inferring speakers from prior utterances and appearance.Source: §4 · p. 5, Figure 4 · p. 5
4.1. Task Difficulty
- Hard-setting candidates come from nearby pages, reducing stylistic clues and requiring greater reliance on context than random candidates in the easy setting.Hard text candidates tend to share character names and entities, while visual candidates become less distinguishable by color and texture.Source: §4.1 · p. 5
5. Models & Experiments
The experiments compare four neural architectures for predicting comic-panel content from preceding panels, with the strongest model using a hierarchical LSTM. Models benefit from multimodal input on text cloze, but all remain behind human performance.
5. Models & Experiments
- Four neural models—text-only, image-only, and two image-text variants—are evaluated, with the best-performing model encoding panels using a hierarchical LSTM.The image-text architecture combines pretrained image features with learned text features to form a context representation that scores candidates.Source: §5 · p. 5, Figure 5 · p. 6
- On text cloze, accuracy increases with pretrained VGG-16 image features alongside text, but adding both modalities is less important on the other tasks, and no architecture beats humans.The full-results caption also reports that preceding-panel context helps across tasks; the best models still lag well behind humans.Source: §5 · p. 6, Table 2 · p. 7
5.1. Model definitions
- Models predict a target panel from preceding panels, using image, OCR text, and textbox coordinates; candidate scores are normalized with softmax and trained with cross-entropy loss.The experiments use three context panels and three candidates, and test and validation comics are unseen during training.Source: §5.1 · p. 6
- The text-only baseline encodes textboxes and panels with hierarchical LSTMs, while the image-only baseline feeds pretrained image features through an LSTM.For text cloze and character coherence, an image-text variant without context panels scores candidates using image features from the target panel.Source: §5.1 · pp. 6–7
- Replacing softmax with contrastive max-margin losses slightly lowers development-set performance, and more sophisticated text encoders show no noticeable improvement.Source: §5.1 · p. 6
6. Error Analysis
Results vary by task: images help text cloze, text contributes less to visual cloze, and preceding-panel context helps across tasks. Error examples show that models can be misled by artwork or miss contextual cues that humans use.
6. Error Analysis
- Image-text models dominate single-modality models on text cloze, while text helps visual cloze less; preceding-panel context helps across tasks, with smaller gains in the hard setting.Differences among models on character coherence are minor.Source: §6 · p. 7
- Humans comfortably outperform all models; their errors mostly reflect poor OCR, such as misspellings, or low image resolution.Source: §6 · p. 7
- In an easy text-cloze example, artwork reveals an airplane absent from the dialogue, allowing the image-text model to reject both incorrect candidates.The text-only model misses the airplane because the context dialogue does not mention it.Source: §6 · p. 7
- In hard text cloze, strong visual associations can override context: dialect and an out-of-vocabulary token obscure the correct choice, while a fight image draws the model toward a misleading candidate.The authors suggest that external world knowledge may help assess whether an utterance fits a situation.Source: §6 · p. 7
7. Related Work
Related work spans multimodal language-and-vision tasks, non-natural imagery, and language-based narrative modeling. COMICS extends this work by computationally modeling comic-book content and connecting panel sequences.
7. Related Work
- Related multimodal work includes image captioning, visual question answering, and matching television characters with visual attributes or speech patterns.These efforts use datasets such as MS COCO and Visual Genome.Source: §7 · p. 7
- Photo-based visual storytelling generates or reorders stories, but its images lack COMICS’s coherent characters and accompanying dialogue.COMICS offers professionally created, character-centric stories driven by dialogue, though OCR errors make its text less suited to generation.Source: §7 · pp. 7–8
- Prior non-natural-image work studied clip-art scene properties, painting recognition and question answering, and comic-element extraction rather than comic content.The paper also notes substantial manga segmentation research.Source: §7 · p. 8
- COMICS is presented as the first effort to computationally model comic-book content rather than only extract its elements.The paper relates this work to the Story Cloze Test and to research on characters and narrative structure in novels.Source: §7 · p. 8
8. Conclusion & Future Work
The paper introduces COMICS and cloze tasks for studying closure in Golden Age comics. Experiments and manual analysis support multimodal and contextual understanding, while future progress toward human performance requires better context use.
8. Conclusion & Future Work
- COMICS contains over 1.2 million panels from Golden Age comic books, and three cloze-style tasks probe how readers connect panels into coherent stories.The paper uses these tasks to explore closure.Source: §8 · p. 8
- Experiments and manual analysis confirm the importance of combining text and image, while context is crucial for predicting narrative or character-centric panel aspects.The findings concern the paper’s COMICS tasks.Source: §8 · p. 8
- Reaching human performance will require better context use; proposed directions include commonsense knowledge, dialogue-to-artwork generation, and transfer learning across natural and non-natural images.The paper poses these as future research directions.Source: §8 · p. 8