Source-linked AI summary
Thinking with Comics: Enhancing Multimodal Reasoning through Structured Visual Storytelling
Andong Chen, Wenxin Zhu, Qiuyu Ding, Yuchen Song, Muyun Yang, Tiejun Zhao
TL;DR
Static images lack temporal structure, while videos impose redundancy and high computational cost. The paper introduces Thinking with Comics, evaluates two comic-based reasoning paths across reasoning and context-understanding tasks, and reports systematic gains with lower cost than video.
Problem
Static images struggle with temporal structure and textual cues, whereas videos add substantial redundancy and computational overhead for multimodal reasoning.
Method
Thinking with Comics uses multi-panel comics as an intermediate representation through end-to-end visualized reasoning or conditioning context for a downstream VLM.
Results
Comics consistently yield performance gains across tasks; TwC achieves 85.8% accuracy on MathVista and 99.4% on DocVQA while outperforming Thinking with Video on MathVista.
Takeaways & Limitations
Comics provide an efficient intermediate visual medium that preserves essential temporal structure while supporting multimodal reasoning and context understanding.
Takeaways & Limitations
In Path 1, reasoning capability is constrained by the image-generation model because reasoning occurs implicitly within it.
Abstract
from arXiv · showhide
Chain-of-Thought reasoning has driven large language models to extend from thinking with text to thinking with images and videos. However, different modalities still have clear limitations: static images struggle to represent temporal structure, while videos introduce substantial redundancy and computational cost. In this work, we propose Thinking with Comics, a visual reasoning paradigm that uses comics as a high information-density medium positioned between images and videos. Comics preserve temporal structure, embedded text, and narrative coherence while requiring significantly lower reasoning cost. We systematically study two reasoning paths based on comics and evaluate them on a range of reasoning tasks and long-context understanding tasks. Experimental results show that Thinking with Comics outperforms Thinking with Images on multi-step temporal and causal reasoning tasks, while remaining substantially more efficient than Thinking with Video. Further analysis indicates that different comic narrative structures and styles consistently affect performance across tasks, suggesting that comics serve as an effective intermediate visual representation for improving multimodal reasoning.
1. Introduction
Thinking with Comics addresses the limitations of static images and videos by using structured comic panels that preserve temporal and textual information at lower reasoning cost. The paper evaluates two comic-based reasoning paths across reasoning and context-understanding tasks, finding systematic gains and task-dependent effects of narrative structure.
- Static images poorly represent temporal structure, whereas videos preserve temporal information at substantial redundancy and computational cost.
- Comics combine temporal logic, embedded text, dynamic reasoning, and higher information density than video panels.The paper positions comics between static images and videos as an intermediate visual reasoning medium.
- The study evaluates End-to-End Visualized Reasoning and Comic as Conditioning Context for VLM across reasoning and long-context understanding tasks.
- Comics consistently yield systematic performance gains across reasoning and context-understanding tasks compared with text, image, and video reasoning paradigms.
- Different narrative structures suit different tasks, with detective-style comics supporting logical reasoning and culture-centric narratives supporting cultural understanding.
- More difficult tasks require more comic panels, while disrupting panel order degrades performance and embedded text reduces semantic ambiguity.
- Thinking with Comics preserves essential temporal structure while requiring substantially lower inference cost than Thinking with Video.
2. Related Works
Related work extends explicit Chain-of-Thought reasoning from text to images and then video. Thinking with Comics builds on these paradigms by using sequential panels to address temporal and cross-modal limitations.
- Chain-of-Thought reasoning improves language-model reasoning by making intermediate reasoning steps explicit.
- Thinking with Images extends explicit reasoning to multimodal models through image inputs and generated intermediate visual representations.
- Thinking with Video addresses the limited temporal information and textual–visual independence of single-image reasoning with temporally unified video generation.
- Vision-generation research has been shaped by diffusion models, including Stable Diffusion for efficient high-resolution image generation.
3. Method
Thinking with Comics externalizes intermediate reasoning into temporally and causally structured panels. The panels either carry the reasoning process directly or condition a downstream multimodal language model.
- Method: Thinking with Comics represents intermediate reasoning as a sequence of comic panels with explicit temporal and causal structure.
- Path I: End-to-End Visualized Reasoning: Path 1 uses comic generation itself as end-to-end reasoning and extracts the answer from the final panel.
- Path II: Comic as Conditioning Context for VLM: Path 2 combines the generated comic with the original question and sends both to a multimodal language model for downstream inference.
- Path I: End-to-End Visualized Reasoning: Path 1 generates panels c1, c2, . . . , cT, with each panel depicting an intermediate reasoning step.
- Path I: End-to-End Visualized Reasoning: The generation process progressively unfolds latent reasoning states into visual panels, and answer extraction reads relevant information from the last panel.
- Path I: End-to-End Visualized Reasoning: Path 1 offers relatively low computational cost and interpretable trajectories, but its reasoning capability remains constrained by the image-generation model.
- Path II: Comic as Conditioning Context for VLM: The two paths differ in whether comic generation performs the reasoning itself or supplies structured context for a downstream VLM.
- Path II: Comic as Conditioning Context for VLM: In Path 2, the comic is an explicit intermediate variable that jointly encodes spatial structure, object relationships, and temporal evolution.
4. Experiments
The experiments evaluate Thinking with Comics across reasoning and long-context understanding benchmarks, using two implementation paths and comparisons with multiple reasoning paradigms. TwC performs strongly across multimodal reasoning, textual mathematics, document understanding, and cultural understanding tasks.
- Evaluation Datasets: Evaluation covers reasoning tasks and long-context understanding tasks, including mathematical, visually grounded, document-level, comic, and cultural benchmarks.Reasoning benchmarks are MATH500, GSM8K, and MathVista; context understanding benchmarks are DocVQA, eBDtheque, and CulturalBench.
- Implementation Paths: The experiments test two TwC paths: End-to-End Visualized Reasoning and Comic as Conditioning Context for MLLM.The first uses comics as the complete reasoning trajectory, while the second supplies comics with the question to an MLLM for joint reasoning.
- Baselines and Metrics: Comparisons include text-only MLLMs, reasoning-oriented LLMs, Thinking with Images methods, and Thinking with Video models under zero-shot, task-agnostic prompting.Accuracy is used for most benchmarks, including MATH500, GSM8K, MathVista, DocVQA, and CulturalBench.
- Main Results: 85.8% accuracy on MathVista shows strong multimodal reasoning performance, significantly outperforming Thinking with Video.TwC remains competitive with strong proprietary models on pure text-based mathematical reasoning benchmarks.
- Main Results: 99.4% accuracy on DocVQA and leading CulturalBench performance, especially on the hard subset, demonstrate broad context-understanding performance.The reported results cover both document-level understanding and contextualized cultural understanding.
5. Analysis Experiment
The analyses examine how narrative style, panel count, temporal order, embedded text, model configuration, and generation cost shape Thinking with Comics performance. Across these factors, structured comic representations support adaptive, temporally grounded, semantically anchored, and cost-efficient reasoning.
- Role-playing Narrative Alignment: A 28.5-point absolute gain raises average accuracy from 64.0 to 92.5 when detective-style comics replace documentary-style comics on MathVista and GSM8K.This is a 44.5% relative improvement over the documentary baseline and identifies detective narratives as effective visual prompts for logical reasoning.
- Scaling the Panels: Accuracy plateaus at 4–6 panels on MATH500 while token cost remains between 1100 and 1300, making this range the reported Pareto-optimal trade-off.Increasing panel counts beyond this range yields rapidly diminishing marginal gains.
- Panel Distribution Across Task Difficulties: TwC allocates one panel to 33.28% of GSM8K samples but uses four panels for 62.82%, while 30.41% of MathVista samples require six panels.The distributions indicate fewer panels for simpler queries and higher panel counts for more difficult visual reasoning.
- The Role of Temporal Sequence in Reasoning: Accuracy declines from 75.0% to 71.5% under temporal perturbations, with missing intermediate panels harming reasoning more than shuffled order.The controlled tests compare Complete Shuffle with Random Intermediate Deletion while preserving relative order among retained panels.
- Ablation on Textual Anchoring: Embedded text improves accuracy by 18.1 points on CulturalBench-Easy, 8.3 points on CulturalBench-Hard, and 13.2 points on MathVista over pure visual panels.Speech bubbles, narration, and symbols provide semantic anchoring that reduces visual ambiguity and cross-modal search complexity.
- Efficiency Analysis of TwC and Think with Video: A 10-second reasoning task costs $1.00 with video generation versus $0.134 with TwC, corresponding to an 86.6% reduction in media generation cost.The cost functions break even at approximately 1.34 seconds, beyond which video-based reasoning is more expensive under the stated prices.
6. Conclusion
The paper introduces Thinking with Comics as a multimodal reasoning paradigm using multi-panel comics as efficient intermediate representations for temporal and multi-step reasoning. It reports improved reasoning performance while avoiding video-generation overhead.
- Thinking with Comics uses multi-panel comics as an efficient intermediate representation for temporal and multi-step reasoning.
- TwC improves reasoning performance while avoiding the overhead associated with video generation.
Impact Statement
The paper positions Thinking with Comics between images and videos as an efficient multimodal reasoning paradigm. It argues that comics reduce redundancy and computational cost while preserving temporal and narrative structure, with deployment requiring attention to narrative style and cultural conventions.
- Thinking with Comics uses comics as an intermediate representation between images and videos.
- The approach reduces redundancy and computational cost while preserving temporal and narrative structure for long-context and temporal reasoning tasks.
- Future deployment should consider narrative style and cultural conventions to support robust and fair use across diverse settings.
A. Empirical Analysis: Why Comics Are a Privileged Visual Reasoning Medium
The empirical analysis finds that comics provide a stable, coherent scaffold for multi-panel visual reasoning. Global comic generation is more structurally coherent than incremental generation, while the representation formalizes task sufficiency and information-efficiency alongside generation cost.
- A.1. Prompt-Induced Structural Stability in Multi-Panel Visual Generation: The evaluation samples 20 instances each from MATH-500 and MathVista and measures panel-generation success rate and answer accuracy.
- A.1. Prompt-Induced Structural Stability in Multi-Panel Visual Generation: Comic prompts consistently produce structurally complete multi-panel layouts, whereas Non-Comic prompts more often exhibit layout collapse or merged reasoning steps.
- A.2. Global Comic versus Incremental Generation: Global Comic generation produces stronger cross-panel coherence, more stable entity representations, and smoother reasoning progression than incremental generation.
- A.2. Global Comic versus Incremental Generation: A qualitative divisor-of-196 example shows Global Comic maintaining a consistent FactoBot character and smooth logical flow, unlike the incremental non-comic baseline.
- B.1. Representation and Utility: Path II predicts the final answer by combining the question q with an intermediate representation z generated by a visual generator Gθ.
- B.1. Representation and Utility: Information-efficiency evaluates task-relevant information in z relative to its media-generation cost, with comics modeled at fixed per-image cost and video at linear per-second cost.
B.2. Comics Outperform Single Images Due to Temporal Structure and Textual Anchoring
The paper argues that single images can omit answer-relevant temporal states, whereas comics combine selected panels with embedded text to preserve more task-relevant information. Comics also avoid video redundancy and are argued to benefit from smaller domain shift than synthetic sequential images.
- A single image may discard relevant latent states when answers depend on multi-step temporal or causal relations.
- Comics combine selected intermediate panels with embedded bubbles or narration, adding a textual anchoring channel beyond pure visual information.
- Adding bubbles or narration improves robustness and accuracy in the reported ablation.
- Video information grows sublinearly with frame count under temporal redundancy, while video cost grows linearly, reducing efficiency as redundant frames accumulate.
- Comics select K ≪T key states to retain task-relevant information with fewer visual tokens, with accuracy saturating around K ∈[4, 6].
- The analysis attributes comics’ higher generation fidelity to a smaller distribution shift from the generator’s training distribution than ad-hoc synthetic sequential images.
C.1. Path I: End-to-End Comic Reasoning.
Path I uses generated comics to depict end-to-end reasoning, with the final answer visually embedded and extracted from the comic. The supplied prompts specify comic-based reasoning for mathematical, cultural, document, and visual question-answering tasks.
- Path I: End-to-End Comic Reasoning: Path I generates a multi-panel comic as the complete reasoning result, with the final answer typically shown in the last panel.An external answer reader identifies the answer from the generated panels and the original question.
- Path I: End-to-End Comic Reasoning: The prompts require comics to illustrate step-by-step reasoning, highlight relevant document information, or depict authentic cultural contexts leading to an answer.Task-specific prompts cover math, visual reasoning, document question answering, and cultural knowledge or truthfulness judgments.
- Path I: End-to-End Comic Reasoning: The analysis prompts vary comic styles and narrative constraints, including role-playing, slice-of-life, documentary, realistic photo-style, and text-free visual storytelling.These conditions are used across structural-coherence and textual-anchoring experiments.
- Path I: End-to-End Comic Reasoning: The structural-coherence prompt requires a fixed number of coherent panels that introduce the problem, progressively show reasoning, and present the final solution.It also requests consistent elements, mathematical notation, smooth transitions, and a single cohesive comic image.
E.2. Evaluation for external answer reader
The external answer reader is evaluated through manual checks and qualitative examples spanning comic styles and multiple benchmark figures. The examples distinguish documentary, role-playing, and slice-of-life visualizations for problem solving.
- Evaluation for external answer reader: A manual cross-validation study samples 20% of evaluation instances and has three independent annotators identify the final answer from each generated comic.The annotations are compared across annotators to assess answer-reading reliability.
- Evaluation for external answer reader: The qualitative section compares comic styles before presenting examples from reasoning and long-context understanding tasks across five benchmarks.The supplied figure labels include MATH500, MathVista, and GSM8K.
- Evaluation for external answer reader: Documentary style presents realistic problem context and relevant information, whereas role-playing and slice-of-life styles narrate reasoning through roles or everyday scenarios.The styles differ in how the reasoning process is visually situated.
F.3. (Long) Context Understanding Tasks
The supplied examples for long-context understanding include cultural question prompts and figure references for CulturalBench and DocVQA. They cover cultural practices, truthfulness judgments, and document-based visual question answering.
- (Long) Context Understanding Tasks: The cultural examples ask about greetings in Morocco, cultural characteristics in Nigeria, and food-consumption tools in India.The India example presents multiple candidate utensil combinations for evaluation.
- (Long) Context Understanding Tasks: The India question organizes candidate answers as combinations of four listed tools or practices.The options enumerate combinations such as i, iii and iii, iv.