PaperlayerView exact revision on arXiv

Source-linked AI summary

ManGo: Manga Active Narrative Grounding Optimization

Hao Qiu, Junyan Wang, Zheyuan Liu, Lei Fan, Hong Jia, Lianbo Guo, Zhulin Tao

arXiv:2608.29865v1cs.CL

TL;DR

Manga VQA requires active reasoning over distributed evidence across ordered panels, text, characters, and event transitions. ManGo builds question-directed evidence sketches through Active Narrative Sketching and trains this behavior from rollout-derived rewards without human answer or rationale annotations. Across manga understanding benchmarks, it reports improved answer accuracy and evidence grounding, while evaluation remains limited by benchmark diversity.

  • Problem

    Manga VQA requires linking distributed, ordered, multimodal evidence, making passive whole-page encoding insufficient for deciding what to inspect, retain, and when to stop.

  • Method

    ManGo uses Active Narrative Sketching to select panels, extract concise grounded clues, and stop when sufficient evidence is assembled, then applies group-relative training with answer preference and path consistency rewards.

  • Results

    ManGo achieves state-of-the-art performance across open-ended and multiple-choice settings and improves both final answer accuracy and evidence grounding.

  • Takeaways & Limitations

    Active panel-level evidence seeking provides ManGo with a compact, question-directed narrative representation for manga question answering.

  • Takeaways & Limitations

    Evaluation is constrained by limited benchmarks covering only part of manga comprehension across genre, era, language, and cultural context.

Abstract

from arXiv · show

Manga visual question answering requires models to answer questions over panel-based visual narratives, where relevant evidence is distributed across ordered panels, embedded text, recurring characters, and implicit event transitions. This structure makes passive page encoding insufficient, as the model must identify which panels to inspect, what clues to retain, and when the accumulated evidence is sufficient for answering. We propose ManGo (Manga Active Narrative Grounding Optimization), an unsupervised framework for active manga visual question answering. ManGo introduces Active Narrative Sketching (ANS), which iteratively selects panels, extracts concise grounded clues, and decides when to stop, forming a compact question-directed evidence sketch before answer generation. To optimize this behavior without human-annotated answers or rationale paths, ManGo samples multiple ANS rollouts and applies group-relative training with two rewards: answer preference from listwise self-ranking and path consistency from stable ordered panel trajectories. The combined reward is optimized with group-relative policy training, encouraging the model to improve both final answers and the panel-level evidence paths that support them. Experiments on standard manga understanding benchmarks show that ManGo achieves state-of-the-art performance across different settings.

1 Introduction

Manga VQA requires question-dependent reasoning over distributed, ordered visual and textual evidence rather than isolated cues. MANGO addresses this with active panel inspection, compact narrative sketching, and unsupervised group-relative training.

  • Motivation: Manga VQA distributes relevant evidence across ordered panels, embedded text, recurring characters, gestures, and implicit event transitions.Answering may require linking dialogue to characters and reconstructing temporal or causal relations across the page.
  • Motivation: Active evidence-seeking selects which panel to inspect, which clue to retain, and when accumulated evidence is sufficient for answering.This replaces passive whole-page encoding with progressively discovered and connected question-relevant evidence.
  • MANGO: MANGO proposes Active Narrative Sketching, which selects panels, extracts concise grounded clues, and decides whether to continue or answer.The resulting sketch follows evolving evidence needs rather than page order alone and may revisit panels for different clues.
  • Training: Process-level supervision is difficult to scale because active sketching requires panel, clue, relevance, and stopping decisions beyond final-answer labels.MANGO therefore uses questions and panel annotations without human-annotated answers or rationale paths.
  • Training: MANGO derives answer preference and path consistency rewards from multiple sampled rollouts and optimizes them with group-relative policy training.The supplied passage introduces answer-level ranking and complementary path-based training signals.
  • Results: MANGO achieves state-of-the-art performance across open-ended and multiple-choice manga understanding settings.The evaluation uses standard manga understanding benchmarks.

2 Background and Preliminaries

The background frames manga VQA as active, task-conditioned perception over panels and motivates group-relative optimization from multiple sampled trajectories. These preliminaries replace fixed observation encoding and separate value modeling with question-guided evidence updates and reward comparisons.

  • Active Perception: Active perception acquires task-relevant information through goal- and observation-dependent decisions instead of passively encoding a fixed observation.The agent determines what, where, how, or when to sense according to its current goal and observations.
  • Active Perception: In manga VQA, each perceptual action selects one panel and updates a compact evidence sketch.The sketch functions as accumulated perception history for question-guided panel inspection.
  • Group-Relative Policy Optimization: GRPO estimates advantages by comparing multiple responses sampled for the same prompt, avoiding a separately trained value model.It is presented as a critic-free variant of PPO.
  • Group-Relative Policy Optimization: GRPO normalizes each response reward using the mean and standard deviation of rewards within its sampled group.The reward for response Y_i is compared against the group statistics.
  • Group-Relative Policy Optimization: The policy uses a PPO-style clipped objective over generated tokens, with token-level probability ratios and their clipped versions.In this setting, rewards come from sampled answers and panel-level evidence paths rather than annotated answers or rationales.

3 MANGO

MANGO combines Active Narrative Sketching with group-relative training to perform unsupervised manga VQA. ANS builds a compact, question-directed panel evidence path, while rewards jointly assess answer preference and ordered path consistency.

  • 3 MANGO: MANGO is an unsupervised manga VQA framework combining Active Narrative Sketching with group-relative policy optimization.Its policy generates a structured sketch and final answer from a manga page, panel boxes, and question.
  • 3.1 Active Narrative Sketching: ANS iteratively selects panels, extracts concise grounded clues, and decides whether to continue or answer.The accumulated sketch is passed to answer generation after the stopping action.
  • 3.1 Active Narrative Sketching: The narrative evidence sketch stores ordered panel-clue pairs whose clues are local, atomic, visible, and question-relevant.Its evidence order may differ from page reading order and can include revisiting panels to verify relations.
  • 3.2 Unsupervised Reward Modeling: Training samples multiple ANS rollouts per question and constructs answer preference and path consistency rewards without human-annotated answers or rationale paths.Answer preference uses listwise self-ranking of clustered answers, while path consistency evaluates stable evidence paths among sampled rollouts.
  • 3.2 Unsupervised Reward Modeling: Path consistency is order-sensitive: it scores prefixes shared with high-preference reference trajectories, distinguishing identical panel sets arranged in different orders.This matches manga VQA’s dependence on ordered panels and cross-panel temporal or causal relations.
  • 3.3 Group-Relative Training: The combined reward includes answer quality, path consistency, and format validity, then group-relative policy optimization converts rollout rewards into training advantages.Clipping is applied to the token-level policy ratio, and self-ranking samples are used only for reward construction.

4 Experiments

MANGO is evaluated across three manga benchmarks and consistently performs strongly across open-ended, multiple-choice, single-panel, and multi-panel settings. Ablations and analyses attribute its behavior to active sketching, answer preference, path consistency, compact responses, and answer-relevant panel grounding.

  • Experimental Settings: The evaluation uses MangaVQA, ChrOMIC, and MangaUB, covering open-ended, multiple-choice, single-panel, and multi-panel manga understanding.MangaVQA includes Exact Extraction and Descriptive Answering; ChrOMIC includes Multiple-choice and Open-ended settings; MangaUB includes Single-panel and Multi-panel questions.
  • Main Results: MANGO achieves the strongest and most consistent performance across the evaluated open-ended, multiple-choice, single-panel, and multi-panel settings.The comparison includes general perception and active perception baselines.
  • Analysis: MANGO maintains shorter response distributions while concentrating more samples in high score bins than other active perception methods.The response analysis compares token-length distributions within score bins.
  • Main Results: MANGO remains effective on a Qwen2.5-VL checkpoint fine-tuned on MangaOCR data, indicating robustness across backbones and initializations.The checkpoint is evaluated under the same SFT protocol as MangaLMM.
  • Ablation Study: The full model performs best, while ANS, answer preference reward, and path consistency each contribute to performance gains.Answer preference particularly improves open-ended evaluation, and path consistency adds gains across settings.
  • Ablation Study: Self-ranking outperforms clustering- or confidence-based preference rewards and remains more effective than POLAR despite POLAR using ground-truth answers.Self-ranking compares candidate answers from multiple rollouts under the same page and question.
  • Ablation Study: Prefix Tree performs best among consistency estimators because stable ordered prefixes better capture how early panel choices guide later evidence construction.Jaccard matching ignores panel order, whereas sequence-based metrics account for ordered panel paths.
  • Analysis: MANGO benefits more from retaining human-annotated answer-supporting panels and drops more when those panels are masked than with size-matched random panels.The perturbation analysis supports tighter grounding in answer-relevant panels and stable ordered evidence trajectories.

5 Related Work

Related work places manga VQA within a broader shift toward reasoning over structured visual information while emphasizing manga-specific challenges such as ordering, speakers, and multi-panel understanding.

  • Visual Question Answering: VQA research is expanding from object- and attribute-centric questions toward text-rich, multilingual, scientific, ambiguous, and otherwise structured visual contexts.Manga VQA is presented as a less explored specialized case of this broader trend.
  • Manga VQA: Manga and comic benchmarks study reading order, panel sequencing, speaker identification, and multi-panel understanding, with MangaVQA and MangaLMM providing manga-specific resources.These benchmarks and models target abilities that differ from conventional natural-image VQA.

6 Conclusion

MANGO frames manga visual question answering as active panel-level evidence seeking and introduces Active Narrative Sketching with unsupervised group-relative training. Across multiple manga understanding benchmarks, it improves final answer accuracy and evidence grounding.

  • MANGO is an unsupervised framework that answers manga questions by actively sketching question-relevant evidence across panels.Its central components are Active Narrative Sketching and group-relative policy optimization.
  • Active Narrative Sketching selects relevant panels, extracts concise grounded clues, and builds a compact evidence sketch before answering.
  • Answer preference and path consistency rewards are optimized with group-relative training without human-annotated answers or rationale paths.
  • Experiments on multiple manga understanding benchmarks show improvements in both final answer accuracy and evidence grounding.

Limitations

MANGO’s evaluation is constrained by the limited availability and diversity of large-scale manga VQA resources. Existing benchmarks cover only part of manga comprehension across genre, era, language, and cultural context.

  • Limited large-scale manga VQA resources constrain evaluation of MANGO’s performance.
  • Current benchmarks may not fully characterize model behavior across more diverse manga scenarios.The stated gaps include genre, era, language, and cultural context.
  • Broader benchmarks and evaluation protocols are identified as future directions for more comprehensive manga-understanding assessment.

Ethics Statement

The work uses existing manga understanding benchmarks under their dataset usage conditions and does not redistribute raw manga images beyond permitted limits. Training uses sampled ANS rollouts and self-ranking without gold answers.

  • The study follows the usage conditions of the corresponding datasets because manga pages are copyrighted creative works.
  • Raw manga images are not redistributed beyond what the original resources permit.
  • Manual annotations were conducted by the authors, with disagreements resolved through majority voting and no external annotators or crowdworkers.
  • Policy optimization samples ANS rollouts and ranks candidate answers without providing gold answers in either stage.
  • ANS rollouts associate each panel ID with a concise clue while discouraging page-level summaries and unrelated descriptions.
  • Candidate-answer rankings are aggregated with Borda count to produce an answer preference reward.

B.1 Manga Understanding Benchmark

The evaluation covers three manga or comic understanding benchmarks and compares manga-specialized, general-purpose, and actively grounded multimodal models. It also reports implementation settings and inference latency for MANGO.

  • Benchmark suite: The evaluation uses MangaVQA, ChrOMIC, and MangaUB as manga understanding benchmarks.
  • Benchmark suite: MangaVQA contains 526 manually curated question–answer pairs requiring visual content, panel layout, character interactions, and embedded-text understanding.It covers both single-panel and multi-panel reasoning.
  • Benchmark suite: ChrOMIC contains 998 comic pages and evaluates panel reordering, description reordering, and multiple-choice question answering.The benchmark spans Western and Japanese comic styles.
  • Benchmark suite: MangaUB contains 6,585 questions and 18,179 prompts spanning low-level scene recognition and higher-level narrative understanding.Its tasks include location, time, weather, character counting, emotion recognition, panel localization, and next-panel prediction.
  • Compared models: Active-perception comparisons include TreeVGR, DeepEyes, and ViGoRL, which explicitly select, crop, or inspect visual regions during reasoning.
  • Efficiency: 5.21 seconds per example is MANGO’s MangaVQA inference latency on one NVIDIA A800 GPU with batch size 1, lower than existing active-perception baselines.The passage attributes additional overhead for DeepEyes and ViGoRL to iterative cropping and visual feedback.

C.2 Comparison with Closed-Source Models

MANGO remains competitive with strong closed-source models under the same evaluation protocol, despite using an 8B open-source backbone. The strongest closed-source models still achieve higher overall performance.

  • MangaVQA open-ended responses receive scores from 1 to 10 using an LLM judge, while ChrOMIC uses accuracy in its original multiple-choice setting.
  • MANGO uses an 8B open-source backbone yet outperforms several closed-source models on both manga benchmarks.The comparison uses the same evaluation protocol.
  • The strongest closed-source models achieve higher overall performance than MANGO.

C.3 Evidence Behavior of ANS

ANS selects compact, answer-relevant panel subsets and organizes their clues into an ordered evidence flow. Its original sketch ordering supports progressively improving answers more reliably than reversed or random orderings.

  • Panel-level evidence relevance: Keeping only ANS-selected panels performs much better than keeping the same number of random panels, while masking them causes a larger degradation.These perturbations associate ANS-selected panels with answer-relevant evidence.
  • Panel-level evidence relevance: ANS uses an average of 2.4 panels from 11.3 panels per page, preserving answer-related information in a compact subset.
  • Evidence sketch accumulation: When four-step sketches contain the same panel-clue steps, original ordering improves steadily as prefixes grow, whereas reverse and random orderings saturate or fluctuate.The analysis progressively reveals prefixes with k = 1, . . . , 4 steps before generating the final answer.
  • Evidence sketch accumulation: The ordering result suggests that ANS contributes a question-directed evidence flow beyond merely selecting relevant clues.
Loading 2608.29865v1…