Source-linked AI summary
QG-CoC: Question-Guided Chain-of-Captions for Large Multimodal Models
Kuei-Chun Kao, Hsu Tzu-Yin, Yunqi Hong, Ruochen Wang, Cho-Jui Hsieh
TL;DR
MLLMs still struggle with complex multimodal tasks such as multi-image relationships, while many prompting approaches are focused on single-image or constrained settings. The paper proposes QG-CoC, which decomposes questions and uses sub-question-guided captions to build answers. It reports consistent gains across multi-image scenarios and generalization to single-image settings, with the full model reaching 68.2 accuracy on MUIR and 55.4 on MMIU.
Problem
MLLMs can struggle with complex multimodal tasks, including temporal, spatial, and multi-image relationships, and existing prompting approaches are not general across diverse multi-image scenarios.
Source: §1 · p. 1Method
QG-CoC decomposes a question into sub-questions, generates captions guided by each one, and combines sub-question–sub-answer pairs to answer the original problem.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The authors present QG-CoC as a strong baseline for future multimodal understanding tasks across single- and multi-image scenarios.
Source: §1 · p. 2QG-CoC relies on advanced MLLM captioning ability and may deteriorate with less advanced language models or more challenging tasks; the experiments do not cover all models and frameworks.
Source: Limitations · pp. 9–10
Abstract
from arXiv · showhide
Recently, Multimodal Large Language Models (MLLMs) encounter two key issues in multi-image contexts: (1) a lack of fine-grained perception across disparate images, and (2) a diminished capability to effectively reason over and synthesize information from multiple visual inputs. However, while various prompting methods aim to describe visual content, many existing studies focus primarily on single-image settings or specific, constrained scenarios. This leaves a critical gap in understanding and addressing how MLLMs tackle more general and complex multi-image reasoning tasks. Thus, we first extensively investigate how current prompting methods perceive fine-grained visual details and process visual information when dealing with multiple images. Our findings reveal that existing prompting methods fall short in attending to needed clues and seamlessly integrating perception and reasoning. Inspired by the findings, we propose a new zero-shot prompting method, Question-Guided Chain-of-Captions (QG-CoC), a generalized prompting approach that effectively handles problems with an arbitrary number of images. We evaluate our method on various open-source and closed-source MLLMs for multi-image and single-image benchmarks. Experimental results indicate that QG-CoC demonstrates competitive performance across tasks and exhibits robust improvements in the challenging scenarios where existing prompting methods fail.
1 Introduction
MLLMs show promise on single-image tasks but still struggle with complex multimodal tasks, including multi-image relationships. The paper analyzes prompting strategies and introduces QG-CoC, a zero-shot method for an arbitrary number of images that performs strongly across multi-image and single-image settings.
Motivation
- MLLMs show promise on single-image tasks but still struggle with complex multimodal understanding, including temporal, spatial, and multi-image relationships.The cited empirical studies identify these as areas of difficulty.Source: §1 · p. 1
- Existing prompting approaches often represent visual scenes as text for reasoning, but may miss key information needed to relate multiple images and are not general across diverse multi-image scenarios.The paper identifies multi-perspective, multi-relation, and multi-understanding settings as examples.Source: §1 · p. 1
Approach
- QG-CoC decomposes a question into sub-questions, generates captions conditioned on each sub-question, and uses sub-question–sub-answer pairs as prior knowledge for the final response.This process is introduced following a preliminary analysis of captioning strategies and adapted prompting methods.Source: §1 · pp. 1–2
Contributions
- QG-CoC is a zero-shot prompting method for an arbitrary number of images, and the paper reports consistent gains in multi-image scenarios and generalization to single-image scenarios across open- and closed-source models.The authors also describe it as a strong baseline for future multimodal understanding tasks.Source: §1 · p. 2
2 Related Work
Prior multimodal prompting work transforms visual input into text or uses tools, while benchmarks increasingly assess multi-image reasoning. This paper focuses on a prompting strategy that does not require fine-tuning, separate visual modules, or external tools.
Multimodal Prompting Methods
- Multimodal prompting methods transform images into captions, graphs, or bounding boxes, or use agent-style pipelines with tools; these approaches can still leave models to integrate spatial, temporal, and contextual cues across images.The paper positions its framework as multimodal reasoning without fine-tuning or separate visual modules or external tools.Source: §2 · p. 2
Multimodal Understanding Benchmarks
- Many multimodal benchmarks focus on single-image settings, while newer benchmarks assess multi-image tasks such as comparison, video understanding, and grounding.The paper says existing MLLMs often need specific prompting to address multi-image problems, while common performance-enhancement techniques use supervised fine-tuning.Source: §2 · p. 2
3 Preliminaries
The preliminary analyses compare captioning choices and adaptations of existing prompting methods for multi-image tasks. Detailed, individual, question-guided captions perform better in the reported comparisons, while adapted methods show difficulties with implicit context and complex reasoning.
3.1 Analysis on Different Captioning Strategies under Multi-Image
- The captioning analysis compares concise versus detailed captions, individual versus summarized captions, and captions with versus without the question across eight settings and both open- and closed-source models.Results for each control factor are averaged over four relevant variations.Source: §3.1 · p. 3
- Detailed captions improve multi-image accuracy over concise captions, and individual captions outperform a single summary across the image set.The analysis attributes the detailed-caption advantage to richer descriptions and the individual-caption advantage to retaining image-specific information.Source: §3.1 · pp. 3–4, Table 1 · p. 3
- Question-guided captions improve overall multi-image task accuracy compared with captions generated from images alone.The analysis says they focus on task-relevant visual elements.Source: Table 1 · p. 3, §3.1 · p. 4
3.2 Adjusting Existing Prompting Methods to Multi-Image Scenarios
- Adapted DDCoT, CCoT, and CoCoT can identify entities and explicit relations but struggle with implicit context and complex reasoning across an arbitrary number of images.The reported examples include missing spatial context, unrelated object descriptions, and vague descriptions of similarities and differences.Source: §3.2 · p. 4
- The study motivates a zero-shot QG-CoC approach that balances detail, specificity, and relevance for multi-image contexts.The subsection introduces it in response to limitations observed in adapted prompting methods.Source: §3.2 · p. 4
4 Question-Guided Chain-of-Captions
QG-CoC structures multi-image reasoning around question decomposition, targeted captioning, and answer integration. Each stage connects the original question to image evidence through sub-questions and their answers.
Question decomposition
- QG-CoC first decomposes a complex question into simpler sub-questions targeting specific aspects of the images.Examples include a subject’s action, outcome, or reaction.Source: §4 · p. 5
Question-guided captioning
- For each sub-question, the MLLM generates a targeted caption describing relevant visual evidence such as objects, actions, effects, or scene changes.The captions connect each reasoning step to image content.Source: §4 · p. 5
Answer integration
- The model answers each sub-question from its captions, then combines those answers into a final answer to the original question.The paper says this step-by-step process improves prediction accuracy and explainability.Source: §4 · p. 5
5 Experimental Results
The experiments use closed- and open-source MLLMs and compare QG-CoC with prompting baselines, including a no-prompt baseline.
5 Experimental Results
- Experiments include GPT-4o and Gemini-1.5-Flash as closed-source models, and Mantis-idefics2-8B, LLaVA-OneVision-7B, and Qwen2.5-VL-7B as open-source models.The open-source models support multiple image inputs.Source: §5.1 · p. 5
- The open-source models have limited capacity to process and follow long prompts that generate additional context in the first stage.Gemini-1.5-Flash serves as oracle captioning for open-source model evaluation.Source: §5.1 · p. 5
- The default baseline applies each model without prompt engineering; QG-CoC is also compared with five state-of-the-art methods.Source: §5.1 · p. 5
MUIR MMIU ScienceQA MMMU MMBench
The evaluation spans multi-image benchmarks and single-image benchmarks, comparing QG-CoC with several prompting baselines. QG-CoC is reported to perform strongly across both benchmark types, with sub-question-targeted captions distinguishing it from the alternatives.
5.2 Main Results
- QG-CoC demonstrates strong performance across multi-image and single-image benchmarks, as reported in Table 2.Table 2 reports answer accuracy for different models and prompting methods.Source: Table 2 · p. 6, §5.2 · p. 7
MUIR MMIU ScienceQA MMMU MMBench
- The compared baselines include detailed captioning, question-guided detailed captioning, DDCoT, CCoT, and CoCoT, alongside a no-prompt baseline.The baselines generate textual representations in a first step, then answer using the images, question, and generated text.Source: MUIR MMIU ScienceQA MMMU MMBench · p. 6
- MuirBench covers 12 diverse multi-image tasks, while MMIU spans 7 relationship types and 52 tasks; single-image evaluation includes MMMU, MMBench, and ScienceQA.The authors report overall averages across MuirBench tasks and MMIU tasks, and filter some low-quality MMIU spatial and semantic tasks.Source: MUIR MMIU ScienceQA MMMU MMBench · p. 7
5.2 Main Results
- QG-CoC targets captions to sub-questions, making them more focused than general detailed captions and typically more effective than a single question-guided captioning pass.It decomposes the main question before generating captions for each sub-question.Source: §5.2 · p. 7
- QG-CoC frequently outperforms DDCoT and consistently exceeds CCoT, using targeted captions to ground sub-problems in visual evidence and keep extracted information task-relevant.Source: §5.2 · p. 7
- QG-CoC generally achieves higher accuracy than CoCoT, whose image-similarity and difference descriptions may be less suited to tasks beyond comparison.The authors characterize QG-CoC as a more general framework for varied reasoning needs.Source: §5.2 · p. 7
6 Discussion
The discussion analyzes performance across image relationships, component ablations, error patterns, and computational overhead. It reports gains from the full method alongside persistent reasoning challenges and added inference costs.
6 Discussion
- Across MMIU image relationships, direct prompting generally fares better on low-level semantic tasks, QG-CoC supplies more spatial information, and all methods struggle with reasoning-intensive temporal tasks.QG-CoC outperforms existing methods on the described high-level semantic tasks, including causality reasoning and emotion recognition.Source: §6 · p. 8, Figure 4 · p. 8
- 68.2 on MUIR and 55.4 on MMIU are the full QG-CoC model’s reported accuracies, the highest among the Gemini-1.5-Flash ablation variants.On MUIR, accuracy rises from 66.0 to 66.5 with question decomposition, then to 67.2 with question-guided captioning before reaching 68.2 with the full model.Source: Table 3 · p. 8, §6 · p. 8
- 35.0% of sampled error cases are reasoning failures, 33.3% reflect inaccurate question understanding, and 31.7% reflect failures to capture image details.The analysis samples 10 errors per task, totaling 120 Gemini-1.5-Flash errors on MuirBench; reasoning failures can persist despite accurate context.Source: Table 4 · p. 8, §6 · p. 9
- Error types vary by MUIR task: holistic multi-image tasks are dominated by reasoning errors, fine-grained perception tasks by perception errors, and abstract tasks show more question-understanding errors.Reasoning across multiple images remains the most significant challenge in the reported breakdown.Source: Table 5 · p. 9, §6 · p. 9
- QG-CoC increases token usage and runtime; for open-source models, it yields reported gains of +12% for LLaVA-OV and +5% for Mantis.The authors describe this as a trade-off for performance improvements and say the overhead is comparable to other two-stage methods.Source: Table 6 · p. 9, §6 · p. 9
7 Conclusion
QG-CoC decomposes a problem, generates captions guided by each sub-question, then uses the sub-question–sub-answer pairs to answer the original problem. Experiments report advantages across MLLMs and benchmarks.
7 Conclusion
- QG-CoC decomposes the problem, captions each image guided by a sub-question, then uses sub-question–sub-answer pairs as prior knowledge to answer the original problem.The experiments report advantages across different MLLMs and benchmarks.Source: §7 · p. 9
Limitations
The paper presents QG-CoC as a strong baseline, while noting limits in model coverage and dependence on advanced MLLM captioning. The appendices give default experiment settings and show prompts for the captioning settings and methods.
Limitations
- The work is a strong baseline, not coverage of all models and frameworks; reliance on advanced MLLM captioning may hurt performance with less advanced language models or harder tasks.The authors suggest exploring more diverse, complex scenarios, including geometric shapes and 2D or 3D spatial information.Source: Limitations · pp. 9–10
A Model Hyperparameters
- Experimental hyperparameters use default values for consistency, with model-specific generation parameters detailed in Table 7.Source: A Model Hyperparameters · p. 11
B.1 Full Model Prompt
- The full model prompts for different captioning settings are shown in Figure 6.Source: §B.1 · p. 11
C.1 Full Model Prompt
- The full model prompts for different methods are shown in Figure 7.Source: §C.1 · p. 11
D.1 Full Model Prompt
- The full model prompt for QG-CoC is shown in Figure 8.Source: §D.1 · p. 11
D.2 Full Quantitative Results Across Various Image Relationships
Across MMIU and MUIR, the reported findings remain consistent: QG-CoC outperforms other prompting methods. Task-level performance differences are generally larger for open-source models.
D.2 Full Quantitative Results Across Various Image Relationships
- QG-CoC outperforms other prompting methods across MMIU and MUIR, with task performance reported in Tables 8 and 9 and Figure 5.Source: §D.2 · p. 11
- Across datasets and prompting methods, task-level performance differences are generally larger for open-source than closed-source models.Source: §D.2 · p. 11
D.3 Qualitative Analysis of Error Cases
The examples show three distinct error points: interpreting the question and decomposing it, captioning image content, and reasoning in the final response.
D.3 Qualitative Analysis of Error Cases
- In E1, misreading “tortoise” as “duck” leads the model to decompose the question into incorrect sub-questions.Source: §D.3 · p. 11
- In E2, the model miscounts the L shape as four squares instead of three.Source: §D.3 · p. 11
- In E3, accurate sub-questions and captions identify “a person walking” as the difference, but the final reasoning is still incorrect.Source: §D.3 · p. 11