Source-linked AI summary
ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions
Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, Mohamed Elhoseiny
TL;DR
AI research has emphasized answering questions, although answer usefulness depends on question quality. ChatCaptioner prompts ChatGPT to ask BLIP-2 informative image questions and summarize its answers; its captions received more human votes for image information and identified more objects than BLIP-2 alone.
Problem
AI research has primarily developed question-answering models, although their usefulness depends heavily on question quality and insightful questions have relied on human questioners.
Source: §1 · p. 2Method
Results
Three times as many votes from human evaluators favored ChatCaptioner for richer image information, and it identified 53% more objects than BLIP-2 alone.
Source: §1 · p. 2Takeaways & Limitations
Prompted LLMs can serve as zero-shot automatic questioners, and iterative questioning can acquire more informative image responses.
Source: §1 · p. 2Caption correctness depends on BLIP-2's answers, and a small portion remain incorrect despite an uncertainty prompt.
Source: §4.4 · p. 8
Abstract
from arXiv · showhide
Asking insightful questions is crucial for acquiring knowledge and expanding our understanding of the world. However, the importance of questioning has been largely overlooked in AI research, where models have been primarily developed to answer questions. With the recent advancements of large language models (LLMs) like ChatGPT, we discover their capability to ask high-quality questions when provided with a suitable prompt. This discovery presents a new opportunity to develop an automatic questioning system. In this paper, we introduce ChatCaptioner, a novel automatic-questioning method deployed in image captioning. Here, ChatGPT is prompted to ask a series of informative questions about images to BLIP-2, a strong vision question-answering model. By keeping acquiring new visual information from BLIP-2's answers, ChatCaptioner is able to generate more enriched image descriptions. We conduct human-subject evaluations on common image caption datasets such as COCO, Conceptual Caption, and WikiArt, and compare ChatCaptioner with BLIP-2 as well as ground truth. Our results demonstrate that ChatCaptioner's captions are significantly more informative, receiving three times as many votes from human evaluators for providing the most image information. Besides, ChatCaptioner identifies 53% more objects within the image than BLIP-2 alone measured by WordNet synset matching. Code is available at https://github.com/Vision-CAIR/ChatCaptioner
1. Introduction
The introduction argues that AI has focused on answering questions even though useful answers depend on question quality. ChatCaptioner uses ChatGPT to question BLIP-2 about images, then summarizes its answers; evaluations report richer captions than BLIP-2 alone.
1. Introduction
- AI research has primarily developed question-answering models, whose usefulness depends heavily on receiving high-quality questions.The introduction motivates automatic questioning as a way to guide models toward more informative answers.Source: §1 · p. 2
- Prompted LLMs such as ChatGPT can keep asking contextually relevant questions, making automatic questioning feasible.The introduction connects this capability to LLMs' zero-shot instruction-following ability.Source: §1 · p. 2
- ChatCaptioner has ChatGPT ask informative image questions, BLIP-2 answer them, and ChatGPT summarize the conversation into an enriched caption.ChatGPT cannot see images, so BLIP-2 supplies the visual information.Source: §1 · p. 2
- On sampled images from COCO, WikiArt, and CC, ChatCaptioner received three times as many human-evaluator votes as BLIP-2’s direct captioning for richer image information and identified 53% more objects than BLIP-2 alone within the image.Source: §1 · p. 2
2. Related Works
The related-work section distinguishes ChatCaptioner from approaches that generate isolated questions and situates it among prompt-driven LLMs and vision-language models. Its approach uses LLM instruction-following to continue questioning and BLIP-2 to extract image information.
2. Related Works
- Earlier question-generation methods require a passage and answer, while visual question generation focuses on independent image questions rather than follow-ups based on prior exchanges.ChatCaptioner instead targets knowledge acquisition, asks relevant questions iteratively, and requires no training for questioning.Source: §2 · p. 2
- Prompting research shows that LLMs can perform new tasks through in-context learning, step-by-step prompts, or zero-shot instruction following.ChatCaptioner leverages LLM instruction-following to keep asking image questions.Source: §2 · p. 2
- BLIP-2 uses a lightweight Q-Former to convert visual features into tokens for a frozen LLM, and ChatCaptioner uses its VQA ability to gather added image information.The cited related work also describes advances in image captioning and visual question answering.Source: §2 · pp. 2–3
3. ChatCaptioner
ChatCaptioner combines ChatGPT's iterative questioning with BLIP-2's image-question answering, then summarizes their conversation as a caption. Its prompting and post-processing guide the question and answer turns and the final summary.
3. ChatCaptioner
- ChatGPT asks image questions, BLIP-2 answers using visual information, and ChatGPT summarizes the conversation into a detailed caption.The workflow is illustrated with examples where multiple exchanges add image details beyond BLIP-2's initial description.Source: §3 · p. 3, Figure 3 · p. 4
3.1. Automatic Questioning
- The questioning prompt combines a task instruction, prior chat log, and question instruction; it starts with “Describe the image in detail” and guides one follow-up question at a time.Prompts discourage yes/no questions, while post-processing removes any fabricated answer beginning with “Answer:”.Source: §3.1 · pp. 3, 5
3.2. Question Answering
- BLIP-2 receives the task instruction, chat log, and answer instruction, including a prompt to admit uncertainty rather than imagine image content.Post-processing filters question text that BLIP-2 may append after an answer.Source: §3.2 · p. 5
- Figure 4 contrasts ChatCaptioner's richer image details with its much lower scores under traditional caption metrics.The figure caption presents this as a limitation of those metrics.Source: Figure 4 · p. 5
3.3. Context Summarizing
- After the conversation, a summarization instruction asks ChatGPT to produce a few sentences, ignoring answers of “no” or “not sure” without adding or omitting information.Source: §3.3 · p. 5
4. Experiments
The experiments assess caption informativeness and correctness alongside question diversity. They find richer captions and varied questions, while identifying BLIP-2 errors as a constraint on caption correctness.
4. Experiments
- Human evaluations across COCO, WikiArt, and CC assessed informativeness and accuracy; reference-similarity metrics may undervalue detailed captions, so human assessments were primary.The experiment also analyzed generated-question quality.Source: §4 · pp. 5–6, §4.1 · p. 6
4.1. Information Analysis
- Three to four times as many human votes favored ChatCaptioner over pure BLIP-2 and ground-truth captions for providing the richest image information.The evaluation sampled COCO, WikiArt, and Conceptual Caption images.Source: Table 1 · p. 6, §4.1 · p. 6
- ChatCaptioner identified 586 of 1,154 Pascal VOC objects versus BLIP-2’s 383, a 53% increase in object coverage.Objects were matched using WordNet synset similarity or synset closure.Source: Table 2 · p. 6, §4.1 · p. 6
4.2. Correctness Analysis
- 81% of final captions were deemed correct; BLIP-2 correctly answered 66.7% of ChatGPT’s questions, and its wrong answers caused 94% of incorrect captions.The table summarizes human caption judgments and BLIP-2 answer correctness.Source: Table 3 · p. 6, §4.2 · p. 6
- The uncertainty prompt reduced wrong answers by about 15%; without it, BLIP-2 would answer 118 of 147 uncertain questions incorrectly.Humans judged about 60% of those questions unanswerable from the image.Source: §4.2 · p. 7, Table 4 · p. 7, Figure 5 · p. 7
4.3. Question Analysis
- ChatGPT and InstructGPT rarely repeated questions and generated around 1,400 unique of 1,800; FLAN-T5 11B and OPT 6.7B repeated, while GPT-3 failed to generate meaningful image-related questions.The broader analysis found 1,419 unique questions among 1,800 generated for 200 CC images, spanning varied image aspects.Source: §4.3 · pp. 7–8, Table 5 · p. 8
4.4. Limitation
- Caption correctness depends on BLIP-2, whose answers remain sometimes wrong despite uncertainty prompting; LLM risks include potentially offensive or socially biased conversations and captions.The authors suggest better vision-language models may enhance visual descriptions.Source: §4.4 · p. 8
5. Conclusion
The paper finds that prompted LLMs can ask insightful, diverse questions and presents ChatCaptioner, which uses those questions to guide BLIP-2 toward more comprehensive image information and enriched captions.
5. Conclusion
- Prompted advanced LLMs can pose insightful, diverse questions, motivating ChatCaptioner for image captioning.The paper presents automatic questioning as a way to extract desired information.Source: §5 · pp. 8–9
- ChatCaptioner prompts ChatGPT to keep asking questions that expand its understanding of an image and guide BLIP-2 toward comprehensive image information.The resulting captions are described as more detailed and enriched.Source: §5 · pp. 8–9
- The authors aim to draw attention to automatic questioning’s potential in AI and inspire further research across domains.Source: §5 · p. 9
6. Appendix
The appendix reports a yes/no-question prompt ablation, per-image ChatGPT API costs, human-evaluation interfaces, generated-question examples, and random qualitative samples.
6.1. Cost
- Approximately 0.005 US Dollars per image covered ten question-answer rounds averaging 2,500 tokens each.The appendix specifies gpt-3.5-turbo API pricing at the time of the project.Source: §6.1 · p. 12
6.2. Yes/No Question Ablation
- 33% to 2%: adding the “Avoid asking yes/no questions” prompt reduced yes/no questions in 1,800 questions from 200 random CC samples.The prompt is included in both the task instruction and question instruction.Source: §6.2 · p. 12
6.3. Human Evaluation Interface
- The appendix presents the human-evaluation interface for the information experiments reported in Table 1.Source: §6.3 · p. 13, Figure 8 · p. 13
6.4. Random Examples of Generated Questions in Conceptual Captioning Dataset
- The CC question examples range across object attributes, expressions, counts, surroundings, and scene context.Examples ask about materials, colors, poses, weather, and event settings.Source: §6.4 · p. 14
6.5. More Qualitative Results
- Random qualitative samples are shown from CC, WikiArt, and COCO.Source: Figure 9 · p. 15, Figure 10 · p. 16, Figure 11 · p. 17