Source-linked AI summary
Caption Anything: Interactive Image Description with Diverse Multimodal Controls
Teng Wang, Jinrui Zhang, Junjie Fei, Hao Zheng, Yunlong Tang, Zhe Li, Mingqi Gao, Shanshan Zhao
TL;DR
Controllable image captioning is limited by scarce annotated data and methods that support only predefined controls. CAT combines pretrained image captioners, SAM, and an instruction-tuned LLM, using unified representations for visual and language controls. Qualitative examples demonstrate interactive captioning across controls and extensions to object-centric chatting and image-paragraph captioning.
Problem
Existing controllable image captioning is limited by scarce annotated image-text-control data and support for only predefined controls.
Source: §1 · p. 2Method
CAT combines pretrained image captioners, SAM, and an instruction-tuned LLM, representing visual controls as pixel-level masks and language controls as textual prompts.
Source: §1 · p. 2Results
Qualitative examples show captions responding to visual and language controls, alongside extensions to object-centric chatting and image-paragraph captioning.
Source: §1 · pp. 2–3, §4 · pp. 6, 8Takeaways & Limitations
Abstract
from arXiv · showhide
Controllable image captioning is an emerging multimodal topic that aims to describe the image with natural language following human purpose, $\textit{e.g.}$, looking at the specified regions or telling in a particular text style. State-of-the-art methods are trained on annotated pairs of input controls and output captions. However, the scarcity of such well-annotated multimodal data largely limits their usability and scalability for interactive AI systems. Leveraging unimodal instruction-following foundation models is a promising alternative that benefits from broader sources of data. In this paper, we present Caption AnyThing (CAT), a foundation model augmented image captioning framework supporting a wide range of multimodel controls: 1) visual controls, including points, boxes, and trajectories; 2) language controls, such as sentiment, length, language, and factuality. Powered by Segment Anything Model (SAM) and ChatGPT, we unify the visual and language prompts into a modularized framework, enabling the flexible combination between different controls. Extensive case studies demonstrate the user intention alignment capabilities of our framework, shedding light on effective user interaction modeling in vision-language applications. Our code is publicly available at https://github.com/ttengwang/Caption-Anything.
1 Introduction
Existing controllable image captioning is limited by small human-annotated datasets and fixed control menus. CAT uses foundation models and unified control representations to support diverse, combinable controls, with qualitative examples showing interactive captioning and extensions to object-centric chat and image-paragraph captioning.
1 Introduction
- Existing controllable captioning relies on limited annotated image-text-control data and predefined controls, constraining control comprehension and flexibility.Vanilla and dense captioning also lack explicit mechanisms for aligning output with user intention.Source: §1 · p. 2
- CAT combines pretrained image captioners, SAM, and an instruction-tuned LLM to process visual controls and refine captions according to language instructions.SAM generates a pixel-level mask for the selected region; the LLM refines the output sentence.Source: §1 · p. 2
- CAT supports three visual controls and four language controls, unified as pixel-level masks and textual prompts that can be flexibly combined and extended.The listed controls are clicks, boxes, trajectories, sentiment, length, language, and factuality.Source: §1 · p. 2
- Qualitative examples show user-selected objects captioned in preferred styles, with CAT also extended to object-centric chatting and image-paragraph captioning.The latter integrates knowledge from distinct foundation models to generate detailed, logically coherent descriptions.Source: §1 · pp. 2–3
- The paper presents CAT as a training-free foundation-model framework that reduces reliance on annotated data and demonstrates user-interactive capabilities.Its unified control representations are presented as supporting extensibility to new aspects of controllability.Source: §1 · p. 3
2 Related Work
Prior image and dense captioning describe salient content but lack explicit controls aligned with user intent. Related work adds region- and style-based controls, interactive segmentation, and instruction-following LLMs; CAT builds on these directions for interactive image description with diverse controls.
Image Captioning and Dense Captioning
- Image captioning commonly uses encoder-decoder models [36] [1] [12] [18] [17] [38], while dense captioning localizes descriptions for salient objects [13] [47] [48] [43].Prior work includes multi-scale contextual information sharing [48] and a unified dense-captioning and object-detection framework [43].Source: §2 · p. 3
Controllable Image Captioning
Interactive Image Segmentation
Large Language Models
- LLMs have transfer capabilities across language tasks and can interact with users; GPT-3 prompted subsequent LLM work [50] [4] [35] [30].The passage describes GPT-3 as a 175B-parameter model that demonstrated few-shot learning potential.Source: §2 · p. 3
- Human-feedback tuning supports LLMs' adaptation to language instructions [28] [26] [34], motivating the use of textual prompts for language controls.The paper applies LLMs to generate image descriptions aligned with user preferences.Source: §2 · p. 4
3 Caption Anything
Caption Anything combines a segmenter, captioner, and text refiner to turn visual prompts into region-focused captions tailored by language instructions. Its framework also supports object-centric chatting and image paragraph captioning.
3 Caption Anything
- CAT uses a segmenter, captioner, and text refiner to map visual controls to masks, generate raw captions, and apply language controls.The visual chain-of-thought step helps the captioner focus on the user-selected object.Source: §3 · p. 4, Figure 3 · p. 4
- SAM converts visual prompts such as points, boxes, and trajectories into pixel-level masks for user-interested regions.The captioner then describes the specified region using the original image and mask.Source: §3 · p. 4
- BLIP2 serves as the captioner, using frozen pretrained image encoders and LLMs with a querying transformer for zero-shot captioning.The querying transformer bridges the modality gap.Source: §3 · p. 4
- ChatGPT refines raw captions according to user instructions, and the text refiner can be replaced with open-source LLMs.The cited alternatives include LLaMA, OPT-IML, and BLOOM.Source: §3 · pp. 4–5
- Visual chain-of-thought first identifies the selected object against a white background, then uses that text and the cropped image to generate the final caption.This stepwise process is intended to keep background content from distracting the captioner.Source: §3 · p. 5
- For object-centric chatting, CAT combines an object mask and user query with a visual question-answering model and ChatGPT.LangChain acts as the control hub for predicting API calls.Source: §3 · p. 5
- For paragraph captioning, CAT combines dense captions and OCR-identified scene text, then prompts ChatGPT to summarize them coherently.Dense captions are generated by segmenting image contents and captioning each object through the CAT pipeline.Source: §3 · p. 5
4 Experiments
Qualitative examples show CAT generating captions from point, trajectory, and box controls, adapting language style, and supporting object-focused conversations and paragraph descriptions. The examples also report that visual chain-of-thought improves focus on selected objects.
Visual Controls
- Point, trajectory, and box controls let CAT identify and describe corresponding objects across varied image content.The reported examples characterize point-based object identification as accurate and show trajectory- and box-based controls as supported alternatives.Source: §4 · p. 6
Language Controls
- Language controls produce styles ranging from casual and conversational to formal and informative, including sentiment and factuality adjustments.Figure 6 also shows multiple languages.Source: §4 · p. 6
Object-Centric Chatting
- CAT performs visual question answering and conversations focused on objects selected within an image.The examples are presented as evidence of object-centric chatting.Source: §4 · pp. 6, 8, Figure 7 · p. 7
Visual Chain-of-Thought
- Visual chain-of-thought improves caption focus and reveals more intricate details about the selected object when direct inference is affected by background content.The reported examples connect this stepwise process with better alignment to user intent.Source: Figure 8 · p. 7, §4 · p. 8
Caption Everything in a Paragraph
- Generated image paragraphs cover most objects and textual information in a scene and are generally accurate and logically coherent.Some paragraphs also include reasoning information.Source: Figure 9 · p. 7, §4 · p. 8
5 Conclusion
Caption AnyThing (CAT) combines pre-trained image captioners, SAM, and an instruction-tuned LLM in a training-free controllable image-captioning framework. Its unified control representations support flexible adaptation and extension, and the experiments demonstrate strong user-interactive capabilities.
5 Conclusion
- CAT combines pre-trained image captioners with SAM and an instruction-tuned LLM to provide a training-free solution for controllable image captioning.The framework is presented as an adaptable solution for controllable image captioning tasks.Source: §5 · p. 8
- CAT’s unified representation of control signals supports flexibility, scalability, and extension with new aspects of controllability.Source: §5 · p. 8
- Experiments demonstrate strong user-interactive capabilities for CAT.Source: §5 · p. 8