FlexCap: Describe Anything in Images in Controllable Detail

• Specifying a region and desired caption word count lets FlexCap bridge holistic image understanding and localized inquiry with spatially precise, semantically rich descriptions.The paper calls the ability to zoom in and out across visual content “flexible captioning.”Source: §1 · p. 1
• The training data pipeline produces 200 million YFCC100M and 32 billion WebLI image-box-caption triplets from captioned web images.The triplets pair an image, a proposed region, and a caption of a particular length.Source: §1 · pp. 1–2
• FlexCap’s descriptions are competitive with state-of-the-art VLMs on zero-shot image and video VQA, and its dense-captioning methods outperform baselines under comparable scenarios.The paper presents both outcomes as key technical contributions.Source: §1 · p. 2
• FlexCap’s localized textual descriptions can be supplied to LLMs to support visual question answering and dialog.The authors contrast this approach with providing visual features as input tokens to LLMs.Source: §1 · p. 2

Length conditioning

• A requested-length token conditions caption generation, allowing the model to vary information in descriptions and users to control output length.For captions sharing a prefix, next-word training can favor the end token over a differing continuation, depending on dataset statistics.Source: §2 · p. 2
• 30.8% of caption pairs share a prefix without length conditioning, versus 11.1% with a length-conditioning token.The token also helps distinguish captions with the same prefix during generation.Source: §2 · p. 3

Architecture

• FlexCap combines image-patch features and bounding-box coordinates as context for a causal transformer text decoder.The 590M-parameter model uses a 12-layer, 768-dimensional decoder; its vision encoder is initialized with SigLIP weights and remains trainable.Source: §2 · p. 3, Figure 2 · p. 3

Training

• FlexCap predicts caption tokens autoregressively with a next-token classification loss, ignoring padded tokens.Training runs for about 400K steps with AdamW, a batch size of 4096, 224 × 224 images, and up to eight boxes per image.Source: §2 · pp. 3–4

Inference

• At inference, FlexCap takes an image, target box, and desired length, then decodes until the end token or maximum decoding steps.Experiments use greedy decoding unless otherwise stated; beam search, temperature sampling, and nucleus sampling can also generate multiple captions.Source: §2 · p. 4

Dataset generation

• The pipeline extracts caption n-grams of one to eight words, filters incomplete phrases, then matches them to OWL-ViT region proposals at similarity scores above 0.1.Multiple matching n-grams can provide different descriptions for one box.Source: §3 · p. 4

WebLI

• WebLI yields 32 billion image-box-caption triplets from 2 billion images without new manual annotation, and its bottom-up vocabulary uses informative person labels such as baby, nurse, and baseball player.Source: §3 · p. 4

YFCC100M

• YFCC100M produces about 0.2 billion triplets from roughly 11 million images with at least one valid box, averaging 20 boxes per image.This is 160 times smaller than the WebLI localized captions dataset.Source: §3 · p. 4
• Because OWL-ViT and the CLIP YFCC100M subset are public, their localized captions dataset can be generated with open-source models and serves as a reproducible benchmark.WebLI is not publicly available yet.Source: §3 · p. 4

4.1 Correctness and Compliance of Generated Captions

• 13% mAP boost follows from producing multiple captions per box, and FlexCap outperforms contrastively trained approaches in zero-shot MS-COCO region classification.The authors suggest direct comparison of text embeddings may explain the boost, but identify this only as a possible reason.Source: §4.1 · p. 5
• FlexCap’s generated captions are mostly compliant with their target lengths, while longer examples add context, attributes, and alternative nouns.Source: §4.1 · p. 5

4.2 Visual Question Answering

• FlexCap-LLM outperforms zero-shot baselines on VQAv2, OKVQA, and VizWiz, and all recent GQA baselines except InstructBLIP.Source: §4.2 · pp. 6–7
• FlexCap-LLM significantly exceeds Flamingo on zero-shot MSRVTT-QA and MSVD-QA video question answering.The system samples eight frames uniformly, captions objects and regions in each, and combines those captions for the LLM.Source: §4.2 · pp. 7–8

4.3 Dense Captioning

• On Visual Genome, FlexCap performs better than other evaluated approaches when captioning ground-truth boxes and GRIT proposals.The GRIT-proposal comparison is against GRIT and uses 448 × 448 resolution, which is lower than the resolution used by GRIT.Source: §4.3 · p. 8

4.4 Open-Ended Object Detection

• Localize-then-describe with FlexCap achieves higher object recall and better matching scores than describe-then-localize, especially for small and medium objects.The authors attribute the difference mainly to VLMs describing salient objects rather than exhaustively listing scene objects.Source: §4.4 · p. 8, Figure 6 · p. 9

4.5 Discussion

• Prefixes let FlexCap extract targeted properties and other box-level information, including color, material, actions, uses, OCR, book details, and photo locations.The same box can yield different information depending on the prefix.Source: §4.5 · pp. 8–9
• The authors identify biases inherited from OWL-ViT and alt-text, and note that FlexCap-LLM combines a VLM and LLM rather than being end-to-end trainable.Source: §4.5 · p. 9

VQA

• FlexCap’s localized descriptions provide human-interpretable image representations for LLM-based VQA, with comparable state-of-the-art benchmark performance and a tool-based approach complementary to ViperGPT [44].Unlike approaches that provide visual features directly to LLMs, FlexCap leaves reasoning to the LLM and uses FlexCap to supply visual information.Source: §5 · pp. 9–10

Open-vocabulary detection

• Open-vocabulary detectors match text queries to boxes, whereas FlexCap generates descriptions for boxes and removes the need to manually provide per-image or per-dataset queries.RegionCLIP [61] instead associates manually defined text queries with bounding boxes.Source: §5 · p. 10

Dense captioning

• Dense captioning describes localized image regions, but its descriptive richness is often limited by existing image-captioning datasets [28] [32].FlexCap uses web-scale noisy image-text pairs to target more diverse descriptions in controllable detail.Source: §5 · p. 10

Length-controlled captioning

• Unlike ZeroCap and LIC [13], FlexCap supports region-level control at an exact word count rather than post-hoc stopping or coarse whole-image length levels.A separate approach [48] uses an image captioner followed by ChatGPT post-processing; FlexCap uses length tokens to modulate captioner output.Source: §5 · p. 10

6 Conclusion

• FlexCap describes localized image regions with captions whose richness can be controlled.The conclusion also reports that its localize-then-describe approach is effective relative to describe-then-localize approaches.Source: §6 · p. 10
• The training dataset uses noisy web-scale image-text pairs and open-vocabulary object detection to produce diverse image-box-caption examples at scale.The conclusion reports that FlexCap benefits from localized-caption dataset-size scaling, model-size scaling, and contrastive pretraining.Source: §6 · p. 10
• The paper reports strong performance on visual question answering and dense captioning and describes using FlexCap representations to connect images and videos to LLMs.It also reports that FlexCap complies with length conditioning.Source: §6 · p. 10

Dataset scale

• WebLI yields approximately 32 billion image-box-caption triplets, while the CLIP subset of YFCC100M yields approximately 0.2 billion.WebLI averages 4.19 unique boxes and 4.04 captions per box per image; YFCC100M averages 6.3 unique boxes and 3.3 captions per box.Source: A Localized Captions Dataset Details · p. 15

Caption-length distribution

• Caption lengths are not uniformly distributed in the WebLI dataset because there are more n-grams of length 1 to sample than length 8.Figure 9 covers caption-length distributions for region captions from WebLI and YFCC100M image-text pairs.Source: A Localized Captions Dataset Details · p. 15, Figure 9 · p. 15

N-gram Filtering

• N-gram filtering removes uninformative captions and n-grams with unsuitable opening or ending words to reduce web-data noise and support grammatical, informative captions.The filters remove captions made only of uninformative terms and n-grams starting or ending with words that sentences usually do not.Source: A Localized Captions Dataset Details · p. 15

Dataset Samples

• Dataset samples show boxes captioned from image alt-text, which may be clipped in the displayed examples and therefore not show the text behind every box caption.The paper next discusses how captions of varying lengths are matched with image objects.Source: A Localized Captions Dataset Details · p. 15

Large-scale Contrastive Pre-Training

• Contrastively pretrained weights yield better performance than training the ViT-B/16 backbone and text decoder from scratch in the reported region-captioning experiments.The study evaluates GT box captioning and dense captioning on GRiT proposals on Visual Genome.Source: B Ablations and Analysis · p. 15, Table 5 · p. 17

Qualitative analysis

• Figure 11 presents examples of objects that FlexCap can find with OWL-ViT but LLAVA does not.Source: Figure 11 · p. 17

Data Scaling

• Approximately 5% mAP improvement for GT box captioning is reported with a larger region-caption dataset, even after contrastive pretraining.The models use a contrastively pretrained ViT-B/16 encoder; Table 5b reports the data-scaling experiment on Visual Genome.Source: B Ablations and Analysis · p. 17, Table 5 · p. 17

Model Scaling

• The larger SO-ViT/14 (428M) model performs better than ViT-B/16 (85M) in the model-scaling experiment.Both models start from contrastively pretrained WebLI weights; Table 5c reports this comparison.Source: B Ablations and Analysis · p. 17, Table 5 · p. 17

C Open-Ended Object Detection

• FlexCap + OWL-ViTv2 is significantly better on medium and small objects and marginally better on large objects than LLAVA 1.5 [33] + OWL-ViTv2 [35].The comparison evaluates recall of Visual Genome ground-truth regions.Source: C Open-Ended Object Detection · p. 17

D Different prefixes with same image

• The figure illustrates diverse captions for one full-image box using the prefixes “a,” “the photo was taken in,” and “notice.”Source: Figure 12 · p. 18
• The prefix “Notice” leads captions to highlight noteworthy aspects that unprefixed captions omit.The examples also show the model identifying a country, possibly from a train logo.Source: D Different prefixes with same image · p. 18

E Visual Dialog

• FlexCap and OWL-ViTv2 describe the top 128 detected boxes once, then the system supplies conversation turns as context to the LLM for each query.Source: E Visual Dialog · p. 18
• The examples show the system reading image text, recognizing material, and localizing objects of interest.Source: E Visual Dialog · p. 18
• Because object captions are computed only at the conversation’s start, later turns add no overhead from querying a large VLM.Source: E Visual Dialog · p. 18

F Effect of Length on FlexCapLLM

• Increasing generated-caption length improves performance on both VQAv2 and GQA.Source: F Effect of Length on FlexCapLLM · p. 18
• The experiment evaluates captions of increasing lengths on the VQAv2 validation split.Source: F Effect of Length on FlexCapLLM · p. 18
• Table 6 reports the effect of caption lengths on FlexCap-LLM.Source: Table 6 · p. 18

G FlexCapLLM Details

• The VQA pipeline converts an image into localized text and passes it to an LLM with a task preamble and question.The figure describes FlexCap captions for image regions paired with box proposals as LLM input.Source: G FlexCapLLM Details · p. 18, Figure 14 · p. 19
• The prompt can include image dimensions, full-image descriptions, and object descriptions paired with bounding-box coordinates.Source: G FlexCapLLM Details · pp. 19–20

VizWiz

• For VizWiz, the prompt includes detection confidence; low-confidence detections are described as often making a question unanswerable, which the LLM may answer explicitly.Source: VizWiz · p. 20

MSRVTT and MSVD

• The video prompt describes detected objects by frame, iterating over frame indices before combining the prompt with the question.Source: MSRVTT and MSVD · p. 20