PaperlayerView exact revision on arXiv

Source-linked AI summary

Describe Anything: Detailed Localized Image and Video Captioning

Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming-Yu Liu, Trevor Darrell, Adam Yala, Yin Cui

arXiv:2504.16072v1cs.CVcs.AI

TL;DR

Detailed localized captioning remains challenging because vision-language models must describe specific image and video regions while preserving both fine-grained details and broader context. DAM combines focal prompting and a localized vision backbone, while DLC-SDP supplies richer training data and DLC-Bench evaluates captions without reference captions. The resulting system achieves state-of-the-art performance on seven benchmarks covering multiple localized captioning granularities.

  • Problem

    Detailed and accurate descriptions for specific image and video regions remain an open problem, especially when models must capture fine-grained visual content and dynamic interactions.

  • Method

    DAM uses a focal prompt and localized vision backbone, DLC-SDP builds localized-caption data from segmentation and unlabeled web images, and DLC-Bench uses attribute-based evaluation without reference captions.

  • Results

    DAM achieves state-of-the-art performance on seven benchmarks spanning keyword-level, phrase-level, and detailed multi-sentence localized image and video captioning.

  • Takeaways & Limitations

    The paper provides a model, data pipeline, and benchmark for multi-granular localized captioning in images and videos.

  • Takeaways & Limitations

    The task formulation uses binary masks as localization input, with other localization forms transformed into masks through segmentation models.

Abstract

from arXiv · show

Generating detailed and accurate descriptions for specific regions in images and videos remains a fundamental challenge for vision-language models. We introduce the Describe Anything Model (DAM), a model designed for detailed localized captioning (DLC). DAM preserves both local details and global context through two key innovations: a focal prompt, which ensures high-resolution encoding of targeted regions, and a localized vision backbone, which integrates precise localization with its broader context. To tackle the scarcity of high-quality DLC data, we propose a Semi-supervised learning (SSL)-based Data Pipeline (DLC-SDP). DLC-SDP starts with existing segmentation datasets and expands to unlabeled web images using SSL. We introduce DLC-Bench, a benchmark designed to evaluate DLC without relying on reference captions. DAM sets new state-of-the-art on 7 benchmarks spanning keyword-level, phrase-level, and detailed multi-sentence localized image and video captioning.

1. Introduction

Detailed localized captioning remains difficult because models must preserve fine-grained regional details and contextual information across images and videos. The paper addresses these challenges with DAM, DLC-SDP, and DLC-Bench, achieving state-of-the-art results across seven benchmarks.

  • Motivation: Localized captioning remains open because image and video models must describe specific regions while capturing actions, motions, and interactions.The task could support fine-grained grounded image/video understanding and generation.
  • Challenges: Existing systems often lose fine-grained details from global representations, while cropping improves detail but removes contextual cues.These failures are especially problematic for small objects in complex scenes.
  • Challenges: High-quality DLC data is scarce because common datasets provide short phrases, while bounding-box-based synthetic data may not precisely define regions of interest.The paper identifies this scarcity as a separate obstacle to training rich localized descriptions.
  • Solutions: DAM combines a focal prompt with a localized vision backbone to encode targeted regions densely while integrating global context.The design is intended to preserve local detail and context, including for small objects in complex scenes.
  • Solutions: DLC-SDP expands segmentation annotations and unlabeled web images into diverse localized captions through a two-stage semi-supervised pipeline.A VLM expands mask-associated keywords into detailed captions, followed by self-training and multi-granularity summarization.
  • Solutions: DLC-Bench evaluates localized captions using predefined positive and negative regional attributes without relying on comprehensive reference captions.This is designed to provide more flexible and accurate evaluation of informative, precise descriptions.
  • Results: DAM achieves state-of-the-art performance on seven benchmarks spanning keyword-level, phrase-level, and detailed multi-sentence localized image and video captioning.The reported results cover multiple granularities and both image and video settings.

2. Related Work

Related work has expanded vision-language models toward regional captioning, but existing methods still struggle with precise localization, intricate regional details, and reliable evaluation. DAM addresses these limitations with explicit user-specified focal prompting, a localized vision backbone, richer SSL-curated data, and attribute-based benchmarking.

  • Vision-Language Models: General VLMs lack precise localization capabilities, limiting their ability to generate regional descriptions.Localized captioning requires fine-grained descriptions rather than only image-level captions.
  • Localized Image Captioning: Region-aware methods add referring inputs, but they still struggle to capture intricate details in the specified regions, especially for small objects.Prior systems may rely on global image embeddings or referring tokens that provide insufficient regional detail.
  • Data: Existing localized-captioning datasets often contain short phrases, while bounding-box-based synthetic generation introduces additional caveats.The paper instead uses human-annotated and unlabeled data for richer regional descriptions.
  • Benchmarking Localized Captioning: Reference-based language metrics may not correlate well with factual correctness or quality for detailed localized descriptions.This motivates evaluating captions through regional attributes rather than comprehensive reference matching.
  • Vision Models with Focus: Unlike model-driven focal-attention methods, DAM's focal prompt explicitly prioritizes user-specified regions, including non-salient objects.DAM combines focal prompting with a localized vision backbone to balance regional detail and global context.

3. DAM: Describe Anything Model

DAM generates detailed localized descriptions by combining high-resolution focal views with global context. Its focal prompt and localized vision backbone encode masks explicitly, fuse regional and global features, and support image and video captioning.

  • 3. DAM: Describe Anything Model: DAM uses a focal prompt and localized vision backbone to balance local detail with contextual information.The focal prompt includes the full image and mask plus a contextual focal crop and mask.
  • 3.2.1. Focal Prompt: The focal prompt expands the mask bounding box, crops the image and mask, and retains surrounding context around the specified region.The expansion factor α controls the added context, with clipping at image boundaries and a 48-pixel minimum dimension.
  • 3.2.2. Localized Vision Backbone: DAM embeds images and binary masks in spatial alignment, using separate global and regional vision encoders for localization and feature extraction.The mask is integrated through a one-channel patch embedding layer, while the focal branch produces regional features from the cropped image and mask.
  • 3.2.2. Localized Vision Backbone: Gated cross-attention lets regional features attend to global features, integrating broader context into the focal representation.The adapters are inserted into each transformer block of the regional vision encoder, with learnable scaling parameters initialized to zero.
  • 3.3. Video Captioning: The fused visual features and textual prompt tokens are passed to a large language model to generate detailed, context-aware descriptions.The same design extends to videos by concatenating features from multiple masked frames before language-model decoding.
  • 3.2.2. Localized Vision Backbone: Zero-initialized new modules preserve the pretrained VLM’s initial behavior and allow adaptation without rerunning pretraining.DAM remains efficient because its proposed components do not increase the vision-token sequence length; training uses approximately 1.5M samples.

4. DLC-SDP: SSL-based Data Pipeline

DLC-SDP addresses limited high-quality localized-caption data with a two-stage semi-supervised pipeline. It expands annotated segmentation data with web images, filters generated samples, and supports multiple caption granularities.

  • 4. DLC-SDP: SSL-based Data Pipeline: DLC-SDP builds a large and diverse dataset for detailed localized descriptions through two-stage semi-supervised learning.The pipeline addresses the limited availability of high-quality training data.
  • 4.1. Supervised Data Construction: The first stage expands keywords from annotated segmentation datasets into detailed captions for mask-referred regions using a VLM.Keywords include object class names, part names, and entities, while the masks provide high-quality localization.
  • 4.1. Supervised Data Construction: The model is trained to predict detailed descriptions without receiving the initial keywords as inputs.This matches inference conditions where class labels are unavailable and is intended to improve data quality over direct VLM prompting.
  • 4.2. Semi-supervised Data Expansion: The second stage self-trains on unlabeled web images through mask generation, description generation, confidence filtering, and data expansion.Open-vocabulary segmentation produces masks, DAM generates descriptions, and CLIP-based filtering retains high-quality samples.
  • 4.2. Semi-supervised Data Expansion: An LLM summarizes detailed descriptions into shorter forms, enabling caption outputs from phrases and short sentences to multi-sentence narratives.The self-training stage also broadens object categories and data diversity beyond the initial supervised dataset.
  • 4.3. Results: DAM trained with DLC-SDP outperforms GPT-4o and o1, demonstrating the reported effectiveness of the data pipeline.The comparison is reported for two strong closed-source baselines.

5. DLC-Bench: Benchmark for DLC

DLC-Bench evaluates detailed localized captions by testing whether descriptions include region-specific details while avoiding factual errors and irrelevant content. Its reference-free pipeline combines masked-region caption generation with LLM-based scoring using curated positive and negative questions.

  • Benchmark design: DLC-Bench evaluates descriptions for rich relevant details while penalizing factual errors and information about irrelevant regions.
  • Evaluation pipeline: The model first generates a detailed description for each masked benchmark region, after which an LLM judge assesses it.
  • Question design: Positive questions test whether specified object-part attributes are accurately included; omissions receive no points, while factual errors incur penalties.
  • Question design: Negative questions test whether descriptions avoid absent attributes and details irrelevant to the specified region.
  • Question design: The benchmark uses 892 manually verified questions covering diverse attributes and potential hallucinations.

6. Results

DAM achieves state-of-the-art localized captioning across image and video tasks, granularities, and evaluation settings. Its qualitative and ablation results support detailed region descriptions under challenging visual conditions.

  • Quantitative Results: DAM achieves state-of-the-art performance on 7 benchmarks spanning keyword-, phrase-, and detailed localized image and video captioning.The benchmarks include both in-domain and zero-shot settings.
  • Quantitative Results: 73.2% semantic IoU and 84.2% semantic similarity on PACO exceed the previous best by 23.2% and 8.5%, respectively.PACO includes full objects and parts in complex scenes.
  • Quantitative Results: 12.3% relative improvement over the previous best on 5 metrics is achieved in zero-shot Flickr30k Entities phrase-level captioning.The task requires brief regional descriptions containing object identification and attributes.
  • Quantitative Results: 33.4% and 13.1% relative improvements over the previous best are achieved on Ref-L4 short- and long-captioning metrics, respectively.Ref-L4 evaluates detailed descriptions ranging from long sentences to multiple sentences.
  • Quantitative Results: DLC-Bench results show DAM surpassing general, region-specific, API-only, and thinking-mode models on positive and negative accuracy.DLC-Bench evaluates detailed descriptions without comprehensive reference captions.
  • Quantitative Results: 19.8% relative improvement over the previous best is achieved on HC-STVG detailed localized video captioning.DAM also surpasses the previous best on VideoRefer in both zero-shot and in-domain settings.
  • Qualitative Results: Qualitative results show accurate detailed descriptions for user-specified video objects under motion and occlusion, controllable granularity, and coherent multi-view 3D object descriptions.DAM can integrate information from multiple frames and describe stationary objects.
  • Ablations: Focal prompting improves accuracy to 67.3%, compared with 48.7% using only the full image, 60.1% using local crops, and 63.2% with cross-attention.Naive concatenation performs at 42.4%, while focal crops reach 65.4%; SSL with 10% unannotated SA-1B images also improves accuracy to 67.3%.

7. Discussions and Conclusion

The paper presents DAM, DLC-SDP, and DLC-Bench as a combined approach to detailed localized captioning. DAM achieves state-of-the-art performance across seven multi-granular regional captioning benchmarks.

  • Discussions and Conclusion: DAM balances local detail and global context through a focal prompt and localized vision backbone for image and video captioning.DLC-SDP expands training data using segmentation datasets and unlabeled web images, while DLC-Bench uses attribute-based evaluation without reference captions.

A. Challenges in Generating Detailed Localized Descriptions with Off-the-Shelf VLMs

Off-the-shelf VLMs struggle to produce detailed descriptions of specified regions because precise localization conflicts with retaining contextual information. Existing text, crop, and visual-marking strategies each introduce localization or context problems.

  • Limitations of Off-the-Shelf VLMs: Current VLMs excel at global image descriptions but generally lack accurate mechanisms for specifying and localizing regions of interest.Text-based localization can require cumbersome referring phrases and still mislocalize the intended object.
  • Limitations of Off-the-Shelf VLMs: Cropping or masking forces attention onto a region but removes contextual information, often confusing the model and impairing object identification.This exposes the tension between regional detail and the context needed for correct recognition.
  • Limitations of Off-the-Shelf VLMs: Overlaying points, scribbles, contours, or alpha masks can blend into complex scenes, especially around small objects, and may make images out-of-distribution.These effects can confuse VLMs and disrupt their original output quality.
  • DAM’s Design Response: DAM accepts localization as a separate 2D-space input, making the target explicit while preserving the image’s original distribution.The approach is intended to support accurate localization in complex scenes without distracting markings.

B. Evaluation Benchmarks

The paper evaluates localized captioning across keyword, phrase, and detailed image and video tasks using in-domain and zero-shot benchmarks. It also introduces DLC-Bench to assess regional descriptions through predefined attributes rather than comprehensive reference captions.

  • Benchmark Coverage: DAM is evaluated on 7 benchmarks covering LVIS, PACO, Flickr30k Entities, Ref-L4, DLC-Bench, HC-STVG, and VideoRefer.The set spans keyword-level, phrase-level, detailed image, and detailed video captioning.
  • Keyword-Level Captioning: Open-class keyword-level benchmarks require freeform keywords identifying objects or parts, with PACO additionally distinguishing full objects from object parts.LVIS uses object class names, while PACO can require both object and part names.
  • Phrase-Level Captioning: Phrase-level captioning evaluates brief regional descriptions containing object identification and attributes, commonly using CIDEr, METEOR, BLEU, ROUGE_L, and SPICE.Flickr30k Entities is evaluated zero-shot because its training entities are excluded from model training.
  • Detailed Captioning: Detailed localized captioning requires long-sentence or multi-sentence descriptions covering regional objects, parts, attributes, and relationships.Ref-L4 evaluates this capability with short-captioning metrics and CLAIR against reference captions.

B.4. Detailed Localized Video Captioning Benchmarks

The video-captioning benchmarks evaluate localized descriptions across spatial-temporal caption quality and four VideoRefer-Bench dimensions, but reference captions can misclassify correct details as hallucinations.

  • HC-STVG evaluates localized video captions with CIDER, METEOR, BLEU, ROUGE_L, and SPICE.
  • VideoRefer-Bench scores Subject Correspondence, Appearance Description, Temporal Description, and Hallucination Detection in zero-shot and in-domain settings.
  • Boxes can ambiguously specify video regions, whereas masks provide more precise references for localized description.
  • Reference captions may omit valid regional details, causing caption metrics to label correct predictions as hallucinations.
  • In one example, DAM correctly described gesture and expression details that the reference caption omitted, producing an invalidly low hallucination-detection score.

C.4. Potential Limitations

DAM is specialized for detailed localized captioning rather than broad vision-language tasks, and it can still misrecognize objects or confuse combined object and camera motion. Its DLC-Bench evaluation scores correct details while penalizing incorrect, hallucinated, or mislocalized content.

  • Scope: DAM is optimized for multi-granular localized image and video descriptions, not general vision-language tasks.The model prioritizes in-depth localized analysis over breadth of general vision-language understanding.
  • Failure cases: DAM can misrecognize a region, such as classifying a frog-shaped slipper as a frog.This is identified as a failure case in the paper’s qualitative discussion.
  • Failure cases: DAM can confuse object motion with camera motion, incorrectly describing a person as pulling the body upward.The example illustrates a limitation in interpreting motion when both object and camera movement are present.
  • Evaluation: DLC-Bench rewards correct region details and penalizes incorrect details, hallucinations, and mislocalization.Positive and negative question scores are normalized separately and averaged into an overall score.

E. Additional Ablation Studies

The ablations attribute DAM’s DLC performance to its architecture, while testing prompt augmentation, joint image-video training, and Set-of-Marks prompting. The results favor DAM’s focal and localized visual design, with smaller effects from training and prompting choices.

  • Architecture design: DAM’s architecture outperforms VP-SPHINX when both models are trained on the same curated DLC data.The comparison supports the contribution of the focal prompt and localized visual backbone beyond training-data differences.
  • Prompt augmentation: Prompt augmentation decreases overall benchmark accuracy by 0.6%, while improving instruction following for prompts with added output requirements.Negative-question performance remains unchanged, and the non-augmented model is used for benchmark evaluations.
  • Image-video training: Joint image-video training slightly improves detailed localized image captioning over image-only training with the same 3B model size.The joint-training variant combines localized image and video description tasks.
  • Set-of-Marks prompting: Set-of-Marks prompting produces degraded DLC-Bench accuracy compared with the prompt engineering used in DAM’s annotation pipeline.The paper attributes this to marks blending into objects or backgrounds and sometimes masking object parts in complex scenes.
  • Additional capabilities: DAM shows emerging zero-shot regional question-answering abilities despite not being trained on regional QA datasets.Examples include identifying object properties and recognizing a strawberry within a masked region.

G.4. Qualitative Comparisons with Strong Baselines

Qualitative comparisons show DAM producing more accurate and detailed localized descriptions than strong image and video baselines. Its examples cover contextual object recognition, fine attributes, occlusion, and motion under challenging visual conditions.

  • Localized image captioning: DAM gives detailed descriptions of masked objects while preserving broader contextual understanding in image comparisons.GPT-4o is vague or misrecognizes objects, while VP-SPHINX misses details such as color and texture; DAM correctly describes the seal and towel.
  • Localized video captioning: DAM more accurately describes object identity, appearance, and motion in videos than GPT-4o, Qwen2.5-VL-7B, and VideoRefer-7B.The comparisons include correctly tracking a sheep’s identity and full movement and describing a cow’s motion in detail.
  • Localized image captioning: DAM uses context to recognize difficult localized objects, including a roller blind, while excluding objects outside the masked region.Additional LVIS examples include precise descriptions of a sofa cushion and giraffe without mentioning nearby out-of-region objects.
  • Evaluation data: The DLC training data contains 1.46M annotated regions across 819k images, plus 94k regions across 37k videos.Image data combines segmentation datasets and SSL expansion, while video regions represent instances across multiple frames.
  • Localized video captioning: DAM describes small objects and slight or stationary motion despite large object or camera motion, and it handles partial occlusion.The video examples come from SA-V and MOSE datasets.
Loading 2504.16072v1…