PaperlayerView exact revision on arXiv

Source-linked AI summary

One Patch to Caption Them All: A Unified Zero-Shot Captioning Framework

Lorenzo Bianchi, Giacomo Pacini, Fabio Carrara, Nicola Messina, Giuseppe Amato, Fabrizio Falchi

arXiv:2510.02898v5cs.CV

TL;DR

Supervised region captioning depends on manually written captions and does not scale well. This paper shifts zero-shot captioning to patches, aggregating their features for arbitrary regions; the resulting framework achieves state-of-the-art performance across regional captioning tasks, including dense and region-based captioning.

  • Problem

    Region-level captioning requires manually labeled captions for image regions, and the paper identifies captioning patches without direct patch-level supervision as a challenge.

    Source: §1 · p. 1

  • Method

    The framework treats patches as captioning units and aggregates frozen vision-language backbone features to represent arbitrary regions, without region-level supervision.

    Source: §1 · p. 1, §3.1 · p. 4, §3.2 · p. 4

  • Results

    The patch-centric framework sets the state of the art on various regional captioning tasks, including dense and region-based captioning, and the newly proposed trace task.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The framework bridges local and global image captioning in a zero-shot setting, with one backbone forward pass supporting captions for multiple regions.

    Source: §5 · p. 8

    The model still lags behind fully supervised, task-specific approaches; patch context is fixed by the backbone, and modality-gap noise can cause hallucinations.

    Source: §5 · p. 8

Abstract

from arXiv · show

Zero-shot captioners are recently proposed models that utilize common-space vision-language representations to caption images without relying on paired image-text data. To caption an image, they proceed by textually decoding a text-aligned image feature, but they limit their scope to global representations and whole-image captions. We present a unified framework for zero-shot captioning that shifts from an image-centric to a patch-centric paradigm, enabling the captioning of arbitrary regions without the need of region-level supervision. Instead of relying on global image representations, we treat individual patches as atomic captioning units and aggregate them to describe arbitrary regions, from single patches to non-contiguous areas and entire images. We analyze the key ingredients that enable current latent captioners to work in our novel proposed framework. Experiments demonstrate that backbones producing meaningful, dense visual features, such as DINO, are key to achieving state-of-the-art performance in multiple region-based captioning tasks. Compared to other baselines and state-of-the-art competitors, our models achieve better performance on zero-shot dense captioning and region-set captioning. We also introduce a new trace captioning task that further demonstrates the effectiveness of patch-wise semantic representations for flexible caption generation. Project page at https://paciosoft.com/Patch-ioner/ .

1. Introduction

The paper addresses the cost and scalability limits of supervised region captioning by treating image patches as captioning units and aggregating them into arbitrary regions. It studies how existing zero-shot captioners can support this approach across region scales.

1. Introduction

  • Supervised region captioning requires manually written captions for box sequences or sets, a process the authors describe as poorly scalable.The paper also identifies how to caption patches without direct patch-level ground-truth supervision as a central challenge.Source: §1 · p. 1
  • The framework treats a patch as the basic captioning unit and aggregates frozen-backbone features to caption arbitrary, potentially non-contiguous regions without region-level supervision.Its zero-shot setup assumes text-only training data is available, but paired text-region data is not.Source: §1 · p. 1
  • The authors analyze decoder and aggregation choices, emphasizing backbones such as DINO that produce meaningful, localized patch representations.Their framework adapts existing zero-shot captioners and examines modality-gap mitigation strategies for interpreting visual features without paired image-text data.Source: §1 · p. 2
  • Patch-centric zero-shot captioning reaches state-of-the-art or comparable results across dense, region-set, and image captioning, and introduces mouse-trace captioning.The framework is evaluated across four zero-shot captioning tasks spanning region granularities.Source: §1 · p. 2

2. Related Work

  • Prior zero-shot captioners generally decode from global representations, which are poorly suited to localized semantic detail.The literature distinguishes global CLIP-based captioning approaches from dense-representation work addressing patch-level language alignment.Source: §2 · pp. 2–3
  • Existing controllable and dense captioning systems often require ground-truth box supervision, while localized narratives provide timed mouse tracks for track-level evaluation.The paper proposes a unified zero-shot framework spanning image- to patch-level captioning.Source: §2 · p. 3

3. A Patch-centric Framework

The framework makes image patches the basic captioning units, then aggregates them into arbitrary regions for text-only zero-shot decoding. This factorization supports multiple region granularities without region–text or image–text supervision.

3. A Patch-centric Framework

  • A frozen vision–language backbone produces language-aligned patch embeddings; aggregating a region’s patches and decoding the result yields its caption.The decoder is trained on text data, while region selection is handled separately from text decoding.Source: §3 · pp. 3–4

3.1. Motivation and Formulation

  • Traditional early region injection requires region-caption annotations; the proposed factorization separates image encoding, parameter-free region aggregation, and text decoding.The formulation is t = ϕ(aggR(ψ(I), R)).Source: §3.1 · p. 4
  • Mean aggregation combines the selected patch embeddings, allowing regions to be boxes, masks, traces, single patches, or full-image grids.The authors report that the specific aggregation choice has limited impact in regional captioning.Source: §3.1 · p. 4
  • The decoder reconstructs captions from text embeddings, but a modality gap separates visual and textual representations, prompting projection, input-perturbation, and diffusion-based mitigation strategies.These strategies respectively map visual features into the text subspace, train decoder robustness to another subspace, or iteratively refine visual features toward the text subspace.Source: §3.1 · p. 4
  • One backbone forward pass supplies patch features that can be reused across multiple regions, and aggregation supports arbitrary patch sets.The framework is designed for text-only training without paired image–text samples for the decoder.Source: §3.1 · p. 4

3.2. From Patches to Regions

  • A region is a set of patches whose embeddings are averaged, so existing regional tasks differ in patch selection rather than requiring task-specific model changes.For traces, mouse points map to patches whose embeddings are averaged; this supports free-form, user-specified regions and localized descriptions.Source: §3.2 · pp. 4–5

4. Experiments

Experiments find that DINOv2-based backbones, especially Talk2DINO, provide the strongest patch features in the tested framework. The resulting patch-centric models outperform baselines on fine-grained trace and dense captioning, achieve state-of-the-art region-set results, and remain competitive on whole-image captioning.

4.1. Datasets and Metrics

  • Evaluation covers trace, dense, region-set, and image captioning using CIDEr and RefPAC, with mAP for dense captioning and CLIP-Score for image captioning.Trace captioning uses cleaned COCO and Flickr30K subsets; dense captioning uses ground-truth boxes, omitting box proposal.Source: §4.1 · pp. 5–6

4.2. Backbone Selection

  • DINOv2-based models perform best in backbone selection, with Talk2DINO the most effective encoder; standard CLIP performs poorly and CLIP adaptations improve results.The authors attribute the pattern to the importance of semantically rich patch-level features.Source: §4.2 · p. 6

4.3. Comparison with SOTA

  • Patch-centric models significantly outperform all baselines across metrics on trace and dense captioning, including crop-based adaptations and region-supervised AlphaCLIP.For dense captioning, crop-based adaptations provide stronger baselines than applying the same models to global CLS representations.Source: §4.3 · p. 7
  • Patch aggregation achieves state-of-the-art region-set captioning, outperforming zero-shot and region-supervised models while producing coherent captions from arbitrary, potentially disjoint regions.The passage notes a narrower gap because region-set captions align more closely with image-level captioning than strictly localized descriptions.Source: §4.3 · p. 7
  • On whole-image captioning, patch-centric models remain competitive with strong zero-shot captioners but trail dedicated image-centric models such as MERCap and EntroCap.Attention-based weighting is reported when helpful for this task; structured external knowledge or filtered retrieval improves fluency and informativeness for noise-based decoders.Source: §4.3 · p. 7

5. Conclusions

The paper concludes that patch-centric zero-shot captioning supports individual patches and arbitrary region aggregations, with strong results on regional tasks. It also identifies limits from fixed patch context, modality-gap noise, and a remaining performance gap against fully supervised methods.

5. Conclusions

  • The framework sets the state of the art on regional captioning tasks, including dense, region-based, and newly proposed trace captioning.Table 2 compares Talk2DINO Patch-ioner with zero-shot methods across trace, dense, region-set, and image captioning.Source: §5 · p. 7, Table 2 · p. 8
  • Patch-centric captioning covers individual patches and arbitrary aggregations without region or image supervision, and one backbone pass can support multiple regions.The conclusion describes this flexibility as facilitating viability in interactive applications.Source: §5 · pp. 7–8
  • Figure 2 contrasts captions from image-level and CLIP-based regional captioners with Talk2DINO across tasks from finer to coarser granularity.The caption identifies DeCap, DeCap (Crop), and CLIP + Mem. as the compared alternatives.Source: Figure 2 · p. 8
  • Patch context is fixed by the backbone rather than adapted to user intent, and modality-gap noise can cause hallucinations.The authors propose improving patch-level semantics or refining the patch-to-text projection as future work.Source: §5 · p. 8

6. Implementation Details

Implementation uses text-trained GPT-2-style decoders and several vision–language backbones, including CLIP variants and DINO-based models. The appendix specifies training settings and backbone characteristics.

6. Implementation Details

  • The memory-based configuration trains a four-layer, four-head prefix GPT-2-style decoder on COCO training captions, which also form an approximately 500k-text memory bank.Training uses 10 epochs, learning rate 10^-5, batch size 64, and AdamW with weight decay 0.01.Source: §6 · p. 13
  • External knowledge-based models train the same GPT-2-style decoder for 15 epochs with batch size 80 and learning rate 2 × 10^-5.The setup also uses Gaussian noise variance 16 × 10^-3 to replicate the cited experimental settings.Source: §6 · p. 13

7. Backbone Details

  • The tested backbones range from CLIP and dense-task adaptations to SigLIP2, DINO.txt, and Talk2DINO, with different patch resolutions and approaches to language-aligned patch features.SigLIP2 applies its MAP head to each patch token; DINO.txt aligns global and patch-level representations with language, while Talk2DINO maps CLIP text embeddings into DINOv2’s patch space.Source: §7 · p. 13

8. Comparison with LMMs

The comparison evaluates Patch-ioner against supervised LMMs adapted to regional tasks, finding its strongest relative advantages on fine-grained regions and broader region sets, while LMMs generally lead on whole-image captioning. It also contrasts their training requirements, model size, and inference costs.

8.1. LMM Adaptation for Regional Tasks

  • LMMs are adapted to regional tasks through visual prompting with marked images or cropping around the target region.Visual prompting retains the full image context; cropping provides a tight region crop.Source: §8.1 · pp. 13–14

8.2. Results Analysis

  • Patch-ioner outperforms LMMs on fine-grained Dense and Trace captioning and on Region-Set captioning, while LMMs generally perform better on whole-image captioning.The comparison spans regional captioning tasks as well as global image captioning.Source: Table 3 · p. 14, §8.2 · p. 14

8.3. Comparison Highlights

  • Patch-ioner is engineered for multi-granularity regional captioning, while LMMs primarily target global image understanding.Adapting LMMs through prompting or cropping limits their precision on fine-grained regional tasks.Source: §8.3 · p. 14
  • Patch-ioner uses a strict zero-shot setup, training its text decoder on text alone rather than paired image-text data.The compared LMMs are trained on massive image-text pair datasets.Source: §8 · p. 13, §8.3 · p. 14
  • Talk2DINO uses 0.21B parameters, compared with 3B to 4B parameters for the tested LMMs.Patch-ioner extracts patch features once per image and reuses them across regions, whereas adapted LMMs run inference for each annotation.Source: §8.3 · pp. 14–15

9. Ablations on Text Decoder Architectures

The section reports that ReLaion text training improves Patch-ioner over COCO-trained baselines on most tasks, with benefits that grow with subset size. Decoder ablations find no consistent gains from larger text decoders.

9. Ablations on Text Decoder Architectures

  • Increasing text-decoder capacity does not yield consistent gains; GPT-2 small performs strongly on Trace and Region-Set CIDEr, while Qwen3 1.7B has the lowest CIDEr across tasks.The tested decoders were trained on the COCO train Karpathy split, containing 566,747 captions.Source: §9 · p. 15

10.1. Selected dataset: ReLaion 600M

  • The ReLaion ablation uses its higher-quality “best caption” field and matches COCO training steps with a 5.6M-sample subset.The 28.3M-sample subset corresponds to five times as many training steps.Source: §10.1 · p. 15

10.2. Discussion

  • A single ReLaion epoch improves Trace and Dense results, while the 28.3M-sample subset further boosts Region-Set CIDEr.Moving from COCO to ReLaion 5.6M yields +2.4 CIDEr on Trace and +1.7 on Dense.Source: §10.2 · pp. 15–17
  • Using 500k ReLaion captions as the projection memory bank reduces performance across many tasks compared with the COCO memory bank.The authors suggest COCO memory better matches COCO ground-truth captions, particularly syntactically.Source: §10.2 · p. 17
  • ReLaion-trained Patch-ioner outperforms COCO-trained baselines on most tasks, with advantages that grow as the ReLaion subset increases.The reported setup includes training the GPT-2 decoder on different datasets and evaluating CIDEr and RefPAC-S across four captioning tasks.Source: §10.2 · p. 17

11. Additional Results: Aggregation Strategies, Input Resolution, Text Collection

These additional experiments compare patch aggregation, input resolution, and text-memory choices. Results vary by task: simple averaging works best for Trace, attention weighting for image captioning, while weighting choices have little effect in Dense and Region-Set captioning.

Input Resolution

  • The experiments vary patch aggregation functions and image resolution, including 518x518 Talk2DINO inputs that yield 37 patches per side.The tested CLIP configurations include 224x224 inputs with 7 or 14 patches per side and 592x592 inputs with 37 per side.Source: §11 · p. 17

Trace Captioning

  • For Trace captioning, simple averaging of trace patches performs best among the framework’s tested aggregation strategies.Gaussian weighting is omitted because sparse, discontinuous traces do not define the rectangular region it requires.Source: §11 · p. 17

Dense Captioning and Region-Set Captioning

  • Changing the weighting strategy has little effect on Patch-ioner performance in Dense or Region-Set captioning.In Dense captioning, crop-based region baselines are the strongest baselines; the Region-Set gap between zero-shot image captioners and Patch-ioner narrows.Source: §11 · pp. 17–18

Image Captioning

  • Attention-weighted patch aggregation is the most effective tested strategy for standard image captioning.This agrees with Talk2DINO’s suggested use of attention-weighted patch means for global tasks.Source: §11 · p. 18

Memory Bank

  • Adding ground-truth test captions to the memory bank improves performance by only +0.5%, suggesting robustness to memory-bank text selection.The comparison treats this configuration as an approximate upper bound when varying the memory text collection.Source: §11 · p. 18

12. Modality Gap: Projection to Textual Space vs Training with Noise

The comparison finds that both memory-based projection and noise-trained decoders outperform a baseline with no modality-gap mitigation. Memory is selected as the default because it performs better on Dense and Region-Set captioning and is more stable across settings.

Training with Noise

  • The comparison tests a text-memory projection against decoder training with Gaussian noise added to textual embeddings at σ2 = 0.08.The decoder input space is Talk2DINO textual space aligned to DINOv2 with registers.Source: §12 · pp. 18–19

Modality Gap Mitigation

  • Memory outperforms Noise on Dense and Region-Set captioning, while Trace metrics are mixed and Image Captioning results vary by dataset and configuration.On Trace, Memory is slightly better on RefPAC-S while Noise is marginally better on CIDEr, ROUGE-L, METEOR, and BLEU@4.Source: §12 · p. 20
  • Both Memory and Noise outperform the no-mitigation baseline across the reported tasks.The table compares Memory-based Projection, a Noise-trained Decoder, and a baseline without mitigation.Source: §12 · p. 20, Table 10 · p. 21

Chosen Strategy

  • Memory is the default modality-gap strategy because it performs better on Dense and Region-Set tasks and is stable across evaluation settings.Noise remains competitive across multiple tasks.Source: §12 · pp. 20–21

13. Trace Captioning Benchmark Generation

The benchmark converts Localized Narratives mouse traces and speech transcriptions into concise captions for the specific image areas those traces cover. It aligns traces with individual sentences, trims transitions, and rephrases captions to focus on objective regional descriptions.

13. Trace Captioning Benchmark Generation

  • The benchmark is built from Localized Narratives, which pairs mouse traces with speech transcriptions describing image objects as annotators move the pointer over them.The target is one concise caption for the area covered by a trace, not a temporally ordered multi-sentence narrative.Source: §13 · p. 21
  • Descriptions are split into individual sentences and their traces aligned; each trace is trimmed by removing its first and last 15% of points.This removes intermediate periods associated with transitions between image regions.Source: §13 · p. 21
  • Llama3 8B rephrases captions into concise, objective descriptions of the traced region and removes vague or subjective phrasing.Irrelevant sentences are discarded along with their associated traces.Source: §13 · p. 21
  • Figure 5 shows the full prompt used to guide the Llama model in refining and cleaning descriptions, while Figure 6 illustrates initial narrative samples transformed into final trace-captioning samples through trace splitting and caption rephrasing.Source: §13 · p. 21

14. Learned Patch Aggregation via Attention

The learned aggregator improves results on tasks matching its training granularity, but those gains do not consistently transfer across region granularities. The paper therefore favors fixed, parameter-free aggregation for a unified zero-shot framework.

14. Learned Patch Aggregation via Attention

  • The default region representation is the mean of the patch embeddings selected for that region.This is the fixed aggregation strategy used in the main paper.Source: §14 · p. 21
  • The learned alternative uses a single transformer-style attention layer to summarize the region's patch embeddings with a [CLS] token.The attention block processes the patch set jointly and outputs the aggregated region information.Source: §14 · p. 21
  • The learned variant adds an attention-based summary to the mean representation, with α initialized to 0.1 so aggregation begins close to the mean.The learnable scalar α controls the attention summary's contribution.Source: §14 · p. 21
  • The attention aggregator is trained for one epoch with region-level supervision on the COCO Trace Captioning training set, while the backbone and decoder remain frozen.The attention layer and α are optimized with AdamW at learning rate 10^-4.Source: §14 · p. 22
  • Learned aggregation yields substantial gains on trace and dense captioning, while fixed aggregation is more balanced on region-set and image captioning.Table 11 reports the learned-versus-fixed aggregation experiment; the text does not provide numerical values here.Source: Table 11 · p. 23, §14 · p. 23
  • With no region-level annotations assumed, parameter-free aggregation remains preferable for robustness across captioning granularities.The authors attribute this preference to the learned aggregator's gains not consistently transferring to tasks with different training granularities.Source: §14 · p. 23

15. More Qualitative Results

The qualitative results compare model predictions and baselines across captioning tasks from finer to coarser regions. The examples also clarify that region-set captions can track image-level annotations rather than localized content.

15. More Qualitative Results

  • The first rows include single-patch captioning examples, for which the paper reports no quantitative results because annotated data are unavailable..Source: §15 · p. 23
  • Region-set captions tend to align more with image-level captioning than with localized regions because COCO Entities ground-truth captions originate from COCO image-level annotations [10].Source: §15 · p. 23
  • Figures 7 and 8 compare four model predictions with DeCap, DeCap (Crop), ZeroCap, and CLOSE from finer to coarser tasks.Trace times are color-coded from red at the start to yellow at the end; GT denotes the ground-truth caption.Source: Figure 7 · p. 24, Figure 8 · p. 25
Loading 2510.02898v5…