PaperlayerView exact revision on arXiv

Source-linked AI summary

A Glance Is All You Need: Single-Pass Fine-Grained Image Captioning with SimLoss

Suryaansh Jain, Rahasya Barkur, Vishal G, Ryan Rossi, Franck Dernoncourt, Jack Wang, Koustava Goswami, Nedim Lipka, Puneet Mathur, Samyadeep Basu, Seunghyun Yoon

arXiv:2609.00591v1cs.CV

TL;DR

Fine-grained captioning demands details that fluent VLMs often omit, while multi-stage systems recover them with higher latency and unreliable caption targets. SimLoss provides reference-free embedding-space supervision before decoding, and SimLoss FFT nearly matches CapMAS in F1 while achieving the highest precision with single-pass inference; SimLoss GRPO achieves the highest recall.

  • Problem

    Fluent VLM captions often omit visually specific details, while ordinary and pipeline-generated captions provide limited or unreliable supervision for fine-grained adaptation.

  • Method

    SimLoss aligns a trainable VLM’s projected hidden-state representation with a frozen image embedding using contrastive supervision before text decoding.

  • Results

    SimLoss FFT achieves the highest precision, reaches 0.7023 F1 versus CapMAS’s 0.7025, and retains single-pass inference, while SimLoss GRPO achieves the highest recall.

  • Takeaways & Limitations

    Embedding-space supervision can recover nearly all of CapMAS’s F1 benefit without ground-truth or pipeline-generated caption targets and at single-pass latency.

  • Takeaways & Limitations

    SimLoss depends on the frozen embedding teacher, whose biases and visual granularity determine which distinctions receive reward.

Abstract

from arXiv · show

An image may be worth a thousand words, but most captioning models describe it in only a few. Modern vision-language models produce fluent high-level captions, yet routinely miss the attributes, counts, textures, materials, and spatial relations that make an image visually specific. Recent multi-stage systems recover some of these details through generation, decomposition, verification, and rewriting, but they do so at the expense of substantially higher inference latency. We propose SimLoss, a reference-free embedding-space objective for single-pass fine-grained image captioning. SimLoss trains a vision-language model to align its projected hidden-state representation with a frozen image embedding through an InfoNCE contrastive loss, supplying a dense visual supervision signal before any text is decoded, and requiring neither human-written fine-grained captions nor pseudo-captions from a multi-stage pipeline. We instantiate it as SimLoss FFT, which backpropagates through a locally available embedding model, and SimLoss GRPO, which treats that model as a black-box reward. Compared with single-pass, multi-stage verification, reward-optimized, and perception-aware baselines, the fully differentiable fine-tuning variant, SimLoss FFT, achieves the highest precision while nearly matching the F1 score of the multi-stage method, all while retaining single-pass inference and running roughly 20 times faster than the multi-stage pipeline. The reward-based variant SimLoss GRPO attains the strongest recall. Together, these results show that embedding-space supervision can recover the quality of multi-stage verification at the latency of a single-pass captioner.

1 Introduction

Fine-grained captioning seeks visually specific details that fluent VLMs often omit, while multi-stage methods recover coverage at substantial inference cost. SimLoss addresses this trade-off with reference-free embedding supervision for single-pass captioning.

  • Fine-grained captions preserve attributes, materials, counts, textures, and spatial relationships that generic captions often omit.
  • VLMs can produce broadly correct captions that remain visually incomplete by omitting distinctive object details.
  • Detailed grounded captions matter for assistive technology, embodied robotics, and clinical image interpretation.
  • Multi-stage systems generate, decompose, verify, and rewrite captions, improving factuality or coverage but increasing inference cost.
  • SimLoss aligns projected VLM hidden states with frozen image embeddings before decoding, without human-written fine-grained captions or pseudo-captions.
  • SimLoss FFT is the most precise evaluated method and nearly matches CapMAS in F1 at roughly 20× lower latency.

2 Related Work

Related work spans controllable and hyper-detailed captioning, inference-time verification, similarity-based supervision, and contrastive objectives. SimLoss differs by applying contrastive alignment between a frozen embedding model and a captioner’s hidden state.

  • Earlier controllable captioning methods targeted attributes, relations, and scene structure beyond salient objects.
  • Hyper-detailed benchmarks show that strong VLMs remain incomplete or inconsistent when generating dense visual descriptions.
  • CapMAS and related methods add decomposition, verification, rewriting, detection, or visual question-answering stages to improve caption factuality or coverage.
  • Image-text similarity and reward methods provide reference-free supervision, but naive optimization can reduce faithfulness and increase hallucination.
  • SimLoss uses InfoNCE between a frozen embedding model and a captioner’s pooled hidden state, shaping generation rather than a retrieval encoder.

3 Problem and Motivation

Fine-grained captioning requires coverage of discriminative visual evidence while avoiding unsupported details, but conventional captions are too coarse and incomplete as adaptation targets. SimLoss uses frozen image embeddings as a continuous proxy for preserving image information before decoding.

  • 3.1 Detailed Captioning Requires Coverage and Grounding: Fine-grained captions should preserve visible attributes, counts, textures, materials, object parts, and spatial relations while maintaining factual grounding.
  • 3.2 Caption Sources Differ in Detail and Reliability: MS COCO captions support generic scene description but are intentionally concise, whereas IIW evaluates much richer descriptive content.
  • 3.2 Caption Sources Differ in Detail and Reliability: 186.0-word CapMAS captions average 29.3 atomic propositions, including 7.0 judged false, giving a mean factuality ratio of 0.766.
  • 3.4 Captions as Noisy Channels: SimLoss treats captioning as a lossy textual channel and aligns continuous representations to retain information that distinguishes the source image from alternatives.
  • 3.4 Captions as Noisy Channels: Cross-entropy can reward imitation of a short reference caption even when it omits visually present details.
  • 3.5 Embedding-Space Similarity Across Caption Sources: SimLoss adapts using COCO images while discarding their captions and avoiding IIW descriptions or CapMAS captions as adaptation targets.
  • 3.5 Embedding-Space Similarity Across Caption Sources: In the COCO diagnostic, image-to-human-caption similarity is 0.4794, compared with 0.6982 for image-to-baseline-generation similarity.

4 Method

SimLoss trains a captioner by contrastively aligning projected hidden-state representations with frozen image embeddings, then removes the auxiliary components for single-pass inference. FFT supports differentiable training, while GRPO extends the objective to black-box embedding models.

  • 4.1 SimLoss: Embedding-Space Distillation: SimLoss uses a frozen image encoder, pooled VLM hidden states, a learned projector, and InfoNCE to align each image with its matching representation against in-batch negatives.
  • 4.1 SimLoss: Embedding-Space Distillation: At inference, the embedding encoder, projector, and contrastive loss are removed, leaving single-pass generation.
  • 4.1 SimLoss: Embedding-Space Distillation: The objective encourages representations to retain attributes, counts, textures, materials, and spatial relations that distinguish the source image.
  • 4.2 SimLoss Fully Differentiable Fine-Tuning (FFT): SimLoss FFT backpropagates through the projector into LoRA adapters while keeping the base model and embedding encoder frozen.
  • 4.3 SimLoss GRPO: The black-box setting motivates SimLoss GRPO, which samples captions, scores image-caption embedding similarity, and updates policy likelihoods using relative group advantages.
  • Baselines: FeedQuill instead uses PPO with a composite reward combining judge-based F1, global image-text alignment, and pseudoreference similarity.
  • Baselines: SimLoss FFT uses representation matching rather than caption references or judge-model rewards.

5 Baselines

The evaluation compares SimLoss with plain captioning, inference-time verification, reward-based reinforcement learning, and perception-aware policy optimization. These baselines test whether fine-grained captioning improves through external verification, reward optimization, or explicit visual perturbation.

  • The baseline families comprise plain zero-shot captioning, inference-time verification, reward-based RL, and perception-aware policy optimization.
  • The plain baseline is the unadapted Qwen2.5-VL-7B captioner operating without multi-stage verification or additional training.
  • CapMAS samples, merges, decomposes, verifies, and rewrites captions across five inference stages.
  • FeedQuill decomposes captions into atomic information units and optimizes their judge-derived precision–recall F1 reward with PPO.
  • PAPO adds a perception loss based on KL divergence between outputs for original and randomly masked images, with an object-aware masking extension evaluated here.

6 Experiments

Experiments train SimLoss on COCO without caption targets and evaluate fine-grained captioning on IIW-400. SimLoss FFT leads precision and nearly matches CapMAS in F1 with single-pass inference, while GRPO leads recall; FFT is about 20× faster than CapMAS.

  • Training and setup: SimLoss training discards COCO captions and uses images, prompts, and embedding-based adaptation signals instead.
  • Training and setup: All trainable methods use 7B-class Qwen vision-language models with LoRA; FFT uses Qwen3-VL-Embed during training, whereas GRPO uses Gemini 2 Embed as a black-box reward.
  • Evaluation: IIW-400 evaluation reports CLAIR, precision, recall, F1, caption length, and matched A100 latency per image.
  • Main results: 0.8485 precision is highest for SimLoss FFT, while its 0.7023 F1 nearly matches CapMAS’s 0.7025.SimLoss FFT exceeds CapMAS precision of 0.8467 by a small margin.
  • Main results: SimLoss FFT improves precision from 0.7884 to 0.8485 while reducing mean caption length from 347 to 114 words.Its caption-length standard deviation is 13 versus 46 for the baseline, while recall remains approximately flat.
  • Main results: 0.6015 recall is highest for SimLoss GRPO, but its 0.8227 precision and 0.6949 F1 are below SimLoss FFT.
  • Latency and quality–latency tradeoff: 5.77 seconds per image gives SimLoss FFT a 20.0× measured speedup over CapMAS at 115.31 seconds, with nearly the same F1.FFT requires one deployed VLM pass because its embedding encoder and projector are training-only.
  • Qualitative analysis: Qualitatively, SimLoss more consistently captures scene depth, paired objects, visible text, and materials with less repetition, but both variants mistake an abstract driftwood moose for a deer.

7 Conclusion

SimLoss adapts a VLM by aligning projected hidden states with frozen image embeddings before generation, without ground-truth caption targets. FFT delivers the highest precision and nearly matches CapMAS in F1 with single-pass inference, while GRPO delivers the highest recall; the remaining gap suggests explicit verification remains useful.

  • SimLoss aligns projected hidden-state representations with frozen image embeddings, adapting a VLM without ground-truth caption targets.
  • SimLoss FFT achieves the highest precision among evaluated methods and nearly matches CapMAS in F1 while retaining single-pass inference.
  • The remaining gap to CapMAS suggests explicit verification is still useful for balancing precision and recall.

A Analysis

SimLoss explains fine-grained captioning performance through the supervision signal used during training and inference. SimLoss FFT offers nearly CapMAS-level F1 with stronger precision and shorter captions, while GRPO favors recall.

  • Mechanistic analysis: SimLoss FFT directly aligns hidden representations with a frozen image embedding, providing dense supervision before generation.This avoids copying pseudo-reference wording or relying on sampled-text updates.
  • Mechanistic analysis: 0.6015 recall is achieved by SimLoss GRPO, whose black-box reward encourages broader visual coverage but reduces precision relative to FFT.Because the reward is applied after discrete generation, optimization is less direct.
  • Quality–latency trade-offs: 0.7025 for CapMAS versus 0.7023 for SimLoss FFT, leaving the single-pass model nearly tied in F1.CapMAS retains a marginal F1 advantage, while SimLoss FFT occupies a more favorable quality-latency point.
  • Baseline comparisons: SimLoss FFT provides a direct differentiable alignment objective, unlike judge-based reward methods whose signals are noisy or indirect for fine-grained visual recall.PAPO and object-aware masking can also depend on model variants, masking strategy, incomplete detections, or lost context.
  • Limitations of competing supervision: LLM-judge supervision is constrained by agreeableness bias: true-positive rates approach 96%, while true-negative rates remain below 25%.This weakness affects both proposition verification and PPO reward supervision.
  • Conciseness as efficient visual encoding: SimLoss FFT averages 114 words, compared with 347 for the plain baseline and 189 for CapMAS, while achieving the highest precision and near-best F1.The comparison indicates that quality is not explained by generating longer captions.

B Qualitative Analysis

The qualitative analysis finds that SimLoss captions recover fine-grained object, material, text, and scene-structure details that baselines often omit, while sometimes over-specifying uncertain categories.

  • Scene structure: SimLoss FFT and GRPO organize the mountain image into foreground, middle-ground, and background layers, covering more spatial questions than baselines.The layered structure is identified as the main driver of coverage gain on this image.
  • Scene structure: SimLoss FFT records ground-level evidence beneath the flowering tree—green grass and fallen leaves—that no baseline captures.PAPO and PAPO+YOLO identify dogwood flowers, but omit the ground.
  • Reading visible text: SimLoss variants correctly transcribe “MILSPED AML,” while baselines either omit the label or misread and expand it without evidence.SimLoss GRPO additionally identifies the platform’s tactile paving strip.
  • Specificity and coverage: SimLoss FFT gives both mosaic hats symmetric, detailed descriptions and identifies the multicolored tile border that other methods omit.It notes that both hats have red bands and red decorative objects on top.
  • Specificity of object description: SimLoss FFT names medieval garments precisely, including a chainmail hauberk, red surcoat, white sash, gauntlets, and boots.No baseline reaches this terminology level of specificity.
  • Material specificity: SimLoss FFT identifies pine needles and small pine cones on the decaying log, while baselines hallucinate moss, twigs, or hedge about the material.SimLoss GRPO further distinguishes dry light-brown needles from green ones.
  • Specificity and errors: On the car image, SimLoss methods venture Porsche identifications, illustrating that added specificity can improve coverage but creates fine-category risk.On the driftwood sculpture, both methods describe details but misclassify a moose as a deer, unlike conservative baselines.
  • Precision and hallucination: On one mountain image, SimLoss FFT reaches 100% measured precision versus 96.3% for Plain VLM and 85.2% for CapMAS, though over-specification remains possible.The passage characterizes this improvement as a tendency rather than a guarantee.

C Caption Length versus Quality

Caption-length comparisons show that SimLoss FFT can preserve coverage with shorter outputs, while longer baseline captions add repetition, speculation, and imposed structure rather than answerable content.

  • Caption length and quality: SimLoss FFT produces the shortest captions while achieving the highest precision, indicating that brevity and factuality need not conflict.Figures 11 and 12 compare cases where coverage is identical but caption length and quality differ.
  • Figure 11: 100% precision with 123 words matches the 20/23 coverage of the other methods, versus 90.9% precision for Plain VLM.The baseline uses nearly a third of its words for repetition and unsupported setting speculation.
  • Figure 12: 100% precision with 106 words accompanies identical 18/24 coverage, compared with 94.1% for Plain VLM and 89.7% for CapMAS.The longer baselines add summaries, speculation, and imposed structure without measured coverage gain.

D Prompts

The prompts section specifies the captioning, evaluation, and CapMAS inference procedures, including atomic-fact precision, caption-only recall, and holistic CLAIR scoring.

  • D Prompts: The prompt appendix includes runtime substitutions such as {caption}, {reference_caption}, {propositions}, and literal newline markers.It groups prompts for captioning, evaluation, CapMAS, and reward or judge procedures.
  • D.1 Captioning Prompts: All methods use the single-pass captioning prompt “Describe this image in detail,” while CapMAS first samples five diverse detailed-description prompts.The five prompts vary wording but all request detailed image descriptions.
  • D.2 Evaluation Protocol and Prompts: Precision decomposes a candidate caption into atomic propositions and uses a judge to mark each proposition supported or unsupported by the image and reference.The displayed precision equation averages the binary judgments over extracted propositions.
  • D.2 Evaluation Protocol and Prompts: F1 is reported as the harmonic mean of precision and recall, while CLAIR rates candidate-reference same-image likelihood from 0 to 100.Table reporting divides CLAIR by 100, and references and questions are used only for evaluation.
  • D.2 Evaluation Protocol and Prompts: The evaluation prompt extracts every caption component into atomic propositions before proposition-level verification.The verification stage judges each extracted proposition True or False against the image and reference caption.
  • D.2 Evaluation Protocol and Prompts: Recall answers human-verified, image-derived multiple-choice questions from the caption alone, withholding the image and returning “I don’t know” when necessary.The protocol measures recoverable information without allowing outside knowledge or question-based inference.
  • D.3 CapMAS Inference-Pipeline Prompts: CapMAS samples five captions, merges them into one comprehensive description, then decomposes the result into propositions for later fact-checking and rewriting.The merge prompt requests important details without repetition and restricts content to what the captions mention.
Loading 2609.00591v1…