PaperlayerView exact revision on arXiv

Source-linked AI summary

Controllable Image Captioning with Prompt-Conditioned Scene Rewards

Jongyeop Hyun, Taeyoung Kim, Hyounghun Kim

arXiv:2609.00709v1cs.CVcs.CLcs.LG

TL;DR

LVLMs generate fluent captions but offer limited reliable control over semantic emphasis. FOCUS uses natural-language prompts to weight scene-graph-aligned rewards during caption optimization, and SCOPE evaluates contrastive controllability. Experiments on two VLM backbones show improved controllability and fine-grained caption quality without architectural modifications.

  • Problem

    LVLM captioners offer limited reliable control over whether descriptions emphasize attributes, relations, or particular image regions.

  • Method

    FOCUS parses captions into scene-graph components and uses prompt-conditioned signed weights to optimize requested content and suppress off-scope content with GRPO.

  • Results

    Experiments on two VLM backbones show improved controllability and fine-grained caption quality without architectural modifications.

  • Takeaways & Limitations

    SCOPE provides contrastive Include/Avoid evaluation, while prompt-conditioned scene-graph control is presented as a practical direction for controllable image captioning.

  • Takeaways & Limitations

    FOCUS adds computational overhead through scene-graph parsing and LLM-based evaluation, and its scalability to significantly larger models remains unvalidated.

Abstract

from arXiv · show

Large Vision-Language Models produce fluent image descriptions but offer limited semantic control: users cannot reliably specify whether captions should emphasize attributes, relations, or particular image regions. We present Fine-grained Captioning Control Using Scene Rewards (FoCUS), a controllable image captioning method that lets users steer captions toward specific semantic emphases through natural-language control prompts. The core idea is a prompt-conditioned control objective based on scene-graph-aligned component scores. Generated captions are parsed and aligned to scene-graph components such as objects, attributes, and relations. These components are differentially weighted, including negative weights, according to the requested emphasis. We optimize this objective with GRPO and further improve its reliability through a stricter object validity threshold and reasoning-based verification for attribute and relation scoring. To evaluate controllability, we introduce Semantic Control and Precision Evaluation (SCoPE), a benchmark with contrastive Include/Avoid constraints for measuring both target content coverage and out-of-scope suppression. Experiments on two VLM backbones show that FoCUS consistently improves controllability and fine-grained caption quality without degrading general caption performance.

1 Introduction

FOCUS addresses the limited reliability of natural-language control in image captioning by using prompt-conditioned scene-graph rewards to emphasize requested semantics and suppress off-scope content. It also introduces SCOPE to evaluate this contrastive controllability.

  • Motivation: Large Vision-Language Models often produce fluent descriptions but provide limited, unreliable control over whether captions emphasize attributes, relations, or image regions.Zero-shot prompting can default to generic scene descriptions instead of the requested semantic focus.
  • Motivation: Natural-language prompting is a more natural interface than structured controls, but models may drift toward generic high-probability content.Prior methods commonly use length tokens, regions or boxes, or formal semantic graphs at inference time.
  • Motivation: Scene graphs provide interpretable objects, attributes, and relations, but prior work mainly used them for post-hoc evaluation or prompt-agnostic training.FOCUS instead uses these components as a prompt-conditioned control signal.
  • FOCUS: FOCUS aggregates scene-graph component scores with prompt-specific signed weights, rewarding requested content and penalizing off-scope content without architectural changes or structured side inputs.The objective is optimized with GRPO, alongside stricter object-validity checks and reasoning-based verification for attributes and relations.
  • Evaluation: SCOPE evaluates controllable captioning with category-specific Include and Avoid constraints that measure target coverage, off-scope suppression, and factual consistency.Its contrastive design addresses a gap in standard overall-quality and component-wise captioning evaluations.

2 Related Work

Related work shows that controllable captioning has relied mainly on structured inference controls, while scene-graph feedback and caption benchmarks have not fully addressed natural-language, contrastive semantic control. FOCUS extends scene-graph rewards with prompt-conditioned signed weighting and SCOPE measures both requested-content coverage and off-scope suppression.

  • Large Vision-Language Models and Detailed Captioning: Modern LVLMs produce fluent, detailed captions but typically default to a single “best overall” description with limited control over semantic emphasis.Relevant emphases include attributes, relations, and foreground or background content.
  • Controllable Image Captioning: Prior controllable captioning methods commonly use structured inference controls such as length signals, regions, boxes, masks, or formal semantic specifications.FOCUS instead uses natural-language prompts without additional structured inputs at inference time or architectural changes.
  • Training Objectives for Caption Alignment: Captioning objectives and metrics have incorporated scene-graph components for finer object-, attribute-, and relation-level feedback.Examples include SPICE, CAPTURE, CompreCap, and compositional supervision in SC-Captioner.
  • Training Objectives for Caption Alignment: FOCUS extends scene-graph supervision by combining object, attribute, and relation scores with prompt-specific signed weights.This formulation targets user-specified semantic emphases.
  • Captioning Benchmarks and Fine-grained Evaluation: SCOPE introduces category-specific Include and Avoid constraints to measure coverage and suppression, addressing the lack of explicit contrastive controllability tests.These measures correspond to maximizing requested content while suppressing off-scope content.

3 Method

FOCUS trains controllable captions with a prompt-conditioned objective that scores scene-graph objects, attributes, and relations, rewarding requested emphasis while penalizing off-scope content. It improves reward reliability through stricter object matching and reasoning-based verification, then optimizes the objective with SFT followed by GRPO.

  • Prompt-Conditioned Control Objective: FOCUS decomposes generated captions into scene-graph objects, attributes, and relations, then combines their scores with prompt-specific signed weights.Positive weights reward requested content, while negative weights penalize off-scope content.
  • Scene-Graph-Aligned Component Scores: Object scoring matches extracted caption nouns to annotated scene-graph categories using mutual-best Sentence-BERT matches with validity threshold τ = 0.5.The object score is the fraction of ground-truth objects successfully matched.
  • Scene-Graph-Aligned Component Scores: Attribute scoring uses a CoT-based Qwen3-30B-A3B-Instruct judge to assign 0–5 alignment scores for matched objects and averages them.If no objects are matched, the attribute score is set to 0.
  • Scene-Graph-Aligned Component Scores: Relation scoring gathers subcaptions containing relation participants and uses the same CoT-based judge to score directional correctness from 0 to 5.Relations are evaluated over annotated directed relations, with the relation score set to 0 when none exist.
  • Prompt-Conditioned Control Objective: FOCUS supports General, Attribute-, Relation-, Foreground-, and Background-Focused prompts, with signed weights adapting component emphasis and suppression.Foreground and background scores aggregate object, attribute, and relation scores over salient-central or contextual-peripheral object subsets.
  • Optimization: The prompt-conditioned reward is optimized with a two-stage SFT+GRPO pipeline using group-relative advantages from multiple sampled captions.Training covers all five prompt categories for robustness across control settings.

4 Benchmark: SCOPE

SCOPE evaluates controllable captioning contrastively: captions should cover requested facts while suppressing complementary off-scope facts. Its verified Include/Avoid benchmark supports Coverage, Adherence, Faithfulness, and an aggregated overall score, with strong human alignment.

  • Benchmark Construction: SCOPE addresses the gap in standard captioning evaluation by testing requested-content coverage and off-scope suppression through category-specific Include and Avoid lists.The lists are derived from curated captions and target contrastive controllability.
  • Benchmark Construction: The pipeline constructs SCOPE through category-specific caption generation, automated focus refinement, and atomic-fact extraction from verified captions.Figure 3 presents data construction above the evaluation framework.
  • Benchmark Construction: SCOPE uses 189 manually curated, non-overlapping images from COCO, CompreCap, and DOCCI across Attribute, Relation, Foreground, and Background categories.For each image and focus, facts from the selected category form Include, while complementary-focus facts form Avoid.
  • Evaluation Metrics: Coverage measures recall of Include facts, whereas Adherence measures suppression of detected Avoid-list violations.These metrics separately represent target inclusion and out-of-scope avoidance.
  • Evaluation Metrics: Faithfulness penalizes explicit incompatibilities with target atomic facts, counting mismatched properties but not omissions or lack of mention.The verifier considers incompatible values for properties such as color, number, material, and spatial relations.

5 Experiments

Experiments evaluate two VLM backbones with SFT, GRPO-based baselines, and FOCUS across controllability, fine-grained alignment, and general caption quality. SFT+FOCUS achieves the strongest reported overall and category-wise control, while stronger inference prompts have mixed effects.

  • Experimental Setup: Experiments use Qwen2.5-VL-3B-Instruct and InternVL3-2B, training each with supervised fine-tuning followed by GRPO.SFT+FOCUS results are averaged over three runs.
  • Evaluation Setup: Evaluation combines SCOPE controllability, CompreCap-based component scoring, and reference-based captioning on 5,000 DOCCI test images.The protocols cover Include/Avoid control, fine-grained factual alignment, and standard general-caption metrics.
  • Baselines: Baselines include zero-shot prompting, SFT, SFT+CLIP, and SFT+CompreCap, representing pretrained, cross-entropy, holistic-similarity, and fixed scene-graph reward settings.The two GRPO baselines use CLIP similarity or the original CompreCap metric as rewards.

6 Results and Analysis

FOCUS consistently improves contrastive controllability and fine-grained factual alignment across two LVLM backbones, while preserving general caption quality. Ablations indicate that reasoning-based verification and stronger evaluators provide the largest reward-quality gains, whereas token allocation and matching thresholds also affect performance.

  • Main Results: SFT+FOCUS achieves the best overall performance and strongest category-wise controllability across both LVLM backbones.
  • Main Results: Approximately +16 points on Qwen2.5-VL-3B and +11 points on InternVL3-2B improve overall SCOPE controllability relative to zero-shot prompting.
  • Main Results: Approximately +7–9 points over zero-shot overall improve CompreCap performance across both backbones while strengthening fine-grained factual alignment.
  • Main Results: Explicit prompts modestly help several methods on Qwen2.5-VL-3B but not InternVL3-2B, while SFT+FOCUS remains the best-performing method.
  • Token Efficiency: SFT+FOCUS generates approximately 110 tokens, reducing token count by approximately 34% relative to zero-shot while achieving a 43.8% SCOPE Overall improvement.
  • Component-Scoring Ablation: Enabling all three reward components achieves 36.86 (+7.60), a 26.0% relative improvement over the baseline configuration.
  • Hyperparameter Analysis: Across evaluators, Overall scores span roughly 34.6–36.9, with stronger judges performing better; τ = 0.5 offers the best Overall result across a broad stable range.

7 Conclusion

FOCUS uses prompt-conditioned, scene-graph-aligned rewards to emphasize requested content and suppress off-scope content, and SCOPE evaluates this behavior with contrastive Include/Avoid constraints. Across two VLM backbones, the method improves controllability and fine-grained caption quality without architectural modifications.

  • FOCUS differentially weights scene-graph component scores according to natural-language prompts to emphasize requested content and suppress off-scope content.
  • SCOPE evaluates semantic control through contrastive Include/Avoid constraints.
  • Experiments on two VLM backbones improve controllability and fine-grained caption quality without requiring architectural modifications.

Limitations

The paper identifies computational, scalability, and evaluation limitations for FOCUS and SCOPE. These include training overhead, possible parsing-error propagation, unvalidated scaling to much larger models, and residual LLM-judge bias.

  • FOCUS incurs computational overhead from scene-graph parsing and LLM-based training evaluation, and parsing errors may propagate into the reward signal.
  • The approach has been demonstrated on two VLM backbones, but scalability to significantly larger models remains unvalidated.
  • SCOPE relies on LLM-based evaluation whose biases and inconsistencies may miss subtle semantic distinctions despite reasonable human alignment.

Ethics Statement

The research followed the ACL Ethics Policy and used publicly available datasets, models, images, and automated pipelines. Human validation was conducted through MTurk under standard crowdsourcing guidelines, with no negative ethical outcomes anticipated.

  • The study adhered to the ACL Ethics Policy and used publicly available datasets and vision-language models.
  • SCOPE used publicly available images and automated generation pipelines for benchmark construction.
  • Human validation was conducted through MTurk following standard ethical guidelines for crowdsourced annotation.
  • The authors anticipated no negative ethical outcomes from improving controllability in image captioning systems.

H.4 Robustness to Judge Choice

Robustness analyses indicate that SCOPE’s human alignment and FOCUS’s performance gains are not narrowly dependent on one evaluator or random seed. Results remain stable across judge families, model sizes, fixed evaluators, and repeated runs.

  • All four tested judges remain strongly and significantly correlated with human preferences, with modest variation across evaluator family and size.Pairwise judge agreement is also consistently high, with Spearman’s ρ between 0.64 and 0.73.
  • Qwen3-32B-Thinking has the highest agreement with MTurk preferences and is distinct from the training-time reward judge.
  • SFT+FOCUS achieves the best SCOPE Overall score under three fixed evaluators, including weaker and backbone-mismatched judges.
  • Across three random seeds, the overall-score standard deviation remains below 0.5 for both backbones on both SCOPE and CompreCap.Variation is slightly larger in foreground/background-focused categories but does not affect the main conclusions.

I.4 SCOPE Robustness to Source Composition and Sampling

Additional evaluations test whether FOCUS’s gains depend on source composition, sampling, prompt strength, reward weighting, or penalty magnitude. The results support robust controllability gains while showing that calibrated negative penalties and signed prompt-conditioned weights matter.

  • Source composition and sampling: SCOPE includes 189 images, with 35 from CompreCap and 154 from COCO and DOCCI, none overlapping training images.Source-stratified scores and 95% nonparametric image-bootstrap confidence intervals assess composition and finite-set uncertainty.
  • General-caption quality: SFT+FOCUS largely maintains general-caption quality across two backbones while consistently improving fine-grained factual alignment under the general prompt.For InternVL3-2B, CIDEr and ROUGE-L decrease slightly while CAPTURE remains essentially unchanged.
  • Engineered prompts: SFT+FOCUS achieves the highest Overall score on both backbones and performs best across all four control categories under engineered prompts.
  • Reward-weighting ablation: Both fixed-weight and target-only positive-weighting alternatives underperform FOCUS in Overall score and show less balanced category profiles.They can score higher on Foreground but substantially lag on Attribute, Relation, and Background.
  • Penalty magnitude: Very large negative penalties may produce refusal-like outputs, whereas a zero penalty increases scope drift and off-scope background details.The authors therefore adopt a moderate default penalty rather than an extreme value.

K SCOPE and CompreCap Benchmark for Commercial Models

SCOPE evaluates contrastive controllability through category-specific Include/Avoid constraints, while CompreCap measures component-wise factual alignment. Commercial-model results show that strong factual alignment does not always imply strong controllability.

  • Benchmark protocols: SCOPE measures controllability through category-specific Include and Avoid constraints, while CompreCap evaluates component-wise factual alignment.
  • SCOPE results: Larger commercial LVLM variants generally achieve higher SCOPE controllability scores across model families.GPT-5 rises from Overall 41.20 for nano to 49.74 for the largest setting, while Gemini 2.5 rises from 39.34 for Flash Lite to 55.86 for Pro.
  • CompreCap results: CompreCap scores vary less across commercial model variants than SCOPE scores, indicating lower sensitivity to model size for component-wise alignment.
  • Interpretation: Models that perform well on CompreCap do not always dominate on SCOPE, which explicitly measures off-scope suppression.
  • Qualitative comparison: FOCUS qualitative comparisons show better concentration on requested semantics and suppression of out-of-category content than zero-shot generation.
Loading 2609.00709v1…