PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic-Conditional Diffusion Networks for Image Captioning

Jianjie Luo, Yehao Li, Yingwei Pan, Ting Yao, Jianlin Feng, Hongyang Chao, Tao Mei

arXiv:2212.03099v1cs.CVcs.CLcs.MM

TL;DR

Image captioning diffusion models must capture dependencies among discrete words while aligning captions with visual content. SCD-Net conditions a diffusion process on retrieved image semantics and guides it with self-critical learning; experiments report better performance than state-of-the-art non-autoregressive approaches and a competitive autoregressive Transformer with the same encoder-decoder structure.

  • Problem

    Non-autoregressive captioning can underperform autoregressive methods, with word repetition or omissions when sequential dependencies are under-exploited, while diffusion captioning had also overlooked these issues.

    Source: §1 · pp. 1–2

  • Method

    SCD-Net uses retrieved image-relevant semantics as a prior for cascaded Diffusion Transformers and applies guided self-critical sequence learning to transfer knowledge from a standard autoregressive Transformer.

    Source: §1 · p. 2

  • Results

    SCD-Net outperforms state-of-the-art non-autoregressive approaches and the competitive autoregressive method sharing its Transformer encoder-decoder structure.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The results demonstrate the encouraging potential of diffusion models for image captioning.

    Source: §5 · p. 9

Abstract

from arXiv · show

Recent advances on text-to-image generation have witnessed the rise of diffusion models which act as powerful generative models. Nevertheless, it is not trivial to exploit such latent variable models to capture the dependency among discrete words and meanwhile pursue complex visual-language alignment in image captioning. In this paper, we break the deeply rooted conventions in learning Transformer-based encoder-decoder, and propose a new diffusion model based paradigm tailored for image captioning, namely Semantic-Conditional Diffusion Networks (SCD-Net). Technically, for each input image, we first search the semantically relevant sentences via cross-modal retrieval model to convey the comprehensive semantic information. The rich semantics are further regarded as semantic prior to trigger the learning of Diffusion Transformer, which produces the output sentence in a diffusion process. In SCD-Net, multiple Diffusion Transformer structures are stacked to progressively strengthen the output sentence with better visional-language alignment and linguistical coherence in a cascaded manner. Furthermore, to stabilize the diffusion process, a new self-critical sequence training strategy is designed to guide the learning of SCD-Net with the knowledge of a standard autoregressive Transformer model. Extensive experiments on COCO dataset demonstrate the promising potential of using diffusion models in the challenging image captioning task. Source code is available at \url{https://github.com/YehLi/xmodaler/tree/master/configs/image_caption/scdnet}.

1. Introduction

Autoregressive captioning is effective but computationally costly, while non-autoregressive and diffusion approaches face repetition, omission, and sentence-quality challenges. SCD-Net conditions diffusion on image-relevant semantics and uses guided self-critical learning to improve alignment and coherence.

1. Introduction

  • Autoregressive captioning generates words sequentially with unidirectional textual message passing and computational costs that scale quadratically with sentence length.Non-autoregressive methods instead emit words in parallel, but can underperform when sequential dependencies are under-exploited.Source: §1 · p. 1
  • Conventional diffusion captioning can match basic autoregressive models, but still overlooks word repetition and omission.It represents words as binary bits and uses a continuous diffusion process to denoise noise-augmented sentences.Source: §1 · p. 2
  • SCD-Net conditions reverse diffusion transitions on image semantics to encourage visual-language alignment and reduce omitted semantic words.It retrieves semantically relevant words for each image and incorporates them as a semantic prior.Source: §1 · p. 2
  • Cascaded Diffusion Transformers progressively enhance captions, while guided self-critical sequence learning transfers standard autoregressive Transformer knowledge through sentence-level reward.The strategy is intended to stabilize and boost the diffusion process.Source: §1 · p. 2
  • The paper reports extensive COCO experiments as verification of SCD-Net's potential for image captioning.The contributions include a semantic-conditional diffusion process and guided self-critical sequence learning.Source: §1 · p. 2

2. Related Work

Related work spans autoregressive captioning models and approaches that generate words or groups in parallel. SCD-Net extends diffusion-based non-autoregressive captioning with semantic conditioning and guided sentence-level learning.

2.1. Autoregressive Image Captioning

  • RNN captioners developed from CNN-RNN systems toward attention, semantic attributes, object-level attention, object relations, and scene-graph semantics.Examples include Up-Down, GCN-LSTM, and SGAE.Source: §2.1 · p. 2
  • Transformer captioning work adds spatial relations, persistent memory, mesh-like layer connections, higher-order modal interactions, and parsed-tree structure.The cited approaches include the primary Transformer, Meshed-Memory Transformer, X-Transformer, and APN.Source: §2.1 · pp. 2–3

2.2. Non-Autoregressive Image Captioning

  • Non-autoregressive captioning includes parallel decoding, masked decoding, coarse-to-fine object refinement, semi-autoregressive groups, and diffusion over binary text.Bit Diffusion remains inferior to state-of-the-art autoregressive Transformers, and sentence-level reward optimization for diffusion had not been explored.Source: §2.2 · p. 3
  • SCD-Net extends diffusion-based non-autoregressive captioning with semantic-conditional diffusion and guided self-critical learning using sentence-level reward.The paper presents these additions as strengthening alignment and stabilizing and boosting diffusion.Source: §2.2 · p. 3

3. Method

SCD-Net applies continuous diffusion to caption generation, using retrieved semantic information to condition denoising. It combines Transformer-based caption processing with cascaded refinement and teacher-guided sequence training.

3.1. Problem Formulation

  • Caption words are converted to binary-bit vectors, progressively noised, then reconstructed by reverse latent-state transitions and quantized back to bits.The model is trained to reconstruct the sentence representation from noisy states conditioned on detected visual objects.Source: §3.1 · p. 3

3.2. Diffusion Transformer

  • Each Diffusion Transformer uses a Visual Encoder and mask-free bidirectional Sentence Decoder; decoder word distributions map to bit vectors, with cross-entropy added to bit loss.Figure 2 also shows a Semantic Transformer within each Diffusion Transformer.Source: Figure 2 · p. 4, §3.2 · p. 4

3.3. Semantic Condition

  • A retrieved sentence from the training pool supplies semantic features that condition diffusion, addressing alignment and word omission concerns.The semantic Transformer encodes the current latent state with the retrieved semantic prior before sentence decoding.Source: §3.3 · pp. 4–5

3.4. Cascaded Diffusion Transformer

  • Cascaded Diffusion Transformers progressively refine captions, with later modules conditioned on earlier sentence predictions and their outputs fused at each timestep.The first module uses the semantic condition; subsequent modules take the preceding module’s sentence prediction as an additional cue.Source: §3.4 · p. 5

3.5. Guided Self-Critical Sequence Training

  • Guided self-critical training anchors sampling with an autoregressive teacher caption, addressing multi-step sampling and independent-word reward challenges.After diffusion saturates, the method can replace the teacher caption with a diffusion estimate if its CIDEr score is higher.Source: §3.5 · p. 5

4. Experiments

COCO experiments compare SCD-Net with autoregressive and non-autoregressive captioners and test semantic conditioning, guided training, and cascaded designs. SCD-Net leads the reported comparisons, while ablations examine the contribution of each design.

4.1. Datasets and Experimental Settings

  • The COCO Karpathy split uses 113,287 training, 5,000 validation, and 5,000 test images; the study also reports official test-server results.Source: §4.1 · p. 6, §4 · p. 6

4.2. Performance Comparison

  • 118.0% CIDEr (+4.7%) and 21.6% SPICE (+0.6%) under cross-entropy beat Transformer; guided training also exceeds M² Transformer by 0.4% on CIDEr and SPICE.Source: §4.2 · p. 6, Table 1 · p. 7
  • On the official test server, single-model SCD-Net surpasses single-model baselines across most metrics and is comparable to some autoregressive ensembles.Source: §4.2 · p. 6, Table 2 · p. 7
  • Across five examples, Transformer captions can omit key words and Bit Diffusion can repeat them, while SCD-Net produces more accurate and descriptive captions.Source: §4.2 · p. 7, Figure 3 · p. 8

4.3. Experimental Analysis

  • Semantic conditioning boosts Base; ordinary SCST slightly improves CIDEr and SPICE but lowers other scores, whereas GSCST improves all metrics and cascading yields the best performance.Source: §4.3 · pp. 7–8, Table 3 · p. 8
  • In a single Diffusion Transformer, performance is relatively smooth with 3–4 blocks, drops slightly at 5, and is best at three blocks with less computation.Source: Table 4 · p. 8, §4.3 · p. 8
  • With 1–3 cascaded Diffusion Transformers, performance generally improves; it is less sensitive to the count above one, and the authors select two.Source: Table 5 · p. 8, §4.3 · p. 8

5. Conclusion

The conclusion presents SCD-Net as a diffusion-based image-captioning approach that strengthens visual-language alignment and linguistic coherence. It highlights semantic conditioning, guided self-critical training, and promising results against non-autoregressive and Transformer-based autoregressive methods.

  • SCD-Net empirically outperforms state-of-the-art non-autoregressive approaches and a competitive autoregressive method with the same Transformer encoder-decoder structure.Source: §5 · p. 9
  • Semantic conditioning upgrades diffusion with an additional semantic prior to strengthen visual-language alignment.Source: §5 · p. 9
  • Guided self-critical sequence training is designed to stabilize and boost diffusion for image captioning.Source: §5 · p. 9
  • The authors conclude that these results demonstrate the encouraging potential of diffusion models for image captioning.Source: §5 · p. 9
Loading 2212.03099v1…