Source-linked AI summary
Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner
Tseng-Hung Chen, Yuan-Hong Liao, Ching-Yao Chuang, Wan-Ting Hsu, Jianlong Fu, Min Sun
TL;DR
Cross-domain captioning must bridge image and sentence shifts without paired target-domain training data. The paper uses unpaired target data with adversarial critics and critic-based inference planning, reporting consistent performance across four targets, including a 21.8% CIDEr-D improvement on CUB-200.
Problem
Cross-domain captioning asks how to transfer from paired source-domain data to target domains with image and sentence shifts but no paired target-domain training data.
Source: §1 · p. 1Method
The captioner is adversarially trained with domain and multi-modal critics using unpaired target data, then uses critic-based planning at inference.
Source: §1 · p. 2Results
Across four target datasets, the method performs consistently well; CUB-200 adaptation improves CIDEr-D by 21.8%, and critic use at inference adds a 4.5% boost.
Source: §1 · p. 2Takeaways & Limitations
The method outperforms baselines across four target domains, including two with large and two with regular domain shifts.
Source: §5 · p. 8Direct training on all paired target-domain data performs worse than fine-tuning.
Source: §4.2 · p. 7
Abstract
from arXiv · showhide
Impressive image captioning results are achieved in domains with plenty of training image and sentence pairs (e.g., MSCOCO). However, transferring to a target domain with significant domain shifts but no paired training data (referred to as cross-domain image captioning) remains largely unexplored. We propose a novel adversarial training procedure to leverage unpaired data in the target domain. Two critic networks are introduced to guide the captioner, namely domain critic and multi-modal critic. The domain critic assesses whether the generated sentences are indistinguishable from sentences in the target domain. The multi-modal critic assesses whether an image and its generated sentence are a valid pair. During training, the critics and captioner act as adversaries -- captioner aims to generate indistinguishable sentences, whereas critics aim at distinguishing them. The assessment improves the captioner through policy gradient updates. During inference, we further propose a novel critic-based planning method to select high-quality sentences without additional supervision (e.g., tags). To evaluate, we use MSCOCO as the source domain and four other datasets (CUB-200-2011, Oxford-102, TGIF, and Flickr30k) as the target domains. Our method consistently performs well on all datasets. In particular, on CUB-200-2011, we achieve 21.8% CIDEr-D improvement after adaptation. Utilizing critics during inference further gives another 4.5% boost.
1. Introduction
Cross-domain captioning seeks to transfer from paired source-domain data to target domains without costly target-domain image–sentence pairing, despite shifts in both images and sentence styles. The paper introduces adversarial critics and critic-based inference planning, reporting improvements across four target datasets.
Motivation
- Paired image-sentence data supports strong domain-specific captioning, but collecting such pairs in every domain is costly.The cross-domain setting instead uses paired source data and can avoid paired training data in the target domain.Source: §1 · p. 1
- MSCOCO and CUB-200 differ in both image content and sentence style, with CUB captions describing bird parts in detail rather than mainly location, color, and size.These shifts pose the central challenge of adapting a source-trained captioner to target-domain images.Source: §1 · pp. 1–2
- Prior approaches use unpaired images and web sentences for word-level changes or test-time taggers for word selection, rather than changing overall sentence style.The paper contrasts these approaches with adversarial adaptation using unpaired target-domain images and sentences.Source: §1 · p. 2
Approach
- A domain critic assesses target-domain sentence style, while a multi-modal critic assesses image-caption pairing; policy gradients train the captioner against both.At inference, critic-based planning selects sentences without additional supervision such as tags.Source: §1 · p. 2
Results
- Across four target datasets, the method performs consistently well; CUB-200 adaptation improves CIDEr-D by 21.8%, with critic use at inference adding a 4.5% boost.The paper also reports significant improvement over source-domain-only captioning on four publicly available datasets.Source: §1 · p. 2
2. Related Work
Prior work spans neural visual description, sequence-training methods, learned reward models, and domain adaptation. The paper positions cross-domain captioning as an unaddressed application of adversarial domain adaptation and notes that common metric-based policy-gradient methods require target ground-truth sentences.
Visual description generation
- Captioning commonly encodes images with CNNs and decodes captions with RNNs; spatial attention [35] learns where to look, while video description also uses LSTM encoders [9] [36] [40].The captioner is described as a standard CNN-RNN architecture [33], feeding each predicted word into the next inference step.Source: §2 · p. 2, Figure 2 · p. 3
Addressing exposure bias
Reward modeling
Domain adaptation
- Adversarial domain adaptation has been applied to sentiment analysis [1], feature alignment [10], street-scene segmentation [15], and cross-city road scenes [6], but not cross-domain captioning.Conventional methods also align domains using distance measures such as MMD [25] and CMD [39].Source: §2 · pp. 2–3
3. Cross-domain Image Captioning
Cross-domain captioning uses paired image–sentence data in a source domain and separate unpaired images and sentences in the target domain. The method trains a captioner with two critics and uses critic-based planning to select words at inference.
3. Cross-domain Image Captioning
- Target-domain captions should match target-sentence style and remain relevant to their images, although target image–sentence pairs are unavailable.The setup provides paired data in the source domain and separate image and sentence sets in the target domain.Source: §3 · p. 3
3.1. Captioner as an Agent
- The captioner generates words sequentially from an image and partial sentence, using policy-gradient updates based on expected future reward.Monte Carlo rollout estimates that reward by sampling possible sentence continuations.Source: §3.1 · pp. 3–4
3.2. Critics
- The domain critic scores target-domain sentence style, while the multi-modal critic scores image–sentence relevance; their probabilities jointly determine sentence reward.The domain critic classifies sentences as source, target, or generated, and the multi-modal critic classifies pairs as paired, unpaired, or generated.Source: §3.2 · pp. 4–5
3.3. Adversarial Training
- Captioner and critics iteratively pursue competing goals: the captioner seeks high reward, while the critics distinguish generated outputs from target and source data.The captioner is first pretrained with source-domain cross-entropy; critics are updated more often than the captioner during adversarial training.Source: §3.3 · p. 5
3.4. Critic-based Planning
- At inference, critic-based planning uses expected reward to choose among candidate words when the policy’s top choices are sufficiently close.The method can produce the same sentences as greedy search and typically gives further gains on datasets with large domain shifts.Source: §3.4 · p. 6
4. Experiments
The experiments evaluate adaptation from MSCOCO to four target domains with different degrees of domain shift. The method improves over source-pretrained and DCC baselines, critic-based planning often improves decoding, and the ablation favors using both critics.
4. Experiments
- The study evaluates MSCOCO pre-training followed by adversarial adaptation on unpaired target-domain data, using four target datasets spanning different domain shifts.The experimental setup also compares against DCC, source-pretrained models, and target-domain fine-tuning with paired data.Source: §4 · p. 6, §4.2 · pp. 6–7, §4.1 · p. 6
4.2. Experimental Results
- CUB-200 and Oxford-102 pose fine-grained description shifts, while TGIF differs in verb usage and Flickr30k tests a subtler style shift.For TGIF, motion verbs account for 30% versus 19% in MSCOCO.Source: §4.2 · p. 7
- Across fine-grained, verb-shifted, and stylistically similar targets, adaptation improves over source-pretrained and DCC baselines.The reported results also show gains for the reverse Flickr30k-to-MSCOCO route.Source: §4.2 · p. 7, Table 1 · p. 7
- Critic-based planning often outperforms greedy search and typically beam search of size 2, with its clearest gains on large-shift datasets.It chooses among the top two policy words; critics determine 16.2% of words in CUB-200 and 9.4% in Oxford-102.Source: §4.2 · p. 7, Table 2 · p. 8
4.3. Ablation Study
- Combining the multi-modal and domain critics gives the best performance across evaluation metrics; the multi-modal critic alone misses target sentence style.Using the domain critic alone contributes significantly.Source: §4.3 · p. 8, Table 3 · p. 8
5. Conclusion
The paper proposes adversarial training between a captioner and critics for cross-domain captioning, plus critic-based planning at inference. It reports consistent improvements over baselines across four target datasets and identifies combining multiple critics in a plug-and-play fashion as future work.
5. Conclusion
- The method consistently outperforms baselines across four target datasets with large or regular domain shifts.The authors propose improving flexibility by combining multiple critics in a plug-and-play fashion.Source: §5 · p. 8
- Adversarial training pits the captioner against critics to support cross-domain image captioning.The paper also proposes critic-based planning to improve caption generation during testing.Source: §5 · p. 8
- Future work aims to make the method more flexible by combining multiple critics in a plug-and-play fashion.Source: §5 · p. 8