PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Knowledge Alignment with Reinforcement Learning

Youngjae Yu, Jiwan Chung, Heeseung Yun, Jack Hessel, JaeSung Park, Ximing Lu, Prithviraj Ammanabrolu, Rowan Zellers, Ronan Le Bras, Gunhee Kim, Yejin Choi

arXiv:2205.12630v1cs.CLcs.CV

TL;DR

Text-only language models have strong zero-shot capabilities, but multimodal inputs remain outside their input format. ESPER uses reinforcement learning to align multimodal inputs with language-model generations without paired image-caption supervision. It outperforms prior unsupervised methods on zero-shot image/audio captioning and demonstrates diverse generation across additional tasks.

  • Problem

    Large pretrained language models show zero-shot capability, but models such as GPT-3 accept only textual prompts rather than multimodal inputs.

    Source: §1 · p. 1

  • Method

    ESPER trains a lightweight visual-to-language encoder with reinforcement learning rewards while keeping CLIP and GPT-2 parameters frozen.

    Source: §1 · pp. 1–2, §2 · p. 2

  • Results

    ESPER outperforms prior unsupervised methods in zero-shot image/audio captioning quality and inference speed, including a 14.6-point CIDEr improvement over Laina et al. (2019) on COCO unpaired captioning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ESPER exhibits zero-shot adaptability for visual news, visual dialogue, and diverse-style image generation, and can learn about audio inputs using an audio-based reward.

    Source: §1 · p. 2

    The audio-based training signal appears less accurate than image CLIP, though ESPER still generates audio-relevant and plausible captions.

    Source: §3.2 · p. 6

Abstract

from arXiv · show

Large language models readily adapt to novel settings, even without task-specific training data. Can their zero-shot capacity be extended to multimodal inputs? In this work, we propose ESPER which extends language-only zero-shot models to unseen multimodal tasks, like image and audio captioning. Our key novelty is to use reinforcement learning to align multimodal inputs to language model generations without direct supervision: for example, in the image case our reward optimization relies only on cosine similarity derived from CLIP, and thus requires no additional explicitly paired (image, caption) data. Because the parameters of the language model are left unchanged, the model maintains its capacity for zero-shot generalization. Experiments demonstrate that ESPER outperforms baselines and prior work on a variety of zero-shot tasks; these include a new benchmark we collect+release, ESP dataset, which tasks models with generating several diversely-styled captions for each image.

1 Introduction

ESPER extends text-only zero-shot language models to multimodal inputs, using reinforcement learning to align visual features with language generation. It outperforms prior unsupervised methods on image/audio captioning and supports diverse zero-shot generation tasks.

1 Introduction

  • GPT-3 demonstrates broad zero-shot language capabilities but accepts only textual prompts, leaving multimodal inputs outside its input format..Source: §1 · p. 1
  • ESPER generates diverse, image-conditioned text for zero-shot visual news, dialogue, visual question answering, blog-style posts, and stories.The ESP dataset tests generating texts in different styles for the same image; the work also explores audio inputs with an audio-based reward.Source: §1 · pp. 1–2
  • ESPER combines multimodal prompt tuning with reinforcement learning, updating a lightweight visual-to-language adapter rather than training on paired image-caption data.CLIP similarity scores generated text, while the language model remains frozen.Source: §1 · pp. 1–2
  • Paired visual-linguistic datasets contain literal descriptions that may not match GPT-2's training-text variety, and supervised models may consequently produce less richly styled language or diverse reasoning..Source: §1 · p. 2
  • ESPER outperforms prior unsupervised methods in zero-shot image/audio captioning quality and inference speed, including a 14.6-point CIDEr gain over Laina et al. (2019) on COCO unpaired captioning.It is also 102× faster than Tewel et al. (2021), which uses per-token gradient optimization over partial decodings.Source: §1 · p. 2

2 Method

ESPER connects multimodal inputs to a frozen pretrained language model through a trainable encoder, then uses reinforcement learning to align generated text with the input. Its method also includes rewards to stabilize generation and a text-only style-pretraining variant.

2 Method

  • CLIP and GPT-2 remain frozen while PPO trains ESPER’s encoder, which projects multimodal inputs into GPT-2’s word-embedding space.Gradients pass through the frozen language model to update the encoder.Source: §2 · p. 2, §2.1 · p. 3

2.1 Architecture

  • The encoder maps CLIP image features to a sequence of 10 vectors, which serve as a multimodal prompt concatenated with embedded text prompts.The two-layer MLP maps CLIP dimensions to GPT-2 dimensions, then expands the representation to the fixed-length sequence.Source: §2.1 · p. 3

2.2 Training

  • PPO treats GPT-2 as a policy generating text from image-plus-text prompts and optimizes a CLIP cosine-similarity reward because CLIP provides no per-token feedback.Training uses random sampling with temperature 0.7; rewards also include KL divergence, text-only likelihood, and a repetition penalty to stabilize language generation.Source: §2.2 · pp. 3–4

2.3 ESPER-Style

  • ESPER-Style pre-fine-tunes GPT-2 on text corpora with style prefixes such as “news:” before using those generators as backbones for ESPER.The style-prompt training uses text only and does not require multimodal inputs.Source: §2.3 · p. 4

3 Experiments

The experiments evaluate ESPER on unpaired image and audio captioning, downstream visual tasks, and diverse-style generation. Results include gains over comparison systems, while qualitative examples and human ratings also reveal limits and trade-offs.

3 Experiments

  • The experiments use COCO training images unpaired with captions for fair comparison, and evaluate image-text alignment through unpaired captioning and task transfer.The ESP benchmark tests generating multiple writing styles for the same image.Source: §3 · p. 4, §3.1 · p. 5

3.1.2 Finetuning

  • In supervised finetuning, ESPER initialization beats random initialization whether GPT parameters are updated or fixed.The comparison evaluates the encoder's contribution as pretrained weights.Source: §3.1.2 · p. 6

3.2 Evaluation of Auditory Alignment

  • ESPER improves over GPT-2 rollout-and-select baselines on unpaired AudioCaps captioning, and style prompt tuning increases CIDEr by 4.86.The reported audio comparisons include GPT-2-based baselines because the unpaired image baselines require object detectors and cannot be directly applied.Source: §3.2 · p. 6

3.3.1 Visual News

  • On VisualNews, zero-shot ESPER beats text-only and supervised baselines on Bleu-4 and METEOR, but trails the supervised model on CIDEr.The authors attribute the CIDEr gap partly to proper nouns that CLIP has not encountered; adaptor finetuning overcomes this gap and surpasses baselines on CIDEr.Source: §3.3.1 · p. 6

3.3.2 Visual Dialogue

  • On VisDial, zero-shot ESPER improves on the baselines and outperforms supervised ViLBERT-Head in ranking answer candidates.Evaluation ranks 100 candidate answers given an image and the previous dialogue act.Source: §3.3.2 · p. 7

3.4 From One Image to Many Styles

  • On ESP, ESPER outperforms the COCO-supervised CLIPCap-MLP baseline, while the results support the value of both visual-linguistic alignment and explicit style conditioning.The benchmark contains 4.8k captions for 1k COCO test images across five styles; ESPER-Style improves over ESPER-Free.Source: §3 · p. 4, §3.4 · p. 7

3.5 Human Evaluations on ESP dataset

  • Human ratings find ESPER more visually relevant and informative than CLIP-Infer across every prompt, while CLIP-Infer is slightly more fluent on story and instruction prompts.Ratings use a five-point Likert scale from three annotators on 100 randomly selected ESP test images.Source: §3.5 · p. 7, Figure 4 · p. 7

3.6 Qualitative Results

  • Qualitative examples show errors in weather interpretation and visual counting, while recipe prompts not seen in training can yield sensible generations even for non-food images.The examples also show plausible audio-captioning adaptation using Wav2CLIP rewards.Source: §3.6 · p. 8

4 Related Work

Related work spans visual-language pretraining, multimodal prompt tuning, unsupervised captioning, and reinforcement learning for language tasks. ESPER is distinguished as using multimodal reward to align images with a general pretrained language model.

Visual-Language Pretraining

  • Visual-language pretraining includes BERT-style, encoder-decoder, and contrastive models, with vision-text models also extended to audio.TAPM adapts a visual encoder and GPT using a self-supervised causal-order objective for visual stories.Source: §4 · pp. 8–9

Multimodal prompt tuning

  • Multimodal prompt tuning adapts small image-to-text adapters with maximum likelihood, and CLIPCap combines GPT with CLIP image features while fixing GPT weights.ESPER follows the same architecture as CLIPCap and the setup of p-tuning.Source: §4 · p. 9

Unsupervised captioning

  • Prior unsupervised captioning methods use pseudo-pairs from visual concept detectors or image-language embedding spaces, while Tewel et al. (2021) use CLIP alignment to guide inference without further training.Source: §4 · p. 9

Reinforcement learning for language tasks

  • Prior language-task reinforcement learning has been applied to captioning, storytelling, dialogue, interactive QA, text games, and alignment to human preferences.Instruction GPT also applies RL to prompt-conditioned generation quality.Source: §4 · p. 9
  • ESPER is presented as the first method using multimodal reward to align images with pretrained language models; Cho et al. (2022) use CLIP rewards but finetune a trained image-captioning model.ESPER instead targets a general large language model.Source: §4 · p. 9

5 Conclusion

ESPER combines GPT-2 language generation with CLIP multimodal knowledge, training through reinforcement-learning rewards rather than maximum likelihood. The conclusion identifies scaling, multiple reward modalities, and data augmentation as future directions.

5 Conclusion

  • ESPER combines GPT-2 language generation with CLIP multimodal knowledge to create an image-conditioned text generator trained with reinforcement-learning rewards rather than maximum likelihood.The objective can also be used with multimodal prompt tuning and zero-shot captioning with CLIP guidance.Source: §5 · p. 9
  • Future work includes maximizing rewards across multiple modalities, scaling CLIP and GPT-2 backbones, and exploring ESPER for multimodal-reasoning data augmentation.Source: §5 · p. 9

A Language Model Backbones

The backbone appendix describes ESPER-Free’s GPT-2-base setup and ESPER-Style’s text-only style preparation, including the prompt and corpus sources used for style generators.

ESPER-Free

ESPER-Style

  • ESPER-Style prepares style-generator backbones by finetuning GPT-2 on a text-only corpus with style prompts; listed style-prompt and text-corpus sources include COCO Caption, Sentiment140, MDID, and TweetEval.Source: A Language Model Backbones · p. 14

Visual news generation

  • Visual-news generation uses source-specific prompts—bbc:, guardian:, usa today:, and washington post:—to reflect writing-style differences in VisualNews.Source: A Language Model Backbones · p. 14

B Language Model RL Training

The appendix describes rewards and constraints used in language-model RL training, including KL divergence, reference entropy, and repetition penalties. The supplied table captions identify style-wise and human-evaluation results but provide no result values.

KL Divergence

  • KL divergence constrains the online policy relative to the initial language model to help maintain generated-text salience.The optimization uses the per-token difference between online- and initial-policy log likelihoods.Source: B Language Model RL Training · p. 14

Reference Entropy

  • Reference entropy penalizes generated tokens whose text-only negative log likelihood exceeds a length-scaled threshold, using τe = 70 l and gain αe = 0.1.Source: B Language Model RL Training · p. 14

Evaluation tables

  • Table 5 reports style-wise ESP dataset results with Bleu-4, while Table 6 reports three annotators’ average 5-point Likert ratings for visual relevance, informativeness, and fluency.Source: Table 5 · p. 15, Table 6 · p. 15

Repetition Penalty

C Details on ESP dataset

The ESP dataset explores diverse writing styles for describing the same image. Styles encompass choices of text type, structure, and vocabulary that convey a writer’s intended meaning, and may highlight different information in the same visual cue.

  • Styles include choices of text type, structure, and vocabulary used to convey a writer’s intended meaning.Different styles may describe different information from the same visual cue.Source: C Details on ESP dataset · p. 15
  • A sibling’s informal caption about a boy singing in a choir can differ from a local news account of the same concert.The sibling’s caption focuses on the bow tie, while the news account describes the choir’s performance.Source: C Details on ESP dataset · p. 15
  • The dataset collects captions relevant to an image while following specified writing styles.Captions are filtered for adherence to both the image and the requested style.Source: C Details on ESP dataset · p. 15

D ESP dataset Collection Process

The collection uses COCO test-set images and asks annotators to write style-conditioned, image-relevant text. Submissions are monitored and manually checked for relevance to the image and style.

  • Images were randomly selected from the COCO Captions test set using the Karpathy split, with one to five annotated objects.The selection aimed for salient but not noisy context.Source: D ESP dataset Collection Process · p. 15
  • Annotators wrote image-relevant text following a specified style, with a 30-word minimum lowered to 10 words for shorter styles such as social media and news.Source: D ESP dataset Collection Process · p. 15
  • The collection involved 189 workers, with regular monitoring and manual filtering for fluency, style understanding, and caption relevance.Source: D ESP dataset Collection Process · p. 15

E ESP dataset Experiment Details

ESPER adapts to diverse text styles without paired image-text data for those styles, while supervised baselines show limited generalizability even with style prompts. The experiment compares it with a text-only generator and supervised models under specified prompting and length controls.

  • The comparison uses a text-only pretrained style generator and two COCO-caption-supervised baselines sharing ESPER’s architecture.The supervised baselines are evaluated with and without style prompts; without them, the prompt is fixed to “Image of a”.Source: E ESP dataset Experiment Details · p. 16
  • All texts, including ground-truth captions, are truncated to the first 20 GPT-tokenizer byte-pair tokens for comparison with ClipCap-MLP.The passage states that all compared methods use vanilla GPT2’s tokenization scheme, so truncation does not favor a particular approach.Source: E ESP dataset Experiment Details · p. 16
  • ESPER adapts flexibly to each style without paired image-text data for those styles, while supervised baselines show limited generalizability even with style prompts.Table 5 reports scores that include and expand upon the summarized Figure 5 results; the total score is the mean over each style’s metrics, without sample-size weighting.Source: E ESP dataset Experiment Details · p. 16

F Additional Qualitative Samples

The additional qualitative samples show ESPER generations conditioned on custom prompts. The prompts grow progressively to emphasize the randomness of the outputs.

Related papers

Loading 2205.12630v1…