Unifying Multimodal Transformer for Bi-directional Image and Text Generation

1 INTRODUCTION

• Traditional systems use task-specific architectures, and an earlier jointly trained system still requires separate image-to-text and text-to-image architectures, increasing design effort.The paper describes CNN visual encoders with LSTM caption decoders and GAN-based text-to-image generators as typical choices.Source: §1 · p. 2
• Prior Transformer approaches discretize image features, losing information for image-to-text, while teacher-forced cross-entropy training can create exposure bias and test-time error accumulation.The paper identifies these as two challenges for bi-directional generation.Source: §1 · p. 2
• The framework uses dense image features to reduce information loss for image-to-text, discrete features for text-to-image, and sequence-level training to bridge training–testing mismatch.A CLIP-based image-level loss improves text-image consistency, and a CLIP-based metric supports evaluation without extra crowd-sourced labeling.Source: §1 · p. 2
• CLIPScore rises from 72.9 to 77.2 for text-to-image generation, and CIDEr-D rises from 100.9% to 122.6% for image-to-text generation over X-LXMERT.Both comparisons are reported on MS-COCO.Source: §1 · p. 2
• Most Transformer modules are shared and trained iteratively, facilitating shared image-text embedding learning with half the model size of two separate Transformer models.The approach is motivated partly by storage efficiency for multimodal applications on mobile devices.Source: §1 · pp. 1–2

2.1 Image-to-Text Generation

• Image captioning commonly uses CNN encoders with RNN or Transformer decoders; attention lets the decoder selectively integrate relevant visual information, and large-scale vision-language pretraining has advanced captioning.Transformer models are noted for parallel training and modeling global dependencies through attention.Source: §2.1 · p. 2

2.2 Text-to-Image Generation

• Text-to-image work spans pixelCNN, Langevin sampling, variational autoencoders, and GANs, with GANs and multi-stage generation described as dominant approaches; large pretrained Transformers have produced high-fidelity images.DALL-E and CogView are cited as large-scale examples trained on hundreds of millions of image-text pairs.Source: §2.2 · p. 2

2.3 Bi-directional Image-and-Text Generation

• Earlier bi-directional systems include Huang et al.'s separate LSTM captioner and GAN image generator, while X-LXMERT uses a Transformer for several vision-language tasks; this work uses the same model for both generation directions.Figure 2 depicts unified token sequences, multimodal Transformer interactions, and sequence-level training.Source: §2.3 · pp. 2–3, Figure 2 · p. 3

3 MODEL

• The model section introduces a unified multimodal framework and two-level granularity image representations as its central design.Figure 2 provides an overview of the model.Source: §3 · p. 3

3.1 Unified Multimodal Framework

• Image and text are preprocessed as token sequences; the Transformer contextualizes masked input tokens and classifiers predict target tokens, sharing most Transformer parameters but not the output classifiers.A GAN-based image generator converts 8 × 8 image-token predictions into 256 × 256 images for text-to-image generation.Source: §3.1 · p. 3

3.2 Two-level Granularity Image Representations

• Fine-grained Faster R-CNN grid features serve image-to-text generation to reduce information loss, while clustered discrete features provide visual-token labels for text-to-image generation.K-means clustering and nearest-neighbor search produce cluster indices; grid positions use 2-D embeddings of bounding-box positions.Source: §3.2 · p. 3

4 TRAINING

• Two-stage training addresses exposure bias: teacher forcing trains at the token level, then generated sequences are used to bridge the training–testing gap.The strategy applies word-level training to image-to-text generation and grid-level training to text-to-image generation.Source: §4 · p. 4

4.1 Image-to-Text Generation

• Image-to-text training first predicts each ground-truth word from prior words and visual context, then uses SCST to optimize sentence-level metrics such as CIDEr-D.The second stage minimizes the negative expected CIDEr-D score of sampled text sequences.Source: §4.1.1 · p. 4, §4.1.2 · p. 4

4.2 Text-to-Image Generation

• Text-to-image grid-level training predicts multiple masked visual tokens from other ground-truth tokens and text, enabling mask-predict-k sampling in a few steps.The paper gives k=4 as an example and notes that this strategy enables faster inference.Source: §4.2.1 · p. 4
• Image-level training uses CLIP to optimize semantic consistency with the caption rather than matching one reference image, addressing one-to-many mappings and ignored cross-grid relations.Grid-feature similarity, pixel-wise, and perceptual losses showed little improvement in preliminary experiments; image-level training is iterated with teacher forcing because sampling has high variance and teacher forcing encourages sampling coherency.Source: §4.2.2 · p. 4

5.1 Experimental Setup

• The study trains on 82,783 MS-COCO images and evaluates captioning on the 5,000-image Karpathy test split; captioning results are not directly comparable with models trained on 113,287 images.For text-to-image evaluation, it samples 30,000 validation images; grid-based 8x8 features are used for comparison with X-LXMERT.Source: §5.1 · pp. 4–5

5.2 Evaluation Criteria

• Evaluation uses captioning metrics including BLEU, ROUGE-L, METEOR, CIDEr-D, and SPICE, while image generation is assessed for fidelity and image-text alignment.The text-to-image criteria include FID, R-precision, and CLIPScore, which measures cosine similarity between CLIP image and text representations.Source: §5.2 · p. 5

5.3 Ablation Studies

• The unified Transformer outperforms separate Transformers on most text-to-image metrics, while showing comparable image-to-text performance with half the parameter size.Two-level features retain dense representations for image-to-text and discrete representations for text-to-image, whose single-feature alternatives trade off between tasks.Source: §5.3 · pp. 5–6, Table 1 · p. 5

5.4 Quantitative Results

• Across both tasks, Table 2 reports gains over comparison methods; versus X-LXMERT-FT, CIDEr-D rises from 100.9% to 122.6%, and CLIPScore improves by 3.9 points.Against DM-GAN, the model improves CLIPScore by 4.9 and FID by 6.0.Source: Table 2 · p. 6, §5.4 · p. 7

5.3 Ablation Studies

• Removing sequence-level training reduces text-to-image CIDEr-D from 122.6% to 107.9% and image-to-text CLIPScore from 77.2 to 73.4; replacing CLIP loss also hurts text-to-image metrics.The authors attribute sequence-level training's benefit to mitigating the training-testing exposure-bias mismatch; CLIP loss improves semantic consistency without affecting image-to-text performance.Source: §5.3 · p. 6

5.5 Human Evaluation

• In human evaluation, the model surpasses DM-GAN, X-LXMERT, and w/o CLIP by about 37%, 19%, and 29%, respectively, averaged across fidelity and semantic scores.Ten English-proficient volunteers compared generated image-caption pairs from 300 sampled MS-COCO test captions.Source: Figure 3 · p. 7, §5.5 · p. 7

5.6 Qualitative Examples

• Qualitative examples show more accurate, rational captions and images with more realistic objects and stronger text consistency than the compared systems.Compared with the discrete-feature model, the unified model also generates more realistic and smooth images.Source: §5.6 · p. 7

• The unified Transformer alleviates task-specific design effort and improves storage utilization compared with two separate models for bidirectional generation.Source: §6 · p. 7
• Two-level granularity features address information loss from feature discretization, while sequence-level training addresses test-time error accumulation from cross-entropy training.Source: §6 · p. 7
• The authors report qualitative and quantitative evidence of effectiveness and suggest the model could facilitate multimodal industrial applications with desirable performance.Source: §6 · p. 7