PaperlayerView exact revision on arXiv

Source-linked AI summary

FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation

Guangyu Sun, Shlok Kumar Mishra, Wentao Bao, Robert Zhenheng Yang, Xiao Wang, Xiyuan Wang, Yujunrong Ma, Chen Yuan, Max Xiangjun Fan, Jun Xiao, Jianpeng Cheng

arXiv:2609.16591v1cs.CV

TL;DR

Traditional multimodal systems separate representation learning from generation, limiting generative models to downstream use of pretrained embeddings. FLAT jointly trains a shared multimodal encoder with T2I and I2T decoders in a unified flexible-length 1D token space. It supports retrieval, generation, and latent-space operations, with fine-tuned performance reaching reported state-of-the-art baselines across tasks.

  • Problem

    Traditional multimodal pipelines separate representation learning from generation, leaving downstream generative models dependent on pretrained embeddings.

  • Method

    FLAT jointly trains a shared multimodal encoder with T2I and I2T decoders using contrastive and bidirectional generative objectives over flexible-length 1D tokens.

  • Results

    FLAT reaches reported state-of-the-art performance after task-specific fine-tuning across T2I generation, I2T generation, and cross-modal retrieval.

  • Takeaways & Limitations

    FLAT representations function as both semantic descriptors for retrieval and generative conditions while supporting variable-length outputs and latent-space operations.

Abstract

from arXiv · show

Traditional multimodal representation learning and generation are two stages: a contrastive or self-supervised visual encoder is trained first, followed by a separate downstream generative model. This setup bottlenecks generative performance behind frozen embeddings. To bridge this gap, we revisit joint multimodal representation learning and generation to produce linearly interpolatable embeddings that are directly consumable by generative decoders. We present FLAT (Flexible-Length Aligned Transmodal representations), a representation pre-training framework that jointly optimizes a shared multimodal encoder alongside downstream text-to-image (T2I) and image-to-text (I2T) decoders. By combining contrastive alignment with bidirectional cross-modal generative objectives, FLAT ensures its representations function as both discriminative semantic descriptors and generative conditions. Architecturally, FLAT maps visual and textual inputs into a unified continuous 1D sequence space, applying nested dropout over prefix-K tokens to enable dynamic output lengths. A single pre-training stage allows FLAT to perform cross-modal retrieval and generation across variable prefix K, achieving a T2I GenEval score of 71.1. Task-specific fine-tuning aligns model performance with state-of-the-art baselines: 83.1 GenEval on T2I generation; 40.5 BLEU-4 and 138.6 CIDEr on MS-COCO image captioning; and Recall@5 scores of 86.8 (I2T) / 75.8 (T2I) on MS-COCO alongside 98.3 (I2T) / 93.6 (T2I) on Flickr30K. Finally, qualitative evaluations demonstrate that FLAT representations natively support linear interpolation, latent space arithmetic, and zero-shot composed retrieval.

1 Introduction

FLAT jointly trains a shared multimodal encoder with bidirectional image-to-text and text-to-image generation, preserving contrastively aligned representations that also serve as generative conditions. Its variable-length 1D tokens support retrieval and synthesis, reaching strong zero-shot and fine-tuned results across both directions.

  • 1 Introduction: FLAT jointly trains a multimodal encoder with T2I and I2T decoders using contrastive alignment and bidirectional generative objectives.The framework retains an explicit, linearly interpolatable embedding space while optimizing retrieval and generation together.
  • 1 Introduction: Qualitative evaluations show that FLAT representations support linear interpolation, latent-space arithmetic, and zero-shot composed retrieval.
  • 1 Introduction: FLAT resamples visual and textual inputs into a unified continuous 1D representation space rather than using separate modality-specific representations.The representation is contrastively aligned during generative training, supporting both semantic description and decoder conditioning.
  • 1 Introduction: 83.1 GenEval, 40.5 BLEU-4, and 138.6 CIDEr demonstrate strong task-specific T2I and MS-COCO I2T performance after fine-tuning.The same evaluation reports Recall@5 of 86.8/75.8 on MS-COCO and 98.3/93.6 on Flickr30K for I2T/T2I retrieval.
  • 1 Introduction: 71.1 GenEval and zero-shot MS-COCO Recall@5 of 69.1 (I2T) and 64.6 (T2I) are achieved after a single pre-training stage.Varying prefix-K tokens enables one pretrained model to retrieve and synthesize coarse-to-fine cross-modal outputs from the same encoder pass.

2 Related Work

Prior work largely separates multimodal representation learning from generation, while newer approaches jointly train unified models or bypass explicit vision encoders. FLAT builds on compact 1D resampling methods by applying them to both visual and textual inputs within a shared multimodal architecture.

  • 2 Related Work: Earlier multimodal systems commonly use contrastive or self-supervised representations separately from downstream generative models.Examples include frozen visual encoders for I2T and frozen text encoders for T2I systems.
  • 2 Related Work: Joint representation-learning and generation methods combine alignment with image or text generation, including BLIP, CoCa, MAGE, DREAM, and unified multimodal models.
  • 2 Related Work: Resampled 1D representations compress image grids, long text, or image-encoder outputs into compact sequences or summary vectors.Representative approaches include TiTok, FlexTok, GigaTok, AutoCompressor, and BLIP-2.
  • 2 Related Work: FLAT’s architecture uses a shared VLM encoder, register tokens, nested dropout, and a unified 1D continuous representation driving contrastive alignment and both transmodal decoders.

3 Method

FLAT jointly learns a shared multimodal encoder and bidirectional decoders so one continuous 1D representation supports contrastive alignment, retrieval, and generation at variable prefix lengths.

  • Overview: FLAT maps image and text inputs into ordered continuous 1D tokens that are contrastively aligned and consumed by both transmodal decoders.The shared multimodal encoder produces the representation, which serves as the common condition for image-to-text and text-to-image tasks.
  • Representation Encoder: The encoder appends learnable registers, projects their hidden states into shared representations, and uses the same registers and projection for both modalities.The encoder is a pretrained vision-language model prompted to represent the input.
  • Representation Dropout: Nested dropout samples K from {1, 4, 16, 64, 256} and passes only the prefix z:K to contrastive and generative objectives, enabling variable sequence lengths.Sampling one geometric-ladder length per training step avoids wasting optimization on imperceptible length differences.
  • Decoders: The visual representation conditions autoregressive caption generation, while the textual representation conditions rectified-flow image synthesis through decoder-specific soft-token projections.Image synthesis uses cross-attention to integrate textual soft tokens while denoising continuous VAE latents.
  • Training Objective: Training combines bidirectional contrastive loss with symmetrical image-to-text and text-to-image generation losses using the same representation prefix.Fine-tuning can optimize task-specific objectives separately over the encoder and decoders.
  • Variable-Length Generation: A single fine-tuned checkpoint supports GenEval evaluation across prefix lengths, with larger K providing more fine-grained visual details and smaller K capturing core entities and colors.Figure 2 links larger prefixes to spatial relations, multi-object layouts, and attribute bindings.

4 Experiments

FLAT is evaluated as a unified 1D representation for generation, retrieval, probing, and latent-space analysis across variable prefix lengths. Results show strong task performance, stable retrieval under truncation, aligned cross-modal geometry, and emergent interpolation and arithmetic.

  • Generative Evaluation: 83.1 GenEval is achieved on T2I generation with a 2B encoder without prompt rewriting, compared with 0.78 for MetaQuery-L under the same protocol.Single tokens capture basic semantics, while relational and compositional tasks require larger prefixes.
  • Discriminative Evaluation: Retrieval remains virtually invariant across K, with every metric staying within roughly one percentage point across a 256× width reduction.FLAT spans 64 dimensions at K=1 to 16,384 at K=256, and the first token carries sufficient global semantics and discriminative power.
  • Discriminative Evaluation: 73.3% ImageNet-1K top-1 accuracy is obtained with a frozen K=1 token, increasing to 81.2% at K=16 and outperforming purely generative latent spaces and DREAM at equivalent dimensionality.Unsupervised k-means clusters from a single token align with ImageNet classes without label supervision.
  • Geometry Analysis: FLAT reduces CLIP’s cross-modal centroid distance by half with one register token and to one quarter with 256 tokens, producing tighter image-text overlap.The aligned geometry supports smooth cross-modal interpolation, with longer prefixes controlling finer-grained transitions.
  • Geometry Analysis: Zero-shot token arithmetic modifies target concepts while preserving surrounding scene context, and the full training objective shows mutually reinforcing contrastive and generative losses.These representations also support composed retrieval without explicit editing supervision.

5 Conclusion

FLAT introduces flexible-length aligned transmodal tokens learned by jointly training a shared encoder with bidirectional generative decoders. The resulting representations support retrieval, synthesis, and zero-shot latent operations, with task-specific fine-tuning reaching SOTA-level performance.

  • 5 Conclusion: FLAT resamples images and text into flexible-length 1D aligned transmodal tokens using a shared encoder and T2I and I2T decoders.Contrastive alignment and bidirectional generation losses make the tokens useful both for retrieval and as generative conditions.
  • 5 Conclusion: Task-specific fine-tuning pushes T2I and I2T retrieval and generation performance to SOTA levels.The framework also supports variable-length retrieval and synthesis from the learned representation.
  • 5 Conclusion: Qualitative analyses show that FLAT tokens support zero-shot linear interpolation, latent-space arithmetic, and composed retrieval.The evaluation and appendix coverage include zero-shot transfer, latent composition, and expanded qualitative analyses.

B.1 Evaluation protocol

The evaluation protocol measures T2I and I2T generation, bidirectional retrieval, composed retrieval, and frozen-representation probing under controlled prefix-length and task-specific settings. Standard benchmarks and metrics are used across COCO, Flickr30K, CIRR, and ImageNet-1K.

  • B.1 Evaluation protocol: T2I generation is evaluated on all 553 GenEval prompts with four images per prompt, 20 sampling steps, CFG scale 4.5, and no prompt rewriting.The initial noise vector is reused across prefix lengths for controlled comparisons.
  • B.1 Evaluation protocol: I2T captioning uses the MS-COCO Karpathy test split with greedy decoding and reports BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE.These metrics measure n-gram precision, unigram alignment, sequence structure, descriptive similarity, and semantic scene-graph overlap.
  • B.1 Evaluation protocol: Bidirectional retrieval is evaluated on COCO Karpathy 5K and Flickr30K Karpathy 1K test splits using direct recall without post-hoc reranking.Composed retrieval uses 1,000 CIRR queries against a 1,000-image gallery with latent add/remove arithmetic.
  • B.1 Evaluation protocol: Composed retrieval forms edit queries by adding representations for requested phrases and subtracting representations for removed phrases relative to a neutral phrase representation.The resulting composite representation is normalized and scored with a late-interaction similarity function over a sweep of α values.
  • B.1 Evaluation protocol: Linear probing trains a classifier on frozen concatenated prefix-K tokens using all 1,281,167 ImageNet-1K training images and evaluates on 50,000 validation images.Task-specific configurations use direct prefix truncation and uniformly sample K from {1, 4, 16, 64, 256}.

C Evaluation Results of the pre-trained checkpoint

The jointly pre-trained FLAT checkpoint supports text-to-image generation, image-to-text captioning, and bidirectional retrieval without task-specific adaptation. Performance improves with longer prefixes, while captioning metrics are partly limited by stylistic mismatch between pre-training text and COCO references.

  • T2I Generation: 0.71 overall GenEval is achieved by the pre-trained checkpoint, scaling from 0.32 at K=1 to 0.71 at K=256.The evaluation uses the same prompt, sampling, and fixed-noise protocol as the fine-tuned checkpoint without task-specific tuning.
  • I2T Generation: 63.8 CIDEr and 16.3 SPICE are reached for zero-shot MS-COCO captioning at K=256.Larger prefixes steadily improve performance, but scores remain below the fine-tuned checkpoint because pre-training text is more diverse and descriptive than COCO captions.
  • I2T Generation: 99.6% of generated sequences terminate with EOS at K=64, while 27.1% begin with quotation marks and average 10.8 words.Fine-tuning mainly aligns output length and stylistic tone with COCO expectations, so some metric gains reflect format alignment rather than semantic improvement alone.
  • Retrieval: Zero-shot FLAT representations effectively support bidirectional retrieval on MS-COCO and Flickr30K, although performance degrades slightly as prefix length increases.Task-specific fine-tuning substantially mitigates this sensitivity to K.

D Effect of FLAT pre-training on task adaptation

FLAT pre-training substantially improves task-specific generation relative to matched controls, while retrieval gains are smaller and direction-dependent. The comparison uses the same architecture, data, optimization budget, and evaluation protocol for both conditions.

  • T2I synthesis: 0.16–0.24 GenEval improvement is obtained for K ≥4, reaching 0.83 at K=16 and K=64 with FLAT pre-training.The task-only model remains near 0.60 across prefix lengths, whereas pre-training enables gains from wider prefixes.
  • Captioning: 9.0–10.9 BLEU-4 and 31.9–41.2 CIDEr gains are obtained across prefix lengths for captioning after FLAT pre-training.These comparisons use matched 5k-step recipes on the COCO Karpathy train+restval split.
  • Retrieval: Image-to-text R@1 improves by 0.6–1.3 points, while the task-only model is 0.4–0.9 points higher for text-to-image R@1.Retrieval effects are therefore smaller and direction-dependent than the generative-task improvements.

E Qualitative results for FLAT

Qualitative evaluations show coherent variable-length text-to-image behavior, semantically strong retrieval, and characteristic failure modes. Unseen prefix lengths remain usable, while errors involve fine-grained layout, typography, omitted details, or near-miss retrievals.

  • Text-to-image generation: Controlled galleries cover single-object, multi-object, counting, color, spatial-relation, and attribute–object prompts across the full prefix-length ladder.Longer free-form prompts are also evaluated at K=64 across photographic and illustrative styles.
  • Retrieval: Bidirectional retrieval galleries show ground-truth targets near the top, with rank-2 or rank-3 cases representing semantically consistent near misses.Green boxes mark ground-truth targets and the remaining candidates are top-ranked retrievals.
  • Failure cases: Generation preserves subject matter and style but can contain minor layout or typography imperfections, while captioning sometimes omits secondary objects or actions.Retrieval failures are likewise preceded by semantically close alternatives.
  • Prefix-length behavior: Unseen intermediate prefix lengths produce coherent generations that evolve smoothly with K under fixed prompts and initial noise.Direct prefix truncation enables inference at arbitrary integer sequence lengths within 256.

F.2 Prefix-dependent compute

Prefix length affects computation, generation quality, interpolation granularity, and zero-shot extensions. FLAT supports arbitrary-length inference, while training strategies trade off generation and retrieval performance and the representation generalizes to multilingual, video, and composed-retrieval settings.

  • Prefix-dependent compute: Reducing K from 256 to 1 lowers encoder and I2T decoder compute by 50.0%, while T2I compute decreases by 13.8%.T2I cost remains dominated by spatial denoising and VAE decoding rather than conditioning length.
  • Prefix sampling: Geometric prefix sampling favors generation for K ≥4, Replay favors retrieval, and no strategy is universally optimal across metrics and prefix lengths.The comparison uses matched 40k-step checkpoints.
  • Latent interpolation: Longer prefixes yield smoother transitions across multiple attributes, whereas shorter prefixes produce coarser latent blends during interpolation.Additional K=256 walks span identity, appearance, structure, context, and season.
  • Zero-shot extensions: French and Chinese prompts and emoji sequences decode into images consistent with their English counterparts despite English-only training.The representation encoder inherits multilingual capabilities from its LLM backbone.
  • Multi-frame video understanding: At K=64, jointly encoding four sampled video frames preserves persistent objects and scene context in one generated image.The representation aggregates multi-frame context into a unified thumbnail-like visual output.
  • Composed retrieval: K=1 gives the strongest zero-shot CIRR composed-retrieval performance, showing that an edit vector can steer the global representation space through a single token.The evaluation uses a protocol without training on composed queries.

H Ablation on the training loss contributions

FLAT’s three objectives contribute differently across retrieval, captioning, and image generation, while auxiliary objectives generally provide complementary signals. The task-aligned contributions become clearer as the prefix length increases.

  • FLAT evaluates individual and joint contributions by training variants with different subsets of retrieval, T2I-generation, and I2T-generation losses under matched initialization and schedules.The full ablation varies only the active loss components across a common 40k-step schedule.
  • Image-generation loss accounts for at least 95% of GenEval across prefix lengths, while captioning dominates both text-generation metrics from K=16 onward.Contrastive learning increasingly accounts for retrieval performance as the prefix widens.
  • The task-aligned attribution structure sharpens with wider prefixes: contrastive learning drives retrieval, captioning drives text generation, and image generation drives GenEval.The heatmaps normalize each attribution by the total improvement, so each metric column sums to 100%.
  • Minor negative auxiliary attributions appear at K=16 and K=256, showing that non-primary objectives are not uniformly beneficial at every prefix length.Despite these negative entries, the primary task-aligned attributions remain dominant.

I Truncated prefixes and null padding

FLAT compares truncating the prefix sequence with replacing later positions by learned null embeddings. Null padding introduces optimization instability, including large gradient excursions before contrastive loss reaches the random-choice value.

  • Fixed-length null padding replaces positions i ≥K with learned null embeddings, whereas truncation retains only the first K register positions.The null-padding strategy keeps sequence length N fixed.
  • Null padding produces large gradient excursions before its contrastive loss approaches the random-choice value.Figure 26 compares optimization under truncation and null padding using smoothed curves with unsmoothed measurements shown faintly.
  • At step 40k, generated images from prefix truncation and null padding use matched COCO target-caption pairs and diffusion noise for comparison.The figure compares outputs produced under the two sequence-handling strategies.
Loading 2609.16591v1…