PaperlayerView exact revision on arXiv

Source-linked AI summary

Linearly Mapping from Image to Text Space

Jack Merullo, Louis Castricato, Carsten Eickhoff, Ellie Pavlick

arXiv:2209.15162v3cs.CLcs.LG

TL;DR

The paper asks whether frozen text-only language models and vision-only models share conceptual representations that can be aligned linearly. It trains a single projection from image features to soft prompts for frozen language models and finds competitive captioning and visual-question-answering performance, with results depending on linguistic supervision in the image encoder.

  • Problem

    The paper addresses whether text-only language models learn representations of the non-linguistic world and whether their conceptual representations resemble those of vision models.

  • Method

    LiMBeR trains one linear projection from frozen image-encoder representations into the input space of a frozen generative language model as continuous soft prompts.

  • Results

    LiMBeR is competitive with models that tune image and language networks on image-captioning and visual-question-answering benchmarks, while performance depends on linguistic supervision during image-encoder pretraining.

  • Takeaways & Limitations

    The results indicate that language- and vision-trained models can learn structurally similar conceptual spaces, including when the vision encoder is trained only on images.

  • Takeaways & Limitations

    Prompt length was not controlled across image encoders, and language-model runoff remains a potential confound despite the authors’ controls.

Abstract

from arXiv · show

The extent to which text-only language models (LMs) learn to represent features of the non-linguistic world is an open question. Prior work has shown that pretrained LMs can be taught to caption images when a vision model's parameters are optimized to encode images in the language space. We test a stronger hypothesis: that the conceptual representations learned by frozen text-only models and vision-only models are similar enough that this can be achieved with a linear map. We show that the image representations from vision models can be transferred as continuous prompts to frozen LMs by training only a single linear projection. Using these to prompt the LM achieves competitive performance on captioning and visual question answering tasks compared to models that tune both the image encoder and text decoder (such as the MAGMA model). We compare three image encoders with increasing amounts of linguistic supervision seen during pretraining: BEIT (no linguistic information), NF-ResNET (lexical category information), and CLIP (full natural language descriptions). We find that all three encoders perform equally well at transferring visual property information to the language model (e.g., whether an animal is large or small), but that image encoders pretrained with linguistic supervision more saliently encode category information (e.g., distinguishing hippo vs. elephant) and thus perform significantly better on benchmark language-and-vision tasks. Our results indicate that LMs encode conceptual information structurally similarly to vision-based models, even those that are solely trained on images. Code is available here: https://github.com/jmerullo/limber

1 INTRODUCTION

The paper asks whether text-only language models learn representations of the physical world and tests whether image and language representations can be aligned with a linear map. It finds that frozen language models can describe image content, with performance depending on the image encoder’s pretraining supervision.

  • The paper investigates whether text-only language models learn aspects of the physical, non-linguistic world.
  • LiMBeR trains one linear projection from image representations into a frozen language model’s input space as continuous soft prompts.
  • The study compares BEIT, NFRN50, and CLIP, whose image pretraining uses increasing amounts of linguistic supervision.
  • All three encoders transfer image semantics for natural-language generation, but stronger linguistic supervision generally improves performance.
  • The results support structural similarity between conceptual spaces learned from language and vision, while showing that similarity varies with image-encoder supervision.

2 RELATED WORK

Related work establishes that pretrained language and vision models can be combined for vision-language tasks, often by tuning models jointly or converting visual information into text. LiMBeR instead directly stitches image representations to a language model to investigate representational similarity.

  • Prior multimodal systems tune pretrained image and text networks together for captioning and downstream vision-language tasks.
  • Other work feeds text prompts derived from vision-language models into language models, whereas this paper feeds visual representations directly into the language model.
  • The paper uses this direct connection to investigate representational similarity between independently trained image and text models.
  • LiMBeR treats the connection between an image encoder and language model as model stitching through soft prompts.

3 METHOD: LINEARLY MAPPING FROM IMAGE TO TEXT REPRESENTATIONS

LiMBeR freezes an image encoder and language model, training only a linear projection that converts image features into soft prompts. The method compares encoders with different pretraining supervision and evaluates their transfer through captioning and visual question answering.

  • Core method: LiMBeR trains only a single projection P while freezing the image encoder and language model on either side.
  • Core method: The method tests whether conceptual representations from an image encoder and language model can be related through a linear transformation.
  • Prompt construction: Image features are projected into a sequence of language-model-sized soft prompts, using a 12×12 CLIP feature map flattened into 144 prompts.
  • Training procedure: The projection is trained on image captioning to align the image and language representation spaces.
  • Baselines: The study compares pretrained and tuned NFRN50 baselines with MAGMA, which tunes the CLIP encoder, GPT-J, and attention-block adapters.
  • Limitations: Prompt length is not controlled across encoders because computational constraints led to substantially larger k values for CLIP and BEIT.
  • Limitations: The analysis includes a blind visual-question-answering model to control for possible language-model runoff from generated concepts.
  • Qualitative evaluation: Example captioning and zero-shot VQA outputs illustrate transfer without tuning either model and show BEIT’s conceptually related but sometimes incorrect generations.

4 PERFORMANCE ON VISION-LANGUAGE TASKS

LiMBeR tests whether a single learned projection can connect frozen image representations to a language model for captioning and VQA. Performance generally tracks linguistic supervision, while BEIT transfers useful caption semantics but struggles with detailed visual-linguistic reasoning.

  • A single projection lets frozen image representations provide semantic image information that a language model can use for captioning and VQA.
  • Jointly tuning the image encoder and language model is not consistently better, and is often worse, than training the projection with frozen models.
  • CLIP outperforms NFRN50, which outperforms BEIT, linking stronger linguistic supervision during pretraining to better transfer into the language model.
  • BEIT still transfers well for captioning and far outperforms random NFRN50, despite having no linguistic supervision during pretraining.
  • BEIT captions remain semantically related but less lexically precise, and its 4-shot VQA score of 31.72% trails a blind language model's 36.99%.

5 TRANSFER OF VISUAL CONCEPTS

The transfer analysis separates lexical category information from coarse visual properties. Linguistically supervised encoders identify animal categories more accurately, whereas BEIT often produces semantically or perceptually related alternatives.

  • 5.1 TRANSFER OF LEXICAL CATEGORICAL CONCEPTS: For noun categories, BEIT matches or exceeds NFRN50 and CLIP on Wu-Palmer similarity in 4/5 categories despite lower exact noun recall.
  • 5.1 TRANSFER OF LEXICAL CATEGORICAL CONCEPTS: BEIT has lower noun recall than NFRN50 or CLIP but comparable Wu-Palmer similarity in many categories, indicating related rather than exact concepts.
  • 5.1 TRANSFER OF LEXICAL CATEGORICAL CONCEPTS: Linear probes show BEIT encodes fine-grained information less well than NFRN50 or CLIP, although it far outperforms randomly initialized NFRN50.
  • CLIP and NFRN50 correctly name animals in 59% and 43% of captions, compared with 13% for BEIT and 0.4% for random NFRN50.
  • 5.3 TRANSFER OF COARSE-GRAINED PERCEPTUAL CONCEPTS: Mistaken animal captions remain highly similar to the ground truth, with Wu-Palmer averages of 0.8 for BEIT, 0.81 for NFRN50, and 0.8 for CLIP.
  • 5.3 TRANSFER OF COARSE-GRAINED PERCEPTUAL CONCEPTS: NFRN50 and CLIP cluster by animal type, whereas BEIT clusters more by perceptual features such as habitat and flippers.

6 DISCUSSION & FUTURE WORK

The authors interpret LiMBeR's success as evidence that vision and language representations share an approximately linearly mappable structure. They also identify remaining differences as motivation for richer multimodal representations.

  • A linear projection can connect image representations to language-model inputs, revealing an underexplored representational similarity between vision and language.
  • BEIT transfers coarse perceptual information but struggles with lexical distinctions, while linguistic supervision helps image encoders transfer category information.
  • The representation spaces still contain differences that a linear map cannot approximate, motivating multimodal models that combine information from both modalities.
  • LiMBeR is positioned as a baseline for comparing how representations trained in different modalities are similar or different.

7 CONCLUSION

The paper tests whether pretrained image and text representations can be connected through a linear map, enabling frozen language models to process image representations. LiMBeR is competitive with jointly tuned models, but transfer quality depends strongly on linguistic supervision in the image encoder.

  • LiMBeR linearly projects image representations into a language model’s input space so the frozen LM can describe images.The method uses a learned projection without tuning the image encoder or language model.
  • LiMBeR models are competitive with MAGMA on image captioning and visual question answering benchmarks.MAGMA tunes both image and text networks, whereas LiMBeR uses the linear-transfer setup.
  • Transfer quality increases with linguistic supervision: BEIT underperforms classification-trained ResNet, which underperforms CLIP pretrained with natural-language captions.
  • Analysis finds that vision-only representations mostly transfer coarse perceptual concepts, whereas linguistically supervised encoders transfer more conceptual information.

8 REPRODUCIBILITY STATEMENT

The authors release training code and linear-projection weights to support reproducibility with frozen pretrained image and text models.

  • Training code and LiMBeR projection weights are released for reproducing the reported models.The frozen attached models require no additional tuning when using the released projection weights.

A.1 TRAINING DETAILS

LiMBeR trains a projection from image features to language-model prompts using paired images and captions. The training setup extracts encoder features, projects them into the LM space, and optimizes caption generation.

  • Each image-caption pair is encoded by an image encoder, whose features are passed through projection layer P for caption generation.For CLIP, 12×12×3072 feature patches are reshaped to 144×3072 before projection.
  • Models are trained for 15,000 steps on 16 A100 GPUs with effective batch size 2048 and projection learning rate 8 × 10^-4.Optimization uses AdamW with dropout probability 0.1, zero weight decay, and gradient clipping 1.0.
  • The image-encoder summary table defines prompt length as the number of tokens representing an image in the language model.

B CAPTIONING PERFORMANCE

Across captioning, representation analysis, visual question answering, and probing experiments, linguistically supervised image encoders generally transfer more detailed lexical information. BEIT still transfers useful coarse visual and semantic properties, but its category and domain-shift performance is weaker.

  • Captioning performance: Greater linguistic supervision improves captioning, while unfreezing the image encoder does not consistently produce significant gains.
  • Captioning performance: CLIP-based models perform best across SPICE subcategories, including relations, attributes, and objects.The appendix reports a CLIP score of 12.1 versus 9.28 for NFRN50 among LiMBeR models.
  • Captioning performance: BEIT transfers coarse visual and semantic properties but produces vaguer or semantically related incorrect object descriptions.
  • Representation analysis: The representation-similarity experiment compares the structure of image and caption encodings for shared concepts.
  • Visual question answering: VQA breakdowns show no significant question-type pattern, although NFRN50 may perform better on counting questions.The authors state that future work is needed to determine whether this counting trend is noise or significant.
  • Probe analyses: BEIT probe performance drops by 0.3 for animals under the CC3M-to-COCO domain shift, complicating comparisons between probing and transfer.Generated captions may also substitute common words such as “man” or “woman” for the label “person.”

E.3.2 FINE-GRAINED PROBES

Fine-grained probes evaluate image encodings on object-class information across COCO and CIFAR settings. Per-class results use 100 test images, with no significant within-model difference between coarse- and fine-grained probes.

  • FINE-GRAINED PROBES: Per-object-class F1 results use 100 images for each testing-set class.The reported analysis concerns image encodings evaluated through object-class probes.
  • FINE-GRAINED PROBES: No significant intra-model changes appear between coarse-grained and fine-grained probe results.This pattern is reported for the per-object-class CIFAR probe analysis and compared with the COCO probe.
  • FINE-GRAINED PROBES: Captioning F1 for object categories tends to follow image-encoder probe performance, although BEIT transfers much worse from CC3M to COCO.The figure text also notes that BEIT captioning F1 is consistently higher, making conclusions for that model difficult.
Loading 2209.15162v3…