Source-linked AI summary
Incorporating Visual Semantics into Sentence Representations within a Grounded Space
Patrick Bordes, Eloi Zablocki, Laure Soulier, Benjamin Piwowarski, Patrick Gallinari
TL;DR
Sentence grounding is limited by one-to-one alignment assumptions, even though a visual scene can have many descriptions and captions can contain non-visual information. The paper learns an intermediate grounded space with cluster and perceptual objectives, and reports consistent gains over textual baselines across natural-language tasks, with larger improvements for more concrete sentences. Its approach transfers visual structure while relaxing constraints on the textual space.
Problem
One-to-one alignment between textual and visual representations is problematic because scenes admit paraphrases, captions can be ambiguous, and sentences can contain non-visual information.
Method
The model uses an intermediate grounded space trained with complementary cluster and perceptual objectives to transfer visual structure while preserving textual semantics.
Results
The grounded models systematically outperform the textual baseline on all SentEval benchmarks, while grounding gains are larger for more concrete sentences, reaching Δ=+43 for Captions.
Takeaways & Limitations
The approach reports consistent positive results against purely textual baselines across a variety of natural-language tasks.
Takeaways & Limitations
The cluster objective requires images associated with several captions and can push apart sentences describing closely related visual content.
Abstract
from arXiv · showhide
Language grounding is an active field aiming at enriching textual representations with visual information. Generally, textual and visual elements are embedded in the same representation space, which implicitly assumes a one-to-one correspondence between modalities. This hypothesis does not hold when representing words, and becomes problematic when used to learn sentence representations --- the focus of this paper --- as a visual scene can be described by a wide variety of sentences. To overcome this limitation, we propose to transfer visual information to textual representations by learning an intermediate representation space: the grounded space. We further propose two new complementary objectives ensuring that (1) sentences associated with the same visual content are close in the grounded space and (2) similarities between related elements are preserved across modalities. We show that this model outperforms the previous state-of-the-art on classification and semantic relatedness tasks.
1 Introduction
The paper argues that visual grounding can enrich sentence representations, but one-to-one cross-modal alignment is too restrictive for varied language descriptions. It therefore transfers visual structure through an intermediate grounded space using complementary cluster and perceptual information.
- Visual grounding enriches textual representations because text-only models can encode biased or unrealistic descriptions, while language understanding is linked to perceptual experience.
- Existing grounding methods implicitly assume one-to-one correspondence between textual and visual spaces, although images and sentences need not share the same neighborhood structure.
- The model transfers visual structure partially to text by combining cluster information from shared images with perceptual information from high-level image representations.
- A grounded space mediates this transfer, preserving textual semantics while avoiding an overconstrained textual representation.
- The paper evaluates these complementary objectives quantitatively and qualitatively across several transfer tasks against previous grounding methods.
2 Related work
Sentence representation research spans task-specific and universal text encoders, while language-grounding work adds perceptual information to address the mismatch between symbolic language and physical experience. Existing sentence-level approaches align captions directly with images, which can over-constrain textual representations because multiple sentences may describe the same visual content.
- Sentence representations: Sentence representations have been learned with recursive, convolutional, self-attentive, and large-corpus unsupervised methods.The cited examples include Socher et al. (2013), Kalchbrenner et al. (2014), Lin et al. (2017), Conneau et al. (2017), and FastSent (Hill et al., 2016).
- Language grounding: Language-grounding work argues that purely symbolic language models should incorporate physical and perceptual experience.This motivation is reinforced by differences between linguistic and real-world event frequencies, such as murder being mentioned more often than breathing.
- Word-level visual grounding: Visual grounding for words has used either separately learned textual and visual representations or jointly learned multimodal representations.These two lines include sequential techniques and joint methods drawing on multiple sources simultaneously.
- Sentence-level visual grounding: Caption-based sentence models train sentence encoders to predict associated visual features, but direct projection toward images can over-constrain the textual space.IMAGINET and the model of Kiela et al. (2018) align sentence representations with corresponding images; Collell and Moens (2018) report that cross-modal projections may not resemble the target modality in nearest-neighbor comparisons.
- Proposed model: The model overview depicts a cluster loss gathering visually equivalent sentences and a perceptual loss aligning cosine similarities across modalities.The contrastive term in L_C is omitted from the illustration, while the perceptual component is represented by the green arrow and angles.
3 Incorporating visual semantics within an intermediate grounded space
The model transfers visual structure to sentence representations through an intermediate grounded space, using complementary cluster and perceptual objectives while preserving textual semantics.
- Model overview: The approach addresses paraphrase diversity and visual ambiguity by grouping captions for the same image, preserving cross-modal similarity structure, and retaining non-visual textual information.The grounded space is designed to avoid degrading semantics supplied by the textual objective LT.
- Grounded space: The grounded space projects sentence representations into an intermediate space where visual grounding objectives can transfer visual structure without forcing one-to-one text-image correspondence.A projection g maps textual sentence embeddings into the grounded space, while textual and grounding objectives share sentence-encoder parameters.
- Cluster information: Cluster information brings sentences describing the same image closer than sentences associated with different images through a max-margin ranking loss.The loss uses a visually equivalent sentence s+ and a visually different sentence s− sampled for sentence s.
- Perceptual information: Perceptual information preserves similarities between related images and their associated sentences, addressing the tendency of cluster-only training to separate visually related content.This objective uses image content and aligns sentence-space similarity with visual-space similarity.
- Grounded loss: The combined grounding objective is a weighted linear combination of cluster and perceptual losses, with αC and αP controlling their contributions.The grounding-related parameters include the image encoder and the projection function g.
4 Evaluation protocol
The evaluation combines textual and visual corpora, grounding baselines and model variants, semantic and classification benchmarks, and intrinsic measures of grounded-space structure.
- Datasets: The study evaluates sentence representations using BookCorpus text and MS COCO images, with each COCO image paired with five English descriptions.BookCorpus contains 11K books and 74M ordered sentences, while MS COCO contains 118K/5K/41K train/validation/test images.
- Models and baselines: Grounding variants are built from SkipThought and include cluster, perceptual, and combined objectives with either an MLP grounded-space projection or an identity mapping.The experiments also include visual-only variants and compare against Cross-modal Projection, Sequential, and GroundSent baselines.
- Evaluation tasks: Semantic relatedness is measured on STS and SICK using correlations between embedding cosine similarities and human-labeled similarity scores.STS includes captions, news headlines, and forum posts, covering concrete and more abstract sentence sources.
- Evaluation setup: Table 1 compares intrinsic evaluations across grounded, textual, and visual spaces for the model scenarios and baselines.The table distinguishes MLP grounded-space evaluations from textual-space and visual-space evaluations.
- Evaluation tasks: Classification is evaluated with SentEval on sentiment, subjectivity, customer-review, paraphrase, and entailment tasks.The listed benchmarks include MPQA, MR, SUBJ, CR, SST, and MSRP, among others.
- Structural measures: Intrinsic analysis measures nearest-neighbor overlap, cross-modal similarity correlation, within-cluster homogeneity, and between-cluster separation.The structural measures are reported on the MS COCO validation set, using 5K images and 25K captions.
5 Experiments and Results
The experiments probe how grounded-space design transfers visual structure into sentence representations and evaluate the resulting embeddings qualitatively and on transfer tasks. Grounded models improve over textual and prior grounding baselines, with complementary cluster and perceptual objectives producing the strongest overall behavior.
- 5.1 Study of the grounded space: For Captions, grounding improves the semantic-relatedness score by Δ = +43 over T, whereas gains are smaller for News (Δ = +12) and Forum (Δ = +12).The reported pattern associates larger improvements with higher visual concreteness: Captions has c̄ = 3.10, compared with 2.61 for News and 2.39 for Forum.
- 5.1 Study of the grounded space: In qualitative analysis, grounded neighborhoods of a caption are mostly composed of sentences describing the query image or its nearest visual neighbor.The model also retrieves more accurate visual meanings for abstract-word queries than the purely textual model.
- 5.1 Study of the grounded space: Using only cluster information produces the highest Cintra and lowest Cinter, whereas using only perceptual information produces the highest ρvis but the lowest Cintra.These scenarios separate the effects of preserving local cluster structure from correlating textual and visual similarities.
- 5.1 Study of the grounded space: Combining cluster and perceptual information yields higher semantic relatedness and mNNO than corresponding single-objective or non-grounded variants.The combined model also improves both cluster structure, through higher Cintra and lower Cinter, and perceptual alignment, through higher ρvis.
- 5.2 Evaluation on transfer tasks: Grounded models systematically outperform the textual baseline T across all SentEval benchmarks, while prior GS grounding models perform worse than their same-dimensional textual baseline.This establishes the main transfer-task result, with evaluations conducted using 2048-dimensional sentence representations except T1024.
- 5.2 Evaluation on transfer tasks: The joint approach outperforms sequential grounding models across benchmarks, while T + Pg leads on most tasks but Cg-based models perform better on entailment tasks.T + Cg + Pg provides a balance between classification and entailment performance.
6 Conclusion
The paper presents a multimodal model that preserves visual and textual structure while learning grounded sentence representations. It combines perceptual and cluster information through an intermediate grounded space and reports consistent gains over purely textual baselines across natural-language tasks.
- The model preserves the structure of visual and textual spaces to learn grounded sentence representations.
- It combines perceptual and cluster information in an intermediate grounded space that relaxes constraints on the textual space.
- The approach reports consistent positive results against purely textual baselines across a variety of natural language tasks.
- Future work targets visual grounding for complex downstream tasks involving commonsense and reasoning, including question answering and visual dialogue.