PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Black Boxes: Visual Grounding for Text Embeddings

Slimane Larabi

arXiv:2610.18047v1cs.AI

TL;DR

High-dimensional text embeddings are difficult to interpret, while their fixed-size representations can lose fine-grained information as text length increases. The paper maps SBERT caption embeddings to visually inspectable topographic sketches and reports modest retrieval gains from score fusion, though those gains are attributed to task-specific training rather than the sketch’s visual form.

  • Problem

    High-dimensional text embeddings are difficult to interpret, and fixed-size embeddings can lose fine-grained information as text length increases.

    Source: §1 · p. 2

  • Method

    The framework transforms caption representations into topographic sketches that can be visually inspected as an intermediate representation.

    Source: §1 · p. 2, §5 · p. 16

  • Results

    Score fusion improves Top-5 by +1.4 points and MRR@10 by +0.9 points, but the gains are attributed to task-specific training rather than the sketch’s visual form, and there is no evidence the sketch improves accuracy.

    Source: §4.4.2 · p. 16, §5 · p. 17

  • Takeaways & Limitations

    The sketch can serve as an interpretable intermediate that preserves about 97% of latent-space Top-5 retrieval performance, without evidence of an accuracy improvement.

    Source: §4.4.2 · p. 16

    Because sketches are derived from SBERT, they inherit its limitations; the authors propose end-to-end generation from raw text as future work.

    Source: §1 · p. 2, §5 · p. 17

Abstract

from arXiv · show

In this paper, our goal is to represent text embeddings as sketches for three main reasons. First, to visualize text representation and to explore their potential their potential for AI explainability. Second, to investigate the potential benefits of the sketch-based representation for information retrieval. Third, to explore their potential for reasoning, drawing inspiration from the concept of mental imagery. We propose a sketch-based visual grounding framework that transforms text representations into visual topographic sketches. Our approach aims to emphasis the semantics content encoded in the text embeddings through the corresponding sketch. Evaluations are conducted on the MS-COCO dataset to examine the utility of these sketches for text retrieval. The comparative study of SBERT and Sketch-based method reveals that SBERT retains an advantage in strict top-1 retrieval accuracy while achieving comparable performance to the sketch-based method in top-5 retrieval. A cascaded approach combining SBERT and the sketch-based method outperforms SBERT alone, suggesting that the Sketch-based representation could offer significant potential for information retrieval if trained directly on text.

1 Introduction

The paper targets the interpretability limits of high-dimensional text embeddings by making sentence representations visually inspectable as sketches. In retrieval, SBERT retains an advantage at Top-1, sketches approach it at Top-5, and combining the methods improves overall performance.

  • High-dimensional embeddings are difficult to interpret, hindering efforts to improve model explainability.Source: §1 · p. 2
  • The sketch inherits limitations of its SBERT input, including loss of fine-grained information as text length increases and sensitivity to minor variations or perturbations.The authors identify direct, end-to-end sketch generation from text as future work.Source: §1 · p. 2
  • Pure SBERT maintains an advantage in strict Top-1 precision, while the sketch-based representation closely approaches SBERT performance at Top-5 retrieval under comparable settings; combining SBERT and Sketch improves overall performance.Source: §1 · p. 2
  • The framework transforms sentence-vector representations, including image or video captions, into structured topographic sketches.Figure 1 illustrates sketches for two captions describing the same visual content.Source: §1 · p. 2, Figure 1 · p. 3

2 Review of SBERT

SBERT produces sentence embeddings using a bi-encoder architecture, allowing sentences to be encoded independently rather than jointly. The review describes its training objectives for semantic relationships and similarity.

  • SBERT modifies pretrained BERT with Siamese and triplet network structures to derive semantically meaningful sentence embeddings.Source: §2 · p. 3
  • Unlike a cross-encoder that processes sentence pairs together, SBERT encodes sentences independently with shared BERT weights, enabling precomputation and indexing.The passage contrasts O(n) transformer passes for SBERT with O(n^2) for finding the most similar pair using standard BERT.Source: §2 · p. 3
  • SBERT fine-tuning uses classification, regression, or triplet objectives to structure embeddings for semantic relationships or similarity.The triplet objective pulls anchor-positive embeddings together and pushes anchor-negative embeddings apart.Source: §2 · p. 4

3 Mapping SBERT Embeddings to Sketches

The framework maps SBERT caption embeddings to topographic sketches and re-encodes those sketches as retrieval features. Joint training aligns captions describing the same scene while encouraging spatial variation in the generated sketches.

3 Mapping SBERT Embeddings to Sketches

  • The mapping is motivated by mental imagery, with goals of visualizing embedding content, evaluating sketch-based retrieval, and exploring reasoning; only the first two are investigated.Source: §3 · p. 4

3.1 Semantic Text Encoding

  • A pretrained SBERT model encodes each caption as a 384-dimensional vector, which an MLP transforms into a layer-normalized latent vector for the decoder.The MLP uses a 256-dimensional hidden layer, ReLU, and dropout at rate 0.1.Source: §3.1 · p. 5

3.2 Topographical Sketch Generation (Decoder)

  • The sketch decoder expands the latent vector into a spatial canvas, then upsamples and convolves it into a single-channel 416 × 800 grayscale density map.A sigmoid bounds pixel intensities between 0 and 1.Source: §3.2 · p. 5

3.3 Visual Feature Re-Encoding

  • A sketch encoder extracts visual features from the generated sketch and projects them to a 384-dimensional vector used for retrieval.The encoder pools its feature map to a fixed 4 × 8 grid before the linear projection.Source: §3.3 · pp. 5–6

3.4 Joint Optimization Objective

  • Joint optimization combines contrastive alignment of same-scene caption features with a spatial-variation loss to discourage uniform or blank sketches.The overall objective adds the negative spatial variance terms to the contrastive loss, weighted by λ = 0.01.Source: §3.4 · pp. 6–7

4.1 Semantic Information Coded by Sketches

The sketches retain shared semantic content while reflecting changes in captions, including differences in setting, subject, and wording. Captions describing the same image with different wording produce visibly similar sketches.

4.1 Semantic Information Coded by Sketches

  • For captions of a man running, sketches share information for “man is running” while varying with location.The paper describes this pattern for captions changing the setting from a house to a garden or street.Source: §4.1 · p. 8
  • Replacing “man” with “girl,” “boy,” or “elderly man” changes details but retains the global layout, suggesting a common notion of a person.Source: §4.1 · p. 8
  • Dog-running sketches are visually different from the sketches for the man-running captions.Source: §4.1 · p. 8, Figure 4 · p. 9
  • Five captions describing the same image yield visibly similar sketches despite differences in wording.Source: §4.1 · pp. 8–9, Figure 5 · p. 9

4.2 Sketch Variation Under Caption Perturbation

Sketch similarity remains high across the tested caption perturbations, with syntactic reordering most similar to the original and character-level typos least similar. A caption describing a different scene has much lower sketch similarity.

4.2 Sketch Variation Under Caption Perturbation

  • Sketch similarity is computed by extracting 384-dimensional features, applying L2 normalization, and taking cosine similarity.Source: §4.2 · p. 10
  • 0.9920 similarity for syntactic reordering was highest, while 0.8405 for typo and noise perturbation was lowest.The other reported scores are 0.9763 for lexical perturbation and 0.9581 for partial omission.Source: §4.2 · pp. 9–10
  • The authors describe the visual decoder as an abstraction layer that filters word-level variations while maintaining spatial and structural consistency.Source: §4.2 · p. 10
  • A caption about a black cat jumping a fence has sketch similarity of 0.2356 to the original dog-running caption’s sketch.Source: §4.2 · pp. 9–10

4.3 Sketch vs. SBERT Caption-to-Caption Retrieval

The section compares sketch-based caption retrieval with pure SBERT on MS-COCO. Across the reported configurations, SBERT remains strongest, though the sketch features approach it more closely at Precision@5 than at Precision@1.

4.3.1 Experimental Setup

  • The evaluation retains MS-COCO groups with at least three captions and trains the sketch pipeline on subsets of 1,000, 10,000, 20,000, or 30,000 groups.Source: §4.3.1 · p. 10
  • For each evaluation, one caption per sampled group is held out as a query and ranked against captions from the remaining groups; correctness requires a match from the same group.The evaluation samples 500, 1,000, or 5,000 groups; increasing N increases both queries and gallery distractors.Source: §4.3.1 · pp. 10, 12
  • Pure SBERT uses pretrained all-MiniLM-L6-v2 embeddings, while the sketch method maps SBERT embeddings through a semantic encoder, sketch decoder, and convolutional encoder.Source: §4.3.1 · p. 12

4.3.2 Main Results

  • SBERT remains strongest in every configuration, while sketch features place the correct group among the top five almost as reliably but rank the exact best match less precisely.Source: §4.3.2 · p. 12

4.4 Evaluation of SBERT and Skertch in cascade

On MS-COCO caption retrieval, this evaluation compares SBERT, sketch and latent representations, and their combinations. Score fusion improves Top-5 and MRR over SBERT, whereas Sketch alone trails SBERT and the sketch’s interpretability is not accompanied by evidence of accuracy improvement.

4.4 Evaluation of SBERT and Skertch in cascade

  • Evaluation uses COCO caption quintets with held-out queries and a strict same-group correctness criterion; five seeds and paired group-bootstrap intervals assess performance.The strict ground truth can underestimate absolute scores because captions from other images may still be semantically valid matches.Source: §4.4 · pp. 12, 14
  • The study compares SBERT, Sketch, Latent, and an untrained-sketch control, using validation-selected score fusion and a threshold cascade that falls back to Sketch.The untrained control yields no gain, the threshold cascade is equivalent to SBERT, and an oracle indicates partly complementary errors.Source: §4.4 · pp. 13–14, §4.4.1 · p. 15

4.4.1 Results

  • Sketch alone trails SBERT by 13.5 points at Top-1 and 12.6 at Top-5, while Latent is slightly better than Sketch by 1.6 and 1.3 points, respectively.At Top-5, Sketch retains about 97% of Latent performance.Source: §4.4.1 · pp. 14–15
  • Score fusion improves SBERT by 1.36 points at Top-5 (59.59 vs. 58.23) and 0.87 at MRR@10; the +0.59 Top-1 gain is only a trend.The Top-5 gain is positive across all five seeds, while the Top-1 result does not survive Bonferroni correction.Source: §4.4.1 · p. 15, Table 3 · p. 15

4.4.2 Conclusions

  • The sketch is an interpretable intermediate that preserves most Latent Top-5 performance, but the results provide no evidence that it improves accuracy.The conclusion attributes the fusion gain to a learned COCO-trained model complementing a generic encoder, not to the visual sketch itself.Source: §4.4.1 · p. 15, §4.4.2 · p. 16

5 Conclusion

The framework makes SBERT caption embeddings visually inspectable as topographical sketches, but retrieval results show trade-offs rather than an accuracy advantage. Linear score fusion yields modest retrieval gains, while direct sketch generation from raw text remains future work.

5 Conclusion

  • The framework treats sketches as a structural information bottleneck that prioritizes scene-level composition and spatial-layout distributions.Source: §5 · p. 16
  • SBERT leads strict top-1, sketches retain approximately 98% of SBERT top-5 performance, and linear fusion adds +1.4 Top-5 and +0.9 MRR@10 points.Fusion gains were consistent across evaluation seeds.Source: §5 · p. 17
  • Control and ablation studies indicate fusion gains stem from task-specific training, since the latent representation alone achieves comparable gains.Source: §5 · p. 17
  • Sketches provide visual transparency into spatial-layout encodings and serve as an effective mechanism for candidate shortlisting.Source: §5 · p. 17
  • Future work targets end-to-end models that generate sketches directly from raw text to address limitations inherited from intermediate SBERT embeddings.Source: §5 · p. 17

Related papers

Loading 2610.18047v1…