PaperlayerView exact revision on arXiv

Source-linked AI summary

Steerable Visual Representations

Jona Ruthardt, Manu Gaur, Deva Ramanan, Makarand Tapaswi, Yuki M. Asano

arXiv:2604.02327v2cs.CVcs.AI

TL;DR

Existing ViTs provide broad visual features but tend to emphasize salient objects, while multimodal LLMs accept language guidance at the cost of language-centric representations. SteerViT injects text into frozen ViT layers through lightweight cross-attention, producing steerable visual features that preserve representation quality and match or exceed strong baselines across tasks without task-specific training.

  • Problem

    Generic ViT representations can overlook less prominent concepts, while language-guided multimodal representations may lose effectiveness for generic visual tasks.

  • Method

    SteerViT interleaves lightweight trainable cross-attention layers within frozen ViT blocks to condition visual encoding on natural-language prompts.

  • Results

    Across retrieval, personalized object discrimination, and industrial anomaly detection, SteerViT matches or outperforms strong baselines without task-specific training while preserving representation quality.

  • Takeaways & Limitations

    Text prompts enable steerable visual representations to transfer zero-shot across new domains, including personalized object discrimination and industrial anomaly detection.

Abstract

from arXiv · show

Pretrained Vision Transformers (ViTs) such as DINOv2 and MAE provide generic image features that can be applied to a variety of downstream tasks such as retrieval, classification, and segmentation. However, such representations tend to focus on the most salient visual cues in the image, with no way to direct them toward less prominent concepts of interest. In contrast, Multimodal LLMs can be guided with textual prompts, but the resulting representations tend to be language-centric and lose their effectiveness for generic visual tasks. To address this, we introduce Steerable Visual Representations, a new class of visual representations, whose global and local features can be steered with natural language. While most vision-language models (e.g., CLIP) fuse text with visual features after encoding (late fusion), we inject text directly into the layers of the visual encoder (early fusion) via lightweight cross-attention. We introduce benchmarks for measuring representational steerability, and demonstrate that our steerable visual features can focus on any desired objects in an image while preserving the underlying representation quality. Our method also matches or outperforms dedicated approaches on anomaly detection and personalized object discrimination, exhibiting zero-shot generalization to out-of-distribution tasks.

1 Introduction

SteerViT addresses the tension between generic but query-agnostic visual features and language-guided but language-centric multimodal representations. It conditions frozen ViTs on text through lightweight cross-attention, yielding steerable features while preserving representation quality and transferring without task-specific training.

  • Motivation: Generic ViTs support retrieval, classification, and segmentation but often prioritize salient objects over smaller or less prominent concepts.DINOv2 is described as focusing on a dominant cat while neglecting a remote control or bookshelf.
  • Desiderata: Steerable representations should adapt what they encode to text while retaining usefulness across diverse visual tasks.The paper frames steerability and representation quality as its two desiderata.
  • Method: SteerViT interleaves lightweight trainable cross-attention layers within frozen ViT blocks so text directly influences visual encoding.Referential image segmentation is used as the training objective to encourage vision-language alignment.
  • Results: 21M trainable parameters yield a Pareto improvement over prior approaches while preserving the underlying representation quality.The visual and text encoders remain frozen, and the added parameter count is two orders of magnitude below that of MLLMs.
  • Results: Across retrieval, personalized object discrimination, and industrial anomaly detection, SteerViT matches or outperforms strong baselines without task-specific training.The reported transfer includes DINOv2, SAM3, billion-scale MLLMs, and some dedicated methods.
  • Contribution: The framework equips pretrained visual encoders with text-steerable global and local features and supports zero-shot transfer to new domains.The contribution summary cites personalized object discrimination and industrial anomaly detection as examples.

2 Related Work

Prior visual representation families trade off visual quality, steerability, and parameter efficiency. SteerViT instead conditions vision on language to produce general steerable representations that transfer across tasks.

  • Visual representation families: Unimodal self-supervised encoders learn rich visual features but remain query-agnostic, while CLIP and SigLIP use text supervision without text-steerable visual encoders.The related-work comparison distinguishes independent image encoding from text-conditioned feature production.
  • Visual representation families: MLLMs offer moderate steerability and visual quality, but their features reside in language space and require billions of parameters.This positions MLLMs as close to the target desiderata but costly and language-centric.
  • Task-specific methods: TEVI edits CLIP’s final image features for text-to-image retrieval re-ranking, whereas SteerViT is designed as a general framework transferable across many tasks.The distinction is between a retrieval-specific feature-editing method and a broader steerable representation framework.

3 SteerViT: Steering Vision Transformers with Text

SteerViT equips pretrained ViTs with text-steerable visual representations by injecting language through gated cross-attention and training with referential patch segmentation. The architecture freezes the original encoders while adding lightweight multimodal capacity and learns to route prompt information into visual patch tokens.

  • Architecture: A frozen RoBERTa-Large text encoder produces token embeddings that a trainable two-layer MLP projects into the visual embedding space.The projected text tokens serve as keys and values for vision-to-language cross-attention.
  • Architecture: SteerViT augments a pretrained ViT with lightweight cross-attention layers that fuse text prompts into visual patch tokens.The visual encoder remains frozen, while cross-attention layers are interleaved within Transformer blocks.
  • Gated cross-attention: Layer-specific tanh gates, initialized at zero, integrate cross-attention outputs into the ViT residual stream while preserving the frozen ViT at initialization.The gates receive learning signals and can move away from zero during optimization, gradually activating language conditioning.
  • Training objective: Referential segmentation trains the model to use a prompt by predicting which image patches correspond to the referred object or entity.Patch-level supervision uses soft cross-entropy over mask probabilities, avoiding pixel-level decoders and reducing training complexity.
  • Training data: Training uses referential segmentation and grounding data spanning diverse domains and textual styles, totaling 162k images and 2.28M image-text pairs.The mixture includes RefCOCO/+/g, Visual Genome, LVIS, and Mapillary Vistas.

4 Experiments

Experiments show that SteerViT uses text to redirect visual features toward queried concepts while preserving representation quality across retrieval, localization, and transfer settings.

  • CORE retrieval: SteerViT forms object-specific embedding clusters from prompts, whereas DINOv2 produces scene-level or non-separable clusters.Text conditioning reorganizes the embedding space around the queried concept.
  • CORE retrieval: 96.0% retrieval accuracy is achieved by early-fusion SteerViT, compared with 43.7% for vanilla DINOv2 on CORE.SteerViT also exceeds FLAIR’s 81.3% steerability by 14.7 points.
  • MOSAIC localization: 50.2% attention PR-AUC shows that SteerViT focuses on text-specified objects, compared with 14.3% for DINOv2.The effect holds for inconspicuous objects in a four-image mosaic, while the SteerViT [CLS] token remains frozen.
  • Representation quality: SteerViT preserves representation quality while steering, with a Pareto operating point at gate scaling factor ω=0.6 for DINOv2 and SigLIP.Early fusion outperforms late fusion across ViT families; removing the gate reduces FG-CLS, CORE, and PODS by 4.2, 1.4, and 11.0 points.
  • Representation quality: More detailed text prompts improve personalized representation quality and retrieval, demonstrating that conditioning controls feature granularity.The detailed-prompt representation can surpass supervised fine-tuned DINOv2.
  • Transfer and generalization: SteerViT transfers to OOD domains without task-specific training, matching or outperforming dedicated methods across personalized discrimination and anomaly detection.Text conditioning also reorganizes embeddings across semantic abstraction levels and clustering criteria.

5 Conclusion

SteerViT conditions frozen visual encoders on language through lightweight cross-attention, steering global and local features while preserving base representation quality. The conclusion presents text as a lightweight mechanism for extending visual encoders without fine-tuning.

  • Conclusion: SteerViT interleaves lightweight cross-attention layers into frozen ViT blocks to condition visual encoding on language.The approach adds approximately 21M trainable parameters.
  • Conclusion: SteerViT steers both global and local features with text while preserving the base ViT’s representation quality.The paper characterizes this as a Pareto improvement over prior approaches.
  • Conclusion: Text prompts let SteerViT match or surpass dedicated methods on personalized object discrimination and industrial anomaly segmentation without task-specific training.The result generalizes across multiple ViT backbones.
  • Conclusion: Human gaze patterns shift with task-relevant textual prompts, providing inspiration for SteerViT’s early fusion of language into visual encoding.The figure compares gaze when locating a person in a tower window with gaze under no contextual instruction.

B Additional Qualitative and Quantitative Results

Additional results detail the CORE benchmark and show that SteerViT’s prompt-conditioned global features outperform query-agnostic, multimodal, and specialized alternatives across controlled scenes.

  • CORE benchmark: CORE pairs six SUN397 scenes with five contextually inpainted objects per scene to test prompt-conditioned one-vs-all retrieval.The benchmark evaluates whether global features retrieve images sharing both scene and queried object.
  • CORE results: SteerViT performs best across all indoor and outdoor CORE scenes with the correct prompt, while incorrect prompts significantly reduce performance.The benchmark therefore tests whether retrieval responds to the specified object rather than scene similarity alone.
  • Prompt sensitivity: Incorrect-object conditioning sharply lowers SteerViT accuracy, whereas CLIP and SigLIP change negligibly under the same manipulation.This contrast indicates that text actively shapes SteerViT’s visual representation.
  • Qualitative retrieval: SteerViT’s retrieval features incorporate small background objects, unlike DINOv2, which mainly retrieves by global scene appearance.The qualitative comparison covers one indoor and one outdoor scene.

B.1.2 Steerability transfers to real-world conditional retrieval.

SteerViT’s text-driven steering transfers beyond controlled inpainting to real-image conditional retrieval, while representation divergence reveals backbone-dependent integration of text across ViT layers.

  • Real-world retrieval: GeneCIS evaluates zero-shot conditional retrieval in real images, requiring both reference-scene similarity and presence of the queried object.Its gallery includes distractors sharing either scene or object characteristics.
  • Layerwise analysis: Representation divergence from vanilla ViT is non-zero in early-to-mid layers and grows most strongly in later blocks.The profile differs by backbone: SteerDINOv2 departs sharply late, while SteerMAE diverges gradually.
  • Real-world retrieval: SteerViT’s qualitative rankings account for target objects in real-image scenes, whereas DINOv2 remains biased toward scene similarity.The comparison appears in the Focus Object split.

B.2 MOSAIC Benchmark

MOSAIC tests whether text can redirect [CLS] attention from dominant regions toward prompted objects across a four-image collage. SteerViT selectively localizes specified objects, including multiple instances, while text attributes refine entity selection.

  • B.2 MOSAIC Benchmark: MOSAIC measures whether text conditioning steers [CLS] self-attention toward patches containing a prompted object.The benchmark uses four padded PASCAL-VOC images stitched into a 2×2 mosaic, with binary patch-grid masks for the prompted class.
  • B.2 MOSAIC Benchmark: DINOv2 primarily attends to the collage’s dominant object(s), whereas SteerViT attends to objects specified in the prompt.
  • B.2 MOSAIC Benchmark: Text conditioning can steer internal features progressively across the visual encoder’s layers.
  • B.2 MOSAIC Benchmark: Despite single-instance training, SteerViT localizes multiple occurrences of a prompted object class across the mosaic.The reported example shows attention to all instances of “chair” in the middle row.

B.3 Anomaly Segmentation

SteerViT is evaluated for zero-shot anomaly segmentation under an extreme industrial domain shift. It transfers effectively, matching or surpassing dedicated methods on MVTec AD and VisA, while remaining limited for defects without visible training evidence.

  • B.3 Anomaly Segmentation: SteerViT derives zero-shot anomaly maps by upsampling a learned linear segmentation head’s heatmaps from object-conditioned prompts.The evaluation ensembles ten prompts such as “the anomaly in the <object>.”
  • B.3 Anomaly Segmentation: 82.1 PRO on MVTec AD substantially exceeds SAM3 at 54.5 and CLIPseg at 34.6, while approaching FADE at 84.5.
  • B.3 Anomaly Segmentation: On VisA, SteerViT surpasses FADE in ROCP (92.1 vs. 91.5) and PRO (82.0 vs. 79.3).
  • B.3 Anomaly Segmentation: Predicted heatmaps are especially accurate for texture-based inputs but cannot reliably predict some defects lacking visible or non-anomalous training evidence.The passage gives flipped metal nuts and absent scratches as examples.

C.1 Training Objective: Pointing vs. Segmenting

The training objective determines whether SteerViT learns only object location or richer object appearance and extent. Segmentation supervision consistently outperforms pointing across downstream evaluations.

  • C.1 Training Objective: Pointing vs. Segmenting: Pointing supervises a soft target near the bounding-box center, whereas segmentation activates patch tokens overlapping the object.Pointing collapses supervision to one spatial location; segmentation teaches content matching across the object.
  • C.1 Training Objective: Pointing vs. Segmenting: Segmentation consistently outperforms pointing across all reported metrics.
  • C.1 Training Objective: Pointing vs. Segmenting: Segmentation yields gains of +7.3 on FG-CLS, +8.0 on ADE20k, and +12.4 on PODS over pointing.

C.2 Training Duration

Steerability emerges early, while deeper language-dependent capabilities continue improving with longer training. Larger visual and text encoders help, whereas adding a gated FFN increases parameters without improving representation quality and harms transfer.

  • C.2 Training Duration: Within 50k iterations, CORE reaches 95.3% versus 43.7 for frozen DINOv2, while FG-CLS remains 89.6 versus 89.0.
  • C.2 Training Duration: From 50k to 450k iterations, PODS improves from 49.9 to 58.1 and RefCOCOg from 63.4 to 70.6.The results suggest longer training improves deeper language understanding; final models use 500k iterations.
  • C.2 Training Duration: Scaling from ViT-S through ViT-B to ViT-L improves both representational quality and textual understanding.
  • C.2 Training Duration: Using one-fourth of the unique training samples preserves high steerability and feature fidelity.
  • C.2 Training Duration: Adding the gated FFN reduces CORE by 7.2 points and MVTec PRO by 3.4 points for MAE, while increasing adapter parameters by 67%.The FFN is omitted from the final architecture because it is empirically dispensable and harms steerability and transfer.

D.1 Training Data

SteerViT is trained on diverse grounding and segmentation data spanning varied expressions, scene complexities, and visual domains. The resulting dataset contains 162k unique images and 2.28M image-text pairs.

  • D.1 Training Data: The training mixture combines referential segmentation and grounding datasets to diversify visual domains and textual expression styles.RefCOCO, RefCOCO+, and RefCOCOg provide grounded referring expressions, with differences in spatial language and description length.
  • D.1 Training Data: RefCOCO+ excludes spatial language, while RefCOCOg uses longer descriptive expressions to exercise detailed textual understanding.
  • D.1 Training Data: Mapillary Vistas adds street-level imagery with fine-grained panoptic annotations, expanding the visual domain beyond COCO for out-of-distribution generalization.Synthetic referential expressions and accompanying segmentation masks are also included.
  • D.1 Training Data: 162k unique images and 2.28M image-text pairs cover scenes from single objects to dense urban panoramas, expressions from two-word labels to multi-sentence descriptions, and indoor, outdoor, and street-level domains.
Loading 2604.02327v2…