PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-modal Steering Vectors Exist Across Both Image and Text Generation

Donghao Zeng, Nishant Subramani, Tamar Rott Shaham, Benno Krojer

arXiv:2610.36269v1cs.CVcs.CLcs.LG

TL;DR

The paper asks whether concepts and control directions are shared across modalities, given evidence that text and image representations can be separated. It tests zero-shot cross-modal steering in unified multimodal models and finds transfer in both directions, with vectors for the same concepts often pointing in similar directions.

  • Problem

    Whether a steering direction discovered in one modality can causally control generation in another remains open, despite evidence of separation between text and image representations.

    Source: §1 · pp. 1–2

  • Method

    The authors extract steering vectors from minimal pairs in one modality and apply them without adaptation to tokens of either modality in three unified multimodal models.

    Source: §1 · pp. 1–2, §2.2 · p. 3

  • Results

    Steering transfers in both directions across the three models for most concepts; in UniAR, transferred vectors recover 84% of the intra-modality effect on images and 74% on text.

    Source: §1 · p. 2, §3.2 · p. 5

  • Takeaways & Limitations

    Despite a modality gap, vectors for the same concepts often point in similar directions, and closer directions facilitate transfer.

    Source: §1 · p. 3

    The study is limited to internal steering directions in open-weight unified models and synthetic minimal pairs, requiring shared generation weights across modalities.

    Source: §2.1 · p. 3, ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Are concepts and mechanisms unified across modalities in current AI models, or instead implemented separately for each one? We operationalize this broad question of modality unification by studying cross-modal steering: 1) finding vectors that successfully steer generation on tokens of one modality, which then 2) transfer zero-shot to steer generation when added to tokens of another modality. Our study goes beyond simple correlational alignment (e.g. representational similarity) because steering is fundamentally causal. We study cross-modal steering in several unified multimodal models (UMMs), since they possess shared parameters for language and image generation, focusing on these two modalities. Across diverse steered concepts such as emotion and object size, we show that steering vectors transfer zero-shot in both directions (text-to-image & image-to-text). Even for concepts that are highly unique to one modality (e.g. formal vs. colloquial language), steering vectors often induce a consistent, interpretable effect in the other modality, such as formal clothing and design. This transfer has a geometric signature too: despite a clear modality gap separating text and image representations, steering vectors with strong cross-modal transferability tend to point in similar directions for the same concept. Our findings shed light on how vision and language interact inside unified multimodal models and point towards more general, modality-agnostic approaches for controlling their behavior.

1 INTRODUCTION

The paper asks whether a steering direction found in one modality can causally control generation in another, beyond correlational representational alignment. Across three unified multimodal models, steering transfers in both directions, with interpretable cross-modal effects and a geometric signature.

1 INTRODUCTION

  • The question addresses whether steering directions are modality-specific or can control generation across modalities, extending beyond correlational alignment to causal intervention.Prior work cited in the paper reports near-orthogonal text and image representations, mostly unsuccessful text-side knowledge edits in images, and often disjoint task circuits.Source: §1 · pp. 1–2
  • The study tests whether a vector built only from one modality’s tokens can steer generation zero-shot when added to the other modality’s tokens.The authors call this cross-modal steering; Figure 1 depicts transfer without adaptation.Source: §1 · p. 1, Figure 1 · p. 2
  • The authors use unified multimodal models because mainstream paradigms separately parameterize image-to-text and text-to-image generation, while shared parameters alone do not guarantee shared mechanisms.They note prior reports of asymmetric transfer, with text-derived vectors transferring better to images than image-derived vectors do to text.Source: §1 · p. 2
  • Across three unified multimodal models, steering vectors transfer in both directions, with UniAR vectors recovering 84% of the intra-modality effect on images and 74% on text.Text-derived vectors sometimes steer images better than image-derived vectors in Emu3.5 and Liquid.Source: §1 · p. 2
  • Despite a modality gap between text and image representations, vectors for the same concept often point in similar directions, and closer directions facilitate transfer.The paper presents this as a geometric signature of cross-modal steering.Source: §1 · p. 3
  • Cross-modal steering reveals shared causal control directions in modality-specific representations and motivates modality-agnostic control of multimodal generation.Examples include formal-language vectors yielding more formal visual clothing or design, and low-resolution image vectors yielding shorter, more uncertain text.Source: §1 · pp. 2–3

2 CROSS-MODAL STEERING: METHODOLOGY

The methodology tests steering in unified multimodal models whose text and visual tokens share generation weights and an autoregressive stream. It builds contrastive vectors from minimal pairs and applies them, without target-modality adaptation, across selected layers and generation tokens.

2.1 PRELIMINARIES

  • The study defines UMMs as models generating language and visual content in one autoregressive stream with shared generation weights, and tests UniAR, Emu3.5, and Liquid.The models differ in image tokenization: UniAR uses semantic tokens from a language-aligned vision encoder, while Emu3.5 and Liquid use tokens trained to reconstruct pixels.Source: §2.1 · p. 3
  • Steering adds a vector to intermediate hidden states to modulate generation; its strength and sign are controlled by a scalar, while common discovery and application practices differ between language and image generation.Language vectors are commonly found from contrastive pairs or probes and applied during decoding; image directions are often discovered in diffusion latent spaces or learned as adapters and applied during denoising.Source: §2.1 · p. 3

2.2 OUR APPROACH

  • The approach builds vectors from minimally different inputs, then applies the same vector without target-modality data in any of four settings: txt→txt, txt→img, img→txt, and img→img.The two cross-modal settings are the transfer conditions; the within-modality settings serve as references.Source: §2.2 · p. 3
  • For each concept, the method extracts representations from paired inputs that differ in the target pole and computes the steering vector as their average representation difference.Each layer’s vector is normalized to unit length so a single coefficient sets comparable steering strength across layers and concepts.Source: §2.2 · p. 4
  • Text pairs are short declarative prompts differing only in the final attribute token, while image pairs edit a shared baseline toward opposite attribute values while preserving scene features.Text uses the last-token hidden state; image representations are mean-pooled across image tokens.Source: §2.2 · p. 4
  • Steering is added to generated tokens in the shared transformer, using layer ranges and a swept strength coefficient; evaluations cover eight concepts, three layer regions, and 20 prompts per concept.The intervention applies identically to text or image tokens, regardless of the vector’s source modality; layer ranges are chosen to show where modalities align while avoiding all-layer steering.Source: §2.2 · p. 4

3 ARE STEERING DIRECTIONS SHARED?

Cross-modal steering is evaluated by comparing the effect of transferred vectors with native vectors, then tested across concepts, modalities, and model representations. Transfer works in both directions, including for some modality-specific concepts, while vector alignment varies across models and concepts.

3.2 RESULTS

  • In all three models, vectors transfer in both directions for most concepts; in UniAR, middle-layer transfer recovers 84% of image and 74% of text intra-modality steering effects.For every concept except spatial, text-derived vectors recover over 80% of the intra-modality effect at some layer range in all three models.Source: §3.2 · p. 5, Figure 3 · p. 7
  • Text-to-image transfer is stronger: Emu3.5 and Liquid average recovery of 1.10 and 1.20, while image-to-text transfer is widely successful only in UniAR.The authors hypothesize that the high text-to-image recovery in Emu3.5 and Liquid mainly reflects weaker image-derived vectors; their middle-layer image-to-image AUCs are 0.44 and 0.43, versus 0.83 in UniAR.Source: §3.2 · p. 6
  • Emotion transfers most evenly, averaging 0.95 recovery in both directions; size averages 1.12 for text-to-image but 0.32 for image-to-text, while spatial transfer mostly fails.Age transfers into images at 1.02–1.87 across models and layer ranges.Source: §3.2 · p. 6
  • Text fluency drops sharply beyond the strength where concept rate peaks, whereas image coherence remains about 90% of baseline at 0.6 strength, with concept rate already above 90% of its maximum.Text fluency is about three quarters of unsteered output at 0.4 strength, where text concept rate peaks.Source: §3.2 · p. 6

3.3 QUALITATIVE ANALYSIS: MODALITY-SPECIFIC CONCEPTS

  • Language-derived formality and English–French vectors induce visual shifts, including formal clothing and design, ornate French-associated styles, and French-looking but nonexistent words.The language minimal pairs define linguistic formality rather than visual attributes such as suits versus sportswear.Source: §3.3 · pp. 6–7
  • Image-derived resolution and motion-blur vectors alter text detail and wording, while a camera-angle vector steers images cleanly but changes text only marginally.The text effects include shorter sentences and calmer or faster movement-related descriptions; other steering vectors do not produce comparable length reductions.Source: §3.3 · p. 7

3.4 GEOMETRY OF STEERING VECTORS

  • Cosine similarity between modality-derived vectors correlates moderately with transfer recovery overall (Spearman ρ = 0.38, Pearson r = 0.22), but the relationship varies by model and direction.For UniAR, the strongest reported correlation is for text-to-image transfer (ρ = 0.76).Source: §3.4 · p. 8, Figure 5 · p. 8
  • Text and image activations occupy separate narrow spaces, yet steering directions align for five of eight concepts, with cosine similarities of 0.14–0.60 versus a random-direction baseline of ±0.03.In UniAR middle layers, the average happy image is more than ten times closer to the average sad image than to the average happy sentence.Source: §3.4 · pp. 8–9

4 RELATED WORK

Prior work studies vision-language alignment and multimodal internals, but reports mixed evidence for unification. This paper differs from related steering studies by testing both transfer directions in free-form text and image generation within unified models.

Unification and Alignment of Modalities

  • Earlier studies of unified-model internals often report little unification, whereas Wang & Li (2026) report cross-modal steering only from text to images.Their work uses BAGEL with separate language and image generation weights, minimal text inputs, and closed-form QA outputs rather than free-form text.Source: §4 · p. 9

Visual steering

  • Related visual-steering work examines one transfer direction at a time or targets image-conditioned MLLMs and diffusion models.Steering the visual-token stream of an autoregressive image generator remains rarely explored.Source: §4 · p. 9

Textual steering

  • Textual steering uses concept directions in language-model latent spaces, commonly derived from contrastive prompt pairs, task demonstrations, or sparse-autoencoder features.These approaches are grounded in the linear representation hypothesis.Source: §4 · p. 9

5 CONCLUSION

The study demonstrates text-to-image and image-to-text steering in unified multimodal models. It points to modality-agnostic control and interpretability as potential directions, while calling for tests across more model families and concepts.

5 CONCLUSION

  • Cross-modal steering vectors work in both text-to-image and image-to-text directions, motivating future modality-agnostic control and interpretability methods for unified models.The authors suggest safety interventions or knowledge edits may transfer to visual capabilities, and encourage testing other model families and concepts.Source: §5 · p. 9

ETHICS STATEMENT

The work studies internal steering directions in open-weight unified multimodal models using synthetic minimal pairs, and identifies general risks associated with controllable generation.

ETHICS STATEMENT

  • The study is limited to internal steering directions in open-weight unified multimodal models evaluated on synthetic minimal pairs.The authors foresee no direct ethical concerns beyond general controllable-generation risks, including harmful content targeting or exploiting vulnerable populations.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

The authors used AI tools across experiments, data and evaluation, and writing, while reviewing outputs and retaining responsibility for the paper. AI also assisted with coding, but the authors did not inspect agent-written code line by line.

AI USE STATEMENT

  • Coding agents implemented methods, ran most experiments, and helped clean and reformat generated data.Authors discussed code and outputs, reran experiments, and sanity-checked results, but did not review agent-written code line by line.Source: AI USE STATEMENT · p. 10
  • AI tools contributed to methodological discussions and interpretation of qualitative patterns, which authors checked against underlying numbers and generations.The authors report a back-and-forth process and say they did not blindly adopt suggestions.Source: AI USE STATEMENT · p. 10
  • AI models generated minimal pairs and evaluation prompts, which authors inspected before use; Gemini 3.7 Flash judged all reported metrics.Agreement with human raters is reported in Appendix A.8.Source: AI USE STATEMENT · p. 10
  • AI tools helped create figures and identify and summarize related work, while the authors retained primary responsibility for the final paper and checked suggested citations against original sources.They used some isolated sentence-level writing suggestions, more sparingly than AI assistance for coding and experiments.Source: AI USE STATEMENT · p. 10
  • The authors did not use AI to develop theoretical models or formulate or prove mathematical claims, and reviewed all AI-assisted content.They take responsibility for the final work.Source: AI USE STATEMENT · p. 10

A EXPERIMENTAL DETAILS

The experiments construct concept-specific steering vectors from text or image minimal pairs and evaluate their effects under controlled prompts, layer configurations, steering strengths, and blinded pairwise judgments. They also assess how image-concept DreamSim pole scores change with steering strength across models and concepts, including text-derived vectors.

A EXPERIMENTAL DETAILS

  • The study selects language, image-editing, scene-level, and visual-property concepts, anticipating that spatial steering will be hardest.Examples include emotion, age, expression, cleanness, chaos, size, depth, position, and color.Source: A EXPERIMENTAL DETAILS · p. 19

A.1 MINIMAL PAIRS

  • Vectors are mean differences across minimal pairs; text pairs use 100 subjects with only the pole word changed, while image pairs preserve a scene and change the pole.For depth, size, and spatial edits, the study uses Gemini 3 Pro Image and Reve because Gemini 2.5 Flash Image often gets these edits wrong.Source: §A.1 · pp. 19–20

A.2 EVALUATION PROMPTS

  • Each concept has 20 neutral evaluation prompts; image generation uses the full scene description, whereas text generation uses only its subject.The prompts are reused across layer configurations and steering strengths.Source: §A.2 · pp. 23–24

A.3 GENERATION AND DECODING

  • Text outputs use a shared open-ended instruction and steer the subject plus every generated token, while image outputs steer only generated visual tokens.Models retain their default decoding settings except for decoder steps and image resolution, with the seed fixed at 42.Source: §A.3 · pp. 24–25, Table 4 · p. 26

A.4 LAYER AND STEERING STRENGTH

  • The study tests seven layer configurations per model and uses a model- and setting-specific strength grid chosen to produce degenerate samples at its largest magnitudes.Configurations include early, middle, and late single layers and ranges, plus all layers; transferred vectors are injected at the layers from which they were extracted.Source: §A.4 · p. 26, Table 5 · p. 26

A.5 JUDGE SETUP AND PROMPTS

  • Gemini 3.7 Flash judges outputs as A, B, or Tie without seeing steering parameters, using held-out human-agreed examples and a randomly selected target pole.The judge is instructed to assess the attribute itself and choose a candidate only when the difference is clear.Source: §A.5 · p. 27

A.6 DREAMSIM POLE SCORE (IMAGES)

  • The pole score maps an image’s relative DreamSim distance to positive- and negative-pole references onto a 0–100 scale, with 50 equidistant.References are the model’s unsteered image minimally edited toward each pole with Gemini 3 Pro Image.Source: §A.6 · p. 28
  • DreamSim pole scores rise with steering strength for responsive concepts, including with text-derived vectors, while spatial stays at 50.Figure 7 shows this pattern for both image-output settings.Source: §A.6 · p. 29, Figure 7 · p. 29
  • UniAR shows the largest shifts, with age strongest; all concepts except spatial rise except size with the text-derived vector.Emu3.5 and Liquid shift only a few points: color moves most with both vectors, while age moves mainly with the text-derived vector. In Liquid, size and depth stay above 50 without moving, so their offset is not steering.Source: §A.6 · p. 29

A.7 FLUENCY AND COHERENCE JUDGES

The section describes automated judges for text fluency and image coherence, then reports how those quality measures change as steering strength increases. Text fluency collapses beyond the strength where concept rate peaks, while image coherence declines gradually.

Text judge

  • Gemini 3.7 Flash scores each text output from 0 to 2 for fluency, without seeing its prompt, strength, or layer.The evaluation covers 22,400 steered text outputs per model and one unsteered baseline per concept and prompt.Source: §A.7 · pp. 29–30

Image judge

  • For images, a judge sees only the image and rates visible breakage as intact, partly broken, or broken, mapped to scores 2, 1, or 0.The study scores 3,200 steered images per model at the middle layer range and 160 unsteered baselines.Source: §A.7 · p. 30

Results

  • Text fluency holds up to about 0.4 of the strength range, where concept rate peaks, then collapses; image coherence declines gently as concept rate rises.Transferred and intra-modality vectors degrade quality similarly, except image-derived vectors cost more text fluency than text-derived vectors in Emu3.5 and Liquid at middle strengths.Source: §A.7 · pp. 30–31, Figure 8 · p. 31

A.8 HUMAN VALIDATION

Human reviewers and the automated judge evaluate whether a steered output follows a randomly selected pole relative to its unsteered baseline. Their agreement across 320 held-out pairs is κ = 0.62.

Human annotations

  • Human validation covered 320 held-out pairs, with 10 per concept and setting across eight concepts, and three independent reviews per pair.The pairs were outside the evaluated sweeps; reviewers compared a steered output with its α = 0 baseline.Source: §A.8 · p. 31

Annotation interface

  • The interface presented the unsteered baseline and steered output as randomly ordered Candidate A and B, and reviewers chose A, B, or Tie.The asked pole and its opposite were named in the interface.Source: §A.8 · p. 31, Figure 9 · p. 32

Agreement

  • The automated judge and human majority labels agree with Cohen’s κ = 0.62 across 320 held-out pairs.Each pair received three independent reviews, with the asked pole and A/B order randomized; reviewers were blind to steering strength, layer, and which output was steered.Source: §A.8 · pp. 31–32

B DETAILED RESULTS

This section presents concept-rate curves across three models and reports cross-modal recovery by concept, model, and layer range. The figures compare transferred and intra-modality steering at middle layers and across seven layer configurations.

Middle-layer comparisons

  • Figures 10–12 compare transferred and intra-modality vectors at the middle layer range for UniAR, Emu3.5, and Liquid, respectively.Figure 10 repeats UniAR for comparison; its black curve is intra-modality and its colored curve is transferred, with shading marking their AUC difference.Source: B DETAILED RESULTS · p. 32, Figure 10 · p. 33, Figure 11 · p. 34

Cross-modal recovery

  • Table 6 defines cross-modal recovery as transferred-vector AUC divided by intra-modality concept-rate AUC; values above 1.0 indicate stronger transferred steering.Cells are blank when intra-modality AUC is below 0.2, and gray marks recovery below 0.5; bold indicates the mean of the best layer range per model and setting.Source: Table 6 · p. 33

Sweep setup

  • Each sweep generates and judges outputs for every combination of concept, setting, layer configuration, strength, and prompt against an unsteered baseline.Across the figures, the middle column is the unsteered baseline and other columns show steering strength normalized to that sweep’s largest value.Source: B DETAILED RESULTS · p. 33

Layer configurations

  • Figures 13–15 show all seven layer configurations for UniAR, Emu3.5, and Liquid, respectively.Their rows represent concepts and columns represent settings; curves distinguish layer configurations and each starts at rate 0 when α = 0.Source: Figure 13 · p. 35, Figure 14 · p. 36, Figure 15 · p. 37

C DETAILS FOR: GEOMETRY OF STEERING VECTORS

This appendix details how the geometry of steering vectors is compared across modalities, how activations are centered for modality-gap analysis, and how modality-unique concepts were selected and tested.

C DETAILS FOR: GEOMETRY OF STEERING VECTORS

  • The geometry analysis compares per-layer text and image steering-vector cosines with cross-modal steering success, while separately examining modality gaps and concept poles.Tables 7–9 cover correlations, anisotropy, and same-pole versus same-direction comparisons; Figure 16 shows steering-vector and activation cosines.Source: C DETAILS FOR: GEOMETRY OF STEERING… · p. 38, Table 7 · p. 38, Table 9 · p. 39
  • Centering uses unit-length activations and subtracts the midpoint of text and image means; in UniAR, middle-layer random-activation cosines remain high after centering.Before centering, mean cosine is 0.75 for text pairs, 0.74 for image pairs, and 0.65 across modalities; the steering-vector difference itself is unchanged by centering.Source: C DETAILS FOR: GEOMETRY OF STEERING… · p. 38, Table 8 · p. 39, Figure 16 · p. 39

D CONCEPTS UNIQUE TO ONE MODALITY

D.1 ALL ATTEMPTED CONCEPTS

  • Candidate pairs use 100 template-matched text examples or programmatic image operations and paired edits, then must steer their source modality before cross-modal testing.Two image concepts failed at pair construction; operations or small camera moves worked better than edits requiring a scene to be reimagined.Source: §D.1 · p. 40, Table 10 · p. 40

D.2 EXAMPLES FOR THE CONCEPTS THAT WORKED

Examples show that steering can produce interpretable visual changes from text-derived concepts and text changes from image-derived concepts, though some effects become disruptive at high strength.

D.2 EXAMPLES FOR THE CONCEPTS THAT WORKED

  • Text-derived formality changes generated scenes toward formal clothing and design, while stronger steering can wash images out or make them illustrative.The prompts do not mention clothing or language register.Source: Figure 17 · p. 41
  • English–French steering shifts generated scenes toward French or English cultural conventions, but stronger French steering can make images painterly.At moderate strength, French-associated objects remain photographic.Source: Figure 18 · p. 42, Figure 19 · p. 43
  • Camera-height steering changes image viewpoint while keeping the subject fixed, but its effect on free text is weak and does not override the preferred answer.The image examples move between a top-down view and near-ground level; text steering changes next-token probabilities but leaves the most likely token a function word.Source: §D.2 · p. 43, Figure 20 · p. 44
  • Motion-blur steering changes text descriptions toward calmer or faster movement, while image steering first blurs the background and then the full frame.Toward sharp, images gain detail; the text forced-choice example moves “the runner looks” from 6% to 99% blurry.Source: §D.2 · p. 44, Figure 21 · p. 44
  • Low-resolution steering shortens and simplifies text, while image outputs lose detail, become pixelated, and eventually collapse at high steering strength.The cited examples compare generated descriptions and show the low-resolution image progression.Source: §D.2 · pp. 44–45, Table 11 · p. 45, Figure 22 · p. 46

E AVERAGING TEXT AND IMAGE STEERING VECTORS

Averaging text and image steering vectors does not improve on the better single-modality vector. The behind-the-scenes accounts describe model-selection challenges and reflect on the uncertain prospects of unified multimodal models.

E AVERAGING TEXT AND IMAGE STEERING VECTORS

  • No averaged text–image vector improves over the better single-modality vector beyond judge noise, and the best average ties the text vector at the per-pole maximum.The evaluation also finds the same pattern across layer ranges and on the image target, where rates saturate for most vectors.Source: E AVERAGING TEXT AND IMAGE STEERING… · p. 46, Table 12 · p. 46

F.2 SENIOR PERSPECTIVE

  • The project grew from interest in interpretability and vision-language models into a study of cross-modal steering as unified autoregressive models improved.The senior perspective identifies UniAR as a timely model for studying this question.Source: §F.1 · p. 47, §F.2 · p. 48

F.1 JUNIOR PERSPECTIVE

  • Early models were useful for smoke tests, but image-generation artifacts made rigorous conclusions difficult; finding a unified model with strong image generation was a major challenge.The authors describe settling on UniAR after earlier model limitations and encouraging initial results.Source: §F.1 · p. 47

F.2 SENIOR PERSPECTIVE

  • The authors describe unified models as not yet replacing mainstream VLMs or diffusion systems, while noting early evidence that they may not be internally unified.The senior perspective remains cautious about near-term impact but optimistic about the long-term direction.Source: §F.2 · p. 48
  • The senior perspective presents mentorship as a gradual balance of trust, increasing expectations, and stepping in when needed, especially for conference writing.The account also notes the value of adding other senior collaborators.Source: §F.2 · p. 49
Loading 2610.36269v1…