PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Emotion Concepts Generalize Across Sources, Modalities, and Architectures in Vision-Language Models?

Bohao Xing, Xin Liu, Kaishen Yuan, Deng Li, Rong Gao, Guoying Zhao, Xiaolan Fu, Heikki Kälviäinen

arXiv:2609.34742v1cs.CVcs.LG

TL;DR

Prior evidence for structured emotion representations in language models has focused mostly on text and a single architecture, leaving their generalization across sources and modalities unclear. This paper constructs CMES and evaluates emotion vectors across sources and VLMs, finding shared relational structure and causal effects despite differences in vector directions.

  • Problem

    Most existing evidence for structured emotion representations is based on text and a single model, leaving unclear how emotion geometry and causal effects generalize across sources, modalities, and architectures.

    Source: §1 · pp. 1–2

  • Method

    The authors construct CMES, a seven-source collection covering six basic emotions, and analyze emotion vectors across three core VLMs before building a shared subspace.

    Source: §1 · pp. 2–3

  • Results

    Emotion vectors show held-out correspondence and causal effects across sources, modalities, and architectures despite low or near-zero native cosine similarity, and consensus vectors generalize to held-out Gemma-3 models.

    Source: §1 · p. 3, §7 · pp. 9–10

  • Takeaways & Limitations

    Cosine similarity before alignment gives an incomplete picture of generalization, because different VLMs can preserve shared emotion relationships and causal effects despite differing vector directions.

    Source: §7 · pp. 9–10

    The core analyses focus on three approximately 8B-scale VLMs, and generalization to substantially larger models and more diverse model families remains to be tested.

    Source: §G.1 · p. 59

Abstract

from arXiv · show

Recent studies suggest that large language models encode emotion concepts as structured internal representations, but most existing work focuses on text and a single architecture. Therefore, we ask, do emotion concepts generalize across sources, modalities, and architectures in vision--language models (VLMs)? To address this, we construct CMES (Cross-Modal Emotion Stimuli), a multi-source collection of emotion-conditioned stories, real facial expressions, synthetic portraits, and synthetic emotion-evoking scenes. For each stimulus source, we extract a separate set of six Ekman emotion vectors from each of three VLMs. We report four main findings as follows: 1) Image-derived emotion vectors form a low-dimensional geometry similar to that of text-derived vectors. Valence is relatively stable across sources, while arousal varies more. 2) Text- and image-derived emotion vectors have modest cosine similarity but still show held-out cross-modal correspondence. Text-derived vectors can also steer image interpretation. 3) Cross-architecture correspondence remains even when native cosine is near zero. Transformations estimated from generic ImageNet activations recover both correspondence and causal transfer without using the six emotion vectors or their labels. 4) After aligning representations across architectures, we construct a shared emotion subspace that preserves affective geometry and selective steering effects. The corresponding consensus emotion vectors also generalize to a held-out fourth architecture at two model sizes. These results suggest that emotion representations can share relational structure and causal effects across sources, modalities, and architectures, even when individual vector directions differ.

1 INTRODUCTION

The paper asks whether emotion representations in VLMs preserve structure and causal effects across stimulus sources, modalities, and architectures. It introduces CMES and reports shared relational structure despite differences in individual vector directions.

1 INTRODUCTION

  • Most prior evidence on structured emotion representations and emotion-vector interventions comes from text and a single model.The paper also notes that prior work has studied vector transfer for concepts and behavioral attributes across text LLMs, leaving emotion geometry and causal effects across architectures open.Source: §1 · pp. 1–2
  • The paper frames four questions: whether images yield structured emotion vectors, whether text and image vectors correspond, whether effects transfer across architectures, and whether a shared subspace generalizes.The map–territory analogy allows internal representations to differ in form while preserving common structure.Source: §1 · p. 2
  • CMES brings together text stories, real facial expressions, synthetic portraits, and synthetic emotion-evoking scenes organized around the same six basic emotions.The collection includes three independently generated story sources and multiple image sources, helping distinguish source effects from modality effects.Source: §1 · p. 2
  • Core analyses use Qwen3-VL-8B, InternVL3.5-8B, and Ministral-3-8B, and the reported consensus vectors also generalize to held-out Gemma-3 models.The Gemma-3-12B and Gemma-3-27B variants are excluded from consensus construction.Source: §1 · p. 3
  • Across sources, modalities, and architectures, emotion representations share relational structure and causal effects even when individual vector directions differ.The reported findings include held-out correspondence, cross-modal and cross-architecture steering, and consensus vectors that generalize to held-out Gemma-3 models.Source: §1 · p. 3

2 CMES AND EMOTION-VECTOR EXTRACTION

CMES pairs six basic emotion categories with seven text and image stimulus sources. The paper extracts a separate set of emotion vectors for each model and source from residual activations.

2 CMES AND EMOTION-VECTOR EXTRACTION

  • CMES covers six basic emotions across seven sources: three story sets, two real-face datasets, synthetic portraits, and synthetic emotion-evoking scenes.Real-face vectors use training images, while functional evaluations use separate validation or test splits.Source: §2 · p. 3
  • For each model and source, residual activations are averaged over story tokens from position 50 onward or over all visual tokens, then averaged by emotion.The resulting class means are centered across the six emotions, and background directions estimated from neutral stimuli are removed.Source: §2 · p. 3
  • The extraction produces six vectors per model and source after removing leading neutral-PCA directions that explain at least 50% of variance.The neutral PCA directions are denoted by U.Source: §2 · pp. 3–4

3 EMOTION GEOMETRY AND REPRESENTATIONAL CORRESPONDENCE

The six emotion vectors form low-dimensional configurations whose relational geometry varies more with stimulus source than architecture. Despite low native cosine similarity, held-out alignments recover cross-modal and cross-architecture correspondence.

3.1 VISUAL EMOTION VECTORS FORM STRUCTURED LOW-DIMENSIONAL REPRESENTATIONS

  • PC1 explains 30–46% of vector variance and PC1+PC2 explain 55–75%; human valence agreement is generally stronger and more stable than arousal agreement.Both text and image sources include cases with weak and strong arousal agreement.Source: §3.1 · pp. 4–5

3.2 STIMULUS SOURCE CHANGES RELATIONAL GEOMETRY MORE THAN ARCHITECTURE

  • Same-source cross-architecture CKA has median 0.99, versus 0.90 for different sources within an architecture; source changes therefore affect geometry more.The corresponding RSA medians are 0.95 and 0.66, respectively; RAF-DB accounts for much of the strongest source-specific change.Source: §3.2 · p. 5, Figure 4 · p. 5

3.3 CROSS-MODAL CORRESPONDENCE SURVIVES LOW COSINE SIMILARITY

  • Text–image cosine is only 0.15–0.31, yet LOEO held-out cosine reaches 0.29–0.53 for AffectNet and synthetic image sources, with all 27 comparisons above z = 2.RAF-DB shows weaker text–image LOEO correspondence: cosine is 0.08–0.21, with four of nine comparisons exceeding the permutation threshold.Source: §3.3 · pp. 5–6

3.4 CROSS-ARCHITECTURE CORRESPONDENCE SURVIVES NEAR-ORTHOGONAL NATIVE COORDINATES

  • Across architectures, native cosine stays near zero while LOEO cosine ranges from 0.13 to 0.45 with mean 0.31; generic-image mappings also recover correspondence.For AffectNet, neutral-face mapping gives cosine 0.66–0.76 (mean 0.70), and ImageNet mapping gives 0.40–0.55 (mean 0.47), without fitting the six emotion vectors.Source: §3.4 · p. 6

4 CAUSAL FUNCTION ACROSS MODALITIES

The paper tests emotion-vector function by injecting vectors at image-token positions and measuring changes in emotion readouts. Image- and text-derived vectors selectively steer interpretation, with similar effects across modalities.

4 CAUSAL FUNCTION ACROSS MODALITIES

  • Image-token interventions add a scaled unit emotion vector at each image-token position, with strength and sign controlled by s; selective steering raises the injected emotion and suppresses competitors.The default interventions use s = +0.5 or s = −0.5, and outcomes are changes in log-probability for the injected emotion and the five competing emotions.Source: §4 · p. 6

4.1 EMOTION VECTORS SELECTIVELY STEER IMAGE INTERPRETATION

  • In Qwen3-VL, image-derived vectors increase the matched emotion’s log-probability by 4.96 and decrease competing emotions by 1.43 on average; random and emotion-orthogonal controls produce near-zero changes.InternVL3.5 and Ministral-3 show a similar steering pattern, and negative injection reverses the pattern.Source: §4.1 · p. 6

4.2 TEXT-DERIVED VECTORS FUNCTIONALLY SUBSTITUTE FOR IMAGE-DERIVED VECTORS

  • Text-derived vectors produce steering effects similar to image-derived vectors despite text–image cosine similarity near 0.2–0.3, across all three models.Both vector types also suppress competing emotions.Source: §4.2 · p. 6, Figure 5 · p. 7
  • Steering changes emotion predictions on 47% of evaluated RAF-DB images on average, versus 2% for gender, and shifts safe–dangerous predictions in emotion-consistent directions.Continuations describe visual cues that match the injected emotion; removing the centered emotion-vector span impairs recognition more than removing a matched-rank random subspace.Source: §4.2 · p. 7

5 CROSS-ARCHITECTURE TRANSFER

Mappings learned from paired image activations recover cross-architecture emotion correspondence without fitting emotion vectors and restore selective causal transfer across models and modalities.

5.1 ESTIMATING THE COORDINATE TRANSFORMATION WITHOUT THE SIX EMOTION VECTORS

  • Generic ImageNet activations support cross-model emotion correspondence without emotion-dataset images; a mapping learned from the same number of neutral faces reaches 0.70.Additional LOEO alignment further improves held-out correspondence for image-derived and text-derived vectors.Source: §5.1 · pp. 7–8

5.2 MAPPED EMOTION VECTORS TRANSFER CAUSALLY ACROSS ARCHITECTURES

  • Mapped Ministral-3 vectors restore selective Qwen3-VL steering: matched-emotion log-probability rises by 4.11 for neutral-face mappings and 3.33 for ImageNet mappings.Unmapped Ministral-3 image vectors produce essentially no selective effect; mapped text vectors also steer bidirectionally despite cosine similarity of only 0.18–0.29.Source: §5.2 · p. 8, Figure 6 · p. 8

5 CROSS-ARCHITECTURE TRANSFER

  • Paired-image mappings recover vector correspondence and selective causal effects across architectures, supporting shared emotion relationships despite differences in model coordinates.The mappings also improve LOEO results.Source: §5.2 · p. 8

6 A SHARED EMOTION SUBSPACE

The paper aligns three architectures to construct a shared emotion subspace whose consensus vectors preserve affective structure and selective steering, including in a held-out architecture.

6 A SHARED EMOTION SUBSPACE

  • Generalized Procrustes alignment jointly fits orthogonal transformations and a shared template across three models, then averages aligned vectors across models and sources into emotion consensus vectors.Centering the shared vectors yields a subspace spanning distinctions among the six emotions.Source: §6 · pp. 8–9
  • Consensus vectors show emotion-related lexical readouts, steer Qwen3-VL continuations, and retain correspondence and causal effects in held-out Gemma-3-12B and Gemma-3-27B.Neither Gemma model contributes to consensus construction.Source: Figure 7 · p. 9, §6 · p. 9
  • After alignment, shared emotion differences remain with little model-wide variation but substantial source variation and interactions.Mean cosine between original vectors and their shared-subspace projections ranges from 0.59 to 0.64 under neutral-face alignment and 0.57–0.63 under ImageNet alignment.Source: §6 · p. 9
  • The shared projection of each Qwen3-VL emotion vector steers the matched emotion slightly more than the original vector, while its orthogonal remainder has very little effect.Random directions outside the subspace and remainders with shuffled emotion labels also show little selective steering.Source: §6 · p. 9, Figure 7 · p. 9

7 CONCLUSIONS

The paper concludes that VLM emotion representations share relational structure and transferable causal effects even when individual vector directions differ. Its appendices document related work, methods, evaluations, limitations, ethics, and reproducibility.

7 CONCLUSIONS

  • Emotion representations generalize through shared relationships and causal effects despite differing vector directions; alignment recovers correspondence that native cosine can miss.The shared subspace preserves agreement with human valence and arousal, and its projections retain most selective steering effects.Source: §7 · pp. 9–10

APPENDIX

  • The appendices document experimental setup and controls, shared-subspace evaluations, limitations in model and emotion coverage and source dependence, ethics, and reproducibility.They also discuss the need to evaluate practical applications and potential risks of emotion inference and steering.Source: APPENDIX · p. 15

A RELATED WORK

  • Prior work mainly examined text-based emotion geometry and within-model steering; this study tests six-emotion geometry and causal transfer across modalities and architectures.It connects the group’s relational geometry to the causal function of individual vectors.Source: A RELATED WORK · pp. 15–16

B EXPERIMENTAL SETUP AND REPRODUCIBILITY

The setup builds matched six-emotion text and image stimuli, extracts vectors from three core VLMs, and tests their interpretation, steering, and cross-architecture relations. It also specifies reproducibility materials and exploratory lexical readouts.

B.1 MODELS AND INFERENCE

  • The core analyses use Qwen3-VL, InternVL3.5, and Ministral-3; Gemma-3 models are reserved for held-out architecture generalization.The held-out Gemma-3 models have 12B and 27B parameters and do not contribute vectors to the three-model consensus.Source: §B.1 · p. 16

B.2 STIMULUS SOURCES

  • CMES spans three story sources and four image sources, with six Ekman emotions and neutral stimuli in each set.Image sources include AffectNet, RAF-DB, synthetic portraits, and synthetic scenes; evaluation images are held out from vector construction.Source: §B.2 · pp. 16–17

B.3 STIMULUS GENERATION AND PROMPTS

  • Story prompts cross emotion words with topics, while synthetic portrait prompts vary demographic, style, composition, and lighting factors.Synthetic scenes use environment-centric topics; generated images are not manually selected, filtered, or regenerated.Source: §B.3 · p. 17, Figure 8 · p. 18

B.4 ACTIVATION EXTRACTION AND EMOTION-VECTOR CONSTRUCTION

  • Vectors are class-mean activation contrasts, centered across emotions and projected away from neutral-stimulus PCA directions explaining at least 50% of variance.Text activations pool story tokens from position 50 onward; image activations pool visual tokens.Source: §B.4 · pp. 17–18

B.5 STEERING PROTOCOL

  • Steering adds a scaled emotion vector to image-token residual positions and measures changes in next-token emotion-label probabilities.The default uses unit-length vectors at one layer with s = +0.5 or s = −0.5; evaluations use held-out images screened as correctly classified.Source: §B.5 · pp. 18–19

B.6 CROSS-ARCHITECTURE ALIGNMENT

  • Cross-architecture maps are fitted from paired activations without emotion vectors or labels, using neutral faces by default or ImageNet images as an out-of-domain control.Generalized Procrustes alignment places the three core models in a shared frame; Gemma-3 maps are estimated separately.Source: §B.6 · p. 19

B.7 SIMILARITY MEASURES

  • The similarity analyses separate axis alignment, relational geometry, and matched-vector correspondence, with LOEO testing held-out emotion pairs.CKA and RSA compare relational structure; permutation testing uses 500 shuffled label assignments and treats z > 2 as evidence beyond the unlabeled geometry.Source: §B.7 · pp. 19–20

B.8 REPRODUCIBILITY ARTIFACTS; C.1 LOGIT-LENS OF EMOTION VECTORS

C.2 QUALITATIVE EVIDENCE FROM RETRIEVED IMAGES AND PATCH-LEVEL READOUTS

Image-vector retrievals often match target emotions, but retrieval and full-set classification can diverge, especially across datasets. Emotion geometry also varies across layers and sources, with valence more stable than arousal.

C.2 QUALITATIVE EVIDENCE FROM RETRIEVED IMAGES AND PATCH-LEVEL READOUTS

  • 84–92% happiness and 82–90% sadness P@50 are the highest retrieval ranges across models, while anger and surprise are generally lower.Every AffectNet emotion vector exceeds the pool’s approximately 12.5% reference rate.Source: Table 2 · p. 23, §C.2 · p. 23
  • P@50 and six-way recall correlate at ρ = +0.72, but Qwen3-VL disgust has 80% P@50 and only 34% recall.On the same AffectNet validation images, vector classification accuracy is 51.4%, versus 47.6% for direct questioning.Source: §C.2 · p. 23
  • RAF-DB disgust retrieval is weak despite classification recall of 60/48/53%; geometry places disgust nearer happiness and fear nearer anger.Those recall values use a different evaluation dataset from AffectNet and do not establish a controlled comparison of difficulty.Source: §C.2 · pp. 23–24, Table 3 · p. 24
  • Displayed high-similarity image tokens often overlap expressive facial regions, but the maps are qualitative and do not test which regions cause predictions.The images are tightly cropped around faces, so they also do not establish a preference for faces over natural backgrounds.Source: §C.2 · p. 24, Figure 11 · p. 25

C.3 EMOTION-VECTOR STRUCTURE ACROSS DEPTH

  • T2 story geometry has at least 0.89 similarity across every sampled layer pair, whereas AffectNet geometry changes between layer ranges.The comparison uses the 15 unique off-diagonal entries of each six-emotion cosine matrix.Source: §C.3 · p. 25, Figure 12 · p. 26

C.4 PRINCIPAL AXES ACROSS MODELS AND STIMULUS SOURCES

  • PC1 explains 30–46% of variance and PC1+PC2 explain 55–75%, with PC1 more consistently aligned to human valence than PC2 is to arousal.Across the 21 model–source sets, median PC1 correlation is +0.90 with R&M valence; median PC2 correlation is +0.58 with R&M arousal.Source: §C.4 · pp. 25–26

C.5 SET-LEVEL SIMILARITY ACROSS STIMULUS SOURCES

The section compares six-emotion geometry and individual emotion correspondence across stimulus sources and modalities. Source differences, especially those involving RAF-DB, qualify the otherwise broad cross-source and cross-modal patterns.

C.5 SET-LEVEL SIMILARITY ACROSS STIMULUS SOURCES

  • RAF-DB drives much of the source mismatch: within-model median RSA is 0.04 versus 0.75 among other sources, with disgust and fear relationships contributing.The comparisons do not rule out effects of the shared stimulus sample.Source: §C.5 · p. 28
  • CKA is 0.94–1.00 (median 0.99) across architectures at fixed source, versus 0.69–0.98 (median 0.90) across sources at fixed architecture.The distributions overlap, but source changes can produce larger differences in centered six-emotion geometry than architecture changes.Source: §C.5 · p. 28, Figure 15 · p. 30

C.6 SOURCE DEPENDENCE OF THE AROUSAL AXIS

  • Mean arousal agreement ranges from 0.22 for synthetic portraits to 0.89 for RAF-DB, while valence source means range from 0.53 to 0.97.Both text and image sources include weak and strong arousal agreement; AffectNet also varies across models.Source: §C.6 · pp. 30–31
  • Success filtering shifts axis agreement in both directions, not consistently toward human ratings; for AffectNet, InternVL3.5 changes from 0.03 ± 0.03 under size-matched random subsampling to 0.63 after filtering.The selection uses Qwen3-VL predictions and the same selected examples across all three models.Source: §C.6 · p. 31

D.1 WITHIN-MODALITY CONSISTENCY ACROSS STIMULUS SOURCES

  • Across story sources, cosine is 0.60–0.71 and LOEO cosine is 0.59–0.73; image-source correspondence varies more, with RAF-DB comparisons weaker.The three models rank the six image-source pairs almost identically, and all nine RAF-DB comparisons exceed the permutation threshold despite lower LOEO cosine.Source: §D.1 · pp. 33–34

D.2 CORRESPONDENCE BETWEEN TEXT- AND IMAGE-DERIVED VECTORS

  • Across 36 text–image comparisons, native cosine is 0.15–0.31, yet all 27 AffectNet and synthetic-source comparisons exceed z = 2, with LOEO cosine of 0.29–0.53.This held-out test evaluates whether relationships among five emotions predict the sixth.Source: §D.2 · p. 34
  • Full-fit cosine reaches 0.89–0.98 for text–image pairs and 0.86 with random or orthogonal image vectors, while those controls give near-zero LOEO cosine.RAF-DB also has weaker cross-modal LOEO cosine (0.08–0.21), with only four of nine comparisons above z = 2.Source: §D.2 · p. 36
  • Across architectures, synthetic portraits rank highest and RAF-DB lowest in average text–image LOEO cosine, with AffectNet and synthetic scenes between them.This broad ordering appears in each of the three models.Source: §D.2 · p. 36

D.3 CROSS-ARCHITECTURE CORRESPONDENCE AND ALIGNMENT

The section tests whether cross-architecture emotion correspondence and steering survive differences in model coordinates. It also examines whether activation-based mappings and text-derived vectors support held-out correspondence and selective effects.

D.3 CROSS-ARCHITECTURE CORRESPONDENCE AND ALIGNMENT

  • Despite near-zero native cosine, Ministral-3→Qwen3-VL LOEO cosine is positive for all seven sources (0.13–0.45; z = 2.60–3.73), unlike random and orthogonal replacement controls.The correspondence is not equally strong for every emotion; RAF-DB sadness and surprise are close to zero.Source: §D.3 · pp. 36–37
  • With 3,000 paired images, mean cosine rises from 0.38 to 0.70 for neutral-face mapping and from 0.14 to 0.47 for ImageNet mapping.The mappings are fitted without the six emotion vectors or their labels; neutral faces give higher cosine at every tested sample size.Source: Figure 17 · p. 37, §D.3 · p. 37
  • After activation mapping, adding five-pair LOEO raises AffectNet held-out cosine to 0.755 with neutral-face mapping and 0.584 with ImageNet mapping, versus 0.305 for LOEO alone.A random orthogonal map followed by LOEO gives 0.316, while target-orthogonal replacement directions give −0.005.Source: §D.3 · p. 38

E.1 WITHIN-MODEL STEERING WITH IMAGE-DERIVED VECTORS

  • Across all three models, positive image-vector injection raises the matched emotion and suppresses competing emotions on average; negative injection reverses the pattern.Random and orthogonal controls do not reproduce this selective sign reversal.Source: §E.1 · pp. 39, 41, Figure 19 · p. 41

E.2 CROSS-MODAL STEERING WITH TEXT-DERIVED VECTORS

  • Unaligned T3 text vectors selectively steer image predictions across models; in Qwen3-VL, cosine is 0.24 with RAF-DB and 0.25 with AffectNet, yet s = +0.5 raises matched log-probability by 6.07 and 4.77, respectively.Source: §E.2 · p. 41, Table 17 · p. 42

E.3 CROSS-ARCHITECTURE STEERING WITH MAPPED VECTORS

This section examines whether mapped emotion vectors support cross-architecture steering, including its selectivity, robustness, and retained shared structure. It reports that alignment can recover transfer despite modest vector similarity, while mapped consensus vectors selectively steer Qwen3-VL, shape emotion-consistent continuations, and retain emotion-related lexical readouts across architectures; source-specific variation remains consequential.

E.3 CROSS-ARCHITECTURE STEERING WITH MAPPED VECTORS

  • Neutral-face and ImageNet maps restore selective cross-architecture steering for weakly aligned Ministral-3 pairs, including text vectors despite mapped cosine of only 0.18–0.29.For Ministral-3→Qwen3-VL at s = +0.5, neutral-face mapping raises the matched-emotion effect from 0.04 to 4.11 while the competing-emotion mean changes from +0.18 to −0.77. ImageNet mapping yields lower image-vector cosine (0.39–0.49 versus 0.64–0.72) but preserves selective sign reversal; negative injection reverses the response. Mapping does not strengthen steering for the already aligned Qwen3-VL/InternVL3.5 pair.Source: §E.3 · pp. 42, 45, Figure 20 · p. 43, Table 18 · p. 44

E.4 SPECIFICITY, COHERENCE, AND INJECTION POSITION

  • Emotion steering changes emotion predictions far more than gender predictions while preserving coherent continuations, though disgust labels rarely switch and effects vary by injection position.Emotion predictions change on 47% of images and accuracy falls from 0.982 to 0.527, while gender predictions change on 2% and accuracy shifts from 0.961 to 0.950; the evaluation subsets differ slightly. Across 630 continuations per dataset, distinct-1 averages 0.92 on RAF-DB and 0.93 on AffectNet, with no repeated-token run longer than one. Disgust steering changes the readout to the target on only 32 of 3,500 AffectNet images (under 1%); question-token injection is weaker and fails to suppress competing emotions on average, while all-position injection suppresses them most strongly.Source: Table 19 · p. 45, §E.4 · pp. 45, 47, Table 20 · p. 47

E.5 REMOVING THE EMOTION-VECTOR SPAN

  • Full removal of the centered emotion-vector span lowers accuracy from 0.993 to 0.814 on RAF-DB and from 0.986 to 0.781 on AffectNet, unlike dimension-matched random controls.Partial removal and removing a single emotion direction cause smaller decreases; removal does not eliminate recognition, so these directions contribute to but do not capture all supporting information.Source: Table 21 · p. 47, §E.5 · pp. 47–48

E.6 BEYOND THE EMOTION READOUT

  • Positive anger and fear injections shift safe–dangerous predictions toward dangerous and happiness toward safe; steering is strongest for the depicted person but also affects other queried subjects.Negative injection reverses the safe–dangerous effects, while disgust and sadness vary across datasets. Effects on emotion predictions about the assistant and an absent third party show that steering is only partly tied to the depicted subject.Source: §E.6 · pp. 48–49, Table 23 · p. 49

E.7 CAUSAL EFFECT ACROSS DEPTH

  • Steering is stronger before the default layer and weaker near the output, with the default layer’s selectivity 28%, 18%, and 47% below each model’s maximum.Effects use residual-norm scaling; thus the layer comparisons are conditional on that calibration and do not provide a scale-independent ranking.Source: §E.7 · p. 49, Table 24 · p. 50

F.1 SEPARATING SHARED EMOTION STRUCTURE FROM SOURCE AND MODEL VARIATION

  • Aligned consensus vectors define a shared emotion subspace that reconstructs source-specific vectors and tracks human valence and arousal ratings, while substantial residual variation remains.Under neutral-face alignment, emotion accounts for 0.326 of cross-model variance, versus 0.004 for model and 0.003 for modality; residuals and interactions account for 0.668. Mean projection cosines across all sources are 0.64, 0.59, and 0.61 for Qwen3-VL, Ministral-3, and InternVL3.5; R&M correlations are |r| = 0.93 for PC1–valence and 0.96 for PC2–arousal. ImageNet alignment gives a similar pattern but weaker reconstruction, especially for AffectNet vectors.Source: §F.1 · pp. 51–52, Table 25 · p. 52

F.2 CAUSAL LOCALIZATION OF THE SHARED EMOTION SUBSPACE

  • The shared projection retains selective steering: unit-normalized (diag, off) is (+5.39, −1.62), versus (+4.96, −1.43) for the original and (+1.00, −0.14) for the remainder.Without renormalization, projection and remainder effects are (+5.14, −1.47) and (+0.68, −0.14). Equal-norm random-complement and rotated-label remainder controls give (+0.19, +0.22) and (+0.12, +0.03), respectively.Source: §F.2 · p. 52, Figure 24 · p. 53
  • Mapped consensus vectors selectively steered Qwen3-VL (diag +5.60, off −1.64); related information remained outside the shared span, with a smaller effect than the projection.Using Qwen3-VL as the reference for constructing the consensus yielded a similar result: (+5.56, −1.60).Source: §F.2 · p. 53, Table 26 · p. 53

F.3 GENERATED CONTINUATIONS AND LEXICAL READOUTS OF CONSENSUS VECTORS

  • In selected Qwen3-VL examples, all six consensus-vector injections produced continuations that followed the injected emotion and described facial cues consistent with that interpretation.The two illustrated images both had ground-truth label disgust.Source: §F.3 · p. 54, Figure 25 · p. 54
  • ImageNet-aligned consensus vectors promoted emotion-related tokens and suppressed some opposite-emotion terms across all three models, including Ministral-3 despite its near-zero native cosine before alignment.The coordinate maps used ImageNet activations without emotion labels, while the consensus averaged corresponding emotions after mapping.Source: §F.3 · p. 55, Figure 26 · p. 55

F.4 HELD-OUT GENERALIZATION AND MODEL-SIZE ABLATION

Held-out emotion-vector correspondence generalizes to Gemma-3 at both 12B and 27B, despite unequal residual widths. Mapped consensus vectors also retain correspondence and selective steering effects, while larger model size does not consistently improve correspondence.

F.4 HELD-OUT GENERALIZATION AND MODEL-SIZE ABLATION

  • 0.16–0.52 (median 0.36) at 12B and 0.13–0.54 (median 0.36) at 27B are the held-out LOEO cosine ranges; all 42 comparisons at each size exceed threshold.All 24 replacement controls remain below z = 1 at 12B and below z = 2 at 27B.Source: §F.4 · p. 55, Figure 27 · p. 56, Table 27 · p. 57
  • 0.63 and 0.42 (12B), and 0.59 and 0.39 (27B), are the mean cosines between consensus vectors mapped with neutral-face and ImageNet activations, respectively, and Gemma-3’s own vectors.Gemma-3 was excluded from consensus construction, and the maps used paired activations without emotion-vector fitting.Source: §F.4 · pp. 55, 57, Table 28 · p. 58
  • 3.66 for Gemma-3’s original vector, 3.59 for the neutral-face consensus, and 3.35 for the ImageNet consensus are the 12B matched-emotion log-probability changes at s = +0.1.Negative injection lowers the matched emotion and raises competing emotions; mapped consensus does not suppress competing emotions on average.Source: Figure 27 · p. 56, §F.4 · p. 57, Table 28 · p. 58
  • 2.66–2.90 are the positive matched-emotion effects of individual core-model transfers under neutral-face alignment, smaller than the consensus effects despite their higher cosine with Gemma-3’s own vectors.ImageNet alignment shows the same ordering; similarity to the target model’s vector does not fully predict steering strength.Source: §F.4 · pp. 57–58, Table 28 · p. 58
  • At 27B and s = +0.1, the original vector gives (diag, off) = (+4.04, −0.06), while neutral-face and ImageNet consensus mappings give (+3.47, +0.08) and (+3.16, +0.00).Both consensus variants again exceed corresponding single-model transfers in positive matched-emotion effects; negative injection reverses the effect, unlike random and orthogonal controls. Differences in width, depth, and evaluation subsets mean this tests robustness across sizes, not parameter count alone.Source: Figure 27 · p. 56, §F.4 · p. 58, Table 28 · p. 58

G.1 LIMITATIONS

The paper limits its claims to controlled emotion representations and identifies boundaries in model scale, emotion categories, applications, and data. It also outlines future extensions and cautions against interpreting model outputs as reliable judgments of individuals’ emotions.

G.1 LIMITATIONS

  • The study centers on three approximately 8B-scale VLMs and Ekman’s six basic emotions; source-sensitive organization may differ for larger or more diverse models, complex emotions, continuous affect, or other cultural contexts.It does not test whether the findings improve task performance or reduce emotion-related hallucinations.Source: §G.1 · p. 59

G.2 FUTURE WORK

  • Future work will test larger and more diverse models, video and speech, complex emotions, continuous affect, cultural contexts, and whether the vectors improve task accuracy or reduce hallucinations.The authors also plan to compare stimulus sources and assess effects on unrelated tasks.Source: §G.2 · p. 59

G.3 BROADER IMPACT AND ETHICS

  • The authors study internal representations, not people’s true emotional states; dataset, cultural, and source dependencies and possible output manipulation warrant validation, privacy protection, and human oversight.They caution against treating the findings as evidence of reliable assessment in sensitive settings such as mental health, education, or employment.Source: §G.3 · p. 59, H ETHICS STATEMENT · p. 59

H ETHICS STATEMENT

  • AffectNet and RAF-DB images are not redistributed because of privacy concerns, and access remains subject to the original providers’ terms.The study uses existing facial-expression datasets rather than collecting new human-participant data.Source: H ETHICS STATEMENT · p. 59

I REPRODUCIBILITY STATEMENT

The paper documents its experimental procedures and plans to release code, configurations, stimulus sample lists, synthetic-generation seeds, and extracted emotion vectors. It also discloses the use of language and generative models.

I REPRODUCIBILITY STATEMENT

  • Appendix B documents model configurations, stimuli and splits, prompts, activation extraction, vector construction, steering, alignment, and similarity measures; Appendices D–F provide results and controls.Planned releases include code, experiment configurations, stimulus sample lists, synthetic-generation seeds, and extracted emotion vectors; sample identifiers will support reconstruction where data cannot be redistributed.Source: I REPRODUCIBILITY STATEMENT · p. 60

J LLM USAGE STATEMENT

  • LLMs assisted with language polishing, and generative models created the synthetic stories, portraits, and emotion-evoking scenes in CMES.Data sources, generation models, and prompts are documented in Appendices B.2 and B.3.Source: J LLM USAGE STATEMENT · p. 60
Loading 2609.34742v1…