"Not" is Not Lost in Translation: A Causal Analysis of Negation Representations in Multilingual LLMs

1 Introduction

• The authors construct Mini-minus-4 for controlled contrastive pairs and FLORES-minus-4 to test interventions on longer, out-of-domain sentences.Source: §1 · p. 2
• Negation representations are causally controllable and largely shared across languages, with dense directions and SAE features transferring across languages and to broader settings.The reported scope includes unseen languages and naturalistic sentences.Source: §1 · p. 2
• The study examines six open models from three families, using English, French, Vietnamese, and Chinese as its primary languages.Source: §1 · p. 2
• Directions extracted in one language retain strong control in others, and directions extracted from different languages are substantially aligned.Interventions along a contrastive direction can add or remove negation during translation.Source: §1 · p. 2
• A single SAE feature per model can respond selectively to negated contexts and exert causal control over polarity across languages.Source: §1 · p. 2
• Both dense directions and sparse features generalize to unseen languages and naturalistic sentences.Source: §1 · p. 2

2.1 Preliminaries

• The paper studies decoder-only Transformers and uses sparse autoencoders to represent residual activations with sparse codes and decoder-feature vectors.Each decoder vector is treated as a direction in residual-stream space and used for interventions.Source: §2.1 · p. 2

2.2 Dataset Construction

• Mini-minus-4 supplies minimally contrastive pairs in four languages, while FLORES-minus-4 tests intervention effects on naturalistic sentences.Mini-minus-4 has 350 pairs per language, split into 150 extraction, 100 evaluation, and 100 test pairs; FLORES-minus-4 retains 216 items verified in all four languages.Source: §2.2 · pp. 2–3

2.3 Causal Intervention with Contrastive Directions

• Contrastive Activation Addition derives directions from paired activations and adds them during generation, with positive coefficients adding negation and negative coefficients removing it.Interventions affect source-sentence tokens, excluding the instruction and generated output; on FLORES-minus-4 they target main-clause tokens.Source: §2.3 · p. 3

2.4 Causal Intervention with Sparse Features

• The SAE approach selects features more active on negative than positive examples and constructs a steering direction from their decoder vectors.It uses the same sign convention and intervention scope as CAA, with coefficients norm-matched using the best English CAA coefficient at the same layer and position.Source: §2.4 · pp. 3–4

2.5 Intervention Success Evaluation

• The evaluation counts a flip only when polarity changes as intended while target language, non-degeneracy, and content preservation also pass.Examples are eligible only when the baseline translation preserves source polarity; human evaluations agreed with automatic judgments on 96.5% of Mini-minus-4 and 93.7% of FLORES-minus-4 samples.Source: §2.5 · p. 4

3.1 Models

• Six instruction-tuned models span Gemma 3, Qwen3/3.5, and Llama 3.1; SAE steering uses one model per family with a publicly available pretrained SAE.The six models are Gemma 3 1B, 4B, and 12B; Qwen3 4B; Qwen3.5 9B; and Llama 3.1 8B.Source: §3.1 · p. 4

3.2 Translation task

• Zero-shot translation experiments use four primary source-to-English or English-to-Vietnamese settings, varying the source language because target-language changes yield nearly identical directions and similar intervention results.Additional-language tests translate Russian, Hindi, Arabic, and Indonesian into English; Figure 3 reports cross-language transfer flip rates for addition and removal.Source: §3.2 · pp. 4–5, Figure 3 · p. 5

4.1 Negation directions causally control polarity

• Adding or subtracting a contrastive activation direction reverses translation polarity, with ≥80% flip rates across languages for Gemma-3-(4B, 12B) and Qwen3.5-9B.Exceptions include Qwen3-4B addition on French (60.0%), Gemma-3-1B addition on Vietnamese (65.0%), and Llama-3.1-8B removal on Vietnamese (74.5%); random directions typically achieve 0%.Source: Table 1 · p. 5, §4.1 · p. 5

4.2 Negation directions align and transfer across languages

• A single English-extracted direction flips polarity in all four languages, with 48–100% addition and 77–99% removal rates across languages.Directions extracted from any language transfer to the others, though addition is weakest to Vietnamese, especially from French; cross-language and within-language rates track across layers, and language-specific directions have pairwise cosine similarities of 0.48–0.82.Source: §4.2 · pp. 5–6, Figure 4 · p. 6, Figure 6 · p. 6

4.3 Negation is represented in sparse features

• One English-selected SAE feature controls polarity across languages, reaching ≥91% flip rates for Gemma-3-4B and ≥93% for Qwen3.5-9B in both addition and removal.Features respond to negation markers, negated contexts, or both; Llama-3.1-8B’s feature fires less often on Vietnamese, which may explain its weaker control there.Source: Table 2 · p. 7, §4.3 · p. 7, Figure 7 · p. 7

4.4 Negation directions and features generalize to unseen languages and naturalistic sentences

• Across unseen languages, contrastive directions achieve at least 85.6% addition and 88.0% removal, while SAE features reach at least 86.9% for Gemma-3-4B and Qwen3.5-9B and 77.2% for Llama-3.1-8B.On naturalistic FLORES-minus-4 sentences, SAE removal is at least 91.6% across models and languages; CAA removal is at least 86.5% except Llama-3.1-8B on Vietnamese (65.0%), while addition is weaker for Llama-3.1-8B.Source: Table 3 · p. 7, §4.4 · p. 8

Negation in NLP and machine translation

• Behavioral studies report that language models under-use negation cues and that dropped or inverted polarity is a long-documented machine-translation failure mode, while internal multilingual representations remain underexplored.The cited works include Kassner and Schütze (2020), Ettinger (2020), Hossain et al. (2020), Truong et al. (2023), García-Ferrero et al. (2023), Fancellu and Webber (2015), and Tang et al. (2021).Source: §5 · p. 8

Linear representations and activation steering

• Activation steering has been used to manipulate causally effective directions for truth, sentiment, and refusal, and this work uses steering to validate identified negation representations.The passage cites Park et al. (2024), Marks and Tegmark (2024), Tigges et al. (2024), Arditi et al. (2024), Turner et al. (2024), and Rimsky et al. (2024).Source: §5 · p. 8

Sparse feature decomposition

• Prior multilingual SAE findings differ: one study reports more reliable cross-lingual steering from multilingual training, while another finds most features do not causally transfer despite recurring activations.Whether SAE features control behaviors as effectively as dense directions remains contested (Wu et al., 2025b; Jørgensen and Hansen, 2026; Arad et al., 2025).Source: §5 · p. 8

Cross-lingual structure in multilingual LLMs

• Evidence for language-neutral representations is mixed, while adjacent negation and translation studies do not test the same cross-lingual causal polarity-control question.Brinkmann et al. (2025) report weak cross-lingual sharing for negation-associated SAE features and do not test their causal role; this work tests dense directions and sparse features through cross-language interventions.Source: §5 · p. 8

6 Conclusion

• Negation is causally controllable through dense activation directions and individual SAE features, which transfer across languages and generalize to naturalistic sentences and unseen languages.Similarity analysis finds substantial alignment among directions, while inspected SAE features respond to negation in every language.Source: §6 · p. 8

7 Limitations

• Negation analysis covers explicit lexical verbal cues in translation; morphological and implicit negation, and other tasks, remain outside scope.Sparse-feature analysis also relies on pretrained SAE releases and interprets only one selected feature per model.Source: §7 · p. 9

A.1 Massive-Activation Dimensions in Gemma Models

• Gemma massive-activation dimensions inflate cross-lingual cosine similarity, so the authors zero them before similarity analyses but leave them unchanged in intervention experiments.Zeroing them changes intervention flip rates by a mean absolute 0.7, 0.4, and 0.5 percentage points for Gemma-3-1B, 4B, and 12B, respectively.Source: §A.1 · pp. 11–12
• Each Gemma model has one extreme dimension; in Gemma-3-4B and 12B, it exceeds the median dimension by over 2000× and carries 83.9–93.2% of squared norm.The highest-ranked dimensions in Qwen and Llama account for at most 3.8% of direction norm.Source: §A.1 · p. 12, Table 4 · p. 12
• Removing the Gemma-3-4B and 12B dimensions lowers mean pairwise cosine from 0.960 to 0.643 and from 0.965 to 0.673, respectively.For Gemma-3-1B, removal instead raises mean pairwise cosine from 0.540 to 0.642; for Qwen and Llama, the change is at most 0.002.Source: §A.1 · p. 12, Table 4 · p. 12

A.2 Target Language Ablation

• Across six models, directions extracted with different targets have pairwise cosine similarities of 0.9896–0.9986, while steering flip rates differ by a mean 5.1 percentage points from EN→VI.The authors therefore use a single target language per source in the main experiments.Source: §A.2 · p. 13, Table 6 · p. 13

A.3 Random-Direction Control

• Random directions yield 0% addition rates for every model and language, while removal rates reach at most 8%, versus substantially higher CAA flip rates.The authors suggest the asymmetry reflects random perturbations dropping, but not inserting, negation markers.Source: §A.3 · p. 13, Table 7 · p. 13

B.1 Implementation Details

• Generations use greedy decoding, with max_new_tokens of 64 for Mini-minus-4 and 128 for FLORES-minus-4; Qwen models run with thinking mode disabled.The study uses public SAEs across three model families, applying base-model SAEs to instruct-tuned Llama and Qwen variants when instruct SAEs are unavailable.Source: §B.1 · pp. 13–14, Table 8 · p. 14

B.2.1 Using Pre-determined Rules

• Mini-minus-4 intervention outputs must pass language-identification, non-degeneracy, and content-preservation thresholds to count as successful.The criteria require the intended language at fastText confidence ≥0.5, 2–200 words with at least 30% unique tokens, and chrF ≥30 against the opposite-polarity baseline translation.Source: §B.2.1 · p. 14

B.2.2 Using LLM-as-Judge

• On FLORES-minus-4, only baselines that preserve source polarity are eligible; a blind judge then checks polarity change and whether meaning is retained.The reported flip rate is the share of eligible items whose main-clause polarity changes to the target while meaning is retained.Source: §B.2.2 · p. 14

B.3 Human Validation

• 96.5% agreement on Mini-minus-4 and 93.7% on FLORES-minus-4 support the automatic evaluators, although FLORES-minus-4 flip rates may be upward biased.The respective PABAK values are 0.930 and 0.874; the evaluation covered 800 and 1,728 examples.Source: §B.3 · pp. 14–15, Table 9 · p. 15

C.1 Mini-minus-4

• Mini-minus-4 contains 350 minimally contrasting polarity pairs per language, split into extraction, evaluation, and test sets, with predicates disjoint across splits.It covers English, Vietnamese, French, and Chinese; additional held-out sets contain 100 pairs each in Russian, Hindi, Arabic, and Indonesian.Source: §C.1 · p. 15, Table 10 · p. 16

C.2 FLORES-minus-4

• FLORES-minus-4 is built from parallel FLORES-200 sentences through candidate collection, LLM-assisted polarity-flip generation, and verification, yielding 216 four-language items.Verification checks polarity reversal, main-clause scope, grammar, naturalness, and cross-language proposition consistency.Source: §C.2 · p. 16

D.1 Negation Translation Evaluation on FLORES-minus-4

• On FLORES-minus-4, models generally preserve negation accurately, with slightly lower preservation for negative than positive sentences and little fluency difference.LLM-as-judge evaluates polarity preservation; neural and n-gram metrics are used only for overall translation quality.Source: §D.1 · p. 17

D.2 Negation Preservation and Eligible Count on Mini-minus-4

• On Mini-minus-4, all models preserve negation at near-perfect accuracy in most languages; Chinese negative examples are the exception.Baseline polarity preservation determines eligibility for intervention flip-rate calculations.Source: §D.2 · p. 17, Table 11 · p. 17

Model EN VI FR ZH

• For Chinese-to-English, 89.8% of ineligible evaluation examples use predicate antonyms rather than explicit negation markers; subtracting the negation direction reverses meaning in 68.2% of them.The authors retain the marker-based eligibility filter, which they say arguably yields more conservative flip rates.Source: Model EN VI FR ZH · p. 17, Table 12 · p. 18

D.3.1 Setup

• Logistic-regression probes test whether polarity is linearly encoded and whether a probe trained on one language transfers to other languages.Probes predict positive versus negative polarity across model layers, positions, and languages.Source: §D.3.1 · p. 18

D.3.2 Full Probe Accuracy Results

• At both NP and FST positions, mid-layer probes predict polarity at perfect accuracy.The full probe accuracies are reported by layer in Figures 12 and 13.Source: §D.3.2 · p. 18

D.3.3 Probe Transfer Results

• Probes trained in one language transfer at high accuracy for most language pairs.At each model’s selected layer-position pair, probes trained in each language were evaluated across all four languages.Source: §D.3.3 · p. 18

D.4.1 Eval-set Flip Rates for CAA Steering

• Causal flip rates rise from early to mid layers and then fall, with effective control concentrated more narrowly than peak probe accuracy.The typical best steering coefficient is α = 2 or 4, depending on model and language.Source: §D.4.1 · p. 18

D.5.1 α∗used for SAE norm matching

• At the three selected SAE layers, αCAA = 4 is optimal on the EN evaluation set.Source: §D.5.1 · p. 19, Table 15 · p. 19

D.5.2 Selecting Candidate Layers

• Only several layers per model show expected causal effects from SAE steering: Gemma-3-4B layers 4, 7, and 14; Qwen3.5-9B layers 4, 8, 10, 12, and 15; and Llama-3.1-8B layers 4 and 13.Source: §D.5.2 · p. 19

D.5.3 Selecting Features

• Only a small fraction of tested SAE features show strong polarity control, so one highest-performing feature per model was selected for further tests.Source: §D.5.3 · p. 19

D.5.4 Feature Interpretation

• Across languages, selected features primarily respond to negation-related cues: Gemma f331 fires at explicit negation markers, Llama f12611 on negated predicates, and Qwen f43677 responds to both.Llama’s L13 f12611 activates weaker on Vietnamese sources than on other languages, one explanation for its weak causal control there (§4.3).Source: §D.5.4 · p. 19

• The difference between the full SAE direction and CAA depends on the difference in SAE reconstruction error for negative versus positive examples.The decoder bias cancels between the negative and positive reconstructions.Source: E Relationship between CAA and SAE… · p. 20
• The SAE direction is a truncated reconstruction of the contrastive direction using a small number of SAE features.The full SAE direction uses SAE-reconstructed residual activations, whereas CAA uses original residual activations.Source: E Relationship between CAA and SAE… · p. 20