Source-linked AI summary
Hard Negatives Reveal What Easy Negatives Hide: Cross-Lingual Harmfulness Representations Degrade with Resource Tier Under Hard Negatives
Paras Balani, Subhrakanta Panda
TL;DR
Cross-lingual harmfulness probes have appeared to transfer to low-resource languages, but prior evidence relied on easy negatives. This paper tests English-trained probes with both easy and surface-similar hard negatives across nine languages, finding that hard-negative transfer degrades most in low-resource languages.
Problem
Method
The study transfers English-trained harmfulness probes across nine languages using easy unrelated negatives and benign, surface-similar XSTest hard negatives.
Source: §1 · p. 2, §2.3 · pp. 3–4Results
On Qwen2.5-7B-Instruct, mean AUROC drop rises from 0.003 in English to 0.017 in high-resource, 0.042 in mid-resource, and 0.276 in low-resource languages.
Source: §6 · p. 9Takeaways & Limitations
Easy-negative AUROC alone cannot establish that harmfulness representations survive translation; evaluation should include hard negatives before making cross-lingual transfer claims.
Source: §6 · p. 9With nine languages, the data do not identify the remaining mechanism after considering chrF and tokenizer fertility.
Source: §4 · p. 8
Abstract
from arXiv · showhide
Safety alignment in large language models is trained primarily in English, and recent work reports that the underlying harmfulness representation survives translation: English-trained probes separate harmful from harmless prompts almost as well in low-resource languages as in English. This has been taken as evidence that cross-lingual refusal failures mainly reflect calibration rather than representation quality. We show that this conclusion depends on the choice of negative examples. Across nine languages spanning three resource tiers, we replicate near-perfect transfer (AUROC > 0.98) when harmless prompts come from an unrelated distribution (easy negatives). With XSTest contrast prompts, which are benign but surface-similar to harmful requests (hard negatives), transfer collapses in low-resource languages while remaining largely stable in high-resource languages. On Qwen2.5-7B-Instruct, mean AUROC drop increases from 0.003 in English to 0.017 in high-resource, 0.042 in mid-resource, and 0.276 in low-resource languages. The pattern replicates on Aya Expanse. Back-translation chrF controls and a matched-chrF comparison across three languages reduce the likelihood that translation quality explains the effect. The collapse remains after controlling for chrF (partial r = 0.70, p = 0.03). Tokenizer fertility correlates with the collapse and explains part of the resource-tier effect, but not all of it. The results show that easy-negative transfer can coexist with substantial degradation under hard negatives. Easy-negative evaluation alone therefore cannot establish that the harmfulness representation survives translation.
1 Introduction and Related work
Prior cross-lingual probe results suggested that harmfulness representations transfer to low-resource languages, but that conclusion relied on clearly harmless negatives. Testing with surface-similar XSTest contrasts instead reveals substantial low-resource degradation, alongside controls indicating that translation quality alone does not explain the pattern.
1 Introduction and Related work
- Probe performance also depends on readout location and target: harmfulness and refusal can occupy separate directions, and final-token probes can miss earlier safety evidence.Zhao et al. [2025] and Doda [2026] motivate distinguishing the harmfulness representation from refusal behavior and considering where safety evidence is read out.Source: §1 · p. 2
- Earlier work found that topic- and surface-matched benign prompts sharply reduce probe AUROC, but had not tested that collapse across languages.Schwarz [2026] reports 0.996–0.999 AUROC on disjoint sources versus 0.590–0.690 on matched pairs; Llorente-Saguer [2026] reports a large hard-benign gap for ShieldGemma-9B.Source: §1 · p. 2
- Aziz et al. [2026] found near-equal cross-lingual probe separation despite lower refusal, but their easy PolyRefuse negatives leave surface-form shortcuts uncontrolled.They interpreted the gap as calibration failure and showed that target-language threshold resetting recovers much of the lost refusal rate.Source: §1 · p. 2
- Easy negatives preserve >0.98 AUROC transfer, but hard negatives produce 0.20–0.42 AUROC losses in low-resource languages, versus less than 0.04 in high-resource languages.The pattern replicates on Aya Expanse; after controlling for chrF, the resource-tier effect remains (partial r = 0.70, p = 0.03).Source: §1 · p. 2
- Matched-chrF languages show five- to ten-fold differences in collapse, and tokenizer fertility correlates with the effect but is not established as its cause.The authors note that nine languages do not establish which factor dominates; hallucination detectors also show that low-resource representational degradation is not automatic across capabilities.Source: §1 · p. 2
2 Method
The method compares English-trained harmfulness probes across two matched-size negative sets and nine translated target languages. It evaluates transfer using model activations, English-only probe fitting, and condition-specific peak layers.
2.1 Data
- The matched-size sets contrast unrelated AdvBench–Alpaca negatives with XSTest benign contrasts, while a length-only gate checks that hard negatives reduce length-based separation.The length-only classifier reaches 0.688 AUROC on easy negatives and 0.534 on hard negatives; both sets use fixed-seed samples from the same underlying pools.Source: §2.1 · p. 3
- All 450 prompts are translated into nine languages across three resource tiers, and back-translation chrF is measured to assess translation quality as an explanation.The study runs the pipeline on Qwen2.5-7B-Instruct and Aya-Expanse-8B without fine-tuning, extracting activations in a single forward pass.Source: §2.1 · p. 3, §2.2 · p. 3
2.2 Models
- The two open instruction-tuned models supply last-prompt-token activations at every layer, so probes read representations of the input rather than generated responses.The models are Qwen2.5-7B-Instruct and Aya-Expanse-8B; the last-token choice targets harmfulness rather than the subsequent refusal decision.Source: §2.3 · p. 3, §2.2 · p. 3
2.3 Probing setup
- Logistic-regression probes are fit on English only, with all non-English AUROCs measuring transfer and each condition evaluated at its own English-selected peak layer.English scores use five-fold cross-validation, while non-English languages are scored using the probe trained on all English examples; reported transfer intervals use a 5,000-resample stratified bootstrap.Source: §2.3 · pp. 3–4
3 Results
Easy-negative transfer remains nearly intact across languages, but hard negatives reveal a resource-tier gradient in AUROC loss. Controls show that translation quality alone does not account for the pattern, while tokenizer fertility is associated with some of the drop.
3.1 Main result
- 0.977 (Amharic) and 0.991 (Swahili) easy-negative AUROCs fall to 0.561 and 0.786 under hard negatives.High-resource French and Spanish drop only 0.007 and 0.010, while bootstrapped intervals separate low-resource from high-resource languages at the hard-negative layer.Source: §3.1 · pp. 4–5
- Mean easy-to-hard AUROC drop rises from 0.003 in English to 0.017, 0.042, and 0.276 across high-, mid-, and low-resource tiers.Each tier’s mean drop is roughly an order of magnitude larger than the tier above it.Source: §3.1 · p. 4, Table 1 · p. 5
- Aya-Expanse-8B reproduces the tier ordering, with mean AUROC drops of 0.002, 0.008, 0.019, and 0.251 from source through low-resource languages.Per-language drops also track closely between models for Amharic, Swahili, and Telugu.Source: §3.1 · p. 5, Table 2 · p. 6
3.2 Controls
- The raw chrF–drop correlation on Qwen (r = −0.751, p = 0.020) disappears when Amharic is removed, since it is the only point driving that relationship.The remaining eight languages show no clear relationship between translation quality and collapse.Source: §3.2 · p. 6, Figure 3 · p. 6
- At ≈68 chrF, Telugu’s 0.208 drop exceeds Arabic’s 0.039 and Vietnamese’s 0.019; resource tier remains related to drop after chrF control on both models.Partial correlations are 0.695 (p = 0.038) on Qwen and 0.709 (p = 0.032) on Aya.Source: §3.2 · pp. 6–7
- Fertility correlates with AUROC drop (Qwen r = 0.592; Aya r = 0.720), but Hindi has relatively high fertility and drops of only 0.069 and 0.019.With nine languages, neither tier nor fertility clearly dominates once both are included alongside chrF.Source: Figure 4 · p. 7, §3.2 · p. 7
4 Discussion
Hard negatives reveal a resource-tier degradation that easy negatives obscure, challenging the interpretation that cross-lingual refusal gaps are only calibration failures. Fertility explains part, but not all, of the observed variation, and easy-negative monitor validation can mask lost separation in low-resource languages.
4 Discussion
- This challenges Aziz et al. [2026]’s intact-representation reading: removing surface-form shortcuts sharply weakens separation, especially in low-resource languages.Their easy-negative evaluation shows transfer even to low-resource languages.Source: §4 · pp. 7–8
- Aziz et al. [2026]’s threshold reset can recover refusal from a weak residual signal, so recovery does not establish an intact representation.Source: §4 · p. 8
- On Qwen, mean AUROC drop rises from 0.003 in English to 0.017 in high-resource, 0.042 in mid-resource, and 0.276 in low-resource languages.Aya shows a comparable progression.Source: §4 · p. 8
- Matched-chrF comparisons show similar translation quality but five- to ten-fold differences in collapse magnitude across Telugu, Arabic, and Vietnamese.Source: §4 · p. 8
- Fertility explains roughly half the raw tier effect, but Hindi’s smaller collapse despite comparable-or-higher fertility limits a fertility-only account; the remaining mechanism is unidentified.Source: §4 · p. 8
- Easy-negative validation can make a safety monitor appear effective across languages while masking lost separation on hard negatives in low-resource languages.Source: §4 · p. 8
5 Solutions
The proposed solutions address both evaluation and representation quality: test monitors with surface-matched benign prompts, then investigate tokenizer fertility and steering as possible interventions. Both interventions can use the existing probing pipeline.
5 Solutions
- Test cross-lingual safety monitors with benign negatives that match harmful prompts in surface form; XSTest provides this construction in English.Matched prompts can be translated or authored for other languages.Source: §5 · p. 8
- Vocabulary augmentation or tokenizer retraining for high-fertility languages can test whether lower fertility produces smaller AUROC drops.The proposed examples include Telugu, Amharic, and Swahili.Source: §5 · p. 8
- Adding English-derived harmfulness directions to the residual stream at inference time can test whether steering restores hard-negative separation.Recovery would support a signal-strength explanation; no recovery, especially for Amharic, would indicate evidence for a deeper representational deficit.Source: §5 · pp. 8–9
- Both interventions can use the existing layer-wise activations across languages and models, without a new evaluation design.Source: §5 · pp. 8–9
6 Conclusion
Easy negatives reproduce strong cross-lingual probe transfer, but XSTest hard negatives expose substantial degradation in low-resource languages. The findings support evaluating cross-lingual safety representations with hard negatives before claiming that the representation survives translation.
6 Conclusion
- Hard-negative transfer remains strong in high-resource languages but collapses in low-resource languages, with Qwen’s mean AUROC drop reaching 0.276 in low-resource languages.The pattern appears despite easy-negative transfer above 0.98 AUROC across resource tiers.Source: §6 · p. 9
- Easy-negative AUROC cannot distinguish an intact semantic representation from a surface-form shortcut.Source: §6 · p. 9
- Cross-lingual safety representations should be evaluated with hard negatives before claims of transfer are made.Source: §6 · p. 9