PaperlayerView exact revision on arXiv

Source-linked AI summary

LASA: Language-Agnostic Semantic Alignment at the Semantic Bottleneck for LLM Safety

Junxiao Yang, Haoran Liu, Jinzhe Tu, Jiale Cheng, Zhexin Zhang, Shiyao Cui, Jiaqi Weng, Jialing Tao, Hui Xue, Hongning Wang, Han Qiu, Minlie Huang

arXiv:2604.12710v2cs.LGcs.AIcs.CL

TL;DR

LLMs often lose safety robustness in low-resource languages because language-agnostic semantic understanding is paired with language-dominant safety alignment. The paper identifies an intermediate Semantic Bottleneck and anchors LASA safety alignment there, reducing attack success substantially across evaluated models, while noting limits for implicit or multi-step harmful intent.

  • Problem

    Safety performance is strong in high-resource languages but degrades in low-resource languages because semantic understanding generalizes across languages while safety discrimination remains tied to alignment-language distributions.

  • Method

    LASA identifies the Semantic Bottleneck, trains a Safety Semantic Interpreter on its representation, and conditions response generation on the resulting safety signal.

  • Results

    LASA reduces average ASR from 24.7% to 2.8% on LLaMA-3.1-8B-Instruct and remains around 3–4% across Qwen2.5 and Qwen3 Instruct models from 7B to 32B.

  • Takeaways & Limitations

    The findings support anchoring safety understanding in language-agnostic semantic structure rather than surface text to improve safety generalization to unseen low-resource languages.

  • Takeaways & Limitations

    LASA is most effective when harmful intent is explicitly represented at the bottleneck and may fail for implicit or multi-step malicious content.

Abstract

from arXiv · show

Large language models (LLMs) often demonstrate strong safety performance in high-resource languages, yet exhibit severe vulnerabilities when queried in low-resource languages. We attribute this gap to a mismatch between language-agnostic semantic understanding ability and language-dominant safety alignment biased toward high-resource languages. Consistent with this hypothesis, we empirically identify the semantic bottleneck in LLMs, an intermediate layer in which the geometry of model representations is governed primarily by shared semantic content rather than language identity. Building on this observation, we propose Language-Agnostic Semantic Alignment (LASA), which anchors safety alignment directly in semantic bottlenecks. Experiments show that LASA substantially improves safety across all languages: average attack success rate (ASR) drops from 24.7% to 2.8% on LLaMA-3.1-8B-Instruct and remains around 3-4% across Qwen2.5 and Qwen3 Instruct models (7B-32B). Together, our analysis and method offer a representation-level perspective on LLM safety, suggesting that safety alignment requires anchoring safety understanding not in surface text, but in the model's language-agnostic semantic space.

1 Introduction

LLMs can remain vulnerable in low-resource languages despite strong safety in high-resource languages because semantic understanding generalizes more broadly than text-space safety alignment. LASA addresses this mismatch by aligning safety at a language-agnostic semantic bottleneck, substantially improving cross-lingual safety.

  • Motivation: Safety robustness degrades substantially in low-resource languages even when LLMs perform strongly in high-resource languages.
  • Motivation: High-resource-language safety alignment can achieve near-zero ASR on training languages yet leave about 50% ASR on Swahili.
  • Problem formulation: The paper attributes this gap to language-agnostic semantic understanding paired with language-dominant safety alignment inherited from alignment-data distributions.
  • LASA: LASA identifies the Semantic Bottleneck, trains a Safety Semantic Interpreter there, and conditions response generation on its safety-relevant semantic signal.
  • Results: LASA reduces average ASR from 24.7% to 2.8% on LLaMA-3.1-8B-Instruct and keeps ASR around 3–4% across Qwen2.5 and Qwen3 Instruct models.
  • Results: LASA improves safety across languages, particularly on unseen low-resource languages.

2 Related Work

Prior multilingual safety work expands alignment data or transfers safety behavior across languages, while latent-space methods control safety using hidden-state signals. The paper positions LASA against these approaches by focusing alignment on a semantic bottleneck rather than language-dominated final layers.

  • Cross-Lingual Vulnerabilities: Multilingual vulnerabilities are linked to uneven training-language distributions and can be amplified by mixed-language, multilingual-jailbreak, and code-switching attacks.
  • Multilingual Enhancement: Multilingual safety methods directly align target languages using collected or synthesized data and supervised or preference-based fine-tuning.
  • Multilingual Enhancement: Transfer-based methods reduce safety gaps by aligning low-resource languages to high-resource ones through reward shaping and self-distillation.
  • LLM Safety at Latent Space: Latent-space safety research finds separable safe and unsafe regions and uses hidden-state signals for safety control or inference-time guidance.
  • LLM Safety at Latent Space: The paper argues that existing methods intervening on final layers face a low-resource generalization challenge because those layers are strongly language-dominated.

3 Preliminary: The Semantic Bottleneck

The Semantic Bottleneck is the intermediate layer where multilingual representations are organized most strongly by shared semantic content rather than language identity. The paper identifies it by comparing language- and query-based clustering across layers and validates the resulting cross-lingual structure empirically.

  • Definition: The Semantic Bottleneck is an intermediate layer where representation structure is dominated by semantic content rather than language identity.
  • Measurement: The analysis forms language partitions and query partitions over hidden states to compare language organization with shared semantic organization.
  • Measurement: Average Silhouette scores quantify how strongly representations are grouped by each partition, with higher semantic scores indicating stronger grouping by shared meaning.
  • Identification: The bottleneck layer is defined where semantic organization dominates language organization most strongly.
  • Empirical pattern: At the bottleneck, semantically equivalent inputs across languages form tight clusters, while inputs in one language with different meanings remain separate.
  • Empirical pattern: Across layers, semantic clustering follows an inverted U-shaped trajectory while language clustering follows a U-shaped trend, with semantic organization strongest at intermediate layers.
  • Robustness: Similar bottleneck behavior is consistently observed across architectures and model scales.

4 Methodology

LASA identifies a semantic bottleneck, trains a lightweight Safety Semantic Interpreter on its hidden states, and conditions response generation on the resulting safety signal. Experiments examine cross-language semantic safety transfer and its relationship to multilingual capability.

  • Framework: LASA extracts hidden states from the identified semantic bottleneck layer and processes them with a Safety Semantic Interpreter before response generation.The resulting safety-relevant semantic signals condition subsequent generation.
  • Training procedure: The three-stage procedure identifies the bottleneck layer, trains SSI with frozen model parameters, and then performs semantic-conditioned alignment.The algorithm updates SSI using safety labels and subsequently updates the target model using the interpreter’s semantic signal.
  • Safety Semantic Interpreter: SSI maps bottleneck hidden states to a scalar safety logit using a lightweight MLP trained with binary cross-entropy.The method distinguishes benign from malicious queries at the semantic bottleneck.
  • Cross-language validation: SSI-trained safety semantic accuracy is positively correlated with general capability across languages, following a saturation curve on Swahili.The reported Swahili Qwen2.5 Instruct results fit the curve with R2 = 0.988.
  • Cross-language validation: Safety semantic understanding improves with multilingual competence, but gains diminish after sufficient semantic understanding is achieved.The results support anchoring safety at the semantic bottleneck so improved semantic representations can translate into more robust multilingual safety.
  • Semantic-conditioned alignment: LASA incorporates the SSI’s latent safety logit into KTO-style training so generation is conditioned on semantic safety information.This conditioning associates internal safety semantics with appropriate refusal or compliance patterns.

5 Experiments

LASA is evaluated across multilingual safety benchmarks, model scales, and optimization variants, consistently reducing attack success while preserving general performance. Ablations indicate that locating the Safety Semantic Interpreter at the semantic bottleneck is more important than the specific Stage 2 optimizer.

  • Superior Safety Performance: LASA achieves 1.70% average ASR on MultiJail with LLaMA-3.1-8B, versus 21.00% for the vanilla model.It consistently outperforms the evaluated baselines across languages.
  • Robust Generalization to Low-Resource Languages: LASA reduces Swahili ASR on Qwen2.5-7B-Instruct to 13.0%, while most baselines remain near 50% despite near-0.0% English ASR.The evaluation tests generalization from high-resource training languages to low-resource languages without language-specific safety demonstrations.
  • LASA Maintains General Performance: LASA preserves or slightly improves average M-MMLU, MT-Bench, and MGSM performance after safety alignment.The reported comparisons indicate no typical alignment tax on general capabilities.
  • Ablation Study on SSI Layer: 8.0% ASR from training SSI on the final layer is worse than the 4.4% KTO baseline, while performance reaches its minimum near the semantic bottleneck.Safety alignment degrades when SSI is trained on layers closer to the input or output.
  • Ablation Study on Semantic Conditioned Alignment: Average performance varies by approximately 0.65% across SFT, ORPO, and KTO optimization variants, while all LASA variants reduce ASR versus vanilla models.The results suggest that bottleneck identification and SSI-based conditional control contribute more than the choice of Stage 2 optimizer.
  • Results on Different Scale Models: 4% average ASR is maintained across Qwen2.5 and Qwen3 models spanning 7B–32B parameters.Safety performance generally improves with model scale, while gains at 7B are smaller because those models already show relatively strong safety semantic understanding.

6 Analysis and Discussion

Additional analyses examine how the semantic bottleneck scales across architectures, how translation choices affect LASA, and where semantic alignment remains limited. The bottleneck is consistently mid-network, while implicit semantic inference remains challenging.

  • Semantic Bottleneck Location: The semantic bottleneck occurs at approximately 43%–68% of total network depth across models with 28–64 layers.Its relative depth, rather than a fixed layer index, remains stable as model depth changes.
  • Translation Robustness: Google Translate and NLLB produce nearly identical semantic-bottleneck structure and location to GPT-4o.The bottleneck remains observable without meaningful translator-dependent differences.
  • Translation Robustness: Approximately 1.7% ASR is achieved on MultiJail across translators, indicating that LASA’s safety performance is largely unaffected by translation quality.The reported gains consistently outperform baseline methods.
  • Emoji Expressions: High-semantic-similarity emoji prompts retain low ASR because the model can directly access their underlying meaning.The analysis groups emoji prompts by similarity to textual counterparts.
  • Emoji Expressions: ASR increases for low-similarity emoji prompts that require multi-step reasoning to infer harmful meaning.This identifies a limitation for semantic alignment when harmful content is only implicitly conveyed.
  • Representation Analysis: At shallow and near-output layers, English and Swahili representations separate clearly, whereas intermediate layers are dominated by semantic representation.Within-language clusters also show boundaries between harmful and benign queries.

7 Conclusion

The paper attributes multilingual safety gaps to language-biased alignment and proposes LASA as a semantic-space alternative. Its conclusion emphasizes bottleneck location as a central design choice while identifying implicit reasoning and multimodal settings as future boundaries.

  • Conclusion: LASA addresses the mismatch between language-agnostic semantic understanding and safety alignment biased toward high-resource languages.The method identifies semantic bottlenecks and anchors safety alignment directly in semantic space.
  • Conclusion: Aligning safety in semantic-dominant representation spaces enables more principled and scalable multilingual safety than relying solely on language-specific safety data.The paper frames this as a supported implication of its empirical findings.
  • Future Work: Future work includes multi-step reasoning, implicit semantic inference, multimodal semantic spaces, and other forms of LLM alignment.These settings mark the stated extension areas beyond the demonstrated scope.

Limitations

The paper identifies limitations involving evaluation noise, data diversity, implicit harmful intent, safe-completion scenarios, and the binary design of the Safety Semantic Interpreter.

  • GPT-4o-based evaluation may introduce mislabeling because automated annotation noise is difficult to eliminate.On LLaMA-3.1-8B, GPT-4o judgments achieve over 95% agreement with the human average.
  • LASA is most effective when harmful intent is explicitly represented at the semantic bottleneck.Implicit or multi-step malicious content, such as low-similarity emoji prompts, may not activate appropriate safety signals.
  • Overly homogeneous training data can constrain semantic-bottleneck identification and robust safety understanding.The Safety Semantic Interpreter depends more heavily on the underlying data distribution, trading flexibility for lightweight design.
  • The study excludes safe-completion scenarios because available datasets do not support unambiguous harmful-versus-benign classification.The stated target is refusal of harmful queries and safe alternatives when appropriate.
  • The Safety Semantic Interpreter currently uses a binary benign-versus-malicious classifier rather than richer risk representations.Multi-label or continuous risk modeling is left for future exploration.

Ethical Considerations

The paper frames its work as defensive safety research and reports ethical safeguards for its harmful-query experiments. The supplied passages also describe analysis of model representations and parameter overhead, but do not add further ethical requirements.

  • Ethical safeguards: The experiments use harmful queries from established public safety benchmarks, including MultiJail and HarmBench.The paper states that no personally identifiable information or user-generated private data was collected or used.
  • Ethical safeguards: The work focuses exclusively on defensive mechanisms rather than developing new attack vectors.The authors do not release new highly optimized jailbreak prompts and instead target resilience across linguistic boundaries.
  • Representation analysis: The representation analysis evaluates semantic organization across layers using Silhouette scores and t-SNE visualizations.The analysis is repeated across several multilingual instruction-tuned models.
  • Parameter analysis: The SGA Latent Safety Projector maps H to H, and its reported parameter-expansion ratio is 0.26% for Llama-3-8B and 0.10% for Llama-3-70B.The paper presents these ratios as indicating negligible impact on total capacity and inference latency.

D Reliability of ASR Evaluation

The paper validates ASR evaluation through multilingual capability checks, manual comparison with GPT-4o judgments, and translation-tool analyses. These checks support the reliability and translation robustness of the reported safety measurements.

  • Semantic capability: 43.3 and 44.3 are the average multilingual MMLU scores for Qwen-2.5-7B-Instruct and Llama-3.1-8B-Instruct on Swahili, Bengali, and Javanese.These nontrivial scores are used as evidence of adequate semantic understanding in low-resource settings.
  • Evaluator validation: 0.965 is the average accuracy of Direct GPT-4o evaluation in manual validation of 200 Llama-3.1-8B-Instruct QA pairs.The validation concerns the MultiJail dataset.
  • Translation robustness: The semantic-bottleneck phenomenon remains clearly present when GPT-4o translation is replaced by alternative translation tools.The computational procedures are otherwise kept unchanged.
  • Translation robustness: Around 1.7% is the ASR obtained across translation tools on MultiJail, with no significant differences between translators.The paper reports that these results substantially outperform baseline methods.

G Case Analysis on Emoji Expressions

The emoji case analysis shows that semantic alignment works when symbolic prompts closely preserve harmful intent but can fail when intent is indirect and requires multi-step inference.

  • High semantic similarity: High-semantic-similarity emoji prompts preserve a near one-to-one mapping to malicious intent, enabling clear and consistent refusals.The example concerns malware development and propagation.
  • Low semantic similarity: Low-semantic-similarity emoji prompts create a substantial semantic gap between the symbols and the underlying harmful intent.The model may interpret such prompts as benign risk-analysis scenarios rather than illegal instructions.
  • Limitation: Implicit harmful intent conveyed through weak or indirect semantic cues remains a limitation for semantic alignment approaches.These cases require the model to infer intent through multi-step reasoning before safety signals can activate.
  • Method comparison: The qualitative comparisons include responses from Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct across multilingual harmful prompts.The paper compares SGA with vanilla SFT and preference-based baselines.
Loading 2604.12710v2…