PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent-Space Intervention for Cross-Lingual Factual Consistency: Consistency Improvements without Accuracy Drops

Faeze Ghorbanpour, Constanza Fierro, Alexander Fraser, Anders Sogaard

arXiv:2608.28860v1cs.CL

TL;DR

Multilingual models can answer the same factual question differently across languages, motivating a method to align their internal representations. The paper trains cross-lingual autoencoders and applies inference-time latent corrections, finding improved consistency with English across QA formats while AE-based strategies preserve factual accuracy.

  • Problem

    Factual knowledge retrieval and answers remain uneven across languages, creating cross-lingual factual inconsistency.

  • Method

    The paper trains layer-specific autoencoders on parallel multilingual representations and applies inference-time latent-space corrections without modifying model weights.

  • Results

    AE-based intervention improves representation alignment and cross-lingual consistency with English across open-ended and multiple-choice QA, while AE and AE+PCA preserve factual accuracy.

  • Takeaways & Limitations

    Autoencoder reconstruction can improve cross-lingual consistency without an accuracy cost, whereas mean-shift yields larger open-ended gains with reduced accuracy.

  • Takeaways & Limitations

    Evaluation is limited to a focused set of languages and factual relations, so broader benchmarks, languages, and relation types are needed.

Abstract

from arXiv · show

Large Language Models (LLMs) often answer the same factual question differently across languages. We study whether cross-lingual latent-space intervention can reduce this inconsistency. We train layer-specific autoencoders on parallel multilingual representations and apply inference-time corrections to factual QA prompts. We find that latent intervention improves geometric alignment between languages, and that this improvement translates into consistent gains in cross-lingual consistency with English across both open-ended and multiple-choice QA formats, without degrading factual accuracy. In open-ended QA, Spearman's rank correlation between English and non-English languages improves substantially, with gains of 0.16 for English-Arabic and 0.20 for English-Russian pairs. In multiple-choice QA, answer agreement with English improves consistently across both KLAR and mParaRel. Ablations show that AE reconstruction yields consistent gains at no accuracy cost, while PCA projection contributes marginally, and mean-shift produces substantially larger consistency gains in open-ended QA at the cost of some accuracy.

1 Introduction

Factual knowledge is unevenly accessible across languages, producing inconsistent answers to the same factual questions. The paper asks whether aligned latent representations can diagnose and reduce this inconsistency without retraining.

  • Pretraining-data imbalance makes facts frequent in some languages but rare or absent in others.
  • Cross-lingual factual inconsistency occurs when a model answers the same factual query differently across languages.
  • The paper asks whether a comparable cross-lingual representation space can improve factual consistency and support lightweight inference-time corrections.
  • The proposed autoencoder maps multilingual representations into a shared latent space and applies corrections without modifying model weights.
  • Autoencoder-based intervention improves cross-lingual representation alignment and semantic consistency while largely preserving question-answering accuracy.

2 Related Work

Prior work identifies multilingual factual inconsistency as both an uneven-knowledge problem and a representational problem. This paper builds on those diagnostics to evaluate a mitigation method without retraining.

  • Existing work includes alignment matrices, forward-hook steering, shared latent spaces, and English-pivot projection modules.
  • Factual knowledge is unevenly distributed across languages because multilingual pretraining data are imbalanced.
  • Models can remain inconsistent across languages even when prompts are paraphrases of the same fact.
  • Mechanistic analyses suggest that facts occupy a language-independent concept space before diverging into language-specific representations near prediction.
  • The paper uses established benchmarks and metrics to evaluate a mitigation method rather than providing only a diagnostic account.

3 Our Approach

The approach learns a shared latent space from multilingual representations, then applies inference-time corrections through forward hooks. Reconstruction, PCA projection, and mean-shift provide alternative intervention strategies.

  • The method defines inconsistency when a multilingual model predicts different answer choices for the same fact across languages.
  • Cross-lingual Latent Space: A shared latent space reduces language-specific variation by mapping semantically equivalent multilingual representations into closer alignment.
  • Autoencoder Training: The autoencoder objective combines self-reconstruction, parallel-pair alignment, variance, and covariance losses.
  • Autoencoder Training: Variance regularization prevents dimensional collapse, while covariance regularization reduces redundancy in the latent representation.
  • Autoencoder Training: The second training phase freezes the shared encoder and trains one language-specific decoder per language.
  • Inference-time Correction: At inference time, a forward hook encodes a mean-pooled hidden state, applies a latent correction, decodes it, and adds the residual to every token position.
  • Inference-time Correction: The AE variant leaves the latent vector unchanged, whereas the projection variant removes principal components identified from cross-lingual difference vectors.
  • Inference-time Correction: The mean-shift variant moves each language representation toward the cross-lingual mean without selecting a pivot language.

4 Experimental Setup

Experiments assess latent-space alignment and factual QA across diverse languages, models, and benchmarks. The setup compares multiple intervention layers and reports both accuracy and cross-lingual agreement.

  • Datasets: Autoencoder training uses parallel TED sentences, while FLORES+ evaluates latent-space alignment.
  • Datasets: Evaluation covers Arabic, English, Dutch, Russian, and Chinese, plus KLAR and mParaRel factual benchmarks.
  • Datasets: KLAR provides prompts in 17 languages, while mParaRel contains relational-fact paraphrase sets across 45 languages.
  • Models: The study evaluates three open-source 8B-parameter models spanning different multilingual pretraining regimes.
  • Evaluation Metrics: Metrics cover representation alignment, factual accuracy, and cross-lingual agreement for multiple-choice and open-ended QA.
  • Choice of intervention layer: Intervention layers are selected in the final third of each model because representations separate into language-specific clusters toward the network end.

5 Results

AE-based intervention improves cross-lingual representation alignment and consistency with English across open-ended and multiple-choice QA, while weaker strategies preserve factual accuracy. Mean-shift delivers larger open-ended consistency gains but introduces an accuracy trade-off.

  • Representation alignment: AE-based intervention consistently improves cross-lingual representation alignment, especially for typologically distant languages and at later layers.These are the layers where raw representations become increasingly language-specific.
  • Open-ended QA: 24.91 is the average open-ended Spearman rank correlation with English after AE+Mean-shift, versus 6.68 before intervention.English-Arabic improves from 2.52 to 18.17, while English-Russian improves from 4.55 to 24.35.
  • Multiple-choice QA: 85.43% is the average English-pair answer agreement for KLAR after intervention, up from 85.09%.For mParaRel, agreement rises from 86.70% to 86.97%, with gains consistent across English-paired language pairs.
  • Strategy ablations: AE+Mean-shift raises average open-ended rank correlation to 24.91 but lowers average accuracy from 56.77 to 53.45.The largest accuracy declines occur for Dutch and Russian, exposing a consistency–accuracy trade-off.
  • Accuracy: AE and AE+PCA leave open-ended accuracy essentially unchanged, while multiple-choice accuracy changes by less than 0.3 points in all conditions.AE accuracy changes from 56.77 to 56.40 and AE+PCA to 56.30.

6 Conclusion

The study tests whether inference-time intervention in a cross-lingual latent space can reduce factual inconsistency in multilingual language models. Across KLAR and mParaRel, AE-based intervention improves representation alignment and translates into improved cross-lingual consistency, particularly for typologically distant languages and middle-to-late layers.

  • The paper investigates whether inference-time intervention in a cross-lingual latent space can reduce factual inconsistency in multilingual language models.
  • AE-based intervention improves cross-lingual representation alignment across KLAR and mParaRel and translates into improved cross-lingual consistency.
  • The strongest alignment improvements occur for typologically distant languages and at middle-to-late layers.

Limitations

The study’s evidence is limited by its focused language, benchmark, relation, and model-scale coverage, while geometric alignment does not reliably ensure factual accuracy or agreement.

  • Evaluation covers a focused set of languages, factual relations, and benchmarks, so broader testing is needed to assess generality.The authors specifically call for additional benchmarks, languages, and relation types.
  • Experiments use three model families at a single 8B parameter scale, leaving generalization to larger models untested.Larger-scale evaluation would test whether the observed final-third layer heuristic generalizes.
  • The lightweight corrections may miss inconsistency distributed across relation-specific or nonlinear directions.The authors suggest more expressive methods as a possible way to obtain stronger factual gains while retaining interpretability.
  • Latent intervention improves geometric alignment more reliably than factual accuracy or agreement, showing that alignment alone is insufficient for factual consistency.Future interventions should explicitly preserve answer-relevant information while reducing cross-lingual divergence.

Ethical Considerations

The work uses public multilingual resources and inference-time interventions that leave model weights unchanged. It evaluates factual accuracy alongside consistency and releases artifacts to support auditing and assessment.

  • The study uses publicly available multilingual benchmarks and openly licensed models without human subjects or personal data.The work states that it does not involve crowdsourcing or personal-data collection.
  • The method operates at inference time, leaves model weights unchanged, and introduces no new training data or supervision.These properties make the intervention straightforward to audit, disable, or apply selectively.
  • The evaluation reports factual accuracy alongside consistency because agreement across languages alone is not sufficient.The paper frames joint reporting of correctness and consistency as necessary for responsible assessment.
  • The authors release code, trained models, and openly available artifacts for research use and assessment.The stated artifact-use policy follows intended purposes and license terms.

G Loss Ablation

Loss ablations show that reconstruction and explicit alignment are essential to the autoencoder’s latent space: removing them sharply weakens recoverable content, retrieval, or alignment.

  • Removing self-reconstruction collapses the latent space, reducing retrieval accuracy to 0.84 and cosine similarity below raw LLM representations.The result indicates that the latent space retains almost no recoverable content without reconstruction.
  • Removing alignment reduces parallel-sentence retrieval accuracy from 95.46 to 83.22, producing a functional but substantially weaker latent space.The ablation demonstrates that alignment requires explicit supervision rather than emerging from reconstruction alone.
  • The full objective outperforms the raw LLM space on both cosine similarity and parallel-sentence retrieval accuracy.The ablation evaluates these measures on FLORES+ using Llama-3.1-8B.

H Results on Additional Models

Across Aya-expanse-8B, Llama-3.1-8B, and Qwen-3-8B, the main intervention patterns persist, with effective layers in the final third and gains varying with baseline alignment.

  • AE-based intervention improves average consistency with English in both open-ended and multiple-choice formats across all three models.The most effective intervention layers consistently fall in the final third of each network.
  • On Llama-3.1-8B, AE+PCA raises open-ended consistency from 21.64 to 23.47 without reducing accuracy.This model has the lowest baseline cross-lingual agreement among the additional-model comparison.
  • On Llama-3.1-8B, AE+Mean-shift raises multiple-choice agreement from 76.31 to 78.29 without reducing accuracy.The result shows a corresponding gain for the multiple-choice setting.
  • Qwen-3-8B shows smaller gains because its baseline cross-lingual agreement is already high.The paper links gain magnitude to how well aligned a model is before intervention.

I Two-Phase vs. Joint Training

The autoencoder’s two-phase training is stronger than joint training, which degrades both cross-lingual consistency and factual accuracy, especially at earlier layers.

  • Training comparison: Two-phase training outperforms joint training for both cross-lingual consistency and factual accuracy across evaluated layers.Table 6 evaluates the alternatives on Llama-3.1-8B using KLAR.
  • Training comparison: 28.30 points: multiple-choice accuracy falls from 78.12 to 49.82 at L20 under joint training.This is the reported early-layer degradation for multiple-choice factual accuracy.
  • Training comparison: 34.44 points: open-ended accuracy falls from 49.00 to 14.56 at L20 under joint training.The reported drop is severe at the earlier evaluated layer.
  • Training comparison: The joint objective lets language-specific decoders absorb alignment pressure, destabilizing the shared encoder.In two-phase training, freezing the encoder preserves cross-lingual geometry while decoders specialize.
Loading 2608.28860v1…