PaperlayerView exact revision on arXiv

Source-linked AI summary

Same Entity, Different Script: Diagnosing Cross-Script Entity Matching Failures in Language Models

Athira Gopal, Patil Prathamesh Pravin, Ankush Kadu, Arpit Jain, Ashwanth Krishnan

arXiv:2610.31407v1cs.CLcs.AI

TL;DR

Identity judgments can change when the same people’s names are presented in different scripts, complicating multilingual entity matching. The paper tests this with controlled name-form substitutions and behavioral, probing, calibration, and causal analyses. Qwen’s same-person accuracy falls sharply in native-script conditions, while interventions at a middle-layer site improve held-out Marathi accuracy.

  • Problem

    The paper asks whether identity judgments remain consistent across script-associated forms of the same names, a requirement for connecting aliases and other name variants across users’ writing systems.

    Source: §1 · p. 1

  • Method

    The study uses controlled identity verification on audited pairs, varying attested name forms while holding the question, people, and relation fixed, then combines behavioral tests with probes, calibration, and causal interventions.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    35.1% in Lat–Lat versus 8.5% in Dev–Dev: Qwen’s Marathi same-person accuracy drops across 601 pairs, while its held-out accuracy rises from 6.5% to 33.5% after a fact-independent window intervention.

    Source: §1 · p. 1, §8 · p. 8

  • Takeaways & Limitations

    The results distinguish script-dependent discrimination from decision criterion placement and identify the second entity’s middle-layer states as an effective intervention site in Qwen.

    Source: §8 · p. 8

    Probe decodability does not uniquely establish retrieval of an identity association, and the causal intervention site does not locate a stored fact.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Multilingual language models can give different identity judgments when the same people's names are written in different scripts. We study this effect using 601 audited identity pairs with attested Devanagari and Latin forms, holding the question and relation fixed. Qwen2.5-7B's same-person accuracy falls from 35.1% in Latin to 8.5% in Devanagari under Marathi prompts, while different-person accuracy remains above 98%. Evaluation across five models reveals distinct response biases, and reduced tokenization fragmentation does not reliably close the gap. Replicating the design on 374 audited Malayalam pairs built from the same entities reproduces the effect in a second script and language family: same-person accuracy falls to 5.3%, about half the matched Devanagari rate, while the margin still ranks matching above non-matching names at 0.925 AUROC. Latin-trained probes recover label information from Devanagari states, exceeding a string-feature baseline under similarity-matched negatives. Scalar calibration recovers 87% of the raw model-probe balanced-accuracy gap, identifying decision criterion placement as a major contributor. Activation patching localises causal sensitivity to the second entity in Qwen's middle layers; one shared sensitivity direction transfers about half the full-state margin correction. A fact-independent intervention at this site raises held-out Marathi same-person accuracy from 6.5% to 33.5%, with 98.5% different-person accuracy. Together, these results distinguish script-dependent discrimination from criterion placement and identify a causal site for improving identity judgments.

1 Introduction

The paper tests whether identity judgments remain consistent across script-associated forms of the same names, a requirement for multilingual entity matching. It finds a large Devanagari same-person accuracy deficit and connects behavioral results to probes, calibration, and causal interventions.

1 Introduction

  • 35.1% in Lat–Lat versus 8.5% in Dev–Dev: Qwen2.5-7B’s Marathi same-person accuracy drops across 601 audited pairs, while different-person accuracy remains above 98%.The same effect appears in Malayalam, a different script and language family.Source: §1 · p. 1
  • The analysis separates cross-script discrimination from response bias, tests whether tokenization efficiency explains the gap, and measures label information through probes and output-score calibration.Activation patching and subspace interventions test where that information can affect the decision.Source: §1 · p. 1
  • A shared sensitivity direction transfers about half the full-state margin correction, and a fact-independent intervention raises held-out Marathi same-person accuracy more than fivefold.The intervention acts at the second entity’s middle-layer state; the account attributes much of the gain to moving the decision criterion at a causally sensitive site.Source: §1 · pp. 1–2

2 Related work

Related work covers cross-lingual consistency, script and language variation in factual judgments, and methods for probing or intervening on model representations. This study fixes the question language while varying supplied name forms and compares decodability, calibration, and causal transfer on corresponding inputs.

2 Related work

  • Prior work examines consistency across translated queries and references in code-mixed coreferential statements, while this design fixes question language and varies the two supplied names.It follows one identity relation across behavioral and internal measurements.Source: §2 · p. 2
  • Prior work localizes cross-lingual factual inconsistency near language-specific final-layer representations; this study instead finds strong middle-layer sensitivity at the second entity.Comparisons with output calibration and other script conditions clarify what the related linear intervention’s accuracy gain changes.Source: §2 · p. 2
  • The paper combines activation patching and linear probes to compare label decodability, output calibration, and causal transfer on corresponding inputs.The controlled task enables those comparisons across the same inputs.Source: §2 · p. 2

3 Task, data and protocol

The task compares identity judgments while changing the script-associated forms of supplied names and holding the question, people, and relation fixed. Its audited datasets use attested forms, hard negatives, and counterbalanced prompts across Marathi, Hindi, English, and Malayalam.

3 Task, data and protocol

  • The main corpus contains 601 manually audited Wikidata identity pairs, each with observed Devanagari and Latin forms, independently varied across four script configurations.Pairs link canonical names to attested aliases, birth names, pseudonyms, or nicknames.Source: §3 · p. 2
  • Each positive has a different-person hard negative selected by occupation, country, and relation tier; 99.8% of negatives share occupation with the positive.A separate analysis rebuilds negatives for name similarity.Source: §3 · p. 2
  • The audit retains conventional, attested name forms and excludes pairs whose scripts encode different name relationships, avoiding the need to generate unfamiliar names.The retained set includes 404 observed-alias relations and 197 relations involving explicit name properties.Source: §3 · p. 2
  • A separately audited Malayalam corpus retains 374 of 380 items; its splits reuse the seed-reproduced Devanagari split, without transferring probes, vectors, or thresholds.Some Malayalam strings are attested or spelling variants; the remainder were machine-transliterated and audited.Source: §3 · p. 2
  • The main Marathi and Hindi corpora cover the same identities, and each is queried in its own language and English using three paraphrases and both yes/no letter assignments.Scoring uses final-position logit margins, reports positive and negative accuracy separately, and measures threshold-independent ranking with AUROC.Source: §3 · p. 2
  • Five models are evaluated, with the 24B models using the Marathi corpus and Qwen providing the detailed causal analysis.A fixed 361/120/120 target-item split supplies training, validation, and test data; distractors come from a shared pool and can cross splits.Source: §3 · p. 3

4 The script gap

Across scripts, Qwen’s largest behavioral deficit is on matching pairs, while negative accuracy remains high; the gap also appears in Malayalam. Reduced tokenization fragmentation alone does not reliably recover performance.

Qwen’s deficit is concentrated on matching

  • The script gap remains among near-duplicate spellings: for 154 pairs above 80% Latin-form similarity, Marathi accuracy is 10.8% in Dev–Dev and 32.1% in Lat–Lat.The gap is larger among the 58 least-similar pairs, so near-duplicate spellings do not account for the phenomenon.Source: Qwen’s deficit is concentrated on… · p. 3
  • Model response biases differ: Gemma accepts most positives and many negatives, while Qwen strongly favors rejection; the Lat–Lat advantage in d′ excludes zero for Qwen in Marathi, Gemma in all three prompt languages, and Llama in English.Llama’s Marathi and Hindi outputs are excluded because of fixed-letter instruction failures; the 24B model comparison is descriptive because negative accuracies and checkpoint versions differ.Source: Qwen’s deficit is concentrated on… · p. 3
  • Lat–Dev exceeds Lat–Lat positive accuracy by 9.7 points in English and 11.1 in Hindi without better per-template ranking, indicating a role for criterion placement as well as discrimination.This mixed-script ordering also appears under English prompts in the Malayalam comparison: Lat–Mlym is 66.4% versus 62.9% for Lat–Lat.Source: Qwen’s deficit is concentrated on… · pp. 3–4
  • 35.1% in Lat–Lat versus 8.5% in Dev–Dev: Qwen’s Marathi positive accuracy falls across 601 items, while negative accuracy stays at least 98.6%; the gap persists under Hindi and English prompts.Lat–Lat exceeds Dev–Dev in margin AUROC by 0.037 [0.025, 0.049] in Marathi, 0.038 [0.026, 0.051] in Hindi, and 0.023 [0.014, 0.033] in English.Source: Qwen’s deficit is concentrated on… · p. 3, Table 1 · p. 4
  • 5.3% versus 10.4%: native-script same-person accuracy is lower for Malayalam than Devanagari on the same 374 entities, with negative accuracy at least 98.6% in every cell.The Malayalam margin AUROC is 0.925 despite the near-floor same-person accuracy; its script gap is 43.5 points versus 30.6 points for Devanagari.Source: Qwen’s deficit is concentrated on… · pp. 3–4, Table 2 · p. 4

4.2 Tokenization and familiar forms

  • Automatic romanizations reduce mean name length from 16.5 tokens to 7.7–10.4 but yield no reliable recovery; attested Latin forms reproduce Lat–Lat within 0.004 across all nine primary model–language cells.Devanagari names are tokenized roughly three times as finely as Latin names, yet surface features predict script-sensitive items at only 0.56–0.60 out-of-fold AUC, versus 0.86–0.96 after adding relation states.Source: §4.2 · p. 4
  • Automatic schemes reproduce attested Latin spelling for only 0.8–4.5% of names, distinguishing movement into Latin characters from presentation of a familiar Latin name.Gemma and Llama use substantially fewer Devanagari tokens per character than Qwen, yet their behavior still depends on script.Source: §4.2 · p. 5

5 Available information and decision calibration

Latin-trained probes recover label information from Devanagari states, including for model errors, while scalar calibration accounts for much of the model–probe accuracy gap.

5 Available information and decision calibration

  • 0.933 balanced accuracy is reached by the second-entity probe on Marathi Dev–Dev inputs, and layer-averaged probes recover 78.9% of exact Marathi failures.The question-mark probe also reaches 0.925 balanced accuracy and 0.867 same-person recall.Source: §5 · p. 5
  • 0.908 versus 0.796 balanced accuracy favors the question-mark probe over a five-feature string classifier on the partially similarity-matched Marathi test set.On the tightly matched subset, model separation remains within 0.01 of its original value; richer orthographic signals and familiarity are not separated from identity retrieval.Source: §5 · p. 5
  • 0.879 versus 0.525 entity-level balanced accuracy follows a validation-selected constant margin offset, recovering 87% of the distance to the Marathi E2 probe.The probe retains a +0.054 advantage [0.017, 0.092] in Marathi and +0.050 [0.021, 0.079] in English at their selected operating points.Source: §5 · p. 5

6 Localising causal sensitivity

Patching localises strong causal sensitivity to Qwen’s second entity in middle layers; a shared rank-one direction transfers part of the margin correction, but margin movement does not itself ensure answer flips.

6.1 Position, name and computation

  • 4.79 versus 3.78 Marathi margin changes after canonical-name/alias swapping show that the effect follows the second slot, the first position that can attend to both names.E1 remains within 0.42 of zero across all six language–order combinations; nonce names also show E2-over-E1 sensitivity.Source: §6.1 · p. 6
  • 5.41 versus 0.51 Marathi correction from cumulative attention versus MLP patches supports stronger transfer through attention patches.Multi-head patches exceed individual-head patches, consistent with distributed transfer; these interventions are not an additive decomposition of residual effects.Source: §6.1 · p. 6

6.2 A shared direction transfers correction

  • 0.37–0.61 cosine similarity between Dev–Dev and Lat–Lat local gradients across Marathi layers 12–17 motivates testing shared subspaces against random directions.Random directions reach at most 0.004 in magnitude.Source: §6.2 · p. 6
  • Training-item directions are validation-selected at layer 16 and rank one, then evaluated on the frozen held-out paraphrase P2.The sensitivity direction is the normalized mean Dev–Dev gradient over gated training items; alternatives include donor-difference, principal-component, ridge-probe, and random directions.Source: §6.2 · p. 6
  • 1.30 [0.73, 1.95] Marathi margin gain from one sensitivity direction is about half the 2.54 full-state patch gain; the probe direction transfers approximately zero.Negative accuracy remains 1.00 in these subspace tests; the comparison does not establish that the probe direction is unused.Source: §6.2 · p. 6, Table 6 · p. 7
  • Rank-one results concern margin correction, while substantial answer flips require multi-layer intervention because a score shift may not cross the decision threshold.The mean-difference direction produces larger but broader changes, including movement of same-person pairs outside the gate.Source: §6.2 · pp. 6–7

6 Localising causal sensitivity

  • 2.66–3.16 margin gains from E2 patching across languages contrast with little E1 effect; joint layers 12–17 patching flips 50%, 60%, and 71% of held-out items.Matched-negative accuracy remains 100%, and the joint effect falls below one margin point at layers 18–20.Source: §6 · p. 5, Table 5 · p. 7

7 Improving judgments through a shared intervention

A fact-independent, shared intervention applies layer-specific corrections estimated from training items. Held-out same-person accuracy improves, with results depending on selected depth and scale and with ranking effects varying by template.

7 Improving judgments through a shared intervention

  • Layer-specific vectors estimated from 361 training items summarize mean E2 changes associated with replacing either name’s script and are shared across items.Validation selects layers 12–17 and scale by balanced accuracy, using negative accuracy as a tiebreak without imposing a negative-accuracy floor.Source: §7 · p. 7
  • Marathi held-out same-person accuracy rises from 0.065 to 0.335 with 0.985 negative accuracy; English rises from 0.283 to 0.590 with 0.976 negative accuracy.Marathi gains occur on all three templates despite selection on one; applying the frozen Marathi intervention at E1 yields only 0.071 positive accuracy.Source: §7 · p. 7
  • 0.083 test positive accuracy from the earlier single-layer intervention contrasts with validation accuracy rising from 0.083 to 0.233 when widening to layers 12–17.Extending the scale search reaches 0.392; held-out gains follow validation-selected depth and magnitude.Source: §7 · p. 7
  • 0.312 true-positive rate at 0.012 false-positive rate from scalar offset is near the vector’s 0.335 at 0.015, while template-level comparisons show a ranking component.On the tuning template, vector TPR is 0.425 versus 0.308 for threshold adjustment at a matched false-positive rate; ranking also improves on templates 0 and 2.Source: §7 · p. 7, Table 7 · p. 8
  • The frozen vector improves Lat–Lat positive accuracy too, narrowing script disparity in three of four corpus–language arms but widening it by 1.9 points in Marathi.The paper distinguishes decoding, AUROC ranking, and decisions at a reported false-positive rate because improvement in one does not guarantee improvement in the others.Source: §7 · p. 8

8 Conclusion

Controlled name-form substitutions expose cross-script identity-matching failures. In Qwen, a fact-independent intervention at the second entity’s middle-layer site improves held-out Marathi same-person accuracy while retaining high different-person accuracy.

8 Conclusion

  • 33.5% held-out Marathi same-person accuracy, up from 6.5%, with 98.5% different-person accuracy after a fact-independent window intervention.A shared sensitivity direction transfers about half the full-state margin correction.Source: §8 · p. 8
  • Evaluating discrimination and decision criterion across both script conditions makes intervention gains interpretable and provides a basis for improving multilingual identity verification.Script contrast changes discrimination as well as criterion placement, making calibration and script robustness distinct evaluation targets.Source: §8 · p. 8

Limitations

The study’s evidence is bounded by its audited entities and scripts, Qwen-focused mechanistic diagnosis, and dependent held-out evaluations. Its construction and gate procedures also impose specific limits on interpretation.

Limitations

  • The corpus covers 601 relations across Marathi, Hindi, and English prompts, plus 374 of the same relations in Malayalam; other scripts, entity types, and factual relations remain untested.Hindi reuses Marathi identities with different Devanagari renderings; Malayalam evaluation is behavioral on Qwen only, with no probe, patching, or intervention results.Source: Limitations · p. 9
  • Mechanistic findings are detailed for Qwen; the 24B comparison does not isolate Indic post-training, and Sarvam-M’s causal gate contains only one Marathi item.Other models provide behavioral and probe comparisons plus exploratory causal analyses, with different response biases and effective layer ranges.Source: Limitations · p. 9
  • Probe decodability does not uniquely establish identity-association retrieval, and tight similarity matching is available for only 144 of 601 items.Orthographic regularities or familiarity may contribute; nonce names also alter response bias, and the E2 result identifies an intervention site rather than a stored fact’s location.Source: Limitations · p. 9
  • Global distractors can cross data splits, stable causal gates yield only 12–24 held-out items per language, and shared test items make control analyses non-independent replications.The sensitivity-vector steering sweep is validation-only and reported separately from held-out results.Source: Limitations · p. 9

A Corpus construction

  • The retained corpus requires observed Devanagari and Latin forms, and manual auditing excludes identical fields, joined aliases, metadata, and mismatched name relationships.Hindi uses the same entities and byte-identical Latin forms as Marathi, but 528 of 601 Devanagari pairs differ; retained items pass the audit with no recorded issues.Source: A Corpus construction · p. 11
  • Hard negatives favor shared occupation and country attributes, but are not matched on name similarity and were not independently rebuilt across models.The selection scores shared occupation, country, and relation tier; a separate analysis rebuilds negatives for name similarity.Source: A Corpus construction · p. 11

B Prompts

  • Counterbalancing requires both answer-key orders to be correct, while prompt scoring uses final-position logits without sampling but can vary slightly with batch composition.Three paraphrases are used per prompt language; P2 is the hardest in every condition, and bfloat16 batch-shape effects flip 0.4% of positive prompts without changing reported comparisons.Source: B Prompts · pp. 11, 13

C Knowledge and causal gates

  • Causal gates select items that fail in Dev–Dev but succeed in a donor condition across paraphrases and answer keys; Gemma’s acceptance bias prevents its binary gate from firing.The study instead uses a margin gate on negative pairs for Gemma, so the failed binary diagnostic is not evidence that the effect is absent.Source: C Knowledge and causal gates · p. 13

D Full behavioural results

Qwen’s script-dependent accuracy loss is concentrated on same-person judgments, while different-person accuracy remains near ceiling. Token fragmentation is related to the script benefit but does not suffice to explain or reverse it; automatic romanizations reduce fragmentation without reliably recovering performance.

D Full behavioural results

  • The gap spans every paraphrase and substantive relation tier, although paraphrase P2 is markedly harder in every condition.Source: D Full behavioural results · p. 13
  • Devanagari accuracy is similarly low for foreign-origin and Indic-origin names, with no Devanagari advantage for Indic names.Under Marathi prompts, accuracy is 8.3% for 489 foreign-origin entities and 9.4% for 112 Indic-origin entities.Source: D Full behavioural results · p. 13
  • The script effect falls almost entirely on SAME pairs; DIFFERENT accuracy stays near ceiling across script combinations.Source: D Full behavioural results · p. 13, Figure 2 · p. 14

E Sensitivity measure

  • Qwen’s English and Hindi d′ intervals do not exclude zero because the false-alarm floor dominates estimates, while AUC separates the same conditions in every prompt language.Source: E Sensitivity measure · pp. 13, 16

F Fragmentation analyses

  • Fragmentation correlates only weakly with the item-level Latin benefit (ρ = +0.146 over 601 items), despite a monotone quartile dose response.Mean benefit rises from 3.78 to 6.34 across quartiles spanning token ratios 2.12 to 4.49.Source: F Fragmentation analyses · p. 16, Table 17 · p. 17
  • Token ratios poorly predict script-sensitive items (0.56–0.60 AUC), compared with 0.86–0.96 after adding relation states; tested fragmentation reductions did not restore performance.Source: F Fragmentation analyses · p. 16, Table 18 · p. 17

G Transliteration

  • Across schemes, native names average 16.5 tokens; automatic romanizations average 7.7–10.4 versus 5.7 for human Latin, recovering half to four-fifths of the token gap but none of the performance.Automatic schemes exactly match the human Latin field for 0.8–4.5% of names. Substituting the corpus’s own Latin field reproduces Lat–Lat within 0.004 in all nine model-by-language cells.Source: G Transliteration · pp. 16, 18
  • The transliteration condition varies spelling fidelity as well as script, because no correct-but-noncanonical Latin-spelling condition was tested.Mean string similarity is 0.82 for Indic-origin names.Source: G Transliteration · p. 18

H Probing

Cross-script probes reveal graded label information despite incorrect decisions, while causal patching localizes corrective sensitivity to the second entity and shows that sustained interventions can flip answers.

H Probing

  • At layer 3, isolated Devanagari-name retrieval reaches 0.511 and 0.456 top-1 across Qwen’s two name roles, without surrounding context to support retrieval.This is a cleaner identity test than contextual alignment, where the unchanged name can identify the fact.Source: H Probing · p. 18
  • Probe same recall orders failure groups from 0.944 on script-robust items to 0.662 on never-learned items, indicating graded label signal rather than a blanket same bias.Recall is 0.789 on script-sensitive items and 0.913 on other known items; the never-learned group remains above chance.Source: H Probing · p. 19
  • At E2, contextual alignment is 0.97 when the first span changes versus 0.49 when the second changes, but the unchanged name can still identify the fact.Robust and script-sensitive items separate in contextual alignment between layers 3 and 8; this measures fact-context alignment, not isolated-name identity.Source: H Probing · p. 21, Figure 3 · p. 22

I Full patching profiles

  • 57% of Marathi and 77% of English gated items flip under sustained three-site patches, while E2 peaks at layer 16 before the handoff to the question mark at layer 18.The first-entity site has little effect, and adding it to an E2 patch contributes little; the flip rates pool gated items across splits.Source: I Full patching profiles · p. 21

J Components and heads

  • Cumulative attention patching transfers 5.41 Marathi margin points versus 0.51 for the MLP; the top ten heads reach 5.60 versus 0.30 for the individual-head mean.The effect is distributed across several heads rather than concentrated in one sparse mechanism.Source: J Components and heads · p. 23

L Jacobian geometry

  • The Devanagari gradient is smaller than the Latin gradient at every layer in all three prompt languages; their directions grow more alike with depth but never approach closely.Figure 5 depicts the overlapping but noncoincident directions and lower Devanagari sensitivity.Source: L Jacobian geometry · p. 26

M Recovery details

The recovery analysis contrasts limited single-layer gains with larger gains from applying a shared correction across layers, while comparing it with familiar Latin forms and automatic romanization.

M Recovery details

  • Human Latin-form anchoring reaches 0.219 Marathi positive-pair accuracy versus 0.108 for automatic romanization, but anchoring supplies a familiar form in the prompt.The learned vector is the intervention described as requiring no information about the test entity.Source: M Recovery details · pp. 26, 29
  • Held-out Marathi same-person accuracy reaches 0.335 with the shared vector applied across layers 12–17, compared with the 0.065 baseline.The same vector is applied at a larger scale across the layer band; the percentage of the Lat–Lat gap closed is not a measure of closure because the reference also moves.Source: M Recovery details · pp. 26, 29

N The window intervention

The window intervention applies a shared correction across E2’s middle layers, improving held-out accuracy across conditions; the reported gain is largely a shift in decision criterion rather than a replicated repair.

N The window intervention

  • At single-layer scale α=6, windowing raises Marathi positive-pair accuracy from 0.083 to 0.233; extending the scale grid raises it to 0.392 before it falls.Negative-pair accuracy remains 1.000 through α=8 and declines slowly only at larger scales.Source: N The window intervention · p. 29, Table 36 · p. 30
  • The additive vector sums two raw single-script-swap differences, restoring the first-entity component that the direct Lat–Lat minus Dev–Dev difference at E2 largely lacks.At E2, the full difference aligns more with the second-entity swap than with the first-entity swap.Source: N The window intervention · pp. 29, 31
  • The single-layer grid stopped at α=8, roughly half the median distance needed to reach held-out Lat–Lat counterparts along the rank-one direction.The validation-selected scales, 16 for Marathi and 24 for English, fall within or just above the reported median-distance ranges.Source: N The window intervention · p. 31, Table 38 · p. 32
  • The rank-one direction with each item’s own donor coordinate moves margins but flips no gated Marathi item, separating score movement from answer correction.These are held-out rows of the reference-arm comparison.Source: N The window intervention · p. 31, Table 39 · p. 32
  • Accuracy rises in all six held-out template cells, and the correction exceeds the Lat–Lat reference on the hardest Marathi template despite selection on template 0.The vector was estimated on 361 training entities and tested on 120 others.Source: N The window intervention · p. 31, Table 40 · p. 32
  • The held-out split had already been scored four times in both corpora, so these appendix estimates are a second look rather than an independent first evaluation.The authors report the estimates on that basis.Source: N The window intervention · p. 31
  • Across four held-out arms, the window intervention raises accuracy; Marathi comes within 0.9 points of its Lat–Lat reference and the other arms exceed theirs, but the movement is largely a criterion shift.The Hindi positive-pair result pays a 13-point negative-pair-accuracy cost; the table reports replicated movement, not replicated repair.Source: Table 41 · p. 33
  • On validation, the sensitivity direction reaches 0.833 balanced accuracy at α=1, but negative-pair accuracy falls to 0.008 by α=3.This sweep was validation-only; the sensitivity direction uses answer information on training items and a smaller gated training subset than the mean-difference vector.Source: N The window intervention · p. 33

O Subspace sweep

The subspace sweep finds that a rank-one sensitivity direction captures much of the margin effect, while intervention amount and direction shape which items move. Gemma shows a related positional pattern, but its different behavioural gate and weaker localization do not replicate Qwen’s failure mechanism.

O Subspace sweep

  • 0.56, 0.79 and 0.72 in Marathi, and 0.46, 0.71 and 0.57 in Hindi: the rank-one direction carries these fractions of the whole patch across validation folds.The frozen test uses the hardest paraphrase, P2, where Dev–Dev accuracy is 3.2%; the authors treat this as a limit of the evaluation.Source: O Subspace sweep · p. 33
  • 4.00 to 4.54 in Marathi and 4.05 to 5.09 in Hindi: raising rank from one to eight adds little, concentrating the effect in one sensitivity direction.The sweep peaks at layers 15–17 in both languages; the probe and random directions remain near zero.Source: O Subspace sweep · pp. 35–36
  • 0.68 to 1.10 times the gated repair: a constant step moves never-correct items nearly as much, while item-derived steps move them −0.15 to 0.23 times as much.Selectivity also depends on direction: with item-derived amounts, the sensitivity direction moves off-gate items less than the mean-difference direction in all nine folds.Source: Table 43 · p. 34, O Subspace sweep · p. 35
  • 3.55 to 7.00 across Marathi and Hindi fold cells: a shared coordinate moves gated margins at least as much as item-specific coordinates, so amount does not explain the full-versus-additive outcome gap.Negative-pair accuracy stays at 1.00 in the reported comparisons.Source: O Subspace sweep · p. 35
  • 0.065 to 0.335: applying the correction across layers 12–17 raises Marathi accuracy, whereas the single-layer margin result flips almost no items.The paper attributes much of the accuracy gain to moving the decision criterion, not a change specific to the failing condition.Source: O Subspace sweep · pp. 35–36

P Gemma

  • 2.21 in Marathi and 2.07 in English: Gemma’s second-entity effects are smaller than Qwen’s, and its Marathi question-mark effect peaks higher at 4.21.Its gate reads negative-pair errors, so this is not a direct replication of Qwen’s same-person failure mechanism; Gemma’s held-out intervention does not repair positive-pair accuracy.Source: P Gemma · pp. 36–37

Q Llama

Llama’s Marathi results are excluded because some paraphrases do not attempt the task, though its English evaluation retains measurable cross-script effects. Mistral-Small shows second-entity sensitivity at a shallower relative depth than Qwen, and the rebuilt corpus removes name-pairing errors that could confound the script comparison.

Q Llama

  • Llama’s Marathi arm is excluded because one of three paraphrases is informative; the Hindi arm is excluded because one paraphrase is at chance, making the pooled 19.3% Dev–Dev accuracy unrepresentative.A fixed-token response scores 50% across both answer keys regardless of content.Source: Q Llama · p. 37
  • +1.00 [0.90, 1.11]: Llama’s English d′ gap is larger than Gemma’s +0.73, while Qwen has no measurable gap in that prompt language.Held-out probes reach 0.925 across prompt languages and 0.954 on Marathi states, despite Llama being at chance on the Marathi arm.Source: Q Llama · pp. 37, 39
  • 0.94 against 0.56 in Marathi and 0.91 against 0.53 in English: Llama’s in-prompt retrieval shows the same positional asymmetry, but its causal gate is too small for a stronger mechanistic claim.Its component effects are at the noise floor: attention is 0.098 and MLP 0.079 against residual stream 0.482.Source: Q Llama · p. 39

R Mistral-Small causal probe

  • Layer 23 carries 0.01 under Marathi prompts and 0.00 under English; Mistral-Small’s effects peak instead at layers 11 and 18, shallower in relative depth than Qwen’s band.On the depth-matched sub-band, the Marathi second-entity effect has already decayed below the question-mark effect.Source: R Mistral-Small causal probe · p. 40

S Corpus errors identified and removed

  • The rebuild jointly pairs and audits name fields, removing items where Devanagari and Latin encode different names rather than different script forms of the same relation.All reported results use the 601 items that survive the audit; the ten identified mismatches would bias the measured gap in the reported direction.Source: S Corpus errors identified and removed · p. 40

T Surface-similarity baseline

The string baseline can decode labels from Devanagari names, so ordinary probe success does not establish that model states contribute information beyond surface similarity. Under a partially similarity-matched negative map, the probe beats the string baseline, though matching covers only a subset and does not establish retrieved entity knowledge.

T Surface-similarity baseline

  • 0.933 and 0.946 balanced accuracy: the classifier using only Devanagari strings exceeds the probe’s 0.925 and 0.942 on the same held-out items.The classifier uses five name-string features and the same test items as the probe.Source: T Surface-similarity baseline · p. 42, Table 51 · p. 43
  • +0.112 [+0.067, +0.163] in Marathi and +0.083 [+0.025, +0.142] in English: the probe beats the string baseline under the similarity-matched map, unlike under the shipped map.The matched map is partial, and beating the baseline still shows label decoding rather than proving entity association retrieval.Source: T Surface-similarity baseline · p. 42, Table 52 · p. 43
  • 0.942 under Marathi and 0.954 under English: at layer 16, the second entity is the best probe position in both languages and the only position peaking within the 12–17 causal window.This convergence in position and depth is not independent proof of entity retrieval; the probe decodes labels.Source: T Surface-similarity baseline · p. 42, Table 53 · p. 43
  • 22% for the string baseline and 39% for the probe: on 18 held-out Marathi positives with sequence ratio at or below 0.5, the probe does better, but both are weak on this small subset.Most positive pairs are name expansions sharing tokens; genuine low-similarity aliases are a minority.Source: T Surface-similarity baseline · p. 42, Table 54 · p. 43
  • 8.5% in Dev–Dev versus 0.933 balanced accuracy for the string classifier: available surface separability does not mean the model converts that evidence into a correct same-person decision.The measured sequence-ratio separation is 0.423; the authors caution that one similarity measure cannot rule out all surface explanations.Source: T Surface-similarity baseline · pp. 42, 44
  • 0.003 separation is achievable on 144 items, but the best separation across all 601 is 0.184, because highly similar positives are structurally difficult to match with distractors.Matching all five features at once leaves only 30 items, so the authors report a partial matched result rather than a fully matched corpus.Source: T Surface-similarity baseline · p. 44

U Nonce-name control

The appendix tests whether script effects reflect entity matching and documents how accuracy, ranking, interventions, and evaluation protocols behave across conditions, models, and corpora. It also reports Malayalam-corpus knowledge and causal gates, paraphrase results, and consistency across runs in the shared English Lat–Lat arm.

U Nonce-name control

  • Nonce names shrink the English Dev–Dev deficit from 28 points on real names to 5, but near-universal “different” responses make this a bias-shift result, not evidence that the effect requires entities.On nonce names the model answers “different” to at least 0.999 of pairs across conditions and prompt languages; Marathi is at floor, with Lat–Lat accuracy 0.036. Causal patching retains the E2-over-E1 positional signature at roughly a third of the real-name magnitude, while donor quality confounds nonce flip rates. The script gap also persists across similarity bands and is largest for the least similar positive pairs; the relation-tier split is not a valid similarity split.Source: U Nonce-name control · pp. 44–45, Table 56 · p. 45

V Condition ranking against condition accuracy

  • Lat–Dev can be more accurate without better ranking: under English prompts accuracy is 9.7 points higher with no detectable ranking difference, while Hindi template 2 has a 0.131 ranking deficit alongside 11.1 points more accuracy.Under Marathi prompts, Lat–Dev has detectable ranking deficits on two templates and no accuracy gain. On the Hindi corpus, Lat–Dev is 10.5 points more accurate under English prompts; Lat–Lat is a reference point, not the model’s best condition. These cross-condition observations are consistent with criterion placement but do not establish it causally.Source: V Condition ranking against condition… · p. 46, Table 58 · p. 47

W The condition control

  • Applying the frozen vector to Lat–Lat raises its acceptance in every arm by 0.222 to 0.289, so the intervention is not specific to Dev–Dev or a gap-closure measure.The position control finds that changing the addition site matters: the first entity moves accuracy by at most 0.054, while readout results are weaker and less consistent than the second-entity site.Source: W The condition control · p. 46, Table 59 · p. 47, Table 60 · p. 48

X Investigation history

  • Widening the intervention window raises Marathi validation accuracy from 0.083 to 0.233 at α=6; extending the scale grid raises it to 0.392, while the sensitivity direction outperforms the mean-difference vector on validation.The selected Marathi scale is 12 and the peak is 16, beyond the single-layer grid’s limit of 8. The correction vector is nearly orthogonal to the local sensitivity direction, explaining its small single-layer effect; the sensitivity direction’s validation figures are not held-out test results. A reference arm transfers about half the whole-state patch’s margin change but flips no gated Marathi items, though the measurements differ in scope.Source: X Investigation history · pp. 49–50, Table 65 · p. 52

Y Protocol

  • The intervention’s offset is selected on validation under a negative-accuracy floor, whereas the vector has no such floor; the held-out split is reused across four evaluations and is not four independent replications.Vector scales are selected on template 0 and reported across all three templates, making per-template results important; the threshold-only comparison is an oracle diagnostic rather than a deployable rule.Source: Y Protocol · p. 50

Z The 24B comparison

  • Sarvam-M’s script gap is 6.9 points versus Mistral-Small-24B’s 25.8, but Sarvam-M has only one item under the shared causal gate and is not analysed causally.Sarvam-M’s Dev–Dev same-person accuracy is 75.2%, compared with Qwen’s 8.5%; its larger deficit is by prompt language, while Marathi negative-pair accuracy is lower than Mistral-Small’s. Mistral-Small’s exploratory sweep reproduces E2-over-E1 sensitivity.Source: Z The 24B comparison · p. 50

Compute and reproducibility

  • The runs use one NVIDIA A100 80GB in bfloat16, with pinned model revisions and a resumable pipeline whose reported figures are recomputable from result tables.The pipeline scores 57,696 behavioural prompts per model and caches hidden states at four token positions across all layers.Source: Compute and reproducibility · pp. 50–51

Malayalam corpus and behavioural results

  • The Malayalam corpus carries over Latin forms for 380 entities, with 760 Malayalam strings; a speaker audited all 380 items and distractors, and six items were rejected.Selection favoured familiar public figures and is not a representative sample; 171 strings were exactly attested in cached Wikidata records, 18 were spelling or joiner variants, and 571 were machine transliterations. Attestation alone did not establish that paired forms were the same name; the audit addressed that question.Source: Malayalam corpus and behavioural results · pp. 51–52
  • Across 4,488 byte-identical English Lat–Lat prompts, the two runs agreed on 99.6% of trials; all 17 disagreements were within 0.5 margin points of zero.The largest shared-arm margin movement was 1.75 points, and only two prompts moved more than 1.5 points, measuring batch-composition sensitivity of bfloat16 reductions.Source: Malayalam corpus and behavioural results · p. 53
  • For Malayalam prompts, 223 entities were Lat–Lat correct but both-native incorrect, versus 151 in matched Devanagari; only 10 versus 23 were correct in all four configurations.The corresponding known-entity counts were 299 versus 263; no causal experiment was run on the Malayalam corpus.Source: Malayalam corpus and behavioural results · p. 53
  • On 76 held-out entities, the third Malayalam paraphrase yielded 2.0–7.9% same-person accuracy, versus 12.5–47.4% in English on the same question.All three templates were native-checked; the first two paraphrases carry the result, while the third is near the floor in every configuration.Source: Malayalam corpus and behavioural results · p. 53, Table 67 · p. 54
Loading 2610.31407v1…