PaperlayerView exact revision on arXiv

Source-linked AI summary

Same Answer, Different Reasons: Trace-Level Divergence in Multilingual Large Language Models

Rui Sun, Peizhao Mei, Xiewei Cheng, Kaizhuo Chen, Taoran Li, Zhiyuan Yu, Varun Chandrasekaran

arXiv:2610.29092v1cs.AIcs.CL

TL;DR

Existing multilingual evaluations measure final-answer agreement or whole-response similarity, not whether languages provide comparable supporting evidence. The paper introduces a calibrated framework for comparing visible traces as structured sequences. Across benchmarks, holistic similarity overstates trace agreement, and human judgments align with structural rather than holistic similarity.

  • Problem

    Existing multilingual evaluations measure final-answer agreement or whole-response similarity but do not determine whether users receive comparable supporting evidence across languages.

    Source: §1 · pp. 1–2

  • Method

    The framework compares visible reasoning traces as structured sequences using calibrated structural similarity, order consistency, and holistic similarity measures.

    Source: §1 · p. 2

  • Results

    Across evaluations, holistic similarity is consistently higher than structural similarity, and 52.1%–67.3% of top-quartile holistic-similarity examples are not also top-quartile in structural similarity.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Answer-level and holistic evaluations can overstate the consistency of multilingual explanatory artifacts, motivating trace-level evaluation for multilingual deployment.

    Source: §1 · p. 2, §7 · p. 9

    Human validation covers two English-anchored language pairs, with 60 verified-correct trace pairs each, all from one model; the authors do not establish generalization to other languages or models.

    Source: Limitations · p. 11

Abstract

from arXiv · show

Multilingual large language models are increasingly deployed in settings where visible reasoning traces support verification, education, and safety auditing, yet standard evaluations typically measure only final-answer agreement or holistic response similarity. As a result, they establish outcome consistency without determining whether users across languages receive comparable supporting evidence. We introduce a trace-level divergence framework that compares visible reasoning traces as structured sequences, decomposing agreement into structural similarity, order consistency, and holistic similarity, with pair-specific calibration to enable comparison across languages. Across three reasoning benchmarks, seven languages, and six frontier LLMs, the natural baseline---embedding each whole response and comparing the two by cosine---systematically overstates agreement: $52.1\%$--$67.3\%$ of examples in the top quartile of holistic similarity are not in the top quartile of structural similarity. This dissociation is stable across sentence encoders and step segmentations, and blind bilingual annotation studies in two English-anchored language pairs show that human judgments align with structural similarity rather than with the holistic baseline. A same-language control shows that the level of divergence is language-specific: two English traces for paraphrases of the same question agree more closely than an English trace and its counterparts in other languages. Our results show that answer-level and holistic evaluations can substantially overestimate the consistency of multilingual explanatory artifacts, motivating trace-level evaluation for reliable multilingual deployment.

1 Introduction

Existing evaluations capture whether multilingual prompts produce compatible answers or similar whole responses, but not whether they provide comparable supporting evidence. This paper introduces a calibrated trace-level framework and finds that holistic similarity overstates agreement in visible reasoning traces.

1 Introduction

  • Multilingual evaluations focus on final-answer agreement or whole-response similarity, leaving comparable supporting evidence across languages unmeasured.Whole-response comparisons can miss differences in intermediate claims, assumptions, problem decomposition, or execution order.Source: §1 · pp. 1–2
  • The framework compares visible traces as structured sequences using structural similarity S, order consistency τ, and holistic similarity H, with pair-specific calibration.S measures preservation of reasoning content, τ preservation of reasoning order, and H serves as a whole-response baseline.Source: §1 · p. 2
  • 52.1%–67.3% of examples in the top quartile of holistic similarity are not also in the top quartile of structural similarity.Across all model–dataset evaluations, holistic similarity is H = 0.770–0.857, while structural similarity is S = 0.502–0.629 and order consistency is τ = 0.307–0.470, even for matching verified-correct answers.Source: §1 · p. 2
  • Blind bilingual annotations in two English-anchored language pairs align with structural similarity rather than the holistic baseline.The study concerns visible explanatory artifacts, not whether traces faithfully expose a model’s internal computation.Source: §1 · pp. 1–2
  • Two English traces agree more closely than English and other-language counterparts in every tested cell, while divergence also appears within English.The reported comparison uses a same-language control with English paraphrases and the identical pipeline.Source: §1 · p. 2
  • The authors present trace-level evaluation as a complement to outcome-based evaluation for assessing multilingual explanatory artifacts.Their stated motivation is reliable multilingual deployment.Source: §1 · p. 2

2 Problem Formulation

The paper distinguishes agreement in final answers from consistency in the visible reasoning traces accompanying those answers. It formalizes trace divergence across content, order, and whole-response similarity, with sentence similarities calibrated by language pair.

2 Problem Formulation

  • The problem formulation studies whether semantically equivalent prompts in different languages yield consistent visible traces as well as the same answers.It formalizes outcome and trace consistency and introduces calibration for comparing traces across language pairs.Source: §2 · p. 2
  • For each reasoning problem, the setup collects semantic-preserving translations and obtains a response for each language variant.Responses are parsed into final answers and step-structured visible traces.Source: §2 · pp. 2–3

2.1 Outcome and Trace Consistency

  • Outcome consistency asks whether translated prompts produce the same final answer; trace consistency asks whether their visible explanations are comparable.The two notions are complementary because matching answers can accompany different supporting claims, decompositions, or execution order.Source: §2.1 · p. 3
  • Outcome consistency alone provides no information about whether accompanying visible reasoning traces remain consistent across languages.The formulation contrasts final-answer agreement with the supporting claims, decompositions, and order exposed in traces.Source: §2.1 · p. 3
  • The framework represents trace divergence with structural similarity S, order consistency τ, and whole-answer cosine H.S captures preservation of step-level reasoning content, τ captures ordering of corresponding steps, and H compares complete responses without localizing divergence.Source: §2.1 · p. 3

3 Method

The method parses visible responses into steps, calibrates sentence similarities within each language pair, and aligns trace content hierarchically. It derives separate scores for structural similarity and the ordering of matched steps, alongside a whole-response baseline.

3 Method

  • The pipeline parses responses into steps, calibrates sentence similarities per language pair, and builds a step-pair matrix for structural and order scores.The whole-response baseline H is computed separately.Source: §3 · p. 3

3.1 Step Parsing

  • The parser applies the same ordered rules across languages, prioritizing step headers and numbered lists before paragraph boundaries, and stops at the final-answer line.Each non-empty step is divided into sentences with a Unicode-aware regular expression; parsing uses no LLM.Source: §3.1 · p. 3

3.2 Per-Pair Calibration

  • Pair-specific calibration maps the unrelated-sentence floor to 0 and the self-translation ceiling to 1, without equating intermediate values across pairs.Because the map is monotone within a pair, it does not change within-pair ranks used in the reported comparisons.Source: §3.2 · p. 4

3.3 Two-Level Greedy Alignment

  • Hierarchical alignment compares sentences within step pairs, then aggregates step-pair scores into structural similarity S.Bidirectional maxima allow many-to-one step correspondences, accommodating different step granularities; S measures content preservation independently of order.Source: §3.3 · p. 4

3.4 Soft Kendall-τ Order Consistency

  • Order consistency τ is derived from the step-pair matrix by soft correspondence assignments and concordant, discordant, or tied matched positions.The directional scores are symmetrized; τ is interpreted jointly with S and is undefined for empty traces or traces with fewer than two steps.Source: §3.4 · pp. 4–5

4 Experimental Setup

The experiments span three reasoning benchmarks, seven languages, and six models from four families. Metrics are computed after answer- and trace-validity filtering, with question-level bootstrap intervals reported for the benchmark results.

4 Experimental Setup

  • The evaluation covers MMLU-ProX, StrategyQA, and SciBench across Arabic, English, Spanish, Hindi, Russian, Vietnamese, and Chinese.The benchmarks test academic multiple-choice reasoning, yes/no commonsense reasoning, and quantitative STEM reasoning, respectively.Source: §4 · p. 5
  • Six models from four families are evaluated on 500 problems per dataset–model run.Trace metrics are reported after the answer- and trace-validity cascade; one-language-correct cases contribute only to Δacc.Source: §4 · p. 5
  • Table 1 reports problem-level 95% bootstrap confidence intervals from 2,000 resamples, jointly resampling each question’s 21 language pairs.Step scores use LaBSE, holistic similarity uses BGE-M3, and soft Kendall-τ uses sharpening power 3.Source: §4 · p. 5

5 Results

Across the evaluations, final-answer agreement did not ensure comparable reasoning traces: whole-response similarity remained high while step content and order diverged. The analyses show that these metrics capture distinct aspects of traces, divergence levels differ across languages, and structural similarity aligns with human judgments in a small validation study.

5.1 Correct Answers Still Hide Trace Divergence

  • H remains 0.770–0.857 while S is 0.502–0.629 and τ is 0.307–0.470, even when both language outputs have the same verified-correct answer.On MMLU-ProX, across six 500-sample evaluations, S=0.513–0.569, τ=0.348–0.469, and H=0.787–0.800.Source: §5.1 · p. 5

5.2 Holistic Similarity Overstates Consistency

  • 52.1%–67.3% of top-quartile holistic-similarity examples fall outside the top quartile of structural similarity, so whole-response cosine overstates trace agreement.Quartiles are computed within each evaluation and language pair; H−S is treated as a diagnostic contrast, not a calibrated distance.Source: §5.2 · pp. 5–6, Figure 1 · p. 6

5.3 Structural Coverage and Order Are Non-Redundant

  • S, τ, and H capture complementary aspects of traces: step coverage, execution order, and holistic semantic similarity, respectively.τ and H are nearly uncorrelated, while S has only moderate independent association with both; collapsing the metrics obscures task- and model-specific divergence.Source: §5.3 · pp. 6–7, Table 2 · p. 7

5.4 Divergence Is Larger Across Languages Than Within One

  • Two English traces agree more closely than English-anchored cross-lingual traces in all six dataset–model cells, while holistic–structural rank dissociation also occurs within English.The control compares meaning-preserving English paraphrases with cross-lingual pairs; the rank dissociation is 41.2%–60.0% within English versus 50.5%–64.6% across languages.Source: §5.4 · p. 7

5.5 Where Divergence Appears and What It Signals

  • The final step is less aligned than middle steps by 0.042, and low S or τ signals divergence rather than failure.Divergence is larger on harder items; ranking by S places 44.0% of shared-wrong-answer pairs in the least-similar quarter, versus 28.9% for H and 25% at random, at a 2% base rate prioritizing a fixed review budget rather than detecting errors.Source: §5.5 · pp. 7–8

5.6 Robustness and Human Corroboration

  • Across three encoders and three datasets, every evaluation has 50.1%–65.5% of top-quartile-H examples outside the top quartile of S; equal-step-count filtering leaves all 18 evaluations between 49.5% and 71.0%.Absolute calibrated levels are less stable, so the authors interpret S and τ by rank within a fixed encoder.Source: §5.6 · p. 8
  • In two English-anchored pairs, blind human equivalence judgments track S and τ and are unrelated to H, but the studies are small and limited to one model.The studies are presented as corroboration, not independent evidence that S is superior to alternatives.Source: §5.6 · pp. 8–9

6 Related Work

Prior work evaluates multilingual reasoning mainly through outcomes, whole-response similarity, or monolingual step-level scoring. This paper situates trace-level comparison as a cross-lingual extension that measures divergence between two visible traces rather than their faithfulness to internal computation.

Multilingual reasoning evaluation

  • Multilingual evaluations commonly measure accuracy or final-answer consistency, while recent whole-response comparisons still collapse reasoning traces into one representation.This can leave intermediate claims, assumptions, decompositions, or execution order unexamined.Source: §6 · p. 9

Trace-level reasoning evaluation

  • ROSCOE scores a single chain against a reference for quality, whereas this work compares two valid traces and extends step-level reasoning evaluation across languages with pair-specific calibration.The paper also distinguishes its aim from work controlling the language of thought, which measures accuracy cost rather than trace divergence.Source: §6 · p. 9

Faithfulness of the visible trace

  • Trace comparison is orthogonal to whether a visible chain faithfully reflects internal computation; monitors consume the emitted text, including for closed-weight models.Related work also studies cross-lingual rationale transfer without step- or order-level alignment.Source: §6 · p. 9

7 Conclusion

The paper finds that agreement on final answers does not ensure agreement in visible reasoning traces across languages. Its framework separates complementary dimensions of trace consistency, with divergence patterns stable across several analyses and supported by limited human annotation.

7 Conclusion

  • For the same correct answer, supporting evidence can differ by requested language, including steps another reader never sees or the same steps in a different order.This pattern appears in all 21 evaluated language pairs, including the typologically closest.Source: §7 · p. 9
  • Decomposing consistency into S, τ, and H reveals complementary axes and locates where traces part; a single-score evaluation can obscure these distinctions.The rank-based result is stable across sentence encoders and step segmentations, and small blind annotation studies on one model’s traces in two English-anchored pairs are consistent with it.Source: §7 · pp. 9–10
  • Because the dissociation also appears between English traces, the framework extends to process-level consistency beyond cross-lingual settings, including paraphrases, prompts, and decoding strategies.Source: §7 · p. 10

Limitations

The paper’s claims concern visible reasoning traces, not internal computation, and its evidence is bounded by rank-based comparisons, limited validation, and untested downstream effects.

Limitations

  • Visible traces do not establish internal computation faithfulness; activation-level analysis would exclude closed-weight models, including two evaluated models.The authors limit their claims to the traces consumed by verification, education, and auditing pipelines.Source: Limitations · p. 11
  • Encoder changes preserve rank structure rather than absolute scale, and calibration aligns only endpoints; absolute cross-pair values remain descriptive.The same-language control is a level comparison, although paired within question and with calibration exposure measured.Source: Limitations · p. 11
  • Human validation covers two English-anchored language pairs, with 60 verified-correct trace pairs each from one model; generalization beyond those settings is unestablished.The authors do not establish whether the metrics track human judgments for other models or language pairs without English.Source: Limitations · p. 11
  • The same-language control covers two models, about 100 questions per benchmark, and one paraphrase per question, so it does not isolate non-zero-temperature decoding variation.Its effect is unanimous in direction but varies about threefold in size across cells.Source: Limitations · p. 11
  • The association between low structural similarity and shared wrong answers rests on 225 pairs from two language pairs, is largely carried by item difficulty, and supports no causal conclusion.The paper does not show that divergence impairs verification, comprehension, or safety; testing those effects would require human readers.Source: Limitations · p. 11

Ethical Considerations

The section documents safeguards and methodological choices for interpreting multilingual trace comparisons, including prompting and evaluation controls, calibration, parsing, translator-independence checks, and supplementary checks of task-dependent language-pair patterns. It clarifies that the framework measures differences in visible explanations—not reasoning quality or harm—and reports stable metric rankings under an independent translator alongside greater sensitivity for some Hindi-involving pairs.

Ethical Considerations

  • The framework diagnoses whether traces follow different routes to the same answer, not reasoning quality, and its scores should not rank languages, gate access, or serve as training targets.The main analyses use answer-correct pairs, so low divergence scores indicate different routes rather than wrong answers; shared-error associations motivate human review, while equal accuracy can mask unequal explanatory support across languages.Source: Ethical Considerations · p. 11

A The Necessity of Per-Language-Pair Calibration

  • Pair-specific calibration addresses language-dependent cosine floors and ceilings by anchoring unrelated sentences at an empirical floor and self-translations of verified-correct traces at an attainable-similarity ceiling.The ceiling uses the 90th percentile of bidirectional self-translation similarities to reduce sensitivity to unusual translations; constants were fixed before evaluation and held across model–dataset runs.Source: A The Necessity of Per-Language-Pair… · p. 13, §B.2.1 · p. 14, §B.2.2 · p. 14

B.1 Step-Parsing Details

  • The parser applies the same ordered marker rules across languages without a language-specific semantic parser, and a small manual audit found no evident step-boundary errors in inspected traces.The audit checked roughly 20 traces per model–dataset cell, but was too small to bound the residual error rate and supports the automatic diagnostics rather than independently measuring boundary fidelity.Source: §B.1 · p. 13, §B.1.1 · pp. 13–14

B.2.3 What Calibration Does and Does Not Establish

  • Calibration removes pair-dependent endpoint drift but does not guarantee that intermediate values mean the same thing across language pairs, so pooled absolute scores are descriptive.The headline comparison ranks examples within a fixed model–dataset–language-pair cell, where monotone rescaling preserves ordering except when clipping creates ties; separate sensitivity checks vary the ceiling percentile and use an independent-family translator.Source: §B.2.3 · p. 14, §B.2.5 · p. 15, §B.2.6 · p. 15

B.2.4 Rank Invariance of the Headline Statistic

  • Across all three Claude Opus 4.6 evaluations, recomputing the headline rank statistic without calibration reproduced the calibrated values, with every difference below 10^-4.The reported 52.1%–67.3% range is therefore not produced by calibration; calibration supplies a comparable scale, while the within-pair rank statistic is monotone-invariant except for ties from clipping.Source: Table 5 · p. 15, §B.2.4 · p. 15

B.5 Controlled Metric Checks

  • Controlled perturbations test dimension-specific behavior: removing a step should lower structural similarity, while permuting unchanged steps should leave it relatively stable and reduce order consistency.The tie-aware order score is primary because it also penalizes multiple source steps assigned to the same target step; structural and order scores expose different kinds of divergence, neither of which determines whether a difference is harmful.Source: §B.5 · p. 16, §B.4 · p. 16, §B.7 · p. 16

B.6 Whole-Answer Holistic Similarity

  • The whole-response baseline does not align steps or explicitly measure order, and BGE-M3 avoids the truncation that LaBSE’s 256-token limit would impose on many full traces.The framework adapts greedy step matching and soft ordering ideas but calibrates alignment per language pair and reports structure and order separately.Source: §B.6 · p. 16, §B.8 · p. 17

C.1 Datasets, Models, and Inference

  • Evaluations cover seven languages and 21 unordered language pairs, sample 500 problems per model–benchmark run, and retain only trace pairs that pass answer verification and trace review.The benchmarks are MMLU-ProX, StrategyQA, and SciBench; the model suite uses six models with greedy decoding.Source: §C.1 · p. 17

C.2 Prompt Templates

  • A fixed prompt family, greedy decoding, and identical token budgets and retry policies were used across runs; English step headers kept the segmentation cue constant across languages.The parser audit found effectively complete coverage in every language.Source: §C.2 · pp. 18–19

C.3 Outcome-Level Accuracy Audit

  • Final-answer accuracy varied across languages and datasets, and trace-level aggregation included only pairs whose two traces passed answer verification.Figure 2 reports accuracy over the full 500-problem inference sets before the trace-validity filter defining Q in Table 1.Source: §C.3 · p. 19, Figure 2 · p. 20

C.4 Calibration Intervals

  • StrategyQA had the largest sentence-level calibration-floor spread, while SciBench and MMLU-ProX often reached the ceiling cap of cmax = 0.98.Table 6 summarizes the sentence-level intervals used in the main analyses.Source: §C.4 · p. 19, Table 6 · p. 20

D.1 Translator-Independence Probe

  • Across 21 language pairs, original Claude-Opus-ceiling and independent Mistral-Large-3-ceiling estimates correlated at ρC = +0.92, ρS = +0.89, and ρτ = +0.87; pooled mean shifts were ΔS = +0.002 and Δτ = −0.003.Source: §D.1 · p. 21, Table 7 · p. 21
  • Hindi-involving pairs showed the largest downstream metric shifts, with the largest ceiling change instead on hi-zh; τ amplified sensitivity relative to S.The top three |Δτ| shifts were hi-vi (0.036), hi-ru (0.032), and hi-zh (0.021); the top two |ΔS| shifts were hi-vi (0.023) and hi-ru (0.019), while |ΔC| was largest for hi-zh (+0.013).Source: §D.1 · p. 21

D.2 Pair-Level Result Tables

  • Language-pair difficulty varied by task: English–Spanish and Spanish–Russian were consistently among the strongest pairs, while Hindi- and Chinese-involving pairs shifted more across datasets.These pair-level rankings support the reported model–dataset–language interaction pattern rather than a single fixed language-pair ordering.Source: §D.2 · p. 21

D.3 Cross-Dataset and Cross-Model Patterns

Across datasets, languages, and models, holistic similarity often fails to reflect step-level alignment, and the metrics reveal distinct forms of divergence. Parser coverage, matched-step analyses, controlled perturbations, and qualitative examples support interpreting these differences as trace divergence rather than artifacts of parsing or step granularity.

D.3 Cross-Dataset and Cross-Model Patterns

  • Dataset rankings replicate strongly for H, moderately for τ, and less for task-sensitive S; MMLU-ProX, SciBench, and StrategyQA show distinct metric profiles.MMLU-ProX has lowest mean S (0.536) and highest mean τ (0.399); SciBench has highest mean S (0.571) and lower τ (0.373); StrategyQA has highest mean H (0.813) and largest H−S gap (0.263).Source: §D.3 · pp. 21, 23, Table 9 · p. 22
  • Model leaders vary by dataset and metric, and whole-output similarity does not monotonically predict step-level consistency.Claude Opus 4.6 leads S on MMLU-ProX and both S and τ on StrategyQA, while SciBench’s leaders differ by metric; Claude Haiku 4.5 has StrategyQA’s highest H (0.857) and largest H−S gap (0.327).Source: §D.3 · p. 23

D.4 Rank-Based Holistic–Structural Comparison

  • 59.9% of top-quartile whole-answer cosine examples are not also in the top structural quartile, and 12.4% fall in the bottom structural quartile.Mean within-pair Spearman correlation ranges from 0.21 to 0.45 across evaluations.Source: §D.4 · p. 23, Table 10 · p. 24

D.5 Parser Coverage

  • Parser coverage is 100.0% in six languages and 99.8% in Arabic, with no fallback use; parser failure is not a material source of the reported divergence.Coverage measures traces with at least one parsed step; boundary placement is assessed separately.Source: §D.5 · p. 23

D.6 Step-Count-Matched Rank Statistic

  • 49.5%–71.0% of equal-step-count pairs still show top-quartile H without top-quartile S, so step-count mismatch does not account for the dissociation.The matched statistic shifts by an average magnitude of 1.8 percentage points, with a net shift of −0.4 points; mean S rises by 0.003–0.023.Source: §D.6 · pp. 23, 25, Table 11 · p. 24

D.7 Sanity-Check Validation

  • Reversing the same reasoning keeps H and S high but drives τ negative, while abstraction and unrelated substitutions sharply reduce S despite nonzero H.The controlled English–Chinese perturbations test whether the metrics respond to changes in reasoning content and order.Source: §D.7 · p. 25, Table 12 · p. 26

D.8 Sharpening Ablation: Power-Law vs Softmax-Temperature

  • Across sharpening schemes, cross-evaluation rankings of mean τ remain similar, supporting the qualitative conclusions drawn with power-law p=3.Rank correlations are at least 0.85 across choices and at least 0.92 for alternatives within mean-absolute-tau distance 0.12 of p=3; p=2 and p=5 correlate at least 0.98 with p=3.Source: §D.8 · pp. 25–26, Table 13 · p. 26

E.1 Qualitative Trace Examples

  • Verified examples show high holistic similarity alongside weak structural alignment when traces differ in step decomposition or intermediate reasoning.The StrategyQA English–Spanish example has H=0.875, S=0.338, and τ=0.009; the SciBench Hindi–Russian example has τ=0.832 but S=0.269, while the MMLU-ProX English–Chinese example has H=0.857 and S=0.384.Source: §E.1 · p. 27

E.2 Rejected-Sample Case Studies

Rejected examples show that trace similarity and answer correctness are distinct: high similarity can accompany shared errors, while language-specific interpretations or assumptions can produce divergent wrong answers. The accompanying annotation protocol compares reasoning equivalence on verified-correct pairs and samples across the natural ranges of both H and S.

E.2 Rejected-Sample Case Studies

  • Rejected traces have lower S and τ than verified traces while H stays near 0.80, but this diagnostic is not a primary metric because rejected cases can conflate answer error, answer inconsistency, and reasoning divergence.The comparisons cover Claude Opus 4.6 on StrategyQA and MMLU-ProX and DeepSeek-V3.2 on StrategyQA.Source: §E.2 · p. 28
  • High trace similarity can reflect a shared wrong answer: Claude Opus 4.6 gives the same incorrect institutional interpretation in all seven languages, with S=0.756 and H=0.887.DeepSeek-V3.2 also answers incorrectly in all seven languages on this question.Source: §E.2 · p. 28, Table 17 · p. 28
  • Rejected StrategyQA traces also diverge when languages interpret Hooke Sea differently, splitting between an ordinary-sea reading and a fictional-location reading.DeepSeek-V3.2 answers yes in Arabic, English, and Vietnamese, but no in Spanish, Hindi, Russian, and Chinese; S=0.449 and τ=0.221.Source: §E.2 · p. 28, Table 18 · p. 29
  • On a rejected MMLU-ProX item, English, Spanish, and Chinese choose different wrong options despite following the same broad rate-table procedure, with unstated rate assumptions differing across languages.The gold option is B; the traces select A, D, and I respectively.Source: §E.2 · p. 29, Table 19 · p. 29

F.1 Protocol

  • Annotators judge whether two verified-correct traces reach the answer by the same reasoning, recording ordinal equivalence, difference types, and whether differences materially affect verification.They are blinded to S, τ, H, and stratum assignment; each item is independently labeled by at least two annotators.Source: §F.1 · p. 30

F.2 Sampling

  • Crossed sampling across H and S terciles preserves both metrics’ natural ranges, avoiding a design that would suppress H’s variance by holding it high.Sample standard deviations closely match the pool for both metrics, and EN–ES sample H–S correlation is 0.25 versus 0.254 in the pool.Source: §F.2 · p. 30, Table 20 · p. 30

F.3 Correlational Results

In two English-anchored language pairs, structural similarity aligns with human judgments of reasoning equivalence more closely than holistic similarity. The paper also compares a model judge, reports supporting analyses, and describes its annotation guidance and volunteer recruitment.

F.3 Correlational Results

  • Human equivalence judgments correlate more strongly with S than H in both language pairs, with the difference between dependent correlations significant for EN–ES and EN–ZH.Williams’ tests give Δρ=0.41, p=0.007 for EN–ES and Δρ=0.437, p=0.0069 for EN–ZH; pooled ρ(equiv, S)=0.40, p<0.001, N=120.Source: §F.3 · pp. 30–31

F.4 Comparison with an LLM-as-Judge Baseline

  • On 60 EN–ZH pairs, S correlates with human equivalence more strongly than the LLM judge or holistic baseline, while confidence intervals overlap for S and the judge.The authors’ defensible claim is that S is at least as faithful to human judgment as the judge and statistically significant on this sample; the judge is not a substitute for human consensus.Source: §F.4 · p. 31

F.5 Supporting Analyses

  • Mean human equivalence rises monotonically across S terciles and is flat across H terciles; in EN–ES, ROC AUC is 0.81 for S versus 0.54 for H (DeLong p = 0.012).The respective AUC intervals are [0.68, 0.92] and [0.36, 0.71]; the EN–ES positive class holds 11 items, so the result is presented as evidence for dissociation rather than as a deployable screening threshold. Shared-anchor groups are indistinguishable on shared items.Source: §F.5 · pp. 31–32

F.6 Annotation Guide

  • Annotators compare reasoning rather than surface wording, disregard correctness because both answers are verified correct, and distinguish cosmetic differences from changes to a reviewer’s understanding.The guide classifies equivalence as Same, Partially different, or Substantially different and records difference types and materiality.Source: §F.6 · pp. 32–33

F.7 Annotator Recruitment and Consent

  • The studies used adult bilingual volunteers from the authors’ academic and professional networks; no personal data was collected, and ethics-board review was not sought under the stated minimal-risk protocol.Participation was voluntary and stoppable at any time; the authors describe the tasks as comparing model-generated text from public benchmarks.Source: §F.7 · p. 33

G Encoder Sensitivity

The encoder-sensitivity analysis finds that the rank-based mismatch between holistic and structural similarity persists across three encoders, including when both metrics use the same encoder. Absolute calibrated similarity levels vary by encoder, and the same-language control compares English paraphrases with English-anchored cross-lingual pairs under matched conditions.

G Encoder Sensitivity

  • The sensitivity study re-derives pair-specific calibration for each encoder and validates its reimplementation against cached LaBSE results.The LaBSE rerun has Spearman 0.959 on S, mean |∆S| = 0.0044, and unchanged top-quartile mismatch.Source: G Encoder Sensitivity · p. 33
  • 50.1%–65.5% of top-quartile-H examples are not top-quartile-S across all three encoders and datasets, including when H and S use the same encoder.The range closely tracks the 52.1%–67.3% main-evaluation range; the finding also holds with a distilled 118M CPU model.Source: G Encoder Sensitivity · p. 34, Table 21 · p. 34
  • Absolute H−S gaps vary by encoder, so calibrated S and τ values are interpreted by rank within a fixed encoder.The average gap changes from 0.207 to 0.225 under BGE-M3 and from 0.207 to 0.125 under MiniLM; H remains fixed.Source: G Encoder Sensitivity · p. 34, Table 21 · p. 34

H Same-Language Control

  • The encoder study covers one model family and three datasets, and does not test other model families or models below 118M parameters.The same-language control is described as an appendix design and full-results analysis summarized in §5.4.Source: G Encoder Sensitivity · p. 34, H Same-Language Control · p. 35

H.1 Design

  • The control adds meaning-preserving English paraphrases, answered with the same prompt, decoding, parser, retention cascade, and metric settings as translated conditions.It runs on two models across three benchmarks; the same-language pair receives a paraphrase-based ceiling rather than an identity-translation ceiling.Source: §H.1 · p. 35

H.2 Results

In the same-language control, English paraphrase traces agree more closely than English traces paired with other languages in every tested cell. Yet holistic-to-structural rank dissociation also appears within English, and the control’s scope is limited to two models and one paraphrase per question.

H.2 Results

  • English paraphrase traces have higher S than English-anchored cross-lingual counterparts in every cell, with ∆S from +0.041 to +0.136.The effect clears the preset threshold in all four cells with a threshold; subtracting calibration bias leaves ∆S at +0.046 to +0.124.Source: §H.2 · p. 36, Table 22 · p. 37, Table 23 · p. 37
  • Within English, 41.2%–60.0% of top-quartile-H examples are not top-quartile-S, showing holistic-to-stepwise dissociation also occurs within a language.The within-language statistic is lower than the cross-lingual value in five of six cells.Source: §H.2 · p. 36, Table 24 · p. 37
  • The control covers two models, uses one paraphrase per question, and does not test non-zero-temperature resampling to isolate decoding variation.Claude Opus 4.6, the model behind most detailed analyses, is not included.Source: §H.2 · p. 36

I Where Divergence Appears and What It Signals

I.1 Positional Structure

The positional analysis finds that alignment declines toward the end of English traces, with the last step least aligned in every benchmark. The difficulty and correctness analyses find that low structural similarity marks hard items, while difficulty matching weakens the shared-wrong-answer contrast.

I.1 Positional Structure

  • The analysis scores each English step by its best match anywhere in the other trace, without a monotonicity constraint, for pairs with verified-correct answers in both languages.It covers 31,277 English-anchored pairs across 18 evaluations; order is measured separately by τ.Source: §I.1 · p. 38
  • The last English step is least aligned in every benchmark, and the gap widens rather than shrinks among pairs with equal step counts.Step length accounts for part of the terminal gap on StrategyQA and MMLU-ProX, but the gap is flat across SciBench step-length terciles.Source: §I.1 · p. 38, Table 26 · p. 39
  • Across all three benchmarks, SciBench branches earliest and most often, StrategyQA latest, and MMLU-ProX between, at every tested threshold.The branch-point threshold changes the levels, so the analysis interprets only the ordering.Source: §I.1 · p. 38, Table 27 · p. 39
  • Across six target languages, alignment has a flat opening and terminal decline; the decline from first to last quintile is 0.052–0.078.The level is highest for Spanish and lowest overall for Hindi, without a proposed mechanism.Source: §I.1 · p. 38, Table 28 · p. 39

I.2 Difficulty and Correctness Outcomes

  • The positional analysis is descriptive and English-anchored, so it does not establish causality or describe pairs between two non-English languages.The correctness-outcome analysis is also limited to two English-anchored language pairs and validity-filtered questions; τ is not relied on because its shared-wrong-answer contrast changes sign across benchmarks.Source: §I.1 · p. 38, §I.2 · p. 40
  • After matching shared-wrong-answer pairs to both-correct pairs by difficulty k, the S contrast falls from +0.293 to +0.073, with an interval including zero.Matching on dataset and k reduces it further to +0.016; the authors characterize the adjusted value as a lower bound, not an unbiased estimate.Source: §I.2 · pp. 39–40, Table 30 · p. 40
  • The authors conclude that low S marks hard items, where shared errors concentrate, rather than establishing that S tracks correctness beyond difficulty.The shared-wrong-answer group is restricted to questions where English and the target language are both wrong.Source: §I.2 · p. 40
  • Reviewing the least-similar 25% of answer-agreeing pairs targets the 225 shared wrong answers among 11,401 pairs, a 1.97% base rate.The result concerns allocating a fixed review budget; the authors say low precision does not support a screening threshold.Source: §I.2 · p. 40, Table 31 · p. 40

Related papers

Loading 2610.29092v1…