PaperlayerView exact revision on arXiv

Source-linked AI summary

How Order-Sensitive Are LLMs? OrderProbe for Deterministic Structural Reconstruction

Zhaolu Kang, Yingjie He, Kehan Jiang, Leqi Zheng, Jiachen Qian, Qianyuan Zhang, Chunlei Meng, Yujie Feng, Yuan Wang, Stephen Dou, Aming Wu, Pengxiang Zhao, Jiaxin Liu, Guansu Wang, Zeyu Zhang, Lei Wang, Qishi Zhan, Xiaomin He, Meisheng Zhang, Jianyuan Ni, Richeng Xuan

arXiv:2601.08626v4cs.CL

TL;DR

Sentence-level restoration is difficult to evaluate deterministically because scrambled sentences may admit multiple valid reorderings. OrderProbe benchmarks structural reconstruction with fixed four-character expressions and diagnostic measures; across twelve LLMs, zero-shot exact recovery frequently falls below 35%.

  • Problem

    Sentence-level restoration is difficult to evaluate deterministically because scrambled sentences may admit multiple valid reorderings.

    Source: §1 · p. 1

  • Method

    OrderProbe benchmarks structural reconstruction in Chinese, Japanese, and Korean and evaluates models with diagnostic measures beyond recovery accuracy.

    Source: §1 · p. 2

  • Results

    Across twelve models, zero-shot exact reconstruction frequently falls below 35%, and meaning-oriented generation consistently diverges from exact structural recovery.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    The reported gap indicates that structural robustness is not an automatic byproduct of semantic competence.

    Source: §4.2 · p. 6

    The benchmark covers six syntactic categories and fixed four-character expressions in Chinese, Japanese, and Korean, not the full range of linguistic structures.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) excel at semantic understanding, yet their ability to reconstruct internal structure from scrambled inputs remains underexplored. Sentence-level restoration is difficult to evaluate automatically because scrambled sentences often admit multiple valid reorderings. We introduce OrderProbe, a deterministic benchmark for structural reconstruction using fixed four-character expressions in Chinese, Japanese, and Korean, which have a unique canonical order and thus support exact-match scoring. We further propose a diagnostic framework that evaluates models beyond recovery accuracy, including Semantic Accuracy, Logical Validity, Structural Consistency, Robustness, and Information Density. Experiments on twelve widely used LLMs show that structural reconstruction remains difficult even for frontier systems: zero-shot recovery frequently falls below 35%. We also observe a consistent gap between meaning-oriented generation and exact structural reconstruction, suggesting that structural robustness is not an automatic byproduct of semantic competence.

1 Introduction

OrderProbe makes structural reconstruction deterministically evaluable with fixed four-character expressions in Chinese, Japanese, and Korean. It also diagnoses model behavior beyond exact recovery accuracy.

1 Introduction

  • Sentence restoration is ambiguous because scrambled inputs can admit multiple valid reorderings, while existing robustness benchmarks mainly test surface noise or instruction perturbations.These limitations blur structural competence with general fluency and leave uniquely constrained internal-order reconstruction untested.Source: §1 · p. 1
  • OrderProbe uses fixed four-character expressions with a single conventional order to enable exact-match evaluation of canonical reconstruction under controlled permutations.Source: §1 · p. 2
  • LLMs may produce meaning-aligned explanations while failing exact reconstruction, exposing a gap between semantic access and structural recovery.Source: §1 · p. 2
  • The benchmark contains 3,543 curated Chinese, Japanese, and Korean expressions across six syntactic categories and multiple script types.The dataset uses multi-source collection, expert filtering, and hybrid semantic annotation.Source: §1 · p. 2
  • Its diagnostic framework measures Semantic Accuracy, Logical Validity, Structural Consistency, Robustness, and Information Density to analyze failures beyond recovery accuracy.Source: §1 · p. 2
  • Zero-shot recovery often falls below 35% across twelve widely used LLMs, showing that exact reconstruction remains difficult even for strong systems.Source: §1 · p. 2

2 OrderProbe

OrderProbe turns multilingual four-character expressions into a deterministic reconstruction benchmark through collection, expert filtering, semantic-reference construction, and permutation-based perturbation.

2.1 Stage I: Multi-source Collection

  • About 4,000 candidates across ZH-CN, ZH-TW, JA, and KO were gathered from public resources, then screening retained 3,543 canonical benchmark roots.The roots were formed into a linguistically reliable core set for reconstruction.Source: §2.1 · p. 2, §2.2 · p. 3

2.2 Stage II: Expert Filtering

  • Five native-speaker annotators per language filtered non-standard, ambiguous, slang, or non-unique expressions; majority voting and consensus review achieved Fleiss’ κ = 0.835.Items were also removed when permutations could produce another conventional lexicalized entry.Source: §2.2 · p. 3

2.3 Stage III: Semantic Reference Construction

  • Dictionary definitions anchor each semantic reference, expanded with annotator-verified LLM paraphrases following Zheng et al. (2019), Fu et al. (2025), and Li et al. (2024).This hybrid reference set supports semantic and logical evaluation without cross-language translation.Source: §2.3 · p. 3

2.4 Stage IV: Permutation-based Perturbation Generation

  • 23 non-identity permutations per expression preserve the same four constituents while disrupting canonical order and retaining a unique gold target.The permutations are controlled reconstruction stimuli, not intended natural variants.Source: §2.4 · p. 3

2.5 Dataset Statistics

  • The final dataset covers Simplified Chinese, Traditional Chinese, Japanese, and Korean across six syntactic categories.Korean Hangul serves as a phonogrammatic control: syllable-block scrambling preserves syllable composition but offers weaker standalone semantic anchors than logographic scripts.Source: §2.5 · p. 3

3 Evaluation Framework

OrderProbe separates exact structural reconstruction from meaning-oriented generation with a six-metric diagnostic framework. It evaluates outputs across every non-identity permutation to characterize model sensitivity and invariance.

3 Evaluation Framework

  • For 3543 expressions, evaluation covers all 23 non-identity permutations, producing 81,489 perturbed inputs and a six-dimensional diagnostic signature per model.Outputs are scored for exact reconstruction, semantic alignment, and logical entailment, then aggregated across permutations to quantify sensitivity and invariance.Source: §3 · p. 4

3.1 Primary Metric: Recovery Rate

  • Recovery Rate is the global indicator of structural integrity, enabled by four-character expressions’ unique canonical forms for deterministic evaluation.Unlike sentence restoration, which can admit multiple valid reorderings, these expressions support exact canonical-order evaluation.Source: §3.1 · p. 4

3.2 Diagnostic Metrics (Beyond Recovery)

  • Semantic Accuracy and Logical Validity compare explanations with validated meanings, while Structural Consistency measures semantic-score dispersion across permutations.Semantic scoring combines cross-encoder relevance, multilingual embedding similarity, and lexical safeguards; logical scoring uses bidirectional multilingual-NLI entailment and the best reference match. Structural Consistency penalizes both average and worst-case deviations from the best-performing permutation.Source: §3.2 · p. 4
  • Spearman correlations of 0.842–0.931 across controlled diagnostic-pipeline variants indicate that model rankings remained stable.Robustness combines permutation sensitivity and syntactic-category variance via harmonic mean; Information Density uses a brevity penalty to reward informative explanations per token.Source: §3.2 · p. 5

4 Experiments

Across OrderProbe’s experiments, exact canonical reconstruction remains difficult despite models’ ability to produce plausible semantic explanations. Results vary with permutation structure, script typology, syntactic pattern, prompting, and lexical familiarity.

4.1 Experimental Setup

  • The benchmark evaluates 12 LLMs on 3,543 four-character expressions across four script settings, requiring exact-match recovery and a one-line explanation.The experiments use language-matched prompts and compare zero-shot, chain-of-thought, and three-shot settings; a reconstruction-only schema is also evaluated in the appendix.Source: §4.1 · pp. 5–6

4.3 Permutation Sensitivity and Prompting Effects

  • Across 23 permutation configurations, recovery depends on anchor displacement; few-shot prompting improves performance more uniformly than CoT, whose gains can be unstable and cause format drift.Permutations preserving informative characters near canonical neighborhoods yield higher recovery, while dispersed anchors degrade it.Source: Figure 4 · p. 6, §4.3 · pp. 6–7

4.4 Impact of Script Typology

  • Chinese and Japanese recovery exceeds Korean: logographic characters provide local semantic anchors, whereas Hangul scrambling weakens them and can yield high consistency despite low recovery.The results caution that stable explanations under ambiguity do not necessarily indicate successful reconstruction.Source: Table 4 · p. 6, §4.4 · p. 7

4.2 Overall Results and Analysis

  • Zero-shot recovery frequently falls below 35%, while models can produce fluent, semantically plausible explanations without reconstructing canonical order.CoT often improves recovery, but gains vary by model and can reverse under unstable decoding; human recovery is 46.5–88.6% across scripts, while the canonical-input control reaches 100% Recovery by construction.Source: §4.2 · p. 6

4.5 Analysis of Syntactic Patterns

  • Parallel and coordinate patterns are easier because redundancy and symmetry reduce the ordering search space, while subject–predicate and verb–object patterns expose distinct ordering failures.Models may recover a subject–predicate proposition but fail to linearize it canonically; verb–object dependencies are difficult when permutation noise disrupts directionality.Source: Figure 5 · p. 7, §4.5 · p. 7

4.6 Analysis of In-Context Learning

  • Three-shot perturbations disrupt diagnostic components unevenly, and relatively strong recovery can coexist with clear drops on particular diagnostics.Low-recovery models typically decline across multiple dimensions, so recovery alone does not capture the effects of perturbation.Source: Figure 6 · p. 7, §4.6 · p. 7

4.7 Case Study: Literal Interpretation under Structural Perturbation

  • In the Qing Guo Qing Cheng case, some models retrieve the idiomatic metaphor, while others give a plausible but non-canonical literal interpretation.The contrast illustrates why exact reconstruction and diagnostic metrics are both needed.Source: §4.7 · p. 7, Figure 7 · p. 8

4.8 Memorization and Frequency Analysis

  • Recovery improves with idiom frequency, but non-attested synthetic expressions also remain difficult, showing that memorized idiom lookup alone cannot explain reconstruction failures.Frequency contributes to Recovery alongside structural predictors and script/model controls, and its advantage diminishes as semantic signals are removed.Source: §4.8 · p. 8

5 Related Work

Related work studies robustness to perturbed inputs, semantic understanding of idioms and classical language, and positional or structural limitations in Transformers. OrderProbe differs by testing exact canonical-order recovery and complements prior structural findings with evidence that this task remains challenging across Chinese, Japanese, and Korean.

5 Related Work

  • Robustness benchmarks target surface variation, instruction following, or reasoning, while OrderProbe tests exact reconstruction from internally permuted components.Examples include TextFlint (Wang et al., 2021), CheckList (Ribeiro et al., 2020), instruction-following and reasoning evaluations (Li et al., 2024; Lin et al., 2025), and studies of typoglycemia and scrambled text (Yang et al., 2026; Su, 2026; Li et al., 2026c, 2025b,a, 2026b, 2025c, 2026d; Wu et al., 2026; Yu et al., 2026e,d,f,g).Source: §5 · p. 8
  • Idiom and classical-language benchmarks largely assess meaning or use on canonical inputs, unlike OrderProbe’s recovery of conventional four-character-expression order.The cited idiom benchmarks include ChID (Zheng et al., 2019), MMFLD (Lai et al., 2023), Chengyu-Bench (Fu et al., 2025), and Idiom-Trans (Donthi et al., 2025); WenMind likewise emphasizes semantic understanding. OrderProbe evaluates deterministic exact-match recovery (Ye et al., 2026a,c,b; Guo et al., 2026, 2025; Luo et al., 2026, 2024a,b; Li et al., 2026a; Yu et al., 2026b,a,c; Zhang et al., 2026; Zheng et al., 2026a,b,c).Source: §5 · pp. 8–9
  • OrderProbe complements research on positional bias and structural generalization with a controlled multilingual benchmark showing exact reconstruction remains challenging for current LLMs.Prior work reports word-order insensitivity and failures to generalize relational structure under reversal (Sinha et al., 2021; Berglund et al., 2024).Source: §5 · p. 9

6 Conclusion

OrderProbe is a deterministic benchmark for structural reconstruction in Chinese, Japanese, and Korean four-character expressions. Experiments on twelve LLMs show that zero-shot exact recovery often falls below 35%, with results also varying by script typology.

6 Conclusion

  • OrderProbe is a deterministic benchmark for structural reconstruction in Chinese, Japanese, and Korean four-character expressions.The authors release the benchmark and diagnostic framework to support future research on structural robustness in LLMs.Source: §6 · p. 9
  • Zero-shot exact recovery often falls below 35% across experiments on twelve LLMs, exposing a gap between semantic access and exact structural reconstruction.The conclusion reports stronger local anchors in logographic scripts than in Hangul, which serves as a phonogrammatic control with weaker local anchoring.Source: §6 · p. 9
  • Logographic scripts offer stronger local anchors, while Hangul provides a phonogrammatic control with weaker local anchoring.Source: §6 · p. 9
  • The benchmark is intended to complement broader evaluations of order sensitivity.Source: §6 · p. 9

Limitations

OrderProbe’s controlled lexical setting limits the linguistic structures and language systems it covers. Some tested expressions may also be familiar from pretraining, leaving memorization and generalization to novel controls incompletely resolved.

Limitations

  • The benchmark covers six syntactic categories and fixed four-character expressions in Chinese, Japanese, and Korean, not the full range of linguistic structures or alphabetic and morphologically rich systems.The paper identifies broader coverage of those systems as outside its detailed analysis.Source: Limitations · p. 9
  • Some expressions may occur in pretraining corpora, and tests on novel, synthetic, or non-idiomatic controls remain future work despite frequency analysis suggesting memorization alone does not explain results.Thus, reconstruction may partly benefit from memorized lexical knowledge.Source: Limitations · p. 9

Ethics Statement

OrderProbe reports that its data use and construction followed academic-use, consent, provider-policy, and human-verification safeguards, and that the authors foresee no direct societal impacts or misuse risks from release.

Ethics Statement

  • The dataset uses public-domain linguistic resources under academic fair-use principles, and the authors report no copyright restrictions, informed consent, and exclusion of offensive or discriminatory content.The collection followed data usage policies and a rigorous participant protocol.Source: Ethics Statement · p. 9
  • LLMs were used for semantic augmentation in compliance with provider terms, and generated outputs were human-verified for factual accuracy and safety.The authors characterize the benchmark as a diagnostic tool for improving model robustness and foresee no direct societal impacts or misuse risks from release.Source: Ethics Statement · p. 9

A Formal Definitions and Rationale of Evaluation Metrics · A.5 Robustness Metrics · A.8 Robustness of Diagnostic Metrics

OrderProbe’s appendix defines independent metrics for exact reconstruction and distinct explanation qualities, then tests their reliability against human judgments and alternative scoring configurations. The diagnostic framework shows stable ranking patterns overall, with cross-language variation in human–automatic agreement.

A Formal Definitions and Rationale of Evaluation Metrics; A.2 Semantic Accuracy; A.6 Logical Validity; A.7 Information Density; A.13 Generation Protocol and Nondeterminism

  • OrderProbe reports each metric independently, not as an aggregate; verification weights are 0.5 for the cross-encoder, 0.3 for embedding-based measures (BERTScore, STS, cosine similarity), and 0.2 for Fβ as a lexical safeguard.Information Density rewards high information content per token and applies a brevity penalty when outputs exceed reference length without proportional semantic gain. When available, temperature is set to 0, with no task-specific tuning or customized decoding strategies.Source: A Formal Definitions and Rationale of… · p. 13, §A.2 · p. 13, §A.7 · p. 15

A.1 Primary Metric: Recovery Rate; A.3 Structural Consistency; A.4 Degradation Under Internal Reordering

  • Exact Recovery requires the canonical expression, while Structural Consistency measures semantic stability across permutations and degradation metrics compare perturbed performance with the canonical-input baseline.Mean Degradation Rate captures average relative loss; Maximum Degradation Amount captures the worst relative loss.Source: §A.1 · p. 13, §A.3 · p. 14, §A.4 · p. 14

A.5 Robustness Metrics

  • Robustness combines sequential stability under internal permutations with structural stability across six syntactic categories, using mean semantic accuracy and a harmonic-mean composite.Sequential Robustness complements an equally weighted combination of mean and maximum degradation; Structural Robustness rewards consistent accuracy across categories.Source: §A.5 · pp. 14–15

A.8 Robustness of Diagnostic Metrics

  • Across ten scoring variants, model rankings remain stable, with Spearman correlations of 0.842–0.931; backbone, aggregation, and weight substitutions preserve the qualitative findings.The alternatives include unidirectional entailment and bidirectional-mean aggregation, compared with the primary bidirectional-max compatibility score.Source: §A.8 · p. 16, Table 5 · p. 17

A.9 Human Audit Protocol

  • Macro-average human–metric Spearman correlations were 0.740 for Semantic Accuracy, 0.781 for Logical Validity, and 0.703 for Information Density; corresponding MAEs were 0.119, 0.102, and 0.129.The audit sampled 290 ZH-CN, 430 ZH-TW, 170 JA, and 110 KO instances, each independently rated by five language-proficient annotators.Source: §A.9 · p. 16, Table 6 · p. 17

A.10 Alignment Between Human Evaluation and Automated Metrics

  • Human and automated evaluations show consistent directional alignment, and the relative ordering of evaluation dimensions remains stable across languages despite differences in absolute scores.The reported alignment concerns logical coherence, robustness, and internal consistency as well as the broader multidimensional evaluation framework.Source: §A.10 · pp. 16–17, Table 7 · p. 17

A.11 Cross-Lingual Metric Reliability

  • Human–automatic metric correlations are stronger in Chinese settings than in Japanese and Korean, so automated scores are relative indicators rather than absolute judgments, especially in lower-agreement scripts.The appendix suggests weaker lexical overlap, segmentation differences, and reduced compatibility between multilingual semantic models and language-specific idiomatic meanings as possible explanations.Source: §A.11 · p. 17

A.12 Controlled Sanity Check of Diagnostic Metrics

  • Controlled tests show that Semantic Accuracy and Logical Validity stay high for structurally incorrect reconstructions with correct explanations, while verbosity substantially lowers Information Density.Contradictory and unrelated outputs receive substantially lower semantic and logical scores; verbosity has only a small effect on those scores.Source: §A.12 · pp. 17–18, Table 8 · p. 18

B Appendix: Details of Semantic and Logical Evaluation … C.9 Synthetic Control Construction

The appendix details semantic and logical evaluation and presents validation analyses for OrderProbe. Its probes show that recovery remains sensitive to internal order even when explanation quality, expression identity, frequency, and model are controlled.

B Appendix: Details of Semantic and Logical Evaluation; B.1 Semantic Explanation Scoring

  • Semantic accuracy combines interaction-based matching, representation similarity, and content-word Fβ using fixed weights shared across languages.The cross-encoder and original-language sentence embeddings are used alongside an optional BERTScore F1 signal; lexical overlap is auxiliary.Source: §B.1 · pp. 18–19

B.2 Logical Consistency via Natural Language Inference

  • Logical consistency uses bidirectional multilingual NLI, taking the maximum entailment across directions and references with a contradiction adjustment when entailment exceeds a threshold.Scores remain continuous, are not post-hoc calibrated, and are used for relative comparison and averaging.Source: §B.2 · p. 19

B.3 Reference Construction and Evaluation Scope; C.1 Korean Processing and Perturbation Unit; C.2 Human Reconstruction Baseline

  • Reference meanings combine dictionary definitions with paraphrases, while Korean inputs permute intact Hangul syllable blocks and are treated as a phonogrammatic control with weaker standalone semantic cues.The human baseline tests exact recovery on the same scrambled inputs with proficient speakers, disallowing external lookup and recording recovery, completion time, and confidence.Source: §B.3 · p. 19, §C.1 · p. 19, §C.2 · p. 20

C.3 Uncertainty Estimation and Parsing Reliability; C.4 Recovery Breakdowns Across Prompting and Script Settings

  • Recovery uncertainty is estimated with 10,000 stratified cluster-bootstrap replicates resampling canonical expressions; unparseable outputs count as incorrect, with parser failures reported separately.Aggregate recovery values are macro-averages over 12 model-level scores, with confidence intervals resampling canonical roots within script settings and retaining their permutations.Source: §C.3 · p. 20, Table 13 · p. 21, §C.4 · p. 21

C.5 Frequency-Stratified Recovery Pilot; C.6 Expanded Frequency-Stratified Analysis; C Additional Validation and Controls

  • Frequency analyses compare exact recovery across within-language frequency bins and progressively reduce input and prompt semantic support to distinguish lexical familiarity from structural effects.The expanded analysis estimates frequency associations with logistic regression while controlling for script setting and syntactic category.Source: §C.5 · p. 21, §C.6 · p. 21

C.7 Incremental Effects of Frequency and Structure

  • Frequency explains part of recovery, but structural configuration adds explanatory power beyond lexical familiarity; the full model also captures script and model variation.The analysis uses instance-level logistic regressions with exact recovery as the outcome and structural predictors including retained adjacency, positional displacement, and syntactic category.Source: §C.7 · p. 22, Table 21 · p. 22

C.8 Canonical-Explanation Probe

  • Recovery rises with retained local adjacency within fixed expression–model pairs, after controlling for explanation quality, expression identity, frequency, and model.Table 23 reports recovery by retained-local-adjacency level.Source: §C.8 · p. 23, Table 23 · p. 23

C.9 Synthetic Control Construction

  • Synthetic controls are manually checked, non-attested four-character expressions with intended compositional readings, evaluated using the benchmark’s permutation protocol, prompts, and diagnostics.Table 24 reports Recovery Rate and Semantic Accuracy for idiomatic and synthetic expressions, but the supplied passage gives no values.Source: §C.9 · p. 23, Table 24 · p. 23

C.10 Reconstruction-Only Output Ablation … E Literal Interpretation and Idiomatic Retrieval

Reconstruction-only prompting improves exact recovery over the standard output schema, but failures remain; frequency and idiomatic examples show that lexical familiarity and semantic plausibility do not guarantee canonical reconstruction.

C.10 Reconstruction-Only Output Ablation

  • 43.5% recovery with reconstruction-only prompting exceeded the standard-schema zero-shot macro-average of 20.98%, though 56.5% of sampled inputs remained unrecovered.The ablation used 200 stratified benchmark inputs and required only the recovered canonical expression.Source: §C.10 · p. 23

C.11 Error-Category Analysis

  • The error-category analysis sampled 1,440 zero-shot outputs across model, script, and retained-adjacency conditions for mutually exclusive expert labeling.Three language experts assigned categories, with Chinese experts covering both Chinese scripts; category rates used the corresponding sampling fractions.Source: §C.11 · p. 23, Table 26 · p. 24

C.12 Frequency Under Semantic Signal Disruption

  • Recovery declined monotonically from high- to mid- to low-frequency expressions, yet remained non-trivial in the low-frequency bin.This pilot zero-shot analysis averaged exact canonical reconstruction rates across GPT-4o and Qwen-3-8B; bin-wise values are not supplied here.Source: §C.12 · pp. 23–24

D Prompt Templates for Idiom Meaning Inference

  • Baseline idiom-meaning prompts used a fixed two-message scheme with a one-line Chinese output constraint, without task-specific examples or additional guidance.For multilingual settings, prompt content was translated or equivalently reformulated while preserving functional roles and formatting; only prompt text varied across conditions.Source: D Prompt Templates for Idiom Meaning… · p. 24
  • The chain-of-thought condition used a fixed two-line template while keeping decoding settings identical.The template is identified as Table 31.Source: D Prompt Templates for Idiom Meaning… · p. 24

E Literal Interpretation and Idiomatic Retrieval

  • Some models gave fluent, non-canonical literal interpretations of Qing Guo Qing Cheng despite strong lexical cues, while stronger models retrieved its idiomatic metaphor.The failure case illustrates that semantic plausibility and structural reconstruction can diverge.Source: E Literal Interpretation and Idiomatic… · p. 24

F Qualitative Analysis and Case Studies

The section presents qualitative visual analyses of model behavior across experimental settings, showing zero-shot failure modes and comparing CoT and few-shot strategies across four language settings.

Baseline behaviors

  • Figures 8 and 9 show baseline zero-shot behaviors, highlighting hallucination and structural blindness as common failure modes.The visuals include system prompts, user queries, and model outputs.Source: F Qualitative Analysis and Case Studies · p. 24

CoT and few-shot strategies

  • Figures 10–13 compare Chain-of-Thought and few-shot strategies for ZH-CN, ZH-TW, JP, and KO.The visualizations display prompts, queries, and model outputs under the evaluation pipeline’s strict constraints.Source: F Qualitative Analysis and Case Studies · p. 24

G Reference Performance on Canonical Inputs · G.1 English Readings of Prompt Templates

Canonical inputs yielded perfect Recovery because their order was already visible, providing a reference control rather than a reconstruction test. The accompanying English readings document prompt formats and constraints; original-language prompts remained the evaluation inputs.

G Reference Performance on Canonical Inputs

  • 100% Recovery on canonical inputs contrasts with permutation recovery frequently below 35%, isolating the structural-reconstruction difficulty.Table 27 reports the reference evaluation under unperturbed inputs; the control is provided for context, not comparative or statistical claims.Source: G Reference Performance on Canonical… · pp. 24–25, Table 27 · p. 25
  • Non-perfect diagnostic scores evaluate generated explanations’ quality and logical compatibility, not whether models copied the visible canonical expression.Thus, diagnostic values need not equal 1 even when Recovery is perfect.Source: G Reference Performance on Canonical… · pp. 24–25
  • The visible-order control separates difficulty recovering expression order from difficulty generating an explanation when canonical input is intact.Canonical expressions are already in order, so this control is not a non-trivial reconstruction test.Source: G Reference Performance on Canonical… · p. 24

G.1 English Readings of Prompt Templates

  • English readings accompany the multilingual prompts for accessibility while preserving operational constraints; original-language strings remain the evaluation prompts.The constraints include language, output field, and line count.Source: §G.1 · p. 25
  • The baseline English template requests a target-language explanation in exactly one line formatted as “Meaning: [explanation].”Its user prompt asks for a complete explanation without filler.Source: §G.1 · p. 25
  • Few-shot prompts request a two-line analysis, while the CoT template requires stepwise analysis and exactly two specified output lines.The CoT prompt directs models to decompose character meanings, reconstruct likely order, and derive overall meaning.Source: §G.1 · p. 25
Loading 2601.08626v4…