ChLogic: Evaluating Robustness of Logical Reasoning in Chinese Expressions

1 Introduction

• Existing logical-reasoning benchmarks are predominantly English, template-like, or explicitly signaled by connectives, so their high scores may not transfer to less explicit Chinese expressions.The introduction identifies elliptical, rhetorical, idiomatic, multi-sense, colloquial, and pragmatically indirect Chinese as a key stress test.Source: §1 · p. 1
• Chinese expressions challenge structure recovery through flexible negation, condition markers, variable quantifier scope, indirect language, idioms, homophones, and omitted subjects.These forms can obscure whether a conclusion is entailed, contradicted, or merely possible.Source: §1 · p. 1
• CHLOGIC aligns English and Chinese problems built from formal templates, holding logical labels fixed while varying language and surface realization.This controlled design extends monolingual robustness testing to equivalent problems across languages and provides interpretable diagnostics.Source: §1 · p. 1
• GLM-5.1 accuracy drops from 98.30% on English to 78.89% on rhetorical Chinese in the General set, and from 84.70% to 52.30% in the Difficult set.These examples illustrate the persistent English–Chinese performance gap reported across the evaluated models.Source: §1 · p. 2
• Back-translation often improves General-set performance, but has mixed effects on the Difficult and Chinese-only sets.The results suggest Chinese surface realization contributes to some errors, while translation can also simplify, alter, or obscure judgment-relevant information.Source: §1 · p. 2

2 Benchmark Construction

• Each aligned item pairs one English reference with five Chinese realizations, while the Chinese-only set covers 15 phenomenon types with 100 examples each.The aligned sets use YES/NO judgments; the Chinese-only set adds UNKNOWN for genuine underdetermination.Source: §2 · pp. 2–3

2.1 Logical Templates

• The benchmark combines 60 General Propositions across nine template families, 40 Difficult Problems, and 15 Chinese-specific phenomenon groups.The templates span propositional and predicate logic, multi-step rule chains, and relational logic; the difficult problems include quantifier alternation, uniqueness, equality, and countermodels. Chinese-specific groups target surface-to-logical-form normalization, including negation scope and rhetorical questions.Source: §2.1 · pp. 3–4

2.2 LLM-Assisted Generation and Quality Control

• Templates determine logical structure and gold labels, while LLMs generate and verify realizations that human reviewers accept, revise, regenerate, or discard.Quality checks preserve premises, target questions, and labels, remove ambiguity, and ensure Chinese-specific answers do not require world knowledge beyond the text.Source: §2.2 · p. 4

3 Experimental Setup

• The General and Difficult aligned sets use binary YES/NO judgments, while the Chinese-only set uses ternary YES/NO/UNKNOWN judgments.Source: §3 · p. 4
• Qwen3, Ministral, and GLM-5.1 models are tested zero-shot with a shared prompt and deterministic decoding; unmappable outputs count as incorrect.Source: §3 · p. 4
• Accuracy is reported by data set, surface-realization condition, template family, and phenomenon type, with back-translation probes testing whether errors arise from inference or recovering Chinese logical forms.Source: §3 · p. 4

Scaling and model profiles

• 95.06% versus 79.61% average accuracy across five Chinese variants for Qwen3-32B and Qwen3-8B shows scaling helps but does not erase Chinese robustness gaps.Ministral-8B remains competitive on several difficult Chinese variants despite trailing stronger models on general English, suggesting English-standard accuracy and Chinese-expression robustness are related but distinct.Source: §4 · p. 4

4 Results

• 99.07% (Qwen3-32B) on general English contrasts with GLM-5.1’s 78.89% rhetorical and 83.17% perturbed Chinese; Qwen3-32B falls from 96.05% difficult English to 69.35% rhetorical Chinese.Aligned variants share logical structures and gold labels, so the drops are consistent with difficulty recovering logical form from Chinese surface realizations, though they do not isolate a cause.Source: §4 · p. 4, Table 2 · p. 5

Back-translation diagnostics

• 90.53% to 99.10% for Qwen3-8B, 95.73% to 99.30% for Qwen3-32B, and 92.60% to 97.73% for GLM-5.1 after general-set back-translation; difficult-set effects are mixed.Qwen3-8B and Qwen3-14B improve on the Difficult set, while Qwen3-32B and GLM-5.1 decline. Back-translation often helps Chinese-specific ellipsis, temporal-order versus causality, fuzzy quantifiers, and not-all versus all-not, but can hurt irony, homophone shifts, and some comparative structures.Source: Table 3 · p. 5, §4 · p. 5

Template-level difficulty and fallacies

• 94.67% English equivalence accuracy for Ministral-8B falls to 23.56% on standard Chinese and 8.00% on rhetorical Chinese; Qwen3-8B invalid inference falls from 99.75% English to 41.50% rhetorical Chinese.Errors cluster around global scope, truth-condition comparison, and tempting invalid conclusions, including affirming the consequent, denying the antecedent, illicit conversion, and existential fallacy; making compact English fallacy patterns more explicit can improve accuracy.Source: §4 · p. 5

• Strong English logical performance does not guarantee robust Chinese understanding; scaling helps, but a sizable English–Chinese gap persists and model families vary across languages and surface realizations.Source: §5 · p. 6
• CHLOGIC probes surface-to-logical-form normalization before inference, exposing errors involving conditional particles, negation scope, rhetorical questions, commonsense plausibility, and invalid converses.For all five models, back-translated English accuracy on the General aligned set is much higher than standard Chinese accuracy and close to English-reference accuracy.Source: §5 · p. 6
• Template-family and phenomenon-level results expose failure modes that overall accuracy can hide, including weak Chinese robustness and misleadingly high scores from answer bias.Examples include Qwen3-14B's high English accuracy but limited Chinese robustness on the Difficult aligned set, and Qwen3-0.6B's label-distribution-matched answer bias.Source: §5 · p. 6

6 Related Work

• Synthetic reasoning benchmarks and reading-comprehension tests evaluate logical reasoning but do not directly test judgment preservation under complex Chinese realization (Weston et al., 2016; Clark et al., 2020; Tafjord et al., 2021; Han et al., 2022; Liu et al., 2020; Yu et al., 2020).Categorical-syllogism studies also examine template coverage and quantifier-related failures (Zong and Lin, 2024).Source: §6 · p. 6
• Chinese NLU suites and multilingual benchmarks cover broad language understanding, knowledge, and reasoning, but do not isolate shared latent templates across controlled English and Chinese realizations (Xu et al., 2020; Huang et al., 2023; Conneau et al., 2018; Shi et al., 2023; Fabbri et al., 2025).The cited resources include CLUE, C-Eval, XNLI, MGSM, SeaEval, and MultiNRC.Source: §6 · p. 6
• Robustness studies show that high average accuracy can mask brittle reasoning under contrast sets, behavioral tests, symbolic-template variation, and typographical perturbations (Gardner et al., 2020; Ribeiro et al., 2020; Mirzadeh et al., 2024; Gan et al., 2024).CHLOGIC specializes this principle by aligning expression families to formal templates and testing whether judgments remain correct under Chinese realization changes.Source: §6 · p. 6

7 Conclusion

• CHLOGIC is a controlled English–Chinese benchmark for evaluating logical understanding under complex Chinese expressions.The benchmark examines whether logical reasoning remains robust across Chinese surface realizations.Source: §7 · p. 6
• High English logical accuracy does not guarantee robust Chinese understanding, and back-translation’s benefits vary across benchmark sets.Back-translation often improves the General aligned set but is unstable on the Difficult aligned set and mixed for Chinese-specific phenomena.Source: §7 · p. 6
• Scaling improves many general Chinese variants, but rhetorical, perturbed, quantifier-sensitive, and equivalence-law expressions remain difficult.These findings qualify the broader performance improvements across Chinese variants.Source: §7 · p. 6
• The results support a cautious interpretation: Chinese surface realization contributes to some failures, while translation artifacts and model-specific decision behavior also affect outcomes.Source: §7 · p. 6

Limitations

• ChLogic’s template-driven aligned items may not cover all forms of discourse-level reasoning.The paper characterizes ChLogic as a diagnostic benchmark rather than a complete measurement of Chinese reasoning ability.Source: Limitations · p. 6
• The evaluation covers only seven contemporary systems, and results may change as model families are updated.Source: Limitations · p. 6
• Generated Chinese variants may still contain artifacts despite labels being specified before generation and items being manually checked.Source: Limitations · pp. 6–7
• Back-translation may remove Chinese cues or introduce ambiguity on the English side because it is a model-mediated transformation.Source: Limitations · p. 7

A.1 Logical Templates and General Propositions

• The 60-proposition inventory combines common benchmark-tested logical forms with additions targeting Chinese logical robustness and surface variation.Additions include exclusive disjunction, unique existence, quantifier movement, and branching inference; the inventory spans English and five Chinese realizations.Source: §A.1 · p. 10

A.2 Statements Specific to Chinese Context

• Fifteen Chinese-specific phenomenon groups test whether models recover logical structure obscured or shifted by Chinese surface forms.Targets include implication-direction ambiguity, negation scope, vague quantifiers, omitted arguments, rhetorical questions, and pragmatic or orthographic effects.Source: §A.2 · p. 11

A.3 LLM-Assisted Generation and Quality Control

• LLMs generate surface realizations from templates with labels fixed in advance, while quality control checks semantic alignment, label preservation, and ambiguity.Human inspection determines final acceptance, revision, regeneration, or rejection; Chinese-only items are also checked for the target phenomenon and dependence on private world knowledge.Source: §A.3 · p. 11

B Construction Audit and Responsibility Matrix

• DeepSeek-V3 handled generation, revision, and back-translation, Yi-1.5-34B-Chat independently verified items, and human reviewers made final acceptance decisions.The evaluated Qwen3, Ministral, and GLM models were excluded from construction; reviewers checked fluency, alignment, premise and question preservation, label consistency, and ambiguity.Source: B Construction Audit and… · p. 12

C Instance Expansion, Filtering, and Release Structure

• The release contains 3,000 General and 2,000 Difficult latent aligned items, each with one English reference and five Chinese realizations, plus 1,500 Chinese-only items.The six aligned realizations are parallel evaluation conditions, not independent logical items; the benchmark is test-only and has no train, development, or test partitions.Source: C Instance Expansion, Filtering, and… · pp. 12–13

D Label Inventory and Distribution

• Aligned sets use binary YES/NO labels, while the Chinese-only set adds UNKNOWN for genuine underdetermination; label imbalance can inflate accuracy without reliable reasoning.An almost-always-YES strategy can reach 78.33% on General, and NO accounts for 45.00% on Difficult, motivating interpretation alongside label frequencies and template-family behavior.Source: D Label Inventory and Distribution · p. 13

E Bias-Aware Metrics for the General Aligned Set

• The bias-aware analysis covers Qwen3-0.6B and Ministral-3B, selected for clear answer-distribution biases, across 2,350 YES and 650 NO instances.It reports accuracy, balanced accuracy, Macro-F1, per-label precision, recall and F1, confusion counts, and predicted-label distributions.Source: E Bias-Aware Metrics for the General… · p. 13
• Qwen3-0.6B reaches 78.33% accuracy but only 50.00% balanced accuracy and 0 NO recall, reflecting universal YES predictions rather than reliable discrimination.It predicts YES for every item across all six surface-realization conditions.Source: E Bias-Aware Metrics for the General… · p. 13
• Ministral-3B predicts NO for 89.83% of rhetorical-Chinese items, with 100% NO recall, 12.98% YES recall, and 30.92% Macro-F1.Its predicted-NO share is 54.87% in the English-reference condition, illustrating how high accuracy on NO-heavy template families may not indicate uniformly strong reasoning.Source: E Bias-Aware Metrics for the General… · p. 13
• Balanced accuracy, per-label metrics, and confusion counts complement aggregate accuracy by revealing opposite answer-distribution biases in the two models.Qwen3-0.6B defaults to YES, whereas Ministral-3B shifts toward NO under rhetorical Chinese.Source: E Bias-Aware Metrics for the General… · p. 13

F Back-Translation Audit and Interpretation

• Back-translation is diagnostic, not a neutral removal of Chinese surface information, because it may simplify wording, alter rhetorical force or semantic scope, remove cues, or introduce English ambiguity.DeepSeek-V3 produced the back-translated English condition.Source: F Back-Translation Audit and… · pp. 13–14
• Semantic consistency checks compare sampled Chinese items with back-translated English for premise and question preservation, locked-label preservation, negation scope and condition direction, and rhetorical or pragmatic force.The audit gives particular attention to rhetorical questions, irony, and homophone-based shifts; performance changes are interpreted as diagnostic rather than causal evidence about normalization.Source: F Back-Translation Audit and… · p. 14

G Additional Results and Analysis

• English-standard logical accuracy is generally higher than robust Chinese accuracy, with aligned examples pointing to Chinese surface normalization as a major difficulty.Qwen3-32B scores 99.07% on general English, while GLM-5.1 scores 78.89% on general rhetorical Chinese and 83.17% on general perturbed Chinese.Source: G Additional Results and Analysis · p. 14
• On the Difficult aligned set, Qwen3-32B scores 96.05% on English but 69.35% on rhetorical Chinese, where polarity, pragmatic implication, and implicit negation remain challenging.Its other difficult Chinese scores are 83.10% on standard, 87.80% on natural, 86.15% on colloquial, and 85.20% on perturbed Chinese.Source: G Additional Results and Analysis · p. 14
• Ministral-8B is competitive on several Chinese variants despite lower English accuracy, while Ministral-3B’s unexpectedly stronger Difficult-set scores reflect label bias and answer-distribution artifacts.Ministral-8B leads the reported Difficult-set Chinese results on natural, colloquial, rhetorical, and perturbed variants; Ministral-3B’s Difficult English score is 14.03 points above its General English score, not evidence of an advantage on harder logic.Source: G Additional Results and Analysis · pp. 14–15

G.1 Scaling trend within Qwen3

• Scaling Qwen3 from 8B to 32B raises average accuracy across general Chinese variants from 79.61% to 95.06%, with larger gains on rhetorical and perturbed Chinese than on English.Rhetorical Chinese rises from 68.69% to 93.53% and perturbed Chinese from 77.99% to 97.00%, while English increases from 98.40% to 99.07%.Source: §G.1 · p. 15
• On the Difficult aligned set, Qwen3-32B raises average Chinese accuracy to 82.32%, yet still trails its 96.05% English score on rhetorical Chinese at 69.35%.Difficult Chinese averages are 52.77% for Qwen3-8B, 54.73% for Qwen3-14B, and 82.32% for Qwen3-32B; scaling reduces but does not eliminate surface-normalization errors.Source: §G.1 · p. 15

G.2 Logical-template-level difficulty

• Propositional equivalence is a persistent Chinese bottleneck, while invalid-inference errors often arise when surface wording invites pragmatically tempting but logically invalid conclusions.Qwen3-32B reaches 94.44% on rhetorical Chinese equivalence, but Ministral-8B gets 8.00%; Qwen3-8B’s rhetorical Chinese invalid-inference score is 41.50% versus 99.75% on English.Source: §G.2 · p. 15

G.3 Language-dependent fallacy detection and targeted English augmentation

• Fallacy detection varies with language and explicitness: several models do better on standard Chinese than English for common invalid inference, showing that compact English wording can obscure logical form.Ministral-3B scores 91.50% on English and 100% on all five Chinese variants; Ministral-8B scores 73.25% on English and 94.75% on standard Chinese. The denying-a-conjunct example likewise illustrates that a Chinese formulation may make invalidity easier to detect.Source: §G.3 · pp. 15–16
• In a targeted diagnostic case study, explicitly augmented English sharply improves Qwen3 performance on several fallacy patterns, demonstrating sensitivity to how the logical relation is expressed.For branch reasoning, Qwen3-8B improves from 10% to 100% and Qwen3-14B from 0% to 100%; similar gains appear for existential fallacy, illicit major, and illicit minor. The study is not a comprehensive augmentation benchmark, and its conclusion is that fallacy evaluation must control linguistic realization as well as logical schema.Source: §G.3 · p. 16

G.4 Chinese-to-English back-translation experiments

• BT-En brought all five models closer to English-reference accuracy on the General aligned set, with the largest gains in families where Chinese wording affects logical-form recovery.Accuracy changed from 90.53% to 99.10% for Qwen3-8B, 93.60% to 98.90% for Qwen3-14B, 95.73% to 99.30% for Qwen3-32B, 79.83% to 91.20% for Ministral-8B, and 92.60% to 97.73% for GLM-5.1. This is a diagnostic, not a neutral control: translation may simplify wording or alter cues, so it does not establish one causal explanation.Source: §G.4 · pp. 16–17, Table 13 · p. 17
• On the Difficult aligned set, back-translation helped Qwen3-8B, Qwen3-14B, and Ministral-8B but hurt Qwen3-32B and GLM-5.1.The gains were 22.10, 23.47, and 3.10 points, respectively; Qwen3-32B fell from 83.10% to 79.00%, while GLM-5.1 fell from 81.00% to 62.95%.Source: §G.4 · p. 17, Table 14 · p. 18
• On Chinese-only phenomena, back-translation improved four models’ averages but reduced Qwen3-14B’s, and gains and losses varied by phenomenon.Qwen3-8B rose from 67.20% to 72.67%, Qwen3-32B from 55.27% to 58.13%, Ministral-8B from 53.07% to 74.13%, and GLM-5.1 from 76.27% to 88.73%; Qwen3-14B fell from 70.07% to 66.27%. Improvements included ellipsis and fuzzy quantifiers, while irony, homophone-based concept shifts, and some comparative structures could worsen. The findings support a two-stage interpretation: usable reasoning may emerge once structure is explicit, but Chinese normalization remains fragile.Source: §G.4 · pp. 17–18

G.5 Degenerate and partially degenerate behavior

• Qwen3-0.6B’s near-universal YES output and Ministral-3B’s strong NO bias can inflate aggregate accuracy without indicating reliable reasoning.YES accounts for 78.33% of General aligned labels, while NO accounts for 45.00% of Difficult aligned labels; the corresponding NO share on General is 21.67%. Ministral-3B’s NO bias explains why its Difficult-set aggregate can appear stronger despite poor performance on many YES-heavy families.Source: §G.5 · pp. 18–19

H Forty Difficult Logical Problems

• The Difficult aligned set uses 40 logical skeletons adapted from or inspired by textbook exercises in first-order semantics and related topics.The listed topics include quantifier reasoning, relational properties, uniqueness, equality, and countermodels.Source: H Forty Difficult Logical Problems · p. 19, Table 17 · p. 20

I.2 Representative Chinese-Only Examples

• Representative Chinese-only examples test sufficient versus necessary conditions and distinguish uncertainty from logical negation.The examples contrast “only if” with “as long as” and “not necessarily” with “necessarily not.”Source: §I.2 · p. 21

J Generation, Verification, and Evaluation Prompts; J.1 Surface Realization Prompt; J.3 Verification Prompt

• The prompts specify controlled surface realizations and verification against shared premises, questions, latent relations, and labels.The generation prompt defines standard, natural, colloquial, rhetorical, and perturbed Chinese forms; verification returns PASS only when all six realizations preserve the intended relation and label.Source: J Generation, Verification, and… · p. 21, §J.1 · pp. 21–22, §J.3 · p. 22

K Detailed Experimental Tables

• Table 18 reports overall accuracy for the aligned sets and Chinese-only set, which use binary and three-label judgments, respectively.Its caption defines the five Chinese conditions: standard, natural written, colloquial, rhetorical-question, and perturbed Chinese.Source: Table 18 · p. 23
• Table 19 reports accuracy on 15 statements specific to Chinese context; Qwen3-0.6B is unavailable in its phenomenon-level results.Its aggregate output-degeneration behavior is analyzed separately.Source: Table 19 · p. 23
• The overview tables are followed by one family-level summary table for each model, preserving the six English and Chinese conditions.Qwen3-0.6B is a degeneration diagnostic because it tends to answer YES across almost all inputs; Ministral-3B shows label-asymmetric behavior.Source: K Detailed Experimental Tables · p. 23

K.1 Logical-template-family summaries by model

• Seven tables retain full six-condition family-level summaries for each evaluated model.Source: §K.1 · p. 23

K.2 Fine-grained results grouped by logical-template family

• Fine-grained results are regrouped by logical-template family, with each table comparing all seven models.Tables split the Qwen3 models from the Ministral and GLM models; each cell lists English and five Chinese condition accuracies in a fixed order.Source: §K.2 · p. 23