PaperlayerView exact revision on arXiv

Source-linked AI summary

ChLogic: Evaluating Robustness of Logical Reasoning in Chinese Expressions

Peixian Zhou, Yuxu Chen, Chaorui Zhang, Wei Han, Bo Bai, Xueyan Niu

arXiv:2606.17905v1cs.CL

TL;DR

Whether logical reasoning remains robust when English benchmark structures are expressed through diverse Chinese forms is unclear. ChLogic addresses this with an aligned benchmark of equivalent English and Chinese realizations, finding a persistent English–Chinese performance gap and mixed effects from back-translation.

  • Problem

    English-dominated, template-like benchmarks provide limited evidence about recovering equivalent logical structures from elliptical, idiomatic, rhetorical, or pragmatically indirect Chinese expressions.

  • Method

    ChLogic uses formal templates to align one English reference with five Chinese variants, preserving logical structure while increasing linguistic and pragmatic complexity.

  • Results

    98.30% English accuracy fell to 78.89% on rhetorical Chinese in GLM-5.1’s General aligned set, while experiments showed a persistent English–Chinese gap and mixed back-translation effects.

  • Takeaways & Limitations

    ChLogic is a stress test for multilingual logical-reasoning robustness, with outcomes shaped by Chinese surface realization, translation artifacts, and model-specific behavior.

  • Takeaways & Limitations

    ChLogic is template-driven, covers only seven contemporary systems, and may contain artifacts in generated Chinese variants and back-translation.

Abstract

from arXiv · show

Large language models perform increasingly well on standardized logical reasoning benchmarks, but whether this ability remains robust beyond English is unclear. We introduce ChLogic, an English--Chinese aligned benchmark that tests whether models preserve logical reasoning performance when the same latent logical structure is expressed in English and diverse Chinese surface realizations. Built from formal logical templates, the benchmark contains three data sets: (i) the General aligned set, derived from 60 General Propositions across nine template families; (ii) the Difficult aligned set, derived from 40 Difficult Problems; and (iii) the Chinese-only set, covering 15 language-specific phenomenon types. Each aligned item pairs one English reference expression with five Chinese realizations. Experiments on Qwen3, Ministral, and GLM models reveal a persistent English--Chinese performance gap. Back-translation from standard Chinese into English often improves performance on the General aligned set, but produces mixed effects on the Difficult aligned set, where Qwen3-32B and GLM-5.1 perform worse after translation. These results indicate that Chinese surface realization, translation artifacts, and model-specific behavior jointly affect multilingual logical reasoning. Overall, ChLogic provides a useful stress test for the robustness of multilingual reasoning.

1 College of Mathematics, Sichuan University, China

The section lists an affiliation with the College of Mathematics at Sichuan University in China and identifies an additional affiliation with Huawei Technologies’ Theory Lab.

  • The College of Mathematics is affiliated with Sichuan University, China.
  • An additional affiliation is with Theory Lab.
  • Theory Lab is part of 2012 Labs at Huawei Technologies Co., Ltd.

1 Introduction

CHLOGIC evaluates whether logical reasoning remains consistent when equivalent formal structures are expressed through diverse English and Chinese surface forms. Results show a persistent English–Chinese performance gap and mixed effects of translating Chinese expressions back into English.

  • Motivation: Existing logical reasoning benchmarks are predominantly English, template-like, or explicitly signaled by logical connectives.High performance under these conditions does not establish robust recovery of logical structure from less explicit Chinese expressions.
  • Motivation: Chinese stresses structure recovery because logical relations vary across syntax, pragmatics, negation, conditionals, quantifier scope, and rhetorical or idiomatic forms.Markers such as 只要, 只有, 除非, and 否则 can contribute to confusion between necessary and sufficient conditions.
  • Benchmark: CHLOGIC is an English–Chinese aligned benchmark built from formal logical templates to test consistent judgments across equivalent problems expressed differently.Its alignment provides a controlled framework extending monolingual robustness testing from canonical and perturbed inputs to cross-lingual realizations.
  • Results: 98.30% versus 78.89% accuracy shows GLM-5.1’s General aligned performance falling from English questions to rhetorical Chinese variants.In the Difficult aligned set, GLM-5.1 likewise falls from 84.70% in English to 52.30% on rhetorical Chinese.
  • Results: Back-translation often improves the General aligned set but has mixed effects on the Difficult aligned set and Chinese-only set.The findings implicate Chinese surface realization in some errors while indicating that translation may simplify, alter, or obscure judgment-relevant information.

2 Benchmark Construction

CHLOGIC uses a template-first construction workflow to express identical logical structures in one English reference and five increasingly complex Chinese realizations. It comprises aligned General and Difficult subsets plus a Chinese-only set targeting language-specific phenomena, with semantic alignment and labels controlled through verification and human review.

  • Construction workflow: Each item is generated from a logical template and expressed in one English reference form plus five Chinese variants that preserve the same logical structure.The variants are standard, written, colloquial, rhetorical-question, and perturbed Chinese, designed to increase linguistic and pragmatic complexity.
  • Benchmark subsets: The benchmark contains a General aligned set with 60 propositions across nine template families, a Difficult aligned set with 40 problems, and a Chinese-only set with 15 phenomenon types.The Chinese-only set contains 100 examples per phenomenon type, while both aligned sets pair English and Chinese expressions.
  • Logical coverage: General Propositions cover propositional, predicate, multi-step rule-chain, and relational logic, including both valid reasoning and plausible invalid inference patterns.The nine families test connectives, equivalence, inference rules, quantifiers, named-entity reasoning, transitivity, symmetry, and antisymmetry.
  • Logical coverage: Difficult Problems comprise 40 mathematical-logic problems involving quantifier alternation, relational properties, uniqueness, equality, and countermodel construction.Their natural-language realizations use the benchmark’s common judgment format, while appendix labels follow standard first-order semantics with a non-empty domain.
  • Chinese-specific coverage: The Chinese-only set contains 15 groups targeting phenomena such as partial negation, omitted arguments, vague quantifiers, rhetorical questions, ambiguity, irony, idioms, and homophone shifts.These phenomena capture logical relations expressed through Chinese markers, word order, discourse particles, and pragmatic force, including distinctions between 只要A 就B and 只有A 才B.
  • Quality control: Templates determine the logical structure and gold labels, while realizations undergo semantic-alignment checks and human acceptance decisions to preserve premises, questions, and intended labels.LLMs assist generation and verification, but do not assign labels or make final acceptance decisions; rule-based checks identify malformed outputs.

3 Experimental Setup

The experiments evaluate binary or ternary logical judgments across ChLogic subsets using zero-shot Qwen3, Ministral, and GLM-5.1 models, with accuracy reported across conditions and targeted back-translation probes.

  • The General and Difficult aligned sets use binary YES/NO judgments, while the Chinese-only set uses ternary YES/NO/UNKNOWN judgments.
  • Qwen3, Ministral, and GLM-5.1 models are evaluated zero-shot with identical prompts and deterministic decoding.Outputs are normalized to each subset’s valid label inventory, and unmappable outputs count as incorrect.
  • Accuracy is reported by data set, surface-realization condition, logical-template family, and phenomenon type.
  • Chinese-to-English back-translation probes test whether errors primarily reflect formal inference or recovering intended logical forms from Chinese surface expressions.Standard Chinese fields are translated into English and evaluated under the same protocol.

4 Results

Results show a persistent English–Chinese accuracy gap, with Chinese surface variation especially damaging performance on difficult and logically demanding forms. Scaling and back-translation can improve robustness, but neither consistently eliminates the gap across models and datasets.

  • Overall accuracy: 99.07% and 98.30% on general English fall to 78.89% and 83.17% for GLM-5.1’s rhetorical and perturbed Chinese variants, while Qwen3-32B drops from 96.05% on difficult English to 69.35% on difficult rhetorical Chinese.These results demonstrate an English–Chinese gap that is amplified on the Difficult aligned set.
  • Scaling and model profiles: 95.06% versus 79.61%: Qwen3 average accuracy across five Chinese variants rises from Qwen3-8B to Qwen3-32B, but difficult rhetorical Chinese remains 26.70 percentage points below difficult English.Scaling yields the largest gains on rhetorical and perturbed Chinese, yet does not remove Chinese surface-normalization failures.
  • Template-level difficulty and fallacies: 94.67% on English equivalence falls to 23.56% on standard Chinese and 8.00% on rhetorical Chinese for Ministral-8B, making propositional equivalence a major bottleneck.Errors concentrate in tasks requiring global scope tracking, truth-condition comparison, or resistance to pragmatically tempting invalid conclusions.
  • Back-translation diagnostics: 90.53% to 99.10%, 95.73% to 99.30%, and 92.60% to 97.73%: back-translating standard Chinese improves Qwen3-8B, Qwen3-32B, and GLM-5.1 on the General aligned set.On the Difficult aligned set, Qwen3-8B and Qwen3-14B improve substantially, whereas Qwen3-32B and GLM-5.1 decrease after back-translation.

5 Analysis

The analysis finds that strong English logical reasoning does not ensure robust Chinese understanding: models must normalize varied Chinese surface forms before inference, and fine-grained evaluation is needed to reveal language-, template-, and phenomenon-specific failures.

  • Key findings: Strong English logical performance does not guarantee robust Chinese understanding, while scaling helps but leaves a sizable English–Chinese gap.Model families also differ in strengths and weaknesses across languages and surface realizations.
  • Reasoning stages: CHLOGIC requires models to normalize a surface expression into a stable logical form before performing inference over it.By fixing the logical template while varying Chinese realizations, the benchmark stresses normalization, where errors may involve conditional particles and negation scope.
  • Evaluation granularity: Fine-grained evaluation is necessary because overall accuracy can hide opposite failure modes across languages and answer-label distributions.Template-family-level and phenomenon-level results expose these cases and identify Chinese forms requiring improvement.
  • Evaluation granularity: Qwen3-14B illustrates high English accuracy with limited Chinese robustness on the Difficult aligned set, whereas Qwen3-0.6B can receive misleadingly high scores from answer bias matching the label distribution.These examples show why aggregate scores alone are insufficient for diagnosing multilingual logical reasoning.

6 Related Work

Prior logical-reasoning and Chinese or multilingual benchmarks evaluate structured reasoning and language understanding, but do not directly test logical consistency across complex Chinese surface realizations. ChLogic addresses this gap by aligning expression families to formal templates and testing whether judgments remain correct under Chinese realization changes.

  • Logical reasoning benchmarks: Synthetic reasoning benchmarks control logical structure and enable proof-style evaluation, but do not test invariant judgments under complex Chinese surface transformations.Examples include bAbI, RuleTaker, ProofWriter, FOLIO, and categorical-syllogism studies.
  • Chinese and multilingual evaluation: Chinese NLU, broad Chinese evaluation, and multilingual benchmarks measure language understanding, knowledge, reasoning, or cross-lingual performance without isolating identical latent logical templates across controlled English–Chinese realizations.The cited resources include CLUE, C-Eval, XNLI, MGSM, SeaEval, and MultiNRC.
  • Robustness under linguistic variation: Contrast sets, behavioral tests, symbolic-template variation, and typographical perturbations show that high average accuracy can conceal brittle reasoning.These approaches motivate evaluating robustness under linguistic variation.
  • Robustness under linguistic variation: ChLogic specializes robustness testing for logic by aligning each expression family to a formal template and checking whether judgments remain correct after Chinese realization changes.The diagnostic target is invariance of the logical judgment under controlled surface variation.

7 Conclusion

CHLOGIC evaluates logical understanding under complex Chinese expressions and shows that strong English accuracy does not ensure robust Chinese reasoning. Its results also identify persistent difficulties across Chinese variants and instability in back-translation benefits.

  • Benchmark contribution: CHLOGIC is a controlled English–Chinese benchmark for logical understanding under complex Chinese expressions.
  • Main findings: High English logical accuracy does not guarantee robust Chinese understanding across Qwen3, Ministral, and GLM models.
  • Main findings: Scaling improves many general Chinese variants, but rhetorical, perturbed, quantifier-sensitive, and equivalence-law expressions remain difficult.
  • Main findings: Back-translation often improves the General aligned set, but is unstable on the Difficult aligned set and mixed for Chinese-only expressions.

Limitations

ChLogic is a diagnostic benchmark, not a complete measure of Chinese reasoning ability. Its coverage, model set, generated variants, and back-translation procedure impose important limitations.

  • ChLogic’s template-driven aligned items may not cover all forms of discourse-level reasoning.
  • The evaluation includes only seven contemporary systems, and results may change as model families are updated.
  • Despite prespecified labels and manual checks, generated Chinese variants may still contain artifacts.
  • Back-translation is model-mediated and can remove Chinese cues or introduce ambiguity on the English side.

A Additional Benchmark Construction Details … D Label Inventory and Distribution

ChLogic combines a broad 60-proposition inventory, Chinese-specific phenomena, controlled multilingual surface realization, audited construction, and structured release and labeling procedures. Its aligned sets contain parallel English–Chinese conditions, while the Chinese-only set targets language-specific normalization challenges and uses an expanded label inventory.

  • A.1 Logical Templates and General Propositions: The 60-proposition inventory combines common benchmark-tested logical forms with additions targeting Chinese linguistic variation and logical robustness.Additions include exclusive disjunction, unique existence, quantifier-related cases, branching and multi-premise reasoning, non-transitivity, and antisymmetry.
  • A.2 Statements Specific to Chinese Context: The Chinese-only component contains 15 phenomenon groups designed to test form–meaning mismatches caused by compact markers, word order, pragmatic tone, and context dependence.Target phenomena include conditional ambiguity, comparison, temporal-causal confusion, concession and contrast, irony, idioms, rule-style language, and homophone-based concept shifts.
  • A.3 LLM-Assisted Generation and Quality Control: Construction uses fixed logical templates and labels, with LLMs generating one English expression and five Chinese realizations while preserving the same latent relation.Quality control removes or revises items that add or delete premises, alter the question or label, introduce ambiguity, or fail Chinese fluency and intended difficulty checks.
  • B Construction Audit and Responsibility Matrix: DeepSeek-V3 handled generation, revision, and back-translation, Yi-1.5-34B-Chat independently verified items, and human reviewers made final acceptance decisions.Yi-1.5-34B-Chat flagged 300 candidate realizations for revision, while 100 additional items were manually revised and no items were rejected.
  • C Instance Expansion, Filtering, and Release Structure: The aligned benchmark expands 60 General Propositions into 3,000 latent items and 40 Difficult Problems into 2,000 latent items, each rendered as one English reference and five Chinese realizations.The five Chinese conditions are standard, natural written, colloquial, rhetorical-question, and perturbed Chinese; they are parallel evaluation conditions rather than independent logical items.
  • C Instance Expansion, Filtering, and Release Structure: The Chinese-only set has 15 phenomenon types with 100 items each, totaling 1,500 items, and is released as a test-only diagnostic benchmark without train, development, or test partitions.It is generated directly in Chinese, and exact-string checking of the supplied file found no duplicate full-text items.
  • D Label Inventory and Distribution: Aligned sets use binary YES/NO entailment–validity judgments, whereas the Chinese-only set additionally includes UNKNOWN for genuine underdetermination.The aligned templates exclude UNKNOWN because they are constructed to yield determinate validity judgments; label frequencies also make aggregate accuracy sensitive to answer bias.

E Bias-Aware Metrics for the General Aligned Set · F Back-Translation Audit and Interpretation

The General aligned set exposes severe answer-distribution biases that aggregate accuracy can obscure, while back-translation is evaluated as a potentially meaning-altering diagnostic rather than a neutral transformation. The audit checks whether translated items preserve logical content, labels, negation and condition direction, and pragmatic force.

  • E Bias-Aware Metrics for the General Aligned Set: The General aligned set contains 2,350 YES and 650 NO instances from 60 templates, supporting bias-aware evaluation across six surface-realization conditions.Qwen3-0.6B and Ministral-3B were selected because they showed the clearest answer-distribution biases.
  • E Bias-Aware Metrics for the General Aligned Set: 78.33% accuracy but 50.00% balanced accuracy and 0 NO recall show that Qwen3-0.6B relies on a universal-YES strategy.Qwen3-0.6B predicts YES for every item under all six conditions.
  • E Bias-Aware Metrics for the General Aligned Set: Ministral-3B predicts NO for 54.87% of English-reference items, rising to 89.83% under rhetorical Chinese.Its NO recall reaches 100% in rhetorical Chinese, while YES recall falls to 12.98%.
  • E Bias-Aware Metrics for the General Aligned Set: 100% NO recall and 12.98% YES recall yield a 30.92% Macro-F1 for Ministral-3B under rhetorical Chinese.The result shows why high accuracy on NO-heavy template families should not be treated as uniformly strong logical reasoning.
  • F Back-Translation Audit and Interpretation: Back-translation is treated as a diagnostic transformation because it may simplify wording, alter rhetorical force, change logical scope, or introduce English-side ambiguity.Potential changes include negation scope, condition direction, homophonic or culturally grounded cues, and pragmatic force.
  • E Bias-Aware Metrics for the General Aligned Set: Bias-aware performance and per-label precision, recall, and F1 are reported for the General aligned set.Predicted NO is defined as the complement of Predicted YES.
  • F Back-Translation Audit and Interpretation: DeepSeek-V3 generated the back-translated English condition used in the audit.The resulting condition is summarized in Table 7.
  • F Back-Translation Audit and Interpretation: Sampled Chinese items were compared with back-translated English across premise and question preservation, locked labels, logical direction, and pragmatic force.The audit specifically examined rhetorical questions, irony, and homophone-based shifts.

G Additional Results and Analysis … K.2 Fine-grained results grouped by logical-template family

The additional analyses show that Chinese surface realization creates persistent logical-reasoning weaknesses, especially for difficult, rhetorical, and template-specific cases, while scaling and translation can partially recover performance. They also identify label bias, language-dependent fallacy detection, and controlled benchmark construction as important qualifications for interpreting results.

  • G Additional Results and Analysis; G.1 Scaling trend within Qwen3: English-standard accuracy generally exceeds Chinese accuracy, with Qwen3-32B reaching 99.07% on general English but 69.35% on difficult rhetorical Chinese.The gap is attributed primarily to recovering intended logical structure from Chinese surface expressions rather than formal inference alone.
  • G.1 Scaling trend within Qwen3: Scaling Qwen3 from 8B to 32B raises average general Chinese accuracy from 79.61% to 95.06% and difficult Chinese accuracy from 52.77% to 82.32%.Rhetorical Chinese improves from 68.69% to 93.53% in the general set, but difficult rhetorical Chinese remains below difficult English at 69.35% versus 96.05%.
  • G.2 Logical-template-level difficulty: Propositional equivalence is a persistent bottleneck, while invalid-inference accuracy can collapse when Chinese phrasing encourages plausible but invalid conclusions.Qwen3-8B drops from 99.75% on English invalid inference to 41.50% on rhetorical Chinese invalid inference; Qwen3-32B reaches 90.89% on colloquial Chinese equivalence.
  • G.3 Language-dependent fallacy detection and targeted English augmentation: Fallacy detection is language-dependent: Ministral-3B rises from 91.50% on English invalid inference to 100% on every Chinese variant.Equivalent Chinese translations or more explicit English formulations can expose reasoning ability that compact English wording obscures.
  • G.4 Chinese-to-English back-translation experiments: Back-translation substantially improves the General aligned set, including Qwen3-32B from 95.73% to 99.30%, but has mixed effects on difficult problems.On the Difficult aligned set, Qwen3-32B falls from 83.10% to 79.00% and GLM-5.1 from 81.00% to 62.95% after back-translation.
  • G.4 Chinese-to-English back-translation experiments: On Chinese-only phenomena, back-translation improves Qwen3-8B from 67.20% to 72.67% and GLM-5.1 from 76.27% to 88.73%, but lowers Qwen3-14B from 70.07% to 66.27%.The strongest gains include Ministral-8B ellipsis from 20% to 100% and temporal-order versus causality from 0% to 94%.
  • G.5 Degenerate and partially degenerate behavior: Qwen3-0.6B and Ministral-3B show degenerate label strategies rather than stable logical reasoning: Qwen3-0.6B favors YES, while Ministral-3B favors NO.Qwen3-0.6B’s General gold-label distribution is YES 78.33%, NO 21.67%, UNKNOWN 0%; Ministral-3B reaches 100% on all Chinese invalid-inference variants.
  • H Forty Difficult Logical Problems; I Examples of Statements Specific to Chinese Context; I.1 Romanization of the Figure 1 Example; I.2 Representative Chinese-Only Examples; J Generation, Verification, and Evaluation Prompts; J.2 Prompt for Statements Specific to Chinese Context; J.3 Verification Prompt; J.4 Evaluation Prompt; K Detailed Experimental Tables; K.1 Logical-template-family summaries by model; K.2 Fine-grained results grouped by logical-template family: The benchmark uses 40 difficult logical skeletons, 15 Chinese-context phenomenon types, and aligned English–Chinese realizations verified to preserve the same latent relation and label.Evaluation requires exactly one of yes, no, or unknown; detailed tables report English, five Chinese conditions, and family-level results grouped across models.

K.3 Enhancement experiments

The section reports enhancement experiments for Qwen3-8B and Qwen3-14B, alongside fine-grained accuracy breakdowns across logical reasoning categories and Chinese realization types.

  • Fine-grained accuracy is organized by basic connectives, propositional and predicate reasoning, quantifier equivalence, multi-step reasoning, and relation logic.
  • Fine-grained tables report results across English, Ch-Std, Ch-Nat, Ch-Col, Ch-Rhet, and Ch-Pert realizations.
  • Enhancement experiments are reported for Qwen3-8B and Qwen3-14B.
Loading 2606.17905v1…