PaperlayerView exact revision on arXiv

Source-linked AI summary

Competence-Based Evaluation of Logical Reasoning in Large Language Models

Jize Jiang, Chengxiang Zhai

arXiv:2610.22894v1cs.CLcs.AIcs.LG

TL;DR

Single accuracy scores on selected reasoning tasks do not show whether models reliably apply a competence across a problem family. The paper proposes CBE, using generated instances and answer-preserving presentations, and finds that even strong models fail its condition across the tested ordering family. Its ordering setup also does not isolate multistep inference capacity.

  • Problem

    Single benchmark accuracy can conflate language understanding, memorization, shortcuts, and reasoning, leaving limited evidence about robust reasoning capability.

    Source: §1 · p. 1

  • Method

    CBE uses a parameterized generator to scale a task family while holding its competence fixed, then tests correctness across answer-preserving presentations.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    Even strong models fail the CBE condition across the tested ordering family despite reasonably good performance over bounded complexity ranges.

    Source: §8 · p. 8

  • Takeaways & Limitations

    CBE distinguishes gains in aggregate performance from gains in correct, invariant behavior and provides diagnostic evidence to guide future model development.

    Source: §1 · p. 2

    The ordering study does not isolate multistep inference capacity because chain length also measures presentation load and endpoint queries admit a shortcut.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have brought impressive performance across diverse tasks; however, shortcomings remain in logical reasoning, including inconsistencies under linguistic variation. To improve the reasoning capacity of LLMs, we must be able to accurately measure their reasoning capacity. Current benchmarks primarily report single accuracy numbers on selected reasoning tasks. Such a performance-based evaluation methodology says little about a model's generalization capacity in reasoning. To address this limitation, we argue for a different evaluation paradigm, called competence-based evaluation (CBE), which tests whether models reliably apply the same competence across a family of problems. A parameterized generator scales complexity while keeping the required competence fixed, and matched answer-preserving presentations test correctness beyond one formulation. We instantiate CBE on elementary ordering tasks, scaling input size while varying presentations of the same ordering rule. Across six GPT and Qwen models, performance is reasonably good over bounded complexity ranges, but even strong models fail the competence condition across the tested family. Chain-of-thought and GRPO extend strong performance without resolving all tested failures; supervised LoRA shows little held-out transfer. CBE provides diagnostic profiles and an evaluation protocol that can grow with future models.

1 Introduction

The paper argues that benchmark accuracy alone cannot establish robust reasoning, and proposes testing whether a model applies the same competence across a scalable family of problems. In an ordering-task evaluation, even strong models fail this condition across the tested family.

1 Introduction

  • Competence-based evaluation tests reliable application of the same task knowledge across generated instances and answer-preserving presentations.Its parameterized generator scales complexity while keeping the required competence fixed.Source: §1 · pp. 1–2
  • CBE produces a competence profile showing where correct, invariant behavior deteriorates, rather than reducing evaluation to a single score.Paired outputs do not by themselves locate a failure in either internal reasoning stage.Source: §1 · p. 2
  • Fresh generation reduces dependence on published test items, while profiles expose failures by size, distractors, and equivalent reformulations.The evaluation can extend its parameter range as models improve, though familiarity with the generator or learned shortcuts remains possible.Source: §1 · p. 2
  • On front/behind at n = 1024, GPT-4.1 retains 77.5% original-prompt accuracy but answers both matched presentations correctly on only 56.8% of valid pairs.Across the tested ordering family, strong models perform reasonably well over bounded complexity ranges but still fail the CBE condition.Source: §1 · p. 2
  • The paper formalizes CBE and instantiates it on ordering tasks whose rule remains fixed as entity count grows.Endpoint queries admit a shortcut and do not isolate multistep inference; Appendix C illustrates CBE adoption for Horn-clause inference.Source: §1 · p. 2

2 Related Work

Prior work supplies fixed benchmark samples, generated complexity scaling, and invariance tests. CBE distinguishes itself by combining an explicit competence condition with matched presentations across a scalable problem family.

2 Related Work

  • CBE makes correctness across equivalent presentations part of an explicit competence condition tracked as complexity grows within one task family.This differs from evaluating robustness on a chosen set or using an accuracy curve alone.Source: §2 · p. 2, §2.3 · p. 3

2.1 Natural-Language Reasoning Benchmarks

  • Curated reasoning benchmarks provide valuable task-performance samples, while CBE organizes correctness across equivalent presentations and sizes around a specified family.The cited benchmarks include ReClor, LogiQA, AR-LSAT, and FOLIO.Source: §2.1 · p. 3

2.2 Generated and Complexity-Scaled Evaluation

  • Generated and complexity-scaled benchmarks control instance structure or difficulty; CBE adds a competence condition across matched presentations and the resulting family profile.ReCogLab is especially close, generating relational reasoning with entity count, reasoning length, and textual features as parameters.Source: §2.2 · p. 3

2.3 Invariance and Diagnostic Evaluation

  • Invariance tests and robustness studies provide direct precedents, while CBE traces whether their requirements hold as complexity grows in a fixed task family.Prior work includes metamorphic testing, paraphrase consistency, and controlled sensitivity studies.Source: §2.3 · p. 3

3 Competence-Based Evaluation

CBE specifies a competence over a generated problem family and requires correct behavior across answer-preserving presentations. It reads out this behavior through correctness and invariance profiles, with statistical comparisons and uncertainty estimates.

3.1 A Competence and Its Problem Family

  • CBE specifies a generator, presentation set, model, correctness criterion, and competence condition for a fixed task rule.Generator parameters vary instance size or presentation load, while an exact oracle supplies the answer and each presentation preserves it.Source: §3.1 · p. 3
  • The instance-level condition requires correct answers across presentations, and the family-level condition extends that requirement throughout the specified instance and complexity space.Finite evaluation can expose violations but cannot certify exhaustive competence or identify internal causes.Source: §3.1 · p. 3

3.2 Profiles of Correctness and Invariance

  • For paired presentations, R, D, and U represent both-correct, both-wrong, and exactly-one-correct outcomes, separating robust correctness from shared errors and presentation dependence.Mean accuracy is R + U/2, while invariance is R + D; the competence profile tracks these masses across generator settings.Source: §3.2 · p. 3
  • The κn statistic discounts a common-bias independent-answer reference but is undefined for constant outputs and does not identify independence or a failure mechanism.The summary κ̄ averages κn over the grid.Source: §3.2 · p. 3

3.3 Readout and Inference

  • Readout locates regions of deterioration, uses common support for profile identities, and pairs marginal-accuracy tests with bootstrap intervals on matched instances.Accuracy treats invalid responses as wrong on all prompts, whereas pair metrics use jointly parsed answers; McNemar tests use Holm correction.Source: §3.3 · p. 4

4 An Instantiation of CBE: Ordering

The ordering instantiation scales a total-order task while preserving its rule and varying answer-preserving presentations. The base evaluation compares six models under specified sampling and matched-instance conditions.

4.1 The Ordering Family

  • The ordering family samples total orders over n named entities, asks endpoint yes/no queries, and scales n from 4 to 1024 while retaining the same rule.Matched rewrites vary phrasing and shuffle names and premises; the catalog covers 31 vocabularies across six relation families. Endpoint queries admit a shortcut and do not isolate multistep inference.Source: §4.1 · p. 4

4.2 Evaluation Setup and Models

  • The base study uses 200 matched pairs per n across six base models, with GPT-5 at minimal reasoning effort and Qwen thinking disabled.GPT models use six spatial-line vocabularies, while Qwen models cover the full catalog; a separate n = 2048 block changes presentation distribution, so its failures cannot be attributed solely to length.Source: §4.2 · p. 4

5 Competence on Ordering

On ordering tasks, models perform reasonably well over bounded complexity ranges, but matched presentations, vocabulary shifts, and longer chains expose failures that aggregate accuracy can conceal.

5.2 Vocabulary Changes the Profile

  • Changing front/behind to left/right lowers GPT-4.1’s accuracy from 93.1% to 70.9% and κ̄ from 0.75 to 0.33, while Qwen3.5-27B’s κ̄ rises from 0.37 to 0.54.Across relations, performance also varies substantially; nonce direction terms yield at most 0.03 κ̄ for every tested model, even with the converse rule supplied.Source: Table 1 · p. 5, §5.2 · p. 5

5.3 Separating Presentation Factors

  • Reshuffling changes GPT accuracy by at most 1.4 points, while full rewrites cost GPT-4.1 and GPT-5 9.1/11.2 accuracy points and 12.4/9.7 invariance points; distractors and negation also expose paired instability.For GPT-5-mini, distractors yield 14.8% correct-to-wrong changes and 14.6% changes in the reverse direction, leaving aggregate performance almost unchanged.Source: §5.3 · pp. 5–6

5.1 Bounded Performance on Front/Behind

  • Through n = 128, GPT-4.1 and GPT-5 retain at least 94.0% accuracy and 87.9% invariance, but both show lower invariance and robust correctness at n = 1024.GPT-4.1’s accuracy is 77.5% at n = 1024, while robust correctness is 56.8%; the profiles show failures of the competence condition across the tested range.Source: §5.1 · p. 4, Figure 2 · p. 6

5.4 Reasoning Extends the Strong-Performance Range

  • Reasoning effort extends GPT-5.4’s strong range, but high effort at n = 2048 retains 67.0% invariance under a separate presentation condition.For Qwen3.5-27B, finished-answer accuracy reaches 100.0/99.6%, but all-prompt accuracy is 87.0/79.1% and strict invariance is 75.4% when unfinished pairs count as disagreements.Source: §5.4 · pp. 6–7, Figure 3 · p. 7

6 What Training Changes

Training affects held-out ordering performance differently: GRPO improves transfer, while supervised fine-tuning shows little held-out gain, and long-chain presentation failures remain.

6 What Training Changes

  • The interventions are LoRA on Qwen3.5-9B, with front/behind held out; GRPO training excludes spatial-line vocabularies to test transfer to held-out relation words.SFT uses auxiliary relation data, paired presentations, or paired presentations with a consistency loss.Source: §6 · p. 7
  • Supervised variants reach 100% on held-in relations, but front/behind invariance remains 56.6–58.3%, versus 57.3% for the base model; the paired change is −0.4 points with interval [−3.5, +2.5].The consistency term produces little additional held-out improvement.Source: §6 · p. 7
  • Across twelve clean held-out relations, GRPO raises invariance from 60.0% to 83.5% and κ̄ from 0.10 to 0.60, supporting transfer beyond the training vocabulary.The comparison does not establish a general ranking of training methods because the SFT and GRPO data, objectives, and budgets differ.Source: §6 · p. 7
  • GRPO raises held-out front/behind accuracy to 81.2/75.1% and invariance to 73.3%, yet chance-corrected invariance is −0.12, −0.11, and −0.20 at n = 256, 512, and 1024.The profile shows improved pooled performance and cross-vocabulary transfer alongside substantial long-chain presentation failures; training does not meet the competence condition across the tested range.Source: Table 3 · p. 7, Figure 4 · p. 8, §6 · p. 8

7 Discussion and Diagnostic Evidence

The discussion treats competence profiles as a way to distinguish correct consistency from shared errors and uses diagnostic interventions to probe possible contributors without claiming a complete mechanism.

7 Discussion and Diagnostic Evidence

  • Higher invariance can reflect agreement on an incorrect default, so competence profiles distinguish stable correct answers from shared errors.In Qwen3-8B, head ablation raises invariance from 81.0% to 93.0% while original/equivalent accuracy falls from 64.5/59.5% to 59.0/52.0%.Source: §7 · p. 8
  • Open-weight probes and interventions characterize possible contributors to biased outputs, but do not establish a complete mechanism.GRPO strengthens the measured signal relative to the answer default, while longer-chain equivalents show reduced decodability at a measured layer and position.Source: §7 · p. 8
  • CBE makes comparisons and intervention gains interpretable in terms of the behavior they actually support by reporting where correctness survives answer-preserving changes.The protocol specifies a competence, a controlled-complexity problem family, and answer-preserving presentations.Source: §7 · p. 8

8 Conclusion

CBE evaluates whether models apply the same rule reliably across growing inputs and equivalent presentations. Strong models still fail this condition across the tested ordering family, while diagnostic profiles can guide model development and coverage can extend with models.

8 Conclusion

  • The ordering instantiation tests reliable application of the same rule across growing inputs and equivalent presentations, but does not isolate an inference-depth limit.Source: §8 · p. 8
  • Even strong models fail the competence condition across the tested ordering family, despite reasonably good performance over bounded complexity ranges.Better performance alone does not establish competence across the family.Source: §8 · p. 8
  • Competence profiles can guide model development by locating gains across sizes and presentations.Source: §8 · p. 8
  • CBE offers an evaluation practice that extends the complexity range while testing the same competence under matched presentations.It distinguishes an expanded range of success from demonstrated ability throughout the evaluated family.Source: §8 · p. 8

Limitations

The limitations constrain what CBE and the ordering study establish: finite samples cannot certify an unbounded family, and the ordering setup does not isolate multistep inference. Behavioral profiles and tested interventions also leave mechanisms and broader training comparisons unresolved.

Limitations

  • Finite samples can reveal failures but cannot certify correctness throughout an unbounded family, and the ordering study plus Horn-clause example leave broader reasoning competences untested.Richer tasks and wider presentation coverage could clarify how diagnostic profiles generalize.Source: Limitations · p. 9
  • Chain length measures input size and presentation load, while endpoint queries admit a shortcut, so the ordering results do not isolate multistep inference capacity.Length and presentation changes are not always separated; future generators could control derivation depth, distracting information, and linguistic realization independently.Source: Limitations · p. 9
  • Behavioral profiles and linear probes do not uniquely identify internal reasoning processes, and paired metrics condition on valid answers.Agreement should be read alongside completion coverage; broader sampling and causal interventions could connect behavioral failures to mechanisms.Source: Limitations · p. 9
  • The studied reasoning and training interventions leave many routes to improved competence unexplored, and their gains do not establish a general ranking of training methods.Matched comparisons across training objectives, model families, and compute budgets could help identify approaches that extend correct, invariant behavior beyond the trained range.Source: Limitations · p. 9

Ethical Considerations

The authors keep conclusions scoped to the tested family and flag risks of overinterpreting generated-task success and unmeasured environmental costs. The supplied appendix passages clarify paired-outcome metrics, evaluation protocols, and a Horn-inference instantiation.

Ethical Considerations

  • Success on the generated family should not be treated as evidence of general reasoning or deployment readiness, and failures do not establish an absence of all useful reasoning.The authors also note that repeated optimization against a known generator may encourage benchmark-specific strategies; fresh instances alone do not establish transfer.Source: Ethical Considerations · p. 9

A.1 Paired-Outcome Identities

  • Labeled matched pairs identify robust-correct, shared-wrong, and presentation-dependent outcomes, whereas marginal accuracies alone leave robust correctness within a range.A paired flip rate identifies invariance but cannot distinguish shared correct answers from shared errors.Source: §A.1 · p. 12
  • The tables’ pooled output-bias reference additionally penalizes differing presentation yes-rates, and its reported normalization is undefined when the pooled yes-rate is zero or one.The authors caution that the reported κ sign should not be read as the sign of correctness covariance.Source: §A.1 · p. 12
  • A threshold-crossing estimate can be unstable when a profile sits near the threshold, while the profile shape is reported as stable under resampling.At 200 pairs per complexity level, crossing reproduction is only 0.50 for GPT-5 and 0.39 for GPT-4o on front/behind; profile shape remains stable under resampling.Source: §A.1 · p. 13

B.1 Ordering Generator

  • Protocol P redistributes rewritten premises across the chain, unlike Protocol L’s position-linked rewrites; original-presentation accuracy is shared by construction.Equivalent-presentation accuracy is lower under Protocol P in 43 of 50 model-vocabulary cells, by up to 7.7 points.Source: §B.1 · p. 13, B Generators, Protocols, and Training… · p. 13

B.2 Evaluation Protocol

  • The evaluation uses matched-instance bootstrap intervals and corrected McNemar tests for marginal accuracy differences, with pair metrics restricted to jointly parsed answers.Figure 2 is recomputed from archived answers, without new model calls.Source: §B.2 · p. 14

B.3 Training Details

  • The GRPO reward uses correctness and agreement across an instance’s other presentations, with the instance yes-rate also entering the reward definition.Unparseable answers receive zero reward.Source: §B.3 · p. 14

C.1 Mapping the Framework to a New Task

  • The Horn-clause instantiation keeps the correctness criterion and paired-outcome analysis while requiring task-specific latent instances, an oracle, and semantics-preserving renderers.Rule count indexes program size, but proof depth and distractor content co-vary with it, so the profiles do not isolate proof-depth effects.Source: §C.1 · p. 15

C.2 Illustrative Results

Appendix results extend the ordering evaluation across additional models, protocols, training conditions, and relations. They show that high accuracy or agreement can mask presentation sensitivity, shared errors, and limited transfer.

C.2 Illustrative Results

  • Across the illustrative Horn-clause results, Qwen3.5-27B has 82.0% invariance but includes 18.0% pairs where exactly one presentation is correct.The appendix describes this as an adoption example with narrower protocol coverage than the main ordering study.Source: §C.2 · p. 15, Table 5 · p. 16

D Per-Axis Comparison with Prior Work

E.1 Additional Open-Weight Models on a Nine-Relation Core

  • Qwen2.5-32B exceeds Qwen3.5-27B on invariance and chance-corrected invariance across every family in the nine-relation core, but both models fail on nonce direction words.Gemma-4-31B’s parsed-answer invariance can conceal format abandonment under the token cap.Source: §E.1 · p. 19, Table 8 · p. 19

F Results under the Release Protocol

  • The Protocol L ordering results show that GPT-4.1 and GPT-5 keep invariance above 87% through n = 128, then fall steeply, while several models’ high large-n agreement reflects shared-wrong answers.Protocol L results are reported separately because its eight-model suite and n = 2048 level were not rerun under Protocol P.Source: §F.1 · p. 20, F Results under the Release Protocol · p. 20

F.2 Supervised Fine-Tuning under the Release Protocol

  • On the POS relation, LoRA reduces Qwen3.5-9B’s marginal accuracy gap from 6.5 to 0.6 points without raising accuracy, while per-pair invariance falls from 64.6 to 57.2.Under Protocol P, the same adapter is within noise of its base on every axis.Source: §F.2 · pp. 20–21, Table 12 · p. 21

F.3 A Names-List Artifact as a Diagnostic

  • A chain-ordered names list enabled a Qwen3.5-9B LoRA model to reach at least 99% accuracy through n = 2048 on both presentations, but shuffled names reduced accuracy to 62.3/66.1.The base Qwen3.5-27B also scored 90.5 with chain-ordered names versus 77.3 with shuffled names; main results use shuffled names.Source: §F.3 · p. 21, Table 17 · p. 25

F.4 Cross-Relation Replication (DEPTH)

  • Cross-relation LoRA on the held-out DEPTH relation again fails to raise accuracy over its base model.The replication uses objects rather than people and tests chain lengths from 4 to 64.Source: §F.4 · p. 22, Table 18 · p. 25

G.1 Training-Seed Replication

The seed replication examines whether training effects hold across runs and whether an agreement reward contributes beyond correctness. The agreement bonus has little consistent effect relative to seed variation, and both runs retain long-chain failures.

G.1 Training-Seed Replication

  • Both GRPO seeds retain substantial long-chain failures, and seed 43 has higher mean accuracy and chance-corrected invariance than the selected seed 42 across twelve relations.Seed 42 was selected by mean invariance on those held-out relations, so it is not uniformly stronger across metrics.Source: §G.1 · p. 26, Figure 9 · p. 27
  • The nominal GRPO agreement bonus is inert: per-presentation advantage groups make its normalized updates equivalent to correctness-only updates, with logged cosine 1.0000.Nonzero-advantage steps occurred on only 28 of 600 steps for seed 42 and 31 of 600 for seed 43.Source: §G.1 · p. 26
  • Changing the agreement reward yields +1.8 invariance points for seed 42 but +0.2 for seed 43, within the 1.3–2.8-point seed-to-seed spread.Every variant leaves front/behind weakest and nonce direction words near zero chance-corrected invariance.Source: §G.1 · p. 26, Table 20 · p. 27
  • The two-seed evidence does not establish a consistent improvement beyond observed training-seed variation.The evaluation intervals cover evaluation noise only.Source: §G.1 · pp. 26–27

H Identical-Prompt Repeat Agreement

The appendix compares identical-prompt repeat agreement with agreement across equivalent presentations, then adds output probes and interventions. Repeatability, invariance, and correctness capture distinct behaviors; the probes support but do not fully explain presentation dependence.

H.1 Control and Interpretation

  • The repeat control measures run-to-run agreement without changing the prompt; it is a measured reference, not a universal lower bound.A gap from cross-presentation agreement does not causally isolate presentation error because answer uncertainty may differ between presentations.Source: §H.1 · p. 28

H.2 Base Ordering Models

  • For GPT-4.1, identical-prompt repeat agreement is 96.4% versus 87.9% cross-presentation invariance, illustrating sensitivity beyond the measured repeat level.Across six base models, repeat agreement ranges from 64.8% to 96.4% on front/behind.Source: §H.2 · p. 28

H.3 Training Comparisons

  • GRPO raises seed-42 repeat agreement from 64.8% to 93.7%, while invariance rises from 57.3% to 73.3%, leaving substantial presentation failures.The difference between repeat agreement and invariance widens from −7.5 to −20.4 points.Source: §H.3 · p. 28, Table 22 · p. 28

H.4 Reasoning and Additional Models

  • GPT-5.4’s medium-effort condition gives 98.8% repeat agreement and 98.4% invariance, but repeat controls were not collected for every effort condition.The figure compares invariance and original-presentation accuracy by n across two relations and reasoning efforts.Source: §H.4 · p. 28

I Mechanistic Probes and Interventions

  • At short chains, the base Qwen3.5-9B’s gold answer is linearly decodable on lexical relations despite nearly constant “yes” outputs, consistent with a readout default overwhelming usable answer information.The observations do not rule out earlier representational errors.Source: I Mechanistic Probes and Interventions · p. 29
  • For Qwen3.5-9B at layer 26, equivalent-presentation AUROC drops from 0.98 at n = 16 to 0.54 at n = 256, indicating reduced linear decodability at that measured position and layer.This does not prove the answer is absent from other representations or nonlinear readouts.Source: I Mechanistic Probes and Interventions · p. 29
  • Ablating four selected Qwen3-8B heads raises invariance from 81.0% to 93.0% while lowering original/equivalent accuracy from 64.5/59.5% to 59.0/52.0%.The increased agreement accompanies worse task performance, showing why invariance alone is insufficient.Source: I Mechanistic Probes and Interventions · p. 29
  • For front/behind, GRPO increases measured instance signal from 0.63 to 3.60 on originals and from 0.50 to 3.22 on equivalents, while the default remains nonzero on originals.These probes cover one model family, small datasets, and mostly short chains, and do not explain API-model failures.Source: I Mechanistic Probes and Interventions · p. 29

Related papers

Loading 2610.22894v1…