PaperlayerView exact revision on arXiv

Source-linked AI summary

KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Tasks

Kaijing Ma, Xinrun Du, Yunran Wang, Haoran Zhang, Zhoufutu Wen, Xingwei Qu, Jian Yang, Jiaheng Liu, Minghao Liu, Xiang Yue, Wenhao Huang, Ge Zhang

arXiv:2410.06526v3cs.DB

TL;DR

Existing evaluations can struggle to separate rule-based reasoning from reliance on pre-trained knowledge. KOR-Bench evaluates models applying newly defined rules across five task categories; O1-Preview and O1-Mini score higher overall than Claude-3.5-Sonnet and GPT-4o.

  • Problem

    Existing benchmarks may not accurately measure reasoning when performance relies on domain knowledge and learned patterns, motivating evaluation of rule following with less reliance on pre-trained knowledge.

    Source: §1 · p. 2

  • Method

    KOR-Bench evaluates applying newly defined rules through five categories, using manually defined rules designed to avoid common pre-training data.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    O1-Preview scores 72.88% and O1-Mini 70.16%, compared with Claude-3.5-Sonnet at 58.96% and GPT-4o at 58.00%.

    Source: §7 · p. 10

  • Takeaways & Limitations

    KOR-Bench is intended as a challenging benchmark for evaluating reasoning while decoupling it from intrinsic knowledge.

    Source: §7 · p. 10

Abstract

from arXiv · show

In this paper, we introduce Knowledge-Orthogonal Reasoning (KOR), a concept aimed at minimizing reliance on domain-specific knowledge, enabling more accurate evaluation of models' reasoning abilities in out-of-distribution settings. Based on this concept, we propose the Knowledge-Orthogonal Reasoning Benchmark (KOR-Bench), encompassing five task categories: Operation, Logic, Cipher, Puzzle, and Counterfactual. KOR-Bench emphasizes models' effectiveness in applying new rule descriptions to solve novel rule-driven questions. O1-Preview and O1-Mini achieve accuracies of 72.88% and 70.16%, surpassing Claude-3.5-Sonnet and GPT-4o (58.96% and 58.00%), highlighting the effectiveness of KOR-Bench. We perform detailed analyses, identifying bottlenecks in the Cipher task with Stepwise Prompting, where two rounds of Self-Correction yield optimal results. We evaluate performance across three integrated tasks, explore the impact of Tricks on the Puzzle task, and visualize rule-focused attention. Additionally, we conduct an ablation study on dataset size, benchmark correlations, and zero-shot and three-shot "only questions" experiments. KOR-Bench aims to enhance reasoning evaluation and support further research in this area.

1 INTRODUCTION

The paper argues that existing evaluations can blur rule-based reasoning with reliance on learned domain knowledge. It introduces Knowledge-Orthogonal Reasoning and KOR-Bench to assess models applying newly defined rules across five task categories.

1 INTRODUCTION

  • The paper frames reasoning as a fundamental aspect of intelligence that includes understanding complex scenarios, planning, and multi-step execution.It cites research on reasoning capabilities emerging as models reach sufficient scale.Source: §1 · p. 1
  • The introduction motivates rule-based adaptation by describing how people learn new rules and apply them to unfamiliar tasks and out-of-distribution settings.The authors contrast this with models’ difficulties beyond training data and their dependence on learned patterns.Source: §1 · pp. 1–2
  • Existing evaluations may not accurately measure reasoning when performance also reflects reliance on in-domain knowledge and memorized patterns.The paper calls for evaluation of models’ ability to understand and follow new rules while reducing reliance on pre-trained knowledge.Source: §1 · p. 2
  • Knowledge-Orthogonal Reasoning (KOR) separates explicit task rules from domain-specific background knowledge while assessing rule application and problem-solving.The benchmark emphasizes applying newly defined rules to new rule-driven questions rather than retrieval or memorization.Source: §1 · p. 2
  • KOR-Bench organizes its challenges around new symbols, concepts, execution rules, problem-solving frameworks, and story contexts.Its five categories are Operation, Logic, Cipher, Puzzle, and Counterfactual reasoning.Source: §1 · p. 2
  • The authors plan to expand the dataset, explore parameterized rules, deepen inference, refine process evaluation, and develop a multimodal version.They describe the tasks as customized rules and problems intended to challenge models beyond traditional reasoning frameworks.Source: §1 · p. 2

2 RELATED WORK

Related work spans knowledge-dependent benchmarks, information integration, and rule-following evaluation. The paper positions knowledge orthogonality as a way to reduce domain-knowledge influence and assess rule following in unfamiliar settings.

2 RELATED WORK

  • Existing reasoning benchmarks span commonsense, logical, multi-hop, and mathematical reasoning.Source: §2 · p. 2
  • One account of reasoning ability emphasizes both extensive world knowledge and integrating new information into an existing knowledge base.Source: §2 · p. 3
  • Knowledge-dependent benchmarks assess accumulation and recall, while domain knowledge can make it difficult to distinguish reasoning from learned-pattern retrieval.The section names commonsense, science, mathematical, and logical benchmarks as examples of this evaluation landscape.Source: §2 · p. 3
  • Prior work on integrating new information is described as relatively limited, despite evaluations using puzzles and complex planning tasks.The section links such tasks to abilities including logical reasoning, spatial cognition, creative thinking, and decision-making.Source: §2 · p. 3
  • Recent evaluation work is expanding from instruction following toward rule following in dynamic problem-solving settings.The section cites benchmarks addressing general rule following, execution and planning, search, and combinatorial reasoning.Source: §2 · p. 3
  • The paper introduces knowledge orthogonality to reduce domain-specific knowledge’s influence and examine rule following in out-of-distribution scenarios.The stated aim is a more comprehensive and fair evaluation framework.Source: §2 · p. 3

3 KNOWLEDGE-ORTHOGONAL REASONING BENCHMARK

KOR-Bench combines manually defined rules and rule-driven questions across five reasoning categories. Its design covers multiple forms of rule application, from novel mathematical operators and logic to ciphers, puzzles, and fictional counterfactual settings.

3.1 OVERVIEW

  • Each of KOR-Bench’s five categories contains 25 manually defined rules designed to avoid common pre-training data, with 10 problem instances per rule.Source: §3.1 · p. 3

3.2 DATA CONSTRUCTION PROCESS

  • Data construction has three phases—rule design, rule-driven Q&A design, and quality validation—with manual annotation and LLM validation and difficulty filtering.Source: §3.2 · p. 3

3.3.1 OPERATION REASONING

  • Operation tasks define new symbolic operators and associated equations, testing comprehension and mathematical reasoning across difficulty levels and domains.Source: §3.3.1 · p. 4

3.3.2 LOGIC REASONING

  • Logic rules adapt traditional logic with symbolic adjustments and new definitions, and each rule has ten problems of varying difficulty.Source: §3.3.2 · p. 4

3.3.3 CIPHER REASONING

  • Cipher tasks modify traditional and modern encryption methods and test stepwise rule application, including transposition, rotation, and spatial understanding.The methods include altered substitution tables and adjusted encryption steps, with accuracy verified using encryption and decryption programs.Source: §3.3.3 · p. 4

3.3.4 PUZZLE REASONING

  • Puzzle rules span classic paper puzzles, number games, and word games; problems vary in difficulty and often combine multiple abilities.The puzzles are sourced from Braingle and Puzzle Prime, and each rule has ten problems.Source: §3.3.4 · pp. 4–5

3.3.5 COUNTERFACTUAL REASONING

  • Counterfactual tasks use 25 works from anime, television, film, and games as fictional settings, requiring answers based on supplied world rules rather than real-life answers.Source: §3.3.5 · p. 5

3.4 STATISTICS

  • KOR-Bench statistics cover rule counts and lengths, question counts and lengths, and answer formats across five task types.The reported lengths refer to characters; answer formats include numerical response, mathematical expression, textual response, multiple choice, and structured data.Source: §3.4 · pp. 5–6, Table 1 · p. 6

4 EXPERIMENT SETUP

Chat models are evaluated with zero-shot prompts containing newly defined rules and questions, while base models receive three generic question-answer examples per rule. Outputs are parsed and checked against answers using task-specific processing.

4 EXPERIMENT SETUP

  • Chat models use zero-shot prompts, whereas base models receive three generic Q&A examples for each rule.Source: §4 · p. 6
  • Evaluation extracts bracketed answers, applies special processing, and compares outputs with answers, using SymPy to parse and simplify mathematical expressions.The study reports accuracy by task type and over the full test set.Source: §4 · p. 6

5 RESULT ANALYSIS

KOR-Bench results vary by model and task: O1 chat models lead overall, while models outside the O1 series struggle most with Cipher and Puzzle. Error analyses identify task-specific bottlenecks, alongside differences in how models apply rules.

Chat Model Performance

  • 72.88% and 70.16% are the overall accuracies of O1-Preview and O1-Mini, ahead of Claude-3.5-Sonnet at 58.96% and GPT-4o at 58.00%.Table 2 presents frontier-model performance on KOR-Bench.Source: §5 · p. 6, Table 2 · p. 7

Base Model Performance

  • Base-model comparisons show less decline in Logic than in other inference tasks, likely because Logic requires shallower inference.Meta-Llama-3.1-405B has the highest base-model accuracy at 39.68%.Source: §5 · pp. 6–7

Reasoning Process Performance

  • Larger models often use Chain-of-Thought to apply rules step by step, while smaller models may omit it and answer Cipher questions with “Hello World.”Larger models still occasionally make execution errors on complex tasks.Source: §5 · pp. 7–8

Reasoning Tasks Performance

  • 42.80% on Cipher and 16.80% on Puzzle are the respective maximum accuracies outside the O1 series, while models generally perform best on Counterfactual tasks.Models also perform well on Operation and Logic tasks, which typically involve one or two reasoning levels.Source: §5 · p. 8

Single Task Analysis

  • Operation errors concentrate on algebraic unknowns, Logic errors on constructing expressions, Cipher errors on several execution steps, and Puzzle errors on multi-step and spatial reasoning.Cipher errors are most frequent in Position Mapping, Transpose Writing, Mathematical Calculation, Split Connection, and Multi-Step Execution; Puzzle models do better on single-solution tasks.Source: §5 · p. 8

6 FURTHER ANALYSIS

The further analyses examine 16 models, locate Cipher sub-step bottlenecks, test self-correction and complex multi-rule tasks, and report additional Puzzle, benchmark-correlation, and rule-inference experiments.

6 FURTHER ANALYSIS

  • The further analyses selected 16 models, including Claude-3.5-Sonnet, GPT-4o, DeepSeek-V2.5, and large and small models from six series.The selected series were Meta-Llama-3.1, Qwen2.5, Qwen2, Yi, Command-R, and Mistral.Source: §6 · p. 8

6.1 STEPWISE PROMPTING ANALYSIS OF CIPHER TASK BOTTLENECKS

  • Nearly 100% error rates on Rotation, Conditional Filling, and Conditional Reading identify spatial operations as Cipher bottlenecks; Encoding and Partition errors were relatively low.The analysis decomposed five highly erroneous rules into sequential sub-steps, covering nine sub-step types.Source: §6.1 · pp. 8–9, Figure 4 · p. 8
  • 10.36% average accuracy increase followed self-correction across models, with the largest gains in the first two rounds and limited later gains.The experiment allowed up to five rounds; longer histories could exceed the context window and require re-inputting a prior response.Source: §6.1 · p. 9

6.3 ANALYSIS ON COMPLEX TASK PROCESSING

  • Claude-3.5-Sonnet performed best across all complex-task settings, while Mistral-7B-Instruct-v0.3 struggled more with Multi-Q than Multi-R and Multi-RQ.The three settings vary the number of questions and rules, and include random combinations of the five task types.Source: §6.3 · pp. 9–10, Table 3 · p. 10

6.4 MORE EXPERIMENTS AND ANALYSES

  • Additional experiments examined Puzzle performance with a Trick field, rule-focused attention visualization, dataset-size robustness, and benchmark correlations, which aligned more strongly with reasoning-focused benchmarks, particularly MMLU-Pro.Zero-shot models relied on prior knowledge and often struggled with accuracy; in three-shot "only questions" experiments, models inferred rules from three examples and improved performance.Source: §6.4 · p. 10

7 CONCLUSION

The conclusion presents KOR-Bench as an evaluation of reasoning that minimizes pre-existing knowledge, with O1-Preview and O1-Mini outperforming Claude-3.5-Sonnet and GPT-4o. The accompanying material describes reproducibility resources, ethical commitments, future extensions, and formal conditions for knowledge orthogonality.

7 CONCLUSION

  • O1-Preview scored 72.88% and O1-Mini 70.16%, while Claude-3.5-Sonnet scored 58.96% and GPT-4o 58.00% on KOR-Bench.The benchmark is intended to evaluate reasoning while minimizing the influence of pre-existing knowledge.Source: §7 · p. 10

REPRODUCIBILITY STATEMENT

  • The authors plan to release the dataset and code, but note that exact reproduction of some proprietary-model results may be difficult because APIs can change.The planned codebase includes data-loader scripts, evaluation metrics, and experiment-running code.Source: REPRODUCIBILITY STATEMENT · p. 11

ETHICS

  • The authors state that data were collected responsibly, participant privacy maintained, and methodology transparency prioritized to address bias and fairness.Source: ETHICS · p. 11
  • Planned extensions include dynamic Cipher keys and text lengths, more reasoning-process observations, and a visual-domain multimodal version.The authors say dynamic parameters would increase rule flexibility and influence the required reasoning depth.Source: ETHICS · p. 11, REPRODUCIBILITY STATEMENT · p. 11

T satisfies knowledge orthogonality under the following conditions:

  • Knowledge orthogonality requires rules to be independent of background knowledge, with correctness relying substantially more on the task rules than on that knowledge.Background knowledge may support or interfere with deriving an answer but is not central to the reasoning under these conditions.Source: T satisfies knowledge orthogonality… · p. 22

A FORMAL DEFINITION OF “KNOWLEDGE ORTHOGONALITY”

  • The derivation adjustment incorporates K’s influence into reasoning via β, defining P(Q →A | R, K) = P(Q →A | R) · (1 + β), where β ranges from (−1, ϵ] for very small positive ϵ.Source: T satisfies knowledge orthogonality… · p. 22

B DATA CONSTRUCTION PROCESS

KOR-Bench data construction proceeds through rule design, rule-driven question-and-answer design, and quality validation, relying primarily on manual annotation and using LLMs for validation and difficulty filtering.

B DATA CONSTRUCTION PROCESS

  • The construction process has three phases: Rule Design, Rule-Driven Q&A Design, and Quality Validation, with manual annotation as its primary mechanism.LLMs are used for quality validation and difficulty filtering.Source: B DATA CONSTRUCTION PROCESS · p. 23

B.1 RULE DESIGN

  • Rule design extracts core rules from puzzles, textbooks, domain knowledge, or virtual settings, then expands or redefines them with new symbols, concepts, constraints, steps, or story contexts.Source: §B.1 · p. 23

B.2 RULE-DRIVEN Q&A DESIGN

  • Rule-driven Q&A design adapts existing questions to the rules or creates new questions and answers manually or programmatically.Counterfactual problems may be manually crafted, while Cipher problems may be programmatically generated; answers receive specified formats.Source: §B.2 · p. 23

B.3 QUALITY VALIDATION

  • Quality validation combines human review of rules and Q&A pairs with LLM assessment of difficulty and discriminative power, followed by repeated revisions.Frequent failures may indicate excessive difficulty or unclear descriptions, while universally correct answers may indicate overly simple setups or data leakage.Source: §B.3 · p. 23

C.1 RULE DISTRIBUTION ACROSS TASK TYPES

The appendix describes rule distributions and answer formats across task types, GPT-4o token statistics, evaluation and extraction procedures, rule summaries, and prompt templates for Logic tasks.

C.1 RULE DISTRIBUTION ACROSS TASK TYPES

  • Table 4 presents rule counts organized hierarchically across the five task categories, including subcategories and tertiary classifications.Source: Table 4 · p. 24, §C.1 · p. 24

C.2 ANSWER FORMAT DISTRIBUTION ACROSS TASK TYPES

C.3 STATISTICAL OVERVIEW OF INPUT AND OUTPUT TOKENS

  • Table 7 reports GPT-4o’s mean and standard deviation for input and output tokens by task type, using the cl100k_base tokenizer.Source: §C.3 · p. 24, Table 7 · p. 25

C.4 DETAILED EXTRACTION AND EVALUATION

  • Evaluation extracts bracketed answers and applies task-specific handling for multiple answers, mathematical expressions, inequalities, and unordered lists before comparison.Mathematical expressions may be parsed or simplified with SymPy; unordered lists are normalized and sorted when order is irrelevant.Source: §C.4 · p. 26

C.5 SUMMARY OF RULE DESCRIPTIONS FOR FIVE TASK TYPES

D PROMPT TEMPLATES

D.2 LOGIC PROMPT

  • The Logic prompt directs models to identify relevant properties and objects, apply the specified reasoning patterns, and format the answer using the required notation and symbols.Source: §D.2 · p. 33

E ANALYSIS OF REAL-LIFE ANSWER RATIOS FOR COUNTERFACTUAL TASK

Counterfactual accuracy declines as the real-life answer ratio rises; this trend is stronger for base models than chat models. The appendix also describes stepwise analysis of Cipher questions using nine sub-step types.

E ANALYSIS OF REAL-LIFE ANSWER RATIOS FOR COUNTERFACTUAL TASK

F.1 STEPWISE PROMPTING ANALYSIS OF CIPHER TASK BOTTLENECKS

  • The Cipher bottleneck analysis selects five high-error rules (IDs 1, 9, 11, 18, and 23) and breaks questions into consecutive sub-steps.Source: §F.1 · p. 38

F.1.1 DETAILED EXPLANATIONS OF NINE KEY SUB-STEP TYPES

F.1.2 EXAMPLE OF A CIPHER QUESTION SPLIT INTO SUB-STEPS

  • Encryption and decryption questions are illustrated as sequences of different sub-step types.Source: §F.1.2 · p. 39

F.1.3 MODEL RESULTS ON SUB-STEP ERROR RATES

The supplementary analyses examine Cipher sub-step errors, Puzzle tricks, attention to rule text, self-correction, integrated tasks, and model error cases. Most models gain most from two correction rounds, while correction patterns differ by task type.

F.1.3 MODEL RESULTS ON SUB-STEP ERROR RATES

F.2 IMPACT ANALYSIS OF TRICKS ON PUZZLE TASK PERFORMANCE

  • The Puzzle experiment compares accuracy with and without a Trick field, and the figure identifies some models that improve when it is provided.The field is appended to the question; in the example, the trick identifies a column equation yielding the value 1 before the grid is completed.Source: Figure 10 · p. 42, §F.2 · p. 42, §F.2.1 · p. 42

F.3 ATTENTION FOCUS VISUALISATION

  • Rule-focused attention is visualized by mapping scores from the top 50 retrieval heads onto rule text, with the needle field underlined.The authors say these visualizations help explain model outputs, particularly errors; focusing on core rule elements is necessary but not sufficient for accuracy.Source: §F.3 · p. 43

F.4.1 SELF CORRECTION PROMPT TEMPLATE

  • The self-correction prompt reintegrates extracted responses into the dialogue history and allows up to four correction rounds.Source: §F.4.1 · p. 44

F.4.2 IMPACT OF ROUND COUNT ON SELF-CORRECTION ACCURACY

  • Two self-correction rounds give most models their highest gains; counterfactual corrections remain higher across four rounds than those for the other task types.For the other four task types, correction effects are most pronounced in rounds one and two; later effects are less pronounced.Source: §F.4.2 · p. 44

F.5.1 COMPLEX TASK PROCESSING PROMPT

  • Complex-task processing evaluates three settings: one rule with 1–10 questions, 2–3 rules with one question, or 2–3 rules with 1–3 questions.Source: §F.5.1 · p. 46

F.5.2 COMPLEX TASK PROCESSING EVALUATION

  • Complex-task evaluation scores each sub-question individually and computes the overall pass rate from the number that pass.Evaluation stops if the response contains fewer results than there are sub-questions.Source: §F.5.2 · p. 46

G FUN-FILLED ANALYSIS OF SLIP-UPS

G.1 OPERATION ERROR CASES ANALYSIS

This section presents examples of models misapplying task rules or constraints across operation, logic, cipher, and puzzle cases, and examines attention and answer-validation behavior across several task types. It also includes an Attack on Titan-contextualized food-preservation prompt and required answer format, without identifying it as an operation-error finding.

G.1 OPERATION ERROR CASES ANALYSIS

  • The model wrongly treated 14 as not a multiple of 7, selecting the wrong operation rule and returning 24 for 25※5※14.The analysis identifies 14 as a multiple of 7, so the response’s choice of the third rule was incorrect.Source: §G.1 · p. 48
  • Another solution made unreasonable assumptions when solving 25※X※14=4 and concluded X=25 despite its own calculations not supporting that answer.It derived X=25/48, then asserted 25 was the only integer solution; its verification instead produced 24, not 4.Source: §G.1 · p. 49
  • Other operation cases involved an incorrect right-to-left evaluation instead of left-to-right order and a partial calculation error that propagated.Source: §G.1 · pp. 50–51

G.2 LOGIC ERROR CASES ANALYSIS

  • In the logic case, the model identified scenarios that implied four possible answers but omitted cases from its final truth-assignment list.The analysis flags incomplete exploration of possible paths in the final response.Source: §G.2 · p. 54
  • The model missed a stated “hexadecimal” requirement, returning a disjunction of all 16 terms as satisfying the tautology and basic-form conditions.The answer lists m0 through m15, while the analysis attributes the error to insufficiently capturing the specific requirement.Source: §G.2 · pp. 55–56
  • For a syllogism it classified as Figure I and mood AEE, the model recognized AEE was invalid for Figure I but still returned that conclusion without correction.The analysis identifies an early matching error that persisted despite the model noticing the mismatch.Source: §G.2 · pp. 58, 60
  • In another syllogism case, an analytical oversight led the model to misread the graphical representation and choose the wrong logical counterpart.The passage attributes the mismatch to misinterpreting the graphical representation.Source: §G.2 · p. 60
  • The model failed to follow the instruction to symbolize propositions with 0-ary predicates, reflecting difficulty applying the stated symbolization rules.The task explicitly requests 0-ary predicates for the propositions.Source: §G.2 · pp. 61–62

G.3 CIPHER ERROR CASES ANALYSIS

  • The cipher response incorrectly placed R at position 1 and returned [[K]], an error attributed to inaccurate position mapping in a sequence.The supplied outer disk is “QWERTYUIOPASDFGHJZXCVBNMKL”; the model’s response identifies K as the encrypted character.Source: §G.3 · p. 64
  • The substitution-cipher answer decoded to “RHWPXNDXVIX,” but the analysis identifies a sequence-linkage error despite correct individual-letter decoding.The analysis suggests limitations in contextual-information maintenance, memory integration, or serialization strategies.Source: §G.3 · p. 66
  • The transpose response wrote the sequence sideways despite describing alternating down-and-up filling, indicating difficulty with spatial relationships or directional instructions.The analysis points to possible deficiencies in spatial reasoning or directional understanding.Source: §G.3 · p. 70

1. Plaintext: Z

  • The RSA-style encryption response gave “223” for plaintext Z, and the analysis attributes the error to possible weakness in modulo-power arithmetic.Source: §1 · p. 72
  • The multitap decryption mapped initial codes to D, Z, N, M, and P, then marked the remaining codes invalid against the supplied table.Source: §1 · p. 74
  • In the Multitap Code case, the model notices invalid codes and tries to self-correct, but makes wild guesses instead of finding the sequence-splitting error.The revised ciphertext still contains invalid codes.Source: §1 · p. 75

G.4 PUZZLE ERROR CASES ANALYSIS

  • In the word-search case, the model gives a wrong answer directly, without any analysis or reasoning process.The supplied answers list coordinates for the hidden vegetables, but the case analysis characterizes the response as wrong and lacking reasoning.Source: §G.4 · pp. 76–77
  • The anagram response claims “Plea” uses all five letters of “Apple,” although it uses only four and violates the all-letters requirement.The response incorrectly concludes that [[Plea]] satisfies the stated restriction.Source: §G.4 · p. 78
  • The TT + TT = LTC response accepts T=6 despite an inconsistent repeated-T assignment, outputting an unsatisfied answer instead of continuing the search.The analysis identifies failure to check that T has the same value on both sides of the equation.Source: §G.4 · p. 80
  • The weighted-grid response fails because it does not coordinate row and column constraints when populating the matrix.It starts with column constraints but does not sufficiently correlate them with row constraints.Source: §G.4 · p. 82
  • The puzzle response considered row constraints but did not verify or modify column constraints, despite mentioning them.Source: §G.4 · p. 84

G.5 COUNTERFACTUAL ERROR CASES ANALYSIS

  • Counterfactual errors included an answer matching real-life knowledge rather than the intended answer, and another based on wrong proof and misunderstood information.Source: §G.5 · pp. 86, 88

H ATTENTION FOCUS VISUALISATION CASES

  • The attention examples cover one correct and one incorrect answer each from Qwen2.5-7B-Instruct and Qwen2.5-72B-Instruct; darker shading marks attended rule text and underlining marks the manually selected needle.Source: H ATTENTION FOCUS VISUALISATION CASES · p. 89

H.1 CASES FOR QWEN2.5-7B-INSTRUCT

  • In the cipher case, attention favored T, Z, F, and O over G, causing the model to miss the correct answer, “FG”; in the logic case, attention overlapped the needle and the answer was correct.Source: §H.1 · p. 89

H.2 CASES FOR QWEN2.5-72B-INSTRUCT

  • In the puzzle case, the model may prioritize fewer steps or word similarity and skip valid intermediate transformations instead of maintaining the one-letter-change constraint.Source: §H.2 · p. 89
  • In the counterfactual case, focusing on “Ice Burst Stone” lets the model produce the correct answer.Source: §H.2 · p. 89
  • In the logic example, the model explains that the premise is true but the conclusion implies a largest natural number and is false, making the reasoning invalid.Source: §H.2 · pp. 92–93
  • For HEAD to TALE, the response gives a three-step path: HEAD–HEAL–TEAL–TALE.Source: §H.2 · p. 94
  • The prompt asks for the most reliable method for preserving food for long periods, with options of refrigeration, heating, Iceburst Stone, or burial in soil.Iceburst Stone is described as extremely cold and valuable for preservation without modern refrigeration.Source: §H.2 · pp. 95–96
  • The prompt requires the final answer to be a single uppercase option letter wrapped in double square brackets.Source: §H.2 · p. 96

I ABLATION STUDY ON DATASET SIZE

KOR-Bench’s model-score comparisons remain stable across reduced dataset sizes under both sampling strategies, with errors stabilizing at a subset of 20% and little change in score dispersion.

I ABLATION STUDY ON DATASET SIZE

  • At 20% of the full dataset, mean and standard deviation of errors stabilize around 2, while the Gini coefficient varies by at most approximately 0.02.The reported stability holds across both rule-based and category-based sampling strategies.Source: Figure 13 · p. 97, I ABLATION STUDY ON DATASET SIZE · p. 98
  • Rule-based sampling draws from 10 questions per rule to preserve dataset diversity, while category-based sampling draws randomly from 250 questions per category to reduce rule-difficulty effects.The two strategies correspond to the left and right columns of Figure 13, respectively.Source: Figure 13 · p. 97, I ABLATION STUDY ON DATASET SIZE · p. 98
  • The ablation tracks mean accuracy error, error standard deviation, and the Gini coefficient of model scores across subsets.The mean error compares subset accuracy with full-dataset accuracy; the standard deviation captures error variability, and Gini measures score dispersion.Source: I ABLATION STUDY ON DATASET SIZE · pp. 97–98

J CORRELATION ANALYSIS WITH OTHER BENCHMARKS

Across 21 models, KOR-Bench correlates most strongly with reasoning-focused benchmarks, particularly MMLU-Pro; the pattern aligns with its focus on reasoning over prior knowledge.

J CORRELATION ANALYSIS WITH OTHER BENCHMARKS

  • Figure 14 shows correlations between KOR-Bench and six benchmarks using 21 models of varying sizes.The models include GPT-4o, Claude-3.5-Sonnet, and the Qwen, Llama, Yi, Mistral, Phi, and Gemma series.Source: Figure 14 · p. 99, J CORRELATION ANALYSIS WITH OTHER… · p. 99
  • KOR-Bench has its highest correlation with MMLU-Pro and is more closely related to reasoning-focused benchmarks than the other benchmarks examined.The analysis compares score distributions across 21 models and includes MMLU, MMLU-Pro, GSM8K, MATH, HumanEval, and MBPP.Source: J CORRELATION ANALYSIS WITH OTHER… · p. 99, Figure 14 · p. 99
  • The correlation pattern aligns with KOR-Bench’s stated emphasis on logical reasoning while minimizing reliance on prior knowledge.MMLU-Pro places greater emphasis on reasoning than MMLU and includes computational problems requiring logical reasoning.Source: J CORRELATION ANALYSIS WITH OTHER… · p. 99

K ZERO-SHOT AND THREE-SHOT “ONLY QUESTIONS" EXPERIMENTS

The “only questions” experiments test whether models can infer rules without explicit instructions. Three-shot performance varies with task and example relatedness, while zero-shot models struggle.

K ZERO-SHOT AND THREE-SHOT “ONLY QUESTIONS" EXPERIMENTS

  • Without explicit rules, zero-shot models receive a problem alone, while three-shot models receive three examples and answers from which to infer a pattern.The experiments assess pattern recognition and extraction of abstract reasoning rules.Source: K ZERO-SHOT AND THREE-SHOT “ONLY… · p. 100

Result Analysis:

  • In zero-shot, models struggle because the questions and prior knowledge are insufficient; three-shot examples let models infer patterns and solve some problems.The associated tables report performance in zero-shot and three-shot “only questions” settings.Source: Result Analysis · p. 100, Table 14 · p. 100, Table 15 · p. 100
  • Three-shot success varies by task: connected examples help models uncover rules in Counterfactual, Logic, and Operation, whereas abstract, weakly related Cipher and Puzzle examples remain difficult.Source: Result Analysis · p. 100

Related papers

Loading 2410.06526v3…