Source-linked AI summary
General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks
Junlin Liu, Shengnan An, Shuang Zhou, Dan Ma, Shixiong Luo, Ying Xie, Yuan Zhang, Wenling Yuan, Yifan Zhou, Xiaoyu Li, Ziwen Wang, Xuezhi Cao, Xunliang Cai
TL;DR
General reasoning in broad, everyday contexts remains insufficiently evaluated because existing evidence emphasizes specialized domains and some general benchmarks lack diversity or difficulty. General365 addresses this gap with a K-12-bounded benchmark of diverse, challenging problems and evaluates 26 leading LLMs. The top-performing model achieves only 62.8% accuracy, while semantic analysis and fine-grained evaluation identify diversity and persistent reasoning bottlenecks.
Problem
Existing evaluations emphasize domain-specific reasoning, while general benchmarks may lack task diversity or sufficient difficulty for distinguishing models.
Method
General365 constructs a diverse benchmark from human-curated seed problems and model-based augmentation, with K-12 knowledge requirements and rigorous quality audits.
Results
62.8% accuracy was achieved by the state-of-the-art model across evaluations of 26 leading LLMs on General365.
Takeaways & Limitations
Current LLMs still struggle significantly with general reasoning, with Semantic Interference and Optimal Strategy identified as primary performance bottlenecks.
Abstract
from arXiv · showhide
Contemporary large language models (LLMs) have demonstrated remarkable reasoning capabilities, particularly in specialized domains like mathematics and physics. However, their ability to generalize these reasoning skills to more general and broader contexts--often termed general reasoning--remains under-explored. Unlike domain-specific reasoning, general reasoning relies less on expert knowledge but still presents formidable reasoning challenges, such as complex constraints, nested logical branches, and semantic interference. To address this gap, we introduce General365, a benchmark specifically designed to assess general reasoning in LLMs. By restricting background knowledge to a K-12 level, General365 explicitly decouples reasoning from specialized expertise. The benchmark comprises 365 seed problems and 1,095 variant problems across eight categories, ensuring both high difficulty and diversity. Evaluations across 26 leading LLMs reveal that even the top-performing model achieves only 62.8% accuracy, in stark contrast to the near-perfect performances of LLMs in math and physics benchmarks. These results suggest that the reasoning abilities of current LLMs are heavily domain-dependent, leaving significant room for improvement in broader applications. We envision General365 as a catalyst for advancing LLM reasoning beyond domain-specific tasks toward robust, general-purpose real-world scenarios. Code, Dataset, and Leaderboard: https://general365.github.io
1 Introduction
General365 addresses the limited evaluation of general reasoning in LLMs, where everyday logic may not transfer from domain-specific reasoning benchmarks. It introduces a diverse, challenging benchmark designed to measure such reasoning within K-12 knowledge constraints.
- General365 targets general reasoning in broad, real-world contexts rather than reasoning dependent on deep domain-specific knowledge.The benchmark is motivated by concerns that strong performance on specialized competitions may not translate to everyday reasoning.
- 365 manually crafted seed problems expand into 1,095 variants spanning eight challenging categories.Variants alter surface semantics or constraints while preserving core reasoning skills.
- The benchmark restricts required knowledge to the K-12 level so evaluation focuses on reasoning rather than knowledge retrieval.
- Across 26 models, the top-performing model reached only 62.8% accuracy, indicating substantial room for improvement on General365 tasks.The benchmark is intended to support progress toward robust, general-purpose reasoning in real-world scenarios.
- General365’s diversity was validated through semantic-space visualization and lower logical overlap than BBH and BBEH.The analyses assess both semantic dispersion and similarity of reasoning paths between cases.
2 GENERAL365
General365 combines a taxonomy of eight reasoning challenges with a multi-stage construction and grading pipeline. Its dataset emphasizes diversity, difficulty, quality control, and reliable evaluation.
- Challenge taxonomy: The benchmark defines eight challenge categories to capture diverse structural and cognitive demands in general reasoning.Examples include complex constraints and branching and enumeration.
- Dataset construction: Seed problems are manually verified, original, structured with reasoning traces and ground-truth answers, and inspected for quality.
- Dataset construction: Difficulty filtering removes problems easily solved by top-tier models or highly similar to existing benchmarks before diversity augmentation.
- Dataset construction: LLM-generated expansions undergo expert review, difficulty estimation, and diversity auditing to produce the final benchmark.The expansion process generates 10 candidates per seed, retaining valid candidates and supplementing seeds with fewer than three valid expansions.
- Dataset statistics: Complex Constraints is the most represented category, while Probability & Uncertainty is the least represented but still contains over 20 seed problems.
- Grading: A hybrid rule-based and model-based grading framework achieves 99.6% manually verified grading accuracy.The evaluation combines computationally efficient structured checking with flexible model-based assessment.
3 Experiments and Analysis
GENERAL365 evaluates leading LLMs on challenging general-reasoning tasks and reveals substantial performance gaps, while analyses examine efficiency, diversity, and difficulty. The benchmark presents broader and more demanding reasoning challenges than existing comparisons.
- Main Results: 26 leading LLMs were evaluated on GENERAL365, and Gemini-3-Pro achieved the highest accuracy at 62.8%.Most evaluated models failed to reach the basic passing threshold.
- Main Results: Gemini-3-Flash and GPT-5-Thinking followed with accuracies of 60.8% and 58.6%, respectively.
- Reasoning Efficiency: Gemini-3-Pro achieved 62.8% accuracy with approximately 14k average output tokens, compared with roughly 25k–30k tokens for some high-accuracy reasoning models.The analysis describes this as superior reasoning efficiency.
- Diversity Validation: GENERAL365 has a more uniform and expansive query-embedding distribution than BBH and BBEH, whose instances show local clustering.The embedding analysis uses text-embedding-ada-002 followed by t-SNE.
- Diversity Validation: GENERAL365 has a mean reasoning-similarity score of 2.16, whereas BBH and BBEH average 4.71 and 4.80, respectively.Approximately 68.2% of GENERAL365 instances score between 1 and 2, while most BBH and BBEH samples receive the maximum score of 5.
- Difficulty Analysis: Overall accuracy is substantially lower on GENERAL365 than on BBH and BBEH across evaluated models, indicating a higher difficulty floor.GENERAL365 also produces longer outputs despite lower accuracy, reflecting more demanding reasoning requirements.
4 Related Work
Related work has primarily assessed LLM reasoning through specialized domains or through broader-domain benchmarks. These efforts motivate evaluating reasoning driven by logical structure rather than expert knowledge.
- Domain-Specific Reasoning: Mathematics, physics, programming, and related benchmarks assess reasoning using specialized subject knowledge and challenging domain-specific problems.
- General Reasoning: A newer line of work evaluates reasoning in broader everyday domains, where difficulty stems from intricate logical structures rather than domain-specific knowledge retrieval.
5 Conclusion
The paper introduces GENERAL365 as a diverse, challenging benchmark for general reasoning in broad real-world domains. Evaluations show that current LLMs still struggle, with semantic interference and optimal strategy identified as primary bottlenecks.
- Benchmark Contribution: GENERAL365 combines human-curated seed problems with model-based augmentation and rigorous quality audits to create a structurally diverse benchmark.
- Findings: The state-of-the-art model achieved only 62.8% accuracy, indicating substantial remaining difficulty in general reasoning.
- Findings: Fine-grained analysis identifies Semantic Interference and Optimal Strategy as the primary performance bottlenecks for current architectures.
A Representative Examples of Challenge Categories
The examples illustrate diverse General365 challenge categories through tasks requiring recursive search, semantic transformation, implicit inference, optimization, probability, and constrained enumeration.
- Probability & Uncertainty Problem: The coffee-shop puzzle combines seating and coffee-order constraints to count possible arrangements.The stated number of arrangements is 8.
- Branching & Enumeration Problem: The milk-distribution puzzle asks how many valid schemes include milk in Box A under occupancy and sum constraints.The stated answer is 16.
- Recursive & Backtracking Problem: The tunnel problem asks for the minimum checks needed to catch a wounded soldier moving between adjacent tunnels.Its stated answer is 14 checks.
- Semantic Interference Problem: The parallel-universe calendar problem requires reasoning through reversed calendar conventions and policy-based changes across dates.For March 2000 and February 2105, it asks for 10a + b and gives 327.
- Implicit Information Reasoning Problem: The country-operation puzzle requires inferring an implicit rule from population and area rankings before computing the Argentina–Kazakhstan result.The stated answer is 72.
- Optimal Strategy Problem: The fire-truck problem asks for the shortest travel time under speed limits, traffic-light timing, and a guarantee of encountering at least one full red light.The stated shortest time is 7.5 minutes.
- Probability & Uncertainty Problem: The memorial-service puzzle asks the reader to identify the most likely suspect from the described events and available choices.The stated answer is Detective D.
B Prompt Templates
The benchmark uses structured prompts to standardize answer generation, pairwise task-similarity assessment, and final-answer equivalence grading.
- Pairwise Similarity Scoring: The pairwise-similarity prompt compares tasks by underlying logic, knowledge systems, cognitive challenge types, and reasoning trajectories on a 0–5 scale.It analyzes the problem, chain-of-thought reasoning, and final result.
- Query Prompt Templates: The query templates require numerical, select-answer, and text-answer problems to place a standardized final answer at the end of the response.Numerical answers use a boxed format, while select and text answers use designated final-answer markers.
- Final Answer Grading: The grading prompt asks whether a student answer is equivalent to a reference answer and returns either Correct or Incorrect.Decimal approximations must be accurate to at least four decimal places.
C Performance on the Public Subset of GENERAL365
The publicly accessible split is designed to support reproducible community research while retaining a substantial portion of the benchmark’s seed and variant problems.
- Public Subset: The public split randomly samples 180 seed problems and their extended variants, producing 720 evaluation instances.The section reports model performance on this open-access subset in Table 2.
D Robustness to Sampling Variance
The evaluation examines whether high-temperature sampling introduces unstable results across repeated runs and reports consistently small performance deviations.
- Repeated Evaluation: Four independent evaluation runs show maximum accuracy deviations below 3% across representative model architectures.The authors interpret this as evidence that the evaluation metrics remain reliable and reproducible despite high-temperature inference.
- Sampling Strategy: High sampling temperature is used to encourage models to explore diverse reasoning pathways on General365’s difficult tasks.The authors note that this setting inevitably introduces stochasticity into final evaluation outcomes.