PaperlayerView exact revision on arXiv

Source-linked AI summary

GNN-CB: A Graph Neural Network Competition Benchmark for Human and LLM Evaluation

Murad Hossen, Tasneem Selim, Gurur Gamgam, Tuga Yousif, Abderrahmane Kasmi, Ikram Aissiou, Mubaraq Onipede, Faran Taimoor Butt, Sanae Zrigui, Rosa Y. G. Paccotacya-Yanque, Ignatius Balayo, Ikram Elhouiti, Hadil Affes, Bijay Adhikari, Sargam Goyal, Muhammad Ibrahim Isah, Mohammad Idrees Bhat, Samuel Kangoni Matia, Peguy Kem-Meka Tiotsop Kadzue, Maha Trabelsi, Emmanuel Owusu, Vinit, Nour Majdoub, Tamiru Alemnew, Islem Rekik

arXiv:2610.05387v1cs.LGcs.AIcs.CLcs.SE

TL;DR

Existing benchmarks do not evaluate whether LLMs can autonomously solve realistic end-to-end GNN coding tasks. GNN-CB introduces 18 competitions and controlled human and LLM evaluation protocols; under the evaluated protocol, LLMs reach Human Top in only a minority of model-task pairs and vary across competitions.

  • Problem

    Existing coding benchmarks focus on general programming, while graph benchmarks evaluate GNN prediction rather than autonomous LLM performance on end-to-end GNN coding tasks.

    Source: §1 · pp. 2–3

  • Method

    GNN-CB comprises 18 competitions evaluated through a unified hidden-test pipeline, with controlled human evaluation and frozen-prompt LLM protocols using plan-then-code generation and bounded repair.

    Source: §1 · p. 3, §3.3 · p. 5

  • Results

    LLMs surpass or tie Human Top in only 19 of 162 model-task pairs and show substantial performance variance across competitions under the evaluated protocol.

    Source: §4.2 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    GNN-CB provides a competition-based testbed for evaluating human and LLM GNN coding and a practice resource for GNN implementation.

    Source: §1 · p. 3, §6 · p. 9

    The evaluation covers nine models across two execution modes, while each competition’s human evaluation involves 13–17 expert volunteers.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated strong performance on coding and reasoning benchmarks; however, their ability to solve graph-structured machine learning problems remains largely unexplored. In particular, no benchmark currently evaluates whether LLMs can autonomously solve end-to-end Graph Neural Network (GNN) coding tasks under realistic competition settings. To address this gap, this paper introduces GNN-CB, the first competition-based benchmark for evaluating both humans and LLMs on GNN coding tasks. GNN-CB consists of 18 curated competitions spanning node-, edge-, and graph-level prediction across diverse graph categories, domains, and difficulty tiers. All submissions are evaluated through a unified automated pipeline with hidden test sets and standardized scoring. Human participants solve tasks under controlled competition constraints, while LLMs are evaluated using a frozen zero-shot prompting protocol based on a plan-then-code paradigm with bounded execute-and-repair loops. The benchmark additionally supports both non-agent and autonomous agent-based evaluation within the same protocol. Under our evaluated protocol, LLMs rarely match Human Top performance and show less stable performance across competitions. No single model dominates: a few competitions are won by LLMs, yet humans still hold the top score on most tasks. We release GNN-CB as a living benchmark with automated evaluation infrastructure, dynamic leaderboards, and reproducible execution pipelines. Beyond benchmarking, GNN-CB provides a practice-oriented resource for studying GNN implementation across progressively diverse graph-learning tasks. The benchmark and evaluation framework are publicly available at https://basiralab.github.io/GNN-CB/.

1 Introduction

The introduction identifies a gap in evaluating LLMs on realistic, end-to-end GNN coding tasks. It presents GNN-CB and controlled human and LLM protocols for addressing that gap.

1 Introduction

  • Existing coding benchmarks overlook graph-learning workflows, while graph benchmarks assess GNN prediction rather than whether LLMs can implement GNN tasks.GNN problems involve topology, relational pipelines, message-passing architectures, and task-specific constraints.Source: §1 · pp. 2–3
  • GNN-CB introduces 18 human-curated competitions spanning node-, edge-, and graph-level prediction, graph categories, domains, and difficulty tiers.Competitions use standardized repositories, datasets, hidden test sets, and automated evaluation pipelines.Source: §1 · p. 3
  • The benchmark uses controlled human evaluation and frozen zero-shot LLM prompting with plan-then-code generation and bounded execute-and-repair loops.It supports both non-agent and autonomous agent-based LLM evaluation.Source: §1 · p. 3
  • GNN-CB is released as a living benchmark with reproducible pipelines and dynamic leaderboards, and also serves as a practice resource for GNN implementation.The authors describe its use for analyzing performance variability, task difficulty, and implementation challenges.Source: §1 · p. 3

2 Related Work

Related work covers code-generation and reasoning benchmarks alongside graph-learning benchmarks. GNN-CB connects these areas by evaluating human participants and LLMs on GNN coding tasks.

2 Related Work

  • Recent benchmarks have significantly advanced evaluation of LLM-based code generation and reasoning, covering functional correctness, competitive programming, programming languages, stricter testing, repository-level reasoning, data-science workflows, and instruction-following capabilities.HumanEval [Chen et al., 2021] and CodeContests [Li et al., 2022] introduced standardized evaluation for functional correctness and competitive programming; DS-1000 [Lai et al., 2023], SWE-bench, BigCodeBench [Zhuo et al., 2024], and LiveCodeBench [Jain et al., 2024] evaluate repository-level reasoning, data science workflows, and instruction-following capabilities.Source: §2 · p. 3
  • Graph benchmarks standardize GNN prediction and study structural generalization or long-range dependencies, but do not assess LLMs’ performance against humans in GNN competitions.The passage discusses OGB [Hu et al., 2020], Benchmarking GNNs, and LRGB [Dwivedi et al., 2022].Source: §2 · p. 3
  • GNN-CB bridges code-generation and graph-learning research through controlled, reproducible evaluation of humans and LLMs on GNN coding tasks.It is designed as a competition-based benchmark.Source: §2 · p. 3

3 Benchmark Construction

GNN-CB evaluates repository-level graph-learning workflows through 18 competitions and standardized human and LLM protocols. Its LLM protocol uses frozen prompts, plan-then-code generation, bounded repair, and both agent and non-agent execution modes.

3 Benchmark Construction

  • GNN-CB targets complete repository-level graph-learning pipelines, including data processing, graph construction, architecture design, optimization, and hidden-test prediction.The benchmark contains 18 competitions across prediction tasks, graph categories, and difficulty tiers.Source: §3 · p. 3, §3.1 · p. 4

3.1 Competition Design Protocol

  • Competitions span node classification, edge prediction, graph classification, and graph regression across seven graph categories and Easy, Medium, and Hard tiers.Table 1 summarizes competition categories, difficulty, dataset scales, feature dimensions, domains, and evaluation metrics.Source: §3.1 · p. 4, Table 1 · p. 4
  • Each competition has a dedicated repository and automated workflow for validating submissions, scoring, logging execution, and updating leaderboards.Submissions are evaluated on hidden test sets, with secure submission mechanisms varying by repository configuration.Source: §3.1 · pp. 4–5

3.2 Human Evaluation Protocol

  • Human participants solve competitions independently under standardized computational and submission constraints, with final predictions scored through the automated hidden-test pipeline.Participants use GNN models without external solutions; LLM assistance is restricted to low-level utility support.Source: §3.2 · p. 5

3.3 LLM Evaluation Protocol

  • The LLM protocol converts repository information into a frozen prompt, generates a plan and executable code, then runs and evaluates the solution on hidden tests.The same automated infrastructure evaluates human and LLM submissions.Source: §3.3 · p. 5

3.3.1 Frozen Prompt Instantiation

  • Repository-derived information fills predefined prompt slots, and the model returns a structured reasoning plan followed by executable code.Slots can contain the README, repository tree, dataset summary, submission specification, and dependency requirements.Source: §3.3.1 · p. 5

3.3.2 Plan-Then-Code Generation

  • The plan outlines pipeline choices such as feature engineering, architecture, optimization, validation, and submission generation before code implements the solution.The authors describe this plan-then-code strategy as intended to reduce failures such as invalid pipelines, hallucinated APIs, and malformed submissions.Source: §3.3.2 · p. 5

3.3.3 Execution and Repair Loop

  • Generated code runs under CPU-only constraints, and execution tracebacks can trigger a bounded repair loop of at most five attempts.Repair is used for failures including syntax errors, runtime exceptions, dependency issues, and invalid outputs.Source: §3.3.3 · p. 6, §3.3.5 · p. 6

4 Results and Analysis

The benchmark compares human participants and nine LLMs across 18 competitions using standardized metrics and hidden-test evaluation. Under the evaluated protocol, LLMs rarely match Human Top performance, while results vary across tasks.

4 Results and Analysis

  • The evaluation covers a human-participant track and nine LLMs under a frozen plan-then-code protocol, using the same hidden-test pipeline and task-specific metrics.Metrics are Macro-F1, accuracy, and AUC for classification, and MAE/NMAE for regression.Source: §4 · p. 7

4.1 Human Performance Results

  • Across 18 competitions, human performance varied with task difficulty, graph structure, and domain complexity: easy C03 achieved high scores with low variance, while harder C02, C04, and C18 showed lower performance and larger gaps between best and worst submissions.Source: §4.1 · p. 7

4.2 LLM Evaluation Results

  • 19/162 LLM–task pairs meet or exceed Human Top, and mean LLM performance exceeds mean human performance in only 6 of 16 competitions.Table 3 reports scores and gaps to Human Top across nine LLMs and 18 competitions.Source: Table 3 · p. 8, §4.2 · p. 8

4.1 Human Performance Results

  • Human results on regression tasks C14 and C17 show larger relative performance gaps and less stable convergence than most classification settings.The authors associate increasing difficulty with demands for advanced graph reasoning, domain-specific adaptation, and complex optimization.Source: §4.1 · p. 8

4.2 LLM Evaluation Results

  • Open-source models show larger average gaps and greater cross-task variability, but neither reasoning nor chat models have a consistent benchmark-wide advantage within the evaluated proprietary tier.LLM performance gaps are larger on Hard competitions than on Easy and Medium competitions.Source: §4.2 · p. 8
  • Among 20 severe classification underperformance cases, incorrect graph construction accounted for 8 and feature processing or utilization for 7.Indexing or data-reading errors accounted for 3 cases and unsuitable model selection for 2.Source: §4.2 · pp. 8–9

5 Discussion

Across the evaluated setting, LLM performance varies substantially across competitions, and gaps widen on harder tasks. The results show no consistent reasoning-versus-chat advantage and motivate task-aware alternatives to relying on one model.

5 Discussion

  • All evaluated models show substantial performance variance across the 18 competitions, indicating limited transfer of GNN construction and reasoning strategies across heterogeneous settings.Proprietary models have better average gaps than open-source counterparts, but reasoning and chat models show no meaningful within-tier difference.Source: §5 · p. 9
  • Performance gaps widen on Hard competitions, while reasoning capabilities do not confer a systematic advantage under the zero-shot, bounded-repair protocol.The authors suggest task-aware routing or ensembling may be more effective than scaling a single model.Source: §5 · p. 9
  • Competition-style evaluation provides a discriminative measure of practical GNN coding ability and highlights risks of deploying LLM-generated solutions without expert oversight.Source: §5 · p. 9

6 Conclusion

GNN-CB is a competition-based benchmark for evaluating humans and LLMs on GNN coding tasks through a unified hidden-test pipeline. Under its evaluated protocol, LLMs fall short of expert performance on most tasks and vary across heterogeneous problems.

6 Conclusion

  • GNN-CB comprises 18 human-designed competitions spanning graph categories, domains, and difficulty tiers, evaluated through a unified hidden-test pipeline.Source: §6 · p. 9
  • The benchmark applies controlled human evaluation and frozen-prompt LLM evaluation with bounded execute-and-repair iterations across API-only and agent-based settings.Source: §6 · p. 9
  • The public benchmark and dynamic leaderboards support future evaluation, comparison, and hands-on GNN coding practice.Source: §6 · p. 9

Limitations

The evaluation is bounded by the available models, compute, and human cohort, and the results are not estimates of each model’s maximum attainable performance. The benchmark also omits some graph-learning settings and has limited coverage of regression.

Limitations

  • API and compute constraints limit the LLM evaluation to nine models and two execution modes, while each competition has 13–17 expert volunteers.The authors characterize results as a broad, reproducible evaluation under bounded-repair and agent-based protocols, not estimates of maximum model performance.Source: Limitations · p. 9
  • The current release omits graph generation, quantum graph modelling, and self-supervised or reinforcement-learning paradigms.Task-specific metrics also limit direct cross-task comparison, and two regression competitions do not support strong general conclusions about graph-regression performance.Source: Limitations · p. 9
  • Reported results are snapshots of models available as of May 2026, although the authors expect the observed qualitative patterns to remain informative as models advance.Source: Limitations · p. 10

Future Directions for Extended LLM Evaluation

The authors outline extensions to GNN-CB’s frozen zero-shot evaluation, including broader modelling analyses and more rigorous comparisons. They also propose expanding the benchmark’s tasks and tracking capability and consistency over time.

Future Directions for Extended LLM Evaluation

  • Future modelling studies could test few-shot prompting, retrieval-augmented generation, and task-specific fine-tuning, and measure variability with repeated sampling, confidence intervals, and pass@k.The current study reports a single greedy run under a zero-shot, frozen-prompt protocol.Source: Future Directions for Extended LLM… · p. 10
  • Submission validation could support an error taxonomy covering graph construction, message passing, data leakage, class imbalance, and malformed submissions.The proposed analysis would examine where and why LLMs break down beyond aggregate scores.Source: Future Directions for Extended LLM… · p. 10
  • Execution traces could enable token-cost and latency analysis across API-only and agent-based runs, including tool-call counts and repair-loop depth.The authors also propose paired significance tests, bootstrap confidence intervals, and rank-correlation analysis of human–LLM ordering.Source: Future Directions for Extended LLM… · p. 10
  • GNN-CB could expand to graph generation, self-supervised, temporal, and reinforcement-learning tasks, and track whether the capability–consistency gap narrows as models are released.The authors propose longitudinal tracking using the benchmark’s automated infrastructure and dynamic leaderboards.Source: Future Directions for Extended LLM… · p. 10

Ethics Statement

The paper reports participant consent and safeguards concerning participant roles, dataset privacy, licensing, and AI-assistant use. It also cautions against deploying LLM-generated GNN code in safety-critical domains without expert verification.

Ethics Statement

  • All human participants consented to publication of their results; only aggregate counts and per-task scores are reported, with no identifying participant information released.Participants were expert volunteers who co-designed the benchmark and received no monetary compensation.Source: Ethics Statement · p. 10
  • No coauthor competed in a competition they designed, so each competition’s reported human submissions came from participants without a design or curation role in that task.Each competition was designed by one coauthor, who solved a subset of the other 17 competitions.Source: Ethics Statement · p. 10
  • Datasets were used in their released forms; sensitive biomedical data followed their original de-identification protocols and data-use agreements, and only aggregate scores were reported.The GitHub usernames were public and not prediction targets; Elliptic identifiers were pseudo-anonymous, and released datasets contained no free-form user-generated text.Source: Ethics Statement · p. 10
  • Datasets were used under their original public licenses, LLMs under providers’ API terms, and GNN-CB’s repositories, pipeline, and leaderboard are to be released under MIT.Source: Ethics Statement · p. 10
  • Without expert verification, deploying LLM-generated GNN code in safety-critical domains could amplify model failures or propagate dataset biases.The paper releases the benchmark for controlled research and practice and discourages direct deployment without expert review.Source: Ethics Statement · p. 10
  • AI assistants supported editing and coding during paper preparation, but were not used to generate experimental results, benchmark code, or scientific claims.The paper distinguishes these assistants from the LLMs evaluated in the benchmark.Source: Ethics Statement · p. 11

A Competition Specifications

This appendix specifies diverse graph-prediction competitions, their data and evaluation conditions, and the controlled implementation protocol for human participants, using a unified hidden-test evaluation pipeline.

A Competition Specifications

  • Competitions share standardized task, dataset, metric, and submission descriptions, with hidden-test scoring; many also perturb data, alter splits, or convert non-graph data to limit transfer of memorized solutions.Examples include feature noise or permutation, class imbalance and distribution shifts, and similarity-based graph construction; the modifications vary by dataset and application.Source: A Competition Specifications · p. 13

C01: PROVEN-GNN; C03: Real Or Fake!

  • PROVEN-GNN detects vulnerable functions in heterogeneous code graphs, while CGCC classifies urban street-network graphs as organic, grid, or hybrid.Real or Fake! predicts news veracity from hierarchical propagation graphs using article and user nodes, interaction edges, and textual and profile features.Source: C01: PROVEN-GNN · p. 14, C02:CGCC · p. 14, C03: Real Or Fake! · pp. 14–15

C04: GLIMPS-GNN; C05: GNN_BACE

  • GLIMPS-GNN tests whether node classifiers generalize from maternal-plasma cfRNA to unseen placental RNA-seq, while GNN_BACE predicts molecular BACE-1 inhibition.GNN_BACE supplies atom features and connectivity without precomputed molecular descriptors.Source: C04: GLIMPS-GNN · p. 15, C05: GNN_BACE · p. 15

C06: DiaGraph; C07: GRAPE

  • DiaGraph predicts diabetes from patient nodes and similarity edges, whereas GRAPE classifies diabetic retinopathy from spatial retinal-vessel graphs.DiaGraph is transductive node classification; GRAPE is graph-level classification using retinal vessel connectivity.Source: C06: DiaGraph · p. 15, C07: GRAPE · p. 16

C08: Liar Nodes; C09: GraphFMD

  • Liar Nodes predicts cancer status from cell-similarity graphs with intentionally corrupted embeddings, while GraphFMD classifies Bitcoin transactions in a temporal graph.GraphFMD adds Gaussian noise to transaction features and evaluates on later time steps, with illicit transactions comprising only 1–3% of nodes.Source: C08: Liar Nodes · p. 16, C09: GraphFMD · p. 16

C10: NetLinkArena; C11: PARK-GNN

  • NetLinkArena predicts citation links in a perturbed scientific-paper graph, while PARK-GNN classifies Parkinson’s disease from voice-recording graphs built using acoustic similarity and subject links.PARK-GNN derives its graph from acoustic recordings; NetLinkArena adds perturbations and synthetic noise to CiteSeer.Source: C10: NetLinkArena · p. 17, C11: PARK-GNN · p. 17

C12: GNN-CoRA

  • GNN-CoRA classifies publication nodes into seven research topics using text features and citation structure, with Gaussian noise added to node features.Its standard citation-network split has 140 training, 500 validation, and 1,000 hidden test nodes.Source: C12: GNN-CoRA · pp. 17–18

C13 : GTA

  • GTA evaluates molecular graph classification on clean and feature-perturbed test graphs, reporting macro F1 separately for each setting and a robustness gap.Training uses clean features; perturbations combine a constant feature offset and Gaussian noise, and smaller robustness gaps indicate more stable performance.Source: C13 : GTA · p. 18
  • GTA submissions are securely scored on hidden clean and perturbed settings, with leaderboard ranking led by perturbed macro-F1 and then robustness gap.Encrypted predictions are decrypted by an automated pipeline that computes metrics and updates the public leaderboard.Source: C13 : GTA · p. 18

C14: BrainAge

  • BrainAge predicts age from 68-region structural brain graphs, using subject-level age-stratified splits to prevent longitudinal leakage and hidden-test MAE for evaluation.The dataset comprises 431 MRI sessions from 288 cognitively normal subjects; node features are cortical thickness and gray matter volume, and edge weights are streamline counts.Source: C14: BrainAge · pp. 18–19

C15: OctoNode Cup

  • OctoNode Cup classifies GitHub users as Web or Machine Learning Developers on a network of about 37,700 nodes and 289,000 edges, scored by hidden-test Macro F1.The transductive task provides labels for 70% of nodes and features only for the remaining 30%; submissions are encrypted and automatically scored.Source: C15: OctoNode Cup · p. 19
  • Graph4ASD classifies subjects from 200-node rs-fMRI graphs; identity-only features and non-identical site distributions make learning site-invariant connectivity representations necessary.The challenge-specific ABIDE subset has 484 training and 153 evaluation graphs, and hidden-test performance is measured by macro F1.Source: C16: Graph4ASD Challenge · pp. 19–20
  • Mix2Print predicts pressure, temperature, and speed from fully connected bioink formulation graphs, with temperature-regime stratification and NMAE across targets.The dataset contains 423 formulations split into 303 training and 120 test samples; node features encode material identity and concentration.Source: C17: Mix2Print · p. 20

C18: THInC

  • THInC requires inductive four-class cell classification on unseen patient graphs, using image-level 80/20 splits and accuracy on encrypted hidden-test submissions.Graphs are built from histology nuclei joined to five nearest spatial neighbors; features include normalized coordinates and standardized nucleus morphology.Source: C18: THInC · p. 20

B.1 Human Implementations

  • Human submissions follow a reproducibility protocol with three CPU hours and five tuning runs per task, fixed seed 25, GNN-only solutions, and one final submission.Participants use only training and validation data, retain source code and run records, and document search configurations and permitted low-level LLM assistance for audit.Source: §B.1 · p. 21

B.2 Benchmark Composition Attempted by Human Participants

The 18 competitions span varied graph settings, difficulty tiers, task types, and application domains, and were co-designed by universities across 12 countries. The human evaluation pipeline runs from repository fork and encrypted submission to automated scoring against hidden test labels.

B.2 Benchmark Composition Attempted by Human Participants

  • Node and graph classification make up 83.3% of the 18 competitions, which are skewed toward Medium (44.4%) and Hard (38.9%) difficulty tiers.The benchmark is dominated by homogeneous graphs (55.6%) and spans ten application domains; it also includes one edge-prediction and two graph-regression competitions.Source: §B.2 · p. 21

B.3 Geographic Distribution of Participants

  • The 18 participating universities span 12 countries across Africa, the Americas, Asia, and Europe.Countries are annotated with the competitions co-designed by their teams.Source: §B.3 · p. 21

B.4 Human Evaluation Pipeline

  • The human workflow runs from repository fork through encrypted pull-request submission to automated GitHub Actions scoring against hidden test labels.The maintained documentation log records hyperparameter search, runtime, hardware, loss-curve verification, and LLM-assisted utility usage.Source: §B.4 · p. 21

C LLM Evaluation Details

The LLM evaluation framework uses a frozen prompt populated from competition repositories and a plan-then-code format. It supports API-only and agent-based modes, while keeping hidden test labels inaccessible and bounding repair feedback to local execution tracebacks.

C LLM Evaluation Details

  • LLMs never access hidden test labels, and their only pre-submission feedback is local execution tracebacks from bounded execute-and-repair loops.The framework supports API-only evaluation and autonomous agents, with agent sessions governed by their native budgets.Source: C LLM Evaluation Details · p. 21, §C.4 · p. 23
  • The appendix describes frozen prompt construction, a representative plan-then-code example, the prompt template, and API-based orchestration.Table 4 groups evaluated models by access type and functional category and lists their engine identifiers.Source: C LLM Evaluation Details · pp. 21–22
  • Figure 5 summarizes the 18 competitions by graph category, difficulty, task type, and application domain.Its legends list per-slice counts and competition codes.Source: Figure 5 · p. 22

C.2 Plan-Then-Code Example

  • The plan-then-code example for C03 (Real or Fake!) pairs a high-level reasoning plan with executable code.The plan typically covers feature engineering, GNN architecture, optimization, validation, and submission generation.Source: §C.2 · p. 22

C.3 Frozen Prompt Template

  • A shared frozen prompt template is automatically populated from each repository, varying slot contents while preserving the same prompt string across all 18 competitions.Slots include the README, repository tree, dataset summary, submission specification, and allowed libraries.Source: §C.3 · pp. 22–23, §C.1 · p. 22

C.5 Performance Analysis of LLMs Across Competitions

Across difficulty levels, LLMs remain substantially behind human participants, with performance gaps and instability generally worsening on harder classification tasks. The analysis compares human-relative gaps, average scores, model categories, and severe underperformance patterns.

C.5 Performance Analysis of LLMs Across Competitions

  • On Easy and Medium tasks, proprietary models generally show smaller human-relative gaps and more consistent behavior than open-source models.Open-source models show larger negative gaps and higher variability on Medium tasks; GPT 5.4 Codex is an exception to the Easy-task proprietary-model trend.Source: §C.5 · p. 24, Figure 11 · p. 26, Figure 10 · p. 26
  • On Hard tasks, the performance gap widens for nearly all models, and the gap to human participants remains large.Gemini 2.5 Pro drops considerably, while Kimi k2.6 reaches comparable levels on several tasks; no substantial performance difference is observed between chat and reasoning models across difficulty levels.Source: §C.5 · p. 25, Figure 12 · p. 27
  • Humans achieve higher average scores in 10 of 16 classification competitions, while LLM performance also declines from Easy to Hard.Both human and LLM performance decline with increasing difficulty, with larger drops on harder tasks.Source: §C.5 · pp. 25–26, Figure 13 · p. 27
  • Mean scores decline across nearly all models and best–worst variance widens on Medium and Hard tasks.On Easy tasks, proprietary models perform highly with smaller best–worst variability, while open-source models remain competitive but vary more.Source: §C.5 · p. 26, Figure 14 · p. 27, Figure 15 · p. 28
  • Open-source models produced 17 of 20 severe-underperformance cases (85%), but the differences are descriptive given the small, unbalanced model groups.Severe failures occurred in 17/64 (26.6%) open-source pairs and 3/80 (3.8%) proprietary pairs; only one open-source reasoning model was evaluated.Source: §C.5 · p. 28
Loading 2610.05387v1…