PaperlayerView exact revision on arXiv

Source-linked AI summary

AcademiClaw: When Students Set Challenges for AI Agents

Junjie Yu, Pengrui Lu, Weiye Si, Hongliang Lu, Jiabao Wu, Kaiwen Tao, Kun Wang, Lingyu Yang, Qiran Zhang, Xiuting Guo, Xuanyu Wang, Yang Wang, Yanjie Wang, Yi Yang, Zijian Hu, Ziyi Yang, Zonghan Zhou, Binghao Qiang, Borui Zhang, Chenning Li, Enchang Zhang, Feifan Chen, Feng Jian, Fengyin Sun, Hao Qiu, Hao Zheng, Haoran Zhu, Hongyu Liu, Jianbin Deng, Jiaxin Song, Jiaying Chi, Jiayou Shi, Jie Fang, Jinghui Zhong, Jingyu Zhou, Jinze Li, Junfeng Yi, Junyan Yu, Junzhi Xue, Ni Song, Pengyi Chen, Qi Chen, Quansheng Li, Rui Tao, Shenghai Gong, Shenhang Lu, Tianqi Shen, Tianxiang Zhu, Tiehan Kang, Tingyu Li, Wendi Wu, Xiao Shen, Xiao Zhou, Xiaotao Zhang, Xinrong Li, Xuankun Yang, Xun Zhang, Yan Li, Ye Lu, Yi Wang, Yibo Zhou, Yichi Zhang, Yihao Sun, Yijun Huang, Yixin Zhu, Yixuan Wu, Yuchen Sun, Yue Wu, Yuheng Sun, Yukun Li, Yutian Tu, Yuxuan Qin, Yuzhuo Wu, Zeyu Li, Zhengyu Lou, Zhenning Ran, Zizhu He, Pengfei Liu

arXiv:2605.02661v1cs.AIcs.CY

TL;DR

OpenClaw benchmarks had largely evaluated assistant-level tasks, leaving academic-level capabilities insufficiently examined. AcademiClaw addresses this gap with an 80-task, student-sourced benchmark and multidimensional evaluation framework; across six frontier models, the best achieved only a 55% pass rate. The benchmark exposes capability boundaries, behavioral differences, and a weak relationship between token use and quality.

  • Problem

    Existing OpenClaw benchmarks focused on assistant-level scenarios, leaving complex academic and domain-intensive capabilities unexamined.

  • Method

    AcademiClaw constructs 80 bilingual tasks from students’ real workflows, curates them from 230 candidates, and evaluates them in standardized isolated environments with multidimensional rubrics.

  • Results

    The best of six frontier models achieved a 55% pass rate, while analyses found capability boundaries, divergent behavioral phenotypes, and near-zero token–quality correlation.

  • Takeaways & Limitations

    AcademiClaw provides fine-grained diagnostic signals for advancing OpenClaw agents beyond assistant-oriented tasks toward broader academic and professional demands.

  • Takeaways & Limitations

    The task set comes from CS undergraduates at one university, uses single-attempt evaluation, and covers only six frontier models.

Abstract

from arXiv · show

Benchmarks within the OpenClaw ecosystem have thus far evaluated exclusively assistant-level tasks, leaving the academic-level capabilities of OpenClaw largely unexamined. We introduce AcademiClaw, a bilingual benchmark of 80 complex, long-horizon tasks sourced directly from university students' real academic workflows -- homework, research projects, competitions, and personal projects -- that they found current AI agents unable to solve effectively. Curated from 230 student-submitted candidates through rigorous expert review, the final task set spans 25+ professional domains, ranging from olympiad-level mathematics and linguistics problems to GPU-intensive reinforcement learning and full-stack system debugging, with 16 tasks requiring CUDA GPU execution. Each task executes in an isolated Docker sandbox and is scored on task completion by multi-dimensional rubrics combining six complementary techniques, with an independent five-category safety audit providing additional behavioral analysis. Experiments on six frontier models show that even the best achieves only a 55\% pass rate. Further analysis uncovers sharp capability boundaries across task domains, divergent behavioral strategies among models, and a disconnect between token consumption and output quality, providing fine-grained diagnostic signals beyond what aggregate metrics reveal. We hope that AcademiClaw and its open-sourced data and code can serve as a useful resource for the OpenClaw community, driving progress toward agents that are more capable and versatile across the full breadth of real-world academic demands. All data and code are available at https://github.com/GAIR-NLP/AcademiClaw.

1 Introduction

AcademiClaw addresses a gap in OpenClaw evaluation by targeting complex academic work rather than assistant-level routines. Its student-sourced benchmark and model experiments reveal substantial limits and fine-grained behavioral differences.

  • Motivation: Existing OpenClaw benchmarks focused on assistant-level routines, leaving complex academic and professional work systematically unexamined.Examples include email triage, calendar management, project scaffolding, mathematical proofs, GPU-intensive training, and domain-specific analysis.
  • Benchmark contribution: AcademiClaw introduces 80 bilingual, long-horizon tasks sourced from students’ real academic workflows across 25+ domains.The tasks range from olympiad mathematics and linguistics to reinforcement learning and literary knowledge extraction.
  • Evaluation findings: 55% pass rate was achieved by the best model across the benchmark, with passing defined as a score of at least 75 out of 100.Six frontier models were evaluated under identical OpenClaw conditions.
  • Evaluation findings: Over 22% of tasks showed model-to-model score swings of up to 90 points, while olympiad-level problems remained universally unsolved.Agents performed better on generative tasks than on formal reasoning tasks.
  • Behavioral analysis: Token consumption varied by over 5× across models but had near-zero correlation with quality, with r = −0.03.The analysis also identifies read-first, execute-first, and minimalist behavioral phenotypes.

2. Related Work

AcademiClaw extends existing agent benchmarks by evaluating end-user-sourced academic tasks, GPU-intensive workloads, and richer safety and capability dimensions. Its contribution is a diagnostic framework for studying how agents fail and behave across demanding domains.

  • Benchmark landscape: AcademiClaw is the only compared OpenClaw benchmark sourcing tasks from end users, targeting academic-level difficulty, and including GPU-intensive tasks.The comparison distinguishes AcademiClaw from assistant-level and CPU-only benchmark suites.
  • Diagnostic evaluation: AcademiClaw combines six scoring techniques, five-category safety auditing, and trajectory analysis to provide fine-grained diagnostic signals.These mechanisms go beyond aggregate task success by examining capability, safety, and behavior.
  • Related benchmarks: The benchmark comparison positions AcademiClaw within a broader ecosystem that includes code, interactive-environment, and assistant-workflow evaluations.Related benchmarks include SWE-bench, AgentBench, MLE-bench, PinchBench, and Claw-Eval.

3 The AcademiClaw Benchmark

AcademiClaw is a student-sourced benchmark of 80 long-horizon tasks spanning 25+ domains, including bilingual and CUDA workloads. Isolated execution, multidimensional scoring, safety auditing, and trajectory logging support detailed capability diagnosis.

  • Task collection: AcademiClaw contains 80 complex, long-horizon tasks from university students spanning 25+ professional domains.Tasks are packaged with prompts, reference materials, rubrics, and expected deliverables.
  • Task collection: Students contributed problems from real academic workflows, and 230 candidates underwent expert review before final selection.Review examined prompt clarity, rubric correctness, reproducibility, and domain coverage, among other criteria.
  • Task coverage: The final benchmark spans six categories and 25+ domains, with deliberate domain-balance enforcement during curation.The taxonomy is summarized in Table 2.
  • Task coverage: 16 of 80 tasks require CUDA GPU execution, covering machine-learning, computer vision, robotic simulation, and scientific computing.These tasks require environment configuration, GPU-memory management, custom training loops, and device-level debugging.
  • Bilingual coverage: The benchmark includes 49 English and 31 Chinese tasks, with Chinese tasks requiring culturally grounded linguistic competence.Examples include adapting Tang poetry, detecting Shuangpin errors, and scoring Chinese essays.
  • Execution environment: Tasks run in isolated Docker environments with CPU or GPU base images and task-specific dependency layers.A heuristic classifier routes tasks to the appropriate base environment.
  • Evaluation: Each task uses a 3–6-dimensional rubric totaling 100 points and combines six complementary verification techniques.Scoring methods include pattern matching, code execution, LLM judging, vision assessment, browser testing, and structure validation.
  • Evaluation: A five-category safety audit and complete trajectory logging supplement task scores with behavioral information.Logs include tool calls, token consumption, latency, and returned results.

4. Experiments

Experiments show substantial variation in model quality, efficiency, safety, and tool-use behavior under a standardized single-attempt evaluation. Task category often matters more than model identity, while behavioral strategies expose mismatches between token consumption, quality, and safety.

  • Overall Results: 55.0% pass rate is the top result, shared by Claude Opus 4.6 and Claude Sonnet 4.6, while Opus records the highest average score at 71.9.The weakest model averages 63.1, only 8.8 points below Opus; 23 of 80 tasks defeat all six models.
  • Domain and Task-Level Patterns: 26.3 points separate the highest and lowest category means, exceeding the 8.8-point spread between the best and weakest models.STEM Reasoning averages 50.6, while Language & Creativity averages 76.9.
  • Domain and Task-Level Patterns: Competition-level tasks produce near-uniform failures: chemistry scores cluster at 23–27, and full-stack debugging scores exactly 25 for every model.These near-zero-variance outcomes indicate systematic capability gaps rather than stochastic errors.
  • Domain and Task-Level Patterns: Model rankings are category-dependent: GPT-5.4 leads Language & Creativity at 83.7, while Claude Sonnet leads ML & AI Engineering at 74.1.GPT-5.4 spans 83.7 in Language versus 49.4 in Applied, a 34.3-point intra-model gap.
  • Behavioral Strategies: Claude Opus uses a read-first strategy, allocating 41% of tool calls to reading and achieving the highest average score at 1,425K tokens per task.Gemini instead allocates 74.3% of tool calls to execution and consumes 2,857K tokens while scoring 64.3.
  • Behavioral Strategies: Token consumption is effectively uncorrelated with task score across evaluations (r = −0.03, p = 0.49), and pairwise model-score correlations range from 0.275 to 0.729.The results indicate distinct capability profiles and no consistent positive quality return from additional token expenditure.
  • Safety Evaluation: S3 boundary compliance varies by 53 points, with Claude models at 83–85 and Gemini at 31.6 amid 217 HIGH-severity violations.The violations predominantly involve file access outside the designated workspace directory.

5. Conclusion

AcademiClaw exposes safety patterns that aggregate task scores alone miss: boundary compliance drives most inter-model variation, while privilege escalation is broadly controlled and safety remains largely independent of task quality.

  • 5. Conclusion: S4 privilege escalation is the most uniformly safe dimension, with all models scoring 90–98.No model attempts sudo or system-level modifications in more than 2% of tasks.
  • 5. Conclusion: Safety and task quality are largely independent across the six evaluated models.The Pearson correlation has |r| < 0.29, with GPT-5.4 as a marginal exception at r = 0.28, p = 0.01.
  • 5. Conclusion: AcademiClaw provides an academic-level testbed spanning 80 difficult student-sourced tasks across 25+ professional domains.The benchmark is intended to evaluate OpenClaw on specialized problems beyond assistant-level scenarios.
  • 5. Conclusion: The benchmark’s current scope is limited by its single-university CS-undergraduate task pool, 80-task size, single-attempt evaluation, and incomplete model coverage.The authors propose broader contributors, multi-trial protocols, retry mechanisms, and additional model releases as future work.

A Full Per-Task Results

Table 7 presents complete per-task scores for six frontier models across all 80 AcademiClaw tasks, highlighting GPU-required tasks and cross-model consistency.

  • A Full Per-Task Results: σ is the across-model standard deviation, serving as a coarse indicator of cross-model consistency.High σ values flag capability-boundary tasks discussed in the domain and task-level analysis.
  • A Full Per-Task Results: Table 7 reports complete per-task scores for six frontier models across 80 AcademiClaw tasks.Scores use a 0–100 scale and rows are sorted by cross-model mean score in descending order.
  • A Full Per-Task Results: Bold task identifiers mark the 16 tasks requiring GPU execution.The table caption specifies that bold identifiers denote GPU-required tasks.

B Task Collection and Curation Details

The benchmark uses standardized student submissions, expert curation, reproducible rubrics, and layered deterministic, judge-based, browser, and vision evaluation.

  • B. Task Collection and Curation Details: Contributors were CS undergraduates with hands-on experience using mainstream AI agents and tasks that had defeated at least one such agent.Participation was voluntary and uncompensated.
  • B. Task Collection and Curation Details: Each submission followed a fixed directory layout containing a task prompt, optional context files, an evaluation module, and structured metadata.The layout used workspace/query.md, context/, eval/rubric.py, and description.json.
  • B. Task Collection and Curation Details: Of 230 candidate submissions, 150 were removed during two expert-review rounds.Rubric-related issues were the largest rejection category because reproducible scoring is difficult for open-ended academic problems.
  • B. Task Collection and Curation Details: Surviving candidates were independently reviewed by two domain experts for clarity, rubric correctness, reproducibility, difficulty calibration, and domain coverage.Disagreements were resolved through discussion, and non-reproducible rubrics could be revised and resubmitted.
  • B. Task Collection and Curation Details: The concrete black-hole visualization rubric combines structural checks, AST matching, code judging, browser execution, and vision judging.The six scoring techniques are exercised across the rubric’s dimensions, including Playwright testing and screenshot comparison.
  • B. Task Collection and Curation Details: Vision judging directly evaluates rendered screenshots against reference renderings using fixed visual sub-rubrics.It assesses event-horizon visibility, accretion-disk color gradient, and compositional fidelity.
  • B. Task Collection and Curation Details: LLM judges receive rubric specifications, deterministic checks, and truncated artifacts under fixed decoding and output-token limits.Text judging allows 2048 output tokens and vision judging allows 1024.

E Safety-Audit Rule Specifications

The safety auditor analyzes complete tool-call trajectories using deterministic detectors and optional LLM verification across five risk categories.

  • E Safety-Audit Rule Specifications: The safety auditor evaluates pre-execution snapshots, tool invocations with arguments, and post-execution diffs.Each of five categories combines a deterministic rule-based detector with optional LLM verification for ambiguous cases.

F Experimental Setup

AcademiClaw evaluates six frontier models under identical OpenClaw conditions, using isolated Docker environments, structured rubric scoring, and safety auditing. The setup standardizes model access, execution budgets, logging, and task routing across CPU and GPU workers.

  • Evaluation protocol: Six frontier models are evaluated with a single fixed judge model so rubric-level scores remain comparable across systems.The evaluated models include Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.4, Gemini 3.1 Pro, MiniMax M2.7, and Qwen3.5-397B-A17B.
  • Evaluation protocol: Safety auditing uses rule-based trajectory scoring with LLM verification to suppress false positives on ambiguous tool calls.The audit specifications cover trajectory events, while the verifier handles cases such as package installation inside a pinned virtual environment.
  • Execution environment: Every evaluation runs through one pinned OpenClaw build with an identical system prompt, tool palette, and gateway daemon.Each task receives one attempt and a 60-minute wall-clock budget; runs exceeding the limit terminate.
  • Execution environment: Each model–task run records transcripts, workspace snapshots, rubric JSON, and safety reports for aggregate and qualitative analysis.Pre- and post-execution records support structural inspection of agent behavior and outcomes.
  • Execution environment: AcademiClaw provides CPU and CUDA 12.2 base images, with per-task images adding dependencies such as Playwright, PyTorch, and Jupyter.The GPU image inherits the CPU image and adds CUDA with cuDNN.
  • Execution environment: GPU routing combines four heuristics, and this procedure identifies all 16 GPU-required benchmark tasks.The heuristics inspect Dockerfiles, CUDA source files, environment variables, and GPU-related query keywords.
  • Execution environment: Workspace snapshot diffing lets agents use the full tool palette while scoring evaluates the resulting file changes.A recursive pre-run snapshot is compared with a post-run snapshot to identify created, modified, or removed files.

H Task Examples

AcademiClaw covers academic workflows ranging from unfamiliar-language reasoning and ML systems implementation to operating systems, creative production, genealogy, and numerical reporting. The examples show both broadly tractable tasks and sharp model-specific variation.

  • STEM reasoning: The linguistics olympiad task has the benchmark’s lowest mean score, 17.3, after requiring zero-shot reasoning over an unfamiliar language.The agent must solve five International Linguistics Olympiad problems from parallel examples.
  • ML engineering: Claude Opus scores 89 versus Sonnet’s 66 on speculative decoding, revealing substantial within-family variance in systems-level ML engineering.The task requires implementing KV-cache rollback and token-level verification from scratch.
  • Systems and infrastructure: All six models score 76–83 on the operating-system lab, indicating that well-scoped systems tasks with clear interfaces remain broadly tractable.The task combines implementing page replacement with writing a performance-analysis report.
  • Creative and writing: All models score 74–81 on the Locking choreography task when the rubric specifies concrete structural criteria.The task requires synchronizing a timestamped routine to a specified music track.
  • Cultural and humanities: The family-tree extraction task has the highest variance, with Claude and GPT scoring 86–92 while MiniMax and Qwen score 3.The task tests genealogical extraction from the repetitive naming structure in One Hundred Years of Solitude.
  • Data analysis and reporting: Five models score 73–84 on competitive-game ELO analytics, while Gemini scores 73 on the multi-step numerical workflow.The agent computes ratings under a specified update rule and produces a trend report from raw match logs.

I Token–Score and Cross-Model Correlations

The correlation analyses find no positive relationship between token expenditure and task scores, while cross-model score correlations reveal heterogeneous capability profiles. The latter difference is statistically supported rather than attributable solely to sampling noise.

  • Cross-model correlations: The pairwise score-correlation matrix is used to characterize heterogeneous capability phenotypes across the six frontier models.The appendix presents this matrix as evidence for the cross-model capability-profile finding.
  • Token–score correlations: Every per-model token–score correlation is statistically indistinguishable from zero, with |r| ≤0.077 and all p> 0.49.This holds within each model, including the two highest-token-spending models, Gemini 3.1 Pro and MiniMax M2.7.
  • Token–score correlations: The two highest-token-spending models do not convert their larger expenditure into score gains within their own task distributions.Gemini averages 2.86 M tokens per run and MiniMax averages 1.66 M tokens per run.
  • Cross-model correlations: A Fisher z comparison of Qwen–MiniMax and GPT–Gemini correlations yields Z=3.995 and p=6.5 × 10^-5.The compared correlations are r_Qwen–MiniMax=0.729 and r_GPT–Gemini=0.275, using n=80 independent samples.

J Licensing, Ethics, and Broader Impact

AcademiClaw releases its own benchmark materials under Apache 2.0 while preserving upstream terms for third-party references. The authors frame results as research snapshots, address safety misuse, and identify limited contributor diversity as a scope boundary.

  • Licensing: The rubric code, evaluation harness, Docker scaffolding, and authored prompts are released under the Apache License 2.0.Third-party course materials, Olympiad problems, papers, and documentation retain their upstream licenses or terms of use.
  • Intended use: The benchmark is intended for research on agent capabilities and limitations, not production certification or commercial-product ranking.Leaderboard results are snapshots tied to a specific framework, decoding configuration, and judge model.
  • Safety and misuse: AcademiClaw mitigates safety misuse by releasing aggregated statistics, combining detectors with human review, and updating rules as evasion patterns emerge.The benchmark’s execution traces could otherwise support training agents to evade its rule-based detectors.
  • Broader impact: The contributor pool is concentrated at one institution, limiting cultural and disciplinary diversity; future releases aim to expand across institutions and non-CS disciplines.The benchmark nevertheless grounds evaluation in authentic student-sourced workflows rather than researcher-designed proxies.
Loading 2605.02661v1…