PaperlayerView exact revision on arXiv

Source-linked AI summary

Qiushi Engine on AstaBench E2E-Bench-Hard

Wenhao Li, Shuxing Yang, Fujia Chen, Jincheng Mi, Yuang Pan, Rui Zhao, Zichen Li, Junyao Wu, Shenzhan Hong, Yaqi Li, Yize Wang, Kaihao Zhu, Taowen Deng, Junjie Yang, Hongsheng Chen, Yihao Yang

arXiv:2609.08196v1cs.AIcs.SE

TL;DR

The paper examines whether autonomous agents can complete full scientific research cycles despite compounding multi-step failures. It evaluates Qiushi Engine on AstaBench E2E-Bench-Hard using rubric-based scoring and finds strong benchmark performance, while emphasizing limits on causal interpretation and scientific-judgment claims.

  • Problem

    Full scientific research requires coordinated question interpretation, experimental design, coding, execution, analysis, and evidence-supported reporting, yet multi-step failure leaves overall agent quality below expert standards.

  • Method

    The evaluation studies Qiushi Engine, a configurable LLM-backed autonomous research system, using binary rubric-item averages to score samples across 40 benchmark tasks.

  • Results

    Qiushi achieved broad pipeline coverage and a 0.816 official leaderboard score, with losses concentrated in statistical support, external dependencies, explicit metrics, and ablations.

  • Takeaways & Limitations

    The results provide evidence of broad end-to-end research capability under AstaBench, but scientific novelty, human judgment, and component-level causality require separate evaluation.

  • Takeaways & Limitations

    Benchmark comparisons reflect complete system configurations because UI mode, task family, resources, environments, models, and tools were not randomly assigned.

Abstract

from arXiv · show

This report analyzes Qiushi Engine v0.8 across all 40 test tasks in AstaBench E2E-Bench-Hard, a benchmark that requires autonomous agents to carry a research question through experimental design, code implementation, actual execution, result analysis, and report delivery. Qiushi Engine is model-configurable; this evaluation selected DeepSeek deepseek-v4pro-preview as the model backend. The official AstaBench leaderboard records a score of 0.816 and an average benchmark cost of USD 15.209 per task, while the full-precision local recomputation is $81.59 \pm 1.87$. Four tasks satisfied every rubric item, yielding a full-task completion rate of 4/40 = 10% -- 7 percentage points above, and about 3.3 times, the approximately 3% best rate reported for AstaBench's official agents. Across 507 required rubric items, 416 were satisfied (82.1%). Official scoring archives and 40 Meta-Trace records show sustained production and verification of reports, code, and experimental artifacts; the principal gaps lie in repeated runs, external dependencies, specified metrics, and ablation studies. The report explains the benchmark, system workflow, aggregate results, representative cases, and limits of interpretation.

1 Introduction

End-to-end scientific discovery asks agents to complete and connect research design, implementation, execution, analysis, and reporting rather than solve isolated coding or question-answering tasks. This report evaluates Qiushi Engine across all 40 E2E-Bench-Hard tasks and links aggregate outcomes to workflow evidence and limitations.

  • The End-to-End Scientific Discovery Challenge: End-to-end scientific discovery requires experimental design, working code, real-data execution, rigorous analysis, and evidence-supported reporting.
  • The End-to-End Scientific Discovery Challenge: Moderate failures at individual research steps can compound into very low complete-task success.
  • AstaBench and E2E-Bench-Hard: AstaBench spans 11 benchmarks and four categories, including End-to-End Discovery, across more than 2,400 problems.
  • AstaBench and E2E-Bench-Hard: E2E-Bench-Hard evaluates full AI/NLP research pipelines using generated tasks that retain their original complexity rather than applying E2E-Bench’s simplification step.
  • Report Scope: The report connects benchmark scoring, Qiushi’s architecture, submission evidence, aggregate results, workflow-linked case studies, leaderboard context, and methodological limitations.

2 How AstaBench E2E-Bench-Hard Scores Research

E2E-Bench-Hard presents detailed, unsimplified research specifications and scores whether report, code, and artifacts jointly support task-specific rubric items. The resulting score measures supported research facets, not an undifferentiated measure of overall quality.

  • Task Structure: Each task specifies a research direction, hypothesis, variables, comparisons, controls, and required measurements while leaving implementation decisions to the agent.
  • Task Generation: HypER generates Hard tasks from research trends, refines them with paper excerpts, ranks them, and retains ideas passing expert review.
  • Task Generation: E2E-Bench-Hard omits E2E-Bench’s simplification pass, preserving the full complexity of the original research specification.
  • Rubric Design: Rubrics cover implementation, data, measurement, statistical, analysis, model-specific, and reporting criteria; scored-item counts range from 8 to 17.
  • Scoring Mechanism: The scorer extracts report, code, and generated artifacts, then uses an LLM judge to issue binary per-item verdicts based on cross-artifact consistency.
  • Scoring Mechanism: The task-level score is the arithmetic mean of sample scores, while a score of 0.80 means 80% of required research facets were supported, not 80% overall quality.

3 Qiushi Engine Architecture

Qiushi Engine uses a configurable language-model backend and a research loop that plans, builds, executes, repairs, verifies, and communicates evidence. Its Meta-Trace records preserve decisions and process states for later recovery and analysis.

  • Architecture: This evaluation configured Qiushi Engine with DeepSeek deepseek-v4pro-preview as its model backend.
  • Workflow Functions: The public workflow comprises Plan, Build, Run and repair, and Verify and communicate functions.
  • Meta-Trace: Each Meta-Trace step combines a scientific reasoning record with a structured handoff describing the next task, current state, and research phase.
  • Meta-Trace: Meta-Trace provides long-horizon memory and supplies process evidence for the report’s analysis.
  • Research Phases: Explore, Execute, and Express are non-linear phases, with evidence-driven transitions and 184 Repair states across 40 runs.
  • Limitations: The workflow can miss rubric items when dependencies are unavailable, run budgets are insufficient, requirements are misinterpreted, or ablations are deprioritized.

4 Submission Protocol and Model Identity

The evaluation separates model identity, execution and submission records, resource accounting, and scorer evidence. Qiushi outputs were exported from Web UI sessions and scored with the official command-line rubric scorer across 40 tasks.

  • Model Identity: The archive distinguishes the Inspect metadata label from the actual selectable DeepSeek backend used to generate research outputs.
  • Submission Protocol: Qiushi sessions ran through the Web UI, with reports, code, traces, and artifacts exported unchanged and scored using astabench score.
  • Cost and Usage: $15.209 per task is the official AstaBench leaderboard cost reported for the 40-run evaluation.
  • Archive Contents: The archive contains 40 complete sample records with task definitions, inputs, outputs, scorer events, and usage statistics.
  • Run Modes: The 40 runs used Coding for 16 samples, Discovery for 12, Submission for 11, and Report for 1.
  • Interpretation: UI mode was not randomized and was confounded with task choice, so descriptive mode differences do not identify a causal effect on rubric coverage.

5 Aggregate Results

Qiushi Engine achieved strong aggregate rubric coverage across all 40 E2E-Bench-Hard tasks, but perfect task completion remained limited and performance varied with task-specific dependencies and complexity. Resource expenditure showed no observed linear association with score, while clinical NLP was disproportionately affected by unavailable external models.

  • 5.1 Overall Score: 0.8159 ± 0.0187 was Qiushi Engine’s mean task score across 40 samples, equivalent to 81.59 ± 1.87 on the percentage scale.The median was 0.833, with scores ranging from 0.500 to 1.000.
  • 5.1 Overall Score: 4/40 = 10% of tasks satisfied every required rubric item, exceeding the approximately 3% maximum full-task completion rate reported for official agents.This stricter task-level measure differs from item-level rubric fulfillment.
  • 5.2 Score Distribution and Bands: 24/40 (60%) samples achieved rubric coverage of at least 0.80, while four samples scored below 0.70.The score bands included four perfect, eight near-complete, twelve strong, twelve substantial-but-incomplete, and four constrained samples.
  • 5.3 Score–Resource Relationships: Resource volume did not predict score within this evaluation: token and step correlations were r = 0.001 and r = −0.010, respectively.The archive’s non-random tasks and resource allocation do not support causal interpretation.
  • 5.5 Resource and Duration Patterns: The median sample duration was 5.9 hours, with a mean of 9.7 hours and a small number of substantially longer runs linked to extended repair cycles.Examples included idea_10 at 28.9 hours, idea_37 at 54.1 hours, and idea_25 at 49.9 hours.

6 Process Analysis: What Produced High Scores and What Caused Losses

Qiushi’s higher scores were associated with complete core implementations, primary evaluation, and artifact production, while losses concentrated in repeated runs, external dependencies, specified metrics, and ablations. Meta-Trace cases show that repair can support either complete or incomplete outcomes, depending on whether empirical breadth is ultimately covered.

  • Aggregate rubric outcomes: 416/507 rubric items were met (82.1%), spanning implementation, baselines, metrics, statistics, ablations, error analysis, visualization, and documentation.The micro-average is distinct from the official macro-average task score of 81.59%.
  • Aggregate rubric outcomes: Higher-scoring samples generally had fewer zero-scored items, while lower-scoring samples often lost points on identifiable gaps rather than broad failure.For example, idea_10 met 10/15 items despite its lower overall score.
  • Recurring score losses: Repeated runs and statistical support were missed in 16 samples, external-resource components in 11, and ablation or sensitivity analyses in 8.These requirements often occurred late and were vulnerable to step or time limits.
  • Recurring score losses: Qiushi performed consistently well on dataset preparation, core implementation, baselines, integration mechanisms, primary metrics, and report/code/artifact production.These capabilities formed the backbone of working experimental bundles.
  • Step-limited cases: Idea_5 reached the step limit after completing research work, while idea_50 retained unmet dataset and repeated-run items when its limit was reached.The step limit therefore coincided with different outcomes across samples.
  • Step-limited cases: Idea_10’s 75-step run remained low-scoring amid recurring dependency conflicts, import errors, and data-format issues.Additional steps without a strategy change may not have resolved those recurring problems.
  • Repair and score outcomes: High Repair activity accompanied both complete and incomplete runs: idea_42 and idea_5 achieved full coverage, whereas idea_10 remained incomplete after repairs focused on core functionality.The incomplete run left empirical-breadth requirements unmet.

7 Comparison with the Official AstaBench Leaderboard

The official leaderboard places Qiushi Engine above the displayed comparison entries, while the score–cost comparison is subject to configuration and environment caveats.

  • Score comparison: 81.6% versus 65.5%: Qiushi Engine leads Ai2’s customized ReAct configuration using Claude Opus 4.7 by 16.1 percentage points.This compares complete system configurations rather than isolating orchestration, model, tools, or execution environment.
  • Score comparison: 25.4 ± 4.5: the Faker baseline provides a reference point for how plausible fabricated outputs can satisfy rubric items.The comparison is informative for the scoring scale but does not isolate an execution effect.
  • Score–cost comparison: Figure 9 plots AstaBench E2E-Bench-Hard score against average cost per task across 29 leaderboard results.The dashed curve marks the Pareto frontier, and labels retain two-decimal percentages from the 4 September 2026 HTML snapshot.
  • Comparison caveats: The comparison is limited by different environments, since Qiushi uses custom tools while AstaBench agents use the standard Asta Environment.Available packages, compute resources, network access, and APIs therefore differ.
  • Score–cost comparison: The official Qiushi Engine cost is $15.209 per task in the standardized leaderboard display.Table 9 and Figure 9 use the official leaderboard cost field.
  • Comparison caveats: All evaluations use the same claude-sonnet-4-6 LLM-as-judge scorer model.This keeps the rubric evaluation standard consistent across published agents and Qiushi.

8 Discussion

Qiushi achieves broad research-pipeline coverage, with verification and repair supporting cross-artifact evidence, but score losses concentrate in empirical breadth, dependencies, and interpretation errors. The archive-level patterns are descriptive and do not establish causal score predictors.

  • 8.1 What the Score Means: 0.815940 is the official macro-average, while 416/507 = 0.820513 is the micro-average across rubric items.The macro-average weights tasks equally; the micro-average weights tasks with more rubric items more heavily.
  • 8.1 What the Score Means: 82% of rubric items were satisfied, with missed items concentrated in statistical support, external dependencies, and ablation analyses.The score measures concrete, verifiable research-pipeline actions rather than novelty, writing quality, or scientific interpretation.
  • 8.2 The Role of Verification and Repair: 184 Repair states and repeated End checks accompanied a workflow emphasizing verification and repair across artifacts.Perfect-scoring samples recorded hash checking, determinism testing, matched-alpha control verification, and independent result reconstruction.
  • 8.2 The Role of Verification and Repair: Verification consumes steps that could fund repeated runs or ablations, creating a tradeoff that currently favors reliability over experimental breadth.The analysis links this allocation pattern to the frequent absence of statistical support, repeated runs, and ablation studies.
  • 8.3 Score-Loss Mechanisms: Resource-allocation pressure affected 16/40 samples, while external dependency failures caused larger per-sample losses and premature conclusion produced the largest loss when it occurred.Idea_16 lost 6 items because two models were unavailable, whereas idea_12 ended after 13 steps with six unmet items.
  • 8.5 Comparative Interpretation: The score gap between Qiushi and Faker is 56.2 percentage points, but comparisons reflect complete system configurations rather than isolated components.Qiushi scored 81.6 versus Faker’s 25.4, with differences also involving model, system, tool, and execution configurations.

9 Conclusions

Qiushi Engine v0.8 provides broad coverage of unsimplified end-to-end research pipelines on E2E-Bench-Hard. Its strongest results concern core implementation and evidence bundling, while remaining gaps concern statistical support, dependencies, metrics, and ablations.

  • 9 Conclusions: 0.816 is Qiushi’s official leaderboard score with DeepSeek deepseek-v4pro-preview, alongside a standardized cost of $15.209 per task.The comparison applies to the complete Qiushi configuration rather than to the model backend alone.
  • 9 Conclusions: 10% of tasks were completed perfectly, or 4/40, compared with the approximately 3% best full-task completion rate reported for AstaBench’s official agents.This corresponds to a 7-percentage-point gain and roughly a 3.3-fold rate.
  • 9 Conclusions: Core research implementation covered dataset handling, method and baseline construction, primary metric computation, and cross-artifact evidence bundling.The 40-sample inventory includes task-specific gaps in metrics, statistics, required models, and ablation or sensitivity analyses.
  • 9 Conclusions: 16 samples lacked statistical support, 11 lacked core components requiring external resources, 10 lacked explicit metrics, and 8 lacked ablation studies.These counts summarize zero-scored dimensions identified across the 507 rubric items.
  • 9 Conclusions: The benchmark provides evidence of broad pipeline coverage, but scientific novelty, human expert judgment, and causal attribution of system components require separate evaluation.The conclusion therefore concerns benchmarked research-pipeline execution rather than overall scientific quality.

B Rubric Dimension Inventory

The rubric inventory classifies 507 scored items by required research action and links zero-scored dimensions to distinct failure mechanisms. Failures cluster around empirical breadth rather than core implementation.

  • B Rubric Dimension Inventory: 507 scored rubric items were classified by the research action each item requires, with categories recording affected samples, failure mechanisms, and archive examples.Table 11 organizes these categories rather than treating the score as a single undifferentiated quality measure.
  • B Rubric Dimension Inventory: Resource-allocation pressure accounts for the largest share of failures because statistical support, ablation, generalization, and efficiency are typically scheduled late.External dependency failures affect fewer samples but produce the most severe per-sample impact.
  • B Rubric Dimension Inventory: No sample scored zero on all implementation items, including the lowest-scoring idea_12 sample, which satisfied 6 of 12 items.The archive therefore shows high implementation coverage even when empirical breadth remains incomplete.

C Submission Archive Contents

The submission archive combines integrity checks, evaluation outputs, configuration and telemetry records, and parsed Meta-Trace states for all 40 samples. These records support reproducibility and descriptive process analysis, with explicit limits on causal interpretation.

  • C Submission Archive Contents: The archive includes checksums, replay documentation, a 112 MB Inspect log, per-sample scores, aggregate statistics, and evaluation configuration.The Inspect log contains 40 sample JSONs plus journal, header, summary, and reduction entries.
  • C Submission Archive Contents: The aggregate statistics record an E2E-Bench-Hard test score of 0.8159401260504202 and standard error 0.018745023003278058.The archive also stores submission metadata identifying Qiushi Engine v0.8 and the deepseek-v4pro-preview backend.
  • C Submission Archive Contents: Telemetry records per-sample session IDs, steps, turns, tools, LLM calls, duration, and token counts, while source inventories store scores and SHA-256 hashes.These files connect packaged results to their source evaluation logs.
  • C Submission Archive Contents: Meta-Trace states include Process, Repair, End, Verify, Plan, Review, Reflect, and Evaluate for per-sample process analysis.The state-count table covers all 40 samples and is sorted by score.
  • C Submission Archive Contents: Process-state counts describe workflow allocation rather than causal score determinants, and all-zero rows indicate unavailable or unparsed evidence.High-repair runs can have opposite outcomes, such as idea_42 scoring 1.0 and idea_10 scoring 0.667.

E Data Processing Methodology

The report uses reproducible Python scripts to generate figures and tables from the submission archive and 40 Meta-Trace process records.

  • Reproducible Python scripts generate figures and tables from the submission archive and 40 Meta-Trace process records.

E.1 Script Descriptions and Verification Chain

Five sequential Python scripts form a reproducible evidence chain that aggregates source data, extracts evaluation and rubric evidence, and produces analytical outputs. Independent consistency checks confirm the extracted results, including a recomputed score mean of 0.8159401260504202 matching the official value.

  • Five Python scripts read each prior step’s output, forming a complete reproducible evidence chain.
  • 40 samples are aggregated from six JSON files into per-sample records and summary statistics, with verification of the sample count and official score mean.The output contains 40 rows and 16 columns per row.
  • The evaluation log’s 45 entries are parsed to verify task, model, scorer, and per-sample event-chain metadata.The identified model is deepseek/deepseek-v4pro-preview and the scorer is score_rubric.
  • Evidence extraction records task definitions, scores, scorer explanations, event counts, output package sizes, UI telemetry, and Meta-Trace summaries for all 40 samples.The resulting evidence-chain file contains 40 rows with complete scorer explanation text and event chains.
  • 507 rubric-item scores and process-state counts are extracted into long-format and per-sample summaries.Process states include Process, Repair, End, Verify, Plan, Review, Reflect, and Evaluate.
  • 0.8159401260504202 is the recomputed score mean, matching the official value to all significant digits after rubric scores are reaggregated.The recomputation covers 507 scored items, of which 416 were met and 91 were zero-scored.
Loading 2609.08196v1…