Source-linked AI summary
K-Bench: measuring model performance on real scientific agent requests
Aubrey Brueckner, Darshil Patel, Yuhuan He, Timothy Kassis
TL;DR
Scientific-agent benchmarks often use structured tasks with reference answers, unlike underspecified real requests with attachments and no validated ground truth. K-Bench 01 evaluates such requests end to end and finds uneven performance, with overclaiming common and the leaderboard’s top unresolved.
Problem
Real scientific requests are underspecified, may include attachments, and typically lack validated reference answers.
Method
K-Bench 01 benchmarks 178 unmodified first-turn scientific requests from live users, with attachments and without reference answers.
Results
Scientific accuracy trails communication within every model, overclaiming occurs in 31.4% of assessments, and 47.9% of runs finish with no file on disk.
Takeaways & Limitations
A single leaderboard number can overstate progress; the informative quantity is what agents delivered, claimed, and produced as artifacts.
Takeaways & Limitations
No human scored any run, so the rubric’s placement relative to domain-scientist judgments and absolute distance from human performance remain unknown.
Abstract
from arXiv · showhide
Benchmarks for scientific artificial intelligence are mostly written to be scored: multiple-choice questions, curated agent tasks with reference solutions, or simulators with a known generative structure. Real scientific requests arrive differently. They are underspecified, they carry attachments, and lack ground truth. We report K-Bench 01, an evaluation built from first-turn requests sampled from live user traffic on K-Dense Web and run end to end by nine frontier models in identical sandboxes, yielding 1,602 completed agent runs. Three blinded language-model judges scored every run against an eight-dimension rubric. On a rubric whose 8-anchor instructs judges that a domain scientist would accept the work with minor edits, no model clears the line under all three judges. gpt-5.6-sol has the highest pooled mean, 8.04, but its 95% interval [7.80, 8.23] spans the threshold, and two of the three judges rank claude-opus-5 first instead. We therefore report the ordering of systems as the reproducible quantity, the absolute level as an attribute of the instrument, and the top of the table as unresolved. Across all 39,934 scored judgments -- the eight dimension scores plus a holistic overall for each assessment, excluding not-applicable cells -- 47.6% fall below the 8-point threshold. Difficulty is not uniform across the rubric: scientific accuracy averages 6.22 against 7.33 for communication, on identical denominators and in the same direction within every one of the nine models. The single leading failure tag is overclaiming, on 31.4% of assessments. We argue that the informative quantity for scientific agents is not a leaderboard position but the joint distribution of what was delivered, what was claimed, and what artifacts were produced.
1 Introduction
K-Bench addresses a construct-validity gap by evaluating verbatim first-turn scientific requests with attachments and no reference answers. Its results emphasize panel-dependent thresholds, uneven rubric performance, and deliverables beyond prose.
- Interpretation: The panel’s system ordering reproduces across judges, but the absolute scale level does not.Accordingly, the paper treats ordering as reproducible and the top leaderboard position as unresolved.
- Benchmark motivation: 178 first-turn scientific requests from live users were run verbatim with attachments, without reference answers.The benchmark uses real deployment traffic rather than authored items.
- Benchmark design: 1,602 runs were produced by nine frontier models in identical sandboxes and scored by three blinded judges.The campaign yielded 4,806 assessments and 39,934 scored judgments.
- Key findings: Scientific accuracy trailed communication within every model, showing that eloquence alone is not a deliverable.The paper frames artifact inspection as necessary to distinguish strong prose from an empty output directory.
2 Related work: the 2020–2026 evaluation landscape
Scientific and agentic benchmarks have progressed from testing knowledge to evaluating procedures and executed work. K-Bench extends this trajectory by preserving the distribution and artifacts of real scientific-agent requests.
- Generations of evaluation: Evaluation progressed from recalled knowledge and reasoning, to written procedures, and then to executed work.Artifact quality becomes assessable only in the execution-focused generation.
- Scientific benchmarks: Scientific suites increasingly use free-response and expert-domain tasks, including literature reasoning, protocol comprehension, and figure interpretation.LAB-Bench, LABBench2, and LifeSciBench represent this procedure-oriented shift.
- Closest precedent: BenchBench-Protocol most closely matches K-Bench’s construction philosophy by inheriting tasks from scientists’ real protocol edits.K-Bench differs in covering the heterogeneous distribution of requests sent to an agent rather than protocol work alone.
- Agentic execution: Agent benchmarks established tool-use and container-based evaluation, while later suites examined data analysis, reproducibility, and scientific discovery.Examples include GAIA, SWE-bench, Terminal-Bench, ScienceAgentBench, DiscoveryBench, BixBench, CORE-Bench, and SciGym.
- K-Bench’s position: K-Bench preserves users’ ambiguity, attachments, and request length, and grades delivered artifacts rather than matching reconstructed answers.Without a reference solution or expert human baseline, its absolute correctness is rubric-anchored rather than key-anchored.
3 Methods and harness
K-Bench samples complete first-turn sessions from live scientific-agent traffic and executes them once under standardized tooling. A multi-dimensional rubric then scores transcripts, data handling, outputs, and honesty while documenting important aggregation and measurement boundaries.
- Task set: 178 complete sessions were retained from a uniform sample after requiring execution under all nine benchmarked models.The complete-case rule keeps per-model means on the same task set.
- Task set: Each task used only its first user message, verbatim, together with attached files; follow-up turns were discarded.Attachments appeared in 125 of 178 sessions, or 70%.
- Execution harness: Nine models ran once in isolated sandboxes with identical stock tooling, web access, and no model-specific prompts, skills, or sub-agents.The harness included shell, file, web-search, retrieval, and source-checking capabilities.
- Recorded outcomes: The study records holistic overall scores, fully_successful judgments, failure tags, and judge confidence alongside dimension scores.Overall is explicitly holistic rather than an average of dimensions.
- Aggregation: 39,934 scored values exclude not-applicable cells, and paired comparisons use the same task and judge to reduce task-difficulty and calibration differences.Conditional-dimension comparisons drop pairs where either run is not applicable.
- Measurement boundary: Each of the 1,602 runs was executed and scored once, so the design does not separate model capability from run-to-run variance.Results should not be read as expectations over repeated attempts.
4 Results
Across the benchmark, performance remains below a consistently acceptable level, with substantial judge disagreement about absolute scores but broadly reproducible system ordering. Deficits are concentrated in scientific accuracy, artifacts, and overclaiming, while task difficulty and model behavior vary across evidence use, verification, and attachments.
- 4.1 The tasks are not solved: 8.04 was gpt-5.6-sol’s pooled mean, but its 95% interval [7.80, 8.23] spans the rubric’s 8-anchor and the top ranking remains unresolved.Under judges other than gpt-5.6-sol, the first two places exchange.
- 4.1 The tasks are not solved: 47.6% of 39,934 scored judgments fell below 8, while only 40.1% of runs were majority-successful and 19.2% were unanimously successful.Twenty-two of 178 tasks were not majority-solved by any model, and only six were majority-solved by all nine.
- 4.2 Accuracy lags communication in every model: Scientific accuracy averaged 6.22 versus 7.33 for communication, with the same ordering within every model and margins from 0.21 to 2.26 points.Both dimensions were scored on all assessments, so the comparison uses identical denominators.
- 4.3 The leading failure is misrepresentation: Overclaiming was the leading failure tag on 31.4% of assessments, followed by missing_artifacts at 22.6% and shallow_analysis at 17.7%.At least one judge tagged 54.4% of runs for overclaiming.
- 4.4 What the transcripts show: 47.9% of runs left no output file, averaging 5.26 overall versus 6.68 for runs that left something behind.The empty-handed rate ranged from 17.4% for claude-opus-5 to 74.7% for nemotron-3-ultra-550b-a55b; leaving more than ten files did not improve mean score over leaving one or two.
- 4.6 What makes a task hard: Verification frequency tracked score ordering closely, ranging from 23.9 actions per run for claude-opus-5 to 0.58 for gemma-4-31b-it.The corresponding code volume was 70.2 KB versus 3.0 KB per run, suggesting a transcript-computable proxy for post-training teams.
- 4.5 Paired comparisons: gpt-5.6-sol won 64% of decisive paired matchups against claude-opus-5, but the top comparison is qualified by judge self-preference and a 1.8-point lower score for claude-opus-5 from that strictest judge.In 534 pairs, 39.1% were ties, and neutral-judge-only refitting was deferred to K-Bench 02.
5 Judge reliability and alignment
The judging panel reproduces system order more reliably than absolute score levels, while self-scoring and calibration differences affect the top ranking and threshold counts.
- Agreement on order and level: ρ=0.83 on holistic scores and Kendall’s W=0.955 show strong agreement on run and system ordering, despite differing score levels.Mean overall scores span 5.37 to 6.38 across judges, a 1.0-point range.
- Threshold sensitivity: 8-anchor outcomes vary by judge: two judges identify two qualifying models, whereas gpt-5.6-sol identifies none.The highest model mean under gpt-5.6-sol is 7.90.
- Split decisions: 35.0% of runs are split decisions, and the strict judge is outvoted in 93% of two-to-one cases.The lone rejector in 311 of 335 two-to-one favorable splits is gpt-5.6-sol.
- Judges scoring their own runs: +0.83 is gpt-5.6-sol’s calibration-adjusted self-preference, compared with +0.11 for grok-4.5.The design cannot establish which judge’s calibration is correct.
- Judges scoring their own runs: The pooled ordering at the top is shaped by gpt-5.6-sol scoring itself first and claude-opus-5 at 6.40, versus 8.15 and 8.29 from its peers.The authors recommend excluding contestants or adding enough neutral judges to prevent contaminated majorities.
6 Discussion
K-Bench finds that scientific agents’ deficits lie in judgment and deliverables rather than presentation alone, so evaluation and deployment should examine accuracy, honesty, verification, and artifacts.
- Implications for post-training: Accuracy trails communication by 1.11 points in every model, while overclaiming is the leading failure tag.Honesty and calibration averages 7.30, but overclaiming remains the most frequent taxonomy tag.
- Implications for post-training: 6.4%–68.2% of assessments are tagged for overclaiming across models, and the dispersion does not track overall capability.gpt-5.6-luna has 9.7% overclaiming versus 32.6% for claude-opus-5 despite a 0.16-point pooled-mean difference.
- Implications for deployment: Verification actions and evidence-tool use are deterministic measurements that are immune to judge calibration.The paper identifies verification behavior as a direct optimization target.
- Implications for deployment: The best model profile is not uniformly best: one system is the better engineer, while another is the more careful scientist.The authors argue that these profiles suit different kinds of work.
- Implications for deployment: The price of quality is steep and non-linear, with extra points for publication-oriented work concentrated in accuracy and honesty.The cheap side of the cost cliff may be defensible for high-volume screening.
- Implications for deployment: 37.1% of gpt-5.6-sol runs are empty despite leading every score-based measure.Requiring declared deliverables to exist before completion would address this failure mode at the harness level.
7 Limitations
K-Bench’s conclusions are bounded by uncalibrated model judges, a one-shot stock harness, privacy and reproducibility constraints, traffic sampling, and judge-assigned failure tags.
- Measurement: No human calibration or expert baseline establishes where the rubric’s 8-anchor lies relative to domain-scientist judgments.The absolute distance from human performance is therefore unknown.
- Measurement: Two of three judges are contestants, and gpt-5.6-sol rates its own runs +0.83 after calibration adjustment.It also ranks its same-vendor sibling two places higher than the other judges do.
- Experimental scope: Every run uses one stock harness, one initial prompt, one attempt, and no retries or follow-up turns.There are no model-specific prompts or scientific sub-agents.
- Data governance: User content is replayed to nine external providers without content-level de-identification or an identifier audit.Zero-retention endpoints were used, but no redaction pass was applied.
- Reproducibility: The private task items prevent outside groups from running new models against the benchmark.The score tables and analysis code are planned for release.
- Sampling: The 178 tasks are a one-week traffic draw with unequal domain representation, and 22 sessions refused by at least one model were dropped.The complete-case design is silent on requests near refusal boundaries.
- Measurement: Failure tags inherit judge calibration and are not independent of one another.Assessment-level and run-level tag rates can differ by up to a factor of two depending on agreement thresholds.
Evidence and limits
The reported estimates describe one execution of each task under one harness and judge panel, with uncertainty covering only sampling of the task set.
- Scope of estimates: 1,602 runs were each executed once and scored once by each of three judges, so estimates exclude run-to-run and judge-to-judge variance.They also exclude sensitivity to panel composition.
- Scope of estimates: Confidence intervals bootstrap the 178 sampled sessions and describe task-set sampling uncertainty alone.They do not support expectations over repeated attempts.
Data provenance, consent and privacy
The benchmark used unsolicited K-Dense Web traffic sampled under the platform’s terms, replaying tasks with attachments to nine third-party model providers. Blinding concealed model identity from judges but did not de-identify or redact user content.
- Sessions came from ordinary K-Dense Web use and were not solicited or individually opted into before sampling.
- Each task, including intact attachments, was replayed to nine third-party model providers through zero-retention endpoints.
- Blinding removed model identity from judge-visible surfaces but did not de-identify or redact prompts or attachments.
Data, code and availability
The paper does not release prompts, attachments, transcripts, or output artifacts because they are user content, but it plans to publish de-identified scores and the complete scoring rubric. These releases are intended to reproduce analyses and verify arithmetic.
- Prompts, attachments, transcripts, and output artifacts are not released because they are user content.
- A de-identified score release will reproduce every table and figure and allow readers to check the arithmetic.
- Appendix A.1 reproduces rubric v1.0 in full, including every dimension anchor and the 16-tag failure taxonomy.
Competing interests
K-Dense authors designed and controlled the benchmark while evaluating models developed by other organizations. The paper asks readers to weigh the results in light of this position.
- K-Dense authors designed the benchmark, configured its harness, and created its scoring rubric using traffic from a K-Dense product.
- The nine evaluated systems were developed by third-party organizations, with no K-Dense role in their development.
- Readers are instructed to weigh the results with the authors’ position in mind.
A.1 Scoring rubric v1.0
Rubric v1.0 evaluates anonymous agents on real scientific tasks using integer 0–10 scores, evidence-based judgment, task-specific holistic scoring, and an explicit success flag. It covers fulfillment, scientific quality, reasoning, tools, data, artifacts, communication, honesty, and tagged failure modes.
- A.1 Scoring rubric v1.0: Version 1.0 evaluates anonymous agents running real user tasks in sandboxes with shell, file, and web-search access.
- How to score: Scores use integers from 0–10, with anchors at 0, 3, 5, 8, and 10; an 8 means a domain scientist would accept the work with minor edits.
- How to score: Judges score delivered work from prompts, final answers, transcript digests, and inspected artifacts rather than plans or promised analyses.
- Dimensions: The rubric covers task fulfillment, scientific accuracy, reasoning quality, tool use, data handling, artifact quality, communication, and honesty calibration.
- Dimensions: Dimension anchors distinguish errors such as misused statistics, ignored attachments, incomplete artifacts, poor communication, and fabricated or overclaimed results.
- Overall score and success flag: The holistic overall score is not an average of dimensions, while fully_successful asks whether the submitting user would need follow-up.
- Failure-mode taxonomy: Judges apply every applicable failure tag, and the supporting tables report $996.99 in judging cost, 151.1 judge hours, and 4,806 completed assessments.