PaperlayerView exact revision on arXiv

Source-linked AI summary

Scientific Agents: Evaluating Profession-Specific System Prompts on Scientific Tasks

Timothy Kassis

arXiv:2610.00084v1cs.AIcs.CL

TL;DR

Whether profession-specific scientific profiles improve problem solving remains an open question. The paper compares matched profiles with four prompt controls across text benchmarks and tool-using bioinformatics tasks. Profiles show no consistent text-accuracy gain, while raising resource use and lowering solve rates under the tested agentic budgets; longer prompts also experience fewer first-pass provider errors.

  • Problem

    Whether profession-specific profiles improve problem solving remains an open empirical question, distinct from evidence about repository-level instructions or focused skills.

    Source: §1 · pp. 1–2

  • Method

    The study compares matched profiles with baseline, role-sentence, generic-rigor, and mismatched prompts on text benchmarks, and tests tool-using agents on BioMysteryBench.

    Source: §1.1 · p. 2, §3 · p. 3

  • Results

    Across nine text benchmarks, profile–baseline accuracy differed by −0.6 percentage points (95% interval [−1.5, +0.2]); BioMysteryBench solve rates were 46.7% with profiles and 56.7% at baseline.

    Source: §4.1 · p. 6, §4.2 · p. 7

  • Takeaways & Limitations

    For the tested model and tasks, always loading full profession profiles is not an effective default for improving factual accuracy or budget-constrained problem solving.

    Source: §6 · p. 11

    Rule-based grading does not measure open-ended scientific thinking, including experiment design, methods-review thoroughness, or explanation clarity.

    Source: §5 · p. 10

Abstract

from arXiv · show

Detailed profession-specific system prompts raise token use and estimated cost per response without a consistent accuracy gain. We evaluate Scientific Agents, an open-source corpus of 503 profession-specific AGENTS.md profiles, with Gemini 3.8 Flash via OpenRouter in the Pi agent harness. We compare matched profiles with four controls: a minimal baseline ("You are a helpful assistant"), the profile's opening role sentence, a generic scientific rigor guide, and a profile from an unrelated domain. Across nine text-based science benchmarks (4,531 sampled questions, 100 matched profiles), 4,488 items completed all five conditions after API-error retries, scored with automated, rule-based grading. The average profile-baseline accuracy difference is -0.6 percentage points (95% bootstrap interval [-1.5, +0.2] across fixed tasks), and no benchmark shows a statistically clear improvement. Matched profiles produced 1.5-2.3 times as many output tokens and cost 2.2-4.5 times more per successful call. On 60 tool-using BioMysteryBench bioinformatics problems (three runs each for baseline and profile), mean solve rates were 46.7% with the profile and 56.7% at baseline, a difference of -10.0 percentage points (95% interval [-16.7, -3.3]) driven by more frequent token- and time-limit stops under the profile. Longer prompts had one unexpected operational advantage: on SuperGPQA, frequent provider API drops left the short baseline with a correct first-pass answer on only 54.0% of items, against 71.6% with the profile. Generic and mismatched prompts were about as reliable, so this gain comes from prompt length or formatting rather than domain expertise. For the tested model and tasks, loading full profession profiles by default does not improve accuracy and costs considerably more; whether selective retrieval of profile sections or open-ended scientific tasks would change this remains to be tested.

1 Introduction

Profession-specific scientific profiles promise domain-tailored guidance, but whether they improve problem solving remains an open empirical question. The paper tests those profiles against simpler and mismatched prompts while separating scored accuracy from broader scientific work.

1 Introduction

  • Prior prompting studies report mixed effects, and profession-wide profiles differ from focused skills because only some advice applies to any one problem.Earlier work found both benefits and drawbacks from personas, while studies of repository instructions also disagree on task success and efficiency.Source: §1 · p. 2
  • The evaluation compares matched profiles with a role sentence, generic scientific rigor guide, and unrelated profile, holding user prompts constant and scoring objective answers automatically.Compute use is reported alongside correctness.Source: §1 · p. 2

1.1 Contributions

  • The study compares five prompt conditions on 4,488 completed items from nine text-based science benchmarks, covering 100 profiles, and evaluates tool-using agents on 60 bioinformatics problems.The contribution summary reports fewer solved problems and more resource-limit stops with full profiles in the tool-using evaluation.Source: §1.1 · p. 2
  • The findings concern always-on prompting in one model and harness, not open-ended scientific work such as study design or literature synthesis.Source: §1.1 · p. 2

2 The Scientific Agents corpus

Scientific Agents is a 503-profile corpus spanning scientific and engineering fields, with a common structure but additional specialized sections in many files. Its documented creation procedures and profile citations were not independently audited.

2 The Scientific Agents corpus

  • The corpus contains 503 profiles across 11 scientific and engineering domains, and 485 profiles include all ten standard sections while 272 add specialized sections.The profiles cover workflows, tools, experimental controls, troubleshooting, and reporting standards.Source: §2 · pp. 2–3
  • The README describes a field-specific research, synthesis, and review process, but these procedures were not audited and the listed profile citations were not checked.Repository metadata gives a median of 52 cited sources per profile, with an interquartile range of 51–57.Source: §2 · p. 3
  • Profiles range from 10,403 to 48,111 bytes, with a median of 19,290 bytes and 2,549 words; the estimated median is about 4,822 tokens.Source: §2 · p. 3
  • The evaluation injects profile text directly as the system prompt rather than testing the repository’s installable plugin management or automated tool discovery.Benchmark specialties without exact profiles were matched to broader related specialties.Source: §2 · p. 3

3 Experimental design

The design compares five system-prompt conditions on fixed, matched text questions and separately evaluates tool-using bioinformatics agents under fixed execution budgets. Automated scoring, retry handling, and resampling methods define the analyzed outcomes.

3 Experimental design

  • Each text question receives the same user message under baseline, role-sentence, generic-rigor, mismatched-profile, and verbatim matched-profile system prompts.The generic guide uses the corpus’s ten section headings, while the mismatched profile is chosen for similar byte length.Source: §3 · p. 3
  • The planned profile comparisons with persona, generic, and mismatch are practical alternatives, not controls that isolate individual prompt ingredients.The unrelated profile may still share scientific concepts with the target task.Source: §3 · pp. 3–4
  • The text sample contains 4,531 questions across nine tasks, matched to 100 profiles by benchmark topic or protocol keywords; some specialties use broader profiles.The aim is to test the matching strategy as used in practice, not every profile in the catalog.Source: §3 · p. 4
  • Text answers are scored by deterministic rules, and items without valid responses under all five conditions are excluded, leaving 4,488 complete items.Technical failures are retried, but wrong answers are not; reported scores use one successful response per item and condition.Source: §3 · p. 4
  • BioMysteryBench uses 60 rule-gradeable problems with shell and file tools; baseline and matched profile run three times each, while generic runs once.The primary agentic comparison pairs problem-level solve rates averaged over three runs.Source: §3 · p. 5
  • BioMysteryBench runs stop at 600,000 cumulative tokens or 35 minutes and count as zero if stopped before answering; the study tests success within these budgets.Cloud containers were not guaranteed to be new, and interrupted or hung jobs were relaunched.Source: §3 · p. 5
  • Accuracy differences use bootstrap intervals, cluster resampling when subcategory counts permit, and an unweighted mean across the nine text tasks.The figure’s primary comparison shows retained item counts, baseline and matched-profile accuracies, and profile-minus-baseline differences with unadjusted 95% intervals.Source: §3 · p. 5, Figure 1 · p. 6

4 Results

Across text benchmarks, matched profiles showed no consistent accuracy advantage and used substantially more resources. BioMysteryBench solve rates were lower with profiles under the tested budgets, while longer prompts were more reliably delivered on SuperGPQA’s first pass.

4.1 No consistent retained-answer accuracy benefit

  • Across nine text benchmarks, the matched profile’s unweighted mean accuracy difference versus baseline was −0.6 percentage points (95% CI [−1.5, +0.2]), with no statistically clear improvement.Other controls showed similarly small differences against the matched profile: −0.5 points versus the one-sentence persona, −0.9 versus the generic rigor guide, and −0.3 versus an unrelated profile from another domain.Source: §4.1 · p. 6

4.2 Lower agentic solve rate under the tested budgets

  • Budget-limit stops occurred in 88 of 180 profile attempts versus 61 of 180 baseline attempts across three runs.Profile prompts were resent each turn; the filtered run-1 comparison among within-budget attempts does not establish equal reasoning quality.Source: §4.2 · p. 7
  • 46.7% with the profile versus 56.7% at baseline was the mean BioMysteryBench solve rate, a −10.0-point difference (95% interval [−16.7, −3.3]).The planned mean averages three runs on the same 60 problems.Source: §4.2 · p. 7, Figure 2 · p. 8

4.3 Repeated runs and the requested-setting comparison

  • Figure 2 separates run-specific profile–baseline differences from the pooled three-run estimate and groups scoring and stopping categories by run.The categories are mutually exclusive, not independent failure causes or a mediation analysis.Source: Figure 2 · p. 8

4.4 Retained resource use and first-pass delivery

  • 1.5–2.3 times as many output tokens and 2.2–4.5 times the cost per successful query accompanied full profiles across text tasks.Cost estimates exclude retries, network latency, and cloud execution; token counts measure text volume, not scientific depth.Source: §4.4 · p. 8, Figure 3 · p. 9
  • 54.0% of scheduled items received a correct first-pass baseline answer versus 71.6% with the profile, a +17.6-point profile advantage.Generic and mismatched prompts performed similarly, so the result does not indicate a domain-expertise benefit; the mechanism remains unclear.Source: §4.4 · p. 9
  • −0.1 points was the mean profile–baseline difference for end-to-end success, a metric distinct from first-pass delivery and completed-item accuracy.End-to-end success counts unrecoverable technical failures as incorrect.Source: §4.4 · p. 9

5 Discussion and limitations

Full profession profiles increase token use and inference cost without consistent accuracy gains, although longer prompts improve first-pass API reliability on SuperGPQA. The evaluation is bounded by its scoring approach, deployment setting, and data and provider limitations.

5 Discussion and limitations

  • Full profession profiles raise token use and cost without consistent accuracy gains, while long prompts improve first-pass provider reliability on SuperGPQA.The paper distinguishes broad profession-wide profiles from curated, task-specific skills and suggests selective retrieval for future evaluation.Source: §5 · p. 10
  • The findings concern scored accuracy and resource use, not open-ended scientific quality such as experiment design, methods-review thoroughness, or explanatory clarity.Even multi-step procedural benchmarks reduce success to whether the agent produced a specific checkable answer.Source: §5 · p. 10
  • Evidence comes from Gemini 3.8 Flash in Pi; text estimates mostly use one run per item and condition, while other benchmark weightings yield the same conclusions.Repeats cover only a 300-item SuperGPQA subset and BioMysteryBench problems; the requested-setting comparison cannot establish whether system prompts replace internal reasoning.Source: §5 · p. 10
  • BioMysteryBench’s per-execution token and time caps disadvantage longer prompts, and container reuse means profile solve-rate differences cannot be attributed to prompt content alone.Profile text was sent on every turn, leaving less room for multi-turn tool use; incomplete logs cannot rule out carryover from reused containers.Source: §5 · p. 10
  • Rule-based graders can misread nuanced answers, accept guesses, or reject valid paraphrases, while provider failures and possible benchmark memorization further qualify comparisons.Items failing any of the five conditions were excluded; systematic failures could bias comparisons, although reported worst-case bounds do not change the main findings.Source: §5 · p. 10
  • The study did not audit profile accuracy or benchmark training overlap, and the author’s K-Dense affiliation and profile-only release limit independent auditing.Fixed mappings and deterministic code limit analytic discretion but cannot substitute for independent replication or outside audit of raw logs.Source: §5 · p. 11

6 Conclusion

The paper evaluates whether profession-specific profiles improve scientific-agent performance over simpler prompt controls. Full profiles raise costs without consistent accuracy gains, and their use as a default remains unsupported within the tested settings.

6 Conclusion

  • Full profiles raised token use and API cost without a consistent accuracy gain over minimal, generic-rigor, or mismatched prompts, while improving first-pass reliability on SuperGPQA.On tool-using bioinformatics tasks, the profile–baseline solve-rate difference was −10.0 percentage points, with more resource-limit stops and limits on attributing the result to prompt content.Source: §6 · p. 11
  • Comprehensive profession profiles are not an effective default for improving factual accuracy or budget-constrained problem solving in the tested agent setting.Whether selectively retrieving relevant profile sections can retain benefits without the overhead remains future work.Source: §6 · p. 11

A Corpus details

This appendix describes the profile corpus and illustrates prompt formats used across the evaluation. The examples include a profession-specific operating manual and several task-specific answer instructions.

A Corpus details

  • Figure 4 contrasts evaluated-profile counts with catalog totals and profile sizes; Table 2 reports ten standard sections in 485 of 503 files, while the excerpt illustrates domain-specific workflows.The bioinformatician example includes assay-aware analysis, reference checks, troubleshooting, and completion criteria.Source: Figure 4 · p. 14, Table 2 · p. 15, Figure 5 · p. 16

B Prompts and conditions

  • User templates were identical across prompt conditions, with answer-format instructions kept in the user message and placeholder values drawn from fixed sample files.Source: B Prompts and conditions · p. 17

SUPERGPQA

  • The SUPERGPQA prompt requests a multiple-choice answer and requires the final line to use the specified letter format.Source: SUPERGPQA · p. 17

MCQ_GENERIC

  • The MCQ_GENERIC prompt requests one correct multiple-choice answer and uses a placeholder set of answer letters in its required final-line format.Source: MCQ_GENERIC · p. 17

TRUE_FALSE

  • The TRUE_FALSE prompt asks for a truth judgment and requires the final line to be either “Answer: True” or “Answer: False.”Source: TRUE_FALSE · p. 17

NUMERIC

  • The NUMERIC prompts require either a single numerical value without units or a single appropriate term for a masked sentence.Source: NUMERIC · p. 17

BIOPRO_ERR

  • The BIOPRO_ERR prompt asks whether a laboratory-protocol step has a reagent, parameter, or operation error, with a specified CORRECT-or-ERROR final answer.Source: BIOPRO_ERR · pp. 17–18

BIOPRO_ORD

  • The BIOPRO_ORD prompt presents protocol steps in scrambled order and requires their chronological sequence as the final answer.Source: BIOPRO_ORD · p. 18

BMB_AGENT

The Senior Scientist Agent profile presents a cross-disciplinary operating manual for framing, conducting, checking, and reporting scientific work.

How You Frame A Problem

  • The profile treats problem framing as matching the question type to its evidence, identifying rival explanations, and clarifying the unit and decision at stake.It also recommends planning analysis before data collection, de-risking with a small test, and establishing a baseline.Source: BMB_AGENT · pp. 19–20

Tools, Instruments, And Software

  • Tool guidance emphasizes understanding failure modes, validating tools on known inputs, recording configurations, and keeping tool chains short.It also recommends pinning environments and reading automation output carefully.Source: BMB_AGENT · p. 20

Rigor And Critical Thinking

  • The profile calls for controls, uncertainty statements, pre-specified analysis, transparent provenance, calibrated claims, and explicit reporting of unresolved assumptions.It distinguishes reproducibility from replicability and asks researchers to report what evidence cannot show.Source: BMB_AGENT · pp. 21–22

Troubleshooting

  • Troubleshooting begins by reproducing the problem, reducing it to a minimal case, checking a known-good baseline, and inspecting raw observations.The profile also advises checking inputs, labels, units, versions, and settings before pursuing an exciting interpretation.Source: BMB_AGENT · p. 21

C Sampling and item-to-profile mappings

The sampling and mapping procedure fixes benchmark samples and item-to-profile assignments before outcome analysis, with fallback and mismatch rules described alongside the mappings.

Sampling

  • The study sampled 4,531 questions across nine tasks using benchmark-specific selection rules, including fixed per-topic or per-domain caps and task-specific exclusions.Options for ChemBench and BioProBench PQA were shuffled once with fixed per-item seeds.Source: C Sampling and item-to-profile mappings · p. 23

Item-to-profile mappings

  • Item-to-profile mappings were fixed before experiments; assignments were specialist when an exact profile existed and general when a broader profession was used.BioProBench assignments used the first triggered keyword rule, while mismatch partners came from another catalog domain and were closest in byte length.Source: C Sampling and item-to-profile mappings · p. 23
  • Mismatch labels reflect catalog availability rather than validated expertise, and a profile from another domain could still share scientific concepts with the task.The text notes hematology paired with regenerative medicine and dermatology with a brain-computer interface engineer as examples.Source: C Sampling and item-to-profile mappings · p. 23
  • ClimaQA used the climate-scientist profile, while BioMysteryBench used the bioinformatician profile.The mapping table reports profile assignments across the evaluated benchmark items.Source: Table 5 · p. 23, C Sampling and item-to-profile mappings · p. 27

D BioMysteryBench grading and execution

The BioMysteryBench section documents deterministic grading, parser checks, and execution safeguards; the corrected scoring retains a negative profile–baseline solve-rate difference.

Grading

  • The deterministic grader uses normalized answer matching across rule families, and three problems with large assignment-table answers were excluded from the 60-problem evaluated subset.Synthetic tests found that strict variants rejected 31 of 32 adversarial cases while retaining 16 of 17 correct cases; applying them changed no actual benchmark score.Source: D BioMysteryBench grading and execution · pp. 27–28, Table 13 · p. 28

Grading sensitivity

  • Correcting hb029 yields a three-run profile–baseline difference of −10.0 points [−16.7, −3.3], versus −9.4 points [−16.1, −2.8] under frozen code.The corrected implementation is the primary grader, and the checks do not eliminate measurement error.Source: Table 14 · p. 28, D BioMysteryBench grading and execution · p. 29
  • Two parser bugs were found with regression fixtures and fixed without changing ground-truth answers; the correction adjusted hb029 scores across four runs.The corrected implementation is the primary grader.Source: D BioMysteryBench grading and execution · pp. 28–29

Execution checks

  • Restart checks retained a negative profile–baseline difference under both conservative scoring of watchdog-relaunched jobs and using only runs 2 and 3.The reported differences were −10.0 points [−16.7, −3.9] and −9.2 points [−16.7, −2.5], respectively; these checks cannot recover unrecorded attempts.Source: D BioMysteryBench grading and execution · p. 29, Table 16 · p. 30

Execution isolation

  • Shared containers could preserve installed packages or cached files across jobs, and missing container IDs prevent identifying which runs shared state.The authors recommend a fresh, isolated sandbox for every future run.Source: D BioMysteryBench grading and execution · p. 30

E Per-benchmark results

The main analysis uses corrected ClimaQA scoring on the same complete items, while first-pass results check whether retries changed the retained-item comparison. The supplied passages also describe sample exclusions and the scope of the subfield plots.

Per-benchmark results

  • −0.6 points was the task-equal mean profile–baseline difference under corrected scoring, versus −0.7 under frozen scoring.The trailing-comma correction added credit to 32 responses across 7 complete items and changed no other text verdicts.Source: E Per-benchmark results · p. 31
  • Items lacking a successful provider response in at least one condition after retry passes were excluded from every condition; format failures remained in the analysis and scored zero.Source: Table 18 · p. 31
  • 621 baseline and 623 profile answers were correct among 820 items completing all five conditions on the first pass, a +0.24-point difference.The retained-item result is therefore not an artifact of retrying failed calls; retained accuracy and end-to-end success that includes API failures remain different measures.Source: E Per-benchmark results · p. 31
  • Most Figure 6 subfield rows contain only 30 items and are descriptive, without subfield-level intervals or multiplicity correction.Figure 7 continues the same ordering and scale; its subfields are not independent subgroup tests.Source: Figure 6 · p. 34, Figure 7 · p. 35

F Inference targets and sensitivity analyses

Sensitivity analyses compare alternative inference targets, resampling choices, repeated-run variability, match-quality strata, and requested reasoning settings. The supplied results report no adjusted significant text contrast and flag limits on interpreting subgroup and setting comparisons.

Inference targets and sensitivity analyses

  • 4 of 36 unadjusted text-comparison intervals excluded zero, but no contrast was significant after within-benchmark Holm correction (smallest adjusted p = 0.090).The four intervals were MedXpertQA versus baseline and generic, BioProBench ERR versus persona, and ClimaQA versus generic.Source: F Inference targets and sensitivity… · p. 36
  • 7, 4, and 3 of 36 text contrasts excluded zero under cluster, stratified-item, and standard-item bootstrap targets, respectively.The authors report that conclusions do not depend on the resampling model; cluster resampling is unstable with fewer than eight groups.Source: F Inference targets and sensitivity… · p. 36
  • On SuperGPQA repeats, baseline accuracy varied by 1.15 points across runs and profile accuracy by 0.41 points; the three-run average profile–baseline difference was +0.7 points [−1.4, +2.8].The three runs estimate run-to-run spread for these data, not a general noise threshold for other benchmarks.Source: F Inference targets and sensitivity… · pp. 36–37
  • On BioMysteryBench, 5 problems favored the profile and 16 favored baseline across three runs, with a sign-flip permutation test of p = 0.007.Problem outcomes changed across runs for 26.7% of baseline problems and 25.0% of profile problems.Source: F Inference targets and sensitivity… · p. 37
  • MedXpertQA’s specialist estimate was −4.2 [−9.1, 0.0], versus −0.9 [−4.7, +3.0] for general matches; neither interval excludes zero.Topic confounding prevents a causal claim about specificity.Source: F Inference targets and sensitivity… · p. 37, Figure 9 · p. 38
  • Under minimal settings, SuperGPQA accuracy was 71.0% for baseline and 72.4% with the profile (+1.4 [−2.1, +4.9]); the API comparison did not remove internal reasoning.The API rejected the high-reasoning request, while reasoning tokens appeared in most minimal-setting queries, so this is not a test without model reasoning.Source: F Inference targets and sensitivity… · pp. 37, 40, Table 29 · p. 40

G Deviations from the pre-run analysis plan

The authors report deviations from the pre-run plan, including revised repeats and reasoning settings, retry handling, watchdog recovery, and post-plan analyses. Corrected grading implementations were used as primary, with frozen results retained for comparison.

Deviations from the pre-run analysis plan

  • Repeat runs and thinking-level ablations were initially deferred after upstream BYOK charges were twice Pi’s price estimates; the provider rejected high and off settings.The substituted minimal request still produced reasoning tokens in most calls, so no inference about removing model reasoning is drawn.Source: G Deviations from the pre-run analysis… · p. 44
  • Provider-error retries expanded beyond the original rule, and the complete-case exclusion rule remained in place with post-hoc residual-missingness bounds.SuperGPQA’s first pass did not retry these errors; a patched harness allowed up to five attempts per pass, followed by additional fill passes.Source: G Deviations from the pre-run analysis… · p. 44
  • A hard wall-clock watchdog was added after hung tool calls escaped the event-driven deadline, and interrupted or preempted runs were relaunched.The plan requested an aggregate interval, but the planned analysis reports a descriptive aggregate mean; post-plan intervals are sensitivity analyses and no aggregate significance or equivalence claim is made.Source: G Deviations from the pre-run analysis… · p. 45
  • Corrected ClimaQA and BioMysteryBench implementations are primary after identified grading incompatibilities; frozen results are reported for comparison.These corrections changed no samples, mappings, reference answers, or model responses, and grading and regression checks remain deterministic.Source: G Deviations from the pre-run analysis… · p. 45

H Availability and execution provenance

The Scientific Agents corpus is public, but the evaluation materials needed to regenerate or independently audit the reported estimates are not. Execution records also have gaps, and the retained provenance does not identify an immutable upstream model snapshot or fully recover sampling settings.

Availability and execution provenance

  • The Scientific Agents corpus at commit 48dedd2 is the study’s only public software artifact; evaluation code, item-level records, raw outputs, and transcripts are not publicly released.Without the evaluation code and item-level records, readers cannot directly regenerate or independently audit the estimates from the corpus alone.Source: H Availability and execution provenance · p. 45
  • Full execution trajectories cover 412 of 420 retained agentic runs, with the remaining gaps all in run 1.Final answers and tool-argument prefixes exist for every retained run, but earlier interrupted attempts may have been overwritten and their complete histories are unavailable.Source: H Availability and execution provenance · p. 45
  • Saved message envelopes do not identify an immutable upstream backend/model snapshot, and effective sampling values cannot be reconstructed.The harness retained no explicit temperature, top-p, or sampling seed; the warmed model catalog and full container/package snapshots were also not archived.Source: H Availability and execution provenance · pp. 45–46
  • Fresh benchmark downloads need not reproduce retained samples because Hugging Face dataset revisions were not pinned when sampling occurred.Local regeneration and a clean-room LaTeX build were checked, but the authors state these checks are not independent replication.Source: H Availability and execution provenance · p. 46
Loading 2610.00084v1…