Source-linked AI summary
MedSkillAudit: A Domain-Specific Audit Framework for Medical Research Agent Skills
Yingyong Hou, Xinyuan Lao, Huimei Wang, Qianyu Yao, Wei Chen, Bocheng Huang, Fei Sun, Yuxian Lv, Weiqi Lei, Xueqian Wen, Pengfei Xia, Zhujun Tan, Shengyang Xie
TL;DR
Medical research agent skills require safeguards beyond general-purpose evaluation because apparent structural completeness may coexist with scientific unreliability. MedSkillAudit is a domain-specific layered audit framework evaluated against expert review on 75 skills, providing preliminary evidence that structured automated audit may complement human evaluation in pre-deployment governance.
Problem
Medical research agent skills require domain-specific evaluation of scientific integrity, methodological validity, reproducibility, and safety beyond general-purpose quality checks.
Method
MedSkillAudit combines automated pre-screening with a Claude-driven evaluation agent to assess release readiness across 75 skills spanning five medical research-related categories and compare outputs with independent expert review.
Results
57.3% of skills fell below the Limited Release threshold, while structured system–expert agreement exceeded the human inter-rater baseline in divergence magnitude.
Takeaways & Limitations
Domain-specific pre-deployment audit may provide a practical complement to human evaluation in governance workflows for medical research agent skills.
Takeaways & Limitations
The Academic Writing scene override had not yet been implemented, and proposed rubric recalibration requires controlled reevaluation to confirm whether it reduces category-level agreement problems.
Abstract
from arXiv · showhide
Background: Agent skills are increasingly deployed as modular, reusable capability units in AI agent systems. Medical research agent skills require safeguards beyond general-purpose evaluation, including scientific integrity, methodological validity, reproducibility, and boundary safety. This study developed and preliminarily evaluated a domain-specific audit framework for medical research agent skills, with a focus on reliability against expert review. Methods: We developed MedSkillAudit (skill-auditor@1.0), a layered framework assessing skill release readiness before deployment. We evaluated 75 skills across five medical research categories (15 per category). Two experts independently assigned a quality score (0-100), an ordinal release disposition (Production Ready / Limited Release / Beta Only / Reject), and a high-risk failure flag. System-expert agreement was quantified using ICC(2,1) and linearly weighted Cohen's kappa, benchmarked against the human inter-rater baseline. Results: The mean consensus quality score was 72.4 (SD = 13.0); 57.3% of skills fell below the Limited Release threshold. MedSkillAudit achieved ICC(2,1) = 0.449 (95% CI: 0.250-0.610), exceeding the human inter-rater ICC of 0.300. System-consensus score divergence (SD = 9.5) was smaller than inter-expert divergence (SD = 12.4), with no directional bias (Wilcoxon p = 0.613). Protocol Design showed the strongest category-level agreement (ICC = 0.551); Academic Writing showed a negative ICC (-0.567), reflecting a structural rubric-expert mismatch. Conclusions: Domain-specific pre-deployment audit may provide a practical foundation for governing medical research agent skills, complementing general-purpose quality checks with structured audit workflows tailored to scientific use cases.
1 Introduction
Medical research agent skills require domain-specific safeguards because structural completeness and strong apparent performance may still coexist with scientific unreliability. MedSkillAudit addresses this gap by auditing release readiness and comparing its outputs with independent expert review across 75 skills.
- Medical research skills can produce unsupported claims, misaligned analyses, or irreproducible guidance despite appearing structurally complete.
- General-purpose evaluations measure capabilities, task completion, or software quality but do not directly assess whether reusable research skill artifacts are safe and reproducible.
- MedSkillAudit is a domain-specific framework for evaluating medical research agent skills before deployment.
- The study evaluated 75 skills across five medical research-related categories and compared framework outputs with independent expert review.
- Structured automated audit is presented as a potential complement to human evaluation in pre-deployment governance workflows.
2.1 Study Design
The study asked whether MedSkillAudit evaluations align meaningfully with expert review. It used a reliability study comparing system outputs with independent expert assessments across the full 75-skill evaluation set.
- The primary research question was whether MedSkillAudit can generate evaluations that meaningfully align with expert review.
- The study compared system outputs against independent expert review using standard agreement statistics.
- The reliability study covered the full evaluation set of 75 skills.
2.2 Skill Evaluation Set
The evaluation set contained 75 medical research-related skills sampled across five functional categories and multiple development cycles. The design aimed to represent realistic pre-deployment quality variation rather than an optimized showcase corpus.
- 75 skills were evaluated, with 15 sampled from each of five functional categories.
- Skills came from four successive development cycles produced by two independent research teams, with random sampling within each category.
- The sampling strategy targeted realistic pre-deployment quality variation rather than an optimized showcase corpus.
- The categories covered Evidence Insight, Protocol Design, Data Analysis, Academic Writing, and Other general utility skills.
2.3 MedSkillAudit Framework
MedSkillAudit is a layered pre-deployment pipeline combining structural checks, dynamic domain-specific evaluation, veto gates, weighted scoring, and tiered release dispositions. It produces structured audit outputs with scores, veto evidence, and improvement guidance.
- MedSkillAudit combines automated Python pre-screening with a Claude-driven evaluation agent to assess release readiness and generate revision feedback.
- Structural Audit: The structural layer applies four hard gates covering operational stability, structural consistency, result determinism, and system security.
- Domain-Specific Research Audit: The domain-specific gate checks scientific integrity, practice boundaries, methodological baseline, and code usability after dynamic output evaluation.
- Scoring and Release Disposition: Dynamic scoring uses a 40-point generic rubric and a 60-point category-specific rubric across 3, 5, or 7 test inputs scaled to complexity.
- Scoring and Release Disposition: Release tiers are Production Ready, Limited Release, Beta Only, and Reject, with any veto failure overriding the numeric score and producing Reject.
2.4 Expert Review Protocol
Two medical-research experts independently reviewed all 75 skills using a standardized environment and shared evaluation dimensions. Each assigned a continuous quality score, release disposition, and high-risk flag.
- Expert Review Protocol: Two experts independently evaluated each of 75 skills using the MedSkillAudit rubric dimensions.Experts executed representative tasks and reviewed skill outputs in a standardized evaluation environment.
- Expert Review Protocol: Each expert assigned a 0–100 quality score, a four-level release disposition, and a binary high-risk flag.The release scale was shared with the automated system; the high-risk flag indicated observed fail
- Expert Review Protocol: Standardized spreadsheet templates recorded the ratings, with cross-reference verification finding zero score or disposition discrepancies against the primary database.The two experts used different rating formats across skill subsets, but the verification pass found no recorded-rating discrepancies.
2.5 Consensus Derivation
Consensus scores generally averaged the two expert scores, while dispositions and high-risk flags were resolved through explicit adjudication rules when experts disagreed.
- Consensus Derivation: Consensus quality scores were computed as the arithmetic mean of Expert 1 and Expert 2 scores.For S010, the consensus score used Expert 2 alone because Expert 1 assigned no numeric score after all four Structural Veto dimensions failed.
- Consensus Derivation: One-rank disposition disagreements were resolved using the disposition closer to the score-weighted mean, while larger disagreements used the more conservative lower-release disposition.Adjudication was flagged for every case involving expert rank disagreement.
- Consensus Derivation: Consensus high-risk status was Y or N when both experts agreed and Unclear when their flags differed.The rule preserved disagreement rather than forcing a binary consensus flag.
2.6 Statistical Analysis
The study used standard agreement statistics and predefined optimization criteria to assess reliability and identify skills requiring further work.
- Statistical Analysis: Analyses were performed in Python 3.9 using pandas, pingouin, scipy, and scikit-learn.These packages supported the reported data processing and statistical analyses.
- Statistical Analysis: Score agreement used ICC(2,1), while ordinal disposition agreement used linearly weighted Cohen’s κ.ICC(2,1) specified two-way random effects, single measures, and absolute agreement; weighted κ assigned partial credit by rank distance.
- Statistical Analysis: Optimization flags were triggered by Reject status, low Beta Only scores, required adjudication, system–consensus rank gaps ≥2, or high-risk flags Y or Unclear.These criteria were prespecified before identifying skills for optimization.
3 Results
Across 75 skills, baseline quality varied substantially by disposition, category, and execution mode, while MedSkillAudit showed moderate overall agreement with expert consensus but important category-specific mismatches.
- 3.1 Baseline Quality Assessment: 72.4 was the mean consensus quality score, and 57.3% of skills fell below the Limited Release threshold.The modal disposition was Beta Only: 31 skills (41.3%), compared with 17 Production Ready, 15 Limited Release, and 12 Reject.
- 3.1 Baseline Quality Assessment: Protocol Design had the highest mean consensus score at 86.2 ± 3.8, whereas Academic Writing had the lowest at 62.7 ± 7.2.Academic Writing included 5 of 15 Reject skills, while Data Analysis showed the widest score variance at 70.7 ± 15.3.
- Human Inter-Rater Agreement (Baseline): 64.0% of skills required expert adjudication, including all 15 Academic Writing skills versus 1 of 15 Protocol Design skills.The category contrast was 100% versus 6.7% adjudication, respectively.
- 3.1 Baseline Quality Assessment: Prompt-only Mode A scored 77.9 ± 12.9, higher than script-based Mode B at 70.1 ± 13.0 and hybrid Mode D at 70.2 ± 10.4.Mode A also had a greater proportion of Production Ready or Limited Release skills than Modes B and D.
- 3.1.1 Representative Reject Cases: Reject cases included undisclosed mock data, incorrect APIs or species annotations, unimplemented scripts, dependency conflicts, and nonreproducible outputs.Representative failures also involved incomplete logic, structural gaps, and generation failures under varied inputs.
- Human Inter-Rater Agreement (Baseline): Human inter-rater ICC(2,1) was 0.300, with a mean absolute score difference of 13.8 points and exact rank agreement for 28/75 skills.Within-one-rank agreement was achieved for 56/75 skills (74.7%).
- System–Expert Agreement: The mean system–consensus bias was -1.4 with no statistically significant directional bias (Wilcoxon p = 0.613).Bland–Altman limits of agreement were -29.0 to +26.2; negative divergences reflected veto-driven score collapse, while positive divergences reflected structural-completeness rewards despite expert penalties for shallow logic or limited coverage.
- System–Expert Agreement: System–consensus ICC(2,1) was 0.449, exceeding the human inter-rater ICC of 0.300, while absolute-difference variability was lower at SD = 9.5 versus 12.4.Within-one-rank agreement was 82.7% for system–consensus comparisons versus 74.7% for human raters.
4 Discussion
MedSkillAudit is presented as a domain-specific governance layer that operationalizes release readiness for medical research agent skills and exposes rubric–expert mismatches. Its preliminary evaluation supports structured audit as a complement to expert review, while highlighting calibration, category, sample-size, and execution-mode limitations.
- 4.1 A Structured Audit Framework Addresses a Distinct Gap in Medical AI Skill Governance: MedSkillAudit addresses deployment readiness as a distinct governance dimension beyond benchmark capability and general-purpose code-quality evaluation.It evaluates reusable skill artifacts for structural integrity, scientific validity, dynamic execution behavior, and safety boundaries.
- 4.1 A Structured Audit Framework Addresses a Distinct Gap in Medical AI Skill Governance: The two-gate veto architecture separates structural integrity from scientific integrity, treating catastrophic failures differently from mediocre but complete outputs.Structural failures such as crashes or unsanitized exec() calls can trigger deployment risk independently of ordinary score deductions.
- 4.1 A Structured Audit Framework Addresses a Distinct Gap in Medical AI Skill Governance: Dynamic execution testing was emphasized because static analysis cannot reliably detect missing library functions, changed APIs, or dependency conflicts in script-based and hybrid skills.The framework assigned dynamic execution 60% of the weighting for the 53 script-based or hybrid skills.
- 4.3 Framework Calibration Is an Iterative Process: Evidence from v1.0 to v1.1.0: The framework’s limitations were treated as diagnosable rubric mismatches, leading to post hoc refinement in skill-auditor@1.1.0 without controlled evaluation of the revisions.Targeted scene overrides were introduced after systematic biases were identified in scientific-computing contexts.
- 4.3 Framework Calibration Is an Iterative Process: Evidence from v1.0 to v1.1.0: The shared rubric misclassified scientific-computing behavior by penalizing defensive stops, rewarding risky auto-correction, and preferring human-readable recovery guidance over machine-parseable error codes.These biases reflected assumptions about human-operated general-purpose software that did not hold for agent-first Mode B/D contexts.
- 4.4 The Academic Writing ICC as a Diagnostic, Not a Failure: Academic Writing produced a negative ICC of -0.567 because the system and experts assessed different constructs, with the rubric emphasizing operational and stylistic features while experts prioritized artifact quality.The framework’s behavioral testing also surfaced reproducibility failures, incomplete generation under adversarial prompts, and missing disclaimers that spot-checking may miss.
- 4.5 Practical Implications for Skill Governance: MedSkillAudit supports iterative governance through veto-based risk detection, prioritized revision, re-audit, disposition tiers, and auditable JSON records of quality state.Beta Only skills can be staged for limited testing while revisions proceed.
- 4.6 Limitations and Directions for Future Work: The 75-skill evaluation set limits statistical power for stratified analyses, leaving category-specific agreement estimates imprecise and divergence tails less fully characterized.Larger per-category evaluation sets are proposed to improve precision.
5 Data and Code Availability Statement
The evaluation used 75 medical research agent skills from historical development versions, but those version data are not publicly available because of institutional data governance restrictions.
- 75 skills were drawn from historical development versions produced across two independent research teams.Aggregated quality metrics, disposition counts, and anonymized per-category agreement statistics are reported in the article.