Training AI Co-Scientists Using Rubric Rewards

• The approach targets open-ended scientific planning, where outputs are difficult to verify and most research cannot be represented by a predefined executable sandbox.Medicine particularly lacks feasible high-fidelity digital simulators, while unguided real-world trial-and-error can waste resources and raise ethical concerns.Source: §1 · p. 1
• Human researchers retain responsibility for articulating scientific objectives, constraints, novelty, and value, while refining generated plans before implementation.A scientific-apprenticeship framing shifts feedback from costly trial-and-error environments to critique of plans before experiments begin.Source: §1 · p. 2
• The method extracts open-ended research goals and goal-specific rubrics from scientific papers, then trains a plan generator with reinforcement learning using a frozen initial model for self-grading.Structured grading also checks seven general guidelines, creating a generator-verifier gap without expensive human annotation.Source: §1 · p. 2
• Ablations show that both goal-specific rubrics and general guidelines are essential to improvement, and evaluations indicate significant cross-domain generalization toward generalist AI co-scientists.The contribution is improvement across different scientific fields using automated rubric evaluations by frontier-model juries.Source: §1 · p. 2
• Experts preferred finetuned plans for 70% of machine-learning goals (p < 0.01), while frontier-model juries found 12–22% relative improvements across medical papers and new arXiv preprints.The finetuned 30B model was competitive with Grok-4-Thinking but remained behind GPT-5-Thinking.Source: §1 · p. 2
• A scalable extraction recipe produces ResearchPlanGen1 from machine-learning papers, medical papers, and recent arXiv preprints for training AI co-scientists.The dataset contains research goals with stated constraints and preferences, alongside requirements that valid plans must satisfy.Source: §1 · p. 2

2 Methodology

• The system trains a policy πθ to generate rigorous research plans from open-ended goals, using paper-extracted rubrics and a frozen initial model as an automated grader.This avoids expert supervision during development while reserving human evaluation for the final assessment.Source: §2 · pp. 2–3

2.1 Training Data Collection

• Paper-derived samples combine research goals, goal-specific rubrics, and reference solutions, with a selector choosing the highest-scoring candidate after rubric and quality filtering.The recipe is fully automated, scalable, and domain-agnostic; human experts verify rubric quality for the machine-learning subset.Source: §2.1 · pp. 3–4

2.2 Rubric Grading

• The grader marks each rubric item satisfied only when no general guideline violations are identified, scoring plans by the fraction of satisfied items.Goal-specific rubrics remain hidden from the generator, while seven general guidelines target common failures such as vague proposals and overlooked flaws.Source: §2.2 · p. 4

2.3 RL Training with Self-Rewards

• GRPO (Shao et al., 2024) optimizes the policy without a separate value network, using rubric scores from a frozen initial policy in a self-rewarding loop.The objective is to maximize satisfaction of detailed, paper-specific constraints.Source: §2.3 · p. 5
• A strict final-plan length limit penalizes verbosity while allowing unlimited thinking tokens, counteracting rubric incentives and judge bias toward longer outputs.This structural constraint is intended to produce plans that are concise yet complete.Source: §2.3 · p. 5

3.1 Dataset: ResearchPlanGen

• The ResearchPlanGen corpus covers ML, eight quantitative arXiv subjects, and medical research, with contamination-aware and high-stakes-domain test settings.The ML subset contains 6,872 training and 685 test goals; the arXiv subset contains 6,573 training and 1,496 test samples, with arXiv tests drawn from August–September 2025 beyond model release dates.Source: §3.1 · pp. 5–6

3.2 Training Configuration

• Training initializes Qwen-3-30B-Instruct-2507 and applies GRPO with group size G = 8, no KL penalty, and separate models for each domain.Generality is additionally tested with Gemma-3-4B, Llama-3.1-8B, and instruct and thinking variants of Qwen-3-4B.Source: §3.2 · p. 6
• A frozen initial Qwen-3-30B policy grades rubric rewards after listing weaknesses, while missing or over-750-word solution tags receive a penalty.The stricter grading procedure improves alignment with Claude-4-Sonnet, and the initial model follows the required format in > 99% of validation samples.Source: §3.2 · p. 6

3.3 Evaluation Methodology

• Evaluation combines a final human comparison on ML plans with automated jury scoring across medical and arXiv goals using GPT-5-Thinking, Gemini-2.5-Pro, and Claude-4-Sonnet.Twenty-five experts assess 100 ML test samples, with three experts per sample, 45 minutes per annotation, five criteria, and 1–10 overall scores; bootstrap sampling provides uncertainty estimates.Source: §3.3 · p. 6, Table 1 · p. 7

4.1 ML Experts Prefer Our Trained Model’s Plans

• 70.0% ± 5.3% of human annotations preferred finetuned plans over initial-model plans, with unanimous decisions favoring finetuning 31-2.Finetuned plans scored 7.89 ± 0.2 versus 7.31 ± 0.2, and satisfied 79.8 ± 2.5% versus 73.8 ± 2.9% of rubric items.Source: §4.1 · p. 7

4.2 Finetuned Models Improve Across Domains, and Become Competitive with Frontier Models

• 10-15% relative improvements over the initial Qwen-3-30B-A3B-Instruct policy brought finetuned models in line with Grok-4-Thinking, though GPT-5-Thinking remained substantially stronger.Figure 3 reports average full-test-set scores from three frontier-model graders across ML, medical, and arXiv goals.Source: Figure 3 · p. 8, §4.2 · p. 8
• 15% relative improvement on ML tasks and 17.5% on ArXiv tasks came from the medical finetune versus baseline, including an advantage over the ArXiv-specific finetune.Improvements on recent ArXiv papers indicate gains beyond papers the initial model may have encountered during earlier training.Source: §4.2 · pp. 8–9

4.3 Training Ablations

• Disabling KL increased validation performance to 29.7, strengthening the grader to Qwen-3-30B raised it to 23.3, and GRPO with 4B self-rewards improved scores from 12.0 to 21.7.Instance-specific rubrics, nested general-guideline checking, and reference solutions improved the recipe, whereas filtering produced only a minor gain.Source: Table 2 · p. 9, §4.3 · p. 10
• After approximately step 120, Qwen-3-30B self-grading continued improving while Claude-4-Sonnet scores diverged, revealing reward-model overoptimization and motivating the step-100 checkpoint.Later plans included superfluous details, while held-out grading found worsening cost and effort efficiency.Source: §4.3 · pp. 9–10
• Supervised fine-tuning worsened plan quality by encouraging reference-solution imitation while reducing compliance with explicit research-goal requirements.The SFT ablation used extracted reference solutions that were vague and omitted important details.Source: §4.3 · p. 10

AI for Research

• This work targets sound plans for researcher-defined goals, contrasting with end-to-end experiment agents limited to narrow, programmatic environments and with approaches that improve scaffolds for frozen models.It is closer to open-ended hypothesis and idea generation, but focuses on planning quality rather than goal novelty and finetunes model weights rather than only improving scaffolds.Source: §5 · p. 10

Reinforcement Learning with Rubrics

• ResearchPlanGen extends rubric-based reinforcement learning to the open-ended, expertise-intensive task of generating research plans by automatically extracting goals and instance-specific rubrics from expert-created papers.The approach addresses the lack of pre-existing datasets and high cost of expert curation, building on scalable rubric-based data creation for question answering, factuality, report generation, and writing.Source: §5 · p. 10

Aligning with Human Preference via Automated Feedback

• The training recipe uses self-rewards from the initial model rather than a stronger reward model, then validates automated feedback against human judgment through an extensive human study.For cross-model and cross-domain proxy evaluation, a jury of significantly stronger models receives instance-specific rubrics to mitigate reward-model overoptimization.Source: §5 · p. 11

Evaluation limitations

• Human expert preferences were available only for machine-learning goals after final-model development, while remaining results used frontier-model rubric grading instead of executing plans.Plan comparison is subjective at the sample level, although aggregate comparisons provide significant evidence; more accurate, economical, and faster evaluations remain a future goal.Source: §6 · p. 11

Model behavior

• Finetuned plans are preferred overall but become more complicated, likely because Qwen-3-30B-MoE grades itself poorly on cost and effort efficiency relative to frontier models.Practitioners can filter unnecessary steps when using the plans to assist human scientists.Source: §6 · p. 11

Future learning methods

• Future learning algorithms could use structured language feedback from rubrics beyond numeric scores to teach novel research-planning skills.The current training may primarily elicit latent research-plan generation abilities rather than teach new planning skills.Source: §6 · p. 11

Future applications

• Future work could apply rubrics to tool-using co-scientists with search and code execution and test whether rubric training generally elicits planning abilities when execution feedback is slow or difficult.Potential applications range from vacation planning to high-stakes decision making.Source: §6 · p. 11

7 Conclusion

• The method extracts research goals and goal-specific rubrics from scientific papers, then trains a plan generator through self-grading with those rubrics as privileged information.The procedure is designed to preserve generality and complement execution-feedback methods specialized to individual scientific tasks.Source: §7 · p. 11
• Human experts confirmed improved research plans for machine-learning goals, while frontier-model rubric evaluations showed gains for medical goals and new preprints across eight arXiv domains.The evaluations also indicate significant cross-domain generalization.Source: §7 · p. 11
• The broader motivation is to develop generalist AI co-scientists that transfer across diverse scientific problems and connect ideas from seemingly disparate settings.The paper relates this goal to transfer learning and the generality of scientific creative processes.Source: §7 · p. 11

A Additional details

• The appendix provides additional methodological details not included in the main paper, including a notation summary.Table 3 summarizes the paper’s notation.Source: A Additional details · p. 17, Table 3 · p. 17

A.1 General Guidelines

• The reward model receives a full set of general guidelines Γ, documented in Table 4.These guidelines are provided to the reward model θr.Source: §A.1 · p. 17

A.2 Hyperparameters

• Papers longer than 15,000 words are filtered out to avoid context-length issues during LLM-based information extraction.This filtering step is part of sample creation.Source: §A.2 · p. 17
• RL training uses temperature 0.7, learning rate 1 × 10−6, batch size 64, no KL penalty, and model-specific A100 allocations and output-token limits.Actor outputs are capped at 2048 tokens, judge outputs at 8192, and thinking-actor outputs at 8192.Source: §A.2 · p. 17
• Inference uses temperature 0.7 for local Qwen, Gemma, and Llama models, API defaults otherwise, and 28k-token judge outputs; stopping steps are selected by validation performance under Claude-4-Sonnet rubric judgments.The stopping-step results are summarized in Table 5.Source: §A.2 · p. 17

A.3 Filtering for Quality

• Filtering scores research goals and rubric items by guideline compliance, averaging rubric quality over the top k = 10 items.The final quality score is the fraction of guidelines not violated, averaged over the selected rubric items.Source: §A.3 · pp. 17–18
• The procedure checks reference-solution faithfulness because low scores can indicate underspecified goals, overly strict or ambiguous rubrics, or incorrect summaries.Higher reference-solution scores indicate higher-quality samples.Source: §A.3 · p. 18
• From up to three samples per paper, filtering selects the sample with the highest averaged quality across the research goal, grading items, and reference solution.The resulting dataset contains one sample per paper, with different papers used for training and testing.Source: §A.3 · p. 18
• The filtering process significantly improves the quality of final training and evaluation data across criteria and domains.Table 6 compares average scores for all candidate samples with the selected sample per paper, including rubric scores before and after reduction.Source: §A.3 · pp. 18–19, Table 6 · p. 19

B.1 Generalization to Other Model Families

• Finetuning consistently improves plans across all model backbones, with especially large gains for Llama, which surpasses Gemma despite lower starting accuracy.The authors attribute Llama’s gains to training that replaces initially vague plans with more detailed explanations of choices.Source: Figure 5 · p. 20, §B.1 · p. 20
• For Llama and Gemma, training retains Qwen-3-30B as the reward model because preliminary experiments found those models to be poor, lenient judges.The authors argue newer models such as Qwen-3-30B are generally better at evaluation and instruction following.Source: §B.1 · p. 20
• The three-judge jury finds consistent improvement across all model families for ML research goals, though Claude-4-Sonnet often prefers initial plans while GPT-5-Thinking and Gemini-2.5-Pro prefer finetuned plans.The jury consists of GPT-5-Thinking, Claude-4-Sonnet, and Gemini-2.5-Pro, reporting pairwise win rates and overall scores.Source: §B.1 · p. 20, Table 7 · p. 20

B.2 Training Qwen-3-4B Instruct vs Thinking

• The study trains only instruct models because performance was similarly strong at the scale available for training.The authors report observing similar performance before overoptimization begins.Source: §B.2 · p. 20
• No differential gains from additional test-time compute were elicited for Qwen-3 thinking models in this task.The authors hypothesize that training emphasis on mathematical and coding tasks limits the model’s ability to exploit additional test-time compute.Source: §B.2 · p. 20
• Thinking may still benefit research-plan generation at larger scale, despite showing no advantage in this training setting.This is presented as an expectation for future work rather than an observed result.Source: §B.2 · p. 20
• Very similar validation performance is observed between Qwen-3-4B instruct and thinking models trained with the Qwen-3-30B MoE reward model.Claude-4-Sonnet served as the validation grader.Source: Figure 6 · p. 21
• More than 2x compute is required for the thinking model to complete the same number of training steps.Thinking also takes more than 2x compute per step and uses more memory in this setting.Source: §B.2 · p. 20, Figure 6 · p. 21

B.3.1 Effect of Reward Model θr Capability (4B vs 30B MoE)

• Improving reward-model capability is proposed as a future direction that may produce gains continuing to hold under human judgment.The reported KL coefficient for this ablation was 0.001.Source: §B.3.1 · p. 21

B.3.2 Supervised Fine-Tuning Fails to Generalize

• Validation loss fell from 1.984 to 1.127 at learning rate 1e-5, batch size 32, and step 180, while training loss continued decreasing as validation loss worsened afterward.This best-validation checkpoint nevertheless performed significantly worse under the automatic grader, showing failure to generalize on hard long-form outputs.Source: §B.3.2 · p. 21
• SFT significantly worsens performance on the task despite selecting the checkpoint with the best validation loss, according to Claude-4-Sonnet rubric judgements.The checkpoint was chosen from a learning-rate and batch-size sweep using validation loss on the validation set.Source: Table 8 · p. 21, §B.3.2 · p. 21

B.3.3 Effect of Disabling KL

• Disabling the KL penalty in GRPO improves results after a certain level of training.With KL enabled, some guidelines, including detailed and specific solution, worsen with more training; this degradation does not occur when KL is disabled.Source: Figure 8 · p. 22, §B.3.3 · p. 23

B.3.2 Supervised Fine-Tuning Fails to Generalize

• A stronger 30B MoE reward model benefits training relative to a smaller 4B reward model.The comparison is shown with thick lines for the 30B MoE reward model and dashed lines for the 4B reward model.Source: Figure 7 · p. 22
• Providing specific rubrics, generic rubrics, or a reference solution leads to better scores across guidelines, whereas best-of-3 filtering yields no significant improvement.These effects are reported in the corresponding appendix figures.Source: Figure 9 · p. 22, Figure 10 · p. 22, Figure 12 · p. 22
• SFT appears to mimic Llama-4-Maverick’s style and wording, improving validation loss while degrading content and satisfying fewer rubric items relevant to the research goal.The summarized reference solution removes important details and rationale; the reference solution still outperforms the initial model but underperforms RL.Source: §B.3.2 · pp. 21, 23

• Providing both specific and generic rubrics leads to better scores across guidelines.Source: §B.3.4 · p. 23
• Specific + generic rubric optimization is preferred across judges and criteria except feasibility in preference evaluations on human annotation guidelines.This result is supported by both the preference-evaluation table and the accompanying summary.Source: Table 9 · p. 23, §B.3.4 · p. 23
• Providing the reference solution to the reward model θr improves scores across guidelines, while filtering provides only marginal benefit.Source: §B.3.4 · p. 23

B.4 ArXiv Subject-wise Performance Breakdown

• Finetuned models consistently outperform the initial Qwen-3-30B-A3B model across all arXiv subjects, averaged over three judges.The train and test split uses stratified sampling to ensure equal subject distribution.Source: §B.4 · p. 23
• MATH is the most challenging subject for all models, with significantly lower scores than CS and Q-BIO.Formal mathematical domains may benefit from training with verifiable rewards.Source: §B.4 · p. 23
• Med-Ft reaches 37.1% on Q-BIO, while ML-Ft reaches 35.4% on EESS and 35.7% on CS.These strengths likely reflect overlap between biological and biomedical literatures and close relationships between machine learning, EESS, and CS.Source: §B.4 · p. 23
• Improvements across subjects not explicitly targeted by domain-specific finetuning suggest broadly applicable research plan generation principles.This pattern indicates generalization beyond the domains used for finetuning.Source: §B.4 · p. 23

B.5 Analysis: Judge Agreements

• The guideline scores average rubric-item satisfaction across Claude-4-Sonnet, Gemini-2.5-Pro, and GPT-5-Thinking, with the same trends persisting when GPT-5-Thinking is excluded.Although GPT-5-Thinking also serves as one grader, the authors report that this has only a minor effect on the observed trends.Source: §B.5 · p. 23, Table 10 · p. 24

B.5 Analysis: Domain finetunes across general guidelines

• The domain-finetuned model outperforms the initial policy and Grok-4-Thinking on most general guidelines, while GPT-5-Thinking leads across the guidelines.Scores are broken down for ML, arXiv, and PubMed domain finetunes in Figures 13–15 and averaged across three judges.Source: §B.5 · p. 23, Figure 13 · p. 24, Figure 14 · p. 24
• The finetuned model is worse on cost and effort efficiency and equivalent on no ethical issues relative to the initial policy and Grok-4-Thinking.These exceptions are consistent with the overall-plan comparison.Source: §B.5 · p. 23

Human-Judge agreement

• LLM judges agreed with human-identified rubric violations 89–98% of the time, but agreed with human-identified satisfied items only 22–49% of the time.This asymmetry suggests the judges were systematically stricter than human experts, likely because they interpreted partial fulfillment differently.Source: §B.6.1 · pp. 24–25
• GPT-5-Thinking achieved the strongest overall alignment with human majority votes at 75.3% agreement (κ = 0.44), versus 73.9% for Gemini-2.5-Pro and 50.0% for Claude-4-Sonnet.The evaluation used 100 samples with three expert annotators; aggregated LLM majority voting reached 69.2% agreement (κ = 0.30).Source: §B.6.1 · p. 24, Table 11 · p. 25
• GPT-5-Thinking aligned most strongly across preference criteria, especially Predicted Outcomes (κ = 0.41) and Clear Execution (κ = 0.33).Agreement was higher for objective criteria such as Addresses Requirements (78–84%) than subjective criteria such as Clear Execution (50–66%); Claude-4-Sonnet had negative κ for some criteria.Source: §B.6.1 · p. 24, Table 12 · p. 25

Preference evaluation

• κ < 0.30 for preference-based evaluations, with GPT-5-Thinking versus Claude reaching κ = 0.12–0.30 and Gemini-involving pairs generally below κ < 0.15.Raw agreement was 40–87%, but finetuned models winning most comparisons produced high baseline agreement and limited κ signal.Source: §B.6.2 · p. 25

Domain and criterion effects

• ML has the most consistent inter-judge agreement, whereas ArXiv and PubMed show greater variance, including κ = 0.04–0.12 for pairs involving Claude.Across criteria, Feasibility has the highest κ (0.10–0.30), while Soundness and Addresses Requirements have lower κ despite high raw agreement.Source: §B.6.2 · p. 25

Rubric grading

• κ = 0.366–0.424 across domains for rubric satisfaction, with Table 14 showing moderate overall agreement among Claude-4-Sonnet, Gemini-2.5-Pro, and GPT-5-Thinking judges.Agreement is highest for objective criteria such as handles all criteria (κ ≈ 0.57–0.63) and lower for subjective criteria such as well justified rationale and no overlooked flaws (κ < 0.35).Source: §B.6.2 · p. 25, Table 14 · p. 26

Sources of variation

• The agreement patterns hold across Qwen, Llama, and Gemma, indicating that inter-judge agreement is primarily task- and domain-dependent rather than model-dependent.High raw agreement but κ < 0.25 for no ethical issues reflects the high prevalence of ethically sound plans.Source: §B.6.2 · pp. 25–26

B.7 Full Benchmark Evaluation Results

• GPT-5-Thinking is the top-performing model across all domains and judges, while finetuned models improve over base Qwen-3-30B by 3–7 percentage points.Domain-matched finetuning produces the largest gains; Medical scores are consistently higher, and Gemini-2.5-Pro is the most lenient judge.Source: §B.7 · p. 26, Table 16 · p. 27

C.1 Task Description

• Annotators compare two AI-generated research plans for a researcher’s goal, selecting a preference and scoring both plans with a brief justification.The goals include constraints and stated preferences, and the feedback is intended to improve LLMs as brainstorming partners for machine learning research.Source: §C.1 · p. 28, §C.1.1 · p. 28

C.1.1 The Task in a Nutshell

• The task presents the researcher’s goal and two proposed methodologies, followed by preference judgments and rubric-based checks.Annotators first compare the plans, then assess ten rubric items for suitability and whether each plan satisfies them.Source: §C.1.1 · p. 28

C Annotation Guidelines: Evaluating AI Generated Research Plans

• The protocol prohibits annotators from using LLMs, with annotations subject to removal if LLM use is detected.Warnings and prompt injections were added after an initial pilot caught an annotator using an LLM, alongside strict anti-LLM-use policies.Source: §C.1.1 · p. 28, C Annotation Guidelines: Evaluating AI… · p. 28
• The annotation guidelines instruct evaluators to judge plan quality for the supplied goal rather than the goal’s novelty or significance.Annotators should imagine carrying out the research themselves and choose which plan they would prefer to follow.Source: §C.2 · p. 28

C.2 Part 1: Comparing Research Plans

• Part 1 asks which plan better fulfills the stated research goal and would be preferable to follow as the evaluator’s own research plan.The evaluation focuses on the quality of execution for the given goal, not whether the underlying goal is novel or significant.Source: §C.2 · p. 28, §C.1 · p. 28

C.2.1 Criteria and Key Questions

• Plan comparisons assess requirements, soundness, clear execution, feasibility, predicted outcomes, and overall usefulness on a 1–10 scale.The criteria cover constraints and preferences, confounders and competing hypotheses, specificity and justification, unnecessary complexity, goal-relevant outcomes, and usefulness to an average ML graduate student.Source: §C.2.1 · p. 29

C.3 Part 2: Rubric-based grading

• For each rubric item, annotators rate its importance from 1 to 5 and check whether plans A and B satisfy it with Yes or No.They are instructed to judge strictly: any weakness relevant to an item means the plan is not satisfied, and Part 1 ratings must not be changed based on Part 2.Source: §C.3 · p. 29

C.4 Deliverables for each sample

• Each sample receives criterion-wise pairwise preferences, strict satisfaction checks for 10 rubric items, overall scores, justifications, annotator confidence, and rubric-item quality ratings.The annotation protocol covers 500 samples with three annotators each; justifications must include the phrase “light-weight to create”.Source: §C.4 · pp. 29–30

C.5.1 Overall score guidelines

• Plans are scored from 1 to 10, while confidence is scored from 1 to 5 according to annotator expertise, comprehension, and checking of technical details.The overall scale ranges from irrelevant or completely wrong at 1 to perfect at 10, with intermediate scores allowed; confidence ranges from an educated guess at 1 to absolute certainty at 5.Source: §C.5.1 · p. 30, §C.5.2 · p. 30

C.6 Guidelines for Part 1: Criteria-wise preference

• Criterion-wise preferences assess whether plans meet explicit and implicit requirements, remain technically and experimentally sound, support clear execution, remain feasible, and predict convincing outcomes.The criteria penalize missed constraints, unethical or costly steps, unsupported claims, contradictions, confounding, vague implementation details, unnecessary complexity, weak methods, missing controls, and reviewer-prone methodology.Source: §C.6.1 · p. 31, §C.6.2 · p. 31, §C.6.3 · p. 31

C.7.1 Rating quality of each rubric item from 1 to 5

• Rubric-item quality ratings distinguish contradictory or irrelevant checks, optional design choices, important components, and must-have components of the research goal.A rubric item is rated more highly when satisfying it is necessary for a plan to be judged good.Source: §C.7.1 · p. 32

C.7.2 Guidelines for checking whether a plan satisfies a rubric item

• A rubric item is satisfied only when the plan addresses what is required with specific how-and-why detail, justification, efficiency, ethical safety, no overlooked flaws, and consistency.All seven satisfaction conditions must hold, including consideration of the requirement, specificity, rationale, absence of weaknesses, cost and effort efficiency, no ethical or safety concerns, and consistency with the rest of the plan.Source: §C.7.2 · p. 32

D.1.1 Extracting Insights from Papers

• The pipeline extracts up to three novel insights from each paper, uses each insight to create a research scenario and grading rubric, and prompts models to generate concise, executable plans aligned with supplied documents and rubrics.The plan prompt requires addressing goals, constraints, and confounders while explaining how and why each step is needed; document-conditioned plans must follow the paper’s approach without omitting motivation or details.Source: §D.1.1 · p. 33, §D.1.2 · p. 33, §D.1.3 · p. 35

Output Format

• Generated plans must be information-dense, present-tense solutions inside <solution> tags, contain no claimed experimental results, and remain within 750 words.The prompt additionally requires human-readable plans that explain what, how, and why without directly mentioning rubric items or adding self-praise.Source: §D.1.3 · p. 35, Output Format · p. 35

D.1.4 Guideline Verification for Research Goal and Rubric

• A separate expert-evaluation prompt checks generated scenarios and rubrics against specified guidelines using the corresponding insight and generated grading rubric.The verification prompt presents the insight and enumerated rubric items for guideline-level judgment.Source: §D.1.4 · p. 36

D.2.1 Rubric Judge

• The rubric judge evaluates each criterion by checking completeness, implementation specificity, overlooked flaws, rationale, efficiency, ethics, and consistency with the overall plan.The judge is instructed to be skeptical and identify violated desiderata, assigning all seven violations when the plan does not address an item.Source: §D.2.1 · p. 38

D.2.2 Preference Judge

• The preference judge compares randomized plan pairs on requirements, soundness, execution clarity, feasibility, and predicted outcomes, then assigns each plan an overall 1–10 usefulness score.The predicted-outcomes criterion is tied to metrics relevant to the specific research goal and its stated preferences or constraints.Source: §D.2.2 · p. 39

Scenario

• The medical example asks for a fully automated, scalable framework that continually refines tool documentation from LLM–tool interactions as tools evolve.The scenario requires closed-loop feedback, trial-and-error exploration, diverse usage coverage, adaptive stopping, complex-parameter handling, and cross-model generalization.Source: Scenario · p. 41, Grading Rubric · p. 41

Grading Rubric

• The rubric requires the plan to satisfy those scenario constraints, including active trial-and-error, diverse exploration, adaptive stopping, and prevention of overfitting.It also tests whether proposed grading items are important, challenging, unambiguous, plan-focused rather than outcome-focused, and nonredundant.Source: §D.1.4 · p. 36, Grading Rubric · p. 41

E.1.1 Initial vs ML Finetune

• The finetuned model produced a far more detailed solution that satisfied more rubric items, although all three frontier graders identified shortcomings; Gemini-2.5-Pro was more lenient than the others.The qualitative comparison included full grading outputs from all three frontier models.Source: §E.1.1 · p. 41

Model Response: Qwen-3-30B (Base)

• The base model’s Dynamic Documentation Refinement framework logs interactions, detects documentation mismatches, proposes revisions, validates them, versions updates, and injects current documentation into inference.A distributed workflow runs across hundreds of tools, with monitoring, event-triggered updates, and safeguards for false positives, distribution shifts, and tool evolution.Source: Model Response: Qwen-3-30B (Base) · p. 42
• The judge fully satisfied automation, iterative updating, documentation-error correction, and dynamic maintenance, but scored scalability 3/7 because resource-optimization details and cost justification were insufficient.The scalability concern centers on the overhead of real-time embedding computations and continuous monitoring.Source: Model Response: Qwen-3-30B (Base) · p. 43
• The judge scored trial-and-error and diverse exploration 3/7, and found no principled alignment-based stopping mechanism to prevent overfitting.The plan passively monitors existing interactions, omits active experimentation and coverage mechanisms, and relies on multiple observations without specifying adaptive version-alignment or stopping criteria.Source: Model Response: Qwen-3-30B (Base) · p. 44
• The base framework supports iterative documentation updates and dynamic synchronization, but reviewers find its automation and scalability claims undermined by human annotation, instrumentation effort, and computational costs.It receives full satisfaction for iterative updating and dynamic maintenance, while automation and scalability receive conflicting assessments because of the human-trained classifier, tool instrumentation, and expensive processing components.Source: Model Response: Qwen-3-30B (Base) · pp. 45, 46, 48
• The base response is primarily reactive: it lacks active trial-and-error, diverse exploration, adaptive stopping, systematic combinatorial testing, and validated cross-model generalization.Its reliance on historical logs can miss rare scenarios, reinforce narrow usage patterns, permit endless refinements, and overfit documentation to one model’s behavior.Source: Model Response: Qwen-3-30B (Base) · p. 49

Model Response: Qwen-3-30B (Finetuned)

• DocRefine instruments tool interactions, builds usage corpora, detects anomalous behavior, generates feedback-aware revisions, and validates updates through A/B testing with explicit confidence and performance thresholds.Updates require >0.85 confidence and >10% metric improvement, while rollout requires >5% success improvement and bootstrapped p-value < 0.05.Source: Model Response: Qwen-3-30B (Finetuned) · p. 50
• The finetuned design extends refinement across tools and changing APIs using knowledge transfer, schema validation, versioned snapshots, drift triggers, containerized orchestration, and rollback-capable auditing.A cross-tool knowledge graph transfers fixes, while event-driven infrastructure and autoscaling support automated refreshes when schemas or usage patterns change.Source: Model Response: Qwen-3-30B (Finetuned) · p. 50
• Judges consistently credit the framework with full automation and iterative refinement driven by LLM tool-interaction feedback.The design uses Kubernetes and event-driven orchestration, a three-stage refinement loop, telemetry, documentation revision, validation, and rollout without human annotation.Source: Model Response: Qwen-3-30B (Finetuned) · pp. 50, 51, 53, 54, 56
• The framework effectively addresses documentation inconsistencies, incompleteness, inaccuracies, and evolving tool functionality through anomaly detection, RAG-based revision, validation, versioning, and telemetry-diff triggers.These mechanisms ground updates in observed LLM behavior and can refresh documentation when API schemas or behavioral drift change.Source: Model Response: Qwen-3-30B (Finetuned) · pp. 50, 51, 57
• Scalability remains questionable because per-tool agents, continuous telemetry, clustering, knowledge-graph maintenance, and A/B testing may substantially increase resource consumption.Judge scores range from 5/7 to 2/7 to 6/7 across assessments, with concerns about missing optimization strategies and heavyweight infrastructure.Source: Model Response: Qwen-3-30B (Finetuned) · pp. 51, 52, 54, 57
• The response inadequately specifies diversity mechanisms, alignment-based stopping rules, systematic parameter-combination coverage, and modeling of rare or non-outlier errors.Judges note reliance on an LLM simulator without explicit diversity constraints, cease criteria, combinatorial or constraint-based exploration, or comprehensive error modeling.Source: Model Response: Qwen-3-30B (Finetuned) · pp. 52, 53, 55, 56
• Cross-model generalization is weakly supported because the plan lacks explicit evaluation, adaptation, and safeguards against overfitting documentation revisions to a single simulator model.The proposed modular interface and canonical ontology do not establish how different LLM capabilities or simulator-specific biases would be handled.Source: Model Response: Qwen-3-30B (Finetuned) · pp. 53, 56
• The plan scores 7/7 for trial-and-error learning and 4/7 for exploration diversity, while its continuous loop lacks a convergence-based termination condition.It uses a usage simulator, feedback tuples, and a three-stage refinement loop, but completely fails to address exploration diversity.Source: Model Response: Qwen-3-30B (Finetuned) · pp. 57–58
• The finetuned Qwen plan scores 5/7 for complex parameter handling and 4/7 for cross-model generalization because its simulator and distilled Llama-3-8B focus are underspecified or model-specific.Telemetry and anomaly detection address error sources, but the plan does not ensure combinatorial coverage or effectiveness across GPT-4, Claude 3, and Gemini.Source: Model Response: Qwen-3-30B (Finetuned) · p. 58

Model Response: GPT-5-Thinking

• GPT-5-Thinking proposes AutoDocRefiner, which instruments tool use, infers documentation gaps, synthesizes evidence-linked patches, validates them with synthetic tasks and replays, and gates deployment through shadow or A/B testing.Its architecture includes a canonical Tool Spec Schema, telemetry, analysis, patch synthesis, validation, release management, rollback, drift monitoring, and model-family-specific renderers.Source: Model Response: GPT-5-Thinking · pp. 58–60

4. Scalability and Maintenance

• Across the proposed frameworks, validation and maintenance rely on synthetic testing, A/B or shadow deployment, version control, drift detection, and periodic or continuous feedback cycles.The designs also track success rates, errors, latency or calls, documentation coverage, and related tool-use efficiency metrics.Source: Model Response: GPT-5-Thinking · p. 60, §3 · p. 62, §4 · p. 62

Model Response: Gemini-2.5-Pro

• Gemini-2.5-Pro frames the goal as an automated loop that detects documentation-attributable failures, diagnoses them, revises structured documentation, and evaluates task success and efficiency against static documentation.Its ADR-Loop parses documentation, monitors traces, uses verifier and diagnoser models, applies targeted revisions, and periodically health-checks evolving tools.Source: Model Response: Gemini-2.5-Pro · pp. 60–61, §3 · p. 61

5. Evaluation Methodology

• The reported fully satisfied-item counts are 3/10 for Claude-4-Sonnet, 1/10 for GPT-5-Thinking, and 5/10 for Gemini-2.5-Pro.The evaluation compares the three frontier-model responses on the listed criteria.Source: §5 · p. 62

Model Response: Grok-4

• Grok-4’s DocRefine structures human-oriented documentation, simulates diverse LLM-tool interactions, logs outcomes, and uses automated analysis to prioritize targeted updates at scale.The design uses JSON schemas, generated common and edge-case tasks, parallel cloud execution, and LLM-based classification of inconsistencies, omissions, and outdated functionality.Source: Model Response: Grok-4 · p. 64
• Grok-4 proposed a continuous automated loop that simulates LLM-tool interactions, analyzes failures, refines documentation, monitors tool changes, and scales through modular parallel pipelines.Its reported fully satisfied counts were 5/10 for Claude-4-Sonnet, 3/10 for GPT-5-Thinking, and 5/10 for Gemini-2.5-Pro.Source: Model Response: Grok-4 · pp. 64–65

Model Response: Llama-4-Maverick

• Llama-4-Maverick organized refinement into interaction monitoring, documentation refining, and validation components, using distributed computing and automated testing to keep documentation accurate and scalable.Its fully satisfied counts were 2/10 for Claude-4-Sonnet and 0/10 for both GPT-5-Thinking and Gemini-2.5-Pro.Source: Model Response: Llama-4-Maverick · pp. 65–66

E.1.3 Evolution Over Steps

• Later finetuning steps introduced superfluous hallucinated result numbers, suggesting the frozen initial-model grader rewarded specificity that reduced held-out Claude-4-Sonnet performance.The main model stopped at step 100, while outputs from steps 40, 180, and 260 were compared.Source: §E.1.3 · p. 66

Model Response: Step 40

• Step 40’s ToolRefine uses structured execution traces, discrepancy classification, LLM-generated edits, regression checks, triage, versioning, and automated deployment to adapt documentation without routine human intervention.It distinguishes parameter misinterpretation, functional deviation, and silent failure, and reports 3/10 fully satisfied items for Claude-4-Sonnet.Source: Model Response: Step 40 · pp. 66–67

Model Response: Step 180

• Step 180’s DocRefine adds consensus-gated reversible patches, modular knowledge-graph updates, drift detection, asynchronous processing, and safeguards against hallucination and feedback bias.Its proposed evaluation includes 41% average documentation-accuracy gain over six months, 2.3× fewer misuse errors, and 92% recovery of true gaps under 15% synthetic noise.Source: Model Response: Step 180 · pp. 67–68

Model Response: Step 260

• 34% lower tool failure rates and 41% faster mean time-to-success were reported for TDL across 120 tools and 30M+ LLM calls, with significant ablation and human-evaluation gains.The framework also recovered 89% of injected documentation flaws versus 41% for an LLM-only baseline, and its documentation was rated 1.8× more actionable than static documentation.Source: Model Response: Step 260 · p. 69

Model Response: ArXiv Finetune

• The ArXiv-finetuned DocRefine combines a semantic knowledge graph, trace-driven discrepancy diagnosis, multi-criteria edit validation, adaptive triggering, adversarial testing, and self-supervised policy improvement.It evaluates task success, failed-call reduction, and documentation alignment across more than 200 tools against static and rule-based documentation updates.Source: Model Response: ArXiv Finetune · pp. 69–70

Model Response: Medical Finetune

• The Medical-finetuned ToolRefine models documentation as a causally annotated executable knowledge graph and proposes anomaly detection, intervention-based root-cause reasoning, grounded patches, and contract, semantic, and adversarial validation.Its design uses execution traces containing prompts, actions, outcomes, errors, timing, and post-hoc LLM judgments to identify undocumented dependencies and constraints.Source: Model Response: Medical Finetune · p. 71
• The framework addresses documentation drift, hallucinations, anomaly-detection false positives, and scalability through versioning, retrieval augmentation, validation, and adaptive sampling.Source: Model Response: Medical Finetune · p. 71
• ToolRefine combines a causally grounded knowledge graph, closed-loop feedback, and modular agents to support evolving, autonomous tool documentation.Source: Model Response: Medical Finetune · p. 72
• Fully satisfied items were 2/10 for Claude-4-Sonnet, 3/10 for GPT-5-Thinking, and 2/10 for Gemini-2.5-Pro.Source: Model Response: Medical Finetune · p. 72

Model Response: Qwen-3-4B (Base)

• The base-model experiment controls token count, data distribution, architecture, optimization, and preprocessing while evaluating accuracy, reasoning correctness, error correction, statistical significance, mixture ratios, and internal representations.Retry data contains plausible inserted mistakes and corrections, and the design includes cross-validation, loss monitoring, t-tests, and ablations over error-free/retry proportions.Source: Model Response: Qwen-3-4B (Base) · p. 73

Model Response: Qwen-3-4B (SFT)

• The SFT design generates structured mistake-correction pairs with Mistral-7B, matches retry and error-free token counts, and tests validation, generalization, significance, and error-type differences.It also examines whether gains arise from retry data itself or from the model used to generate it.Source: Model Response: Qwen-3-4B (SFT) · pp. 73–74

Model Response: Qwen-3-4B (RL)

• The RL design uses matched 10B-token baseline and retry regimes with controlled problem distributions, training settings, hardware, and held-out evaluation of correctness, reasoning fidelity, and error identification.It additionally specifies no-correction, synthetic-noise, and answer-only controls, teacher verification, repeated trials, cross-validation, and sensitivity analyses.Source: Model Response: Qwen-3-4B (RL) · pp. 74–75
• +1.1% accuracy (p=0.01) occurs on medium-difficulty problems with conceptually grounded, trace-aligned retry data and no loss in final-answer correctness; low-difficulty performance requires filtering.Unfiltered retry data reduced low-difficulty accuracy by 0.8% (p=0.03), while conceptual-error filtering improved the accuracy gap by +1.3% (p=0.007) and BLEU-4 coherence from 31.2 to 38.7.Source: Model Response: Qwen-3-4B (RL) · p. 75

Research Goal

• CECL proposes dual-level contrastive learning over ensemble and channel representations, using pretrained temporal embeddings, augmentations, subject-aware channel handling, and downstream decoding feedback.The loss combines ensemble and channel objectives with a learnable weighting coefficient and avoids reconstructive input modeling.Source: Research Goal · p. 76

Finetuned (ML) Model Output

• The finetuned CECL plan adds learnable subject-specific channel embeddings, hierarchical positive and negative sampling, and a weighted proxy-task loss to make representations structurally robust and predictive.It specifies subject-independent evaluation on four decoding benchmarks, AdamW optimization, and ablations removing channel loss, reconstruction alternatives, fixed embeddings, or structured negatives.Source: Finetuned (ML) Model Output · pp. 77–78

Human Annotations (Part 1)

• Human annotators preferred Plan B’s finetuned CECL approach, scoring it 9 versus 6, 7, and 8 for Plan A across three annotations.They credited its channel-embedding matrix, end-to-end decoding loss, optimization details, and stronger feasibility, while noting risks of downstream overfitting and difficult manifold training.Source: Human Annotations (Part 1) · p. 78

GPT-5-Thinking Model Output

• The GPT-5-Thinking output frames the neural-data objective as learning discriminative, downstream-useful representations that capture ensemble- and channel-level information across heterogeneous electrode configurations.This restates the target that motivates the hierarchical contrastive-loss design.Source: GPT-5-Thinking Model Output · pp. 78–79
• The representation-learning design combines multi-view augmentation, temporal-embedding positive mining, ensemble and channel contrastive losses, relational objectives, predictive coding, and adversarial subject regularization.Permutation-invariant aggregation, channel metadata, channel dropout, memory queues, and leakage prevention target varying electrode configurations and cross-subject robustness.Source: GPT-5-Thinking Model Output · pp. 79–80

Research Goal

• The proposed decomposition framework identifies behaviorally meaningful latent directions through clustering, causal interventions, probing, cross-layer checks, sparsity analysis, and natural-language labeling.A behavioral attribution graph links unexpected behaviors to directions, layers, strengths, and semantic labels, supporting scalable analysis beyond polysemantic neurons.Source: Research Goal · p. 81

Finetuned (ML) Model Output

• The finetuned method uses structured sparse autoencoders, linguistic priors, causal ablations, human evaluation, and cross-instance consistency to recover sparse, disentangled semantic units.The framework builds causal attribution graphs and reports improved interpretability and behavior-modeling accuracy when decomposed units replace polysemantic neurons.Source: Finetuned (ML) Model Output · p. 82
• Rubric satisfaction scores were 3/10 for Claude-4-Sonnet, 5/10 for GPT-5-Thinking, and 3/10 for Gemini-2.5-Pro on the finetuned ML output.For the GPT-5-Thinking output, the reported score was 3/10 for Claude-4-Sonnet, 5/10 for GPT-5-Thinking, and 3/10 for Gemini-2.5-Pro.Source: Finetuned (ML) Model Output · p. 82

Human Annotations (Part 1)

• Human annotators generally preferred Plan B, rating it 7–9/10 versus 5–7/10 for Plan A because structured sparsity, causal testing, and consistency checks improved clarity and reliability.Annotators cited Plan A’s assumptions about clustering and weaker causal validation, while noting Plan B required more computation and setup.Source: Human Annotations (Part 1) · p. 83

Finetuned (ML) Model Output

• A content-aware latent diffusion baseline uses adversarial and consistency constraints, multiple samples, and aggregation to reduce attribution bias and stochasticity.It is evaluated through bias ablations, perturbation stability, downstream pruning fidelity, cross-dataset tests, and expert judgments.Source: Finetuned (ML) Model Output · pp. 86–87
• 5/10 rubric items were fully satisfied by Gemini-2.5-Pro, compared with 1/10 for Claude-4-Sonnet and 0/10 for GPT-5-Thinking.Source: Finetuned (ML) Model Output · p. 87

GPT-5-Thinking Model Output

• A class-neutral counterfactual baseline stays on the data manifold, suppresses the predicted class while preserving non-target evidence, and supports integrated gradients along a manifold path.The design combines classifier neutralization, perceptual proximity, smoothness, and optional generative-prior losses, with amortized generation as a scalable alternative.Source: GPT-5-Thinking Model Output · pp. 87–88
• 6/10 rubric items were fully satisfied by Gemini-2.5-Pro, compared with 0/10 for both Claude-4-Sonnet and GPT-5-Thinking.Source: GPT-5-Thinking Model Output · p. 89

Finetuned (ArXiv) Model Output

• The ArXiv output proposes exact-amplitude, gauge-invariant QED showering for ISR, FSR, and intermediate-W radiation, with Sudakov evolution, hybrid dipoles, and coherent QCD merging.It validates photon distributions, correlations, multiplicities, and spin effects against exact predictions and existing generators.Source: Finetuned (ArXiv) Model Output · pp. 91–92
• 6/10 rubric items were fully satisfied by Claude-4-Sonnet, compared with 5/10 for Gemini-2.5-Pro and 4/10 for GPT-5-Thinking.Source: Finetuned (ArXiv) Model Output · p. 92

Research Goal

• The research goal is to estimate whether Italy’s 2018 hiring incentive produced medium-term employment stability and career effects using longitudinal Tuscany data.The policy targeted conversion of temporary contracts to permanent employment and was temporarily extended to workers under 35.Source: Research Goal · p. 94
• Claude-4-Sonnet and Gemini-2.5-Pro fully satisfied 3/10 items, while GPT-5-Thinking satisfied 0/10 for the research plan evaluating Italy’s 2018 hiring incentive.The plan matches treated and comparison workers, estimates staggered DiD effects on employment stability and career outcomes, and adds synthetic-control, subgroup, event-study, and sensitivity analyses.Source: Research Goal · pp. 94–95

Finetuned (ArXiv) Model Output

• Claude-4-Sonnet and Gemini-2.5-Pro each fully satisfied 7/10 items, while GPT-5-Thinking satisfied 5/10 for the finetuned ArXiv policy plan.The plan uses longitudinal Tuscany employment data, staggered difference-in-differences, propensity-score matching, survival and panel models, instrumental variables, synthetic controls, and extensive falsification and sensitivity tests.Source: Finetuned (ArXiv) Model Output · pp. 95–96

Initial Model Output (Qwen-3-30B)

• Claude-4-Sonnet, GPT-5-Thinking, and Gemini-2.5-Pro fully satisfied 9/10, 7/10, and 8/10 items, respectively, for the initial tidal-current acoustic-mitigation plan.The plan characterizes acoustic proxies, varies MPPT coefficients and switching frequencies, evaluates adaptive MPPT and FM-PWM strategies, and optimizes them subject to maintaining at least 95% of baseline power.Source: Initial Model Output (Qwen-3-30B) · pp. 98–99

Finetuned (ArXiv) Model Output

• Claude-4-Sonnet, GPT-5-Thinking, and Gemini-2.5-Pro fully satisfied 6/10, 4/10, and 2/10 items, respectively, for the finetuned tidal-current plan.It tests adaptive MPPT, variable-frequency switching, harmonic injection, and MPC across stochastic flows, reporting a recommended VFS-plus-harmonic-injection strategy with 28% lower acoustic power index and 1.2% power loss.Source: Finetuned (ArXiv) Model Output · p. 100

GPT-5-Thinking Model Output

• GPT-5-Thinking scored 8/10 on fully satisfied items, versus Gemini-2.5-Pro’s 10/10, while proposed controls combine MPPT smoothing, virtual inertia and damping, blade-passing-frequency rejection, and current-control measures.The cited measures include low-pass MPPT updates with a 0.2–1 Hz cutoff, torque/speed-reference ramp limits, repetitive or internal-model notch control, SVM, dead-time and cross-coupling compensation, and spread-spectrum PWM of ±10–20%.Source: GPT-5-Thinking Model Output · pp. 101–102

Initial Model Output (Qwen-3-30B)

• The initial BanLI plan scored 3/10, 0/10, and 4/10 across Claude-4-Sonnet, GPT-5-Thinking, and Gemini-2.5-Pro, using genomic validation, transformation, phage, methylation, and transcriptomic assays.The integrated design tests whether incomplete BanLI restricts foreign DNA, protects against phage, methylates host DNA, or triggers compensatory defense mechanisms.Source: Initial Model Output (Qwen-3-30B) · p. 103

Finetuned (Medical) Model Output

• The finetuned BanLI plan scored 6/10, 5/10, and 5/10 across Claude-4-Sonnet, GPT-5-Thinking, and Gemini-2.5-Pro, respectively, while integrating genetic, methylomic, phage, and evolutionary tests.It prioritizes REase activity and in vivo phage resistance, uses SMRT sequencing and transformation assays, and models alternative methyltransferases, compensatory defenses, fitness costs, and ecological effects.Source: Finetuned (Medical) Model Output · pp. 104–105

Research Goal

• The extracellular-vesicle plan compares sEV DNA with cfDNA using matched plasma processing, size-exclusion isolation, low-input whole-genome sequencing, and fragment-length bioinformatics.It proposes 30 lung-cancer patients and 30 matched controls, with contamination controls and sequencing at 30–50 million 150-bp paired-end reads per sample.Source: Research Goal · pp. 107–108
• The research goals required robust comparison of plasma sEV DNA with cfDNA in lung cancer and improved TB case detection through more sensitive and specific diagnostics.The sEV goal emphasized genomic composition, mutation burden, contamination control, and statistical comparison; the TB goal focused on smear-negative misclassification and an implementable improvement plan.Source: Research Goal · pp. 108, 111

Finetuned (Medical) Model Output

• Finetuned medical outputs achieved 8/10, 9/10, and 9/10 rubric satisfaction across the reported evaluator scores, exceeding the initial model’s 3/10, 1/10, and 4/10.The scores were reported for Claude-4-Sonnet, GPT-5-Thinking, and Gemini-2.5-Pro, respectively.Source: Finetuned (Medical) Model Output · pp. 109, 111, 113, Initial Model Output (Qwen-3-30B) · p. 112
• The finetuned sEV-DNA plan compares matched sEV and cfDNA while controlling contamination and sequencing confounders, validating results with orthogonal assays and defining a tumor-linked sEV DNA fingerprint.It distinguishes vesicle-internal from external DNA and interprets nucleosomal periodicity or uniform fragmentation mechanistically.Source: Finetuned (Medical) Model Output · pp. 109, 111

Initial Model Output (Qwen-3-30B)

• The initial Qwen-3-30B TB plan used four phases—baseline assessment, comparative diagnostic evaluation, impact modeling, and implementation strategy development—to evaluate alternatives to smear microscopy.It compared Xpert MTB/RIF and LFIA against a culture-plus-Xpert reference standard and modeled detection, delays, costs, and feasibility.Source: Initial Model Output (Qwen-3-30B) · pp. 111–112

Finetuned (Medical) Model Output

• The finetuned TB plan proposes tiered reflex-Xpert diagnosis, reporting that Xpert increases sensitivity from 58% to 92% and reduces false negatives by 67%.The implementation strategy includes technician training, rural sample transport, digital reporting, phased piloting, monitoring, and eventual integration into national guidelines.Source: Finetuned (Medical) Model Output · p. 113
• 5/10, 5/10, and 7/10 were the Fully Satisfied scores assigned by Claude-4-Sonnet, GPT-5-Thinking, and Gemini-2.5-Pro to another finetuned medical output.That output proposes a powered comparison of smear microscopy and Xpert Ultra for pulmonary tuberculosis, including sensitivity, specificity, time-to-treatment, cost, bias control, and implementation monitoring.Source: Finetuned (Medical) Model Output · pp. 114–115
• The H1 transcriptomics plan uses axenic coriander, live H1, heat-killed H1, and cell-free supernatant controls across root and shoot time courses to distinguish early signaling from sustained growth responses.It specifies 96 RNA-seq libraries, four sampling times, four biological replicates, and mechanistic controls at 24 hours and 3 days.Source: Finetuned (Medical) Model Output · p. 118
• The plan integrates RNA-seq, differential-expression and enrichment analyses, WGCNA, hormone and metabolite assays, pharmacological tests, and qRT-PCR to identify and validate molecular drivers of H1-mediated growth.Key hypotheses cover hormone signaling, nutrient acquisition, cell-wall remodeling, redox and defense responses, and sugar metabolism.Source: Finetuned (Medical) Model Output · p. 119
• 6/10, 6/10, and 9/10 were the Fully Satisfied scores assigned by Claude-4-Sonnet, GPT-5-Thinking, and Gemini-2.5-Pro to a finetuned transcriptomics output.The output delivers a catalog of time-resolved differentially expressed genes, coexpression modules linked to colonization and growth, and separation of live-cell, secreted-factor, and MAMP-driven responses.Source: Finetuned (Medical) Model Output · p. 119