PaperlayerView exact revision on arXiv

Source-linked AI summary

Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence

Brian Wang, Bin Feng, Xiaoman Pan, Chenyang An, Felix Liu, Tangqi Fang, Gongbo Sun, Lingfeng Shen, Ning Wang, Handuo Zhang, Feng Chen, Fuchao Yang, Xiang Wang, Jiacheng Lin, Siting Li, Zixuan Liu, Chi Han, Zhenhailong Wang, Kunlun Zhu, Lawrence Zhao, Yueqi Guo, Kailong Wen, Feng Xing, Yiling Guo, Lidong Bing, David Tan, Bo An, Heng Ji, Sheng Wang

arXiv:2608.11341v1cs.AI

TL;DR

Consequential real-world challenges often lack executable, verifiable objectives and success criteria. Apodex Discovery provides environments and process evaluation for extended investigations. Its biomedical environment improved mean normalized prediction scores for GPT-5.5 and GPT-5.6-sol by 2.5 and 7.6 points, respectively, over the same closed-book backbone.

  • Problem

    Frontier models can solve tasks with specified objectives and success criteria, but consequential real-world challenges rarely arrive in executable or verifiable form.

    Source: §1 · p. 4

  • Method

    Apodex combines systematic problem selection, executable environment-task-episode workflows for complete solver systems, and HDS6 process evaluation of investigations.

    Source: §6 · p. 42

  • Results

    2.5 and 7.6 points: the biomedical environment improved mean normalized prediction scores for GPT-5.5 and GPT-5.6-sol, respectively, over the same closed-book backbone; Apodex also exceeded prior human baselines on the most demanding AAV capsid-design tasks.

    Source: §6 · p. 42

  • Takeaways & Limitations

    The framework shifts evaluation from whether a model gives the right answer to whether a solver conducts a reliable, evidence-grounded, self-correcting investigation.

    Source: §6 · p. 42

    The benchmark currently covers 17 executable environments and 218 episodes, representing only a subset of the 20 selected problems and 423-problem registry.

    Source: §6 · p. 42

Abstract

from arXiv · show

Apollo did not reach the Moon merely because its engineers could solve difficult equations. It succeeded by turning a distant ambition into a mission architecture of explicit objectives, simulation, verification, and repeated correction. AI now faces a similar transition: frontier models can solve difficult tasks once the problem, tools, and success criteria are specified, yet consequential real-world challenges rarely arrive in an executable or verifiable form. We introduce Apodex Discovery, a framework for building and evaluating discoverative AI through the heavy-duty solver, a system comprising a foundation model, harness, tools, and control policies that pursues extended, stateful, verifiable investigations. It has three core components. First, a problem-scouting process surveyed 561 industries across 16 sectors, assembled 423 high-value real-world problems, and selected 20 for the initial release. Second, a common environment-task-episode abstraction provides data, tools, constraints, feedback, trajectory recording, and verification of intermediate artifacts and final submissions. Third, HDS6 evaluates Tools, Repair, Alternatives, Coherence, Evidence, and Scope independently of final-task success. In AAV capsid design, Apodex surpassed the published state of the art by 7% across viability, tropism, structure prediction, and generative design. In drug repurposing and reformulation, a task-specific biomedical environment improved the mean normalized prediction score of GPT-5.5 and GPT-5.6-sol by 2.5 and 7.6 points over the same closed-book backbone. Controlled ablations show that the fixed TRACES episode interface enables attribution of performance differences to specific solver components. Apodex Discovery moves AI evaluation beyond predefined benchmarks toward verifiable investigations aimed at genuine discovery.

1. Introduction

Apodex Discovery turns open-ended ambitions into stateful, verifiable investigations by evaluating complete solver systems and their trajectories, not only final answers. Its TRACES benchmark and HDS6 process metric support comparison and diagnosis across real-world problem environments.

1. Introduction

  • Apodex addresses the gap between AI capability on specified tasks and consequential ambitions lacking explicit objectives, environments, and success criteria, targeting investigations that can be checked and improved.Static benchmarks lack persistent environments and action, while interactive benchmarks typically reduce narrowly specified tasks to final success; the proposed evaluation unit is a complete episode of tool-using, evidence-seeking, verifiable, self-correcting problem solving.Source: §1 · pp. 4–5
  • Discoverative AI aims to reach unknown conclusions through a loop of hypothesis, action, verification, and revision, supported by problem formulation, reality-based environments, verification, and repair.The paper releases TRACES, a benchmark of executable environments, episodes, and tasks with hidden verifiers.Source: §1 · pp. 4–5

1.1. What is a Heavy-Duty Solver?

  • A heavy-duty solver is a foundation model with its harness, tools, memory, and control policy, sustaining a long, stateful, self-correcting investigation against an external world.It operates using a problem manifest, a reality-based environment, verification, and repair infrastructure.Source: §1.1 · p. 5

1.1.1. Problem Manifest

  • A problem manifest fixes the problem’s objectives, task decomposition, outcomes and hidden verifier, permitted data and tools, and resource budgets; runnable episodes instantiate it across cases.Intermediate load-bearing results can be checked where they arise, and episodes share the manifest’s tools, verifier, and scoring rule while carrying distinct instance data and hidden ground truth.Source: §1.1.1 · p. 6

1.1.2. Reality-Based Environment

  • A reality-based environment is a stateful, interactive substrate that returns fresh observations as the solver acts, while isolation protects hidden ground truth and a fixed episode interface enables component attribution.The complete system—not the foundation model alone—is evaluated, and identical interfaces permit comparisons of models or harnesses while holding the other components fixed.Source: §1.1.2 · pp. 6–7

1.1.3. Verification Mechanisms

  • Outcome verification scores final and load-bearing intermediate outputs against hidden criteria or measurable proxies, while HDS6 independently assesses trajectories across Tools, Repair, Alternatives, Coherence, Evidence, and Scope.Outcome and process verification are complementary: outcome scores indicate success, while process scores assess whether the investigation was reliable and evidence-grounded, including when definitive ground truth is delayed or unavailable.Source: §1.1.3 · pp. 7–8

1.1.4. Repair Loops

  • Repair loops return verification feedback so solvers can localize failures, correct errors, and re-check fixes within an episode or across attempts.Across attempts, structured repair notes identify capability improvements and faulty steps without revealing the outcome, hidden ground truth, or score.Source: §1.1.4 · p. 8
  • 2.5 and 7.6 points: the biomedical environment improved GPT-5.5 and GPT-5.6-sol mean normalized prediction scores over the same closed-book backbone; Apodex exceeded published AAV state of the art.The AAV comparison covered viability, tropism, structure prediction, and generative design. A two-month, ten-person scouting effort surveyed 561 industries across 16 sectors, assembled 423 high-value problems, and selected 20 for the initial TRACES benchmark; controlled ablations support attribution to solver components.Source: §1.1.4 · p. 8

2. Scouting High-Value Real-World Problems

Apodex Discovery uses a structured, expert-reviewed process to identify consequential real-world problems that demand deep reasoning and can be evaluated through verifiable work. It narrows a broad survey to an initial set of executable investigations, including a prospective drug-repurposing challenge whose conclusions can be tested as evidence emerges.

2.1. Domain Selection

  • The team surveyed 561 industries across 16 sectors, assessing technical fit, value, and commercial access independently, with deep reasoning and tool use capping domain scores.Quantitative scores narrowed the search, while human reviewers selected ten domains across eight sectors, with at most two from any one sector.Source: §2.1 · pp. 9–10

2.2. Question Collection and Screening

  • Candidates were treated as prospective environments, requiring concrete objectives and artifacts, substantive intermediate tasks, measurable verification, runnable instances, evidence sources, and consequential beneficiaries.Problems relying solely on unstructured expert judgment were rejected; claims and outcomes needed enough observable structure for independent inspection and possible falsification.Source: §2.2 · p. 11

2. Scouting High-Value Real-World Problems

  • The scouting process produced 423 high-value problems and selected 20 for the initial TRACES release across frontier-model development, biomedical discovery, clinical translation, scientific engineering, and deployment-oriented intelligence.Candidates were reviewed for reasoning depth, technical feasibility, verifiability, data and tool availability, verification latency, and real-world impact.Source: §2 · p. 9, §2.2 · pp. 11–12

2.3. Reality-Facing Problems in the Initial Benchmark

  • The initial release covers five problem families—AAV capsid design, drug repurposing and reformulation, clinical-trial analysis and translation, LLM data, and LLM training and systems—as extended investigations with inspectable artifacts and external outcomes.Tasks combine long-horizon work, heterogeneous tools, intermediate verification, and outcomes that can ultimately confirm or falsify solver conclusions.Source: §2.3 · pp. 12–13

2.4. Discovery Challenges

  • The first discovery challenge asks solvers to rank compounds from causal survival-screen signatures, defend mechanisms with auditable evidence, and assess transfer to an independent disease screen.For the Parkinson’s disease-model instance, predictions are scored against held-out outcomes and sealed evidence; top candidates may advance to a survival assay whose outcome is not yet known.Source: §2.4 · pp. 13–14

3. Process Verification and Reality-Based Environment

Apodex Discovery makes investigations verifiable through complementary outcome and process checks, then packages environments, tasks, episodes, and verifiers into a runnable, reproducible, repairable evaluation framework.

3.1. Verification

  • Outcome verification scores the final submission against hidden criteria, while process verification judges the recorded trajectory without seeing the outcome or solver identity.Outcome scores measure success but can be sparse or delayed; blind process diagnoses can provide finer-grained evidence about how the solver worked and return repair notes without revealing ground truth.Source: §3.1 · pp. 14–15, Figure 3 · p. 15

3.1.1. Outcome Verification

  • Outcome checks score final submissions and, for decomposed problems, verify intermediate milestones; where definitive outcomes are unavailable, measurable proxies can stand in.Scores are normalized from 0 to 1 between instance-specific trivial baselines and known best results. Explicit leakage checks act as hard gates in environments vulnerable to evaluation-data gaming, disqualifying violations.Source: §3.1.1 · pp. 15, 17

3.1.2. Process Verification

  • HDS6 evaluates recorded trajectories across TRACES: Tools, Repair, Alternatives, Coherence, Evidence, and Scope, with an integrity gate for fabricated tool activity.Its blind process verifier uses 25 subcapability rubrics graded on 0/1/2 bands and load-bearing-step analysis that independently re-solves influential steps; both approaches produce targeted repair feedback.Source: §3.1.2 · pp. 17–19
  • Across 409 judged trajectories, HDS6 and hidden outcome scores were positively correlated in every problem family (Spearman ρ +0.24 to +0.65; pooled ρ +0.51, Pearson r +0.56).The pooled coefficients summarize agreement across tasks with different outcome scales, not calibrated magnitudes; task-level median Spearman ρ was near 0.5 and reached 0.8–0.9 for tasks with clear success signals.Source: §3.1.2 · p. 19, Table 3 · p. 20

3.2. Environments, Tasks, and Episodes

  • Environments, tasks, episodes, and verification teams together form a runnable, reproducible, repairable realization of an open-ended problem; building a faithful environment and trustworthy hidden verifier is itself difficult.An environment exposes actions, tools, budgets, and fresh observations; a task defines the problem or verifiable work unit, and an episode instantiates it with data, budgets, and hidden ground truth.Source: §3.2 · pp. 20–21

3.3. Environments

  • An environment is a constrained operational substrate that responds to solver actions with new observations, rather than a static prompt.Its manifest specifies the target outcome, hidden verifier, data, permitted tools, and resource budgets; episodes instantiate this contract.Source: §3.3 · pp. 21–22

3.3.1. Tasks

  • Tasks decompose real-world problems into scoped, verifiable work units and map which HDS6 capabilities each task can measure.For example, AAV sequence design exercises Repair and Tools through feedback and revision, while viability prediction exercises Evidence and Scope but not Repair.Source: §3.3.1 · pp. 22–23, Table 4 · p. 23

3.3.2. Episodes

  • The fixed episode interface standardizes inputs, tools, budgets, submissions, and hidden evaluation for solvers and baselines, enabling outcome differences to be attributed to solver components.Each episode is a specific run on a problem instance; its recorded trajectory supports process verification, while verifier feedback makes the episode repairable.Source: §3.3.2 · p. 23

4. Results on High-Value Problems

The results evaluate task-specific scientific environments, AAV benchmarks, drug-development procedure design, and LLM engineering tasks, alongside solver configurations, episode-level guidance, and HDS6 process characterization. They also assess outcome-blind HDS6-based repair through aggregate results and case studies.

4.1.1. Task Formulation; 4.1.5. Datasets and evaluation

  • The AAV benchmark tests a four-stage discovery pipeline, using real screening or structural data, out-of-distribution splits, and metered scoring rather than exposing ground-truth labels.Its tasks span viability, tissue tropism, structure prediction, and candidate design, reflecting the linked decisions needed to produce experimentally testable candidates.Source: §4.1.1 · pp. 24–25

4.1.2. A domain-specific environment improves outcomes

  • 0.741 across AAV Tasks 1–3 for claude-opus-4-8 in the domain-specific environment exceeded its 0.716 with Claude Code; Task 3 scores were 0.657 versus 0.595.The advantage also held on Task 4, suggesting that task-specific data, tools, biological constraints, and executable feedback help across the capsid-engineering workflow.Source: §4.1.2 · p. 25, Table 5 · p. 26

4.1.3. Surpassing the published state of the art

  • 0.904 exceeded CAP-PLM’s 0.878 on viability; 0.635 exceeded Fit4Function’s 0.622 on tropism; 0.649 exceeded AlphaFold 3’s 0.605 on structure.On generative design, apodex-1.1 scored 0.180 versus AAVGen (0.109), AAVDiff (0.116), and ALICE (0.110).Source: §4.1.3 · p. 25

4.1.4. Results based on process verification

  • HDS6 scores tracked final biological performance, and structure-task examples linked weak process scores to skipped self-validation and stronger scores to completed validation.GPT-5.5 scored 0.544 on structure prediction, 20% below kimi-k3; its process scores were 1.38 and 0.94 on two instances, versus kimi-k3’s 3.56 and 3.60. Opus models declined Task 4 outputs, consistently triggering biosafety refusal behavior.Source: §4.1.4 · pp. 25–26

4.2.1. Task Formulation; 4.2.2. Scoring Methods

  • DRR forecasts drug–disease clinical-development outcomes from pre-cutoff evidence, using a 100-pair public set and 200-pair hidden test set with temporal filtering and decontamination.Its primary normalized score maps absolute error on exact outcomes and lower-bound outcomes to anchors for a knowledge-free random predictor and a perfect predictor.Source: §4.2.1 · p. 27, §4.2.2 · p. 27

4.2.3. Public-100 Benchmark Results; 4.2.4. HDS6 Trajectory Analysis

  • With the DRR environment, both GPT-5.5 and GPT-5.6-sol improved on every reported metric; HDS6 gains were strongest in tool use, evidence fidelity, and coherence.The outcome comparison used three runs on the public development set and is descriptive, not a definitive test-set estimate. HDS6 profiles were similar across backbones; repair gains are unavailable without observable no-environment correction opportunities.Source: §4.2.3 · p. 29, Table 7 · p. 30, §4.2.4 · p. 30

4.2.5. Repurposing and Reformulation Procedure Design

  • Population procedure-design ratings improved by 0.89 points, raising the macro-average from 3.72 to 3.94 with retrieved trial-eligibility context.Context coverage was sufficient only for population specification; the other three fields were unchanged, and the population comparison measures evidence retrieval and synthesis rather than fully prospective procedure generation.Source: §4.2.5 · pp. 30–31, Table 9 · p. 31

4.3. Large language model problems and ablations; 4.3.1. Solver-configuration ablations

  • LLM engineering environments support two controlled ablation layers: solver configurations vary harnesses and models under fixed interfaces, while episode interventions toggle skill guidance and verifier-driven repair.Their recomputable outcome verifiers enable denser ablations; the current matrix spans 11 environments and six harnesses, with asymmetric model–harness coverage summarized in Table 10.Source: §4.3 · p. 31, §4.3.1 · p. 31, Table 10 · p. 32

4.3.1. Solver-configuration ablations

  • GPT-5.6-sol led balanced model means at 0.595, while the highest aggregate mean across tested configurations was 0.588; packaged harnesses changed model means only marginally.Balanced means use four model-agnostic harnesses across seven environments; the aggregate deployment scores include asymmetric harness pairings.Source: §4.3.1 · pp. 32–33
  • ApodexHarness led average model-agnostic harness performance at 0.548, but no harness dominated across models; it beat OpenHands on both GPT (0.592 vs. 0.560) and Opus (0.611 vs. 0.578).ApodexHarness led on Opus while A-Evolve led on GPT, evidencing model–harness interaction; Table 11 shows representative model-specific harness comparisons.Source: §4.3.1 · pp. 33–34, Table 11 · p. 33

4.3.2. Episode-mechanism ablations

  • Skill guidance raised OpenHands scores on LLM-pretrain-data-probe from 0.442 to 0.557 with Opus-4.8 and from 0.534 to 0.730 with GPT-5.6-sol; verifier guidance also improved OpenHands results.Skill guidance did not improve Claude Code–Opus-4.8, and gains varied by task, model, and intervention; one run per instance and incompletely controlled prompt conditions make differences descriptive.Source: §4.3.2 · pp. 34–35

4.4. Verification-Driven Repair

  • 0.155 higher average outcome scores followed repair across 434 deficient trajectories; nine of ten environments improved, with 204 trajectories improving and 86 declining.The study lacks a matched group of deficient trajectories rerun without repair, so ordinary run-to-run variation cannot be ruled out.Source: Table 13 · p. 36, §4.4 · p. 36

4.4.1. Case study: an under-specified clinical safety report

  • In the clinical safety-report case, repair raised the outcome score from 0.83 to 0.95 by improving evidence traceability from 0.71 to 1.00, while both runs submitted identical quantities.The outcome-blind critique identified an undeclared population choice and missing denominator patient IDs; the repair documented both before submission.Source: §4.4.1 · p. 37, Figure 8 · p. 38

4.4.2. Case study: a named estimator that was never run

  • In the corpus-scouting case, repair replaced an asserted source-size assumption with measured overlap and a Chao1 estimate, bringing the document estimate closer to truth but still overshooting it.The initial estimate was Mhigh = 108 [75, 145], versus 2471 [2129, 2813] after repair; estimation improved, but added measurement reduced efficiency, and the process diagnosis pinpointed the unmeasured source-size link.Source: §4.4.2 · pp. 37, 39

5. Related Work

Related work spans single-answer benchmarks, interactive agents, solver scaffolds, and process evaluation, alongside systems pursuing scientific discovery. TRACES and Apodex Discovery connect these threads through environments that verify intermediate results and assess process quality.

Knowledge and reasoning benchmarks

  • Knowledge and reasoning suites score answers against reference keys, but expose neither tools nor state and capture no solution process.Examples range from MMLU [5] and GPQA [6] to FrontierMath [8] and Humanity’s Last Exam [9].Source: §5 · p. 39

Interactive and agentic benchmarks

  • Interactive benchmarks cover executable tasks and real-world work; TRACES adds verified intermediate results and process-quality signals in a shared environment–task–episode abstraction.The abstraction spans biomedical, clinical, and frontier-model-engineering problems.Source: §5 · pp. 39, 41

Agent scaffolds

  • Agent scaffolds shape solver performance, so this framework evaluates the model and harness jointly and grounds repair in verifier feedback.CRITIC [65] uses external tools for correction, while Huang et al. [66] find self-correction without external feedback may fail to improve or degrade reasoning.Source: §5 · p. 41

Automated and LLM-based evaluation

  • Unlike outcome-facing, typically single-pass judges, the process verifier grounds judgments in trajectory evidence and is blind to both outcome and solver identity.Related evaluators include G-Eval [68], CriticGPT [73], and rubric-based HealthBench [72].Source: §5 · p. 41

Process supervision and verifiable rewards

  • HDS6 extends process evaluation from short arithmetic chains to long tool-using trajectories, using task-specific capability rubrics rather than a single scalar reward.Prior work studies step-level feedback and verifiers, including ProcessBench [78], which measures localization of the first erroneous step.Source: §5 · p. 41

Specification gaming and contamination

  • Hidden verifiers and anti-gaming gates address specification gaming, reward hacking, and benchmark contamination by blocking solver access and shortcut outcomes.These design commitments make the executable environment’s verification resistant to gaming.Source: §5 · p. 41

Autonomous scientific discovery

  • Apodex Discovery supports the shared goal of open-ended scientific discovery by providing reality-based environments, hidden verifiers, and process-level evaluation for consequential problems.Related systems include Coscientist, the AI co-scientist [2], and The AI Scientist [3].Source: §5 · p. 42

6. Conclusion

Apodex Discovery shifts evaluation from answer correctness to the quality of reliable, evidence-grounded, self-correcting investigations. It combines systematic problem selection, executable environments, and process evaluation to support discovery on consequential real-world problems.

6. Conclusion

  • The framework combines high-value problem scouting, executable environments, and HDS6 process evaluation; TRACES releases 17 hidden-verifier environments across 218 episodes, with two supporting verification–repair.The released environments realize a subset of 20 selected problems across biomedicine, clinical translation, and frontier-model engineering.Source: §6 · p. 42
  • In drug repurposing, environments raised GPT-5.5 and GPT-5.6-sol mean normalized prediction scores by 2.5 and 7.6 points; AAV results exceeded prior human baselines on the most demanding tasks.Controlled ablations also attributed performance differences to solver components rather than treating failures as generic model-capability failures.Source: §6 · p. 42
  • Controlled ablations attribute performance differences to the foundation model, harness, guidance, budget handling, and verifier-driven repair.This helps distinguish solver-component effects rather than treating every failed episode as a generic model-capability failure.Source: §6 · p. 42
  • TRACES covers only a subset of the 423-problem registry; process verification needs closer outcome-score alignment and broader application testing, while additional environments are under development.The 17 environments span 218 episodes, and two support the full verification–repair loop.Source: §6 · p. 42

7. Author Contributions and Acknowledgments

The acknowledgments credit Tianqiao Chen with proposing the heavy-duty solver concept, defining HDS6’s six capabilities, and shaping Apodex Discovery’s architecture, alongside discussion and technical contributors.

Author Contributions and Acknowledgments

  • Tianqiao Chen proposed the heavy-duty solver concept, defined HDS6’s six capabilities, and shaped Apodex Discovery’s architecture.The authors also thank him for his vision, guidance, and support.Source: §7 · pp. 42–43
  • The authors acknowledge Yuan Cai, Chengyu Fu, Shiyi Zhang, and Harry Zhang for discussion and contribution.Source: §7 · p. 43
  • The acknowledgments name six core technical contributors and a broader group of 23 contributors.Core technical contributors: Sheng Wang, Brian Wang, Bin Feng, Xiaoman Pan, Chenyang An, and Felix Liu. The broader group: Tangqi Fang, Gongbo Sun, Yueqi Guo, Kailong Wen, Feng Xing, Yiling Guo, Lingfeng Shen, Ning Wang, Handuo Zhang, Feng Chen, Fuchao Yang, Jiacheng Lin, Siting Li, Zixuan Liu, Xiang Wang, Chi Han, Zhenhailong Wang, Kunlun Zhu, Lawrence Zhao, Lidong Bing, David Tan, Bo An, and Heng Ji.Source: §7 · p. 43

A. Domains and questions considered by Apodex Discovery

Apodex Discovery catalogs selected high-value domains and concrete questions for long-chain agents, pairing each problem with a task and a way to check its answer. The examples span market, materials, industrial, health, agriculture, construction, and software problems.

A. Domains and questions considered by Apodex Discovery

  • The section identifies ten selected domains and a twenty-question registry; domain entries address value and long-chain-agent fit, while registry entries specify solver task, answer check, and buyer.Source: Table 14 · p. 51, Table 15 · p. 52, A. Domains and questions considered by… · pp. 52–53
  • Market-data and financial-index problems target high-margin franchises with over $100B in aggregate addressable revenue, with claims settled through fixed-horizon market outcomes.Out-of-sample tracking is native to this domain.Source: A. Domains and questions considered by… · p. 52
  • Materials discovery targets a $10–100B-plus advanced-materials market, while industrial automation targets a nearly $100B critical-infrastructure vertical; both offer explicit qualification or verification routes.Materials inverse design links proposed structures to predicted properties and measurements; industrial control can be formalized and simulated, with plant telemetry as verifier.Source: A. Domains and questions considered by… · p. 52
  • The registry includes investment-thesis generation, crop shortfall forecasting, and code-compliant structural design, checked through out-of-sample tracking, official reports, or automated code and constraint evaluation.The tasks respectively surface overlooked evidence and bull/base/bear scenarios, forecast localized shortfalls sixty days ahead, and minimize embodied carbon under seismic constraints.Source: A. Domains and questions considered by… · p. 53
  • Healthcare questions cover cell-therapy manufacturing optimization and organ-specific AAV capsid design, evaluated using post-infusion outcomes or unseen variants in a leak-proof hold-out.Cell-therapy optimization seeks higher potency and persistence; AAV design uses public sequence–function data.Source: A. Domains and questions considered by… · p. 53
  • Other healthcare questions address biological age and mortality, iPSC-cardiomyocyte differentiation, and rare-disease diagnosis, with checks based on held-out cohorts, donors or mutations, confirmed diagnoses, or trial eligibility.Tasks use multimodal health data, model disease-mutation effects, or produce structured reports from whole-genome or whole-exome data.Source: A. Domains and questions considered by… · p. 53
  • The registry also includes metabolic-response ranking, preclinical-to-human translation prediction, and near-term health-deterioration prediction, checked against twelve-month outcomes, paired held-out drug outcomes, or observed decline and escalation.The tasks rank patient interventions, attribute cross-species causal gaps, or recommend safe next actions.Source: A. Domains and questions considered by… · p. 53
  • Software questions include reconstructing company work state and detecting material events in real-time news, checked against whether flagged commitments remain open or events prove material after the fact.The work-state task also detects unresolved decisions and risks and checks whether the loop closed; news monitoring provides alternative readings.Source: A. Domains and questions considered by… · p. 53

B. Adeno-associated virus capsid design full results

Apodex evaluates AAV capsid design through four tasks—viability, tropism, structure prediction, and generative design—with task-specific metrics, held-out tests, and constraints. The supplied results report that gpt-5.5 has low HDS6 scores across all three structure instances because it skips self-validation and submits the cheapest answer.

B. Adeno-associated virus capsid design full results

  • Viability is measured by AUROC averaged across exact mutation counts, testing extrapolation to variants with at least 13 mutations and to insertions and deletions.The multi-mutation instance trains on variants with at most 12 mutations; the single-mutation instance trains on substitutions. The metric isolates ranking among equally mutated variants, where chance is 0.5.Source: B. Adeno-associated virus capsid… · p. 54, Table 16 · p. 55
  • Tropism prediction tests peptide enrichment across mouse multi-organ, mouse-to-macaque transfer, and human in-vitro instances using rank correlation with measured enrichment.The mouse test peptides are at least Hamming distance 2 from every training peptide; the human instance uses a split of its own library, while macaque peptides come from a separate library.Source: B. Adeno-associated virus capsid… · p. 54, Table 17 · p. 55
  • Structure prediction requires solvers to orchestrate folding and analysis tools in a network-isolated environment, with scores for subunit loops, the 60-mer shell, and antibody or receptor complexes.Evaluation controls leakage with post-cutoff structures, sequence-similarity filtering, and no network access; the loop and shell levels score 12 capsids, and the complex level scores 15 targets.Source: B. Adeno-associated virus capsid… · p. 54, Table 18 · p. 56
  • Generative design scores up to 1,000 peptides per target, counting hits only when they are novel, manufacturable, and on target under tissue-specific or receptor-specific criteria.The oracle may be queried for at most 100 sequences, and reference methods are reproductions under the same 1,000-submission and 100-query episode budget.Source: B. Adeno-associated virus capsid… · p. 54, Table 19 · p. 57
  • gpt-5.5 has low HDS6 scores across all three structure instances, consistently skipping required self-validation and submitting the cheapest answer.It uses one default folding engine without checking or comparison for loops, copies the highest-sequence-identity template for assembly, and hardcodes a severely collapsed complex outcome.Source: B. Adeno-associated virus capsid… · p. 56

C. Full harness × model result matrix

The section reports harness–model scores across 11 LLM environments, with seven environments included in the Section 4.3 aggregates and four excluded under pre-specified criteria. Its results distinguish task performance from submission, infrastructure, and scoring-gate effects.

Full harness × model matrix

  • The full matrix covers 11 environments and six foundation models; four environments were excluded from main-text aggregates for pre-specified coverage, score, compliance, or attribution concerns.The exclusions were nanogpt-speedrun, swe-juice, model-weakness-internal-benchmark, and llm-determinism. Some zero or unavailable cells reflected request limits, harness bugs, compatibility faults, or refusals rather than capability; for example, Codex scored zero on three episodes after exhausting the request limiter.Source: C. Full harness × model result matrix · pp. 58, 60

LLM-determinism

  • In llm-determinism, 8 of 23 recorded zeros were genuine, while none of the no-regression gate’s five firings across 72 episodes was confirmed as a regression.The other zeros were harness artifacts where no scorable submission reached the world, including early stopping, tool-frequency guards, and iteration or wall-clock caps.Source: C. Full harness × model result matrix · p. 61, Table 30 · p. 61

Clinical-sap-tlf

  • Clinical-sap-tlf reported 48 cells after eight environment and contract fixes removed 13 spurious zeros; a zero was usually an audit-gate failure on a numerically correct table.The gate required exact reproduction of all pre-specified numbers and passing every audit requirement; six configurations shared a signature in which five of seven numbers matched but implicit Dunnett-adjusted reference p-values were not specified in the plan.Source: C. Full harness × model result matrix · p. 62, Table 31 · p. 62

Drug-repurposing-reformulation

  • Across 1,700 drug-repurposing episodes, the balanced-grid mean was 0.9178, but submission failures—not judgment quality—drove harness separation.Harness spread was 0.0176 and model spread 0.0212; excluding submission-failure zeros, harness spread fell to 0.0098 and the ordering flipped. GPT had the highest judgment quality (0.9625 excluding zeros) but the most submission failures.Source: C. Full harness × model result matrix · p. 63, Table 32 · p. 63

AAV capsid enrichment

  • Every scored AAV enrichment cell cleared the k-mer lookup floor, and most reached or exceeded the Fit4Function reproduction; macaque transfer was most discriminating.Reference points per track were k-mer lookup 0.182 / 0.052 / 0.000 and Fit4Function reproduction 0.532 / 0.614 / 0.719.Source: C. Full harness × model result matrix · p. 64, Table 33 · p. 64

AAV capsid structure

  • AAV structure results were most discriminating on assembly; loop scores saturated at 0.927 in 11 of 16 cells, while six complex cells sat at the 0.163 unchanged-fold floor.The loop path was deterministic and cached across harnesses; the three tracks were loop, full 60-mer assembly, and VP–ligand complex.Source: C. Full harness × model result matrix · p. 65, Table 34 · p. 65

AAV sequence design

  • Only 4 of 14 evaluated AAV sequence-design configurations beat aavdiff on the six-instance mean.The brain / heart / spinal-cord aavdiff baselines were 0.113 / 0.177 / 0.267; the reported tropism score averages three selectivity bars because the strictest bar is counting-noise dominated.Source: C. Full harness × model result matrix · p. 66, Table 35 · p. 66

AAV viability

  • All 33 scored viability cells passed, Bryant settings broadly exceeded the literature reference, and capability failures were zero.The additive baselines were Bryant 0.885 and Ogden 0.851; the literature reference was cap-PLM/ESM2 ≈0.897 on Bryant. Ogden was harder than Bryant, and oracle settings were harder than labelled settings.Source: C. Full harness × model result matrix · p. 67, Table 36 · p. 67

D. Drug Repurposing and Reformulation Trace Example

The section presents one drug-repurposing and reformulation evaluation episode end to end, including its reasoning trace and process scoring. The episode summary keeps the reference column hidden from the judging panel.

D. Drug Repurposing and Reformulation Trace Example

D.1. The task as given

The brief asks the system to assess Dostarlimab for Stage II breast cancer offline, using bounded sanctioned tools while avoiding access to the held-out outcome. It specifies calibrated promisingness and confidence scores, evidence-backed claims, and a structured submission.

Access constraints

  • Do not retrieve the recent held-out outcome through the open internet; use only capabilities exposed by the selected environment bundle and active access adapter.The brief explicitly permits sanctioned interfaces such as the Stage-1 structured bio-graph provider, but copied documentation does not authorize additional capabilities.Source: §D.1 · p. 68

Task and role

  • Estimate how far Dostarlimab for Stage II breast cancer may progress toward clinical approval and report confidence in that estimate.The brief assigns the analyst role of a drug-repurposing researcher assessing this specific drug–disease pair.Source: §D.1 · pp. 68–69

Investigation workflow

  • Begin with one bounded Stage-1 call and its summary; rerun at a higher tier only for decision-critical gaps, and use Stage 2 only if it could change the score or confidence.A partial provider result is usable bounded coverage when error is null, timed_out is false, and the budget is exhausted; preserve conflicts and operational limits in the conclusion.Source: §D.1 · p. 68

Scoring criteria

  • Promisingness must reflect the clinical-approval spectrum, while confidence estimates prediction quality and should be low when evidence is thin or conflicting.Reference scores run from 0.00 for preclinical or failed Phase 1 to 1.00 for FDA-approved on-label use; successful phase results are lower bounds, while terminal failures target the last passed phase.Source: §D.1 · pp. 68–69

Evidence and output

  • Support each load-bearing claim with a retrieved source and supporting snippet, then submit scores, rationale, and 3–8 sourced key claims in the required JSON contract.The brief requires both a final dreg-result code block and submission of the same canonical JSON object through the execution environment.Source: §D.1 · pp. 69–70

D.2. Full reasoning trace · D.3. Process score

These sections describe a bounded biomedical-tool investigation and its process-scoring rubric. The trace records actions and outputs while keeping private reasoning hidden, and the rubric defines how capabilities are scored from sub-rubric bands.

D.2. Full reasoning trace

  • The trace shows every recorded action, tool call, and returned output, while withholding model reasoning turns from the judging panel.Reasoning turns, tool calls, a failed call, and the final submission are visually distinguished in the trace.Source: §D.2 · p. 70
  • The environment meters and records actions, isolates shell work, restricts domain-tool calls to allowlisted capabilities, and requires final answers through submit.Tool artifacts use virtual paths, and shell work cannot access benchmark data or the host tool workspace.Source: §D.2 · pp. 71–72
  • Both providers exhausted their 150-expansion budgets without timing out or returning errors, but this partial coverage is not negative evidence.The disease anchor remained unresolved, and the report did not reconstruct a connection.Source: §D.2 · pp. 73–74
  • The search found no complete dostarlimab–Stage II Breast Cancer connection: the drug resolved, the disease anchor remained ambiguous, and the report graded support incomplete.No candidate path, connection, or load-bearing evidence was returned; the report retained an unresolved anchor gap.Source: §D.2 · pp. 74–75
  • Clinical trials report 100% clinical complete responses in deficient mismatch repair patients, supporting dostarlimab in colorectal and rectal malignancies.The trace also identifies dMMR and locally advanced rectal cancer as supported indications.Source: §D.2 · p. 76
  • Dostarlimab has moderate-to-high clinical-trial evidence in NSCLC, with durable responses across PD-L1 subgroups.Source: §D.2 · p. 76
  • Prostate cancer evidence is directionally positive but constrained by rare qualifying biomarkers, despite biological plausibility from T cell signature enrichment in high-grade tumors.The trace notes approval for dMMR or high TMB solid tumors.Source: §D.2 · p. 76
  • Dostarlimab binds PD-1 with high affinity (KD 300 pM), blocks PD-L1 and PD-L2 interactions, and supports T cell activation [ref-5] [ref-4] [ref-9] [ref-13].The trace describes conformational rearrangements in PD-1 loops [ref-6].Source: §D.2 · p. 76
  • As of March 2023, seven specified cancer types lacked approved MHRA indications; PTEN mutations may predict poor response in dMMR endometrial cancer, and lung efficacy in the dMMR solid-tumor approval cohort is uncharacterized.Source: §D.2 · p. 76

D.3. Process score

  • A capability score is the weighted mean of its sub-rubric bands (0/1/2), rescaled to [0, 4]; critical sub-rubrics can cap it, while unexercised sub-rubrics are excluded.Table 38 marks unexercised sub-rubrics as nr rather than scoring them zero.Source: Table 38 · p. 81

C Long-horizon State Coherence

The trajectory grounded its final judgment in retrieved evidence and distinguished direct support from class-adjacent plausibility, but it lost coherence credit for failing to anchor the supplied identifiers and connect confidence to evidence strength.

Terminal coherence

  • The confidence value was not tied to a named retrieved quantity or explicitly mapped to indirect, incomplete evidence, contributing to a terminal-coherence score of 1.Promisingness and rationale were mapped to evidence categories, but confidence was not.Source: C Long-horizon State Coherence · p. 82
  • The final judgment used the max-tier query’s zero candidate paths to lower promisingness and treated the PD-L1 hint as class-adjacent plausibility, not direct evidence.Key claims tied conclusions to the extra and max queries, the inspected drug page, and the PD-L1 related-page query.Source: C Long-horizon State Coherence · p. 82

Task decomposition

  • The workflow was dependency-ordered, but scored 1 because it used surface names instead of supplied IDs and left the disease anchor unresolved.The final correctly reported that no direct connection was found in the sanctioned max-tier result, while keeping atezolizumab and triple-negative breast cancer as plausibility context.Source: C Long-horizon State Coherence · p. 82

E Evidence Fidelity

Evidence Fidelity was the strongest dimension: key claims were traceable to retrieved records and separated from inference and unknowns. The submission lost credit because load-bearing claims lacked an explicit recency statement.

E Evidence Fidelity

  • 3.50 / 4 was the Evidence Fidelity score, the strongest dimension; one band was lost because load-bearing evidence lacked an explicit recency statement.The retrieved drug page had an updated stamp, but the answer did not state that status or caveat possible staleness.Source: E Evidence Fidelity · pp. 82–83
  • Key claims cited resolvable tool outputs or inspected pages, with supports traceable to those records and no load-bearing item relying on vague literature or knowledge-graph support.Negative claims pointed to named tool-output fields, while positive mechanism and context claims cited locatable records.Source: E Evidence Fidelity · pp. 82–83
  • The answer separated source-backed findings, inference, and unknowns, describing the mechanism as plausible rather than established and the PD-L1/TNBC hint as class-adjacent, not direct evidence.It also stated that the disease anchor was unresolved and that no sanctioned direct clinical support existed for the exact drug-disease pair.Source: E Evidence Fidelity · p. 82

A Hypothesis Management

The evaluation credits a concrete, disease-relevant checkpoint-blockade mechanism and retention of negative retrieval evidence, while finding that the agent did not formulate a competing no-benefit hypothesis or demonstrate an explicit estimate update.

A Hypothesis Management

  • 3.00 / 4: The submission states a testable mechanism and retains negative retrieval, but lacks a competing no-benefit hypothesis and explicit before/after estimate revision.The negative evidence lowered the score, but the final rationale showed a static synthesis rather than a demonstrated ranking movement.Source: A Hypothesis Management · p. 83

Alternative paths considered

  • 1: The agent advanced PD-1/PD-L1 blockade but did not set it against an explicit no-benefit or symptomatic-only hypothesis.The final acknowledged evidence gaps and other possibilities without placing two mechanistic paths side by side.Source: A Hypothesis Management · p. 83

Contradiction retention

  • 2: The submission retained the max-tier retrieval's incomplete-support finding and used the unresolved disease anchor and lack of direct evidence to lower confidence.It scored below Phase-2-success territory while still acknowledging mechanistic plausibility and class-adjacent support.Source: A Hypothesis Management · p. 83

Hypotheses made explicit

  • The deliverable explicitly links PD-1/PD-L1, plus PD-L2 in key claims, to restored T-cell activity in immunogenic breast-cancer subsets.It describes the exact Stage II Breast Cancer evidence as indirect, while the mechanism is concrete and disease-relevant.Source: A Hypothesis Management · p. 83

Hypothesis update on evidence

  • 1: The rationale used negative retrieval and PD-L1-related evidence, but showed no prior, interim score, or explicit estimate movement tied to a retrieved record.The final score was a static synthesis rather than a demonstrated ranking update.Source: A Hypothesis Management · p. 83

S Boundary & Failure Reasoning

The submission maintained research-use boundaries and acknowledged thin evidence, but its counter-case lacked concrete failure and safety constraints and did not justify its moderate confidence score. Boundary & Failure Reasoning was its weakest dimension at 2.50 / 4.

Boundary & Failure Reasoning

  • 2.50 / 4 was the weakest dimension; the counter-case relied on an evidence gap rather than a concrete biological or clinical reason the repurposing could fail.The submission also did not explain why its confidence value was appropriate.Source: S Boundary & Failure Reasoning · p. 83

Boundary & Scope

  • The submission scored 2 on boundary and scope, maintaining research-use framing and noting no direct clinical support for Stage II breast cancer.It said no complete evidence connection was reconstructed and described PD-L1/TNBC evidence as class-adjacent; it made no patient-treatment recommendation.Source: S Boundary & Failure Reasoning · p. 83

Failure Modes

  • Failure modes scored 1 because the submission did not name a concrete biological or clinical mechanism by which dostarlimab could fail in this setting.It cited class-adjacent evidence and lack of direct clinical support, but did not identify issues such as insufficient target engagement, exposure, or tissue delivery.Source: S Boundary & Failure Reasoning · p. 84

Safety & Feasibility

  • Safety and feasibility scored 1 because no indication-specific constraint for Stage II breast cancer was identified.Generic pharmacokinetic and drug-interaction information did not address tolerability, contraindications, toxicity, administration, or monitoring for this population.Source: S Boundary & Failure Reasoning · p. 84

Uncertainty Calibration

  • Uncertainty calibration scored 1: despite naming evidence gaps, the submission did not justify confidence_score=0.48, which the evaluation treated as moderate rather than low.Its rationale for a 0.3 promisingness value did not explain the separate confidence score.Source: S Boundary & Failure Reasoning · p. 84

T Tool Use & Execution State Management · R Self-correction under Verification

The evaluated agent scored 3.50/4 for tool use and execution, with an avoidable invalid-tier call repaired on the next relevant action. The trace shows retrieval, evidence absorption, stopping discipline, and a repair that did not recur.

T Tool Use & Execution State Management

  • 3.50/4: the agent read manuals, issued the bounded search, reused returned artifacts, and stopped with an explicit sufficiency judgment.One band was lost for the avoidable invalid-parameter attempt at step 12.Source: T Tool Use & Execution State Management · p. 84
  • The agent’s calls were ultimately usable, but an unsupported deep tier and surface-name pair instead of supplied IDs fell short of exact call correctness.The tool rejected deep; the agent retried with max, while the disease resolved as local/unknown.Source: T Tool Use & Execution State Management · p. 84
  • The agent escalated from a weak return to max tier and a relevant PD-L1 query, then stopped when the exact pair still lacked a reconstructed path or load-bearing evidence.It stated that PD-L1 breast-cancer evidence was class-adjacent and gave a below-Phase-2 score rather than looping.Source: T Tool Use & Execution State Management · p. 84
  • The agent opened returned artifacts and reused specific empty-path, drug-page, and PD-L1 hint details in its final claims and support.Because the search returned no paths or edges, there were no edge directions or confidences to absorb.Source: T Tool Use & Execution State Management · p. 84
  • The agent used retrieval for the Dostarlimab–Stage II Breast Cancer question and followed a decision-relevant PD-L1 hint with the correct target ID and stage2 suggest form.Skipping expand was acceptable because the returned summary showed no suggested expansions.Source: T Tool Use & Execution State Management · p. 84

R Self-correction under Verification

  • Four of five self-correction sub-rubrics received top-band scores: the agent recognized its bad argument, minimally fixed it, used the repaired result, and did not repeat the error.Error localisation was not separately graded because the error was corrected on the next action.Source: R Self-correction under Verification · p. 85
  • The invalid deep-tier call qualified as a repairable incident, whereas successful calls returning documented partial coverage did not qualify as tool failures or negative evidence.The agent changed deep to max for the same query; both connection calls succeeded, and the unresolved disease anchor was an operational limitation rather than a repairable call error.Source: R Self-correction under Verification · p. 85

Related papers

Loading 2608.11341v1…