PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolving with Open Worlds: Towards Recursive Self-Improvement for Future Forecasting

Zhipeng Chen, Yunsen Liang, Yanxi Chen, Hongkang Zhao, Lingyun Feng, Ruiyang Ren, Guoxin Chen, Jia Deng, Feng Chang, Chuan Hao, Ran Tao, Jian Yang, Ji-Rong Wen, Bryan Dai, Wayne Xin Zhao

arXiv:2610.29784v1cs.CL

TL;DR

Fixed datasets do not renew challenges or independently judge progress, limiting the study of recursive self-improvement in a changing world. DEEPEVOLVE couples a living forecasting environment, an agentic forecaster, and an evolution engine that learns from prediction trajectories. The system ranks among the top three across four LLMs, and the authors report that its performance improves through co-evolution of the agent and environment.

  • Problem

    Fixed datasets neither renew challenges nor judge progress on their own, while studying recursive self-improvement calls for an interactive, verifiable environment.

    Source: §1 · p. 1

  • Method

    DEEPEVOLVE couples a living source of forecasting tasks, an agent that records prediction trajectories, and an evolution engine that updates parameters, distills skills, and shapes harder tasks.

    Source: §1 · p. 3, §4.1 · p. 17

  • Results

    DEEPEVOLVE ranks among the top three across four LLMs, with BT 62.2 on QWEN3.8-MAX, +10.7 over the strongest baseline, and improves through accumulating live trajectories.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings frame recursive self-improvement as co-evolution of an agent and a living environment, with the environment supplying experience and tasks shaped toward exposed weaknesses.

    Source: §1 · p. 2

    The recursive loop is bounded by the quality and quantity of available experience, while long-horizon evaluation is expensive and stability under non-stationary drift is not guaranteed.

    Source: §3 · p. 16

Abstract

from arXiv · show

Most contemporary artificial intelligence (AI) systems are released with frozen parameters and improve through external updates, whereas an ideal system would keep improving from its own experience, a capacity we call recursive self-improvement (RSI). Yet studying RSI requires an environment that is continuously changing, objectively verifiable, and naturally grounded, posing a fundamental challenge for agent evolution. Here we introduce DeepEvolve, a sustainable evolution agent system grounded in future event prediction, in which agents continuously interact with a dynamically evolving environment and improve by internalizing and externalizing the experience gained through these interactions. A living environment synthesizes verifiable tasks, and a recursive-evolution engine turns the reasoning trajectory behind each prediction back into capability: decoupling search from prediction adapts parameter-level evolution to the contamination-prone live environment, while distilling reusable skills lets the agent evolve from the limited live trajectories such an environment affords. On the external FutureX leaderboard, which evaluates predictions against real-world outcomes and is updated weekly, our self-improved forecaster ranked first across all systems over three consecutive weeks, achieving up to a 195% improvement in rating. On our synthesized evaluation, it ranked among the top three across four base LLMs, outperforming the strongest baseline by up to 10.7 Bradley--Terry points. More broadly, these results suggest that a system built for self-improvement is best understood as an agent that co-evolves with a genuinely living, open-ended environment, pointing to a shift in AI from training static models on fixed data toward cultivating agents that grow stronger from their own experience, with such an open world as the indispensable basis of that growth.

1. Introduction

The paper studies recursive self-improvement (RSI) through real-world event forecasting, where future outcomes provide verifiable feedback in a changing environment. DEEPEVOLVE couples that environment with an evolving forecaster and recursive-evolution engine, reporting improved forecasting performance and continued gains from accumulated experience.

1. Introduction

  • Fixed evaluations neither renew challenges nor judge progress, motivating an environment grounded in real-world events with outcomes that resolve over time.The paper contrasts this setting with prior event and time-series forecasting work focused largely on single-shot accuracy [11, 30].Source: §1 · p. 2
  • The framework distinguishes task-solving capabilities—evidence gathering and reasoning—from the additional capability to change behavior based on past successes and failures.The paper names this proposed framework DEEPEVOLVE.Source: §1 · pp. 2–3
  • DEEPEVOLVE treats recursive self-improvement as compounding across rounds, with parameters, skills, and the environment produced by one round feeding the next.Its approach combines learning from forecasting trajectories with tasks shaped by the agent’s exposed weaknesses.Source: §1 · p. 2
  • DEEPEVOLVE couples a living task environment, an agentic forecaster, and a recursive-evolution engine in a closed loop of agent-environment co-evolution.The environment supplies tasks and resolving outcomes; the forecaster produces predictions and trajectories; the evolution engine learns from those trajectories.Source: §1 · p. 3, Figure 1 · p. 4
  • BRANCHUS ranked among the top three across four LLMs and led its block, with BT 62.2 on QWEN3.8-MAX, +10.7 over the strongest baseline.The authors also report that calibration improves through parameter updates, while skill accumulation and environment co-evolution contribute additional gains.Source: §1 · p. 3
  • The authors argue that an evolving agent and environment may shift AI development from fixed-data training toward cultivating agents that grow through experience.They identify the available experience, evaluation cost, and stability under non-stationary drift as constraints.Source: §1 · p. 3

2. Results

The Results section examines how DeepEvolve combines live future-event forecasting, recursive updates to parameters and reusable skills, and task synthesis. Its evaluations assess how internalizing experience and expanding it through an evolving environment support improvement across rounds.

2.1. Experimental Design and DeepEvolve Framework

  • FutureX supplies weekly live tasks, while PROMETHEUS synthesizes daily questions from fresh news that resolve the following day, enabling more frequent verification against real-world outcomes.PROMETHEUS scores resolved predictions using retrieved outcomes and reports calibration; reported scores average several consecutive days, while FutureX uses its official sliding-window Bradley–Terry model.Source: §2.1 · p. 5

2.2. Recursive evolution lifts DEEPEVOLVE to the strongest forecasting system

  • 62.2 BT led the PROMETHEUS evaluation for BRANCHUS with QWEN3.8-MAX, and BRANCHUS ranked first in every base-LLM block, with baseline margins of +10.7 to +0.5 BT.The four BRANCHUS configurations occupied the top three overall positions. Figure 2 tracks FutureX ratings across four weekly evaluations, where BRANCHUS remained at or near the top as the task distribution refreshed.Source: §2.2 · pp. 5–6, Figure 2 · p. 6, Table 1 · p. 7
  • BRANCHUS’s advantage includes calibration: it achieved the best Brier score in three of four LLM blocks and best relative WIS of 0.737 on QWEN3.8-MAX and 0.865 on GPT-5.6-SOL.Choice accuracy alone did not consistently separate systems, and the paper reports that BRANCHUS remained near the top across four consecutive weekly FutureX evaluations as tasks refreshed.Source: §2.2 · pp. 6–7, Figure 2 · p. 6

2.3. Search-prediction decoupling adapts RFT and RLVR to contamination-prone live environments

  • Under the function-oriented protocol, RLVR raises choice accuracy by 1.3 points on each of two LLMs; RFT raises accuracy by 0.7 to 2.9 points and consistently improves Brier and relative WIS.The protocol caches evidence and trajectories while tasks are open, then trains the predict team on fixed, pre-cached evidence. For GLM-4.7-Flash, first-round RFT changes Brier from 0.651 to 0.636 and relative WIS from 1.02 to 0.99; the second round lowers them to 0.630 and 0.98, respectively.Source: §2.3 · pp. 8–9

2.4. Evolving skills unlock recursive self-improvement from fleeting live trajectories

  • Without changing model weights, distilled skills lowered Brier from 0.135 to 0.122 and MAPE from 2.80% to 2.28%, with gains increasing across three rounds of accumulated live trajectories.The three skill pools contained 88, 186, and 262 trajectories; the Brier-score improvement versus baseline reached 0.007, 0.012, and 0.013, while coverage grew from 168 domains and 173 tasks to 479 domains and 505 tasks.Source: §2.4 · pp. 9–10, Figure 6 · p. 11, Figure 7 · p. 12

2.5. Skill evolution guides evidence gathering while parameter evolution sharpens evidence use

  • On structure-synthesis problems, skills cut tool calls by 44% from 32 to 18 per question while maintaining MAPE; on open problems, they raised calls 13% from 30 to 34 as MAPE fell 32% from 3.40% to 2.31%.Skills shape evidence gathering, whereas RLVR changes how the predict team uses cached evidence; skill extraction is inference-only and cheap but gains plateau after roughly 88 trajectories, while parameter updates cost more and continue improving beyond that ceiling.Source: §2.5 · pp. 11–12, Figure 8 · p. 13

2.6. An evolving environment continuously turns the agent’s own failures into adversarial challenges

  • Weakness-targeted task synthesis made questions harder but answerable: selection accuracy fell from 0.60 to 0.53 and Brier worsened from 0.52 to 0.58, while AUROC edged up from 0.678 to 0.68.ATHENA distills recurring failure modes into task-synthesis skills that PROMETHEUS uses to generate subsequent challenges.Source: §2.6 · p. 13, Figure 9 · p. 14

2.7. A targeted and challenging environment drives sustained system evolution

  • Against a fixed environment, weakness-targeted tasks from the same 262 trajectories increased distilled skills from 1312 to 1403, domains from 320 to 337, and tasks from 331 to 357; hard-task RFT improved every reported metric.Relative to plain RFT, adversarially synthesized hard problems lowered Brier by 1.6%, raised AUROC by 2.7%, and improved relative WIS by 1.0% and Md-sAPE by 4.0%.Source: §2.7 · p. 14, Figure 9 · p. 14
  • Parameters, distilled skills, and a harder environment each contribute to evolution, but no single axis is sufficient.Parameters and skills internalize experience already available, whereas the evolving environment keeps that experience expanding.Source: §2.7 · pp. 14–15
  • Across every metric, RFT on adversarially synthesized hard problems improves over plain RFT.Each metric row is shown on its own scale.Source: Figure 10 · p. 15
  • Joint evolution grounded in information-rich forecasting trajectories lets capability compound across rounds, the form of recursive self-improvement described in the introduction.Source: §2.7 · p. 15

3. Discussion

DEEPEVOLVE combines agent learning with an evolving, verifiable forecasting environment, and its results support sustained improvement from accumulated experience. The discussion also identifies limits in the experience available, the reliability of accumulated skills, and the pace of real-world feedback.

3. Discussion

  • DEEPEVOLVE attains the strongest forecasting performance across diverse baselines, while skill accumulation lifts quality step by step rather than in a single jump.The authors report clear calibration gains and say the system matches or surpasses every alternative.Source: §3 · p. 15
  • The framework couples a living data source, an agentic forecaster, and a recursive-evolution engine, with experience retained in both updated parameters and reusable skills.Separating time-sensitive evidence gathering from prediction lets the agent learn from frozen forecast-time evidence while resolved outcomes provide verifiable rewards.Source: §3 · p. 15
  • Parameter evolution improves reasoning over fixed evidence, while skill evolution improves evidence gathering; together they use scarce live trajectories that are insufficient for stable internal learning alone.The authors describe the two learning routes as complementary and report that the strongest configurations pair a parameter-trained predict team with a mature skill library.Source: §3 · pp. 15–16
  • A fixed task environment eventually saturates and risks contamination, so the system uses new events with outcomes that resolve only in the future to renew experience and support genuine forecasting.ATHENA turns diagnosed trajectory failures into task-synthesis skills, which PROMETHEUS uses to generate tasks targeted at current weaknesses.Source: §3 · p. 16
  • The environment’s teaching signal misses silent weaknesses such as overconfidence, and rounds can be slow and costly when questions take months to resolve.The discussion also identifies sparse or culture-dependent events, skill drift, and room to improve both parameter- and skill-evolution operators.Source: §3 · p. 16
  • The methodology is presented as portable to grounded, verifiable, non-stationary domains, including scientific discovery, software engineering, and strategic planning.The authors argue that the environment’s capacity to keep raising the bar sets the ceiling for recursive self-improvement.Source: §3 · p. 17

4. Approach

DEEPEVOLVE is a closed loop in which the agent and environment improve one another over time. PROMETHEUS supplies evolving tasks, BRANCHUS forecasts and records trajectories, and ATHENA turns those trajectories into updates to the agent and environment.

4.1. Overview

  • DEEPEVOLVE’s three modules form a co-evolving loop: a stronger BRANCHUS faces increasingly demanding PROMETHEUS tasks, while resolved outcomes provide objective rewards.BRANCHUS produces predictions and full trajectories; ATHENA updates its parameters and skills and synthesizes harder tasks for PROMETHEUS.Source: §4.1 · pp. 17–18

4.2. Prometheus: A Living Data Environment

  • PROMETHEUS creates forecasting questions from public calendars, feeds, structured pages, and developing stories through template-based and agent-driven discovery.Its pipeline filters candidates for temporal eligibility, removes duplicates and conflicts, then selects for diversity subject to the available questions.Source: §4.2 · pp. 18–19

4.3. Branchus: Agentic Forecasting System

  • BRANCHUS separates time-sensitive evidence gathering from prediction over cached evidence, recording the reasoning trajectory for each forecast.Its search team gathers and vets evidence; its predict team aggregates search trajectories into a probabilistic answer.Source: §4.3 · pp. 19–20
  • BRANCHUS combines hand-authored seed skills with experience-distilled skills, organized from general principles to concrete procedures for reuse at different levels of specificity.The skill repository can match a question without an exact precedent using a broader level of abstraction.Source: §4.3 · p. 20

4.4. Athena: Recursive Evolution System

  • ATHENA turns BRANCHUS’s trajectories into parameter updates, reusable skills, and harder tasks targeted at observed weaknesses.The parameter and skill operators evolve the agent; the environment operator adapts PROMETHEUS using the same experience stream.Source: §4.4 · pp. 20–21
  • Parameter-level evolution trains the predict team with verifiable rewards on cached search trajectories, preventing answer leakage while allowing predictions to vary over fixed evidence.The resolved outcome supplies the reward without human annotation; separating search from prediction makes updates reflect reasoning over the same information.Source: §4.4 · pp. 20–21
  • Behavior-level evolution evaluates search-team roles without showing the judge the resolved outcome, then extracts, filters, and organizes reusable skills without changing model weights.The four-level skills generalize from a concrete procedure toward question-type, domain, and cross-domain guidance; distillation can be repeated after every round.Source: §4.4 · p. 21

4.5. Models and Implementation

  • The evaluation holds PROMETHEUS questions, tools, and tool descriptions constant across harnesses, varying only harness design and base LLM.It pairs five base LLMs with six harnesses and uses each harness’s official prompt.Source: §4.5 · p. 22

A. Full FutureX Overall Leaderboards

The complete FutureX leaderboards show BRANCHUS’s standing against all entrants across four weekly evaluations. It led the overall rating in the first three weeks and remained among the strongest entrants in the final week.

A. Full FutureX Overall Leaderboards

B.1. 2026 Nobel Prize

BRANCHUS was the only forecaster to list either of two outsider Nobel laureates, while its Literature forecasts consistently ranked the eventual winner in the top two. In Economics, no forecaster listed both laureates, and BRANCHUS’s probability for the laureate it did list was mid-field.

B.1. 2026 Nobel Prize

  • The forecasts were published on X before the announcements, making the predictions timestamped and publicly auditable against the outcomes.The evaluation describes five posts, one per prize.Source: §B.1 · p. 27, Table 3 · p. 28
  • The prizes tested unusually difficult targets: thousands of plausible nominees, sealed deliberations, and frequent selection of an outsider over the consensus front-runner.The paper notes that the realistic candidate pool tracked by any one forecaster is only a fraction of the true pool.Source: §B.1 · pp. 27–28
  • BRANCHUS alone listed either outsider laureate, Halzen in Physics or Pillay in Peace, in one of its three independent runs.No competing system listed either winner or beat the no-forecast baseline in these two cases.Source: §B.1 · p. 28, Figure 12 · p. 29
  • Every BRANCHUS Literature run ranked Anne Carson, the eventual laureate, in the top two, a stability no competitor matched.Only BRANCHUS, OpenClaw, and GPT-6 Pro ranked Carson first in any run; BRANCHUS’s other two runs ranked her second.Source: §B.1 · p. 28
  • No forecaster, including BRANCHUS, listed both Economics laureates; BRANCHUS’s probability for the laureate it listed was mid-field.Every run surfaced only one of the two winners.Source: §B.1 · p. 28

C. Evaluation Metrics

The evaluation characterizes forecasting systems along three axes rather than collapsing performance into one number. It handles choice and numeric questions as separate tracks.

C. Evaluation Metrics

  • The metrics characterize how often a forecaster is right, how trustworthy its probabilities are, and how much compute it spends.The evaluation treats these as three complementary axes rather than a single aggregate measure.Source: C. Evaluation Metrics · p. 31
  • The evaluation separates choice and numeric question families into tracks marked c and n.Each track has its own question set and cardinality.Source: C. Evaluation Metrics · p. 31
  • The notation indexes harness–LLM systems by i in a set I of size M.The metrics are defined over the scored harness–LLM systems.Source: C. Evaluation Metrics · p. 31

C.1. Choice Forecasts

Choice forecasts are evaluated for both decision accuracy and probability quality. The metrics also assess whether confidence distinguishes correct from incorrect answers and matches realized accuracy.

C.1. Choice Forecasts

  • Accuracy is the percentage of questions where the top-probability option matches the resolved option, while Brier score measures squared probability-vector error against the one-hot resolution.Brier retains forecast magnitude, and the full sum over options is used; higher accuracy and lower Brier are better.Source: §C.1 · p. 31
  • Negative log-likelihood scores the probability assigned to the resolved option, with a tail statistic averaging the h = ceil(0.1N_c) worst losses to expose catastrophic single-question losses.Logarithms are natural, so scores are in nats; lower is better for both measures.Source: §C.1 · pp. 31–32
  • The confidence gap is mean confidence on correct answers minus mean confidence on incorrect answers, measuring whether stated confidence carries information about correctness.A gap near zero indicates that it does not.Source: §C.1 · p. 32
  • AUROC treats correctness as the positive class and confidence as the ranking score, while ECE measures count-weighted differences between mean confidence and accuracy across ten equal-width bins.AUROC summarizes correct-versus-incorrect confidence ranking; lower ECE means confidence tracks empirical accuracy more faithfully.Source: §C.1 · p. 32

C.2. Numeric Forecasts

Numeric forecasts are assessed using interval and point-forecast errors, while cross-system comparisons use per-question wins and a combined Bradley–Terry rating.

C.2. Numeric Forecasts

  • WIS scores the central 50%, 80%, and 90% predictive intervals by combining interval width and penalties for outcomes outside them with absolute median error.It is reported in target units, lower is better, and cross-target rankings use within-question comparisons rather than averaging raw WIS.Source: §C.2 · p. 32
  • Md-sAPE is the median across questions of symmetric percentage error based on the predicted mean, limiting the influence of a few large-magnitude targets.It is a percentage; smaller values mean tighter point forecasts, and the error is zero when both prediction and outcome are zero.Source: §C.2 · p. 33

C.3. Relative Comparisons

  • For relative comparisons, systems are scored per question using Brier on choice questions and WIS on numeric questions; lower loss wins, ties count as half, and tie detection uses relative tolerance 10^-10.The numeric WIS win rate averages these outcomes across questions and opponents, with each question and opponent weighted equally.Source: §C.3 · p. 33
  • Bradley–Terry ratings combine choice and numeric tracks with equal contribution, fitting pairwise outcomes by weighted maximum likelihood under a zero-sum constraint.The headline score is 100σ(θ̂_i), the fitted win probability against a neutral opponent; stratum ratings are refit separately and are not comparable across strata.Source: §C.3 · p. 33

C.4. Stratified Diagnostics

Stratified diagnostics group forecasts by difficulty and disagreement, then report both absolute and question-adjusted losses within each group.

C.4. Stratified Diagnostics

  • Difficulty and disagreement are computed from forecasts rather than resolutions, and each is divided into three empirical-quantile tiers within each track.Difficulty measures consensus uncertainty; disagreement measures divergence among systems.Source: §C.4 · p. 34
  • Choice difficulty uses normalized entropy of the averaged, renormalized probability vectors, while numeric difficulty uses the median relative width of systems’ 90% intervals.The resulting tiers are Easy, Medium, and Hard.Source: §C.4 · p. 34
  • Choice disagreement is option-summed population variance of predicted probabilities rescaled for differing option counts; its tiers are Low, Medium, and High.Numeric disagreement is defined in the passage as an axis but its calculation is not specified here.Source: §C.4 · p. 34
  • Within each stratum, absolute error is mean Brier for choice or median 100 WIS/|y_q| for numeric forecasts, alongside question-adjusted losses that remove question-mix effects.Choice losses subtract per-question mean Brier; numeric losses use the log ratio of WIS to the question’s geometric-mean WIS, with specified zero-WIS handling.Source: §C.4 · p. 34

D. Prompt Templates

The appendix presents the runtime prompts for a round of DEEPEVOLVE, from question construction and forecasting through trajectory evaluation and skill synthesis. The templates specify how the modules operate and how learned skills feed back into future tasks.

D. Prompt Templates

  • The prompt appendix follows one recursive-improvement round: PROMETHEUS synthesizes questions, BRANCHUS gathers evidence and predicts, and ATHENA evaluates trajectories and distills skills for future task construction.Runtime placeholders receive question, trajectory, evidence, or skill content; the displayed templates omit some formatting instructions and output schemas.Source: D. Prompt Templates · pp. 34–35

D.1. Prometheus: Question Synthesis Prompt

  • PROMETHEUS construction uses either build for bundled adapters or build-open for agent-discovered drafts, with bounded concurrency, validation, deduplication, ranking, and atomic publication in a shared pipeline.Question count is a soft target; published questions must be unknown at cutoff, objective in answer space, and within the requested resolution horizon, while semantic judgment belongs to the templates and executing agent.Source: §D.1 · p. 36

D.2. Branchus: Forecasting Prompt

  • BRANCHUS separates evidence gathering from probabilistic prediction, conditions both teams on retrieved skills, and instructs forecasts to resolve source conflicts without treating repetition or confidence as verification.Searchers can inspect source steps and evidence; the final predictor may use tools when they could change the forecast and must submit finite probabilities from 0 to 1.Source: §D.2 · pp. 36–37

D.3. Athena: Action-level evaluation prompt

  • ATHENA evaluates each BRANCHUS action against its role-specific rubric without access to the resolved outcome, aggregating each dimension’s judge scores by the median.Observed issues and per-action scores become inputs to skill extraction.Source: §D.3 · p. 38

D.4. Athena: Skill Extraction Prompt

  • ATHENA’s extraction process filters trajectory experiences and organizes surviving know-how into four role-specific levels, from a concrete procedure to a cross-domain principle.Skills encode forecasting procedures rather than answers, preserve role boundaries, and should avoid outcome leakage; unsupported experiences may be returned as unable rather than forced into a skill.Source: §D.4 · pp. 38–39

D.5. Athena: Environment-Level Skill Synthesis Prompt

  • Environment-level skill synthesis turns BRANCHUS’s error profile into task-construction guidance that steers PROMETHEUS toward categories where it has performed poorly.Error rates are prioritization signals, not conclusive evidence of systematic reasoning flaws; generated questions must remain objective and reproducibly resolvable.Source: §D.5 · pp. 39–40

E. Case Study

The case study follows one round of the three-module loop, showing how a synthesized question, a forecasting trajectory, and a distilled skill become inputs to later evolution.

PROMETHEUS question synthesis

  • PROMETHEUS first synthesizes a forecasting question, comparing a baseline draft with one sharpened by a task-synthesis skill.The skill-steered draft targets a weakness in the forecaster.Source: E. Case Study · p. 40

BRANCHUS forecasting

  • BRANCHUS then searches for evidence and produces a probabilistic answer to the synthesized question.The case traces its search and prediction teams through the forecast.Source: E. Case Study · p. 40

ATHENA skill distillation

  • ATHENA distills a reusable skill from the forecasting trajectory, making the round’s question, trajectory, and skill raw materials for the next round.Source: E. Case Study · p. 40

E.1. Prometheus: Skill-Conditioned Question Synthesis

A task-synthesis skill steers PROMETHEUS toward questions in BRANCHUS’s weaker categories while preserving objective, reproducible resolution. Worked examples show this shift across sports and air quality.

Controlled comparison

  • 1.29 vs. 2.15 questions per shared event: skill-steered synthesis emits fewer questions and concentrates on harder numerical forecasts or threshold judgments.The controlled comparison uses the same underlying events and corpus, with and without a task-synthesis skill.Source: §E.1 · p. 40

Weightlifting example

  • The weightlifting example replaces a near-certain gold-medal question with the gold medallist’s total, a numerical outcome referenced to the 268 kg Asian Games Standard.The baseline question is nearly determined by the public entry totals, while the exact total depends on competition attempts and results.Source: §E.1 · p. 41

Air-quality example

  • For Delhi AQI, the skill-steered question asks for the bulletin’s exact Index Value instead of a coarse category already suggested by the recent trend.The comparison holds source material, objectivity, and reproducible resolvability fixed while shifting difficulty toward a BRANCHUS weakness.Source: §E.1 · p. 41

E.2. Branchus: A Worked Prediction

The worked forecast separates the FDA decision from whether it is announced by a cutoff, then applies evidence and outcome-specific timing to estimate the public outcome. The case also contrasts skill-based and parameter-level evolution on the same question.

Evolution configurations

  • BRANCHUS was shown in baseline, skill-augmented, and post-RFT configurations, distinguishing behaviour-level skill use from parameter-level evolution.The skill configuration leaves weights untouched; the RFT configuration uses predict-team weights after fine-tuning.Source: §E.2 · p. 41

Timing evidence

  • Weekend-PDUFA precedents showed approvals public by Friday in 6/6 cases but CRLs in only 2/6, supporting different announcement probabilities by outcome.The team therefore treated “none” largely as CRLs or extensions not yet announced, rather than neutral uncertainty.Source: §E.2 · pp. 43–44

Prediction and resolution

  • Approved was the selected forecast, with probability 0.51; zilurgisertib was in fact approved and announced before the 22:30 ET cutoff.The forecast combined the decision probability with the chance of announcement by the cutoff; the release appeared at 19:00 ET on Friday.Source: §E.2 · p. 44

Search process

  • The search team decomposed the question from the first turn into the FDA decision and whether it would be public by the cutoff, then searched relevant announcement channels and precedents.With the skill, the same evidence took two rounds and one critic call rather than three search rounds.Source: §E.2 · p. 44

Effect of the skill

  • The skill changed evidence structure and weighting without adding evidence: reward rose from 0.827 to 0.854, and search took two rounds instead of three.The forecast shifted from (0.52, 0.16, 0.02, 0.30) to (0.56, 0.16, 0.01, 0.27).Source: §E.2 · p. 45

Effect of RFT

  • After RFT, the team selected approved at 0.55 and kept it selected across reasonable input ranges, with approved between 0.48 and 0.61.The RFT trajectory explicitly updates on pre-window silence and uses outcome-specific public-by-cutoff estimates.Source: §E.2 · p. 46

E.3. Athena: The Distilled Skill

ATHENA evaluates a trajectory, extracts a useful action, and organizes it into a four-level reusable skill. The worked skill generalizes a forecast decomposition from one FDA question to broader decision-and-observation settings.

Skill distillation

  • ATHENA evaluates actions with the outcome withheld, extracts and filters a useful experience, then organizes it into a four-level skill.The concrete procedure remains verbatim at the lowest level while more general guidance is formed above it.Source: §E.3 · p. 46

Evaluation finding

  • The scheduler’s early decomposition earned a high decomposition/coverage score because later steps could attach evidence to the right sub-question; the baseline found the same split only in its final search round.Source: §E.3 · p. 47

Four-level skill

  • The example skill generalizes from separating an FDA decision from its public release by cutoff to a cross-domain principle of separating an event from its observation.Its four levels progress from cross-domain principle to domain and question-type specialization, then a concrete Mirum procedure.Source: §E.3 · p. 47

Related papers

Loading 2610.29784v1…