PaperlayerView exact revision on arXiv

Source-linked AI summary

Aspire: Can Models Self-Evolve from Vague Goals?

Yuhao Wu, Jingyuan Zhang, Jiajun Shi, Yuxuan Zhang, Xinping Lei, Junting Zhou, Zexuan Wang, Yuchen Wu, Huan Zhou, Duo Wang, Yinzhu Piao, Yongchang Peng, Yunfeng Shi, Jin Chen, Zuo Wang, Jinkai Liu, Jiaheng Liu, Wenxuan Zhang, Shen Yan, Wenhao Huang, Ge Zhang

arXiv:2608.31111v1cs.CL

TL;DR

Learning from a broad capability goal requires deciding what to improve, how to improve it, and how to verify progress. ASPIRE benchmarks this challenge, and its studies find that goal interpretation takes more effort while retained gains remain difficult to achieve.

  • Problem

    Autonomous learning from broad capability goals requires deciding what to improve, how, and how to verify it, while humans still translate capability gaps into tasks and benchmarks.

    Source: §1 · pp. 1–2

  • Method

    ASPIRE gives agents a natural-language capability goal with hidden evaluation tasks, letting them choose data and updates and evolve model weights or the agent harness.

    Source: §1 · p. 2

  • Results

    Across three studies, vague-goal prompting shifts work toward goal definition, evaluated checkpoints are more common than retained improvements, and the strongest successor harness remains below a fixed reference.

    Source: §6 · p. 15

  • Takeaways & Limitations

    Completing update workflows is not itself evidence that target-aligned improvements have been retained.

    Source: §1 · p. 3

    Retained weight improvement is nonnegative because of safety rollback, so it is a selection outcome rather than evidence that every attempted update improves capability.

    Source: §3.3 · p. 7

Abstract

from arXiv · show

Many important forms of human learning begin with a vague goal, such as "become a better physicist" or "improve at research." Learners must interpret the goal, identify capability gaps, decide how to learn, and determine whether they have actually improved. In contrast, existing work on LLM self-evolution typically begins with tasks and evaluation metrics specified by humans, reducing self-evolution to optimizing an explicit objective rather than deciding what and how to learn. We introduce ASPIRE, a benchmark for vague-goal-driven self-evolution. ASPIRE provides only a natural-language capability goal while downstream evaluation tasks remain hidden. The agent must operationalize the goal by choosing data and update methods, constructing training and validation signals, and deciding when to evaluate. ASPIRE supports both model-weight and agent-harness evolution in a unified interactive environment and evaluates the resulting systems on a hidden, expert-authored set of 520 items spanning six goals. Our experiments show that vague goals redirect search effort toward goal interpretation. Current agents routinely complete training and harness-editing loops, but weight-level gains remain sparse and unstable, and the strongest evolved harness remains below the engineered Qwen-Agent reference. Agents often train on mismatched data and trust narrow self-evaluations, so local gains fail to transfer to hidden evaluation and continued search and training can erase earlier improvements.

1 Introduction

ASPIRE frames self-evolution from a broad capability direction as jointly deciding what to improve, how to improve it, and how to verify genuine gains.

1 Introduction

  • Unlike task-specified self-evolution, vague-goal learning requires agents to identify capability gaps and decide what to optimize as well as how.Prior systems search for ways to improve objectives whose task format, difficulty, metric, and success criterion have already been specified by humans.Source: §1 · p. 1, Figure 1 · p. 2
  • Because agent-built proxy gains may not transfer to the intended capability, ASPIRE keeps task specifications and evaluation items hidden from agents.The benchmark records agents’ decisions, actions, and self-evaluation trajectories to expose the gap between completing updates and genuine improvement.Source: §1 · pp. 2–3
  • Vague goals shift effort toward operationalization and underperform explicit-task references; weight gains are rare and poorly retained, while evolved harnesses remain below Qwen-Agent.Agents routinely complete data selection, training, checkpoint generation, and harness editing, but executing updates does not reliably produce retained target-aligned improvements.Source: §1 · p. 3
  • ASPIRE formalizes target operationalization and benchmarks weight and harness evolution using hidden evaluation across six capability goals.Its expert-authored evaluation contains 520 items; agents receive only a natural-language goal and cannot access evaluation items, reference answers, or per-item feedback.Source: §1 · pp. 2–3

2 Background

Existing benchmarks typically assume an executable task, feedback signal, and execution scaffold, while deployment requires turning vague needs into objectives, validation, and usable systems. Aspire isolates how broad capability goals are translated into concrete learning plans and realized as capability growth.

2 Background

  • Forward-deployed engineers turn general-purpose models into systems adapted to customer workflows, whose success depends on continued use and improvement rather than benchmark scores.The spread of these roles reflects that a capable model is not yet a working system.Source: §2 · p. 3
  • From the model’s perspective, deployment work involves interpreting vague goals, constructing potentially absent or unreliable feedback, and building or adapting the execution system.That system includes tools, context management, state, lifecycle logic, and verification interfaces.Source: §2 · p. 3
  • Aspire studies how broad deployment needs become learning objectives, data, training plans, and validation signals for weight or harness updates, distinguishing its focus from S3Gym and HarnessDev.S3Gym studies judging and reusing interaction experience under executable verification; HarnessDev studies building and maintaining the systems that carry models.Source: §2 · p. 3

3 Aspire: Self-Evolution Benchmark and Interactive Environment

Aspire makes self-evolution measurable with a controller-side evaluator whose benchmark definition, items, and decomposable reward are hidden from the agent; only bounded aggregate outcomes may be returned. The benchmark tests goal operationalization across model-weight and harness evolution, using a hidden six-goal evaluation set and a managed interactive environment.

3.1 Task Definition and Evolution Surfaces

  • Each vague natural-language goal requires the agent to choose its own data, update objective, and validation criteria, while the campaign keeps its evaluator hidden.A round is a bounded search-and-commit episode with a fixed decision model; the two evolution surfaces change either model weights or the harness while holding the decision model fixed.Source: §3.1 · p. 4

3.2 A Hidden, Query-Limited Evaluation Set

  • The hidden evaluation set contains 520 expert-authored items across six goals, with no item-level results exposed and only permitted aggregate scores returned.The goal slices contain 75 scientific and academic reasoning items, 110 humanities and social-science knowledge items, 100 health and medical reasoning items, 126 mathematical reasoning items, 89 integrated reliability items, and 20 academic and scientific writing task bundles.Source: §3.2 · p. 5
  • Under adaptive feedback, agents may use a bounded number of aggregate scores to guide later updates, while final-only evaluation returns one score after checkpoint submission.The hidden items and judging assets remain inaccessible; training datasets are checked for overlap, and the operational OOD definition requires evaluation items to be independently authored and excluded from agent training and validation data.Source: §3.2 · pp. 5–6

3.3 A Minimal Interactive Environment with Safe Retention

  • A single typed-action tool lets agents select data, updates, parameters, validation, and branching while the controller manages backend execution and constrains actions.Weight candidates follow a data–train–verify path; harness candidates are edited and frozen. The controller records provenance and lineage, and only verified weight candidates or frozen harness versions may be evaluated.Source: §3.3 · pp. 6–7
  • Score-gated weight evolution retains a selected checkpoint only when its raw score change is positive; otherwise, the controller rolls back to the incoming state.This nonnegative retained improvement is a consequence of safety rollback, not evidence that every attempted update improves capability; harness effects may be negative and do not imply deployment or recursive handoff.Source: §3.3 · p. 7

4 Experiments and Analysis

Across three studies, ASPIRE tests how vague goals affect post-training search, weight evolution, and harness evolution. Agents often complete training or editing loops, but gains on hidden evaluations are sparse, and all valid evolved harnesses score below the fixed Qwen-Agent reference.

4.2 RQ1: How Vague Goals Change Post-Training

  • Vague-goal scores are lower overall than explicit-task references, though changes vary by task: Claude scores 27.07 versus 32.90, and GPT-5.6 scores 29.58 versus 36.23.Claude under vague-goal prompting scores higher on HumanEval and BFCL, while GPT-5.6 scores higher on GSM8K.Source: §4.2 · pp. 8–9, Figure 4 · p. 9
  • Matched vague-goal Claude runs show more operationalization-associated effort and less active update time, while explicit systems exhibit different evaluation-feedback profiles.Vague-goal runs record 2,109 more thinking seconds, 0.61 more GPU-idle hours, and 1.27 fewer active-training/evaluation hours per matched pair. Their task-material and proxy-script accesses are 2.98× and 2.39× as dense; tool actions do not increase (188.5 to 179.2). Official GPT-5.6 has 2.63 evaluations per training start versus 0.57 for vague-goal Claude, while official Claude Max has 0.56.Source: §4.2 · pp. 8–9, Figure 4 · p. 9

4.3 RQ2: Self-Directed Weight Evolution from Vague Goals

  • Only 1/12 final-only model–goal means exceeds its base score: Qwen3.5-9B scientific and academic reasoning reaches 48.00, 2.67 points above base.Both runs rise from 45.33 to 48.00; Qwen3.5-4B has 0/6 means above base, and 3/24 individual final checkpoints exceed their base scores.Source: Figure 5 · p. 10, §4.3 · p. 10
  • Only two of 30 adaptive-feedback cells score above base, and only Terra’s Qwen3.5-4B mathematics checkpoint remains after rollback.Terra’s score rises from 17.86 to 20.10. Qwen3.5-4B Self’s science score rises from 44.00 to 45.33 but is not retained; no Self cell retains an above-base model.Source: §4.3 · pp. 10–11, Figure 7 · p. 12
  • Later adaptive checkpoints often recover from regression without surpassing base: 28/62 transitions rise, and later search beats the first evaluated checkpoint in 14/22 cells.For example, Qwen3.5-4B Self mathematics rises from 0.79 to 2.38 while remaining below its 17.86 base; blind final-only continuation also ends in poor checkpoints in three reported runs.Source: §4.3 · pp. 11–12, Figure 7 · p. 12
  • Numeric-label SFT coincides with answer-format collapse: five final checkpoints trained on single-digit targets produce only single-digit outputs and score 0, 0, 0, 6.141, and 0.This is a documented mismatch, not a randomized causal comparison; all 24 final evaluations completed without item errors.Source: §4.3 · pp. 12–13

4.4 RQ3: Harness Evolution

  • All three valid successor harnesses score below Qwen-Agent’s 28.64 task macro and 27.65 example micro; Sol is closest at 27.22/25.97.Terra scores 20.76/20.14 and Luna 19.32/18.33. The Qwen3.5-4B Creator produces no valid harness and receives the predeclared failure score of zero.Source: §4.4 · p. 13, Table 2 · p. 14

5 Related Work

ASPIRE is positioned against prior work that operationalizes self-evolution through explicit signals, tasks, or evaluators, and asks whether agents can instead turn vague goals into learning measured on hidden evaluation.

Self-training and self-evolution

  • Self-evolution methods use self-generated rationales, rewards, problems, or edits, but still measure progress with concrete signals; ASPIRE tests learning from a vague goal.Examples include STaR [31], ReST-EM [22], Self-Rewarding LM [30], SPIN [5], R-Zero [11], Absolute Zero Reasoner [35], and SEAL [37].Source: §5 · p. 14

Automated post-training and ML-agent benchmarks

  • Post-training and ML-agent benchmarks automate data, training, or algorithm choices, but typically retain specified targets, benchmarks, or evaluation scripts.RSIBench-Data finds that most continued searches finish below their own best checkpoint; AI4AI-Bench [6] finds most submissions do not change how the model learns, while PAST-Bench [27] tests retained experience.Source: §5 · p. 14

Broader benchmarks and agent evolution

  • MLE-bench [2], MLAgentBench [12], and RE-Bench [26] evaluate ML experimentation against quantifiable objectives, alongside work on automated pipelines, agent systems, and meta-evolution.Related approaches include AIDE [14], DS-Agent [9], SELA [7], ADAS [10], Darwin Gödel Machine [32], Hyperagents [33], Frontis-MA1 [28], and Red Queen Gödel Machine [13].Source: §5 · p. 14

Operationalized objectives and vague goals

  • Unlike systems that start with operationalized tasks or evaluators, ASPIRE hides downstream tasks; RQ1 tests this goal-side shift, while RQ2 studies externally or self-directed weight evolution.Shadow evaluations find that agents can complete engineering without advancing an unpublished research question [16], while a survey identifies research direction-setting as a bottleneck [3].Source: §5 · pp. 14–15

Memory, workflow, and skill evolution

  • Memory, workflow, and harness methods evolve agent processes; RQ3 asks whether the harness for goal interpretation, tool use, memory, and self-evaluation can evolve with runtime weights fixed.Evo-Memory [25] studies reusable memory, EvoFSM [34] edits workflows, SkillLearnBench [36] finds self-feedback can cause recursive drift, and HELIX [8] and Co-Harness [4] combine harness changes with verified or generated trajectories.Source: §5 · p. 15

6 Conclusion · Appendix · A Benchmark Construction and Experimental Contracts

ASPIRE frames self-evolution from vague goals as deciding what to learn when no agent-visible task or decomposable reward defines progress. Its studies show that completing training loops does not necessarily produce retained improvements or close the capability gap to a fixed reference.

6 Conclusion

  • The highest successor-harness mean remained below a fixed reference with runtime weights unchanged, showing that progress must be judged against the base model, not only the previous checkpoint.Closing the training loop is not the same as closing the capability loop.Source: §6 · p. 15
  • Vague-goal prompting was associated with more goal-definition work, while self-directed runs produced evaluated checkpoints far more often than retained improvements.These findings come from three studies, including an RQ1 process slice.Source: §6 · p. 15
  • Conclusions are bounded by six goals, evaluation-item coverage and scoring quality, one canonical adaptive-feedback run per configuration–goal cell, and controlled-access trace evidence.Broader goal coverage, repeated runs, refreshed evaluation sets, and recursive evolution that controls drift and unrelated capability loss remain future directions.Source: §6 · p. 15

A.1 Goal and Evaluation-Set Statistics

The evaluation set comprises 520 expert-authored items grouped across six vague goals, with a composite goal and a distinct task-bundle structure for academic and scientific writing.

A.1 Goal and Evaluation-Set Statistics

  • The adaptive-feedback protocol evaluates all six goals, with logic, reliability, and instruction following treated as one composite goal rather than separate goals.The current release does not identify those three components as separate goals.Source: §A.1 · p. 19
  • 520 newly authored, domain-expert top-level items are partitioned into mutually exclusive groups for six vague goals.Table 3 presents the six goals and their corresponding evaluation items.Source: Table 3 · p. 19, §A.1 · p. 19
  • Academic and scientific writing has 20 top-level task bundles counted as evaluation items, while its harness evaluator may score multiple examples per task.RQ3 therefore reports both task-macro and example-micro aggregation.Source: §A.1 · p. 19

A.2 Contamination-Control Details · A.3 Prompt Contracts

Contamination control combines expert-authored items, overlap audits, and registration gates, while versioned prompt contracts limit what agents can see and bind campaigns to fixed settings. The evaluation set is independently difficulty-screened and frozen by version.

A.2 Contamination-Control Details

  • Domain experts author items from scratch, using public benchmarks only as archetype and difficulty references.The benchmark references do not supply the evaluation items.Source: §A.2 · p. 19
  • Exact and semantic duplicates are removed across reporting groups, and dataset registration rejects overlap with hidden evaluation items before training.The pipeline also audits overlap against reference benchmarks; the registration gate applies before SFT, GRPO, or continual pretraining.Source: §A.2 · p. 19
  • Seed-2.0, GPT-5.2, and Gemini-3 independently screen candidates for difficulty, while the frozen evaluation set is versioned under a manifest and SHA-256.L0–L3 count model failures; timeouts and service errors are retried, and corrections create a new version rather than changing a running campaign’s set.Source: §A.2 · p. 19

A.3 Prompt Contracts

  • Versioned vague-goal prompts disclose the capability goal, trained model, action contract, budgets, and aggregate-only evaluation scores, but hide items, answers, rubrics, routing identifiers, and per-item feedback.The mathematical goal example requests difficult multi-step reasoning, precise calculation, and clear justification without naming evaluation items.Source: §A.3 · p. 19
  • Explicit-task prompts additionally disclose the approved task format, difficulty anchor, metric, and success criterion while preserving the same evaluation-content boundary.Source: §A.3 · p. 19
  • Campaigns bind setting, prompt-profile version, rendered-prompt SHA-256, and experiment identity, with checks at creation and recovery preventing silent contract or prompt-version changes.A vague-goal run cannot resume under an explicit-task contract or a different prompt version.Source: §A.3 · pp. 19–20

B Complete RQ1 Score and Trajectory Accounting … C.2 Feedback and Submission Protocol

Vague-goal prompting shifts agents toward more deliberation and LoRA use, while the accounting protocols distinguish evaluated, eligible, selected, and retained checkpoints and constrain feedback and submission. Official trajectories provide a separate comparison of evaluation and training activity.

B Complete RQ1 Score and Trajectory Accounting

  • For Claude Opus 4.8, vague-goal prompting was associated with 2,109 more seconds of decision-model thinking, 0.61 more idle hours, and 1.27 fewer active hours per matched run pair.Mean idle time moved from 0.6 to 1.2 hours, and mean active time from 8.0 to 6.7 hours.Source: B Complete RQ1 Score and Trajectory… · p. 20
  • Vague-goal prompting increased normalized density of task-definition reading by 2.98×, evaluation-script inspection by 2.39×, and LoRA/PEFT use by 3.09×.LoRA appeared in 89.8% of vague-goal runs versus 24.1% of matched explicit-task runs, with 33 pairs switching to LoRA use and none switching in the reverse direction.Source: B Complete RQ1 Score and Trajectory… · p. 20
  • Trace-level checks found 188.5 versus 179.2 mean actions per trajectory, so longer vague-goal narratives indicate additional deliberation rather than more executed work.Training starts were counted from concrete launch commands, and these trace-level tool calls are broader than the accepted controller actions defined in Section 3.1.Source: B Complete RQ1 Score and Trajectory… · p. 20
  • Vague-goal Claude Opus 4.8 averaged 3.54 evaluations and 6.17 training starts per trajectory, versus 2.89 and 5.14 for official Claude Opus 4.8 Max.Official GPT-5.6 averaged 39.21 evaluations and 14.91 training starts; the respective evaluations-per-training-start ratios were 0.57, 0.56, and 2.63. Official Claude Opus 4.8 Max had 165,851 thinking characters per trajectory versus 30,895 for official GPT-5.6.Source: B Complete RQ1 Score and Trajectory… · pp. 20–21

C.1 Outcome Accounting and Rollback

  • Only verified checkpoints with registered data, admitted updates, and published artifacts enter evaluated lineage; selection uses fixed eligibility and tie-breaking rules, and retention rolls back to base unless the selected score exceeds it.Best evaluated, selected, and retained outcomes are reported separately because an intermediate high score may come from an incomplete run, while a nonnegative retained score may reflect rollback rather than learning.Source: §C.1 · p. 21

C.2 Feedback and Submission Protocol

  • The feedback protocol allows repeated aggregate-score queries within an allowance, while final-only permits one final-checkpoint evaluation and returns no intermediate score.Experiments use separate versioned settings and current-lineage published checkpoints; accepted reservations remain counted after later failures, and hidden evaluation materials are never returned.Source: §C.2 · p. 21

C.3 Final-Only Protocol: Vague-Goal Weight Evolution

In this protocol, agents face six vague goals but can submit only one final checkpoint per run for hidden evaluation. Final-checkpoint gains are sparse, and rollback retains the base model in most runs.

C.3 Final-Only Protocol: Vague-Goal Weight Evolution

  • The protocol covered 24 runs crossing Qwen3.5-4B Self and Qwen3.5-9B Self with six goals; each run could use 40 GPU-hours but submit only one final checkpoint.Mean is the arithmetic mean of two separately executed final-checkpoint scores, not best-of-two or two samples from one checkpoint.Source: §C.3 · p. 21
  • Raw final-checkpoint macro average fell from 35.735 to 18.609 for 4B and 39.010 to 25.640 for 9B; rollback retained the base model in 21/24 runs.These are raw final-checkpoint scores before rollback.Source: Table 5 · p. 21, §C.3 · p. 22
  • Qwen3.5-4B had 0/6 Mean outcomes above base and Qwen3.5-9B had 1/6; scientific and academic reasoning was the only positive Mean.Its two runs each scored 48.00, but only 26 correct items overlapped and 20/75 items flipped which run was correct.Source: §C.3 · pp. 21–22
  • All 24 final checkpoints completed evaluation on the first judge attempt, and all 2,080 item outputs were nonempty with no evaluation errors.The evaluation used a local process, no-thinking checkpoint inference, and gemini-3.5-flash with an 8,192-token thinking budget and 10% fail-closed threshold.Source: §C.3 · p. 21
  • Five checkpoints used numeric-label MMLU SFT, and all 279 corresponding evaluation outputs were single digits; their final-checkpoint scores were 0, 0, 0, 6.141, and 0.The selected training snapshots contained 21,000 single-digit targets.Source: §C.3 · p. 22
  • Long blind training lineages still ended with low scores: three runs spent 26.570–32.286 GPU-hours across 17–20 jobs and scored 0.159, 2.540, and 0.Because only final checkpoints were evaluated, these results do not identify the best intermediate checkpoint or establish monotonic degradation.Source: §C.3 · p. 22

C.4 Adaptive-Feedback Protocol Results · C.5 Trajectory Evidence Boundary · C.6 Compute and Resource Accounting

Only one adaptive-feedback checkpoint, Terra’s mathematics run, is retained above its base score; the trajectory evidence supports claims about exploration and selection, while compute accounting covers training GPU-hours rather than inference.

C.4 Adaptive-Feedback Protocol Results

  • Only two of 30 adaptive-feedback configuration–goal cells beat base: Qwen3.5-4B Self science rose from 44.00 to 45.33, and Terra mathematics from 17.86 to 20.10.Terra mathematics was the only retained improvement under the complete-run selection rule.Source: §C.4 · p. 22
  • Across 62 consecutive checkpoint transitions, 28 increased, 13 tied, and 21 decreased; later search beat the first checkpoint in 14/22 cells.Only eight of the 22 multi-checkpoint trajectories were non-decreasing, and Terra’s retained mathematics score of 20.10 was followed by 2.78 and then 18.17.Source: §C.4 · p. 22
  • Search profiles varied: Luna evaluated 22 checkpoints, Terra 19, and Sol 33, but Sol produced no checkpoint above its corresponding base score.Luna registered 28 plans and used 89.57 settled GPU-hours; Terra registered 33 plans, including six GRPO updates, and used 91.73 GPU-hours; Sol registered 40 plans and used 76.56 GPU-hours. Sol made 36 evaluation attempts and 23 search actions across 12 distinct dataset names.Source: §C.4 · p. 22
  • Qwen3.5-4B Self imported GSM8K or Hendrycks mathematics data in 30/32 events, while Qwen3.5-9B Self used mathematics data in 11/15 imports.The 4B configuration used GRPO in 4/12 plans and the 9B configuration used SFT in 9/11; these counts describe search allocation, not why checkpoints gain or lose capability. The configurations generated 20,396 and 28,135 agent events but registered 12 and 11 training plans, respectively.Source: §C.4 · p. 22

C.5 Trajectory Evidence Boundary

  • The portable trajectory artifact supports claims about exploration, checkpoint progression, selection, and rollback, but not item-level causal analysis.It excludes hidden evaluation items, raw model messages, command outputs, and per-item judge reasons; deterministic extraction covered all cells and recovered 124 training plans and 95 candidate-evaluation attempts.Source: §C.5 · p. 22

C.6 Compute and Resource Accounting

  • Final-only training used 59.269 GPU-hours for Qwen3.5-4B and 89.967 for Qwen3.5-9B; RQ3 did not update weights.Table 6 summarizes resource accounting, and training GPU-hours exclude inference, judging, deployment, and idle controller time; creator-model and runtime inference therefore fall outside the training-GPU ledger. The adaptive-feedback design is complete at 5×6 but is not a multi-seed estimate.Source: §C.6 · p. 23, Table 6 · p. 23

D RQ3 Creator-Trajectory Case Analysis

The three valid GPT-5.6 creator trajectories show how validation-driven harness changes can fail on hidden tasks or improve failure handling without raising scores. These are case diagnoses from one inspected hidden-set execution per frozen successor, not a stable ranking.

Scope and evidence boundary

  • The analysis covers three valid GPT-5.6 creator trajectories and one inspected hidden-set execution per frozen successor; evaluation items and scores were not returned to creators.Table 2 reports means over three executions, while the task examples diagnose mechanisms in one execution and are not additional repetitions or a stable ranking.Source: D RQ3 Creator-Trajectory Case Analysis · p. 23

Luna

  • Luna stopped after a template reached 8/8 on three similar validation prompts, but on hidden tasks it reframed technical questions as study-design problems instead of answering them.The five-part template covered the research question, design, analysis, risks, and limitations.Source: D RQ3 Creator-Trajectory Case Analysis · p. 23

Terra

  • Terra’s missing final-answer invariant led two tasks to submit partial drafts, with scores dropping from 75.00 to 2.57 and from 52.33 to 2.91 under H0.These cases account for approximately 70% of the aggregate decline in that execution.Source: D RQ3 Creator-Trajectory Case Analysis · p. 23

Sol

  • Sol tested four validation task types over 3 hours 11 minutes, then selected a harness with a completeness check that never triggered on 20 inspected evaluation items.The check supported improved failure semantics rather than a score gain; Sol rejected locally stronger variants for latency, repeated calls, fabricated numbers, or scorer misses.Source: D RQ3 Creator-Trajectory Case Analysis · p. 23

Related papers

Loading 2608.31111v1…