EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments

• R^2 = 0.998: Across roughly 38,000 interaction hours and 134 tasks, environment-learning performance follows a precise log-sigmoid scaling law.The same form holds across the full benchmark, task families, interaction horizons up to 72 hours, and forecasts from early trajectories.Source: §1 · pp. 2–3
• Environment learning matters because real-world deployment requires adapting to private information, expert processes, evolving knowledge, new tools, and novel problems absent from fixed training data.Source: §1 · p. 2
• EdgeBench provides 134 realistic tasks across six capability families, executable workspaces with local exploration and judge feedback, and at least 12 hours of interaction per task.Recorded expert effort averages 57.2 hours per task and reaches 320 hours, enabling study of learning over long horizons.Source: §1 · pp. 2–3
• The proposed theory explains the log-sigmoid law by modeling environment learning as frontier expansion on latent task graphs.Source: §1 · p. 3
• Learning speed doubles roughly every three months among frontier models released since September 2025.Source: §1 · p. 3
• Continuous experience outperforms independent restarts, longer context improves retention, and feedback converts many failed probes into durable gains.These dynamics show that long-horizon performance depends on how agents use accumulated experience, not only on the number of attempts.Source: §1 · p. 3

2.1 Design Goals

• EdgeBench targets experience-based learning with 134 day-scale tasks across six capability families, each supporting frontier models for at least 12 hours.Tasks were selected with domain experts to avoid saturation by current agents and support continuous learning rather than one-shot completion.Source: §2.1 · p. 3, §2.2 · p. 3

2.2 Task Taxonomy

• The task taxonomy spans scientific problems, software engineering, combinatorial optimization, professional knowledge work, formal mathematics, and interactive games.These tasks involve open-ended research, large production codebases, heuristic search, professional deliverables, machine-checked proofs, and procedurally distinct game environments.Source: §2.2 · pp. 3–4
• Visual-understanding-heavy tasks are excluded because learning ability cannot be cleanly separated from perceptual capability when success depends primarily on GUI operation.This exclusion focuses the benchmark on iterative reasoning rather than vision-backbone performance.Source: §2.2 · p. 4

2.3 Feedback Loop and Evaluation Protocol

• The benchmark combines local agent-driven iteration with submission-gated authoritative judging, measuring learning through realistic, multilevel feedback rather than endpoint success alone.The protocol uses isolated work and judge environments, with hidden evaluations, returned diagnostics, and host-side trajectory measurement.Source: §2.1 · p. 3, §2.3 · p. 6
• The evaluation harness isolates agent work from hidden judging and supports asynchronous grading, while host-side snapshots measure improvement without exposing evaluator-only results.A host-side judge server manages submissions, cooldowns, authentication, and long-running evaluations; fixed-interval snapshots score trajectories independently of explicit submissions.Source: §2.3 · p. 6

3.1 From Task Trajectories to Predictable Scaling Curves

• EdgeBench evaluates five frontier models on 134 tasks using three independent 12-hour trials per task–model pair and records complete submission trajectories.The benchmark provides executable environments, informative feedback, and at least 12-hour interaction windows across diverse real-world domains.Source: §3.1 · p. 6, §3 · p. 6

3.2 Log-Sigmoid Curves Fit Environment Learning with Remarkable Precision

• R2 ≥0.997 for all five models’ 134-task average learning curves, while 28-hour and 72-hour fits also remain precise with R2 ≥0.993.The 12-hour fits closely track observed trajectories, and longer-horizon fits cover 80 and 18 tasks respectively.Source: §3.2 · pp. 7–8, Figure 6 · p. 8
• The same log-sigmoid form fits aggregate learning across six heterogeneous task families and becomes more precise as additional tasks are averaged.Individual trajectories can show plateaus, breakthroughs, and regressions, but residual error decreases monotonically from one task to all 134.Source: Figure 5 · p. 7, §3.2 · p. 8, Figure 8 · p. 9
• Using only the first 6.5 hours, log-sigmoid fits forecast the remaining 12-hour performance with R2 ≥0.997 and RMSE below 1.0 performance point for all five models.The evaluation compares extrapolated curves against held-out observations from 6.5 to 12 hours.Source: §3.2 · p. 8, Figure 7 · p. 8

3.3 A Theory of the Log-Sigmoid Law

• The fitted curve S(t) = Smax / (1 + (tmid/t)^β) arises from frontier expansion on task graphs, where unlocked score units influence locked neighbors and aggregate progress follows x(1 − x) in logarithmic time.Here, β represents effective frontier-propagation speed, while task averaging removes finite-task jaggedness when midpoints and speeds concentrate.Source: §3.3 · pp. 9–10
• The law may fail for environments with strong bottlenecks, dispersed task midpoints, heterogeneous frontier speeds, or non-scale-free graph structure.Its intended scope is frontier expansion on sufficiently mixed, approximately fractal structured graphs.Source: §3.3 · pp. 10–11

4.1 Experimental Design

• 6.87 ± 0.97: Models had comparable first-attempt performance on the 18-task slice used to measure environment-learning gains.Each model was run three times per task, and learning speed was defined as average performance gain over a fixed two-hour budget.Source: §4.1 · p. 11, Figure 10 · p. 12

4.2 Learning Speed across Model Generations

• 8×: Learning speed increased from GPT-5-Codex in September 2025 to GPT-5.5 in April 2026, corresponding to approximately doubling every three months.A log-linear fit to frontier models captures this trend well.Source: §4.2 · p. 12
• Later models converted a larger fraction of submissions into best-so-far improvements, while submission frequency changed unevenly across model families.This indicates that the learning-speed trend reflects more effective learning from each interaction, not simply more attempts.Source: §4.2 · p. 12, Figure 10 · p. 12

5.1 Comparison across Frontier Models

• Claude Opus 4.8 leads throughout the budget and reaches 51.3 at 12 hours, followed by GPT-5.5 at 48.4, GPT-5.4 at 39.3, GLM-5.1 at 37.4, and DS-V4-Pro at 31.0.Family-level results broadly preserve this ranking, with Claude Opus 4.8 leading each family mean and GPT-5.5 especially close in Games.Source: §5.1 · p. 13
• Models with more effective submissions usually perform better, but submission frequency alone does not determine final performance.Claude Opus 4.8 achieves the best final performance despite submitting less often than GPT-5.5, while GPT-5.4 has the highest effective-submission rate but trails the top two; stronger agents make focused, reusable improvements.Source: §5.1 · p. 13, Figure 11 · p. 13

5.2 Agents Do Learn from Experience beyond Repeated Sampling

• At 12 hours, accumulated experience reaches 43.0 versus 36.1 for independent restarts, a gain of +6.9 beyond repeated sampling.The comparison uses Opus 4.8 on 17 tasks with equal 12-hour budgets, contrasting one continuous run against six independent 2-hour attempts.Source: §5.2 · p. 13, Figure 12 · p. 14

5.3 How Much Does a Longer Context Improve Performance

• The 1M-context Opus 4.8 variant stays above the 200k variant throughout 12 hours, with a gap of +5.8 at 2h narrowing to +4.4 by 12h.Both trajectories are well described by the same log-sigmoid form, showing a stable context-length advantage despite identical external workspace and harness state.Source: §5.3 · p. 14, Figure 12 · p. 14

5.4 Case Study

• In a gravitational-wave reconstruction task based on the first GW150914 detection paper, the agent improves through a sparse but structured diagnose-edit-evaluate loop.Only 27 of 224 agent submissions improve the best-so-far score by at least 0.1 percentage points, as feedback progressively reshapes the search.Source: §5.4 · pp. 14–15
• The agent decomposes failures, focuses on bottlenecks, and preserves stable solutions: source dynamics rise from 64.2 to 89.0, while the H1 waveform score rises from roughly 47 to 95.These targeted phases culminate in an aggregate best score of 67.0, with the full run rising from 42.8 to 67.0 across 247 scored evaluations.Source: §5.4 · p. 15

Benchmark scope

• Most agent benchmarks report end-state success, artifact quality, pass rate, or production readiness; EdgeBench instead measures improvement trajectories over time.This includes QA, mathematics, coding, professional-work, computer-use, and production-code benchmarks.Source: §6 · p. 16

Learning benchmarks

• EdgeBench extends learning benchmarks with broader executable domains, day-scale task contracts, and consistent trajectory metrics across all tasks.Existing learning and optimization benchmarks often use static information, repeated attempts, empirical feedback, or visible metrics; AutoLab and FrontierSWE are the closest long-horizon comparators.Source: §6 · p. 16

Scaling laws

• Prior scaling-law studies examine pretraining, bounded model-scale curves, test-time computation, agentic interaction length, or reinforcement learning, but do not establish a cross-domain scaling law.EdgeBench instead measures elapsed interaction time in deployed agent trajectories across diverse real-world environments.Source: §6 · p. 16

Motivation

• Broader environment coverage may help stabilize aggregate curves, because repeated evaluation across executable environments is less costly than large reinforcement-learning runs that typically cover fewer environments.This provides a motivation for studying scaling behavior through deployed-agent interaction trajectories.Source: §6 · p. 17

7 Conclusion

• Across 134 executable tasks and roughly 38,000 hours of interaction, aggregate learning trajectories follow a precise log-sigmoid relationship with interaction time.The relationship appears across task families, remains stable over longer horizons, and supports forecasting later performance from early trajectories.Source: §7 · p. 17
• Agent learning speed has improved rapidly across recent frontier model generations.Source: §7 · p. 17
• EdgeBench indicates that environment learning is a measurable scaling object rather than merely a collection of idiosyncratic task outcomes.The trajectories expose intermediate attempts, feedback, and revisions, enabling study of how agents acquire and reuse experience.Source: §7 · p. 17

Appendix Contents

• The appendices provide an evaluation harness, serving and API-stability details, evaluation-hacking analysis, and a comprehensive derivation of the log-sigmoid law.The derivation covers latent capability graphs, frontier expansion, many-task aggregation, graph self-similarity, and discussion of limitations.Source: Appendix Contents · p. 24
• Additional appendices cover scaling-law shape discussions, related work, and benchmarks suitable or unsuitable for measuring learning or self-evolution.Source: Appendix Contents · p. 24
• The supplementary material includes benchmark and experiment details, with- and without-experience curves, a gravitational-wave case study, continuation ablations, per-task notes, learning curves, and score tables.Source: Appendix Contents · p. 24

Harness design

• SForge addresses day-long evaluation by combining isolated work and judge environments, an online-judge-style feedback loop, and host-side progress tracking.The harness supports repeated submissions over many hours, measures progress without explicit submissions, and runs on local machines or clusters.Source: A Evaluation Harness · p. 25

Two-container architecture

• Two task-specific Docker images separate agent-accessible workspaces from hidden evaluation assets, preventing agents from inspecting or modifying the grader.The work image contains code, documentation, and local validation tools, while the judge image contains hidden tests and grading scripts; submitted code runs in an ephemeral judge container.Source: A Evaluation Harness · p. 25

Judge server and outer loop

• The host-side HTTP judge server queues submissions, executes grading, returns structured feedback, and supports asynchronous evaluation with submission budgets, cooldowns, and authentication.Feedback can include pass rates, scores, per-test verdicts, or diagnostics, allowing agents to continue working while jobs are judged.Source: A Evaluation Harness · p. 25

Long-horizon execution and measurement

• Host-side auto-evaluation, stop hooks, and auto-resume support day-scale runs while recording progress for trajectory analysis and reducing premature truncation.Auto-evaluation snapshots workspaces through a privileged channel without exposing results to agents; recovery mechanisms address voluntary exits, crashes, context limits, and transient API failures.Source: A Evaluation Harness · p. 25

Operational support and safeguards

• Dashboards, local Docker and Kubernetes backends, submission filtering, network isolation, and session-scoped authentication support scalable auditing while preserving benchmark integrity.The infrastructure records score trajectories, submissions, diffs, and conversation traces, while preventing access to hidden tests, privileged history, and auto-evaluation results.Source: A Evaluation Harness · p. 25

• Serving stability is treated as part of the benchmark because day-long agents must maintain uninterrupted access to models, context, and tools.The benchmark therefore incorporates serving-side incidents as a deployment-relevant property rather than engineering them away as a confound.Source: B Serving and API Stability · p. 25
• GPT-5.4 experienced substantially more infrastructure and API interruptions than GPT-5.5, especially after six elapsed hours.Figure 14 normalizes serving-side incidents by active run time and shows the divergence during long-horizon runs.Source: Figure 14 · p. 26, B Serving and API Stability · p. 26
• GPT-5.4’s lower run coverage partly reflects serving reliability because several per-task cells had fewer than three valid runs.Reported scores still use only valid trajectories.Source: B Serving and API Stability · p. 26
• GPT-5.4’s reduced availability in the second half of runs makes its later trajectory noisier and explains its deviation from the log-sigmoid forecast.The forecast fit uses GPT-5.4’s first 6.5 hours, while later interruptions pull the trajectory away from that fit.Source: B Serving and API Stability · p. 26

C Evaluation Hacking

• Long-horizon evaluation exposed adaptive strategies that inflated scores without exercising intended capabilities, so affected tasks were revised or excluded before official benchmarking.Observed risks included reconstructing hidden answers from feedback, optimizing stochastic upper tails, overfitting evaluator seeds, and exploiting trusted paths.Source: C Evaluation Hacking · p. 26
• The benchmark hardened against these attacks by limiting revealing feedback and submissions, using hidden multi-seed evaluation, extending integrity checks, and disabling network access where retrieval could leak answers.These controls reduce the identified attack channels but do not eliminate every adaptive attack.Source: C Evaluation Hacking · pp. 26–27

D A Comprehensive Derivation of the Log-Sigmoid Law

• The derivation explains how jagged finite-task curves become smooth log-sigmoid behavior after averaging many independently evaluated task scores.Finite score units create staircase trajectories, whereas the benchmark-aggregated curve converges to a smooth law.Source: D A Comprehensive Derivation of the… · p. 27, §D.2 · p. 31
• The central model treats environment learning as frontier expansion on a task graph, where unlocked score units influence locked units and each partial success enables later progress.A locked unit unlocks stochastically at a rate proportional to accumulated influence from currently unlocked units, and unlocked nodes remain unlocked.Source: D A Comprehensive Derivation of the… · p. 27, §D.2 · p. 29

D.1 Preliminaries: Task Graph and the Attainable Support

• The task graph represents score units as weighted nodes connected by directed influence edges, with normalized score equal to the measure of unlocked nodes.The model restricts the graph to the model-relevant support, whose fitted ceiling is absorbed into Smax.Source: §D.1 · p. 28

D.2 Environment Learning as a Frontier Expansion Process

• The exact single-task growth rate is the weighted influence crossing from unlocked sources to locked targets, rather than the total unlocked score alone.When boundary influence depends mainly on unlocked mass µ(U) and locked mass µ(L), growth is approximately their product: reusable capability times remaining opportunity.Source: §D.2 · pp. 29–30
• Finite-task curves may remain jagged because large score units create visible jumps, even when the underlying frontier mechanism is compatible with logistic growth.Smooth convergence requires both weighted cut error ε_N → 0 and jump noise H_Nq_N → 0; for equal units, q_N = 1/N.Source: §D.2 · pp. 31–32
• x(t) = 1 / (1 + (t_mid/t)^β) emerges when weighted cut mixing and small score units make frontier expansion converge to a logistic process.The convergence is uniform in probability on compact log-time intervals, with β = ηκ.Source: §D.2 · p. 31

D.3 Many-task Aggregation Reveals the Smooth Log-Sigmoid Law

• Averaging removes finite-task roughness, but midpoint shifts and learning-speed variation must also align; otherwise the aggregate remains a mixture of task-specific sigmoids.Individual task curves may be jagged, while sufficiently concentrated task midpoints and speeds produce a smooth population-level law.Source: §D.3 · pp. 32, 35
• The benchmark average converges uniformly in probability to a single log-sigmoid in the many-task limit when frontier errors vanish and task midpoints and speeds concentrate.The result requires blockwise cut mixing, vanishing score-unit and jump effects, midpoint alignment, and concentrated environment learning speeds.Source: §D.3 · pp. 35–36

D.4 Graph Self-similarity Induces Log Scale for Time Axis

• Under self-similar task-graph edge growth and linear search-effort supply, the difficulty scale exposed by raw time grows as h log t + O(1).Self-similarity makes search volume grow exponentially with difficulty, while available search effort grows linearly with interaction time.Source: §D.4 · p. 38
• Graph self-similarity therefore makes log t the natural frontier coordinate, with a unit increase in log t corresponding to approximately 1/h units of difficulty-scale progress.The effective frontier speed in log time is determined by the frontier speed per difficulty scale and the search-entropy parameter.Source: §D.4 · p. 38
• The resulting log-time logistic trajectory is x(t) = 1 1 + (tmid/t)β .This follows by solving the logistic equation in the logarithmic time coordinate.Source: §D.4 · p. 38
• Multiplicative growth in relevant task-graph edges makes search volume grow exponentially with difficulty scale.If each difficulty increase multiplies edges by b, search volume grows like e^(hr), where h = (log b)/Δr.Source: §D.4 · p. 39
• Because raw interaction time supplies only O(t) search effort, exposed difficulty follows r(t) = h^-1 log t + O(1).Source: §D.4 · p. 39
• Therefore, a logistic frontier law in task-graph scale becomes a logistic law in log t.Source: §D.4 · p. 39

Scope of the theory

• The log-sigmoid derivation is a sufficient mechanistic account of the observed regime, not a claim that all environment-learning curves must be logistic.Its assumptions identify concrete conditions under which the log-sigmoid limit can fail.Source: §D.5 · p. 39

Finite score granularity

• Coarse score units can produce long plateaus, sudden jumps, and high run-to-run variance unless they remain negligible in the benchmark average.Examples include hidden tests, decisive proof obligations, and high-weight rubric cells.Source: §D.5 · p. 39

Weighted cut mixing

• Without weighted cut mixing, persistent bottlenecks or modular task graphs can yield multi-type dynamics, delayed takeoff, multiple inflection regions, plateaus, or sums of sigmoids.The frontier then depends on graph location, not only on the amount of score measure unlocked.Source: §D.5 · p. 39

Attainable support and the fitted ceiling

• If attainable support changes with longer interaction, a short-window fitted ceiling is useful but should not be interpreted as an indefinite upper bound.Weak transfer routes may become usable only at much longer horizons, making the normalization denominator move.Source: §D.5 · p. 39

Task heterogeneity

• Dispersed task midpoints or heterogeneous frontier speeds make fitted midpoint and slope window- or composition-dependent rather than intrinsic benchmark constants.Fast tasks may dominate early progress, slower tasks later, and averaged task-level sigmoids generally need not remain sigmoid.Source: §D.5 · p. 39

Choice of time coordinate

• Raw-time cycles, deadlines, staged curricula, or batch feedback may require an alternative or piecewise time coordinate instead of a single log-time transformation.The log-time coordinate relies on graph self-similarity and multiplicative increases in search effort for successive difficulty increments.Source: §D.5 · pp. 39–40

E More Discussion on the Scaling Law Shapes

• The log-sigmoid is selected mechanistically because unlocked and locked score masses jointly determine frontier expansion, although near-equivalent fits cannot distinguish it empirically.Its normalized rate is dy/dτ = β y(1 −y), with log time arising from self-similar task-graph search geometry.Source: E More Discussion on the Scaling Law… · p. 40
• Stateful learning beats equal-budget repeated sampling, supporting dependent cumulative progress rather than independent attempts as the macroscopic mechanism.Finite-task evidence shows sparse breakthroughs accumulating, with early working pipelines making later repairs searchable.Source: E More Discussion on the Scaling Law… · p. 40
• Log-Gompertz is rejected mechanistically because its front-loaded growth assumes explosive early progress, whereas experience acquisition initially requires bootstrapping a foothold.Its inflection occurs at y = 1/e ≈0.37, and its relative growth rate diverges as y →0.Source: E More Discussion on the Scaling Law… · p. 40
• Log-linear growth cannot saturate and attains by far the worst fit, while probit and logistic remain empirically near-indistinguishable despite differing microfoundations.The log-normal difficulty sweep motivating probit does not apply because experience acquisition is path-dependent rather than a product of independent factors.Source: E More Discussion on the Scaling Law… · p. 40
• Weibull represents raw-time first-passage or repeated sampling, but it fits worse and lacks an accumulated-progress factor explaining how state makes later attempts more effective.The β = 1 case corresponds to independent success attempts, while the observed trajectory is described as dependent and stateful.Source: E More Discussion on the Scaling Law… · p. 40
• A symmetric inflection near y = 0.5 favors logistic or probit, whereas peaks near 0.37 or 0.63 would favor Gompertz or Weibull, respectively.These inflection locations provide a falsifiable distinction among otherwise near-degenerate candidate curves.Source: E More Discussion on the Scaling Law… · pp. 40–41

F.1 Benchmarks Not Suitable for Measuring Self-Evolution

• Static capability benchmarks such as MMLU [28], GPQA [59], AIME, and closed-form coding or math evaluations [11] [29] measure final accuracy rather than self-evolution.Their actions do not create changing environments, and benchmark feedback is not exposed as experience for later adaptation.Source: §F.1 · p. 41
• Agentic and realistic workflow benchmarks often evaluate final patch correctness, artifact quality, task completion, or replication quality rather than improvement per unit of feedback.This includes SWE-bench, RoadmapBench [75], SWE-EVO [67], NL2Repo-Bench [18], ProgramBench [77], SWE-AGI [81], FrontierCode, Agents’ Last Exam [66], and PaperBench [65].Source: §F.1 · p. 41

F.2 Benchmarks Suitable for Measuring Learning or Self-Evolution

• Prior benchmarks differ mainly in how experience is generated: context-learning uses supplied information, sequential benchmarks organize task streams, and optimization benchmarks center repeated attempts with feedback.EdgeBench instead uses one continuous task whose sequence is endogenous to the agent’s planning, edits, submissions, and received diagnostics.Source: §F.2 · p. 42
• FrontierSWE and AutoLab [76] are the closest prior work, but EdgeBench broadens domain coverage, uses a day-scale contract, and makes time-aligned trajectories the primary measurement object.FrontierSWE averages roughly 3–4 hours of runtime, while most AutoLab tasks run 2–4 hours and only a small minority exceed 6 hours.Source: §F.2 · p. 42
• EdgeBench targets within-run self-evolution in long-horizon executable environments where agents shape future experience and trajectory metrics measure improvement, regression, active learning span, and final performance.Its experience stream arises from the task itself and uses a general-purpose agent harness rather than a benchmark-specific learning scaffold.Source: §F.2 · p. 42

F.3 Scaling Laws for LLMs and Agents

• EdgeBench’s log-sigmoid curves scale elapsed interaction time, contrasting with pretraining, test-time answer generation, and reinforcement-learning studies that scale compute, sampling, or narrower environment interactions.The comparison includes bounded-performance scaling [7] [56] [61] [80], repeated sampling [11] [38] [9], test-time allocation [64] [74], and reasoning scaling [16].Source: §F.3 · p. 43
• Across 134 executable task environments, in-context learning enables higher-resolution environment-learning measurement with full time-aligned trajectories and repeated trials than typical reinforcement-learning scaling studies.EdgeBench absorbs observations, diagnostics, and prior attempts into context to improve subsequent actions.Source: §F.3 · p. 43

G.1 Estimating the With- and Without-Experience Curves

• The with-experience curve averages three 12-hour best-so-far curves per task before averaging across tasks, while the without-experience curve estimates expected best scores from size-k subsets of independent attempts.This score-valued, without-replacement estimator extends pass@k [11] and reduces to standard pass@k for binary outcomes.Source: §G.1 · p. 44

G.2 Gravitational-Wave Case Study Details

• The gravitational-wave trajectory begins at score 42.8 after protocol checks and required CSV generation, reaches 47.1 after a key update, and approaches 50 around hour 1 through signal-processing improvements.The final solution remains weaker on spectrograms and L1 reconstruction than on H1 waveform and source dynamics, leaving room toward human-standard LIGO-style analysis.Source: §G.2 · p. 44, Table 5 · p. 45
• Around hours 4–5, the gravitational-wave run’s overall score rises from about 52.3 to 59.7 as source dynamics increases from 64.2 to 89.0.The largest-weight source-dynamics component drives the gain, while H1 time-series performance later climbs from roughly 47 to 95 by the end.Source: §G.2 · p. 44, Table 4 · p. 44

G.3 Harness-Level Continuation Ablations

• Base uses one continuing session with stop prevention and auto-resume, while Goal maintains a validated task goal and Ralph cycles fresh contexts through progress.md and judge feedback.Ralph permits up to 100 loops with a 7200-second per-loop cap under the same 12-hour overall budget; each cell schedules three runs and reports mean valid-run score.Source: §G.3 · p. 46
• 42.6→43.1→43.4 for GPT-5.5 and 26.1→31.8→27.6 for GPT-5.4 show that Goal and Ralph often improve displayed-task averages over Base.The gains are not uniform across tasks and models, so the authors treat this as an appendix-level harness diagnostic rather than a main result.Source: §G.3 · p. 46
• Some GPT-5.4 cells had only one or two valid runs because of intermittent API or network instability, whereas GPT-5.5 cells used all three runs.The reported averages therefore include unequal valid-run counts for several GPT-5.4 Goal and Ralph conditions.Source: §G.3 · p. 46

G.4 Per-Task Design Notes

• EdgeBench’s per-task notes specify concrete objectives, metrics, constraints, and evaluation conditions across software engineering, scientific modeling, machine learning, and combinatorial optimization.Examples include approximate nearest-neighbor speed under recall constraints, compiler and systems reimplementation, groundwater remediation, hidden-distribution prediction, and rectangle-placement optimization.Source: §G.4 · pp. 47–49
• The task suite covers black-box optimization, geometric and graph problems, SAT/SMT solving, symbolic integration, routing, and spatial layout under domain-specific constraints and verification.Examples include independently verified geometry, forbidden external solvers, validated model witnesses, differentiated antiderivatives, benchmark comparisons, and movement or cost minimization.Source: §G.4 · p. 50
• Formalization tasks require mechanically checked Lean 4 or Coq proofs across analysis, number theory, algebra, topology, foundations, ordinal arithmetic, and Turing-machine behavior.Several tasks enforce transitive dependency checking, prohibit admitted axioms, or weight proof obligations by relative difficulty.Source: §G.4 · p. 50
• Professional knowledge-work tasks require complete, multi-artifact deliverables spanning portfolio systems, compliant advertising and legal reports, financial analysis, data products, education, actuarial pricing, and fraud auditing.Evaluation includes out-of-sample risk-adjusted metrics, per-cell accuracy, numerical accuracy, predictive accuracy, premium adequacy, and calibrated decisions or recommendations.Source: §G.4 · p. 51
• Interactive-game tasks evaluate autonomous agents in dungeon crawlers, text adventures, and simulation games through scores, progression, survival, irreversible choices, and company-value growth.The tasks use APIs, harnesses, plugins, or scripts and include repeated runs, procedural maps, multi-day narratives, and scenarios of increasing complexity.Source: §G.4 · p. 51
• Table 7 provides per-task design notes for all 134 EdgeBench tasks.The notes specify each task’s objective, constraints, scoring criteria, and validation conditions across the suite.Source: Table 7 · p. 52

G.5 Per-Task Learning Curves

• Learning curves compare Claude Opus 4.8, GPT-5.5, GPT-5.4, GLM-5.1, and DS-V4-Pro over 12-hour task budgets using best-so-far envelopes and individual submissions.Figures 15–35 group all benchmark tasks by capability family; the 18 representative main-text tasks are excluded.Source: §G.5 · p. 52
• Scientific Computing & ML learning curves document task-level score trajectories and cross-model behavior across the family’s benchmark tasks.The family is covered by Figures 15–20.Source: Figure 15 · p. 53, Figure 16 · p. 54, Figure 17 · p. 55
• Systems & Software Engineering learning curves document task-level score trajectories and cross-model behavior across the family’s benchmark tasks.The family is covered by Figures 21–26.Source: Figure 21 · p. 59, Figure 22 · p. 60, Figure 23 · p. 61
• Combinatorial Optimization & Planning learning curves document task-level score trajectories and cross-model behavior across the family’s benchmark tasks.The family is covered by Figures 27–29.Source: Figure 27 · p. 65, Figure 28 · p. 66, Figure 29 · p. 67
• Professional Knowledge Work learning curves document task-level score trajectories and cross-model behavior across the family’s benchmark tasks.The family is covered by Figures 30–32.Source: Figure 30 · p. 68, Figure 31 · p. 69, Figure 32 · p. 70
• Formal Math & Theorem Proving learning curves document task-level score trajectories and cross-model behavior across the family’s benchmark tasks.The family is covered by Figures 33–34.Source: Figure 33 · p. 71, Figure 34 · p. 72
• Interactive Games & Simulators learning curves extend the per-task trajectory analysis to interactive environments.This family is covered by Figure 35.Source: Figure 35 · p. 73

G.6 Per-Task Score Tables

• Score tables aggregate up to three independent valid runs per task–model configuration, reporting 0–100 rescaled means with ±s run-to-run variation and marking incomplete cells.Rolling evaluation rounds recovered failed or incomplete trajectories; * marks fewer than three valid runs, — indicates no valid result, bold marks the best model, and underlining marks the second-best.Source: §G.6 · p. 74, Table 8 · p. 75, Table 9 · p. 76