PaperlayerView exact revision on arXiv

Source-linked AI summary

The Endless Exam: Mathematical Constructions from Today's Models toward Superintelligence

Muhan Zhang

arXiv:2609.24555v2cs.AImath.HO

TL;DR

Many benchmarks need scores that distinguish progress beyond human answers and tasks that can grow harder as models advance. The Endless Exam evaluates parameterised mathematical construction problems with automated verification and uncapped quality scores. Its results distinguish model performance, although none of the 30 published-frontier references is surpassed.

  • Problem

    Benchmarks may cease to distinguish leading models when they reach ceilings, creating a need for tasks that retain room for improvement and scores that capture advances beyond current answers.

    Source: §1 · p. 1

  • Method

    The Endless Exam evaluates 69 instances across fourteen parameterised construction families, verifies submissions, and scores them against published frontiers or verified construction baselines without capping improvements at 1.

    Source: §1 · pp. 1–2, §4.1 · p. 5

  • Results

    None of the 30 published-frontier references was surpassed; with tools at high effort, overall scores reached 143.16 for Astra, 119.35 for Luna, and 180.06 for Opus 5.5.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Uncapped scores and size–quality curves provide a way to track construction progress before and beyond published frontiers as task sizes increase.

    Source: §7 · p. 9

    The main comparison uses one evaluation per instance and configuration, so its instance-bootstrap intervals do not measure run-to-run variability.

    Source: §6 · p. 9

Abstract

from arXiv · show

We introduce the Endless Exam, a benchmark spanning fourteen parameterised families of mathematical construction problems, with verifiable scores that distinguish progress before and beyond published mathematical frontiers. Each submitted object is checked automatically for validity and assigned a relative quality score against a published frontier or construction baseline, without capping improvements at 1. The benchmark draws long-term challenges from open mathematical problems and generates larger instances by varying their parameters. Compact certificates allow large constructions to be verified without listing every element. Across nine models evaluated on 69 distinct instances, continuous quality scores distinguish performance even though none of the 30 published-frontier references is surpassed. Size-quality curves show how construction quality changes as problem size increases. We release the generators, verifiers, references, model responses and analysis to support continued measurement before and beyond human frontiers.

1 INTRODUCTION

The Endless Exam addresses the need for mathematical benchmarks that can distinguish progress beyond current frontiers and offer harder tasks as models advance. It applies verifiable, parameterised construction problems across fourteen families and evaluates models on 69 instances.

1 INTRODUCTION

  • Construction benchmarks can combine problems that may require new ideas with validity checks and quality measurements that run automatically.Varying parameters supplies harder instances under the same mathematical rules.Source: §1 · p. 1
  • The Endless Exam evaluates mathematical construction problems across fourteen parameterised families and 69 instances, with larger tasks generated by changing problem parameters.For progression-free sets, the evaluator checks for forbidden triples and counts points; the paper applies the same approach to graphs and codes.Source: §1 · p. 1
  • Published constructions remain far below proven bounds in some families, including seven-colour Schur partitions and progression-free sets in F6.The cited bounds are 8.1 and 11.6 times the best published construction sizes, respectively, and the optima remain unknown.Source: §1 · p. 1
  • The benchmark evaluates nine models in fourteen tool-free configurations and adds high-effort, code- and web-enabled evaluations for three models.The additional tool-assisted models are Astra, Luna, and Opus 5.5.Source: §1 · pp. 1–2
  • Its contributions include uncapped relative-quality scores, expandable construction families, and compact certificates for verifying constructions too large to enumerate.The suite evaluates 69 instances; submitted codes include one with over a trillion codewords.Source: §1 · p. 2

2 RELATED WORK

Prior work includes mathematical reasoning benchmarks, construction-verification tasks, collections of open problems, and systems for mathematical discovery. The Endless Exam combines graded construction scores with parameterised tasks to compare quality across problems and sizes.

Mathematical reasoning benchmarks

  • Mathematical benchmarks span expert problem solving, research-level proofs, Olympiad answers, and abstract reasoning.Examples include Humanity’s Last Exam, FrontierMath, IMProofBench, First Proof, IMO-AnswerBench, and ARC-AGI.Source: §2 · p. 2

Verifiable constructions and problem generation

  • MathConstruct varies construction-task parameters, while MathConstraint generates constraint problems and checks submitted constructions.The Endless Exam assigns graded scores so better valid constructions receive higher scores and quality can be compared across problem sizes.Source: §2 · p. 2

Open mathematical problems

  • Open-problem collections track unsolved questions and bounds, while other benchmarks verify solutions or derive parameterised tasks from mathematical literature.The Endless Exam instead scores construction quality on an uncapped scale, distinguishing improvement even after a published frontier is surpassed.Source: §2 · pp. 2–3

Mathematical discovery systems

  • FunSearch, AlphaEvolve, and AlphaTensor have improved mathematical constructions; the Endless Exam provides a common framework for comparing progress across systems, families, and sizes.Its framework combines relative quality, an uncapped overall score, and size–quality curves.Source: §2 · p. 3

3 THE EXAM

The Endless Exam uses automatically checked construction tasks whose parameters can vary while their mathematical rules remain fixed. Its 69 instances span fourteen families, with published-frontier references for some tasks and verified construction baselines for the rest.

3.1 DESIGN PRINCIPLES

  • Tasks specify mathematical constraints and an objective; the evaluator checks validity and computes the objective directly without requiring the optimum or using a model judge.Compact descriptions and certificates allow verification of objects too large to enumerate, while increasing parameters creates larger and potentially harder instances.Source: §3.1 · p. 3

3.2 FAMILIES AND INSTANCES

  • The fourteen families cover several areas, and the 69 instances use published frontiers for 30 tasks and verified algorithm-generated baselines for the remaining 39.All references have verified constructions and stay fixed within bench-v1.0; the mathematical optimum is unknown for each main-evaluation instance.Source: §3.2 · pp. 3–4

3.3 REPRESENTATIVE TASKS

  • Representative tasks include degree–diameter graphs, point placements in a square, and ternary trifference codes, each with a verifier suited to its construction.The code verifier checks the three-string condition algebraically and counts codewords from independent generator-matrix rows without enumerating the full code.Source: §3.3 · p. 4

3.4 ANSWER FORMATS AND VERIFICATION

  • Answers may be explicit objects or compact descriptions, which verifiers expand or certify through component checks and algebraic conditions.A second verifier independently checks validity and objective values; both verifiers agree on evaluated model constructions and all 69 references.Source: §3.4 · p. 4

3.5 EVALUATION PROTOCOL

  • Tool-free evaluations give each configuration one response per instance under a 128k output-token budget, while tool-assisted evaluations allow code and web access under resource limits.The tool-assisted track gives three high-effort models four CPU threads, 16 GiB memory, and a two-hour wall-clock limit.Source: §3.5 · p. 5

4 SCORING

The scoring framework compares verified constructions with fixed references using relative quality, then aggregates instance scores and measures progress toward proven bounds. It reports published-frontier and construction-baseline groups separately.

4.1 RELATIVE QUALITY AND OVERALL SCORE

  • Relative quality compares a valid maximisation answer as a/h(p) and a valid minimisation answer as h(p)/a; invalid or empty responses score zero.A score of 1 matches the reference, and improvements increase the ratio without a cap.Source: §4.1 · p. 5
  • For published-frontier instances, h(p) is the cited construction’s objective value; otherwise, it is the best verified algorithm-generated construction value.Exceeding a published frontier yields a candidate mathematical record.Source: §4.1 · p. 5
  • Scores of 1.049 and 1.064 distinguish eight-colour Schur constructions of 5,286 and 5,362 elements against the historical reference of 5,041.The benchmark uses 5,362 as its current eight-colour reference.Source: §4.1 · p. 5
  • The overall score is 100 times mean relative quality across 69 instances, so a score of 100 matches the references on average.The paper also reports group means with 95% confidence intervals bootstrapped over instances.Source: §4.1 · p. 5

4.2 GAP CLOSED

  • Gap closed measures logarithmic progress from a reference toward a proven bound, assigning 0 to no improvement and 1 to reaching the bound.The paper averages it over 64 instances with proven bounds; five LABS instances with a conjectured target are reported separately.Source: §4.2 · pp. 5–6

5 RESULTS

Tool-assisted models improve on construction baselines but surpass none of the 30 published-frontier references. In tested larger instances, construction quality can fall relative to references, while increasing search time yields limited gains across search-resistant families.

5.1 DISTINGUISHING TODAY’S MODELS

  • Tool-free overall scores span 7.55–91.90, while mean relative quality on published-frontier instances spans 0.07–0.76 despite a 0% breakthrough rate across all fourteen configurations.GPT-6 Astra at high effort reaches 0.76 (95% CI 0.69–0.83) on those instances.Source: §5.1 · p. 6, Table 2 · p. 7
  • With tools at high effort, overall scores reached 143.16 for GPT-6 Astra, 119.35 for GPT-5.6 Luna and 180.06 for Claude Opus 5.5, yet none surpassed a published frontier.The models improved on construction baselines for 33, 25 and 33 instances, respectively; Table 2 reports results on 69 instances, with its final three rows using code and web access.Source: Table 2 · p. 7, §5.1 · pp. 7–8

5.2 QUALITY AS PROBLEM SIZE INCREASES

  • If current instances become easy, larger dimensions, graph diameters or codeword lengths can provide more demanding tasks under the same mathematical rules.Size–quality curves compare construction quality with references as task size grows; Figure 4 shows curves with a 128k output-token budget.Source: §5.2 · p. 8, Figure 4 · p. 9
  • At graph diameter 10, mean relative quality ranges from 0.00 for DeepSeek V4.1 Flash to 0.30 for GPT-6 Astra; all four models’ spherical-code scores also fall from dimensions 10 to 16.Some models fail to produce valid larger constructions, while others return valid constructions that fall further below the reference.Source: §5.2 · p. 8, Figure 4 · p. 9
  • For 12-dimensional cap sets, Astra’s 12,544-point constructions reach relative quality 0.97 against the 12,928-point reference, but only 24.8% of the 50,571-point proven upper bound.In six dimensions, 112 points meet both the reference and proven upper bound; the gap between references and bounds also grows across graph and Schur size series.Source: §5.2 · p. 8

5.3 SENSITIVITY TO COMPUTATIONAL SEARCH BUDGET

  • Across eight search-resistant families, increasing search budgets from 10 to 600 seconds improves baseline quality by less than 10% on every measured instance; Heilbronn benefits more.Trifference is excluded because its baseline procedure does not use the search-time budget.Source: §5.3 · p. 8
  • The search-budget measurements describe the tested search algorithms, whereas tool-assisted models can use broader methods and discovery systems can develop new algorithms.Future benchmark versions can add larger instances while retaining earlier instances and references for score comparability.Source: §5.3 · pp. 8–9

6 LIMITATIONS

The evaluation covers selected models and reasoning-effort levels, with one main-comparison evaluation per instance and configuration. Its bootstrap intervals do not measure run-to-run variability, and answer formats and verification budgets constrain what can be evaluated.

6 LIMITATIONS

  • The study tests selected models and reasoning-effort levels, using one evaluation per instance and configuration in the main comparison.Repeated evaluations would quantify run-to-run variability, which instance-bootstrap intervals do not measure.Source: §6 · p. 9
  • Instance-bootstrap confidence intervals do not measure run-to-run variability, which repeated evaluations would quantify.Source: §6 · p. 9
  • Answer formats and verification budgets limit which constructions can be evaluated; future versions can extend certificate formats alongside instance sizes.Earlier releases would be preserved for comparison.Source: §6 · p. 9

7 CONCLUSION

The Endless Exam combines uncapped scores, parameterised problem families and compact certificates to measure progress before and beyond published frontiers. Tool-assisted models improve over construction baselines without surpassing any of the 30 published-frontier references, while size–quality curves extend comparisons to larger tasks.

7 CONCLUSION

  • The benchmark combines uncapped scores, parameterised families and compact certificates to measure mathematical progress before and beyond published frontiers.Source: §7 · p. 9
  • At high effort with tools, GPT-6 Astra, GPT-5.6 Luna and Claude Opus 5.5 score 143.16, 119.35 and 180.06, improving beyond construction baselines while surpassing none of the 30 published-frontier references.Source: §7 · p. 9
  • Historical Schur constructions and size–quality curves provide ways to continue measurement toward superintelligence, including on larger tasks.Source: §7 · p. 9

REPRODUCIBILITY STATEMENT

The repository supplies materials to reproduce the paper’s scores, tables and figures. The appendices catalogue task families and parameters, constructions, detailed results, verification, references, evaluation settings and tool-assisted results.

REPRODUCIBILITY STATEMENT

  • The repository provides generators, verifiers, reference constructions, model responses, inference settings and analysis scripts to reproduce every score, table and figure.Instances are specified by family, parameters and random seed; published constructions link to sources, with independent verifiers and regression tests included.Source: REPRODUCIBILITY STATEMENT · p. 10

APPENDIX

  • The appendices cover task families and parameters, representative constructions, detailed results, verification, references and bounds, evaluation settings, and code-and-web-access results.Source: APPENDIX · p. 14

A TASK CATALOGUE

The catalogue presents fourteen construction families with objectives, size parameters, checked examples, and reference context; its labels describe reference-search sensitivity.

A TASK CATALOGUE

  • Family pages pair objectives and size parameters with checked examples and references; the catalogue labels search sensitivity R, S, or U, with some references parameter-dependent.Spherical codes and Heilbronn each include two variants.Source: A TASK CATALOGUE · p. 14, Table 3 · p. 14

A.1 Cap sets; A.2 Corner-free sets

  • Cap sets avoid three-term arithmetic progressions as dimension grows, while corner-free grid sets forbid prescribed corners as grid size increases.Products lift finite caps; progression-free difference sets provide corner-free constructions at many sizes.Source: §A.1 · p. 15, §A.2 · p. 15

A.3 Schur colourings; A.4 Progression-free sets over finite fields

  • Schur colourings maximize an interval without monochromatic x + y = z, whereas finite-field sets avoid three-term progressions as the field or dimension varies.Recursive templates extend colourings; products and algebraic constructions provide finite-field candidates across dimensions.Source: §A.3 · p. 16, §A.4 · p. 16

A.5 Spherical codes; A.6 Low-autocorrelation binary sequences

  • Spherical codes pack vectors subject to angle constraints, while binary sequences are tested for high merit factor as their length increases.Dimension and angle control spherical packing; the sequence family tests whether construction quality persists at longer lengths.Source: §A.5 · p. 17, §A.6 · p. 17

A.7 Heilbronn triangles; A.8 Degree–diameter graphs

  • Heilbronn tasks maximize the smallest triangle area in square or triangular domains; degree–diameter graphs maximize vertices under degree and diameter limits.The two geometry domains are evaluated separately, while varying graph degree and diameter generates new instances.Source: §A.7 · p. 18, §A.8 · p. 18

A.9 Linear-equation-free sets; A.10 Covering designs

  • Linear-equation-free sets exclude nontrivial solutions to a selected relation, while covering designs minimize blocks needed to cover every required subset.Coefficients and interval size set the former task; (v, k, t) determine the universe, block size, and subsets in the latter.Source: §A.9 · p. 19, §A.10 · p. 19

A.11 Matrix multiplication; A.12 Mutually orthogonal Latin squares

  • Matrix multiplication minimizes scalar products in a bilinear scheme, while mutually orthogonal Latin-square tasks maximize the number of compatible squares of a given order.Block composition extends multiplication schemes; algebraic constructions and combinations of smaller designs supply Latin-square candidates.Source: §A.11 · p. 20, §A.12 · p. 20

A.13 Shannon codes; A.14 Trifference codes

  • Shannon codes require pairwise cyclic separation, whereas trifference codes require every triple to display all three symbols in at least m coordinates.Finite factors certify large Shannon-code products; linear and concatenated constructions provide compact trifference-code certificates.Source: §A.13 · p. 21, §A.14 · p. 21

B.1 CAP SETS: A SMALL INGREDIENT AT TWICE THE DIMENSION

The examples combine mathematical constructions from smaller ingredients with automatic verification, while the appendix compares model quality across references and problem sizes.

B.1 CAP SETS: A SMALL INGREDIENT AT TWICE THE DIMENSION

  • 12,544 points from a product of two 112-point caps form a valid 12-dimensional cap set, scoring 0.970 against the 12,928-point published frontier.The product construction makes dimensions add and cardinalities multiply.Source: §B.1 · p. 22

B.2 SHANNON CODES: BETTER FACTORS PRODUCE A LARGER CODE

  • Qwen3.8-27B builds a valid 1,384,128,720,100-word Shannon code at length 24, scoring 0.713 against a 1,941,100,559,147-word baseline and exceeding two 1012-word answers by 38.4%.Its factors are checked separately, and their product size is computed exactly.Source: §B.2 · p. 23

B.3 TRIFFERENCE: TEN ROWS DESCRIBE 59,049 CODEWORDS

  • 59,049 trifference codewords score 3 times the 19,683-word baseline, with a short certificate and verifier checks that avoid listing every triple.A 10 × 64 ternary generator matrix describes the code, and both verifiers accepted it.Source: §B.3 · p. 24

B.4 SCHUR COLOURINGS: SUCCESSIVE ADVANCES ON ONE SCALE

  • The current eight-colour Schur reference is 5,362, while Table 4 scores selected published constructions against the historical reference N0 = 5041.Relative quality preserves differences in magnitude that a binary success measure would collapse.Source: Published constructions · p. 25, Table 4 · p. 25

C.1 QUALITY ACROSS THE TOOL-FREE MODEL COMPARISON

  • Every tool-free configuration has a zero breakthrough rate on 30 published-frontier instances, yet mean relative quality ranges from 0.07 to 0.76 across those instances.The comparison also includes 39 construction-baseline instances.Source: §C.1 · p. 26

C.2, C.7–C.8 PARAMETER RANGES AND PROBLEM-SIZE EXPERIMENTS

  • The appendix separates small familiar instances, larger settings, and the main evaluation into ranges A1, A2, and A3; family coverage varies across groups.Problem-size experiments report mean relative quality at each size, including zero scores.Source: §C.2 · p. 27, Table 6 · p. 33

C.3 MODEL SIZE AND REASONING EFFORT

  • On 25 shared smaller instances, Astra’s mean relative quality is 0.87 at medium effort, 1.01 at high, and 1.06 at xhigh.On the main 69 instances, Astra medium and high return 68 and 69 valid constructions, respectively.Source: §C.3 · p. 28

C.4–C.6, C.9 SIZE–QUALITY CURVES

  • Size–quality curves track constructions as parameters grow; Astra’s degree-four graph grows from 192 to 195 vertices between diameters 5 and 7 while its reference grows faster.For codes, the lower panels also express size as per-coordinate growth for Shannon and ternary rate for trifference.Source: §C.4 · p. 29, §C.6 · p. 31, Figure 11 · p. 31

C.10 CODE-FAMILY RESULTS

Code-family results include trifference constructions above baseline, while the appendices describe failure cases, verification methods, and limits on representable constructions.

C.10 CODE-FAMILY RESULTS

  • Three tool-free configurations produce 59,049 trifference codewords at length 64, three times baseline; each matches baseline at the other three lengths.Astra medium and Claude Fable 5.1 medium also match baselines at all four lengths, certifying 321 codewords at length 192.Source: Table 7 · p. 35, §C.10 · p. 35

C.11 FAILURES AND ANSWER FORMATS

  • A zero score can reflect invalid or malformed output, an empty answer, or budget exhaustion; a malformed length-96 trifference response could be reformatted as a valid 19,683-word code but was scored as submitted.The cited DeepSeek response supplied generator rows as integer arrays rather than the required digit strings.Source: §C.11 · p. 37

C.12 SMALLER TRIFFERENCE INSTANCES

  • On four smaller trifference instances, Astra matches baselines with 729-word codes at lengths 30, 36, and 42, and triples the reference with 6,561 words at length 56.Explicit representations handle these smaller constructions; algebraic certificates extend evaluation beyond what can be listed in a response.Source: §C.12 · p. 37, Table 10 · p. 37

C.13–C.14 SHARED-INSTANCE AND FAMILY-SELECTION COMPARISONS

  • Including Kakeya changes mean relative quality only on construction-baseline instances and leaves published-frontier results unchanged; the Opus 5 comparison covers 45 shared instances.Those 45 comprise 15 published-frontier and 30 construction-baseline instances.Source: §C.14 · p. 38, Table 12 · p. 38, Table 11 · p. 38

C.15 GAP CLOSED AND PROGRESS TOWARD THE LABS TARGET

  • Gap closed measures logarithmic progress from the reference toward a proven bound, while the five LABS instances use a conjectured target and are excluded from the proven-bound mean.Invalid answers and valid answers that do not exceed the reference contribute zero.Source: §C.15 · p. 39, Table 13 · p. 39

D.1–D.2 VERIFIER VALIDATION AND COMPACT REPRESENTATIONS

  • Verifier tests reject deliberately invalid cases and agree with exhaustive checks on small codes and certificate expansions; representation expansion and verification have a 60-second limit.Descriptions generally expand before verification, subject to element and family-specific limits.Source: §D.1 · p. 40, §D.2 · p. 40

D.3–D.4 CODE PRODUCTS AND LINEAR TRIFFERENCE CERTIFICATES

  • Shannon product certificates verify factors to establish separation and compute exact product size; trifference certificates check linear codes algebraically without enumerating all codewords.The linear trifference certificate specifies a ternary generator matrix, and verification checks rank and distance conditions.Source: §D.3 · p. 40, §D.4 · p. 41

D.5 AND E.1 REED–SOLOMON LIMITS AND VERIFIED REFERENCES

  • At length 64, accepted formats cap trifference constructions at 531,441 words, a representation limit rather than a known mathematical optimum; reference constructions attain quality 1 on all 69 instances.Reference objects pass both verifiers within 60 seconds.Source: §D.5 · p. 41, §E.1 · p. 42, Table 15 · p. 42

E.5 REFERENCE SEARCHES AND ALGEBRAIC CONSTRUCTIONS

Tool and web access improve overall model scores, while the paper compares reference searches, specifies instance and evaluation protocols, and reports access differences across tool-assisted runs.

E.5 REFERENCE SEARCHES AND ALGEBRAIC CONSTRUCTIONS

  • Longer search improves some raw objectives, but an expert construction can keep the effective baseline unchanged; for ten-dimensional cap sets, search reaches 1,202 points while the product construction remains 2,240.The reported gains concern the specific search procedures tested.Source: §E.5 · p. 44, Table 19 · p. 44, Figure 13 · p. 45

F INSTANCE PARAMETERS AND RESOURCE LIMITS

  • The 69-instance main evaluation uses parameter ranges selected to leave room between construction baselines and bounds or targets, while Kakeya is treated separately as a control.The A3 screening threshold is a ratio of at least 1.4; answers must fit representation limits and verify within 60 seconds.Source: F INSTANCE PARAMETERS AND RESOURCE… · p. 46, Table 20 · p. 46

G.1 INFERENCE SETTINGS

  • Tool-free runs retain the first completed response or generation-budget stop; invalid answers and budget exhaustion score zero without retries.All configurations receive identical mathematical prompts and format instructions, and no follow-up prompts extend completed answers.Source: §G.1 · p. 49

H CODE AND WEB ACCESS

  • Code and web access improve all three models’ overall scores; Opus 5.5 has the highest overall score, while Astra comes closest to published frontiers.The tool-assisted comparison includes family- and instance-level analyses.Source: H CODE AND WEB ACCESS · p. 51

H.1 PROTOCOL

  • The tool-assisted protocol scores a saved construction only if it is written within the time limit and passes verification; benchmark materials were online during Opus evaluation but not Astra or Luna evaluation.An audit of Opus interactions found no evidence of direct access to those materials.Source: §H.1 · p. 51

H.2 CONSTRUCTION QUALITY

Across the 69-instance evaluations, Astra and Luna matched many published frontiers without exceeding any, while Opus averaged 2.428 relative quality on construction baselines. The section also reports token-use accounting and per-instance scores.

H.2 CONSTRUCTION QUALITY

  • 0.985 mean relative quality on published frontiers and 2.428 on construction baselines; Opus matched 23 published references and exceeded 33 construction baselines.Opus returned 69 valid constructions, improving 62 and tying 7 against its high-effort tool-free answers.Source: §H.2 · p. 51
  • 0.998 for Astra and 0.961 for Luna on published frontiers, matching 29 and 23 references without exceeding any; on 39 construction baselines, they average 1.765 and 1.372.Source: §H.2 · p. 51, Table 23 · p. 52
  • 531,441 codewords at length 64 reached the bench-v1.0 format maximum, not a known mathematical optimum; Opus remained first when any single family was excluded.The instance and ten Heilbronn instances accounted for more than its net score advantage over Astra, while the other 58 instances slightly favoured Astra.Source: §H.2 · p. 52

H.3 TOKEN USAGE

  • Opus’s deadline-stopped token-use total is a lower bound because aggregate usage was unavailable and retained streamed usage was incomplete.Token totals include reasoning tokens and, for Claude Code, native web-search model usage; they exclude external CPU computation and search-service work.Source: §H.3 · p. 52, Table 24 · p. 52

H.4 INDIVIDUAL INSTANCES

  • The release includes every submitted object and its verification result.Source: §H.4 · p. 53
  • Table 25 reports per-instance relative quality for Astra, Luna, and Opus; P denotes published frontier, C construction baseline, and dagger marks protocol-zero evaluations.Source: Table 25 · p. 53

Related papers

Loading 2609.24555v2…