PaperlayerView exact revision on arXiv

Source-linked AI summary

Priced Guidance: Can Language Models Generate Future Research Ideas?

Kaiyue Wen, Tengyu Ma, Percy Liang

arXiv:2610.04976v1cs.LG

TL;DR

Assessing research-idea generation is difficult because perceived novelty need not translate into better outcomes, while existing benchmarks provide guidance without quantifying its information content. Priced Guidance measures the cost of guiding a model to recover a target idea and connects that cost to unguided acceptance probability. In evaluation, an ensemble of three generators further reduces median compression cost, while the benchmark’s judge-based, at-least-once measure does not test whether ideas are true or work.

  • Problem

    Evaluating LLM-generated ideas is difficult because perceived novelty does not necessarily translate into better research outcomes, and existing benchmarks provide guidance without quantifying its information content.

    Source: §1 · p. 1

  • Method

    Priced Guidance has a generator ask probabilistic multiple-choice questions, while a guide with access to the target selects answers whose information costs measure the guidance needed for idea recovery.

    Source: §1 · pp. 1–2

  • Results

    A uniform ensemble of Fable 5.1, Opus 5, and Astra further reduces median compression cost; GLM 5.3’s median is 102.4 bits versus 5,712 bits with gzip.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Lower compression cost indicates a stronger idea generator, and the framework links guided cost to a lower bound on unguided judge-acceptance probability.

    Source: §1 · p. 2

    The benchmark relies on LLM paper summaries and judges, and measures whether a model produces a target idea at least once rather than whether hypotheses are true or ideas work.

    Source: §6 · p. 14

Abstract

from arXiv · show

We evaluate language models' capability to generate novel research ideas through the lens of compression. We aim to lower-bound the potentially tiny probability that a language model generates the essence of a future research idea without any hints. Rather than estimate this probability through expensive repeated sampling, our Priced Guidance framework measures the compression cost: how many additional bits of information are needed to guide the model to recover the target idea. We prove that if the model can recover the target idea with at most K bits of guidance in expectation, then it can generate the idea without any guidance with probability at least $2^{-K}$. In our framework, the language model, called the generator, can pose a sequence of multiple-choice questions and specify a probability distribution over possible answers. A guide, which is a language model with access to the target idea, selects answers. If a selected answer has prior probability p, the generator pays $-\log_2 p$ bits. The generator aims to produce an idea that matches the essence of the target idea with minimal cumulative cost. This cumulative cost equals, up to an additive constant, the number of bits of information sent by the guide. Using this methodology, we evaluate five generator LLMs (Opus 5, Fable 5.1, GPT-6 Astra, GPT-5.6 Sol, and GLM 5.3) on the core ideas in 87 recent high-quality deep learning papers and we use LLM as judge to determine whether the generated idea matches the target in terms of the central research object and defining mechanism. Fable 5.1 achieves the lowest median compression cost at 69.9 bits, substantially lower than gzip's median of 5,712 bits for losslessly compressing the summary of the target idea. A uniform ensemble of Fable 5.1, Opus 5, and Astra further reduces the median compression cost to 55.8 bits and improves the generation probability lower bound by 18,000 times.

1 Introduction

Priced Guidance measures idea-generation capability by the information needed to guide a model toward a future paper’s core idea. It connects that cost to unguided generation probability and evaluates the framework on recent research papers.

1 Introduction

  • Priced Guidance measures how many additional bits are needed to guide a model toward the essence of a target idea, drawing on prediction–compression duality.Ideas a model predicts well require less additional information to specify; surprising ideas require more.Source: §1 · p. 1
  • Each bid pairs a question with answer probabilities; a guide with access to the target selects an answer, which costs the generator −log2 p bits.The interaction ends when the generator submits an idea and an LLM judge assesses whether it matches the target’s essence.Source: §1 · pp. 1–2
  • With expected guided cost at most K bits, the unguided generator passes the judge with probability at least 2^-K.For fixed generator randomness and a judge that accepts the idea with probability one, Figure 1 illustrates the 2^-K lower bound.Source: §1 · p. 2, Figure 1 · p. 2
  • A 10-bit reduction in guided-run cost improves the upper bound on expected unguided generations needed for judge acceptance by roughly 1000×.The bound should not be read as requiring much lower compression costs before a model can produce high-quality ideas autonomously.Source: §1 · p. 3
  • The evaluation uses five generators on 87 recent papers, with an LLM judge checking recovery of the research direction and central research object.The benchmark addresses the difficulty that perceived novelty does not necessarily translate into better research outcomes; earlier benchmarks supplied keywords or parent papers without quantifying the guidance.Source: §1 · pp. 1, 3
  • A uniform ensemble of Fable 5.1, Opus 5, and Astra further reduces median compression cost; frontier models outperform GLM 5.3, especially on harder papers.For GLM 5.3, median cost is 102.4 bits, versus 1,147.1 bits for same-model verbatim compression and 5,712 bits with gzip.Source: §1 · p. 3

2 Priced Guidance

Priced Guidance measures how much information a guide must provide for a generator to recover a target idea, and connects that cost to unguided generation. Its protocol prices guide-selected answers and extends to multiple candidates, recovery, and ensembles.

2.1 Priced Guidance Protocol

  • A guided run repeats bids, guide selections, and updates until the generator submits an idea; the judge evaluates whether it is equivalent to the target.The guide selects an offered action using its knowledge of the target, while the unguided run samples from the generator’s own bid distribution; both otherwise follow the same process.Source: §2.1 · p. 4
  • The cost K combines the costs of selected answers with a judge cost for the final idea’s pass rate; unsuccessful or zero-pass submissions have infinite cost.Trajectories can succeed with different question sequences or final wordings, and the theorem bounds unguided success using guided-run costs.Source: §2.1 · p. 4

2.2 Priced Guidance Theorem

  • Theorems connect compression cost to both encoding and unguided success: guided actions take at most K(τ) + O(1) bits, while acceptance probability is lower-bounded using guided-run costs.The framework gives compression cost two interpretations: a code length for guide-selected actions and a bound on the probability that an unguided run produces an accepted idea.Source: §2.2 · pp. 4–5

2.3 Extensions: multiple candidates, recovery, and ensembling

  • Extensions support multiple candidate ideas, retries through recovery at an added cost, and ensembles that select a generator member with fixed probability.For an ensemble, choosing member i adds −log2 wi bits; the experiments use wi = 1/M.Source: §2.3 · p. 5

3 Experiment Instantiation

The experiments instantiate Priced Guidance to recover paper contributions, using LLM judges and a scaffolded generator-guide protocol. A SkillNet walkthrough illustrates how guidance moves from a research field to a proposal that passes the Essence judge.

3 Experiment Instantiation

  • The experiment compresses paper ideas from gold summaries, with budgets of $50 for generators and $15 each for guides and judges; these constraints make reported compression costs upper bounds.Targets are Top-1 Hugging Face Daily Papers, with 40 development papers and 87 test papers after title-based memorization screening.Source: §3 · p. 6

3.1 Idea-Equivalence Judging

  • The Essence judge requires a candidate to recover the target’s central research object and defining mechanism, while the Directional judge checks research direction, contribution type, and a coherent plausible idea.GPT-5.5 serves as both judges, distinct from the generators; the Essence judge uses three evaluations by default, versus one for the Directional judge.Source: §3.1 · p. 6

3.2 Generator and Guide Scaffolds

  • A lightweight heuristic scaffold offers question-sequence strategies, while a fixed taxonomy and contribution categories guide opening bids across targets.GPT-5.6 Sol generated the shared 3,035-node taxonomy and 11 categories; every generator receives them for every target. The guide selects a strategy and then its answer, with choices priced by their probabilities.Source: §3.2 · pp. 6–7, Figure 3 · p. 7

3.3 Example Walkthrough

  • At 64.1 bits, the SkillNet proposal passed all three Essence evaluations after identifying “skill ontology” as the missing organizing structure.The candidate passed the Directional judge at 48.2 bits after describing a shared hub with LLM-as-judge curation; Essence-stage guidance added 15.9 bits.Source: §3.3 · p. 8

3.4 Data Curation and Memorization Audit

  • Targets came from Top-1 papers published March 2–September 6, 2026, with title-based screening to remove papers recalled by generators from their titles alone.The development set spans March 2–April 30, and the 87-paper test set spans May 1–September 6.Source: §3.4 · p. 8

4 Main Results

The three-model ensemble achieves the lowest reported median compression cost, while individual-model results vary with judge strictness and paper difficulty. The results also show that most information is spent acquiring candidate concepts, and that costs are not explained simply by paper length or date.

Compression costs

  • 102.4 bits is GLM 5.3’s median Essence compression cost, versus 1,147.1 bits for verbatim gold-summary compression with the same model and 5,712 bits with gzip.The Directional median for GLM 5.3 is 59.9 bits; both Priced Guidance costs are more than an order of magnitude lower than verbatim compression.Source: §4 · p. 8

Model comparisons

  • Under the Essence judge, Fable 5.1, Opus 5 high, and Astra nearly tie at median costs of 69.9, 70.4, and 71.6 bits; 80% coverage requires 104.3, 141.3, and 111.0 bits, respectively.Source: §4 · p. 8

Cost patterns

  • Figure 4’s successful Fable 5 trajectory for SkillNet adds 15.9 bits after promotion from Directional to Essence, identifying the skill ontology and passing at 64.1 bits.Across the 87 test papers, the paper reports coverage curves and median and 80th-percentile costs under both judges.Source: §4 · p. 8, Figure 4 · p. 9

Where the information is spent

  • Letter hints account for 23.7% of completed Directional keyword selections but 51.8% of keyword costs; on final Essence paths, they account for 20.3% of selections and 45.2% of keyword costs.The authors suggest that much of the information acquires useful candidate concepts outside the models’ initial guesses.Source: §4 · p. 9

Headroom from ensembling

  • 43.2 bits under the Directional judge and 55.8 bits under the Essence judge are the ensemble’s median compression costs, improving on the best individual models by 4.3 and 14.1 bits, respectively.The uniform ensemble averages 2^-K scores for Fable 5.1, Opus 5, and Astra; unsuccessful runs contribute zero.Source: §4 · p. 9

Cost patterns

  • Compression cost shows no significant correlation with gold-summary or full-paper length, and ensemble cost shows no significant correlation with paper date over the test period.Technical reports tend to have high compression cost; the Kimi K3 target is the only test paper for which none of the five generators passes the Directional judge in the test runs.Source: §4 · p. 10

5 Ablations

Development-set ablations test generator, guide, reasoning, candidate-count, and scaffold choices. Costs fall with post-training, retained reasoning, custom compaction, and ensembling, while extra candidates help Directional but not Essence costs.

5 Ablations

  • Updating GLM 5.2 to GLM 5.3 reduces median compression cost from 100.6 to 53.4 bits on the development benchmark.The models share the same pretraining base model.Source: §5 · p. 10
  • Replacing the Fable 5 guide with Opus 5 changes Opus 5’s median compression cost only from 45.0 to 44.9 bits.Source: §5 · p. 11
  • Opus max effort raises median cost from 45.0 to 54.6 bits as coverage rises from 39/40 to 40/40; Fable’s cost shifts from 48.9 to 46.6 bits.The Fable effort comparison is indicative because its high- and max-effort runs use different guides and prompts.Source: §5 · p. 11, Figure 7 · p. 12
  • Ensembling two Opus runs lowers Directional median cost from 45.0 to 41.6 bits and Essence median cost from 72.4 to 70.2 bits.Directional coverage rises from 39/40 to 40/40 papers.Source: §5 · pp. 11–12
  • Across independent Opus runs, Directional medians are 45.0 and 46.4 bits, while individual-paper costs differ and are only moderately correlated.Essence medians are 72.4 and 77.5 bits; the passage attributes ensemble gains to randomness across independent runs.Source: §5 · p. 12
  • Eight candidate ideas lower Astra’s Directional median cost from 63.7 to 53.1 bits, while Essence costs stay close at 75.3 versus 74.9 bits and both cover 39/40 papers.The eight-candidate configuration lowers Directional cost on 32 of 40 papers.Source: §5 · p. 12
  • Custom natural-language compaction lowers Astra’s Essence median cost from 74.9 to 68.1 bits versus native Codex compaction, while Directional costs remain nearly unchanged at 53.1 versus 51.7 bits.Essence coverage rises from 39/40 to 40/40 papers.Source: §5 · p. 13
  • Retaining reasoning history lowers median Directional cost from 69.9 to 63.7 bits and raises coverage from 37/40 to 40/40 papers.The comparison is with an earlier Astra scaffold that discards messages and reasoning between calls.Source: §5 · p. 13

6 Limitations and Discussion

Priced Guidance provides a compression benchmark for research idea generation, with compression cost jointly shaped by the generator and guide. The discussion notes limits in the scaffold and benchmark, while framing the metric as a measure of generation capability rather than a threshold for autonomous generation.

6 Limitations and Discussion

  • Compression cost measures generation capability, but does not establish a threshold below which autonomous generation becomes possible.The authors note that high-quality outputs can occur even when compressing a particular target requires thousands of bits.Source: §6 · pp. 13–14
  • Lower compression cost is a joint generator-guide problem: stronger generators favor target-reaching trajectories, while stronger guides find cheaper ones.The work presents this as a compression benchmark for research idea generation.Source: §6 · pp. 13–14
  • The heuristic generator scaffold is likely far from optimal, lacks arbitrary tool calls, and could improve through ensembling or post-training.The authors report that inference-time ensembling and generator post-training already lower compression cost.Source: §6 · p. 14
  • The benchmark relies on LLM summaries and judges, so generators could potentially exploit the judge; the authors observed vague phrasing but no more serious judge hacking.The benchmark assesses whether a model can generate a target idea at least once, not whether hypotheses are true or ideas work.Source: §6 · p. 14

7 Related Work

Prior work studies LLM research-idea generation, judges idea quality, forecasts later research with partial information, and automates research tasks. This paper positions priced guidance as a way to quantify the information needed to recover a target idea and as a complement to research-execution evaluation.

Related work on idea generation

  • LLM systems propose scientific hypotheses and directions through scaffolding, while SciMON and ResearchAgent add literature retrieval or academic graphs and review agents [Wang et al., 2023, Baek et al., 2024].Zhao et al. [2026] is identified as closest because it uses target-paper guidance to create training trajectories for idea generators.Source: §7 · pp. 14–15

Idea-quality evaluation

  • LLM-judged proposal benchmarks report higher novelty ratings than human proposals but also limitations in diversity and self-evaluation, an ideation–execution gap, and judge disagreement with experts [Si et al., 2024, Si et al., 2025, Sinhahajari et al., 2026, Zheng et al., 2023, Wataoka et al., 2024, Baig et al., 2026, Si et al., 2026].These findings motivate evaluating idea generation without treating novelty judgments as research outcomes.Source: §7 · p. 15

Future-research forecasting

  • Future-research benchmarks provide partial information such as keywords or parent papers, whereas priced guidance quantifies the bits needed to guide a model toward a target idea’s essence [Krenn et al., 2022, Li et al., 2026a, Jiang, 2026, Chen et al., 2026].The paper frames this as quantifying the value of partial information such as keywords or cited works [Yang and Bing, 2026].Source: §7 · p. 15

Automated scientific research

  • Idea-generation evaluation is presented as orthogonal to automatic science execution [Lu et al., 2024, Chen et al., 2024], which includes systems and benchmarks for experiments, discovery, replication, and research engineering.Examples include the AI Scientist, ScienceAgentBench, DiscoveryBench, PaperBench, and RE-Bench.Source: §7 · p. 15

Information theory and rare-event estimation

  • Priced guidance relates to semantic uncertainty, online algorithms with advice, and rare-event estimation, with selected-answer cost defined as −log2 p.The protocol’s expected guided score 2−K lower-bounds unguided success.Source: §7 · p. 15

A.1 Proof of Theorem 2.1

The proof encodes guided answer sequences as a prefix code once the generator’s target-independent random tape is fixed. The bid probabilities determine code lengths, allowing the guide’s answers to be reconstructed without a separate stopping signal.

A.1 Proof of Theorem 2.1

  • Fixing the generator’s target-independent random tape makes each guided trajectory a path through a deterministic tree.The proof represents the generator’s randomness in initialization and transitions as independent draws on a random tape.Source: §A.1 · p. 21
  • A complete answer sequence cannot be a proper prefix of another because the generator’s state determines whether it requests another answer.Guide decisions that change this state must be communicated through an offered answer, so no separate stopping signal is needed.Source: §A.1 · p. 21
  • The selected answer sequence has interval length 2^-C(τ), enabling a binary prefix code of length at most ⌈C(τ)⌉ + 1.The code uses the generator’s bid probabilities and works across targets and guides for the fixed tape.Source: §A.1 · p. 21

A.2 Proof of Theorem 2.2

The proof relates guided answer costs to unguided acceptance by replaying the same generator randomness and averaging over trajectories. In the next-token special case, the compression bound is exact, while the measured event depends on the fixed-length setup.

A.2 Proof of Theorem 2.2

  • Assuming executions terminate almost surely, unguided execution with the same random tape selects a guided trajectory’s actions with probability 2^-C(τ) and reproduces its idea.The proof averages over guided trajectories and generator and guide randomness, with the fresh judge’s pass rate included in K.Source: §A.2 · p. 21

A.3 Next-token prediction as a special case

  • For next-token prediction, the unguided probability of the target string equals the expected guided score and PPLθ(y | u)^−n.The guide selects each target token, so its answer costs sum to the string’s negative log-likelihood.Source: §A.3 · p. 22
  • The theorem’s event is fixed-length string generation; including an end-of-sequence token adds its probability and corresponding compression cost.With a deterministic judge accepting any idea-equivalent output, the exact-string probability remains a lower bound on acceptance probability.Source: §A.3 · p. 22

A.4 ELBO interpretation

The ELBO interpretation expresses the log acceptance probability as an evidence lower bound from the guide’s expected log-likelihood cost. The bound is exact when the guided trajectory law matches the unguided law conditioned on acceptance.

A.4 ELBO interpretation

  • The derivation uses the KL divergence between guided trajectories and unguided trajectories conditioned on acceptance to relate guide cost to acceptance probability.The finite-divergence assumption permits rearrangement; KL nonnegativity yields the lower bound.Source: §A.4 · p. 23
  • −Eτ∼Pguide[Lguide(τ)] is an evidence lower bound on log acceptance probability, exact if and only if the guided and acceptance-conditioned unguided trajectory laws agree.The identity assumes almost-sure termination, almost-sure guided success, and finite KL divergence.Source: §A.4 · p. 23
  • For a deterministic guide whose runs succeed almost surely, guide log-likelihood cost equals compression cost, reducing the ELBO to −Eτ∼Pguide[K(τ)].A guide restricted to offered answers may not realize the acceptance-conditioned law required for equality.Source: §A.4 · p. 23

A.5 Proofs of extensions

The extension proofs reduce modified generator constructions to the priced-guidance theorem by averaging out target-independent auxiliary randomness. They apply this reduction to candidate selection, recovery, ensembles, and promotion while specifying the relevant costs and assumptions.

A.5 Proofs of extensions

  • Target-independent auxiliary randomness turns each extension into a meta-generator; averaging it out recovers the corresponding experiment’s acceptance probability.The common reduction applies the priced-guidance theorem conditional on the auxiliary randomness, then averages and uses Jensen’s inequality.Source: §A.5 · pp. 23–24

A.5.1 Multiple candidate ideas

  • For multiple candidate ideas, the terminal evaluator samples a weighted candidate and applies a fresh judge, preserving the original experiment’s success probability.The candidate list and weights are determined by the trajectory.Source: §A.5.1 · p. 24

A.5.2 Recovery

  • Indexed recovery charges only the returned path, but occurrence indices count all requests; its bound applies under saved-state, independence, and initialization assumptions.The left side is the acceptance probability of one fresh unguided run; missing ideas or zero pass rates have infinite cost.Source: §A.5.2 · pp. 24–25

A.5.3 Ensembling models and seeds

  • Ensembling fixed, target-independent member weights bounds the mixture’s success probability using member costs, with failed or missing members assigned infinite cost.The observed effective-cost denominator remains the full ensemble size, and a finite-sample probability guarantee requires accounting for statistical uncertainty.Source: §A.5.3 · pp. 25–26

A.5.4 Promotion cost

  • Promotion adds the costs of inserted stay, switch, or Submit choices to the recovered Essence path’s answer, occurrence, and judge cost.Inserted stay or switch choices cost the negative base-2 log of their bid probability; an inserted Submit also includes its occurrence index when applicable.Source: §A.5.4 · p. 26

(a) Reuse the promoting submission

(b) Resume at the promoting question

(c) Resume at another question

The promotion bound charges the recovered Essence path plus the choices needed to replay its stage switch. It bounds the replay scaffold’s success probability; transferring it to the original judge-triggered two-stage scaffold requires a further comparison.

(c) Resume at another question

  • When resuming at another question, replay inserts a switch immediately before the first retained Essence answer, preserving the question and its bid probabilities.The switch costs −log2 of its fixed bid probability; each Directional question also receives an inserted stay.Source: Figure 10 · p. 27
  • The replay scaffold’s success probability is lower-bounded using the recovered path cost Kπ plus promotion cost cprom.The bound is over guided recovery runs and applies under the recovery assumptions.Source: (c) Resume at another question · p. 27
  • The result bounds the replay scaffold, not directly the original judge-triggered two-stage scaffold; transferring it requires comparing their success probabilities.Source: (c) Resume at another question · p. 27
  • Across 354 successful test runs, promotion cost had a 2.1-bit median and 6.9-bit maximum; the fixed extra Submit cost 1.90 bits.The switch/stay probabilities were 0.01 and 0.99, respectively, and all reported Essence costs include promotion cost.Source: (c) Resume at another question · p. 27

B Additional Experimental Details

This appendix details the memorization audit and compression baselines, along with the research-idea recovery protocol and its priced guidance process. It specifies how route choices, submissions, corrections, draft updates, judging criteria, and protocol safeguards govern recovery of a target idea’s defining mechanism.

B.1 Memorization audit

  • 80 is the memorization threshold; excluding papers where any generator scores at least 80 leaves 40 development papers and 87 test papers with no overlap.In the title-only recall audit of 849 selections, mean recall fell from 87.9 in November to 52.6 in December, motivating the operational threshold.Source: §B.1 · p. 28

B.2 Compression cost over time

  • Both ensemble compression-cost correlations with listing date are close to zero across the approximately four-month, 87-paper test set.Figure 12 plots Directional and Essence costs for the uniform mixture of Fable 5.1, Opus 5, and Astra.Source: Figure 12 · p. 29, §B.2 · p. 29

B.3 Verbatim GLM 5.3 baseline

  • 1,147.1 bits is GLM 5.3’s median negative log-likelihood for gold summaries scored without target-specific context.The score covers summary tokens only, using low reasoning effort and an empty reasoning trace; high and max effort yield medians of 1,153.5 and 1,194.5 bits.Source: §B.3 · p. 29

B.4 Lossless gzip baseline

  • 5,712 bits (714 bytes) is the median gzip length for the 87 gold summaries, versus 10,008 bits for uncompressed text.All summaries decompress exactly; this baseline changes both compressor and reconstruction criterion, so it does not isolate recovering an idea’s essence rather than its wording.Source: §B.4 · p. 29

C.1 Generator scaffold details

  • The generator scaffold uses taxonomy and contribution-category bids, maintains a candidate idea and shared whiteboard, and promotes successful Directional runs to an Essence stage.Directional submissions contain eight candidates, while a newly authored Essence submission contains only the current candidate; Essence adds a correction strategy to multiple-choice and keyword strategies.Source: §C.1 · pp. 30–31

C.2 Guide scaffold details

  • The guide selects target-supported answers from priced options, using private memory and, when useful, private Judge previews to decide whether to continue or submit.Only selected answers incur compression cost; on recovery, the guide can revisit an earlier question while retaining private memory, and kept rollouts incur an additional cost.Source: §C.2 · p. 31

C.3 Generator-visible state; C.4 Shared generator preamble

  • The generator-visible state carries confirmed facts and a current draft, while the preamble defines the target as the contribution category, core problem, and main object.The whiteboard tracks belief and open uncertainties, and the top idea must integrate confirmed facts into one coherent mechanism; the guide’s reasoning and detailed Judge feedback remain hidden.Source: C Prompts and Agent Protocol · p. 30, §C.3 · p. 32, §C.4 · pp. 32–33

C.5 Recognition-stage goals and route menus; C.6 Route selection and state update

  • The Directional stage accepts the right research direction with a plausible mechanism, whereas Essence requires the recognizable main-object essence and its defining mechanism at least coarsely.Current stage menus use multiple-choice, keyword, and submit routes in Directional, adding correction in Essence; dispatch replays cached, generator-authored downstream questions after route purchase.Source: §C.5 · pp. 33–34, §C.6 · p. 34

C.6 Route selection and state update

  • The route list uses the active stage’s menu, and each cached question is bound to its source stage, draft, fact-ledger hash, and question hash.The state-update prompt asks the generator to integrate reliable priced answers without dropping confirmed facts and to treat rejected alternatives as negative evidence.Source: §C.6 · p. 34

C.7 Directional eight-candidate submission prompt

  • Before each Directional dispatch with a nonempty draft, the generator creates exactly eight distinct, complete candidate ideas with positive weights, which are normalized and cached before the guide selects a route.Candidates should express substantively different plausible interpretations when evidence leaves uncertainty, and must not invent names, exact numbers, or evaluation results.Source: §C.7 · p. 35
  • A passing Directional slate incurs an additional cost of -log2 of the total normalized probability mass of passing candidates; all eight cached candidates are judged separately and sent unchanged.An unselected slate changes neither the active draft nor confirmed facts.Source: §C.7 · p. 35

C.8 Atomic multiple-choice prompt

  • Multiple-choice prompts ask one atomic question about a high-value unresolved axis, with exactly eight concrete values; selecting a value adds only the deterministic public fact “axis: value.”The options should be mutually exclusive at one semantic level and span plausible values rather than synonyms or cosmetic variants.Source: §C.8 · pp. 35–36

C.9 Keyword prompts

  • The keyword route first prices a 16-way semantic-category slate, then a 64-phrase slate conditioned on the selected category, with optional 16-way completion slates after purchased prefix reveals.Source: §C.9 · p. 36

C.10 Correction prompts

  • Semantic correction first selects a proposition already entailed by the draft, then proposes replace, refine, or delete patches; it cannot insert an independent missing claim.Replace/refine patches must commit a complete after-claim and resulting draft with only one localized contiguous edit; deletion adds no proposition.Source: §C.10 · pp. 38–39

C.11 Guide prompt and per-turn input

  • The guide knows the private target but communicates only by choosing offered options, and must select wholly true propositions while seeking a passing submission at low information cost.The guide is instructed to treat option probabilities as costs, not truth scores, and to minimize total information cost without sacrificing truthfulness.Source: §C.11 · pp. 39–40
  • After each answer, the target-blind Generator updates its whiteboard and draft from paid facts, while private judge previews advise the guide without affecting scores.A real submission reveals the exact private judge gap; rejected submission costs are largely refunded at checkout, so testing a submission is usually cheaper than speculative correction.Source: §C.11 · pp. 42–43

C.12 Judge prompts

  • The Directional coarse screen accepts peer-quality ideas in the same research direction even with a different mechanism; fine judging also requires the gold’s specific problem and distinctive advance.Fine Directional acceptance requires both direction alignment and equal sensibility; merely restating the prior method the gold improves upon does not qualify.Source: §C.12 · p. 44
  • Essence judging requires the same setting and defining mechanism, while allowing omissions or approximations of secondary details such as datasets, configurations, and exact formulas.A candidate that substitutes or contradicts the core mechanism fails even if it pursues the same goal or broad area.Source: §C.12 · p. 45

C.13 Protocol-level safeguards

  • Arena safeguards validate probability distributions, log events, isolate generator and guide processes, restrict private target access, and prevent unchanged failed submissions from being retried.Checkout rewinds generator state and active path cost but not guide memory or global repetition counters; failed submissions require authorized checkout, and unchanged Judge-visible content blocks resubmission.Source: §C.13 · p. 45

C.14 Scaffold variants and prompt differences in the ablations

Section 5 ablations vary scaffold settings, context management, and reasoning history across configurations. Several comparisons bundle multiple changes and therefore measure complete configurations rather than isolated components.

C.14 Scaffold variants and prompt differences in the ablations

  • Across the ablations, median costs are empirical 50%-coverage costs over 40 development papers, with unsuccessful runs assigned infinite cost.Paired statistics use only papers where both configurations pass the relevant judge.Source: §C.14 · p. 45
  • Every Section 5 configuration retains explicit working state, while scaffold settings distinguish reasoning history across generator calls.The earlier scaffold retains confirmed facts, the whiteboard, the current idea, and bounded histories, but not earlier messages or private reasoning.Source: §C.14 · pp. 45–46, Table 3 · p. 46
  • The eight-candidate scaffold adds a Directional expansion call that produces eight weighted ideas normalized into probabilities; the single-idea scaffold submits its current draft with probability one.In the eight-candidate version, the current top idea becomes the starting hypothesis for expansion.Source: §C.14 · p. 46
  • The custom context-management arm adds a summary call and replaces the longer fixed Codex instruction, so results compare complete context-management configurations.Its initial summary allowance is 12,000 characters with compaction triggered at an estimated 100,000 tokens; allowances later increase for some continuation runs.Source: §C.14 · p. 47
  • The history-retaining Astra configuration uses native Codex context management and one Directional candidate, unlike the API baseline, which does not retain earlier conversation messages or reasoning.The comparison also changes effort settings, instructions, and generation budgets; the API baseline was selectively continued, while three unresolved cases are operational outcomes rather than scientific judgments.Source: §C.14 · pp. 45, 47
  • Fable high- and max-effort configurations use different guides and guide wording, while their generator prompts remain identical.The high-effort guide uses revised justification and state-summary instructions; the max-effort configuration uses the earlier wording.Source: §C.14 · pp. 47–48

D Per-Paper Results

This section reports per-paper Directional and Essence compression costs across the 87-paper test set, including individual costs in bits and entries with infinite costs.

D Per-Paper Results

  • Tables 4 and 5 report Directional and Essence compression costs in bits for the 87 test papers, with Essence costs including the Directional stage and promotion cost.Papers are listed in the same order in both tables, sorted by their Hugging Face Daily Papers listing date.Source: D Per-Paper Results · p. 49, Table 4 · p. 49, Table 5 · p. 54
  • An ∞ entry denotes that no validated idea passing the relevant Directional or Essence judge was obtained within the experiment budget.Table 4 applies this convention to Directional costs and Table 5 to Essence costs.Source: Table 4 · p. 49, D Per-Paper Results · p. 49, Table 5 · p. 54
  • Agentic Abstention and Orca each have an infinite cost among their five reported entries.Their listed cost sequences are 56.8, ∞, 46.9, 45.2, 49.1 bits and 48.7, ∞, 34.2, 83.8, 36.3 bits, respectively.Source: D Per-Paper Results · p. 56
  • Program-as-Weights and Video Generation Models each have an infinite cost among their five reported entries.Their listed sequences are 57.8, 72.4, 45.9, 70.2, ∞ bits and 89.1, 85.3, 58.2, 68.5, ∞ bits, respectively.Source: D Per-Paper Results · p. 56
  • AREX has infinite costs in three of its five reported entries.Its listed sequence is ∞, 105.3, ∞, 162.7, ∞ bits.Source: D Per-Paper Results · p. 56

E SkillNet Case Study: generator Proposals and Gold

The SkillNet proposals describe a shared, agent-independent skill hub with LLM curation, and the essence proposal adds an ontology for organizing skills. The motivation section presents SkillNet as an infrastructure for creating, evaluating, relating, and sharing skills.

E SkillNet Case Study: generator Proposals and Gold

  • SkillNet is proposed as a shared skill library with LLM-judge curation, enabling cross-agent skill reuse without model weight updates.The library is decoupled from any specific agent and presented as analogous to a package registry or model hub for skills.Source: E SkillNet Case Study: generator… · pp. 58–59
  • The proposed hub accepts skills distilled from trajectories, vets and maintains them with an LLM judge, organizes them in the ontology, and retrieves relevant skills for new tasks.Retrieval can use ontology-guided navigation or embedding-based similarity to condition planning and actions.Source: E SkillNet Case Study: generator… · p. 59
  • The essence proposal adds a skill ontology: a hierarchical taxonomy linking skills through specialization, dependencies, and composition.The proposal organizes how skills are stored, related, abstracted, and navigated.Source: E SkillNet Case Study: generator… · pp. 59–60

Problem & motivation

  • Agents lack a unified way to turn episodic experience into reusable skills, while existing repositories rely on manual curation and indirect quality signals.The described result is fragmented, redundant, and unvalidated skills.Source: Problem & motivation · p. 60
  • SkillNet is presented as an open full-lifecycle infrastructure that creates skills from varied sources, evaluates them on five intrinsic dimensions, and connects them in a typed relation graph.The release includes 200k+ candidate and 150k+ curated skills, plus a website/API and Python library with CLI.Source: Problem & motivation · p. 60
Loading 2610.04976v1…