PaperlayerView exact revision on arXiv

Source-linked AI summary

ULTRADISCOVERY: Abductive Exploration in an Interconnected, Epistemically Open Universe

Weihan Li, Tianshi Zheng, Yangqiu Song, Ginny Y. Wong, Simon See

arXiv:2610.03092v1cs.LGcs.AI

TL;DR

Open-world discovery requires constructing a representation and composing evidence across contexts, demands that existing benchmarks rarely separate. ULTRADISCOVERY varies both factors in one five-domain world. Its results show that stronger investigation does not reliably produce a transferable explanation.

  • Problem

    Open-world discovery requires agents to identify observations that share a regularity, combine them, and construct a representation for expressing it, while existing benchmarks often leave that representation predetermined.

    Source: §1 · pp. 1–2

  • Method

    ULTRADISCOVERY varies whether the representation is open or disclosed and whether evidence is distributed or aligned within one latent world.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    Disclosure shifts agents toward intervention more than evidence-grounded discovery, and accumulated evidence often fails to become a transferable account.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The results locate the difficulty in composing evidence into a representation that transfers across domains.

    Source: §6 · p. 8

    Evidence comes from one authored world and language-model agents; worlds built by others and human runs are needed to assess how far findings extend.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Scientific discovery often begins when scattered clues call for a new way of describing the world. Such abductive exploration can require constructing the representation in which an explanation is stated, when the world is epistemically open, and composing evidence scattered across contexts, when it is structurally interconnected. Existing benchmarks rarely separate these two demands or control them independently. We introduce ULTRADISCOVERY, an interactive world of five domains in which an agent revises an initially successful theory and predicts the outcome of an unseen cross-domain intervention. A $2 \times 2$ design leaves the representation open or discloses it, and leaves the evidence distributed or aligns it, with the latent dynamics fixed. With the representation open, agents across eleven models often retract the axiom they were taught, and none introduces the unobserved entity or rewrites the variables that a replacement requires. Disclosure triples intervention requests and adds about one of the eighteen findings the world affords, and alignment adds less. Two vendor-harness systems carry discovery into more domains, and one of them rewrites the variables in Open episodes. No system makes the exact prediction within 200 paid actions. At larger budgets one exact prediction appears with both aids, while every Open episode remains inexact. The results locate the difficulty in the step from accumulating evidence to composing it into a representation that transfers.

1 Introduction

ULTRADISCOVERY treats discovery as both constructing an explanatory representation and composing evidence across domains, then varies those burdens independently within one latent world.

1 Introduction

  • Open-world discovery requires agents to determine which observations share a regularity, combine relevant information, and construct a representation in which that regularity can be expressed.These steps precede much of the search that current systems already perform well.Source: §1 · p. 1
  • Many discovery benchmarks leave agents searching within a largely human-designed hypothesis space, so representation construction remains weakly measured.When variables and explanatory forms are supplied, the cost of constructing a representation disappears from the measured problem.Source: §1 · p. 2
  • ULTRADISCOVERY separates representation construction from composing evidence across contexts in a five-domain world where local anomalies reveal a shared mechanism only when related.Its 2×2 design varies whether the representation is open or disclosed and whether evidence is distributed or aligned, holding the latent world fixed.Source: §1 · p. 2
  • Disclosure triples intervention requests and adds about one of eighteen findings; no system predicts exactly within 200 paid actions, while one succeeds at 600 with both aids.The supplied representation also changes how agents investigate more than how much evidence-grounded discovery they achieve.Source: §1 · p. 3
  • 46.1% of a model’s episodes on average retract the taught axiom, but baseline agents never postulate the unobserved entity or rewrite the variables.Vendor-harness systems reach further domains, and one rewrites variables in Open episodes.Source: §1 · p. 3

2 Related work

Prior work studies interactive scientific discovery, representation formation, and agent scaffolds; ULTRADISCOVERY independently controls representation openness and evidence organization in one world.

2.1 Interactive scientific discovery benchmarks

  • Many interactive discovery benchmarks specify entities, variables, or interfaces in advance; others entangle representation formation with exploration or leave cross-context evidence organization uncontrolled.ULTRADISCOVERY varies representational openness and evidence organization independently within the same latent world.Source: §2.1 · p. 3

2.2 Representation formation and abduction

  • ULTRADISCOVERY operationalizes representation formation as an experimental cost by comparing open and disclosed representations.Prior computational discovery systems usually search within a fixed representational substrate.Source: §2.2 · p. 3

2.3 Scaffolds for discovery agents

  • ULTRADISCOVERY compares self-critique, falsification, and evolutionary scaffolds in the two Open conditions while holding the world and interaction protocol fixed.The study treats scaffolding as an experimental factor because scaffold effectiveness can depend on how candidate spaces are constructed and evaluated.Source: §2.3 · p. 3

3 The ULTRADISCOVERY universe

ULTRADISCOVERY is a five-domain, repeatable world where agents must redescribe a taught theory, compose cross-domain evidence, and predict an unseen intervention.

3.1 A world that must be redescribed

  • The world spans five domains and repeats every 24 ticks; a free prologue teaches a doctrine that is exact on its own cases but cannot explain the stall.Interventions and observations use raw physical acts and instrument readings, without scores or hints.Source: §3.1 · p. 3, Figure 3 · p. 4

3.2 Episodes and the terminal prediction

  • After 200 paid actions, agents commit three integer readings for an intervention at two sites that has never been attempted.The standing order arrives with ten actions remaining, making the prediction a transfer test of the agent’s cross-domain account.Source: §3.2 · p. 4

3.3 Epistemic structure

  • Eighteen epistemic blocks comprise 15 domain-local clues and three cross-domain bridges, with hyperedges linking evidence to representation changes and terminal-prediction readings.The three representation changes concern retracting an axiom, postulating an unrendered entity, and changing the variables; HC gathers five sources for the terminal readings.Source: §3.3 · p. 4

3.4 Separating the two burdens

  • Four variants hold the latent world fixed while varying representation disclosure and evidence alignment; Open × Distributed places both burdens on the agent.Disclosure supplies a qualitative representation document, while alignment provides tick-aligned cross-domain transcripts and intervention records.Source: §3.4 · pp. 4–5

3.5 Validation

  • At the contract budget, no episode assembles more than 3 of five HC sources, and a model given the complete reference record still makes no exact prediction.The Open episodes collectively contain evidence for all 18 blocks, separating access to readings from constructing an account that composes them.Source: §3.5 · p. 5

4 Experimental setup

The experiments compare paired model performances across four variants, examine inquiry scaffolds, and score discovery, representation change, evidence composition, and transfer.

4.1 Models and paired evaluation

  • Eleven models from six families play four variants on twelve paired instances; two vendor-harness systems run four instances per variant alongside the baseline comparison.The vendor systems differ from the baseline in model and harness together.Source: §4.1 · p. 5

4.2 Organising inquiry

  • The baseline exposes the event digest, workspace, recalled records, and current view, while self-critique, falsification, and evolution scaffold account revision, discriminating tests, and candidate search.Figure 5 compares four baseline conditions with pooled vendor-harness systems, plotting grounded blocks and evidence held separately.Source: §4.2 · pp. 5–6, Figure 5 · p. 6

4.3 Metrics

  • Discovery AUC averages grounded blocks over 200 actions, distinguishing early documented discoveries from the same count reached later; evidence counts acquired blocks whether stated or not.A block is grounded only when its evidence is produced and its inference is stated in the workspace.Source: §4.3 · p. 6
  • RC scores representation changes, XC grounded cross-domain bridges, and Reach held sources in the largest hyperedge; exact prediction requires all three committed readings to be correct.Interventions counts intervention requests.Source: §4.3 · p. 6

5 Results

Agents often reject the taught doctrine without constructing its replacement, and neither disclosure nor scaffolding closes the gap between evidence and transferable explanation. Vendor-harness systems and larger budgets extend discovery, but exact prediction remains elusive, especially in Open episodes.

5.1 Rejecting the doctrine does not yield its replacement

  • At 200 paid actions, no episode makes the exact prediction, and even complete evidence rarely yields a stated representation change.At larger budgets, one exact prediction appears in Disclosed × Aligned, while every extended Open episode remains inexact.Source: §5.1 · pp. 6–7, §5.4 · p. 8
  • In Open episodes, agents retract the taught axiom in 46.1% of a model’s episodes on average, yet none postulates the unobserved entity or rewrites variables.Only 6 main-comparison episodes state a representation change with complete evidence; none with complete H2 or H3 evidence states the entity or variables.Source: §5.1 · p. 7

5.3 Scaffolding increases testing more readily than discovery

  • Across five models in Open conditions, scaffold gains on Blocks stay below half a block: +0.23 for self-critique, +0.26 for falsification and +0.18 for evolution.Falsification reduces episodes without an intervention request from 18 of 39 to 5 of 39, but only 3 of 118 scaffolded episodes hold complete hyperedge evidence and 1 states its change.Source: Figure 6 · p. 7, §5.3 · p. 8

5.2 A supplied representation changes inquiry more than discovery

  • Disclosure raises intervention requests from 5.0 to 14.8 per episode with distributed evidence and adds +1.0 grounded block on average (∆repr = +1.0, [0.8, 1.2]).Alignment raises evidence held by +0.6 blocks and Blocks by +0.5 ([0.3, 0.7]), while intervention requests change by +0.3 ([−0.7, 1.2]); estimated gains are approximately additive.Source: §5.2 · p. 7

5.4 Vendor-harness systems extend discovery

  • Vendor-harness systems ground 7.6 (Claude Fable 5.1) and 7.1 (GPT-6 Astra) blocks per episode, versus 3.5 for the eleven models on the same instances.They ground fen blocks in 13 of 16 episodes each, but consign blocks in only 2; Claude Fable 5.1 rewrites variables in 5 of 8 Open episodes, while the unobserved entity remains unstated in Open episodes.Source: §5.4 · p. 8
  • On one instance, the vendor systems used 600 paid actions and GPT-5.6 Sol used 2000; 7 of 12 extended episodes held complete hyperedge evidence, and Claude Fable 5.1 made one exact Disclosed × Aligned prediction.Every extended Open episode remained inexact; the exact-prediction episode held evidence for 4 of 5 designed-route blocks and stated 2 of 3 changes.Source: §5.4 · p. 8

6 Discussion and conclusion

ULTRADISCOVERY separates constructing an explanatory representation from composing evidence across domains. Its results distinguish progress in inquiry and evidence from building an account that transfers.

6 Discussion and conclusion

  • A transferable explanation requires more than accumulating evidence: observations from several domains must be held together and expressed through a representation beyond the taught theory’s terms.The terminal prediction tests transfer by asking for an account of an unseen intervention, whether reached by the designed route or the agent’s own route.Source: §6 · p. 8
  • A supplied representation organizes inquiry by naming mechanism types to test, while the evidence needed to establish one still has to be assembled.Disclosure therefore shifts intervention more than discovery, and disclosed agents state changes whose sources they seldom hold.Source: §6 · p. 8
  • Falsification scaffolding obtains more tests, but grounded discovery barely moves; vendor-harness systems and larger budgets carry inquiry further while Open episodes still leave the unobserved entity unstated.The remaining step is to hold evidence from several domains together and build an entity or variables beyond the taught theory’s vocabulary.Source: §6 · p. 8

Limitations

The paper’s findings are measured in one authored world with language-model agents, under operational definitions of openness and workspace-based scoring. The appendix describes the world, clocks, interface, taught doctrine, local anomalies and shared mechanism that delimit this setting.

Limitations

  • The study measures language-model agents in one authored world; other worlds and human-volunteer runs are needed to assess generalization and calibrate difficulty for people.Disclosure grants structural knowledge together with vocabulary, alignment changes evidence access and experimental budget, and unwritten inferences are outside the metrics’ view.Source: Limitations · p. 9

A.1 The setting

  • The five-domain world has fifteen sites, 36 objects and 11 instruments, with some instruments reading objects on other hulks and five sites lacking instruments.The mission asks agents to explain a stall across the formation and predict an unseen intervention, while leaving the representation to the agent.Source: §A.1 · p. 12, Figure 7 · p. 13

A.2 The pass and the two clocks

  • Each pass lasts 24 ticks and resets objects and instruments, while notes and event history persist; paid actions and world ticks are separate clocks.Each action costs one paid action, but waiting can span up to 24 ticks, whereas moving, reading or intervening takes one tick.Source: §A.2 · p. 12, Table 2 · p. 12

A.3 The interface

  • The interface gives unmarked readings without scores or progress signals, and the agent alone authors bounded free-text notes that determine which variables, entities and laws it has stated.The agent can inspect past events through recall, while a refused or deferred act gives no reason.Source: §A.3 · pp. 12–13

A.4 The doctrine and the prologue

  • The prologue teaches a doctrine that predicts its own cases exactly, making adoption a reasonable start and local repair a natural response to later anomalies.A five-item check measures prologue reading; almost all counted episodes answered all five items.Source: §A.4 · p. 13

A.5 Where the doctrine stops

  • Each domain’s anomaly admits a local repair, but the patches share three-domain hyperedges whose regularity requires holding evidence together and changing the doctrine’s representation.Blocks counts findings, Reach records how much evidence for a change an episode holds, and RC records whether notes state the change.Source: §A.5 · p. 14

A.6 The mechanism

  • The shared mechanism assigns hidden four-coordinate labels to individuals, limits each to one act per tick, and uses scheduled rewrites that can merge or split bodies, producing the recurring stall.Instruments show body-specific transforms of individual states, while the audit counter reports the number of individuals; the doctrine is exact on singleton objects but fails where bodies share an individual.Source: §A.6 · pp. 14–15

A.7 The epistemic structure

ULTRADISCOVERY encodes discovery as evidence blocks linked by cross-domain hyperedges, with three links licensing frame changes and a fourth routing to a transfer prediction. Its variants hold latent dynamics fixed while changing representation disclosure and evidence alignment.

A.7 The epistemic structure

  • Eighteen blocks feed four hyperedges: three license representation changes, while the closing edge links five blocks to the designed route for a terminal prediction.Each change requires composing evidence from three domains; the source blocks alone admit accounts short of the change they jointly explain.Source: §A.7 · pp. 15–16

A.8 The representation changes and their hyperedges, worked

  • Across H1–H3, three-domain evidence supports retracting the axiom, postulating a hidden multi-coordinate entity, or rewriting laws over label coordinates and rotation position.Local repairs fit individual sources, but the shared regularity in H1 is one act per tick across distinct substrates.Source: §A.8 · pp. 16–17

A.9 The standing order

  • The closing hyperedge’s five blocks compose a route to three readings after an unseen simultaneous stamp and retool, testing forecasts across three domains.The order is sealed before the caretakers act, and the commitment is one attempt.Source: §A.8 · p. 17, §A.9 · p. 17

A.10 Four variants of one world

  • Four variants share the world: Open leaves variables, entities, and laws to the agent; Disclosed supplies target-frame text; Aligned relocates evidence without changing its content.Distributed records remain at their source instruments, while Aligned transcripts post readings from the previous pass.Source: §A.10 · p. 18

A.11 Explanatory frames and openness

  • A frame pairs world structure with expressible laws; openness withholds the target frame, while disclosure supplies only part of it, and structural interconnection concerns evidence rather than physical interactions.The frame fixes entities, variables, admissible relations, and constitutive constraints; disclosed materials leave instance laws and the standing order to be found.Source: §A.11 · pp. 18–19

B Measurement and scoring

  • Scoring combines deterministic event-log evidence detectors with workspace statements, and both records accumulate across paid actions.A block’s detector records produced evidence; the workspace judge checks whether the notes state the inference it licenses.Source: B Measurement and scoring · p. 19

B.1 The metrics

The metrics distinguish evidence acquisition, stated discovery, cross-domain reach, and exact transfer. Validation checks the detectors, judge, and evidence routes; even with a full record, the inference probe does not produce an exact tuple.

B.1 The metrics

  • Blocks counts evidence plus stated inference, Evidence counts produced evidence even when unstated, and XC counts grounded bridge inferences; discovery AUC dates when evidence and statement are both recorded.For four blocks under disclosure, grounding also enters the AUC timestamp.Source: §B.1 · pp. 19–20
  • Reach measures the most sources held for one representation-change hyperedge; RC counts stated changes, interventions count requests, and exact prediction scores 1 only when all three readings match.Reach ranges from 0 to 3, with 3 indicating complete evidence for a change.Source: §B.1 · p. 20

B.2 Grounding under disclosure

  • Under disclosure, four blocks require grounding as well as evidence and a statement, but this requirement changes no main-comparison episode count and leaves Δrepr on Blocks at +1.0 ([0.8, 1.2]).The grounding leg of RC1 is not included in reported counts.Source: §B.2 · pp. 20–21

B.3 Behavioural detectors

  • Detectors are deterministic event-log rules; timestamps distinguish world time from acquisition order, transcripts preserve observations but do not perform acts, and LD1 requires live readings.CA3 and XC1 pair a logged reading or act with an expectation stated in the notes.Source: §B.3 · p. 21

B.4 The workspace judge

  • A language-model judge checks statements and grounding in notes while allowing equivalent Open vocabularies; its vendor-harness change labels matched reference labels in 91 of 96 pairs (κ = 0.88).The judge uses separate statement and grounding verdicts, validated against literal spans of the notes.Source: §B.4 · pp. 21, 23

C.1 Reachability, shortcuts and evidence structure

  • Validation separates legal-budget reachability from shortcut channels and checks evidence structure; shortcut probes report exact-tuple frequency for one model and channel over tested instances.Reachability is assessed against the evidence needed for the target and its cost in paid actions.Source: §C.1 · p. 23

C.2 Checks and outcomes

  • Reserved validation instances and score checks test whether document restatement, transcript relabeling, or judge agreement could affect reported outcomes; model probes use instances disjoint from evaluation.The checks report that bare document restatement earns no grounded hits on the five stated criteria and relabeling posted rows changes no verdict.Source: §C.2 · p. 24

C.3 Reachability

  • 0/30 exact tuples per variant followed a full-record inference probe, despite a 39-action reference route and Open episodes holding evidence for all 18 blocks.The probe received the route’s 166-event record; the Aligned transcripts add one paid action for S1, compared with at least twelve live actions over two passes.Source: §C.3 · p. 24

C.4 Shortcuts

The validation checks probe potential shortcuts and confirm that the variants differ only through declared surfaces. The implementation holds the paid-action loop fixed while adding inquiry scaffolds.

C.4 Shortcuts

  • 0 of 120 prior-knowledge commitments were exact, and none of 60 vocabulary-only replies stated any of the seven mechanism laws.With the prologue withheld, the comprehension check averaged 1.95 of 5 over 20 samples; no sample reached its required four answers.Source: §C.4 · pp. 24–25
  • 114 of 116 shortcut checks passed, and every tested strategy that committed a tuple was exact on at most 15% of instances.The two non-passing checks registered vocabulary from the declared Disclosed document; the reserved instances realised 106 distinct answers.Source: §C.4 · p. 25

C.5 Evidence structure

  • All four variants use the same latent world and yield the same answer, while a reference route costs 39 paid actions in each.Scripted falsifiers can discard every designed local-repair decoy within budget.Source: §C.5 · p. 26

C.6 A validation pattern for discovery worlds

  • The proposed validation pattern combines a single-source comparison, a priced reference route, channel probes, shortcut strategies, answer-space checks, and a fixed scorer.A full-record probe distinguishes acquiring readings from constructing an account of them.Source: §C.6 · p. 26

D.1 The baseline loop; D.2 Self-critique

  • The baseline gives agents an unstructured workspace and full access to observed records; self-critique adds a revisable four-sentence account, puzzles, and contradicted beliefs.The agent may use, edit, or ignore the scaffolded summary.Source: §D.1 · p. 26, §D.2 · p. 26

D.3 Falsification

  • Falsification screens concrete, discriminating tests and retires an account after two refutations, while leaving action choice to the agent.A contradicted prediction is the sole basis for a refutation mark.Source: §D.3 · pp. 26–27, Figure 12 · p. 27

D.4 Evolution

  • Evolution seeds and mutates candidate mechanisms, retains the four best by retrodiction against the digest, and asks the agent to choose actions that distinguish leading candidates.The episode’s final commitment is the external verdict; candidate fitness is scored against readings already observed.Source: §D.4 · p. 27

D.5 Parity and separation; D.6 Budgets; D.7 Excluded episodes; D.8 Vendor-harness systems

  • Scaffolds add unpaid calls but preserve paid-action parity; episodes allow 200 paid actions, exclude specified filter or malformed-reply failures, and report vendor-harness systems separately.The vendor systems use Claude Code and Codex, so model and harness change together.Source: §D.5 · pp. 27–28, §D.6 · p. 28, §D.7 · p. 28

D.9 Larger budgets

Larger budgets extended evidence gathering for the vendor-harness systems, but exact prediction remained rare: only one extended episode was exact, and every Open episode remained inexact.

D.9 Larger budgets

  • One extended episode made an exact prediction: Claude Fable 5.1 in Disclosed × Aligned; every other extended episode, including all Open episodes, was inexact.The exact episode held evidence for four of the five designed-route blocks and stated two of the three representation changes.Source: §D.9 · p. 29
  • Fable held evidence for 11.2 blocks per episode against 8.8 in its twins, whereas Astra held 9.0 against 9.5 and grounded 7.0 against 7.8.Sol held 8.2 blocks against 6.2 in its twins, then spent 92.3% of its paid actions waiting after its last evidence by action 168.Source: §D.9 · pp. 29–30
  • Fable and Astra continued producing evidence beyond 200 actions, until actions 316 and 460, respectively, and each requested about twice its twin’s interventions.Their intervention requests averaged 112.0 and 77.5 per episode, against 50.2 and 42.5 for their twins, respectively.Source: §D.9 · pp. 29–30, Figure 13 · p. 30

E.1 Discovery over the hypergraph

Episodes often held evidence across the hypergraph but missed complete source sets, with consign evidence a recurring gap. When a hyperedge was complete, whether its representation change was stated depended on condition and change.

E.1 Discovery over the hypergraph

  • Open episodes collectively held evidence for all 18 blocks, but no episode held more than three of the five HC sources; consign evidence was the most frequent missing source for H2 and H3.Consign evidence appeared in 7.8% of main-comparison episodes, and consign blocks were grounded in 1.0%.Source: §E.1 · p. 31, Figure 14 · p. 31, Figure 15 · p. 32
  • Under disclosure, every complete hyperedge came with its stated change; in Open episodes, complete H1 sources accompanied retraction, while most complete H2 or H3 cases left the entity or variable change unstated.The exception was a Claude Fable 5.1 episode that stated the change of variables.Source: §E.1 · p. 31
  • The baseline loop grounded 88.5% of its blocks before the standing order, while 69.8% of vendor-harness grounded blocks were grounded before it.The widest evidence-to-grounding gap was at TR1, whose passive reading was commonly held but seldom stated.Source: §E.1 · pp. 31–32
  • Only 11 of 512 main-comparison episodes held a complete hyperedge, and six stated its supported change; vendor-harness systems held complete hyperedges in two and four episodes, stating changes in two and three.The table counts episodes by system, condition, and hyperedge; an episode may contribute to multiple hyperedges.Source: §E.1 · p. 31, Table 13 · p. 32

E.2 The exact prediction

At the larger budget, Claude Fable 5.1 made the sole exact prediction in Disclosed × Aligned; the contract-budget trajectories show how exploration can yield either a revised account or an untested explanation that fails transfer.

E.2 The exact prediction

  • Claude Fable 5.1 made the exact prediction at 600 paid actions in Disclosed × Aligned, holding four of the five designed-route blocks and stating two of three representation changes.Its notes derived the three readings from the rewriting law, schedule, and other mechanism details; the order’s pass read 16, 1, and 20.Source: §E.2 · pp. 32–33, Figure 17 · p. 33

E.2 The exact prediction; E.3 Four trajectories, from observation to transfer

  • Field (a) was derived without FE3’s matched-conservation evidence, showing that the designed route to the readings was sufficient but not the only route.At the contract budget no episode made the exact prediction, so the paper examines four full trajectories in two same-instance pairs.Source: §E.2 · p. 33, §E.3 · p. 33

E.4 An account revised, then overextended

  • Sol withdrew a causal interpretation after its feed test matched an unmanipulated pass, then treated cull contrasts as evidence for a broader registry-line account.After yielding its remaining inquiry budget, it committed (0, 8, 16) against the true (35, 3, 18).Source: §E.4 · p. 34

E.5 An explanation without an intervention

  • Flash made no intervention, turned a recorded fall in the audit counter into an explanation of the stall, and left that explanation untested before committing (9, 1, 16) against (35, 3, 18).Its proposed threshold and causal role were not tested.Source: §E.5 · p. 34

E.6 A working account, rewritten

Fable turns pass-by-pass observations into a working account and tests it across repeated experiments, but its final transfer prediction misses one reading.

E.6 A working account, rewritten

  • Its pass notes held readings, actions, and effects while marking a working model as untested.Source: §E.6 · p. 34
  • Fable rewrote its pass record into numbered observations, then repeated a stamp to confirm a result and gathered all three H3 sources in the consign domain.Source: §E.6 · p. 35
  • Fable committed (16, 1, 21) against the true (16, 1, 20), with its first two readings exact and its listed risk including the true third value.Source: §E.6 · p. 35

E.7 A budget yielded after testing

The trajectories expose different routes to an inexact transfer prediction, while the agent-facing materials show the mission and interface context in which they unfold.

E.7 A budget yielded after testing

  • Astra distinguishes an issued maintenance request from proof of effect, then uses a neighboring-feeder experiment to carry a result to the named feeder.Source: §E.7 · p. 35
  • After yielding 71 actions and running a final peer experiment, Astra committed (14, 10, 20) against (16, 1, 20), getting only the third reading right.Source: §E.7 · p. 35

E.8 What the four trajectories separate

  • Sol and Flash share an inexact endpoint but diverge in evidence: Flash holds 3 blocks and grounds 0, while Sol holds 9, grounds 6, and reaches 2.Source: §E.8 · pp. 35–36
  • Vendor-harness shortfalls echo both routes despite more evidence in hand: Fable grounds 9 of 10 blocks and reaches 3, while Astra grounds 7 of 8 and reaches 2.Source: §E.8 · p. 36

F Agent-facing materials

F.1 Opening scene and mission

  • The mission is to explain a recurring drift-wide stall and predict three readings for an unseen, two-site standing-order intervention before it occurs.Source: §F.1 · p. 37
  • Site descriptions appear on entry, and Aligned variants append a posted transcript to the site description.Source: §F.1 · p. 37

F.2 Doctrine and prologue

This section specifies the standing-and-share doctrine, its five-beat confirmation, and the disclosed mechanism, prediction checkpoints, interaction rules, and scaffold prompts, alongside agent scaffolding and workspace judging. It also uses Outer Wilds to illustrate discovery through interlocking clues and player-built representations, including knowledge preserved across loops and the distinction between epistemic difficulty and embodied flight.

F.2 Doctrine and prologue

  • Five exact predictions across three sites confirm the standing-and-share doctrine through feeding, rest, culling, retooling, and stamping.The prologue also identifies waiting as one of the tested actions.Source: §F.2 · p. 38

F.3 Disclosed document

  • The disclosed handoff describes hidden four-coordinate labels, shared identity, queued acts, coordinate rewrites, and object-specific monotone readouts, while leaving key dynamics unspecified.It states that well-formed laws concern label coordinates, queue position, and act counts rather than per-object rates, shares, or standings.Source: §F.3 · p. 38

F.4 Comprehension check

  • The free pre-flight check tests five predictions, drawing numeric answers from the prologue and applying the retooling rule to an untouched printer.It must be answered before the first action; it contains no answer key.Source: §F.4 · pp. 38–39

F.5 Commit window and standing order

  • At ten paid actions remaining, the agent must spend one action to commit to three readings under a sealed standing order, adjudicated exactly.The order joins a stamped beacon with consignment-q, retools feeder-3, and asks for feeder intake six ticks later, beacon response timing, and the audit counter seven ticks later.Source: §F.5 · p. 39

F.6 Refusals and deferrals

  • Non-taking acts receive fixed refusal or deferral messages, with one refusal per intervention verb, one deferral per domain, and a separate sealed-object refusal.The passages give examples for feed, release, stamp, retool, cull, the Fen deferral, and sealed objects.Source: §F.6 · p. 39

F.7 Interface contract

  • The interface requires JSON replies, charges each valid action its stated cost, and provides persistent notes and filtered recall; only disclosed variants add the document to the fixed opening.The prompt’s current view includes available actions, blocked actions, budgets, the commit window, loop, and view.Source: §F.7 · pp. 39–40

F.8 Scaffold prompts

  • Scaffold calls use notes and recent log excerpts to support review, rival hypotheses, test screening, refutation checks, and new candidate proposals without choosing actions.The scaffold is instructed not to invent unsupported readings or events; silence alone does not refute an account.Source: §F.8 · pp. 40–41
  • The scaffold presents a maintained working summary, test slate, and candidate set, with unpaid advice about tests while leaving the action choice to the agent.Candidate scoring and test-screen advice do not spend a paid action.Source: §F.8 · pp. 41–42

F.9 Brief for the vendor-harness systems

  • The vendor-harness brief requires agents to work through play.py, maintain their inquiry in the workspace, avoid restarting, and stop when the episode ends.The brief prohibits reading files or directories outside the episode directory and using the network or web.Source: §F.9 · p. 42

F.10 Workspace judge prompt

  • The workspace judge checks whether notes state each criterion and separately whether the claim is grounded, requiring literal quotes; the 21 criteria are printed in Table 14.The prompt instructs the judge to assess only what the notes say, not whether a claim is true.Source: §F.10 · p. 42, Table 14 · p. 43

G.1 The game

  • Outer Wilds resets its planets and objects every twenty-two-minute loop but preserves discoveries in a ship’s log that links information by place and as a web of rumours.The player explores a small solar system with tools including a signalscope, camera probe, and translator.Source: §G.1 · p. 45

G.2 Clues that interlock across planets

  • Three rules taught on separate planets combine into a procedure for reaching the Quantum Moon, while the ending requires carrying knowledge gathered across planets to reach the Eye.Other cross-planet clues explain anglerfish behavior and features of Giant’s Deep.Source: §G.2 · p. 45

G.3 A representation the player builds

  • Players build conclusions beyond a log that records only certain facts; the Ash Twin tower puzzle is solved by revising alignment from the receiver to the body’s center.The twin planets share a center, so the tower fires when Ember Twin passes overhead.Source: §G.3 · pp. 45–46
  • Signals from seeds and the camera probe let the player build a route through Dark Bramble to the Vessel.The signalscope maps the topology; firing the scout through one seed makes its signal reappear at another, leading to the Vessel.Source: §G.3 · p. 46

G.4 Progress stored in the player

  • Knowledge persists across loops: a player who has read the Eye’s coordinates can use them to finish the final, one-life run.Removing the warp core ends the loop; finishing in one loop is possible in principle but extremely unlikely without advance knowledge (Beachum, 2021).Source: §G.4 · p. 46

G.5 What players find hard

  • Players find both discovery and disorientation challenging, while ULTRADISCOVERY sets aside embodied flight difficulty to retain epistemic difficulty (Caldwell, 2019; Myers, 2019; Beachum, 2013).Getting lost appears among both a review’s likes and dislikes; the benchmark makes any site one goto away (Section 3.1).Source: §G.5 · pp. 46–47
  • Players and the game’s writer guard discoveries against spoilers, distinguishing a received answer from one the player constructs (Stark, 2019; Myers, 2019; Beachum, 2021).Source: §G.5 · p. 47

G.6 What the game suggests for benchmark design

ULTRADISCOVERY turns the two demands illustrated by Outer Wilds—building a representation and composing scattered evidence—into measurable properties that can be separated. Unlike the game’s ending, its standing order asks agents to give the whole account in one committed prediction.

G.6 What the game suggests for benchmark design

  • The benchmark makes epistemic openness and structural interconnectedness measurable through hyperedges spanning three domains and instruments that report in their own words.It also includes a pass and reset, plus a notebook that survives every reset.Source: §G.6 · p. 47
  • Its factorial design varies representation construction and evidence composition one at a time, while the standing order demands the whole account at once.The game’s ending also asks players for the whole account.Source: §G.6 · p. 47
  • The epigraph’s tenets, “to seek out” and “to understand,” name the ending’s demand on players and ULTRADISCOVERY’s demand for one committed prediction.Source: §G.6 · p. 47
Loading 2610.03092v1…