PaperlayerView exact revision on arXiv

Source-linked AI summary

SHELDON: Reasoning through Dual Search over Hypothesis and Observation Spaces

Harshit Joshi, Priyank Shethia, Monica S. Lam

arXiv:2610.18025v1cs.AIcs.LG

TL;DR

Existing reasoning approaches often collect evidence in response to current hypotheses, leaving phenomena outside their scope unexamined. SHELDON jointly explores hypothesis and observation spaces, and outperforms evaluated baselines across diagnosis, abstract reasoning, and optimization tasks by up to 26.7 points.

  • Problem

    Existing approaches guide observation collection by current hypotheses, which can leave phenomena outside their scope unexamined.

    Source: §1 · p. 2

  • Method

    SHELDON separately represents and jointly explores hypothesis and observation spaces, using hypotheses to guide evidence collection and observations to motivate new explanations.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    Across diagnosis, abstract reasoning, and optimization, SHELDON outperforms evaluated baselines by up to 26.7 points.

    Source: §1 · p. 2, §7 · p. 16

  • Takeaways & Limitations

    Ablations show that explicitly modeling the observation space is critical to SHELDON's performance.

    Source: §1 · p. 2, §7 · p. 16

Abstract

from arXiv · show

Test-time scaling has enabled LLMs to solve increasingly complex reasoning, discovery, and optimization tasks. However, existing approaches often gather observations while seeking evidence for current hypotheses, then generate new hypotheses conditioned on those hypotheses and their associated evidence. This couples observation discovery to existing hypotheses, favoring local refinements over exploration of unknown hypotheses regions. Inspired by the hypothetico-deductive method (Hempel, 1945) and dual-space models of scientific discovery (Klahr & Dunbar, 1988), we introduce SHELDON, a scientific reasoning harness that separately represents and jointly explores hypothesis and observation spaces. By actively seeking new observations along with new hypotheses, SHELDON can discover observations that its current hypotheses do not explain. These unexplained observations expose gaps in its understanding, motivate new investigations, and guide the generation of hypotheses beyond refinements of existing ones. Across diagnosis, abstract reasoning, and optimization benchmarks, SHELDON outperforms evaluated baselines by up to 26.7 points. Additional analysis shows that SHELDON generates up to 4.5x more novel solutions.

1 INTRODUCTION

SHELDON addresses the tendency of existing reasoning approaches to collect evidence around current hypotheses, leaving phenomena outside their scope unexamined. It jointly explores hypothesis and observation spaces and outperforms evaluated baselines across several task types.

1 INTRODUCTION

  • Existing coding, evolutionary, and specialized research-agent approaches can remain anchored to sequential histories, initial populations, or evidence collection guided by current hypotheses.The paper argues that these approaches may leave phenomena beyond existing hypotheses unexamined, even when current hypotheses are compared.Source: §1 · p. 2
  • SHELDON separately represents and jointly explores hypothesis and observation spaces, using unexplained observations to expose gaps and motivate qualitatively different hypotheses.Hypotheses guide evidence collection, while observations help assess competing explanations and identify new research directions.Source: §1 · p. 2
  • Up to 26.7 points, SHELDON outperforms evaluated baselines across diagnosis, abstract reasoning, and optimization tasks.On QNN Circuit Topology, it reaches 1.00, compared with 0.85 for the strongest agent baseline and 0.917 for the hand-written reference topology.Source: §1 · p. 2
  • Compared with GoS, SHELDON increases the proportion of ground-truth causes discovered by 29.8 percentage points and causes investigated by 34.4 percentage points.Ablations removing the observation space reduce performance by 14.0 points on DiagnosisArena and 13.34 points on ORCA-Bench.Source: §1 · p. 2

2 RELATED WORK

Prior evolutionary and research-agent systems structure search around candidate solutions, strategies, or evidence attached to existing hypotheses. SHELDON additionally models observations as a space that can prompt new hypotheses and distinguish among candidates.

Evolutionary Algorithms

  • Evolutionary methods evolve candidate solutions or search strategies, whereas SHELDON also models observations that can motivate new hypotheses and discriminate among candidates.Examples include FunSearch, AlphaEvolve (Novikov et al., 2025), and EvoX (Liu et al., 2026a).Source: §2 · pp. 2–3

Research Agents

  • Research agents such as GoS (Luo et al., 2026), ScientistOne (Meng et al., 2026), and Arbor (Jin et al., 2026) organize evidence around hypotheses or artifacts, while SHELDON couples observation-driven hypothesis generation with cross-hypothesis investigation.GoS focuses investigation on the highest-probability hypothesis; SHELDON performs comparative analysis across hypotheses.Source: §2 · p. 3
  • Unlike several listed evolutionary and research-agent methods, SHELDON can operate without an explicit quantitative scoring function, using observations as an implicit scoring function.The paper points to medical diagnosis and system root-cause analysis benchmarks as examples.Source: §2 · p. 3

3 PROBLEM FORMULATION

The paper formulates investigation as an agent's sequential interaction with case material and tools under a budget. The final solution is selected to maximize an available or estimated quality function.

3 PROBLEM FORMULATION

  • An investigative problem consists of a statement, case material, available tools, and a solution quality function that may be unavailable to the agent.Case material may include logs, seed programs, execution traces, or measurements; tools may include code execution, database access, or simulators.Source: §3 · p. 3
  • An investigation task gives an LLM agent a budget to interact sequentially with case material and tools, taking actions and receiving observations at a cost.The agent uses the true quality function when available and otherwise a generated estimate.Source: §3 · p. 3
  • After T steps, the agent computes a final solution intended to maximize estimated solution quality subject to the interaction-cost budget.The estimated quality function is the true function when available, or a generated estimate otherwise.Source: §3 · p. 3

4 SHELDON: A HARNESS FOR SCIENTIFIC REASONING

SHELDON represents hypotheses and observations as connected search spaces and iterates between exploration, candidate testing, comparison, and refinement. This design supports evidence-guided investigation while opening directions beyond the current hypotheses.

4.1 DUAL-SPACE REPRESENTATION

  • SHELDON tracks changing hypothesis and observation spaces alongside working sets, while Figure 2 depicts broader observation coverage opening additional hypothesis regions.The figure contrasts early restricted exploration with later exploration of multiple regions and distinct strategies.Source: Figure 2 · p. 4, §4.1 · p. 4
  • Hypotheses are candidate explanations, solutions, or ideas organized along axes and regions that expose qualitatively different investigative directions.The observation space holds obtainable evidence and evidence-grounded insights, organized by possible sources, time ranges, entities, behaviors, scales, or tools.Source: §4.1 · p. 4

4.2 DUAL-SPACE REASONING PROCESS

  • Evidence relations indicate whether observations support, oppose, are neutral to, or remain unassessed for a hypothesis; new observations are provisional until encountered again and cannot refute a hypothesis while provisional.A set-aside hypothesis cannot be reconsidered until its cooldown has elapsed.Source: §4.2 · p. 5
  • EXPLORE examines case material, tools, and the current state to construct spaces and propose working hypotheses and observations.It may revisit set-aside hypotheses after their cooldown to diversify candidates and reduce concentration in dominant regions.Source: §4.2 · pp. 5–6
  • The ORCA-Bench example shows EXPLORE combining charge-error timing and an invalid-token observation into a new payment-failure hypothesis, alongside refinements of earlier hypotheses.SHELDON stops when its budget or round limit is reached, or when a round produces no new hypotheses, observations, or investigative directions, then returns the best-supported candidate.Source: Figure 4 · p. 6, §4.2 · p. 6
  • CANDIDATEANALYSIS tests each hypothesis's assumptions, components, and failure conditions, gathering supporting or refuting observations and, when available, executing its quality function.The process inspects artifacts, scores, or failures and updates the evidence relations for each candidate.Source: §4.2 · pp. 5–6
  • COMPARATIVEANALYSIS tests the working hypotheses jointly, seeking observations that distinguish candidates, expose shared failure modes, or reveal missing distinctions.Figure 3 presents comparative analysis as one stage in SHELDON's alternating reasoning process.Source: §4.2 · pp. 5–6, Figure 3 · p. 5
  • EXPLOIT uses observations to refine or repair hypotheses and classifies them as RETAINED or SETASIDE for future exploration.The next round uses updated hypothesis and observation sets and their maps to balance promising regions with underrepresented ones.Source: §4.2 · pp. 5, 7

5 EXPERIMENTS

SHELDON is evaluated on ten reasoning, discovery, and optimization tasks against coding, reasoning, evolutionary, research, and autonomous-agent baselines. It reports the highest scores across four diagnostic and reasoning tasks, with results also reported for abstract reasoning, molecular optimization, and system optimization.

5 EXPERIMENTS

  • The evaluation covers ten tasks with and without explicit scoring functions and compares SHELDON with coding, reasoning, evolutionary, research, and autonomous agents.All methods use GPT-5.6-terra for reasoning; system-optimization methods are evaluated on the same hardware.Source: §5 · pp. 7–8

5.1 PERFORMANCE ON DIVERSE BENCHMARKS

  • SHELDON achieves the highest reported score on all four diagnostic and reasoning tasks, outperforming coding, reasoning, recursive, and meta-evolution baselines.The strongest baseline varies by domain: RLM leads on DiagnosisArena, while EvoX leads on ARC-AGI-2 and QNN Circuit Topology.Source: Table 2 · p. 8, §5.1 · p. 8
  • SHELDON scores 84.00 on DiagnosisArena versus RLM’s 78.00 and Codex’s 76.00, and reaches 66.67 on ORCA-Bench versus the strongest baseline’s 40.00.The ORCA-Bench margin is 26.67 points.Source: §5.1 · p. 8
  • On ARC-AGI-2, SHELDON scores 59.17 versus EvoX’s 47.50; on QNN Circuit Topology, it reaches 1.00 and discovers a ten-gate, two-qubit classifier.The ARC-AGI-2 comparison uses a $2.5 budget per task.Source: §5.1 · p. 8
  • SHELDON achieves the best score on four of six molecular targets and exceeds the DOCKSTRING library-best reference on three.CORAL scores highest on PPARG and PGR, while SHELDON reaches 9.47 and 9.99, respectively.Source: §5.1 · p. 8, Table 3 · p. 9

5.2 SYSTEM OPTIMIZATION TASKS

  • Across five ADRS system-optimization tasks, SHELDON has best or tied-best mean and best-run performance.Its transaction-scheduling and LLM_SQL strategies are task-specific; on Cloudcast, SHELDON’s best and mean transfer costs are 618.07 and 618.09, versus Codex’s 618.10 and 677.10.Source: Table 4 · p. 9, §5.2 · p. 9

6 ANALYSIS

The analyses examine how SHELDON’s components, exploration patterns, and shared observation space contribute to investigation across optimization, diagnosis, and abstract reasoning tasks. They find broad search across candidate and hypothesis spaces, with observations supporting comparison and new directions.

6 ANALYSIS

  • Removing comparative analysis or the observation space consistently degrades performance; continued exploration matters especially on ORCA-Bench and QNN topology.Comparative analysis also improves results by contrasting hypotheses and seeking distinguishing evidence.Source: §6 · pp. 9–10, Table 5 · p. 10
  • Winning ideas emerge at different times: some tasks need exploitation within an early-opened region, while others require continued exploration to find the decisive direction.QNN is exploitation-heavy: its relevant region is available from round one, but its best score comes after several refinement rounds.Source: §6 · pp. 10–11, Figure 6 · p. 11

LLM_SQL

  • SHELDON recovers 73.7%–99.9% of seeded improvement without a seed program, with the largest gaps on QNN topology and LLM_SQL.The reported recovery values are 73.7% for QNN Circuit Topology, 79.3% for LLM_SQL, 95.5% for transaction scheduling, 96.6% for EPLB, 97.6% for Cloudcast, and 99.9% for PRISM.Source: LLM_SQL · p. 11, Figure 7 · p. 11

6.1 BREADTH OF EXPLORATION

  • At k=20, SHELDON’s median hull share exceeds the strongest baseline on five of six tasks by factors of 1.4 to 20.9.Cloudcast is the exception: EvoX covers more, while SHELDON’s score is marginally better, 618.07 vs 618.08.Source: §6.1 · p. 12, Figure 10 · p. 13
  • After 20 programs, SHELDON leads the strongest baseline in cumulative novelty on every task except Cloudcast, by up to 4.5× on QNN.Cumulative novelty sums each evaluated program’s embedding distance to the nearest previously evaluated program.Source: §6.1 · p. 12, Figure 11 · p. 13
  • Across twelve optimization tasks, hypotheses span regions, improvements arise from three to seven regions per task, and the final best hypothesis is new on eight tasks.Improvements can come from new hypotheses, local refinements, or refinements after each round.Source: §6.1 · pp. 12–13, Figure 12 · p. 14
  • At radius 0.35, SHELDON investigates more distinct ideas than GoS on 47 of 48 DiagnosisArena tasks and 26 of 27 ORCA-Bench tasks.Median counts are 5 versus 1.5 on DiagnosisArena and 6 versus 1 on ORCA-Bench; the direction holds at radii 0.25, 0.35, and 0.5.Source: §6.1 · p. 13, Figure 13 · p. 14

6.2 WHAT THE OBSERVATION SPACE CONTRIBUTES

  • Observations can assess multiple hypotheses, refute existing ones, connect distinct regions, and seed later hypotheses, including directions beyond a new hypothesis’s parent lineage.On ARC-AGI-2, 204 of 782 post-round-one hypotheses explicitly cite observations; 158 cite an observation from testing a different hypothesis, and 37 cite one from analyzing a non-parent hypothesis. New observation regions also open during search, and 12 of 16 initially empty QNN regions are later populated.Source: §6.2 · p. 15, Figure 14 · p. 15, Figure 15 · p. 15

7 CONCLUSION

SHELDON jointly searches hypothesis and observation spaces, treating observations as first-class objects. Across diagnosis, abstract reasoning, and optimization tasks, it outperforms strong agentic baselines, and ablations indicate that explicitly modeling observations is critical to performance.

7 CONCLUSION

  • SHELDON jointly searches hypothesis and observation spaces, treating observations as first-class objects.The harness supports systematic exploration, comparison, falsification, and refinement of competing explanations.Source: §7 · pp. 15–16
  • Across diagnosis, abstract reasoning, and optimization tasks, SHELDON consistently outperforms strong agentic baselines.Ablations show that explicitly modeling the observation space is critical to its performance.Source: §7 · p. 16
  • Coupling hypothesis and observation search is a promising direction for building scientific and general-purpose reasoning agents.Source: §7 · p. 16

A ADDITIONAL EXPERIMENT SETUP

The appendix standardizes task access and resource constraints across several methods, then specifies baseline implementations and the seedless evaluation setting.

A ADDITIONAL EXPERIMENT SETUP

  • SHELDON, ReAct, Codex, and GoS receive the same task statement, tools, per-task budget, and GPT-5.6-terra model at high reasoning effort.Optimization methods are scored on the best measured artifact, re-evaluated in test mode; all ablations run for eight rounds.Source: A ADDITIONAL EXPERIMENT SETUP · p. 20
  • ReAct follows a Thought/Action/Observation protocol, while Codex runs in a sandbox and is stopped at the per-task budget.GoS uses the released implementation with changes to engine, output limit, step limits, and ORCA-Bench prompts; it is evaluated only on DiagnosisArena and ORCA-Bench.Source: A ADDITIONAL EXPERIMENT SETUP · pp. 20–21
  • The seedless evaluation removes the initial seed program but retains tool-use and submission scaffolding.Recovery is measured as the fraction of seeded improvement recovered; a value of 1 means the seedless run matches the seeded run.Source: A ADDITIONAL EXPERIMENT SETUP · p. 21

B.1 EVALUATION HARDWARE

System-optimization baselines were evaluated on a specified dual-processor Intel Xeon machine.

B.1 EVALUATION HARDWARE

  • The evaluation hardware uses two Intel Xeon E5-2660 processors running at 2.20 GHz.Source: §B.1 · p. 22
  • The machine provides 16 physical cores and 32 logical CPUs.Source: §B.1 · p. 22
  • The processors are organized across two NUMA nodes, with eight cores and two threads per core per socket.Source: §B.1 · p. 22

B.2 SETUP

The setup applies SHELDON and baselines to reasoning and optimization tasks under task-specific evaluation and budget constraints. The prompts separate planning, exploration, testing, evidence comparison, revision, and final answering.

B.2 SETUP

  • ARC-AGI-2 evaluates 120 public tasks at $2.5 per task, while ADRS and QNN impose a $15 budget and maximum of 100 evaluations for each baseline and SHELDON.Molecular optimization allows up to 1,000 docking evaluations per target under a $40 cap; DiagnosisArena provides $10 per task.Source: §B.2 · pp. 22–23
  • QNN circuit topology is scored by held-out accuracy; SHELDON’s ten-gate circuit scores 1.00, compared with 0.917 for the hand-written reference.The discovered circuit has depth six and seven trainable rotations.Source: §B.2 · p. 22, Figure 16 · p. 23

C PROMPTS

  • The prompts are templates with fixed submission schemas for hypotheses, observations, and other stage outputs.Tool calls interleave with each step until its submit tool is called.Source: C PROMPTS · p. 26

C.1 PLAN

  • A one-time planning call separates the task situation from the deliverable and its requirements without solving the task.Each part of the statement is assigned to either situation or deliverable.Source: §C.1 · p. 26

C.2 ROUND CONVERSATION (SYSTEM)

  • Each round explores and proposes hypotheses, tests them individually, gathers global evidence, and revises them using the carried-forward state.In the first round, the model opens the map from the task alone.Source: §C.2 · p. 26, §C.3 · p. 26

C.3 EXPLORE

  • EXPLORE maps hypothesis and observation regions, then proposes testable hypotheses in underexplored regions, favoring distinct mechanisms over variants.When hypotheses win on different task parts or scores plateau, proposals include a combination or eligible revival.Source: §C.3 · p. 27

C.4 SELECT

  • SELECT chooses at most the available number of pending hypotheses for testing, favoring tests that resolve open evidence and spreading choices across regions.Kept hypotheses carry forward without using a test slot; unselected candidates can be revived later.Source: §C.4 · pp. 27–28

C.7 EXPLOIT (REVISE)

  • EXPLOIT decides whether each tested hypothesis is kept, repaired, refined, or shelved, weighing supporting and refuting observations.The final answer uses the best-supported hypotheses and observations that ruled alternatives out; it must not treat an unrefuted claim as established.Source: §C.7 · p. 29, §C.8 · p. 30

C.5 CANDIDATE ANALYSIS (LOCAL TEST)

Candidate analysis tests each hypothesis through artifact-building and evidence gathering; global comparison checks competing predictions, while the recorded state links hypotheses and observations to regions and evidence relations.

C.5 CANDIDATE ANALYSIS (LOCAL TEST)

  • Candidate analysis builds and evaluates an artifact iteratively, seeks evidence that supports or refutes the claim, and revises it only when evidence contradicts it or a small change predicts improvement.A finding that an approach does not help is an observation refuting the claim, not itself a reason to revise it.Source: §C.5 · p. 28

C.6 COMPARATIVE ANALYSIS (GLOBAL TEST)

  • Global analysis inspects artifacts and evidence for flawed experiments, then seeks observations that distinguish competing predictions and records support, refutation, or neutrality per hypothesis.A hypothesis that loses overall may still win on a particular task part.Source: §C.6 · p. 29

D STATE REPRESENTATION

  • The state records hypothesis and observation spaces, working hypotheses, and evidence relations; observations retain sources and per-hypothesis verdicts, and space maps expose exploration counts.Regions are grouped under axes, and refinements remain linked to their parent hypotheses.Source: D STATE REPRESENTATION · pp. 30–31

E COVERAGE ANALYSIS

The coverage analysis measures how broadly methods explore their solution spaces using multiple metrics and task-specific representations. These measures describe where a method looked, not solution quality, and do not account for cost.

E COVERAGE ANALYSIS

  • Solutions are complete candidates submitted to evaluators; identical texts within a run are deduplicated before embedding so repeats do not count as exploration.Programs, ARC-AGI-2 transform functions, and DOCKSTRING SMILES strings supply the respective solution representations.Source: E COVERAGE ANALYSIS · p. 31
  • Text-task hypotheses are reduced to labels and embedded; a label counts as a new idea when its cosine distance from every earlier label exceeds radius r.The analysis reports r ∈ {0.25, 0.35, 0.5} and omits tasks with no labels.Source: E COVERAGE ANALYSIS · p. 32
  • The two-dimensional plane retains a task-varying fraction of variance, and hulls proxy breadth rather than measure it; neither breadth measure reflects findings or cost.The authors read breadth measures alongside task scores.Source: E COVERAGE ANALYSIS · p. 32
  • 18 of 24 task–metric pairs: SHELDON is wider; Cloudcast is the consistent exception, with its best solutions concentrated in a small part of the space.Figure 18 compares SHELDON’s median with the strongest baseline’s median across four breadth metrics.Source: Figure 18 · p. 33, E COVERAGE ANALYSIS · p. 33
  • The analysis measures breadth with matched-sample hull share at k=20, cosine diversity, cumulative novelty after 20 programs, and spread among the five best programs.The four metrics provide complementary views of exploration; cosine diversity checks the hull measure.Source: E COVERAGE ANALYSIS · p. 33

F COST

SHELDON uses more than single-agent baselines on diagnosis and incident analysis, but its higher accuracy than GoS at a similar budget indicates that cost alone does not account for the comparison.

F COST

  • Cost estimates use GPT-5.6 Terra list prices of $2.50 per million input tokens, $0.25 per million cached input tokens, and $15.00 per million output tokens.Source: F COST · p. 33
  • At a $10 per-task limit, SHELDON spends more than single-agent baselines, while scoring higher than GoS on both benchmarks at a similar amount.The authors attribute the spending to running several investigations and comparing multiple hypotheses before answering.Source: F COST · p. 33
  • Table 6 reports cost and wall-clock time per task for 50 DiagnosisArena cases and 30 ORCA-Bench incidents.The reported costs are computed from token counts using one pricing rule.Source: Table 6 · p. 33, F COST · p. 33

G EXPERIMENT CONFIGURATION

Each run is specified by one configuration file; Cloudcast is shown as a representative ADRS configuration, with other benchmarks varying selected settings while retaining the same method hyperparameters.

G EXPERIMENT CONFIGURATION

  • Cloudcast is shown as representative; other benchmark runs change benchmark options, budget, and round cap while keeping method hyperparameters the same.max_usd is the per-task budget in US dollars; null lifts the cap.Source: G EXPERIMENT CONFIGURATION · p. 34
  • The Cloudcast configuration uses hypodeducto with gpt-5.6-terra at high effort and a $15 per-task budget.Source: G EXPERIMENT CONFIGURATION · p. 34
  • Its parameters specify 8 rounds, 5 initial candidates, and 3 exploration candidates, among other controls.Table 7 explains the method hyperparameters.Source: G EXPERIMENT CONFIGURATION · p. 34, Table 7 · p. 34

Related papers

Loading 2610.18025v1…