PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLMs Anticipate Their Cruxes?

David Reber, James Reber, Ari Holtzman, Victor Veitch

arXiv:2610.21006v1cs.CLcs.LG

TL;DR

Long model rationales make it difficult to identify the critical questions behind decisions. The paper compares a priori, single-rollout, and many-rollout crux extraction, finding that the cheaper approaches are inadequate and multiple rollouts are needed for high-stakes decisions.

  • Problem

    Because model rationales can be lengthy, it is difficult to isolate the critical subquestions behind a recommendation, and it is unclear whether a priori or single-rollout methods identify them adequately.

    Source: §1 · p. 2

  • Method

    The paper extracts cruxes by contrasting many rollouts and clustering paraphrases, using a reranker estimated to require 86 times less compute than a local judge.

    Source: §1 · p. 2

  • Results

    A priori crux predictions do worse than a random shuffle, and single-rollout extraction typically misses many cruxes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For high-stakes decisions, the paper concludes that multiple-rollout extraction is necessary.

    Source: §1 · p. 2

    Whether a posteriori cruxes help people make better forecasts remains untested and would require a human study.

    Source: §7 · p. 10

Abstract

from arXiv · show

To use language models to help us make decisions we would like to understand the rationale behind the language model's outputs. In principle, the language model provides its rationale as part of its text output. However, the sheer volume of text makes this hard to use in practice. In this paper, we are interested in extracting the cruxes that lead a model to recommend one action over another. We take a crux to be a sub-question that a model's rollouts disagree on, where changing its value may have altered the conclusion. A crux is therefore a property of the distribution of a model's rollouts, not of any one rollout. We consider three strategies of increasing computational cost. First, we just ask a model to identify, a priori, what the cruxes of a decision will be. Second, we have the model actually go through the process of generating a response and then extract the cruxes that arose in its reasoning. Finally, we generate many responses and extract the cruxes by comparing the reasoning across the rollouts. We make this last procedure computationally tractable with cheap calls to a reranker, an estimated 86 times less compute than a judge. We then conduct an extensive empirical evaluation to assess whether the computationally inexpensive options are sufficient. They are not. Across domains and models, we find that a model's a priori assessment of the cruxes has little relation to the factors that are actually identified as important a posteriori. Using a single rollout helps, but is still inadequate for identifying the full set of cruxes. Accordingly, for high-stakes decisions (and to understand the space of language model decision making) the many rollout procedure is unavoidable.

1 INTRODUCTION

The paper asks whether simple a priori or single-rollout methods can identify the cruxes behind a model’s decision, and introduces comparison across multiple rollouts as an alternative. The simpler methods prove inadequate: a priori predictions do worse than a random shuffle, while one rollout typically misses cruxes.

1 INTRODUCTION

  • The paper targets the problem of reducing a model’s long rationale to critical subquestions that expose assumptions and reasoning shaping its conclusion.For example, whether inputs skipped by a coding agent matter could be an unstated crux in deciding whether to accept its change.Source: §1 · p. 2
  • A priori cruxes name questions to investigate before deciding, while single-rollout cruxes are extracted after the model generates one decision response.The authors ask whether either inexpensive approach is adequate.Source: §1 · p. 2
  • The proposed method extracts cruxes by contrasting many rollouts, on the premise that crux questions should show variation across independent reasoning.A question that receives the same belief across rollouts is not treated as a crux, unlike one on which the rollouts vary.Source: §1 · p. 2
  • The evaluation clusters paraphrased cruxes and compares overlap against a permutation baseline across forecasting sources, model families, and semantic-similarity definitions.A reranker matches crux pairs at an estimated 86 times less compute than a local judge while agreeing on most pairs and rarely merging items it judged distinct.Source: §1 · p. 2
  • A priori crux predictions do worse than a random shuffle, and a single rollout typically misses many cruxes, making multiple rollouts necessary for high-stakes decisions.The comparison controls for the same cruxes by randomly reassigning their labels between the a priori and multiple-rollout sets.Source: §1 · p. 2

2 RELATED WORK

The paper distinguishes its question from related work on model introspection, claim extraction, and forecasting systems: it tests whether predicted cruxes match the factual questions on which a model’s own forecasts disagree. The illustrated examples show much less overlap than a permutation baseline.

2 RELATED WORK

  • Prior introspection work studies what models know, their tendencies, or whether their reasoning is faithful, but does not ask whether they can predict which factual questions their forecasts will disagree on.The passage cites Kadavath et al. (2022), Lin et al. (2022), Kapoor et al. (2024), and others on introspection, and Wei et al. (2022), Turpin et al. (2023), Chen et al. (2025), and Chua & Evans (2025) on reasoning faithfulness.Source: §2 · pp. 2–3
  • In the basketball and bankruptcy examples, predicted cruxes match rollout-disagreement questions less often than the permutation baseline.The baseline randomly reassigns the same cruxes between the a priori and multiple-rollout sets.Source: Figure 2 · p. 3
  • Claim extraction breaks text into claims, while self-consistency uses variation among outputs to improve an answer; this paper instead compares two forecasts and retains claims on which they differ.The passage cites Bar-Haim et al. (2020), Min et al. (2023), and Wei et al. (2024) on claim extraction, and Wang et al. (2023) and Du et al. (2024) on self-consistency.Source: §2 · p. 3
  • Forecasting systems seek evidence to improve forecasts, whereas this paper takes forecasts as given and asks how to find their cruxes.The passage cites Halawi et al. (2024) and Murphy (2026) on forecasting systems.Source: §2 · p. 3
  • CruxBench scores named cruxes by how much their resolution moves a prediction market, while this paper tests whether named cruxes match disagreements among the model’s own forecasts.The passage reports a strongest CruxBench model score of 0.016 bits, narrowly above a random-resolution-time baseline of 0.007.Source: §2 · p. 3

3 DATA SOURCES

The study evaluates crux extraction on binary-event forecasts from Metaculus and March Madness, two settings with different sources of rollout variation. Metaculus rollouts vary through both sampling and independent web searches; March Madness holds evidence fixed to focus on model variation.

3 DATA SOURCES

  • The study uses binary-event forecasting, where a model assigns a probability to one of two mutually exclusive outcomes, and forecasts are produced through research, factor selection, and evidence weighting.A forecasting agent searches the web, picks the most important factors, then weighs evidence on each factor to write an event probability.Source: §3 · p. 3

3.1 METACULUS

  • The 15 Metaculus events span geopolitics, economics, science, and technology, with 29 or 30 live rollouts per event using separate web searches.The rollouts vary through model sampling and the search results they receive; the forecaster is gpt-5-nano, while gpt-5.4 predicts cruxes in advance and gpt-5.4-mini extracts them.Source: §3.1 · p. 4

3.2 MARCH MADNESS

  • The March Madness study covers 59 games, using scouting reports gathered before the tournament so every rollout reasons from fixed evidence.Three reports per team use different analyst personas and are reused across matchups; the main runs use gpt-5.4-nano and three open-weight models, with nine rollouts per game.Source: §3.2 · p. 4

4 CRUX EXTRACTION METHODS

The paper compares a priori, single-rollout, and multiple-rollout crux extraction for a fixed forecaster. Multiple-rollout extraction contrasts pairs of responses to find factual questions where they take different sides, then clusters paraphrases; many pairs yield substantially fewer distinct cruxes because cruxes recur.

4 CRUX EXTRACTION METHODS

  • The three methods are a priori prediction, extraction from one rollout, and extraction by comparing many rollouts; the experiment treats the forecaster as part of the data.A different model may act as predictor or extractor, and Figure 3 illustrates the a priori and multiple-rollout prompts for one game.Source: §4 · p. 4, Figure 3 · p. 5
  • The a priori method predicts which cruxes rollouts will disagree on without seeing any rollout, while the single-rollout method extracts cruxes from one randomly selected response.The a priori input differs by source: Metaculus receives only the question text, while March Madness receives all three scouting reports; single-rollout extraction yields about 12 cruxes per game and 6 per event.Source: §4 · p. 5
  • Multiple-rollout extraction contrasts every pair of responses to find factual questions on which the two take different sides, restricting cruxes to factors relevant to the model’s sampling distribution.Each crux is stored as a yes/no question with both possible answers, one for each side a forecast could take.Source: §4 · pp. 5–6
  • Cruxes recur across rollout pairs, so distinct cruxes grow about linearly rather than quadratically: March Madness has about 3.4 per rollout, from 4 at two rollouts to 28 at nine, versus 36 pairs.Five of nine rollouts yield 61% of the distinct cruxes from all nine; seven yield 83%, and subsets of Metaculus rollouts find most cruxes from all twelve, sometimes more.Source: §4 · p. 6, Figure 4 · p. 6

5 EVALUATION METHODOLOGY

The evaluation clusters semantically similar cruxes and compares the resulting sets against a permutation baseline. It uses a reranker to make pairwise matching more affordable and checks whether the observed gap persists across matching choices and set sizes.

5 EVALUATION METHODOLOGY

  • Clustering pooled cruxes by semantic similarity makes set overlap measurable despite paraphrases and differing numbers of extracted cruxes.Average-linkage clustering merges groups by mean pairwise distance; persistence selects a cut, without using which set a crux came from.Source: §5 · p. 6, §5.1 · p. 7

5.1 EQUIVALENCE BETWEEN CRUXES

  • The reranker replaces a generative judge for pairwise equivalence scoring, and the March Madness check agreed with the judge at a small fraction of its cost.It scored 751,515 pairs in 12 GPU-hours; the reranker and clustering were checked on 2,000 pairs.Source: §5.1 · p. 7
  • Equivalence is tested at three levels: same underlying disagreement, same resolution, or same broader dispute.The main-text results use same-resolution matching unless another level is specified.Source: §5.1 · p. 7
  • Naming up to 25 cruxes on each side still does not close the predicted-versus-discovered overlap gap.The figure shows the share of clusters unique to each set and shared by both, alongside the shuffled-label baseline.Source: Figure 5 · p. 8

5.2 PERMUTATION BASELINE TO ADDRESS SINGLETON-CLUSTER BIAS

  • A permutation baseline holds clusters fixed while randomly reassigning set labels, correcting for singleton clusters that cannot be shared.The gap is the baseline minus observed overlap; a gap near zero means the sets cannot be distinguished by their clusters.Source: §5.2 · p. 8

6 RESULTS

Across the tested comparisons, a priori cruxes align poorly with multiple-rollout cruxes, while a single rollout helps but remains incomplete. The gap persists across sources, equivalence strictness, specificity checks, model families, and role assignments.

6.1 PREDICTED AGAINST DISCOVERED CRUXES

  • 13.4% ± 1.6% overlap versus a 39.2% baseline in March Madness; single-rollout overlap is closer at 18.8% versus 25.1%, but coverage remains short.The single-rollout cruxes cover 28.6% of multiple-rollout cruxes against a 37.8% baseline.Source: §6.1 · pp. 8–9

6.2 SOURCES AND CONTESTED GAMES

  • 39.9% ± 4.0% overlap versus a 70.9% baseline across two sources, and 16.0% ± 2.5% versus 40.5% in contested games; neither source nor winner agreement explains the gap.Thirteen of 15 Metaculus events and 26 of 29 contested games fall below their own baselines.Source: §6.2 · p. 9

6.3 STRICTNESS OF EQUIVALENCE

  • At 25 cruxes, the gap ranges from 25.9 to 36.1 points across three equivalence levels; stricter or looser matching does not close it.The result holds in both sources.Source: §6.3 · p. 9

6.4 SPECIFICITY AND COARSER CLUSTERS

  • The specific prompt narrows the March Madness gap to 16.4 points from 24.7, while overlap stays about 17% and the baseline falls from about 42% to 33%.The remaining gap holds in 58 of 59 games; coarser clustering, vaguer rewrites, and evidence-based matching also leave a gap.Source: §6.4 · p. 9

6.5 OTHER MODELS AND ROLES

  • The gap holds in direction across three open-weight model families, and all 27 predictor–extractor–level combinations fall below baseline.Models predict their own forecasts’ cruxes at most marginally better than those of other models’ forecasts.Source: §6.5 · pp. 9–10

7 DISCUSSION

The discussion concludes that a priori crux prediction differs substantially from extracting cruxes across rollouts, and that the latter is affordable with a reranker. It also identifies limits on the findings and describes how overlap with multiple-rollout cruxes is scored.

7 DISCUSSION

  • The multiple-rollout method extracts a posteriori cruxes substantially differently from advance predictions, while its reranker costs an estimated 86 times less compute than a judge.The discussion presents it as an unsupervised way to extract cruxes from the model's rollouts.Source: §7 · p. 10
  • Across 59 games and 15 events, mostly from GPT models, the result's direction is firm but its size is not; a second draw shifts the Metaculus gap by about ten points.The discussion also notes that the predictor is small and, for Metaculus, did not see web results available to the rollouts.Source: §7 · p. 10
  • Whether a posteriori cruxes help people make better forecasts remains untested and would require a human study.Source: §7 · p. 10
  • The evaluation covers forecasts of single events, leaving open how to extract cruxes for open-ended requests such as reports, plans, and code changes.Such requests lack the forecast number that gives two rollouts a direct basis for disagreement.Source: §7 · p. 10
  • The authors report using generative AI for tasks including experiment execution, dataset work, and prose editing, and release artifacts and data for reproducing analyses from recorded model outputs.The reproducibility statement specifies tolerances for reranker scores and overlap results.Source: AI USE STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · pp. 10–11

A.1 THE OVERLAP, ITS BASELINE AND THE OTHER SCORES

  • Overlap measures the share of clusters containing cruxes from both sets; its baseline randomly reassigns set labels and averages overlap across 1,000 shuffles.A positive gap means the sets overlap less often than shuffled labels would; each game or event is paired with its own baseline.Source: §A.1 · p. 13
  • Coverage measures the share of one set whose clusters also contain cruxes from the other, while split-half checks compare random halves of a set.For single-rollout scoring, cruxes extracted from pairs including that rollout are removed, and a priori cruxes are rescored against the reduced set.Source: §A.1 · p. 13

A.2 DESIGN DETAIL FOR THE CHECKS

The checks illustrate differences in the content and specificity of cruxes, while probing alternative explanations involving weighting, wording, information sources, and matching strictness. Several comparisons remain bounded by the design, including the use of different extractors in other-model runs.

What the gap does not show

  • The gap does not show that any a priori crux is false, and replacing the extractor shifts the gap by about 11 points at 25 cruxes.Other-model runs use different extractors, so they are compared with the main result in direction only.Source: §A.2 · pp. 13–15

Two open objections

  • Equal weighting cannot distinguish missed cruxes by their decision value; weighting by value of information is left for future work.A specificity prompt narrows the March Madness gap from 24.7 to 16.4 points at 25 cruxes, but the overlap stays about 17%, so the remaining gap is not attributed to wording.Source: §A.2 · p. 14

Sources and contested games

  • The gap's size may reflect differences in sources, topics, rollout counts, prompts, or model roles, and contested-game checks cannot isolate cruxes from winner-disagreeing pairs.Some March Madness splits may trace to report wording; in Metaculus the predictor sees less than the rollouts do.Source: §A.2 · p. 14

Strictness and coarser clusters

  • Matching checks vary strictness: coarser matching lets cruxes share less to cluster together, while each cut receives its own baseline.The same-resolution sweep loosens matching without changing either set; because scores are clustered and cut anew, a looser instruction need not produce larger clusters.Source: §A.2 · p. 14

Choosing the instructions

  • The three instructions were selected from ten candidates by agreement with a language-model judge.Source: §A.2 · p. 15

Qualitative patterns

  • March Madness multiple-rollout cruxes include specific assumptions about steals and turnovers, while the a priori example asks whether free throws become decisive; Metaculus examples include web-found facts absent from the prompt.For Frontier, multiple-rollout cruxes cite Spirit Airlines' restructuring and Frontier's Altman Z-score, which the rollouts found through web search but the predictor did not see.Source: §A.2 · p. 15

Specific prompts

  • In March Madness, 66% of a priori cruxes make a claim and 32% state a value, while 91% of multiple-rollout cruxes state a value.In Metaculus, 89% of a priori cruxes make a claim and 44% of multiple-rollout cruxes state a value; almost no crux on either side names only a bare factor.Source: §A.2 · p. 15

A.3 ROBUSTNESS ACROSS LEVELS OF STRICTNESS

Predicted cruxes remain below the permutation baseline at all three matching strictness levels in both sources. The gap changes in size, but neither a stricter nor a looser definition closes it.

A.3 ROBUSTNESS ACROSS LEVELS OF STRICTNESS

  • At k = 25, overlaps are significantly below each level’s own permutation baseline at all three strictness levels in March Madness (59 games) and Metaculus (15 events).Source: §A.3 · p. 15, Table 1 · p. 16
  • The three matching levels ask whether cruxes concern the same question, same resolution or same dispute.For example, two oil-demand questions match at every level, while an electric-car question joins the demand cruxes only at the broad dispute level.Source: §A.3 · pp. 15, 17
  • At 25 cruxes, March Madness overlap is 5.6%, 13.4% and 12.8% against baselines of 33.4%, 39.2% and 41.2%, respectively.The three pairs correspond to same-question, same-resolution and same-dispute matching.Source: Figure 6 · p. 16
  • The three reranker instructions were selected by balanced accuracy against a language-model judge, but judge labels were not checked against human labels.The judge’s accuracy on cross-set pairs could not be measured from the validation sample.Source: §A.3 · p. 17

A.4 COARSER CUTS AND SPLIT HALVES

Coarsening the clustering does not eliminate the gap below the permutation baseline, and the two sets’ cruxes match each other less than cruxes within either set do. The discovered cruxes are not simply too scattered to match.

A.4 COARSER CUTS AND SPLIT HALVES

  • In March Madness, predicted and multiple-rollout cruxes remain below baseline at every cut coarser than one crux per cluster.The tree is clustered once and then cut at varying points, so only the cut changes.Source: §A.4 · p. 18
  • Two halves of each crux set overlap more than the a priori and multiple-rollout sets overlap with each other.At 25 cruxes, halves of the multiple-rollout set sit just above the cross-set permutation baseline, while the cross-set overlap is far below it.Source: §A.4 · p. 18
  • In Metaculus, the 25-crux overlap is 39.9% ± 4.0% against a 70.9% baseline, and 13 of 15 events fall below their own baseline.A second draw of a priori cruxes changes the gap’s size, but both draws remain below baseline across strictness levels and set sizes.Source: §A.4 · p. 20
  • The overlap varies across events, with a standard deviation of 15.6 points.The Metaculus gap’s direction is stable across two draws, while its size is unstable with only 15 events.Source: §A.4 · p. 20
  • The March Madness overlap rises from 6.3% ± 1.9% at 5 cruxes against a 13.5% baseline to 13.4% at 25 against 39.2%.Overlap varies widely across games, with a standard deviation of 12.1 points.Source: §A.4 · p. 20

A.5 SPECIFIC PROMPTS AND COARSENING

Prompt specificity, coarsening and evidence-based matching do not close the gap between predicted and multiple-rollout cruxes. The most specific prompt narrows it mainly by lowering the baseline.

A.5 SPECIFIC PROMPTS AND COARSENING

  • In March Madness, the specific prompt leaves overlap at 0.169 while lowering the baseline from 0.417 to 0.333.The specific prompt narrows the gap by about a third through the lower baseline; the gap remains 16.4 points.Source: §A.5 · pp. 20, 22, Table 3 · p. 21
  • No prompt or coarsening arm closes the gap; the smallest reported gap is 11.0 points against its own baseline.That minimum is significant at p = 2.3 × 10^-9.Source: §A.5 · p. 21, Table 4 · p. 22
  • Matching cruxes by their written evidence rather than wording still leaves predicted cruxes below baseline.The evidence statements also differ: 75% of multiple-rollout cruxes name a season statistic, rank or record, versus 11% of a priori cruxes.Source: Table 5 · p. 22, §A.5 · p. 22
  • The specific prompt produces cruxes stating values 97% of the time, compared with 32% under the original prompt.The exemplar prompt narrows the gap further, but changes writing style along with specificity.Source: Table 2 · p. 21, §A.5 · p. 22
  • Rewriting multiple-rollout cruxes to be vaguer or restricting both sets to equal-specificity cruxes leaves the gap essentially unchanged.The respective March Madness gaps are 26.9 versus 25.8 points and 22.4 versus 23.9 points.Source: §A.5 · p. 22, Table 4 · p. 22

A.6 THE SINGLE-ROLLOUT CRUXES AGAINST THE MULTIPLE-ROLLOUT CRUXES

Cruxes extracted from one forecast come closer to the multiple-rollout baseline than a priori predictions, but their overlap does not capture all features of the pooled cruxes. Containment also shows a substantial coverage shortfall for a priori cruxes.

A.6 THE SINGLE-ROLLOUT CRUXES AGAINST THE MULTIPLE-ROLLOUT CRUXES

  • Single-rollout cruxes come much closer to baseline than a priori cruxes, with gaps 20.4 points smaller in March Madness and 26.3 points smaller in Metaculus.The March Madness comparison has p = 3 × 10^-15 and the Metaculus comparison p = 6 × 10^-5.Source: §A.6 · pp. 23–24, Table 7 · p. 24
  • On stripped multiple-rollout cruxes, the single-rollout gap at k = 25 is 6.4 points in March Madness and −4.5 in Metaculus, where it is not significant.By contrast, a priori gaps are 28.1 and 33.3 points under strict matching in the respective sources.Source: §A.6 · p. 24
  • Overlap at baseline means exchangeability at the clustering’s resolution, not equivalence in crux recurrence, forecast contrast or pool coverage.The overlap score does not measure those three properties.Source: §A.6 · p. 24
  • A priori cruxes cover less of the multiple-rollout pool than baseline in both sources, while single-rollout coverage also falls short in March Madness.On stripped pools, coverage gaps for a priori versus single-rollout cruxes are 42.7 versus 9.2 points in March Madness and 13.4 versus −3.3 in Metaculus.Source: §A.6 · pp. 24–25, Table 8 · p. 25

A.7 OPEN-WEIGHT REPLICATION PROTOCOL

Open-weight replications test whether crux prediction remains below baseline when the forecaster, predictor, and extractor roles are varied. The appendix also checks robustness to repeated or malformed cruxes and describes the models used in each role.

A.7 OPEN-WEIGHT REPLICATION PROTOCOL

  • Removing repeated cruxes moved reported gaps by at most 1.1 points, without changing direction or significance.Repeats occurred only among multiple-rollout cruxes; none appeared in a priori sets.Source: §A.7 · p. 25
  • Differences between runs of about two points or less can arise from the draw of cruxes alone, and row gaps should not be compared across models.The rows use different set sizes, and both overlap and its baseline change with set size.Source: §A.7 · pp. 25–26
  • Swapping the extractor on fixed forecasts shrank the gap by 11.15 points at k = 25, while all 12 cells remained below baseline.The two extractors produced about 2.7 times different crux counts per game: 160 for gpt-5.4-nano versus 59 for qwen3:32b.Source: §A.7 · p. 26
  • All 27 predictor–extractor cells fell below baseline, by 7.7–21.5 points at k = 12 and 12.2–32.0 points at k = 25.The grid used three predictors and three extractors at all three levels.Source: §A.7 · p. 26
  • A model’s advantage on its own forecasts was at most marginal, and the failure was not specific to a model reading itself.Self-knowledge reached significance in two of six level-and-k combinations and idiom in one, versus an extractor effect of 6–14 points.Source: §A.7 · pp. 26–27

A.8 REGENERATING THE PREDICTED CRUXES

  • The earlier three-sequence prompt corrupted 13.3% of generated cruxes, but the defect did not produce the finding.The reported March Madness cruxes were regenerated as JSON objects; Metaculus cruxes were not regenerated, and the gap against the corrupted set was 3–5 points smaller.Source: §A.8 · p. 27

B.1 MODELS BEHIND EACH ROLE

  • March Madness assigns forecasting, prediction, and extraction to one model, whereas Metaculus assigns each role to a different model.The same reranker decides whether two cruxes match in every experiment.Source: §B.1 · p. 27, Table 10 · p. 27

B.2 COMPUTATIONAL COSTS

This section covers the computational-cost discussion and the Metaculus data, prompt variants, and extraction criteria used to elicit and identify cruxes in forecasts and contrasting arguments. It includes prompts about disputed factual components, causal links, forecast failures, and March Madness factors, as well as scouting-report persona instructions.

B.2 COMPUTATIONAL COSTS

  • 6.4 GPU-hours scored 271,950 pairs in 1,087,800 reranker passes at no dollar cost; a local qwen3:32b judge was estimated to require 550 GPU-hours, 86 times the reranker.Per game, the five a priori calls cost about $0.012, single-rollout extraction $0.0013, and 36 multiple-rollout comparisons an estimated $0.06 to $0.08 at batch list prices; the rollouts themselves were not metered.Source: §B.2 · pp. 27–28

C.1 METACULUS DATA

  • Thirty rollouts were generated per Metaculus event, with 449 of 450 stored across the 15 events listed in Table 12.The automated system used gpt-5-nano with web search; it produced estimates from outside, inside, and contrarian prompts, combined them by a log-odds mean, then used critique and calibration calls to produce the final probability.Source: §C.1 · p. 29, Table 12 · p. 29

C.2 PROMPTS

  • The prompt appendix covers eleven Metaculus prediction strategies, five March Madness variants, three extraction prompts, and three scouting-report persona prompts.All eleven strategies feed a priori cruxes, and calls used the API default temperature with no other sampling overrides.Source: §C.2 · pp. 29–30

C.2.1 PREDICTION PROMPTS FOR METACULUS (IDS 0–10)

  • The Metaculus prompt asks models to anticipate cruxes across likely independent forecasts, defining pairwise cruxes as factual questions whose answers could change conclusions.It directs the model to cluster similar pairwise questions into representative binary questions with contradictory premises, capturing disagreement across the forecast distribution.Source: §C.2.1 · pp. 30–31
  • One prompt asks the model to argue both high- and low-probability cases, then identify where their factual assumptions diverge.Each crux is framed as a neutral factual question with directly contradictory premises.Source: §C.2.1 · p. 33
  • Another prompt identifies uncertain factors, asks whether informed forecasters would disagree, and prioritizes disagreements that most affect probability estimates.It then asks for each crux as a neutral factual question with opposing premises.Source: §C.2.1 · p. 34
  • A Socratic prompt directs the model to question assumptions and evidence, then identify factual questions that would split forecasters into different camps.The prompts also request a sketch of plausible forecast ranges and reasoning.Source: §C.2.1 · pp. 34–35
  • One Metaculus prompt decomposes the main question into factual sub-questions and treats those forecasters would disagree on as cruxes.It directs the model to identify 4–8 sub-questions and prioritize decision-relevant disagreements that would shift the probability estimate.Source: §C.2.1 · p. 36
  • Another Metaculus prompt scripts a debate between high- and low-probability forecasters, then extracts the specific factual questions revealed as sources of disagreement.The debate proceeds for 3–4 exchanges, becoming more specific about where the forecasters disagree.Source: §C.2.1 · p. 36
  • A causal prompt identifies influential factors, assesses uncertainty in causal links, and defines cruxes as the most uncertain, influential links or variables.It emphasizes uncertain, high-leverage causal factors whose mistaken interpretation would most change the forecast.Source: §C.2.1 · p. 37
  • A pre-mortem prompt imagines the consensus forecast was too high or too low and identifies the uncertain facts distinguishing those failure scenarios from a correct forecast.It frames cruxes as factual questions where forecasters would disagree and whose answers could legitimately shift the forecast.Source: §C.2.1 · p. 37

C.2.2 PREDICTION PROMPTS FOR MARCH MADNESS (IDS 20–24)

  • For March Madness, the prompt seeks objective, uncertain facts that could change the predicted winner, requiring contrasts across six categories of matchup factors.These include statistical matchups, coaching and strategy, momentum and psychology, health and conditioning, historical and situational factors, and stylistic clashes.Source: §C.2.2 · p. 38
  • The March Madness prompts seek many distinct disagreement factors, framed as objective yes/no questions with mutually exclusive alternatives.One prompt calls for at least 12 factors and advises against early self-censorship; another asks analysts to examine upset factors, fragile assumptions, and hidden variables.Source: §C.2.2 · p. 39
  • A separate prompt uses four expert lenses—statistics, tactics, scouting, and tournament history—to generate and rank uncertain factors.Each analyst contributes two to three factors from their domain.Source: §C.2.2 · p. 40

C.2.3 EXTRACTION PROMPTS

  • Extraction prompts define cruxes as clear, objective upstream questions whose answer options are mutually exclusive factual claims, not restatements of arguments or conclusions.The Metaculus prompt operationalizes extraction by listing atomic claims from each argument and classifying pairs, with contradictions as the focus for forming cruxes.Source: §C.2.3 · pp. 40–42
  • The extraction prompt defines cruxes as specific, objective factual disagreements, merges contradictions about the same underlying fact, and requires a crux when contradictions exist.It allows an empty list only when the arguments agree on all world-facts.Source: §C.2.3 · p. 42

C.2.4 SINGLE-FORECAST EXTRACTION PROMPT

  • The single-forecast prompt extracts upstream factual premises whose reversal would change the forecast, requiring neutral, mutually exclusive options grounded in objective world facts.It excludes stylistic or methodological features and ranks premises by how much their reversal would move the forecast.Source: §C.2.4 · pp. 42–44

C.2.5 SCOUTING-REPORT PERSONA PROMPTS

  • Three persona prompts—traditional analyst, quantitative forecaster, and matchup analyst—are templates called once per team per persona using gpt-5.4-mini with web search enabled.The prompts request detailed scouting, statistical, or tactical reports for the 2026 NCAA Tournament.Source: §C.2.5 · pp. 44–45
  • The prompt requests current 2026 season data and specific examples covering tactics, pace, player tendencies, transition play, and mid-series adjustments.It also asks for actionable advice against the team, supported by named players or games.Source: §C.2.5 · p. 45
  • Every claim in the requested report must be backed by a specific statistic, named player, or named game.The prompt sets a minimum length of 600 words and requires the response to be valid JSON only.Source: §C.2.5 · p. 45

D.1 EXAMPLE SCOUTING REPORTS

The example scouting reports present three perspectives on Iowa State, each emphasizing different evidence and potential strengths or vulnerabilities. They disagree on some facts, including the team’s record.

D.1 EXAMPLE SCOUTING REPORTS

  • The three reports cover Iowa State’s season and players, statistical profile, and playing style and matchup vulnerabilities.The analyst recounts players and recent games, the statistician lists rankings and performance figures, and the matchup analyst describes style and ways an opponent could beat the team.Source: §D.1 · pp. 46–47
  • The reports disagree on Iowa State’s record: the analyst and statistician give 27–7, while the matchup analyst gives 26–6.The reports are shortened and condensed in the paper’s presentation.Source: §D.1 · pp. 46–47
  • The analyst highlights Iowa State’s 46 forced turnovers across two Arizona State meetings, while the statistician flags its 67.2% free-throw shooting as a vulnerability.The matchup analyst separately points to rebounding and paint volume as ways opponents can disrupt the team.Source: §D.1 · pp. 46–47

D.2 ONE METACULUS ROLLOUT

The section presents one selected Metaculus rollout and examples of its forecast rationale for Frontier’s possible bankruptcy. Other rollout examples show different probability estimates and factor balances.

D.2 ONE METACULUS ROLLOUT

  • The seeded single-rollout rule selects one of 30 stored rollouts, which estimates an 8% probability that Frontier files before May 1, 2026.Its justification balances distress signals against liquidity and the absence of a filing by the target date.Source: §D.2 · pp. 47–49
  • One rollout treats Frontier’s Altman Z-score of 0.38 as a strong distress signal, while its roughly $889 million in available liquidity offsets risk.The rollout also cites Spirit Airlines’ Chapter 11 filing as a factor elevating industry stress.Source: §D.2 · p. 48
  • Frontier’s estimated filing probability is 0.46, a mid-range estimate with substantial uncertainty: distress signals from Frontier and peers raise risk, while liquidity strength and no filing by May 1, 2026 reduce it.The estimate is anchored near a distress benchmark of approximately 45.7% over 24 months.Source: §D.2 · p. 49
  • The ensemble example reports a 16% aggregate, then objections citing liquidity and no filing reduce its assessment to 10%.The response retains non-zero risk because distress signals and industry dynamics remain.Source: §D.2 · p. 50

D.3 THE FIGURE 1 DEMONSTRATION

The paper illustrates its crux-extraction pipeline with a separate forecasting run, then distinguishes its question from related work on introspection, claim decomposition, aggregation, and forecasting. The related-work discussion emphasizes that the paper tests whether a model can predict which factual questions its own forecasts will disagree on.

D.3 THE FIGURE 1 DEMONSTRATION

  • The demonstration uses 30 Claude Opus 5.5 forecasts to form 435 pairs and extracts 86 unique cruxes, but the run is not reproducible and no paper result depends on it.The figure shows four selected cruxes and three selected pairs; the examples were chosen for readability, not at random.Source: §D.3 · p. 50

E.1 EXTENDED RELATED WORK

  • Unlike work identifying consequential reasoning steps within one chain of thought, this paper asks whether a model can predict factual disagreements across its rollouts.The a priori cruxes are called predictions rather than introspection because the predictor sees no rollout from which to infer them.Source: §E.1 · p. 51
  • The crux-extraction method compares two rollouts at once and keeps only claims on which they take different sides, unlike general claim decomposition.Claim-decomposition examples include Key Point Analysis, FActScore, and SAFE.Source: §E.1 · p. 51
  • Self-consistency and debate seek better answers from multiple outputs; this paper instead breaks their disagreement into cruxes and asks whether those cruxes were predictable.Prior forecasting systems combine independent runs, whereas this paper studies the cruxes on which the runs differ.Source: §E.1 · p. 51

Related papers

Loading 2610.21006v1…