Source-linked AI summary
Bayesian Intelligence from the Outside
Alex Smolin, Bryan Wilder
TL;DR
Because internal information is opaque, it is difficult to determine from language-model behavior whether reasoning is internally consistent. The paper models inference as Bayesian updating on prompt-induced internal experiments and characterizes the resulting report distributions, information comparisons, and aggregates. It finds that joint hard exclusions constrain rationalizability, while report frequencies and positive probabilities do not; coarse reports also leave aggregation highly underdetermined.
Problem
Internal information driving language-model answers is opaque, making it difficult to test whether their reasoning is internally consistent from observable behavior.
Source: §1 · pp. 1–2Method
Results
Reports are rationalizable exactly when some state remains possible under every report; report frequencies and positive probability values add no restrictions, and the paper also characterizes information orders and coarse-report aggregates.
Source: §1 · p. 2Takeaways & Limitations
Bayesian rationality is difficult to reject from reports alone, and coarse reports can leave the Bayesian aggregate underdetermined.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Inferring intelligence from observable behavior is a foundational challenge in artificial intelligence. We develop a theory of Bayesian intelligence for agents such as language models. Each prompt induces a possibly imperfect internal experiment; the agent updates a full-support prior by Bayes' rule and faithfully reports its posterior over the possible answers to the question. Repetitions draw fresh, independent outcomes from the same unobserved experiment at one fixed state. We show that the agent's behavior admits this explanation if and only if its reports are not fully contradictory, i.e., some state remains possible under every report across all prompts. Report frequencies and the sizes of positive probabilities impose no further restrictions. We further propose and characterize the behavioral implications of an intelligence order that makes the behavior of two agents consistent with one agent having access to a more informative experiment: there should exist a coupling of report distributions such that the more informative agent's report excludes every answer excluded by its counterpart. Finally, we show the difficulty of aggregating coarse reports from intelligent agents: unless the agent reports a belief about the complete state of the world, the optimal aggregation can assign arbitrary weights to states that have not been excluded. These results provide a basis for understanding when agents' behavior is intelligent and highlight the difficulty of rejecting Bayesian rationality.
1 Introduction
The paper asks whether language-model behavior can be rationalized as Bayesian updating when internal information is opaque and prompt-dependent. It characterizes the observable restrictions on rationalizability, information comparisons, and aggregation, and distinguishes this black-box approach from prior evaluations that impose more structure on reports or information.
1 Introduction
- The framework treats inference-time computation as internal information and tests whether final reports are Bayes-consistent without assuming chain-of-thought explanations are faithful.This separates the quality of acquired information from whether the agent updates correctly; the internal experiment may be imperfect and prompt-dependent.Source: §1 · pp. 1–2
- Bayesian rationalizability holds exactly when some complete state remains possible under every report across prompts; report frequencies and positive-probability values impose no further restrictions.The characterization establishes compatibility with Bayesian reasoning, not how an agent actually reasons.Source: §1 · p. 2
- A plausible information order requires reports to share a possible true answer and admit a coupling where the more informative report excludes at least the answers excluded by its counterpart.With only interior reports, either order is compatible, regardless of confidence or variability.Source: §1 · p. 2
- With coarse reports, a Bayesian aggregate must exclude every answer excluded by a report but can assign arbitrary weights to the remaining answers.Arithmetic averaging can violate this restriction; affine log-odds pooling respects it, and is optimal in the Gaussian intelligence model.Source: §1 · p. 2
1.1 Related Literature
- Existing evaluations test particular forms of consistency or inference-task performance, while this paper allows categorical answers and prompt-specific experiments with unrestricted information relationships.The cited work includes tests of probability identities and evidence-based updating, as well as benchmarks of probabilistic judgments.Source: §1.1 · p. 3
- Unlike results that average over states, this paper conditions on one realized state and studies joint feasibility across prompt-dependent experiments, yielding weaker restrictions.The discussion also relates this setting to work on unrestricted information structures and misspecified signal models.Source: §1.1 · pp. 3–4
2 Model
The model treats an agent’s response to a prompt as a report of its posterior after a prompt-dependent internal experiment. Repeated prompts use the same experiment at a fixed state, while the model assumes truthful posterior reporting and correctly specified Bayesian updating.
2 Model
- The agent begins with a full-support prior over a finite state space and each prompt induces a statistical experiment whose signal informs its posterior.The prior can be interpreted as the agent’s knowledge before additional computation.Source: §2 · p. 4
- The model focuses on prompts requesting the agent’s posterior over possible answers, where each answer corresponds to a set of compatible states.The answer partition may be coarse; some results explicitly assume each answer is compatible with at least two states.Source: §2 · p. 5
- Obedience requires the agent to report its requested posterior, isolating Bayesian reasoning from reporting errors or strategic distortions.The assumption strengthens the finding that Bayesian reasoning is difficult to reject.Source: §2 · p. 5
- Across repetitions, the state and prompt-specific experiment remain fixed, but fresh conditionally independent signals can generate stochastic reports.The model requires a full-support prior and correctly specified signal distributions, and permits experiments to depend arbitrarily on prompts.Source: §2 · pp. 5–6
3 Plausible Report Distributions
The section characterizes when repeated reports across prompts can arise from Bayesian updating at one fixed true state. The decisive restriction is joint consistency of hard exclusions, not report frequencies or positive probability values.
3 Plausible Report Distributions
- A collection of report distributions is Bayes-plausible exactly when at least one complete state remains compatible with every report.At the true state, Bayes’ rule and a full-support prior require every reported answer to retain positive posterior probability; conversely, a common prior and prompt-specific experiments can rationalize a shared compatible state.Source: §3 · p. 6
- Positive report probabilities and their frequencies impose no further restrictions, so any collection of reports with full support is Bayes-plausible.The restriction concerns whether hard exclusions are jointly consistent across prompts and reports.Source: §3 · p. 6
- The same characterization applies to adaptive prompting: the response tree is plausible exactly when one state remains compatible with every report at every reachable node.Consequently, no martingale properties should be expected.Source: §3 · p. 6
- The running example’s two report distributions are both Bayes-plausible despite agent 2’s more confident and less consistent reports.Each displayed report occurs with probability 1/2; the reports are 0.4 and 0.6 for agent 1 and 0.1 and 0.9 for agent 2.Source: §3 · p. 7, Figure 1 · p. 7
- Replacing a zero report probability with any sufficiently small positive value preserves plausibility, but setting it to zero can force a candidate true answer and make reports incompatible.The logical implication changes at zero, however small the preceding positive probability was.Source: §3 · p. 7
4 Plausible Intelligence Order
The section defines a behavioral test for whether one agent’s reports are compatible with a more informative experiment than another’s. Under a coarseness condition, compatibility reduces to shared possible answers and a coupling that respects report supports.
4 Plausible Intelligence Order
- A plausible intelligence order exists exactly when reports share a possible true answer and can be coupled so the more informative report’s support is contained in the less informative report’s support.The characterization assumes each answer is compatible with at least two complete states.Source: §4 · p. 8
- The support conditions make testing the proposed order a finite linear feasibility problem rather than a search over state-contingent experiments and garblings.The shared-answer requirement reflects that every report at the true state must assign positive probability to the same answer.Source: §4 · p. 8
- The coupling condition is equivalent to first-order stochastic dominance of induced report-face distributions under face inclusion.Only the supports of reported beliefs matter for this representation.Source: §4 · p. 8
- For binary answers, dominance holds exactly when the two distributions’ supports contain at most one extreme belief and the more informative distribution assigns at least as much mass to each extreme belief.This corollary assumes every answer is compatible with at least two complete states.Source: §4 · p. 8
- The same dominance conditions apply when agents have different priors because the order depends only on report supports.The paper gives differing training datasets as an example of why priors might differ.Source: §4 · p. 8
- Plausible dominance establishes that some compatible experiments have the proposed order, not that the agents’ actual experiments do; its interpretation also relies on obedience.A disobedient agent mechanically reporting the same extreme belief can appear to dominate distributions containing only interior reports.Source: §4 · pp. 8–9
- The example’s report histograms are compatible with opposite information rankings, so the histograms alone cannot distinguish which agent is more informative.The construction allows unreported distinctions between states that share the same answer.Source: §4 · p. 9
- As agent 2’s upper report approaches 1, either ranking remains plausible, but at 1 only agent 2 can plausibly be more informative.At 1, agent 2 sometimes rules out Y = 0 while agent 1 excludes neither answer.Source: §4 · p. 9
5 Plausible Information Aggregation
The section characterizes what Bayesian aggregation can infer from repeated reports when the experiment is unknown. For coarse reports, every aggregate that preserves all reported exclusions is possible, while full-state reports permit a more informative aggregation formula.
5 Plausible Information Aggregation
- With at least two distinct reports and at least two complete states per answer, a fixed-prior aggregate is possible exactly when its support is contained in every report’s support.The condition also characterizes possible aggregates when the prior is unknown.Source: §5 · p. 10
- Any answer excluded by one report must also be excluded by the aggregate, but the aggregate may assign arbitrary weights to answers no report excludes.This is the only guaranteed inference when the experiment is unknown.Source: §5 · p. 10
- With full-state reports and a known prior, the Bayesian aggregate is a prior-corrected sum of reported log odds, even when the experiment is unknown.The formula applies to reports that are posteriors over the complete state.Source: §5 · p. 10
- Coarse reports are generally dependent when conditioned only on the answer, because repeated reports are independent conditional on the complete state.Coarse reports provide prior-weighted averages of statewise likelihood ratios, which do not determine a history’s likelihood without the experiment.Source: §5 · p. 10
- Arithmetic averaging violates the support restriction whenever one report assigns an answer probability zero and another assigns it positive probability; affine log-odds aggregation respects it and is optimal in a Gaussian model.Source: §5 · p. 10
- Two experiments with the same report distribution and prior can yield Bayesian aggregates converging almost surely to opposite answers, even though their empirical means converge to 1/2.The experiments are fixed across repetitions and generate the same distribution over every finite report history.Source: §5 · pp. 10–11
- Replacing an arbitrarily small positive probability with an exact zero forces the aggregate to assign zero probability to that answer once the report occurs.Source: §5 · p. 11
6 Future Directions
The appendix reduces prompt-dependent reports to posteriors over partition cells, develops feasibility restrictions and constructions for known-prior reports, and proves the main report-consistency result. Future work considers structured aggregation, answer revision, and indirect tests of internal experiments.
6 Future Directions
- Future work could structure aggregation for self-consistency and semantic uncertainty, study information retained through revision, and indirectly elicit internal experiments; rigorous conclusions require explicit assumptions.The proposed indirect test elicits a belief and probabilities for possible evidence, then supplies evidence and asks for an updated belief.Source: §6 · p. 11
A.1 Reduction to Reports on a Partition
- When a prompt’s response function is constant on partition fibers, its report factors through the posterior over partition cells without changing the experiment.The partition maps each state to its cell label, and the response is a function of the posterior over those labels.Source: §A.1 · p. 16
A.2 Known Prior and Full-State Reports
- The known-prior construction implements lifted posterior reports with signals and a residual signal that never occurs at the candidate truth.The induced report distribution is exactly the target distribution.Source: §A.2 · pp. 16–17
- With a fixed full-support prior and candidate truth, a report distribution is feasible exactly when its odds-moment inequalities hold for every state; in the binary case, this is the full restriction.The inverse-probability restriction is sharper than the implied mean truth-drifting condition.Source: §A.2 · pp. 16–17
A.3 Proof of Theorem 1
- For Theorem 1’s sufficiency proof, choose a state compatible with every report, lift answer-level reports to full-state posteriors positive there, and select a common prior implementing each prompt distribution.The construction uses the lifted reports’ odds moments to choose positive prior weights.Source: §A.3 · p. 18
A.4 Proof of Theorem 2
The proof constructs experiments that realize coupled report distributions and uses a garbling to obtain the less informative agent’s reports. Its implementation step turns abstract signal probabilities and reports into an experiment.
A.4 Proof of Theorem 2
- A coupling pairs the two agents’ reports so that the more informative report rules out no answer retained by its counterpart, and garbling yields the less informative distribution.The constructed ungarbled and garbled report distributions are τ2 and τ1, respectively.Source: §A.4 · pp. 18, 20
- The implementation step realizes abstract signals when each signal’s ex ante report mass on the true answer is at least its probability at the candidate truth.The proof chooses full-support prior masses across answer cells and assigns the remaining mass within the true-answer cell.Source: §A.4 · p. 19
- The construction creates paired signals for positive coupling mass, adds residual signals with zero probability at the candidate truth, and garbles each pair to its less-informative report.A final signal with an interior report ensures the mean report is interior.Source: §A.4 · p. 19
A.5 Proof of the Different-Priors Remark
The proof shows that the intelligence-order characterization remains valid when agents may have different full-support priors, provided each answer corresponds to at least two states.
A.5 Proof of the Different-Priors Remark
- Allowing separate full-support priors leaves the common-possible-answer and support-inclusion coupling conditions sufficient and necessary when every answer cell has at least two states.The sufficiency construction remains admissible because a common prior is allowed even when different priors are permitted.Source: §A.5 · p. 20
- A report’s posterior support depends on experiment likelihoods, not positive prior weights, so support inclusion remains necessary across agents with different priors.The proof derives this from Bayes’ rule under each agent’s full-support prior.Source: §A.5 · p. 20
A.6 Proof of Corollary 1
The appendix characterizes a special binary-answer case of the intelligence order and derives aggregation results for full-posterior reports.
A.6 Proof of Corollary 1
- With two extreme beliefs, the common-truth condition requires at most one extreme belief across both report supports, and support-respecting coupling requires τ2(νk) ≥ τ1(νk) at each extreme.The converse pairs mass at each extreme first, then couples residual mass to nonextreme reports.Source: §A.6 · p. 20
- When the true-answer cell contains only one state, the coupling condition includes a coordinatewise inequality that implies support inclusion for each positively paired report.The proof obtains the inequality by summing over signals inducing the less-informative report and retaining pairs occurring at the truth.Source: §A.6 · pp. 20–21
A.7 Full-Posterior Aggregation
- For full-posterior reports, the aggregation lemma derives an aggregate formula and its log-odds form for every history with positive probability.The proof obtains the log-odds expression by normalizing the aggregate formula.Source: §A.7 · p. 22
A.8 Proof of Theorem 3
The proof constructs internal posteriors that implement observed reports under a fixed prior, then characterizes aggregation for identical and varying reports. Affine log-odds rules provide a Bayes-plausible aggregation family, including the Gaussian-optimal rule.
A.8 Proof of Theorem 3
- For varying reports, any Bayes-plausible aggregate must exclude every answer excluded by at least one report.If an answer receives positive aggregate weight, every observed report must assign it positive probability.Source: §A.8 · p. 23, §A.10 · p. 24
- A fixed-prior implementation realizes collections of full-state posterior lifts satisfying the proof’s condition, with residual signals producing no observed report.The resulting report history has positive probability at any state where every observed report assigns positive conditional probability.Source: §A.8 · p. 23
A.9 Proof of Lemma 4
- For at least two identical reports, Bayes-plausible aggregates can vary across weights on the report’s supported answers, subject to the lemma’s stated condition.The construction reaches every intermediate cell weight; with one report, the unique aggregate is the report itself.Source: §A.8 · p. 24, §A.9 · p. 24
A.10 Affine Log-Odds Aggregation
- Affine log-odds aggregation is Bayes-plausible under the theorem’s hypotheses, with parameters encoding prior knowledge and learning patterns.The equal-weight logarithmic opinion pool is one choice, and affine log-odds aggregation is Bayes-optimal in the Gaussian intelligence model.Source: §A.10 · p. 25
A.11 Gaussian Intelligence
The Gaussian intelligence model makes repeated signals exchangeable given the coarser answer and yields an affine log-odds posterior aggregate. In the homogeneous case, the aggregation weight depends on report count and signal correlation, with independent and identical-report cases as endpoints.
A.11 Gaussian Intelligence
- The Gaussian model allows reports to remain correlated conditional on the coarser answer because they share an unrevealed component of the full state.The signals are independent conditional on the full state and exchangeable conditional on the answer.Source: §A.11 · p. 25
- The average log-odds vector is sufficient for the answer because its centered residuals are independent of the average and their distribution does not depend on the answer.The signal itself is the posterior log-odds vector.Source: §A.11 · p. 26
- Gaussian aggregation yields posterior log odds that are affine in the observed reports’ log odds.The affine parameters depend on the prior and the model’s covariance matrices.Source: §A.11 · p. 26
- lT = (1 − λT)l0 + λT¯lT, with λT = T / (1 + (T − 1)ρ), in the homogeneous Gaussian case.The covariance matrix Σ must be proportional to G for the aggregation matrix to be a scalar multiple of the identity when T ≥ 2.Source: §A.11 · p. 26
- With Σ = 0, aggregation reduces to the conditionally independent-report formula; with Σ = G, all reports coincide and the aggregate equals one report.Source: §A.11 · p. 27
B.1 Opposite Intelligence Orders
The examples show that observable report distributions need not determine an intelligence order or the aggregate from repeated reports. Even with the same prior and report distribution, fixed experiments can produce aggregates converging to opposite answers when the realized states differ.
B.1 Opposite Intelligence Orders
- E2 strictly dominates E1: under the common prior (5/6, 1/6), ex ante squared-error Bayes risk for predicting Y is 7/100 with E2 versus 8/100 with E1.Source: §B.1 · p. 27
- A small change to the endpoint example leaves one direction of plausible dominance but makes the reverse impossible; removing its shared possible answer violates Theorem 1.Source: §B.1 · p. 28
B.2 Aggregation with Arbitrarily Many Reports
- Two fixed experiments with the same prior and report distribution can produce opposite limiting aggregates because their realized states differ.Both induce reports (1/10, 9/10, 3/22) with ex ante signal probabilities (1/24, 1/24, 11/12).Source: §B.2 · p. 28
- Under either realized report distribution, both reports eventually appear almost surely, while the aggregate converges to 1 under E+ and 0 under E−.Source: §B.2 · p. 28