PaperlayerView exact revision on arXiv

Source-linked AI summary

A latent dimension of Condorcet's jury theorem for multiple AI advisers

Kazutoshi Sasahara, Aoi Naito, Ryo Fujie

arXiv:2609.14438v2cs.CYcs.AIcs.HCcs.MA

TL;DR

Multiple AI advisers can improve majority reliability while making disagreement more visible to users who cannot verify the answers. The paper analyzes this tension with a binomial model, finding that reliability and visible dissent both approach certainty at different rates; even ideal advisers can disagree without indicating aggregation failure.

  • Problem

    When users cannot reliably assess AI answers themselves, they may consult multiple advisers, but the majority-reliability view does not capture the vote distribution those users see.

    Source: Introduction · p. 1

  • Method

    The paper studies binary questions with a ground truth using a binomial model of K independent advisers with equal accuracy p above chance.

    Source: Introduction · p. 1

  • Results

    Aggregate reliability and visible dissent both approach certainty as the panel grows, but at different rates that cross at a particular adviser accuracy.

    Source: Introduction · p. 1

  • Takeaways & Limitations

    Disagreement can arise even among ideal advisers, so it does not by itself indicate aggregation failure.

    Source: Introduction · p. 2

    The base model assumes independent advisers with equal accuracy, an assumption made for clarity that is not required by the theorem.

    Source: Introduction · p. 2

Abstract

from arXiv · show

When the same question is asked of multiple AI advisers, as in self-consistency and LLM-as-a-judge panels, Condorcet's jury theorem predicts that adding independent, competent advisers makes the majority more reliable. The theorem, however, has a latent dimension when viewed from the user's vantage: adding advisers also makes disagreement more visible. A binomial model reveals that this ``visible dissent'' becomes nearly inevitable as the number of advisers grows, and that reliability and disagreement approach certainty at rates that cross at an adviser accuracy of 4/5 (0.8); below it, visible dissent eventually becomes more likely than a correct majority. Even ideal panels of independent and competent advisers can be correct in aggregate but appear divided; such disagreement does not by itself indicate aggregation failure. The way advisers split also provides a common basis for predictive multiplicity, reconciliation load, and reliance miscalibration. These results separate aggregation from disclosure and turn the latter into testable questions about how disagreement should be presented and interpreted.

Introduction

The paper revisits Condorcet’s jury theorem from the user’s vantage: a growing panel can make the majority more reliable while making disagreement more visible. It studies this tension with a binomial model of independent advisers and notes that even ideal advisers can disagree.

Introduction

  • People may consult multiple AI advisers about claims they cannot directly verify, including diagnostic systems assessing a medical scan.A larger panel can improve majority reliability while making disagreements harder for users to ignore.Source: Introduction · p. 1
  • For independent advisers more accurate than chance, Condorcet’s jury theorem predicts that aggregate reliability approaches certainty as the panel grows.The same logic underlies wisdom-of-crowds approaches and AI methods including self-consistency and LLM-as-a-judge panels.Source: Introduction · p. 1
  • The usual theorem evaluates a majority against ground truth, discarding the vote distribution users observe; distributing trust across multiple anchors creates new uncertainties.This is the paper’s “multi-trust-anchor problem.”Source: Introduction · p. 1
  • Both aggregate reliability and visible dissent approach certainty as the panel grows, but at different rates that cross at a particular adviser accuracy.The system scores the majority against ground truth, while the user sees only the verdicts.Source: Introduction · p. 1, Figure 1 · p. 2
  • The model considers K independent advisers with equal accuracy p, where 1/2 < p < 1, in binary questions with a ground truth.Equal accuracy is assumed for clarity, not required by the theorem; disagreement can arise even among ideal advisers.Source: Introduction · pp. 1–2

A latent dimension: what the user observes

The paper operationalizes visible dissent as a minimal split of at least two votes on each side, then examines how its probability relates to aggregate reliability. Under the model, splits become nearly certain for imperfect advisers, while their relative likelihood depends on adviser accuracy.

A latent dimension: what the user observes

  • Visible dissent is a split with at least two votes on each side, chosen to represent competing positions rather than a lone dissenter.The threshold is an operational definition; it does not claim that lone dissent never matters.Source: A latent dimension: what the user… · p. 2
  • The minimal split can be generalized by replacing the threshold of two votes with a fixed fraction r of the panel on each side.Its complement comprises zero, one, all-but-one, or all advisers on one side.Source: A latent dimension: what the user… · p. 2
  • For every fixed imperfect accuracy in the base model, visible dissent approaches 1 as K grows, and its probability is symmetric in p and q.Near-unanimous panels become rare under the binomial distribution.Source: A latent dimension: what the user… · p. 2
  • At p = 0.9 and K = 20, majority correctness exceeds 99.999% while users see a split about 61% of the time.For any fixed imperfect accuracy, splits approach certainty as the panel grows, although higher accuracy delays the increase.Source: A latent dimension: what the user… · p. 2
  • At p = 0.8 visible dissent never overtakes a correct majority; it first does at K = 14 for p = 0.7 and K = 6 for p = 0.6.These are overlapping events in the same panel, not competing outcomes.Source: Figure 2 · p. 3
  • As the panel grows, the minority’s share of votes approaches q even as the majority becomes increasingly likely to be correct.The dissenting fraction m is the proportion supporting the minority answer.Source: A latent dimension: what the user… · p. 3

When visible dissent outpaces reliability

Aggregate reliability and visible dissent both approach certainty, but their convergence rates cross at adviser accuracy p = 4/5. Below that boundary, visible dissent eventually becomes at least as likely as a correct majority, defining a finite disclosure reference scale.

When visible dissent outpaces reliability

  • At p = 4/5, the convergence-rate ordering changes: above it aggregate reliability converges faster, while below it visible dissent converges faster.The boundary is unchanged by fixed-count split thresholds or fixed weights, though those choices can shift the first finite crossover.Source: When visible dissent outpaces… · p. 3
  • Figure 3 shows the rate crossing at p = 4/5 and plots K∗(p), which diverges as p approaches 4/5 from below.Under fair tie-breaking, K∗ is 6, 14, and 32 at p = 0.6, 0.7, and 0.75.Source: Figure 3 · p. 4
  • For p < 4/5, visible dissent can become at least as likely as a correct majority; at or above 4/5, a correct majority remains more likely for every panel size.The probabilities overlap, so their ordering compares incorrect-majority splits with correct near-unanimous panels.Source: When visible dissent outpaces… · p. 4
  • K∗(p) marks the first panel size where visible dissent is at least as likely as a correct majority; no such crossover exists for p ≥ 4/5.Near the boundary from below, K∗ grows rapidly; at p = 0.6 and 0.7 it is 6 and 14, respectively.Source: When visible dissent outpaces… · p. 4
  • The rate boundary and K∗ provide quantitative reference points for studying how systems should present verdicts that become more reliable in aggregate yet more likely to appear divided.The design question arises even before adviser independence or competence fails.Source: When visible dissent outpaces… · p. 4

Two vantages on the same disagreement

Prior work examines disagreement from collective-performance and user-interpretation vantages, while this paper compares aggregate reliability with visible dissent under classical assumptions. It presents the comparison and its reference quantities as a basis for disclosure design.

Two vantages on the same disagreement

  • Prior work on disagreement addresses distinct questions, including predictive multiplicity, vote-based inference of competence and truth, AI aggregation, and how observed agreement affects user reliance.Table 1 organizes perspectives around collective performance and users’ exposure to and interpretation of advisers’ outputs; these vantages can overlap within a study.Source: Two vantages on the same disagreement · p. 4, Table 1 · p. 5
  • The comparison is characterized by the p = 4/5 accuracy boundary and disclosure reference scale K∗, and the authors state it has not been developed in prior work.The quantities apply to independent, imperfect advisers with the same accuracy, including jurors, sensors, and models.Source: Two vantages on the same disagreement · p. 4
  • The paper compares visible-dissent probability with aggregate reliability as a panel grows, even under classical assumptions favourable to aggregation.Table 1 characterizes the work as examining user exposure to disagreement while comparing the two quantities.Source: Two vantages on the same disagreement · p. 4, Table 1 · p. 5

Three perspectives on the same vote split

The vote split links three perspectives: predictive multiplicity, reconciliation load, and reliance miscalibration. The base model predicts some split-related stimuli, while behavioral responses remain outside it except under an explicitly stated monotone-response hypothesis.

Three perspectives on the same vote split

  • Predictive multiplicity treats the vote split as disagreement, reconciliation load as the dissenting fraction, and reliance miscalibration as responses to splits or unanimity.The base model gives each literature a predictive counterpart in advance, although each has measured its quantity after the fact.Source: Three perspectives on the same vote… · p. 5
  • For reconciliation load and under-reliance, the model specifies user stimuli, but predicting behavioral responses assumes they do not decrease as their corresponding stimulus increases.The authors call this the monotone-response hypothesis and leave its empirical test to future work.Source: Three perspectives on the same vote… · p. 5
  • At p = 0.7 and K = 14, majority accuracy falls from 0.938 before observing the vote to 0.845 after an 8–6 split, so reduced trust can be calibrated.Many splits occur despite a correct majority, and may be interpreted as evidence of uncertainty.Source: Three perspectives on the same vote… · p. 5
  • An all-wrong panel and an all-correct panel produce the same unanimous vote; under independence, unanimity is correct with probability 1/(1+(q/p)^K).Over-trust can arise when correlated errors inflate all-wrong unanimity above the independence baseline but users treat signals as independent.Source: Three perspectives on the same vote… · p. 5
  • The model places the stimuli for reconciliation load and under-reliance on the same panel-size scale, while the all-wrong probability q^K is negligible at K∗ under independence.K∗ marks the first crossover between visible dissent and aggregate reliability.Source: Three perspectives on the same vote… · p. 5

From null model to exact sampling law

The paper uses an independence baseline to distinguish aggregation from the user-visible vote split, then applies it to repeated sampling and disclosure decisions. The model also marks boundaries: below-chance accuracy can make majority voting confidently wrong, and real panels may violate its assumptions.

From null model to exact sampling law

  • The vote split supplies three model-derived quantities: any disagreement, expected dissenting fraction, and all-wrong unanimity.The link between expected dissenting fraction and reconciliation load relies on the monotone-response hypothesis; the curves are not behavioural responses.Source: Figure 4 · p. 6
  • The independence baseline separates residual dependence from easy questions and unequal accuracies; after adjustment, a remaining split shortfall is evidence of residual dependence.In the simulated correlated panels, dependence also lowers the split rate, but that does not follow from positive pairwise correlation alone.Source: From null model to exact sampling law · p. 6
  • Repeated fixed-setting samples mapped to binary verdicts are i.i.d. with correctness probability p, making Pvis(K, p) an exact law rather than a baseline.Changes in prompts, model versions, or serving conditions can break the i.i.d. assumption.Source: From null model to exact sampling law · p. 6
  • Below chance, i.i.d. samples can make majority voting increasingly likely to select the wrong answer, while the split distribution cannot warn because it is symmetric in p and q.After adjusting for difficulty and unequal accuracies, a lower-than-expected split rate provides evidence of residual dependence.Source: From null model to exact sampling law · p. 6
  • At p ≥ 4/5, splits never overtake a correct majority; below 4/5, K* is the first panel size where a split is at least as likely as a correct majority.The framework proposes testing presentation policies across the two orderings, while leaving the ordering unassigned when the uncertainty interval for p spans both.Source: From null model to exact sampling law · p. 7
  • The proposed policy comparisons measure judgement accuracy, reliance calibration conditional on the observed vote, and reconciliation load.The outlined policies range from individual verdicts to verdicts with vote margin and confidence, or an openly reconciled answer with access to the full panel.Source: From null model to exact sampling law · p. 7

A framework for the disclosure layer

The framework treats adding advisers and showing their verdicts as separate design decisions. It uses the split and a reference panel size to frame how systems present advice, while cautioning that domain-average accuracy is not an item-level trigger.

A framework for the disclosure layer

  • Adding advisers and showing their verdicts are distinct design decisions addressed at the disclosure layer.The model supplies the expected dissenting fraction m → q and reference scale K*.Source: A framework for the disclosure layer · p. 7
  • For self-consistency, the base model yields K* exactly; pipelines that do not vote fall under the framework only after outputs are converted to verdicts and majority-voted..Source: A framework for the disclosure layer · p. 7
  • A domain-average accuracy is only a guide, not an item-level trigger, because mixed item difficulties can reverse the ordering predicted for a uniform accuracy.At K = 14, a mixture of p = 0.51 and p = 0.89 items (mean 0.7) reverses the ordering for items uniformly at p = 0.7.Source: A framework for the disclosure layer · p. 7
  • The split event is side-blind, whereas near-agreement points to a side because a near-unanimous panel is correct with probability approaching one.The system should explain disagreement expected under the base model while retaining information in the observed vote margin.Source: A framework for the disclosure layer · pp. 7–8

Discussion

The discussion separates majority reliability from the interpretation and disclosure of disagreement, including when model assumptions fail. It identifies empirical tests and scope boundaries for applying the framework.

Discussion

  • Even independent, competent advisers can produce visible disagreement as majority reliability rises, so disagreement alone does not establish aggregation failure.The 4/5 rate boundary and K* make the disclosure question concrete.Source: Discussion · p. 8
  • The proposed empirical work tests presentation effects around K* and compares real panels with the independence baseline, while distinguishing model results from behavioural hypotheses.A practical first step is recording pre-aggregation vote distributions and split rates alongside majority accuracy.Source: Discussion · p. 8
  • Dependence can suppress visible dissent, while worse-than-chance advisers can make majority voting increasingly confidently wrong as panel size grows.A dissent deficit supports evidence of residual dependence only after adjusting for item difficulty and unequal accuracies, and its magnitude needs calibration.Source: Discussion · p. 8
  • Extending the framework to interactive panels requires tracking how interaction changes both the verdict distribution and its interpretation.When agents revise one another's answers, interaction itself can shape agreement.Source: Discussion · p. 8
  • The framework applies to questions with ground truth; deciding whether a question is in scope is a governance decision, not one to leave to an AI system alone.Treating a contested normative issue as settled can hide legitimate disagreement from users.Source: Discussion · p. 8
  • Beyond visible dissent, users may not know whether agreement arose independently or from advisers failing together; more complex systems add further layers of trust and disclosure.The framework describes comparing observed split rates with the independence baseline as a first diagnostic.Source: Discussion · p. 8

Declarations

The authors describe their contributions and report that no empirical datasets were generated or analysed. Numerical analyses derive from closed forms and synthetic simulations.

Declarations

  • K.S. conceived and designed the study, developed the theory and analysis, and wrote the manuscript.A.N. contributed to study design by connecting the framework to human–AI reliance research and interpreting user-side quantities.Source: Declarations · p. 8
  • R.F. contributed to the theoretical analysis design and checked derivations in the Supplementary Information.All authors reviewed and approved the manuscript.Source: Declarations · p. 9
  • No empirical datasets were generated or analysed; numerical analyses derive from closed forms and synthetic simulations.Scripts reproducing numerical calculations and simulation figures are available in the stated code repository.Source: Declarations · p. 9

Supplementary Information for “A latent dimension of Condorcet’s jury theorem for

The supplementary material analyzes visible dissent and aggregate reliability in a homogeneous, independent binary-adviser model. It establishes the split probability’s behavior and its symmetry, distinct from the ground-truth-dependent majority measure.

Base model

  • The base model assumes K independent advisers answer a binary question with shared accuracy p, where 1/2 < p < 1.User-side quantities depend on the vote split; only aggregate reliability and K∗ require the ground-truth label.Source: multiple AI advisers” · p. 12

Supplementary Note 1 | Aggregate reliability Pagg

  • Under fair tie-breaking, aggregate reliability approaches one as the panel grows, with the strong law giving the convergence result.Fair tie-breaking also makes Pagg constant across each consecutive odd–even pair, which can affect the ordering near K∗.Source: multiple AI advisers” · p. 12

Supplementary Note 2 | Visible dissent Pvis

  • For K ≥ 4, visible dissent is the complement of four near-unanimous configurations; it is zero for K ≤ 3 and approaches one for every p < 1.The closed form is symmetric in p and q, because relabeling which side is correct does not change whether the panel is split.Source: multiple AI advisers” · p. 12

Symmetry of visible dissent

  • Visible dissent is side-blind, so Pvis(K,q) = Pvis(K,p), while aggregate reliability is antisymmetric under swapping p and q.Conditional on a near-unanimous panel, the indicated side is correct with probability approaching one as K grows.Source: multiple AI advisers” · p. 12

Supplementary Note 3 | Dissenting fraction

The supplementary analyses compare the rates of majority reliability and visible dissent, then qualify the comparison for heterogeneous panels and dependent or difficulty-mixed data. They also specify how to interpret crossings and use dissent as an independence baseline.

Convergence rates and the p = 4/5 line

  • For equal-accuracy independent advisers, the rates cross at p = 4/5: below it visible dissent eventually exceeds a correct majority, while at or above it reliability stays ahead.At p ≥ 4/5, no finite K∗ exists; the rate boundary is unchanged by fixed weighting of residual probabilities or by alternative split definitions that exclude unanimity.Source: Supplementary Note 3 | Dissenting… · pp. 13, 15

Probability comparison

Heterogeneous panels

  • For heterogeneous independent panels, mean accuracy alone does not locate the crossing; the full accuracy profile is needed to compute Pvis and Pagg.A panel with half its advisers at 0.999 accuracy and half at 0.55 has mean accuracy about 0.77, yet its correct majority exceeded a split at every tested size through K = 400.Source: Supplementary Note 3 | Dissenting… · pp. 13–14

Disclosure reference scale K∗

  • K∗ is the first panel size at which visible dissent is at least as likely as a correct majority; for p = 0.7 it is 14 under fair tie-breaking.The crossing can differ by parity, and strict tie handling can produce a smaller K∗ than fair tie-breaking.Source: Supplementary Note 3 | Dissenting… · p. 14

Diagnostic procedure

  • A dissent deficit is evidence of residual dependence only after accounting for sampling variation, accuracy uncertainty, item difficulty, and adviser heterogeneity.For homogeneous reference panels, the remaining deficit can be summarized by an effective panel size Keff; this is a composite diagnostic when heterogeneity is unresolved.Source: Supplementary Note 3 | Dissenting… · p. 15

Diagnostic limitations

  • Item difficulty and unequal adviser accuracies can both mimic a dissent deficit, and difficulty stratification usually requires per-item labels.Whether changes in split rate with panel size can separate dependence from difficulty mixing without labels remains open.Source: Supplementary Note 3 | Dissenting… · p. 15

Difficulty mixtures

  • At the same mean accuracy of 0.7 and K = 14, homogeneous and difficulty-mixed item populations have opposite Pvis–Pagg orderings.Thus mean accuracy alone determines neither the ordering at a given size nor the location of a crossing.Source: Supplementary Note 3 | Dissenting… · p. 15, Table 2 · p. 16

Pvis as a null model

  • Positive correlation can reduce visible splits below the independence baseline while inflating the all-wrong unanimity rate in the illustrated synthetic panels.At p = 0.7 and K∗ = 14, the baseline split rate is about 0.95, versus about 82.5% for ρ = 0.2 and 63% for ρ = 0.5; these are simulations, not real-panel data.Source: Supplementary Note 3 | Dissenting… · p. 15, Figure 2 · p. 16

Supplementary Note 7 | Three user-side quantities and the monotone-response hypothesis

The paper connects vote splits to predictive multiplicity, reconciliation load, and reliance responses, while distinguishing mathematical quantities from behavioral hypotheses. It specifies stimulus orderings that can be tested, not behavioral outcomes already established.

Predictive multiplicity

  • Predictive multiplicity maps directly to the panel’s observable disagreement rate, defined either as any non-unanimous vote or as the minimal split Pvis.The two definitions differ only in one-dissenter outcomes and share the same asymptotic rate.Source: Supplementary Note 7 | Three user-side… · p. 16

Reconciliation load

  • Reconciliation load is linked to the expected dissenting fraction E[m(K)], which approaches q as the panel grows.The monotone-response hypothesis assumes expected load does not decrease as E[m(K)] increases.Source: Supplementary Note 7 | Three user-side… · p. 17

Reliance miscalibration

  • Visible dissent is the proposed stimulus for under-reliance, while unanimously wrong advice supplies the observable cue relevant to over-trust.Under independence, the all-wrong probability is q^K; whether users over-trust apparent unanimity is an empirical question.Source: Supplementary Note 7 | Three user-side… · p. 17

Testing the monotone-response hypothesis

  • Holding p fixed, increasing K raises Pvis and does not decrease E[m(K)], but whether reliance and reconciliation responses follow those stimuli is untested.The proposed test varies (K,p) and measures reliance and reported load against the predicted stimuli, accounting for moderators such as task domain and user expertise.Source: Supplementary Note 7 | Three user-side… · p. 17

Confidence conditional on the observed vote

  • Given a fixed p and observed absolute vote margin d ≥ 1, the majority is correct with probability 1/(1 + (q/p)^d).For p = 0.7 and K = 14, aggregate accuracy is 0.938 but is 0.845 conditional on an 8–6 split.Source: Supplementary Note 7 | Three user-side… · p. 17
Loading 2609.14438v2…