PaperlayerView exact revision on arXiv

Source-linked AI summary

Auditing Pairwise Equivalence Judgments: Self-Critique Effects and Diversity Measurement in Multi-Agent Hypothesis Generation

Ji Young Byun, Anthony Hu, Jesse Rogers, Roujia Wang, Manasa Kesapragada, Falgun Shah

arXiv:2610.04133v1cs.AI

TL;DR

Because experimental ground truth is often unavailable when scientific hypotheses are proposed, evaluating refinement and diversity requires judging whether hypothesis pairs describe the same mechanism. This paper compares critique depths and equivalence rules, finding that the estimated effects and diversity depend on those pairwise judgments.

  • Problem

    Before experimental ground truth exists, the effect of self-critique and the diversity of delivered hypotheses are difficult to assess, although both depend on deciding whether pairs share a mechanism.

    Source: §1 · p. 2

  • Method

    The study reruns a multi-agent workflow from fixed opening hypotheses at different critique depths and audits three equivalence rules with controlled meaning-preserving and causal-chain perturbations.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    34.5 pp of extra divergence occurs from 0 to 1 critique round versus 1.3 pp from 1 to 5, while First Step Swapped pairs are judged different by 83% under the LLM judge, 8% under embeddings, and 0% under TF–IDF.

    Source: §6 · p. 7

  • Takeaways & Limitations

    Pairwise equivalence is a measurement choice: its definition affects both estimated self-critique effects and measured hypothesis diversity.

    Source: §1 · p. 2, §6 · p. 7

    The study covers four proprietary instances and five repetitions per critique depth, with no biological ground truth to establish correctness or biologically correct grouping.

    Source: §6 · p. 7

Abstract

from arXiv · show

Multi-agent systems built on large language models (LLMs) are increasingly applied to scientific discovery and hypothesis generation. Both the effect of refinement and the diversity of the delivered set are hard to interpret before experimental ground truth exists, and both are typically reported by deciding whether pairs of generated hypotheses describe the same underlying mechanism. We study two evaluation questions that rest on this pairwise equivalence judgment: (1) how much self-critique changes delivered hypotheses beyond run-to-run variability, and (2) how the equivalence rule used to group hypotheses affects measured diversity. Across four proprietary instances, we hold opening hypotheses fixed, rerun the downstream workflow with 0, 1, and 5 critique rounds, and score matched hypothesis pairs with an LLM-as-a-judge. Relative to matched same-depth reruns, moving from 0 to 1 round produces 34.5 percentage points (pp) of additional mechanism-level divergence, whereas 1 to 5 rounds adds 1.3 pp. We then compare three equivalence rules: term frequency--inverse document frequency (TF--IDF) similarity, dense embeddings, and the same LLM-as-a-judge. We construct controlled hypothesis pairs that either preserve the causal explanation through wording or biological-terminology changes, or replace one component of the causal chain while holding the rest fixed. All three rules are invariant to meaning-preserving edits, but when the initiating event is replaced, the LLM-as-a-judge identifies 83% of valid pairs as different mechanisms, versus 0% for TF--IDF and 8% for embeddings; varying only the rubric that defines same mechanism moves this figure from 38% to 96%. Together, these results show that pairwise equivalence judgments are a measurement choice: how mechanism equivalence is defined affects both the estimated effect of self-critique and the measured diversity of generated hypotheses.

1 Introduction

Because scientific hypotheses often lack experimental ground truth when proposed, evaluating them requires criteria applied directly to the generated hypotheses. This study uses pairwise mechanism equivalence to examine self-critique effects and measured diversity.

1 Introduction

  • The study compares equivalence rules on controlled pairs that preserve meaning through some edits and alter a causal-chain component through others.No labels establish whether generated hypotheses share a mechanism.Source: §1 · p. 2
  • Pairwise equivalence judgments provide a shared measurement basis for estimating critique-related change beyond rerun variability and deciding which hypotheses count as distinct mechanisms.These are two evaluation questions often treated separately.Source: §1 · p. 2
  • 34.5 pp of extra divergence follows the introduction of one critique round, compared with 1.3 pp from moving from one to five rounds, relative to matched same-depth reruns.The opening hypotheses were held fixed across downstream workflow reruns.Source: §1 · p. 2
  • For First Step Swapped pairs, the LLM-as-a-judge identifies 83% as different mechanisms, versus 8% for embeddings and 0% for TF–IDF; rubric choice moves its rate from 38% to 96%.All three rules behave alike on meaning-preserving edits.Source: §1 · p. 2

2 Related Work

Prior work distributes hypothesis generation and refinement across single-agent and multi-agent designs. Without a reference answer, evaluation commonly compares generated outputs, including through pairwise LLM judgments.

LLM agents for scientific hypothesis generation

  • Multi-agent scientific systems assign specialized roles to generation, reflection, ranking, evolution, and consolidation, or combine literature-grounded generation with experimental analysis.Other designs simulate a research team that generates, discusses, and assesses ideas before selecting one.Source: §2 · p. 3

Self-critique and iterative refinement

  • Self-critique can use model feedback or environmental signals, while multi-agent workflows may assign critique to a separate instance before producing a final response.Its effect varies by setting: intrinsic self-correction may not improve reasoning, and debate does not always outperform simpler ensembling.Source: §2 · p. 3

Comparative evaluation without a reference

  • Without an external reference, evaluation commonly compares generations pairwise, often asking an LLM judge which output is better.The judge itself lacks a reference for evaluation, motivating scrutiny of its behavior.Source: §2 · p. 3

3 Problem Formulation: Evaluation without Experimental Ground Truth

The paper formalizes evaluation when experimental ground truth is unavailable by comparing hypotheses as causal chains and applying pairwise equivalence rules. It defines critique-related divergence relative to same-depth reruns and treats grouping rules as a choice in diversity measurement.

3 Problem Formulation: Evaluation without Experimental Ground Truth

  • An equivalence rule returns whether two hypotheses express the same mechanism, enabling comparison when biological ground truth is unavailable.Each hypothesis is a causal chain from an initiating event through intermediate biological steps to the experimental finding.Source: §3 · p. 3
  • Run divergence is the fraction of matched hypothesis pairs judged to describe different mechanisms; zero means all matched pairs are judged the same.Larger values indicate greater mechanism-level divergence between runs.Source: §3 · pp. 3–4
  • Extra divergence compares mean cross-depth divergence with the mean divergence of same-depth run pairs, accounting for changes from rerunning configurations.A positive value indicates cross-depth runs diverge more than expected from corresponding same-depth variability.Source: §3 · p. 4
  • The opening hypotheses are held fixed so critique depth is the only experimentally varied configuration.This makes the cross-depth comparison specific to critique depth rather than a different starting set.Source: §3 · p. 4
  • Measured diversity is obtained by grouping hypotheses under TF–IDF similarity, dense embeddings, or an LLM-as-a-judge and counting the resulting clusters.The candidate rules are audited with perturbations that either preserve the mechanism or replace a causal-chain component.Source: §3 · p. 4

4 Experimental Setup

The experiments apply the multi-agent workflow to four proprietary instances and repeat hypothesis generation and refinement at critique depths up to five rounds. Evaluation is conducted post hoc, separately from generation.

4 Experimental Setup

  • Four independent proprietary instances each receive five independent hypothesis-generation and refinement runs, using retrieved evidence and up to five self-critique rounds.The instances are anonymized as A–D.Source: §4 · p. 4
  • Generation uses Claude Sonnet 4.6 for proposer and critic, while post-hoc equivalence judgments use Claude Opus 4.6 at temperature 0.Dense representations use Amazon Titan Text Embeddings V2, and judgments are repeated across three model families for sensitivity analysis.Source: §4 · p. 4
  • Evaluation is post hoc and does not influence hypothesis generation or refinement.The models used for evaluation provide no feedback to the generation process.Source: §4 · pp. 4–5
  • Figure 2 contrasts same-depth rerun divergence across 0, 1, and 5 rounds with extra cross-depth divergence measured against corresponding same-depth variability.Points represent individual instances and bars the aggregate across instances.Source: Figure 2 · p. 5

5 Results

The results show that most critique-depth divergence appears when critique is first introduced, while measured diversity depends on how pairwise mechanism equivalence is judged. Controlled perturbations also show that explicit judge criteria increase sensitivity to causal changes while preserving invariance to meaning-preserving edits.

5.1 Critique-depth effects relative to run-to-run variability

  • 61% of 332 critique demands were conceded, but this engagement does not establish how much critique changed final hypotheses or whether changes were scientifically beneficial.Another 27% were answered, 9% partially addressed, and 3% dropped.Source: §5.1 · p. 5
  • 34.5 pp of extra divergence followed the move from 0 to 1 critique round, compared with 1.3 pp from 1 to 5 rounds, relative to matched same-depth reruns.The study also reports 39.8 pp of extra divergence from 0 to 5 rounds.Source: §5.1 · p. 5

5.2 Measured diversity depends on the equivalence rule

  • 83% of First Step Swapped pairs were judged different by the LLM-as-a-judge, versus 8% for embeddings and 0% for TF–IDF; all three rules treated meaning-preserving edits as the same mechanism.For Middle Step Swapped pairs, the judge marked 13% different, while TF–IDF and embeddings marked none different.Source: §5.2 · p. 6
  • 96% of First Step Swapped pairs were judged different with the step-by-step rubric, compared with 83% with the full rubric and 38% with no rubric.All three prompts remained invariant to meaning-preserving edits; reversing presentation order changed decisions for 5.6% of pairs under each prompt.Source: §5.2 · p. 6

6 Conclusion

The study finds that critique-depth effects and measured diversity depend on pairwise mechanism-equivalence judgments, with most additional divergence occurring when critique is first introduced. Its conclusions describe workflow outputs and evaluation behavior, not biological correctness, and the authors identify evaluator design and scale as boundaries.

6 Conclusion

  • 34.5 pp of extra divergence followed the move from 0 to 1 critique round, while 1 to 5 rounds added 1.3 pp; for First Step Swapped pairs, different-mechanism judgments ranged from 0% for TF–IDF to 83% for the LLM-as-a-judge.Changing only the judge rubric moved the latter figure from 38% to 96%.Source: §6 · p. 7
  • The study covers four proprietary instances and five repetitions per critique depth, without biological ground truth to establish scientific correctness or biologically correct mechanism groupings.Its findings characterize workflow outputs and evaluation methods.Source: §6 · p. 7
  • Measured diversity remains sensitive to evaluator design, and the controlled perturbations cover only a limited set of semantic and causal changes.The authors note that judge model, prompt specification, presentation order, and similarity thresholds affect judgments.Source: §6 · p. 7
  • The authors propose selective or cascaded comparisons and distillation to reduce evaluation cost, alongside expert validation and prospective experiments to assess biological meaning.They also propose broader perturbation suites and independently specified rubrics.Source: §6 · pp. 7–8

A Critique-Depth Effects Relative to Run-to-Run Variability

The first critique round produces substantial extra mechanism-level divergence beyond matched same-depth reruns, while later rounds add little; the pattern persists across judge models.

A Critique-Depth Effects Relative to Run-to-Run Variability

  • With opening hypotheses fixed, the downstream workflow is replayed at 0, 1, and 5 critique rounds, and matched delivered hypotheses are judged for mechanism divergence.Extra divergence subtracts mean same-depth rerun divergence from mean cross-depth divergence.Source: A Critique-Depth Effects Relative to… · p. 11
  • Five runs at each depth across four instances yield 60 runs, 420 run-pair comparisons, and 4,863 matched hypothesis pairs.Each instance begins with five fixed opening hypotheses.Source: A Critique-Depth Effects Relative to… · p. 11
  • For each instance and depth comparison, the permutation test considers all 126 ways to divide ten runs into two groups of five, then averages instance-specific extra-divergence estimates.The two-sided p value is the proportion of partitions with an absolute extra-divergence value at least as large as observed.Source: A Critique-Depth Effects Relative to… · p. 11
  • 34.5 pp for K = 0 versus K = 1 and 39.8 pp for K = 0 versus K = 5 exceed the 1.3 pp for K = 1 versus K = 5.These instance-averaged extra-divergence values are measured relative to corresponding same-depth rerun divergence.Source: Table 1 · p. 11
  • Across three judges, extra divergence ranges from 34.5 to 37.3 pp for K = 0 versus K = 1 and 39.8 to 43.3 pp for K = 0 versus K = 5; both comparisons have p < 0.0001.For K = 1 versus K = 5, the range is 0.5 to 2.3 pp.Source: A Critique-Depth Effects Relative to… · p. 11

B Pairwise Mechanism Comparison Methods

The study compares lexical similarity, dense embeddings, and an LLM judge as rules for deciding whether hypothesis pairs describe the same mechanism, and checks whether judge choice affects critique-depth comparisons.

B Pairwise Mechanism Comparison Methods

  • TF–IDF classifies pairs with cosine similarity of at least 0.25 as the same mechanism, using sublinear term-frequency scaling and English stopword removal.The compared methods receive the same hypothesis statement, mechanistic explanation, and causal chain.Source: B Pairwise Mechanism Comparison Methods · p. 11
  • Across Opus 4.6, Sonnet 4.6, and GPT-5.5, extra divergence remains much larger for K = 0 versus K = 1 or K = 5 than for K = 1 versus K = 5.The reported ranges are 34.5–37.3 pp, 39.8–43.3 pp, and 0.5–2.3 pp, respectively.Source: Figure 5 · p. 12
  • Dense embeddings use a 0.77 cosine-similarity threshold, while the LLM judge returns a binary decision by majority across three judgments, including a presentation-order reversal.The embedding threshold matches the marginal same-pair rate range of the other methods; it does not use individual pair decisions as labels.Source: B Pairwise Mechanism Comparison Methods · p. 12

C Controlled Perturbations for Grouping Decisions

Controlled edited hypothesis pairs test whether comparison methods preserve same-mechanism judgments under wording changes and respond to specified causal-chain changes.

C Controlled Perturbations for Grouping Decisions

  • Wording Only rewrites the mechanistic explanation without changing the causal chain, while Wording + Chain also rewrites the chain using biological aliases but preserves its ordered entities and causal relations.Both perturbations are intended to preserve the same mechanism.Source: C Controlled Perturbations for… · p. 12
  • First Step Swapped replaces the initiating event while holding the downstream chain fixed; Middle Step Swapped replaces one intermediate step while preserving the initiating event and terminal finding.Source: C Controlled Perturbations for… · p. 12
  • Variants are generated at temperature 0.3 and retained only if deterministic checks confirm the intended chain preservation or exactly one designated step change.Constructions that fail validation are excluded.Source: C Controlled Perturbations for… · p. 12

D Sensitivity Analyses for Pairwise Comparison Methods

The comparison methods do not fully agree across threshold settings, while repeated judgments and judge-model comparisons show high consistency with a small presentation-order effect.

D Sensitivity Analyses for Pairwise Comparison Methods

  • At selected thresholds, the three methods reach 85%–89% raw agreement and κ ∈ [0.59, 0.74], but no evaluated threshold yields complete agreement.The highest chance-corrected agreement with another method is κ = 0.74.Source: D Sensitivity Analyses for Pairwise… · pp. 12–13
  • On 180 generated-set pairs, same-order repetition changes no full-rubric decisions, order reversal changes 5 (3%), and three judge models agree at 96.1%–98.3% with κ = 0.86–0.94.Source: D Sensitivity Analyses for Pairwise… · p. 13
  • Across full, step-by-step, and no-rubric prompts, exact repetition changes no decisions; order reversal changes 7 decisions per prompt, with affected pairs differing across prompts.Two affected pairs are shared by the full and step-by-step rubrics, and none are shared with the no-rubric prompt.Source: D Sensitivity Analyses for Pairwise… · p. 13

E LLM-as-a-judge Prompts

The appendix specifies two LLM judging tasks: assessing whether replies engage critiques and deciding whether hypothesis pairs describe the same mechanism. For mechanism judgments, it presents a full rubric, a component-level variant, and a no-rubric variant.

E.1 Critique-engagement judge

  • The critique-engagement judge evaluates each demand in a challenge separately, labeling the proposer’s response as answered, conceded, partially addressed, or dropped.Its output is a JSON list of demand labels.Source: §E.1 · p. 13
  • The critique-engagement task judges whether a reply addresses a critic’s specific challenge, not whether either side is scientifically right or the mechanism is plausible.The prompt frames each example as a critic challenging a mechanistic hypothesis and its generator responding.Source: §E.1 · p. 13
  • “Answered” engages a demand with a reason, evidence, or stated ground for refusal; “conceded” withdraws, corrects, or downgrades the claim.“Partial” engages but leaves part unmet, while “dropped” means the demand is not taken up in the reply.Source: §E.1 · p. 13

E.1 Critique-engagement judge; E.2 LLM-as-a-judge and rubric variants

  • The prompts return JSON: the critique judge outputs demand labels, while the mechanism variants return an overall verdict, with the component-level version also reporting its three comparisons.Source: §E.1 · p. 13, §E.2 · pp. 14–15

E.2 LLM-as-a-judge and rubric variants

  • The mechanism judge compares two hypotheses about the same instance and observed finding, with full-rubric, component-level, and no-rubric prompt variants.The component-level variant records initiating-event, route, and terminal-outcome judgments before an overall verdict; the no-rubric variant requests only a verdict.Source: §E.2 · pp. 13–15
  • Under the full rubric, hypotheses are the same mechanism when they share the initiating event and essential route, allowing different levels of detail.The component-level output records judgments for the initiating event, route, terminal outcome, and overall verdict.Source: §E.2 · p. 14
  • The rubric treats a different initiating event, a genuinely different load-bearing route, or opposite direction or causal orientation as a different mechanism.The initiating event can differ even when downstream language overlaps.Source: §E.2 · p. 14
  • Wording and naming changes, added downstream biomarkers, and different levels of detail do not alone make a mechanism different.Source: §E.2 · p. 14

Related papers

Loading 2610.04133v1…