PaperlayerView exact revision on arXiv

Source-linked AI summary

Do "True" and "False" Add Up to One? Complement Consistency in LLM-Agent Opinion Simulations

Anouar Imel, Jin Qi

arXiv:2610.32874v1cs.AIcs.CLcs.MAcs.SI

TL;DR

LLM-agent simulations use generated probabilities as states, but it is unclear whether those reports remain compatible when the same information is expressed as a complementary claim. This paper tests that consistency in agent updates and examines how discrepancies behave under feedback and repair. Complement discrepancies appear across the tested settings, and symmetrization enforces consistency while its factual-loss effects under feedback depend on the model.

  • Problem

    Whether an agent’s probability update is compatible with an equivalent complementary description is a separate question from robustness to simulation design choices.

    Source: Introduction · p. 1

  • Method

    The paper tests updates by transforming the claim and all claim-relative probabilities, then studies paired reports, feedback trajectories, and symmetrization.

    Source: Introduction · p. 1, The Complement Test · p. 3

  • Results

    Complement discrepancies can enter later feedback, while symmetrization enforces complementary outputs and its effect on factual loss under feedback depends on the model.

    Source: Introduction · pp. 1–2, Discussion · p. 10

  • Takeaways & Limitations

    Simulations that treat generated probabilities as agent states should evaluate consistency separately from schedule robustness and factual loss.

    Source: Discussion · pp. 10–11

    The experimental arms are not matched for compute: averaging requires more calls than the baseline and reminder arms.

    Source: Closed-Loop Results · p. 10

Abstract

from arXiv · show

In LLM-agent opinion simulations, each agent reports the probability that a claim is true after seeing its own previous probability and the probabilities reported by its neighbors, and that number becomes its state in the next round. If the number describes the agent's position rather than the sentence it was shown, then rewriting the claim as its complement and replacing every probability p with 1 - p should return the complementary answer, so the two reports should sum to one. We test this with three small open models: Qwen3-4B, Llama-3.2-3B and Gemma-3-4B. On 2,304 agent states taken from a running network simulation, Qwen's paired reports sum to 1.18 on average, and on 48 real fact-checking claims from AVeriTeC they again exceed one. A controlled benchmark of 24 worlds, each with exactly two possible outcomes, shows the same failure in all three models. The discrepancy is two to ten times larger than the change produced by a same-meaning paraphrase, and it persists under repeated sampling and across peer group sizes and report strengths. When the reports are fed back into two populations that start from complementary states, the populations do not remain mirror images: their divergence is positive after five rounds for every model. Averaging each answer with the complement of its paired answer restores consistency by construction and lowers Brier loss averaged over the two wordings, but at the final round it beats the better of the two original wordings for only one of the three models. A schedule-by-targeting experiment on the same model gives a small, stable interaction because such sweeps keep the claim wording fixed. Consistency under rewording therefore needs its own test before simulated probabilities are read as beliefs

Introduction

Simulation design choices do not answer whether an agent’s update gives compatible probabilities to complementary descriptions. The paper tests that local property and examines whether its failure carries into later simulation outcomes.

Introduction

  • A direct test complements the claim and every claim-relative probability while keeping factual evidence unchanged; reports should then sum to one.A local mismatch changes the numerical state supplied to later updates.Source: Introduction · p. 1
  • The evidence links local paired probes to simulation outcomes through a network experiment, real-claim probes, and a controlled benchmark.The controlled benchmark separates complement discrepancies from paraphrase sensitivity, explicit negation, and repeated-sample variation.Source: Introduction · p. 1
  • Three small open models fail complement consistency on running-simulation states, real fact-checking claims, and a controlled benchmark with exactly two outcomes per world.The discrepancy exceeds same-meaning paraphrase sensitivity, persists under repeated sampling and varied peer information, and can enter later rounds.Source: Introduction · p. 1
  • Averaging each report with the complement of its paired report restores consistency by construction, but its effect on factual loss under feedback depends on the model.The schedule-by-targeting sweep holds claim wording fixed, so it addresses a separate question from complement discrepancy.Source: Introduction · pp. 1–2, Figure 1 · p. 2

Related Work

Prior work examines probability judgments, wording effects, coherence repair, and social influence. This paper connects wording dependence to the numerical states used in agent simulations.

Probability judgments and logical consistency

  • Probability-coherence and identity studies motivate checking elicited numbers against relationships implied by questions.Logical consistency has also been used as an evaluation signal when ground truth is difficult to assess (Fluri, Paleka, and Tramèr 2023).Source: Related Work · p. 2

Wording and response bias

  • Meaning-preserving prompt changes can shift outputs, while the paper’s complement test transforms the claim and every claim-relative probability together.The transformation fixes the correct response by arithmetic.Source: Related Work · p. 2

Consistency repair and elicitation

  • Related coherence-repair work evaluates logically related forecasts, whereas this paper tests how symmetrized states affect later requests and factual loss under feedback (Andrews and Sarkar 2026).Verbalized confidence differs from conditional token probabilities, and the paper’s same-polarity control preserves numerical inputs while complementation changes them.Source: Related Work · p. 2

Agent simulation and social influence

  • Networked language-agent studies examine opinion dynamics and peer pressure, while scheduling is an established simulation choice (Mehdizadeh and Hilbert 2025).Averaging numerical reports differs from self-consistency reasoning over sampled paths.Source: Related Work · pp. 2–3

Update Requests and Consistency

The paper defines complement consistency by transforming a claim and all claim-relative probabilities, then distinguishes consistency of mean reports from consistency of individual draws. It also describes how local updates relate to complementary trajectories and factual loss.

The Complement Test

  • Complement consistency requires paired reports for a claim and its exhaustive alternative to sum to one after the agent’s and peers’ probabilities are complemented.The peer vector retains peer identity and order, while evidence facts remain unchanged; the transformation must be involutive on the defined state.Source: The Complement Test · p. 3

Mean Reports and Individual Draws

  • For stochastic decoding, complementary means can coexist with nonzero individual-draw discrepancy, and equal means are weaker than distributional complement consistency.The paired-draw analysis uses independently elicited distributions; its cross-products are dependent, and the plug-in estimate of mean discrepancy is biased upward.Source: Mean Reports and Individual Draws · p. 3

Synchronous Feedback and Factual Loss

  • With the same graph, evidence, and wording-independent execution rules, complement-consistent local updates preserve complementary states across synchronous rounds.After paths diverge, later trajectory divergence measures accumulated separation rather than a one-step error; factual loss is reported by orientation and as an equal-weight mean, and consistency is not accuracy.Source: Synchronous Feedback and Factual Loss · pp. 3–4

Experimental Design

The studies combine network simulations, paired response probes, and controlled tests to examine how update schedules, evidence, peer input, and wording shape model probabilities. They also specify how models, repair conditions, and uncertainty are handled.

Simulation, Preserved States, and Real Claims

  • The network experiment varies evidence targeting and update schedules, while paired probes test 2,304 post-delivery states and 48 fact-checking claims.The network uses 16 matched claims, 12 world seeds, 12 agents, and six rounds; preserved-state probes evaluate one update without propagation, and the real-claim study crosses initial positions, peer opposition, evidence, wording, audience, and peer order.Source: Simulation, Preserved States, and Real… · p. 4

Attraction, Accuracy, and Evidence Access

  • The analysis distinguishes peer attraction from accuracy change and defines evidence protection as reducing endpoint decline under misleading rather than agreeing peers.Positive attraction means movement toward the peer mean; positive Brier change means higher loss. The evidence-protection contrast is distinct from one-step accuracy gain.Source: Attraction, Accuracy, and Evidence… · pp. 4–5

Materials and Models

  • The controlled benchmark uses 24 exhaustive binary worlds, while model configurations differ in size, quantization, and prompt format.World descriptions were checked programmatically for exhaustive, mutually exclusive outcomes, but wording was not validated by human raters; model comparisons therefore do not isolate architecture.Source: Materials and Models · p. 5, Table 1 · p. 5

Controls, Feedback, and Repairs

  • Follow-up studies test repeated sampling, prompt controls, propagation, repairs, and peer counts using six worlds, with reports fed back as states without smoothing or memory.The peer study crosses counts of 2, 4, and 8 and three symmetric report pairs; because count changes prompt length and report strength changes distance from the prior, neither factor is isolated.Source: Controls, Feedback, and Repairs · pp. 5–6

Weighting and Uncertainty

Results

Across models and conditions, complement discrepancies exceed same-polarity paraphrase sensitivity, persist across repeated sampling and peer settings, and enter later feedback trajectories. Scheduling and targeting results address a separate property and do not establish complement consistency.

Scheduling and Targeting Measure a Different Property

  • Targeting intervals cross zero, while delivery lowers final Brier-loss point estimates; under degree targeting, final variance is 0.0049 for synchronous and 0.0024 for fixed sequential updating.Figure 2 tracks population Brier loss across six rounds and update schedules, with one-time source delivery at round two.Source: Scheduling and Targeting Measure a… · p. 6, Figure 2 · p. 7

Complement Tests on Simulation States and Real Claims

  • On preserved simulation states, mean signed and absolute discrepancies are 0.1846 and 0.1899; on real claims, they are 0.1867 and 0.3259.Restoring the source changes the preserved-state signed mean to −0.1138; real-claim results vary by label, with larger signed and absolute means for Supported claims.Source: Complement Tests on Simulation States… · p. 7

Attraction, Accuracy, and Consistency Separate Empirically

  • Source restoration reduces peer attraction and harmful-attraction frequency but does not establish average accuracy gain; persuasion, evidence protection, and accuracy remain distinct outcomes.The source-test Brier-gain estimate and real-claim evidence-protection estimate both have intervals spanning zero.Source: Attraction, Accuracy, and Consistency… · pp. 7–8

Complement Discrepancy and Paraphrase Sensitivity

  • Complement discrepancy exceeds same-polarity paraphrase sensitivity in all six model/evidence cases, while explicit negation shifts discrepancy differently across models.Not every request fails: with visible evidence, 215 of 288 Qwen pairs and 210 of 288 Llama pairs are exactly consistent.Source: Complement Discrepancy and Paraphrase… · p. 8

Repetition and Prompt Controls

  • Repeated sampling preserves mean-level discrepancy; prompt effects vary by model, and reordering evidence items can change reports without adding information.For Llama with visible evidence, mean-level discrepancy is 0.168, versus 0.184 for independently paired draws and 0.200 under greedy decoding.Source: Repetition and Prompt Controls · p. 8

Paired Feedback Trajectories

  • All paired populations develop positive final divergence; with visible evidence, Qwen’s divergence stays roughly flat while Llama’s and Gemma’s grow.The results show wording dependence enters feedback, but is not always amplified; an average-state simulator preserves complementary paths to rounding error.Source: Paired Feedback Trajectories · p. 8

Sensitivity to Peer Count and Reports

  • All 54 model/evidence/peer-count/report combinations show positive mean discrepancy, although 624 of 1,944 pairs are exactly consistent; heterogeneous peer groups remain untested.For Llama, increasing peers from four to eight with 0.2/0.8 reports lowers visible discrepancy by 0.042 but raises orientation-averaged Brier loss by 0.008.Source: Sensitivity to Peer Count and Reports · p. 8

Averaging, Symmetrization, and Utility

For a fixed pair of complementary requests, symmetrization guarantees complementary reports and cannot increase orientation-averaged Brier loss. In closed-loop tests, it lowers loss overall, but does not consistently outperform the better original orientation, and the comparison uses differing call budgets.

What the Algebra Guarantees

  • For fixed complementary-label pairs, symmetrization cannot increase orientation-averaged Brier loss, though it need not improve either orientation; averaging repeated reports need not remove mean-level discrepancy.The one-step guarantee does not determine losses after repaired outputs change future states.Source: What the Algebra Guarantees · pp. 8, 10, Figure 5 · p. 10

Closed-Loop Results

  • Symmetrization lowers final and round-averaged visible Brier loss, averaged over orientations, for all three models, but does not beat both baseline orientations.Qwen’s final loss and Llama’s final and round-averaged losses are worse than their respective better baseline orientations.Source: Closed-Loop Results · p. 10
  • The complement reminder improves orientation-averaged endpoints for Qwen and Llama but worsens Gemma’s; for Llama, the complementary orientation improves from 0.345 to 0.209.The original Llama orientation changes from 0.071 to 0.074.Source: Closed-Loop Results · p. 10
  • Symmetrization uses two calls for one repaired answer and its complementary display, so the repair arms are not matched for compute.Same-orientation greedy calls reproduce baseline paths for the requests visited, not necessarily every possible prompt.Source: Closed-Loop Results · p. 10

Discussion

Network robustness and wording consistency measure different properties: fixed-wording sweeps do not establish compatible reports for complementary descriptions. Symmetrization lowers loss averaged across wordings, but does not always outperform the better original wording.

Local consistency and collective robustness

  • A small, stable schedule-by-targeting interaction does not establish complement consistency because the sweep keeps claim wording fixed.The converse of the stated sufficiency result is false, so the two kinds of evidence must be collected separately.Source: Discussion · p. 10

Consistency and factual utility

  • Symmetrization lowers loss averaged over the two wordings, but sometimes performs worse than the better original wording.Its factual value under feedback should therefore be judged using wording-specific losses and whole trajectories.Source: Discussion · p. 10, Figure 6 · p. 11

Implications for evaluation

  • Wording differences can enter feedback without always being amplified, so both orientations should be propagated under identical execution rules.Evaluation should also pair consistency checks with same-polarity paraphrase controls, distinguish mean-report effects from variation across draws, and assess factual loss and call cost for repairs.Source: Discussion · pp. 10–11

Conclusion

The tested agent simulations produce incompatible probability reports for equivalent complementary descriptions. The discrepancy appears across simulation states, real claims, and a controlled benchmark, and symmetrization's lower average loss does not always beat the better original wording.

Conclusion

  • Equivalent complementary descriptions produce incompatible probability reports in the tested agent simulations.The discrepancy appears on states from a running simulation and real fact-checking claims, exceeds matched paraphrase sensitivity in the controlled benchmark, and can persist after reports become later inputs.Source: Conclusion · p. 11
  • Symmetrization enforces complementary outputs, but lower loss averaged over orientations can coexist with higher loss than the better baseline orientation.Consistency and factual utility therefore remain distinct considerations under feedback.Source: Conclusion · p. 11
  • Simulations that treat generated probabilities as agent states should report consistency and factualSource: Conclusion · p. 11

Appendix A: Prompt Templates and Example Inputs

The prompts define each response as a probability that the exact displayed claim is true, with the participant’s prior treated as a position and peer reports as positions rather than evidence. They also specify controlled complementary requests, prompt variations, and feedback procedures.

Prompt templates

  • Responses are constrained to a JSON object with one field, p_claim_true, whose value is a number from 0 to 1 representing the exact displayed claim’s truth probability.The examples use 0.9 for strong generic support, 0.1 for strong refutation, and 0.5 for balanced information; they define output direction, not experimental answers.Source: Appendix A · p. 12
  • The controlled benchmark prompts treat the initial probability as the participant’s current position, peer reports as other participants’ positions, and source evidence as usable only when displayed.They instruct the model not to retrieve outside facts or infer hidden experimental metadata.Source: Appendix A · p. 12
  • Complementary requests keep the task facts, evidence block, and audience sentence identical outside the observation, which is filled for each orientation.When no evidence is present, both requests use the same replacement line stating that no source evidence is visible.Source: Appendix A · pp. 12–13
  • Prompt controls are applied separately: neutral wording rephrases the task paragraph, evidence prose rewrites the evidence block, and evidence reordering changes field order without changing content.The complement reminder is appended to the task paragraph and also serves as a repair arm.Source: Appendix A · p. 13
  • For real claims, the observation contains an AVeriTeC claim and source question-answer evidence; the public condition says the response will be shared with the peer group.This setup omits the task-facts block and removes agent labels from peer reports.Source: Appendix A · p. 13
  • In network feedback, agents see their own and neighbors’ complete prior-round reports with unchanged evidence; symmetrization feeds back R = (q0 + 1 − q1)/2 and displays 1 − R for the complement.The baseline and reminder arms feed back a single report, while the averaging arm feeds back the mean of two identical greedy calls.Source: Appendix A · p. 13

Appendix B: Construction of the Controlled Benchmark

The controlled benchmark uses worlds with two exhaustive, mutually exclusive descriptions at a fixed referent and time. Each pair concerns the same observed fact, and an explicit-negation control substitutes a negated original description for the complement.

Appendix B: Construction of the Controlled Benchmark

  • Both descriptions in a world concern the same observed fact, allowing one evidence block to verify either description and avoiding open-world existential–universal asymmetry.The passage contrasts this setup with the asymmetry between existential and universal claims in open-world statements.Source: Appendix B · p. 13
  • The explicit-negation control retains the original description and replaces its complement with a negation, such as “is not red” instead of “is blue.”Source: Appendix B · p. 13
  • Each controlled-benchmark world fixes a referent and time and defines two exhaustive, mutually exclusive outcomes.Table 6 groups the worlds into six structural families, with four worlds in each family.Source: Table 6 · p. 13, Appendix B · p. 13

Appendix C: Supplementary Results

Supplementary tests find that complement discrepancies persist across prompt controls, peer configurations, and explicit negation. Repair and real-claim results also show that consistency behavior differs by orientation and label.

Discrepancy by family

  • No single structural family produces the discrepancy: all six show it without evidence, while evidence-condition patterns differ between Qwen and Gemma.Qwen’s discrepancy with evidence is concentrated in event order, set membership, and threshold; Gemma’s is spread across all six families.Source: Appendix C · p. 13

Explicit negation

  • Explicit negation leaves the main discrepancy result intact, although changes vary by model and evidence condition.Without evidence, most changes are below 0.15 in magnitude; with evidence, Qwen changes in both directions, Llama’s remain small, and Gemma’s are mostly negative.Source: Appendix C · p. 13

Repeated sampling

  • With 80 draws per request, mean-report and independently paired-draw discrepancies coincide except for Llama with visible evidence.The orientation intervals jointly cover with probability at least 0.95.Source: Appendix C · p. 13

Prompt controls

  • No prompt control reduces complement discrepancy for all three models in both evidence conditions, and reductions do not consistently lower visible Brier loss.Rendering Qwen’s evidence as prose lowers its discrepancy but raises its loss.Source: Appendix C · p. 13

Peer count and report strength

  • All 54 peer-count and report-strength cells have positive discrepancy, with model-specific patterns across peer settings.Llama’s discrepancy generally falls as reports move toward 0.4/0.6; Gemma’s is highest with eight peers at 0.1/0.9.Source: Appendix C · p. 13

Complementary pair illustration

  • Figure 7 illustrates a complementary request pair and the criterion that its two reports sum to one.The figure marks changes between requests and labels the outputs q0 and q1.Source: Figure 7 · p. 14

Repair by orientation

  • After five rounds, symmetrization gives both orientations the same loss, while reminder effects differ by orientation and model.The reminder brings Qwen’s original-orientation loss to zero but leaves its complementary loss at 0.229; for Gemma it raises complementary loss.Source: Table 7 · p. 14, Appendix C · p. 14

Real claims by label

  • For refuted real claims, signed discrepancy is small despite large absolute discrepancy because pairs with opposite signs cancel.Figure 13 shows visible evidence mainly helps agents starting on the wrong side, with little protection for those starting on the right.Source: Table 8 · p. 14, Appendix C · p. 14

Appendix D: Implementation Details

The implementation appendix specifies model execution and simulation procedures, alongside figure encodings for supplementary analyses.

Model execution

  • Models run in llama.cpp with a 4,096-token context and at most 192 generated tokens, with greedy or temperature-0.2 sampling settings specified.Llama’s date is fixed across compared requests; Gemma uses a JSON schema and accepts exactly one finite number in [0, 1].Source: Appendix D · pp. 14–15

Supplementary figure encodings

  • Figures 8–10 encode discrepancy by structural family, change under explicit negation, and mean-report versus paired-draw discrepancy, respectively.Figure 8 distinguishes models and evidence conditions; Figure 10 uses circles and diamonds for its two discrepancy measures.Source: Figure 8 · p. 15, Figure 9 · p. 16, Figure 10 · p. 17

Simulation and uncertainty procedures

  • Network procedures specify initial probabilities, evidence targeting, randomized peer order, matched response seeds, and bootstrap resampling.Real-claim bootstrap resampling is within label; peer orders are averaged before absolute values, losses, or flip indicators are computed.Source: Appendix D · pp. 15–16

Feedback implementation

  • Feedback updates use four radius-two neighbors sorted by agent index, read the complete prior state, and pass reports on without rounding or smoothing.Source: Appendix D · p. 16

Supplementary figure encodings

  • Figures 11–13 encode prompt-control changes, peer-count and report-strength cells, and truth-directed probabilities across peer opposition and audiences.Figure 13 separates initial positions aligned with or opposed to the dataset label and shows 95% claim-bootstrap intervals.Source: Figure 11 · p. 18, Figure 12 · p. 19, Figure 13 · p. 20

Related papers

Loading 2610.32874v1…