PaperlayerView exact revision on arXiv

Source-linked AI summary

Copies or Sources? Measuring How LLM Aggregators Count Restated Evidence in Multi-Agent Systems

Jianxin Gao, Runze Li, Tianyi Yu, Liangwei Ren, Bohan Chen, Zining Wang

arXiv:2610.06192v1cs.AI

TL;DR

LLM aggregators must decide whether repeated messages represent independent evidence, although multi-agent systems routinely restate observations. The paper measures the weight of copies across controlled logs, web documents, and agent-written logs, and finds that copy inflation can affect decisions; merging copies at the aggregator or avoiding restatements in the protocol reduces it.

  • Problem

    LLM multi-agent systems routinely restate observations, raising the question of whether aggregators count distinct sources or repeated statements.

    Source: §1 · p. 1

  • Method

    The paper converts reported probabilities into independent-reading units to measure copy weight and implied decisions across controlled logs, web documents, and agent-written logs.

    Source: §1 · p. 2

  • Results

    Across the three testbeds, models count forwarded copies as evidence, and one reading stated three times crosses the commitment boundary in 4.7%–39.8% of controlled logs, depending on the model.

    Source: §4.4 · p. 5, §7 · p. 8

  • Takeaways & Limitations

    Merging copies at the aggregator or preventing restatements in the protocol removes most inflation while preserving evidence from genuine observers.

    Source: §1 · p. 2, §7 · p. 8

    The controlled logs assume correct, explicit provenance and resolvable references; forged identifiers, altered relays, or shared source failures require an oracle that models those dependencies.

    Source: §7 · p. 8

Abstract

from arXiv · show

Multi-agent systems built on large language models (LLMs) restate observations as a matter of course: relays forward them, shared boards repeat them and discussion rounds echo them. An aggregator that pools such messages should count sources, not statements. We convert a reported probability into units of independent readings, which assigns every restatement a copy weight, 0 for an aggregator that counts sources and 1 for one that counts every statement, and yields the implied decision under any cost structure. Three testbeds hold the evidence fixed and vary how it is restated: message logs with an exact Bayesian oracle, web documents with appended copies, and logs written by LLM agent teams under four communication protocols. Across four models from three providers, a forwarded copy counts for 0.06 to 0.42 of a new reading, mostly because some replies count every statement. On 5% to 40% of logs that state one reading three times, the reported belief implies an early commitment that the oracle never makes. The models that count copies least and most on controlled logs do so on web copies and agent-written logs as well. A one-paragraph declaration of what a copy contributes brings the copy weight on controlled logs to 0.08 or less. A rule that has agents refer to readings instead of restating them cuts belief-implied early commitment from 11.2% to 1.1% and preserves genuine corroboration.

1 Introduction

LLM multi-agent systems routinely relay and restate observations, leaving aggregators to decide whether repeated messages represent new evidence. This paper measures how much copies count, whether that affects decisions, and whether the model or protocol controls the effect.

1 Introduction

  • The same observation can appear as several messages in relay chains, shared memories, discussions, hierarchies, and near-duplicate retrieval.The aggregator may be a judge, planner, router, or final answerer.Source: §1 · p. 1
  • A restatement adds no evidence under the source-counting norm, but LLM pipelines receive provenance as text and may not act on it.Repeated content is known to sway LLM answers.Source: §1 · p. 1
  • The paper measures restatements in units of independent readings, assigning copy weight 0 when copies count as sources and 1 when every statement counts as a new reading.The belief-based measure also yields implied decisions under different cost structures.Source: §1 · p. 1
  • Three testbeds hold evidence fixed while varying restatements: controlled logs with an exact Bayesian oracle, copied web documents, and LLM-agent logs under four protocols.Four models from three providers answer the same conditions.Source: §1 · p. 2
  • The paper reports that forwarded copies receive partial weight across models and that copy-related belief inflation can push single-reading logs past a commitment boundary.It examines merging copies at the aggregator or preventing restatements in the protocol.Source: §1 · p. 2

2 Measuring How Much a Copy Counts

The section defines a source-counting posterior and measures reported beliefs as independent readings, then uses that measure to connect copy counting with decisions and protocol restatement counts.

2.1 Aggregation with restated evidence

  • The source-counting oracle forms the posterior from distinct readings, regardless of how often the log restates them.A message restates a reading when its value can be recovered; a pointer alone does not restate it.Source: §2.1 · p. 2

2.2 Evidence in readings and the copy weight

  • Reported probabilities are converted into equivalent independent readings, and copy weight measures the average extra evidence per restatement relative to independent evidence.The source-counting and every-statement extremes correspond to weights 0 and 1; the measure also distinguishes agreeing independent readings from repeated ones.Source: §2.2 · p. 2

2.3 From beliefs to decisions

  • The commit boundary expresses the evidence threshold implied by the cost of querying, so reported beliefs can be assessed for commitment before the oracle would commit.On agreeing-reading logs, early commitment occurs when oracle evidence is below the boundary but reported evidence is above it.Source: §2.3 · p. 2

2.4 Separating the model from the protocol

  • Under copy-count sufficiency, protocol effects on excess evidence depend on the number of copies, while a protocol that states each reading once leaves copy weight with no copies to affect.The paper predicts that the weight follows statement count rather than record count, and that copy weight summarizes how excess evidence changes with copy count.Source: §2.4 · p. 3, Figure 2 · p. 3

3 Testbeds and Models

The experiments compare controlled message logs, web documents, and agent-written logs while varying how a fixed source of evidence is restated. Paired conditions and an exact source-counting reference support comparisons across the testbeds.

3 Testbeds and Models

  • T1 uses 1,000 controlled sensing items across five domains, five reliabilities, two polarities, and 20 instances, rendered in 27 forms including copies, pointers, padding, and independent readings.The testbed also includes dose series and conflict probes; the exact oracle supplies source-counting evidence.Source: §3 · p. 3
  • T2 uses 500 ConflictingQA items to compare appended exact, republished, or summarized copies of a focal web document with two distinct focal-answer documents.Because web documents lack an exact posterior, the comparison measures the copy-induced log-odds shift relative to the shift from a distinct document.Source: §3 · pp. 3–4
  • T3 tests four agent communication protocols on 500 world configurations with one or three observers, running logs both with and without a reference rule.The aggregator can answer directly, with a declaration, or from a list of distinct readings; an annotation pass labels restatements.Source: §3 · p. 4
  • All four models answer paired conditions on the same items, and pre-specified contrasts are significant after correction while no-value controls are equivalent to the single report within 0.10 readings.The models are GPT-6 Luna, GPT-6.1 Sol, Gemini 3.8 Flash, and DeepSeek V4.1 Flash.Source: §3 · p. 4
  • On controlled logs, single-report evidence is 0.99–1.00 readings and three-independent-reading evidence is 2.83–2.96, supporting the scale’s endpoint validity.Two additional replies move condition means by at most 0.07 readings.Source: §3 · p. 4

4 Copy Counting in Three Testbeds

Across controlled logs, web documents, and agent-written logs, aggregators treat restatements as evidence. On controlled and agent-written logs, this can make one reading imply an early commitment that the oracle does not make.

4.1 Every model counts a copy as part of a reading

  • Two forwarded copies raise reported evidence to 1.12 (Sol), 1.42 (Gemini), 1.55 (Luna), and 1.84 (DeepSeek) readings; three statements can outvote two opposing readings.Copy weights are 0.06 (Sol), 0.21 (Gemini), 0.27 (Luna), and 0.42 (DeepSeek); at eight statements, models report 1.30 to 2.53 readings.Source: §4.1 · pp. 4–5

4.2 The weight follows restated values

  • Records that point to the original or pad the log leave evidence at one reading, while packed relay traces and copies raise it.Three independent readings give 2.82 to 2.96 readings across formats, with or without provenance labels.Source: §4.2 · p. 5

4.3 Copy counting is mostly all or nothing

  • 47% (DeepSeek) to 91% (Sol) of reports on Copies stay within 0.15 readings of the one-reading posterior; full-counting reports carry 71% to 84% of excess evidence.Asked the same log three times, models count copies in some replies but not others for 16% to 42% of logs.Source: §4.3 · p. 5

4.4 Copies tip decisions

  • 4.7% (Sol), 19.1% (Gemini), 24.6% (Luna), and 39.8% (DeepSeek) of three-copy logs cross the two-reading commit boundary, though the oracle never commits.When models choose actions themselves, they commit on Copies 6 to 35 percentage points more often than on the single report.Source: Figure 3 · p. 5, §4.4 · p. 5

4.5 Web copies count too

  • 0.59 to 0.93 log-odds is the shift from two distinct supporting documents; exact duplicates count for ρ = 0.14 to 0.32 of that shift, versus 0.40 to 0.71 for summaries.Republished copies count for ρ = 0.26 to 0.49; the farther a copy departs from its original, the more it counts.Source: §4.5 · p. 6, Table 2 · p. 6

4.6 Agent teams restate by default

  • 11.2% of pooled one-observer agent logs imply early commitment, reaching 34% for DeepSeek with six or more restatements.Non-observers restate 71% of messages; evidence rises per restatement by 0.04 (Sol), 0.11 (Gemini), 0.12 (Luna), and 0.22 (DeepSeek), with Sol lowest and DeepSeek highest across testbeds.Source: Figure 5 · p. 6, §4.6 · pp. 6–7

5 Merging Copies Before the Decision

The paper compares merging restated evidence at the aggregator with preventing restatements in the protocol, then tests whether these fixes preserve genuine corroboration. The protocol rule lowers early commitment while keeping the distinction between one and three observers.

5.1 Merging at the aggregator

  • On controlled logs, declarations reduce w3 to 0.08 or less; on web summaries, copy weight remains 0.19 to 0.47.Extracting distinct readings first brings one-observer agent logs to 0.95 to 0.99 readings, with no early commitment in 2,000 such logs per model.Source: §5.1 · p. 7

5.2 Merging in the protocol

  • The reference rule lowers restating messages from 71% to 10%, but hierarchical summaries still restate readings, with median count 2.In the hierarchy, this leaves evidence inflated by up to 0.22 readings for DeepSeek.Source: §5.2 · p. 7
  • 11.2% to 1.1%: the reference rule cuts pooled early commitment while preserving 0.92 to 0.99 discrimination between three and one observers.Under the rule, no relay loses a reading.Source: §5.2 · p. 7

5.3 Genuine corroboration survives merging

  • With three genuine observers, merged evidence stays within 0.06 readings under the reference rule and 0.02 under the declaration for three models.With actions, the declaration keeps commitment on three independent readings at 89% to 99%.Source: §5.3 · pp. 7–8

6 Related Work

Prior work studies information exchange, repetition effects, and dependence in aggregation. This paper measures repetition in units of observations, connects it to decisions, and traces restatements to agent protocols.

Information flow in LLM multi-agent systems

  • Multi-agent research studies debate, voting, communication topology, and correlated errors, while this paper tests whether the component reading messages counts each source once [34] [57].Related work also examines false majorities in shared memories and assumptions about agents’ interchangeable roles [43] [21].Source: §6 · p. 8

Repetition, conflict and sources in LLM context

  • Repeated retrieved evidence can shift model answers, and paraphrases of one argument flip answers more often than distinct supporting documents.This paper expresses repetition in observation units, follows it to decisions with explicit costs, and traces it to agent protocols.Source: §6 · p. 8

Aggregation under dependence

  • Prior approaches handle dependent sources with explicit models or rules, whereas this paper measures whether an LLM honors dependence conveyed only in text [65].It also measures deviations from Bayesian posteriors in units of independent observations.Source: §6 · p. 8

7 Conclusion

The paper concludes that LLM aggregators give restated evidence nonzero weight, while avoiding restatements can prevent most inflation. The controlled logs assume honest, explicit provenance; forged identifiers and shared failure modes require an oracle that models dependencies.

7 Conclusion

  • The conclusion recommends having agents refer to readings already in the log; controlled-log conclusions assume honest provenance, while forged identifiers or shared failure modes require a dependency-aware oracle.Merging copies before the decision removes most inflation, whether the aggregator merges them or the protocol avoids creating them.Source: §7 · p. 8

A Overview; A.1 Models and settings

  • Each episode produces one recorded value, and the study pairs contrasts within items; four named models are used, with their call settings listed separately.The models are Luna, Sol, Gemini and DeepSeek.Source: A Overview · p. 11, §A.1 · p. 11

A.2 Replies and parsing

  • Replies are parsed from a required final line; unparsable or truncated replies are retried twice, then counted as missing if they still fail.Probability outputs outside [0, 1] count as failures.Source: §A.2 · p. 11

B.1 World, oracle and items

  • The controlled oracle uses a binary hidden state, a 1/2 prior and conditionally independent sensor readings; the study constructs 1,000 items across domains, reliabilities and polarities.The oracle sums evidence over distinct readings identified by reading_id.Source: §B.1 · p. 11

B.2 Records and forms

  • The controlled testbed includes 27 log forms, with conflict-probe ordering varied by item parity and records within one block ordered by time and sensor identifier.Forwards point to the report received by sensor 1’s original report in display order.Source: §B.2 · p. 11, Table 4 · p. 13

C.1 Items

  • The web testbed selects up to 500 items from controversial yes-or-no questions, using same-stance documents from distinct domains and an opposite-stance document.Items are deterministically ordered, and bootstrap intervals resample questions because items from one question share documents.Source: §C.1 · p. 11

C.2 Copies and conditions

  • Web conditions compare duplicate, republished-copy, summary and distinct-document versions, each with and without a declaration.Republished copies retain the source title and text with a republication line; summaries are generated by Sol and stance-checked by Gemini.Source: §C.2 · pp. 11–13

D.1 Configurations, roles and protocols

  • The agent-team testbed crosses one- and three-observer evidence structures with four protocols and two communication-rule settings.The episodes are also scored under no-note, declaration, extract-first and true-list aggregator arms.Source: §D.1 · p. 13

D.2 Annotation

Transcript annotations identify which messages state readings and which are claims, producing counts used to characterize restatement. Three-observer results include only complete logs, and a sampled annotation check found high agreement.

D.2 Annotation

  • Sol annotates each transcript once, recording readings stated by each message and whether it is flagged as a claim.The derived counts include messages stating a reading, claim messages, and non-observer messages that restate readings.Source: §D.2 · p. 13
  • A three-observer log is complete only when every observer’s reading is stated at least once; only complete logs are used for three-observer results.The share of incomplete logs is reported by protocol.Source: §D.2 · p. 13
  • In a 240-message annotation check, the two authors agreed on all messages about both reading statements and CLAIM labels.The annotation matched the authors’ reading-statement labels on 98.8% of messages.Source: §D.2 · pp. 13–14
  • The protocol schedules define each turn as one call to the worker model and specify what the speaker sees.The schedules are listed in Table 5.Source: Table 5 · p. 14

E Measures and tests

The measures express reported beliefs in independent-reading units and distinguish copy weight from discrimination. The section also specifies clipping, control criteria and statistical procedures.

E Measures and tests

  • Probabilities are clipped to [0.001, 0.999], limiting measurable evidence; results that depend on eight readings are also reported for q≤0.70.At q=0.75 and q=0.80, eight readings cannot be registered under this clipping limit.Source: E Measures and tests · p. 14
  • Reported evidence converts probability to reading units; copy weight measures added evidence per copy, while discrimination contrasts independent readings with copies.The synthesis uses T1 copy weight, the T3 slope and per-copy-type T2 ρ.Source: E Measures and tests · p. 14
  • Belief-implied early commitment is the share of one-reading reports above a decision boundary, while unnecessary queries measure reports at or below it on independent evidence.The section also defines measures for conflict probes, actions, the T3 slope, the reference rule and T2 copy effects.Source: E Measures and tests · p. 14
  • Null controls count as equivalent only when the 90% paired interval lies within ±0.10 readings; paired contrasts require both values to be present..Source: E Measures and tests · p. 14
  • The pre-specified test families use paired Wilcoxon tests, exact McNemar tests or Page’s trend test, with bootstrap intervals over items, configurations or questions.Holm correction is applied within each test family.Source: Table 6 · p. 15

F Full results

The full-results section describes the interval procedure and how zero observed events are handled; it points to a separate section for pre-specified tests.

F Full results

  • Reported intervals are 95% percentile bootstrap intervals based on 10,000 resamples over items, configurations or questions.The resampling unit varies by testbed.Source: F Full results · p. 16
  • No interval is shown when an event has zero observations; with n independent units and no event, the one-sided 95% upper bound is about 3/n.The bootstrap cannot bound a rate that is zero in the sample.Source: F Full results · p. 16
  • The pre-specified tests are described in Section F.5.Source: F Full results · p. 16

F.1.1 Copy weight, discrimination and conflict probes (Sections 4.1

The appendix documents controlled-log measures, decision analyses, note and reasoning comparisons, web-evidence shifts, and agent-team transcripts and restatement patterns.

F.1.1 Copy weight, discrimination and conflict probes (Sections 4.1

  • Tables 7–10 report controlled-log evidence, copy weights, discrimination, conflict probes, and breakdowns by reliability and domain.Figure 1 plots dose series; Table 8 compares forms that repeat one reading with independent readings.Source: §F.1.1 · p. 16

F.1.2 Full counting and repeated replies (Section 4.3 of the main paper).

  • Tables 11–14 describe full-counting replies, clipping at 0.999, eight-statement results, replicate samples, and repeated replies to the same log.Table 11 also reports the share of excess evidence carried by reports that count every statement.Source: §F.1.2 · p. 16

F.1.3 Decisions (Section 4.4 of the main paper).

  • Tables 15–17 and Figure 2 report belief-implied commitment across boundaries, median-of-three replies, emitted actions, and policy-control comparisons.The tables also cover unnecessary queries and the correspondence between emitted actions and belief-implied commitment.Source: §F.1.3 · p. 16

F.1.4 Notes and reasoning effort (Section 5.1 of the main paper).

  • Tables 18–21 compare notes and reasoning settings using changes in reported evidence, copy weight, copied-minority responses, and model reasoning controls.The note comparisons include Claims and Packed; Sol and Gemini cannot switch reasoning off.Source: §F.1.4 · p. 16

F.2 T2 Web evidence (Sections 4.5 and 5.1 of the main paper)

  • Table 22 reports how adding two documents shifts answers toward a focal web document, copy-type weights, and declaration-related shift changes.Intervals resample the 299 questions behind the 500 items.Source: §F.2 · p. 16

F.3 T3 Agent teams (Sections 4.6 and 5.1 to 5.3 of the main paper)

  • Tables 23–29 describe agent-team transcripts, restatement shares, early commitment, reported evidence, discrimination, and restatement-count analyses across protocols and models.Three-observer results use complete logs; only the relay chain without the rule produces incomplete logs.Source: §F.3 · p. 16

F.4 Copy weight on one scale (Section 4.6 of the main paper)

This appendix section places controlled-log, web-copy, and agent-team copy measures on a common scale and documents the statistical tests and controls used.

F.4 Copy weight on one scale (Section 4.6 of the main paper)

  • Table 30 collects copy-weight measures for controlled logs, agent-team restatements, and web-document copy types in units of an independent source.Figure 1 plots reported evidence against the number of statements of one reading and independent readings, with oracle lines and 95% intervals.Source: §F.4 · p. 16, Figure 1 · p. 17
  • Figure 2 plots belief-implied early commitment against the decision boundary, and Figure 3 plots evidence and commitment against agent restatement count.Figure 2 marks the action-channel boundary at e_b=2; Figure 3 pools protocols for one observer without a note.Source: Figure 2 · p. 18, Figure 3 · p. 19

F.5 Tests

  • Table 31 lists pre-specified contrasts and corrected p-values, while Table 32 tests whether null controls are equivalent to Single within ±0.10 readings.The tests include paired Wilcoxon, exact McNemar, Page trend, and cluster-bootstrap procedures.Source: §F.5 · p. 18

DeepSeek V4.1 Flash

The model-specific appendix reports controlled-log commitment and note-related changes in reported evidence for DeepSeek V4.1 Flash, GPT-6 Luna, and GPT-6.1 Sol.

DeepSeek V4.1 Flash

  • 34.1% of DeepSeek V4.1 Flash Copies (prose) logs imply early commitment, compared with 3.6% for Independent logs.The corresponding Copies (prose, no labels) and Packed rates are 43.0% and 46.7%.Source: DeepSeek V4.1 Flash · p. 22
  • For DeepSeek V4.1 Flash, Declaration: records changes mean evidence on Copies by −0.60 against no note.The same table reports changes of 0.01 on Independent and −0.47 on Copied minority.Source: GPT-6 Luna · p. 24, DeepSeek V4.1 Flash · p. 24

GPT-6.1 Sol

Gemini 3.8 Flash

This section describes the procedures and annotation approach used to study message-log, web-document, and agent-team testbeds, including how reported readings are distinguished from restatements. It also defines the oracle decision boundary and a reference rule for agents, illustrating how restatements versus references affect the number of readings attributed to an observation.

G.1 T1: System and user prompts

  • The T1 oracle queries after one reading regardless of restatements and commits after three agreeing independent readings.The sensor reliability is chosen so the commit boundary corresponds approximately to two-reading posterior evidence, ê_b ≈ 2.Source: §G.1 · p. 34
  • T1 notes distinguish lossless forwarded copies and sensorless relays from counting instructions, while placebo notes describe irrelevant display or delivery details.The declaration-and-instruction condition combines the records declaration with the counting instruction.Source: Table 33 · p. 34, §G.1 · p. 35

G.2 T1: Record templates

  • T1 varies record formats, including explicit references, equality claims, packed relay traces, heartbeats, unlabeled forwards, and prose reports.A reference points to the earlier report without repeating its value; packed records state values in the trace and again at the top level.Source: §G.2 · p. 35

G.3 T1: A complete episode

  • A worked T1 episode converts a probability of 0.78 into 1.49 implied readings and a copy weight of about 0.25, against an oracle value of one reading.The example contains one sensor reading and two forwards of that same reading.Source: §G.3 · p. 36

G.4 T2: Evidence, paraphrase and stance-check prompts

  • T2 asks for probabilities from web documents and tests summaries rewritten by another outlet, regenerating summaries whose stance differs from the original.The prompt tells the model that duplicates, republished copies, and summaries are not independent sources.Source: §G.4 · pp. 36–37

G.5 T3: Worker prompts

  • The T3 reference rule tells workers to state each reading in full once and refer to an existing report without repeating its value or agreement.The aggregator conditions include no note, a declaration that sensorless agents cannot add readings, and an extract-first two-step procedure that parses distinct readings before estimating probability.Source: §G.5 · p. 37, §G.6 · p. 38

G.7 T3: Annotation prompt

  • The T3 annotation prompt labels which readings each message states, counting explicit values and equality assertions but not bare references.The aggregator receives a rendered message-bus log; the roster can specify one or three observers.Source: §G.6 · p. 37, §G.7 · p. 38
  • The T3 annotation prompt records each message’s sensor readings or “none” in order and flags claims of verification or more supporting sources than the log reports.It requires one line per message and identifies sensors by their roster identifiers.Source: §G.7 · p. 38

G.8 T3: A complete episode

  • In a Relay example, four messages restating one OD-85=STORM reading yield μ1 = 4, while a reference-style chain yields μ1 = 1.The example describes four aggregator calls on the log, including no note, a declaration, and two extract-first steps.Source: §G.8 · p. 39

Related papers

Loading 2610.06192v1…