Source-linked AI summary
Beyond Final Outcomes: Quantifying Claim-Level Information Dynamics in Multi-Agent LLM Interaction
Haocheng Jiang, Herun Wan, Yongyuan Peng, Minnan Luo
TL;DR
Final task outcomes alone do not show which claims become shared, what new claims arise, or what persists during multi-agent interaction. RIVER decomposes outputs into claims and evaluates convergence, exploration, and retention; strategy comparisons reveal different information-flow trade-offs.
Problem
Final outcomes cannot distinguish whether agents share the same information, introduce the same new claims, or retain claims over time.
Source: §1 · p. 1Method
RIVER evaluates claim-level convergence, exploration, and retention, including source preservation and the persistence of new claims.
Source: §1 · p. 1Results
Takeaways & Limitations
Collective information dynamics are better understood through underlying information-flow mechanisms than through final outcomes.
Source: §7 · p. 8Experiments use one generation model and five fixed personas, leaving the generality of the observed patterns across models and persona compositions unexamined.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
When LLM-powered agents interact, information is repeatedly shared, expanded, and forgotten, yet multi-agent evaluation typically focuses on final task outcomes. We study such interaction as claim-level information dynamics and propose RIVER, which decomposes agent outputs into atomic claims and evaluates three complementary dimensions: convergence, exploration, and retention. Convergence captures cross-agent claim correspondence and stance alignment; exploration captures semantic breadth and extension beyond the source; and retention tracks source preservation and the persistence of new claims across stages and lineages. We evaluate five interaction strategies on 40 controlled news-generation cases. Repeatedly pooled interaction promotes convergence but narrows the information space, whereas distributed relay yields broader, less shared information. Further comparisons show that method-specific constraints increase exploration while weakening convergence and retention. Our findings suggest that collective information dynamics are better explained by underlying information-flow mechanisms than by simple downstream performance.
1 Introduction
Final task outcomes do not reveal which claims become shared, what new information is introduced, or what persists during interaction. RIVER evaluates these information dynamics through convergence, exploration, and retention, and the reported strategy comparisons show distinct trade-offs across these dimensions.
1 Introduction
- Groups with the same final judgment may express different shared information, introduce different new claims, and retain different claims over time.Final accuracy, task success, or output similarity cannot distinguish these processes.Source: §1 · p. 1
- RIVER evaluates multi-agent information flow through convergence, exploration, and retention.The framework examines shared information, expansion beyond the source, and persistence of original and new claims.Source: §1 · pp. 1–2
- The study compares Free, NGT, Delphi, Policy Delphi, and Brainwriting across 40 news instances, separating Framework from Full settings for all but Free.The comparison examines whether information dynamics differ with communication structure or added behavioral constraints.Source: §1 · p. 2
- Repeatedly pooled interaction promotes convergence while concentrating information, whereas Brainwriting’s distributed relay preserves broader but less shared information.Adding method-specific constraints increases exploration but weakens convergence and retention.Source: §1 · p. 2
2 Related Work
Prior work studies LLM-agent capabilities, applications, and interaction structures, while benchmarks evaluate task performance and collaboration. This paper focuses on measuring how information becomes shared, expands beyond its source, and persists or disappears during interaction.
LLM-based Multi-Agent Systems
- LLM-based multi-agent systems use communication, role assignment, and mutual feedback across applications including reasoning, software development, content generation, and misinformation detection.Benchmarks commonly measure answer accuracy, issue resolution, or task success, alongside collaboration, coordination, and collective reasoning.Source: §2 · p. 3
Collective Intelligence and Structured Information Exchange
- Structured group-discussion methods include Nominal Group Technique, Delphi, Policy Delphi, and Brainwriting.Recent LLM studies use agents to simulate information propagation and support group ideation.Source: §2 · p. 3
- This work measures how information becomes shared, expands beyond the initial source, and is retained or lost throughout multi-agent interaction.Prior evaluation has also examined truth deviation during information propagation.Source: §2 · p. 3
3 RIVER: Claim-level Information Dynamics Evaluation
RIVER evaluates multi-agent interaction through claim-level convergence, exploration, and retention measures. Its convergence measure is reported to align with human assessments and outperform direct prompting in validation.
3 RIVER: Claim-level Information Dynamics Evaluation
- RIVER decomposes agent statements into atomic claims, compares corresponding claims across agents and stages, and aggregates those relations across convergence, exploration, and retention.The dimensions use Informational Unity, Unique Claim Count and Extension Rate, and Original News Retention, Survival Rate, and Claim Survival Length, respectively.Source: §3 · p. 3
3.1 Preliminary
- The method decomposes outputs into one-information-unit claims, judges source support and semantic correspondence, and uses 0.75 thresholds to classify extensions and effective correspondences.An extension introduces information beyond the source, such as a new fact or cause.Source: §3.1 · pp. 3–4
3.2 Convergence Dimension
- Informational Unity captures whether agents express corresponding specific information with aligned stances; directed matching accounts for information one agent states but another omits.Stance alignment decreases as stance scores diverge.Source: §3.2 · p. 4
3.3 Exploration Dimension
- Exploration measures the range of distinct claims and the share of claims extending beyond the source, grouping source-supported claims and extensions separately.The greedy clustering procedure prioritizes claims with more corresponding claims from other agents and disallows two claims from the same agent in one cluster.Source: §3.3 · pp. 4–5
3.4 Retention Dimension
- Retention combines source-information representation (OR), cross-stage preservation of extensions (SR), and how long extension information sets remain traceable (SL).Information sets connect corresponding claims across agents at the same position and extensions along specified propagation paths between adjacent positions.Source: §3.4 · p. 5
3.5 Validation
- Informational Unity aligns with human information-agreement assessments across three datasets, outperforms direct prompting on DBS-eval, and remains consistent across LLM evaluators.The reported evaluator comparison varies the semantic judge while keeping the decomposed claims fixed.Source: Table 1 · p. 6, Table 2 · p. 6, §3.5 · p. 6
4 Experimental Settings
The experiments compare five interaction strategies on controlled news discussions, separating each structured method’s core framework from its additional constraints. They use RIVER to evaluate information dynamics across these settings.
Evaluation
- Table 3 reports correlations of Informational Unity scores across LLM evaluators and describes RIVER as robust under different evaluators.The evaluator comparison fixes decomposed claims while varying the semantic judge.Source: Table 3 · p. 6
Discussion Setting
- The study evaluates five interaction strategies on 40 news cases, using the same generation settings across strategies.For each classical interaction method, it compares a Full setting with a Framework setting that removes additional constraints.Source: §4 · p. 6
Interaction Conditions
- The five strategies are Free, NGT, Delphi, Policy Delphi, and Brainwriting; the first three structured methods pool information, while Brainwriting uses distributed relay.Free is the control condition, without a prescribed interaction procedure or behavioral constraints.Source: §4 · p. 6
Discussion Setting
- Each case runs four independent cycles of four stages, followed by a final private summary stage.Each cycle starts independently from the same source news.Source: §4 · p. 6
Evaluation Metrics
- The main evaluation uses gemini-2.5-flash-lite, with each news case treated as an independent statistical unit.Aggregate metrics are reported in the main text, with computation rules and full trajectories provided in appendices.Source: §4 · p. 6
5 Results
Repeated pooling tends to increase convergence while narrowing the information space, whereas Brainwriting relay maintains broader but less-shared information. Additional method-specific constraints increase exploration while weakening convergence and retention.
Overall results
- Pooling increases information sharing while narrowing the information space, whereas Brainwriting relay maintains broader but less-shared information.Table 4 reports aggregate metrics across 40 cases and five strategies; Figure 2 plots U, CC, and OR trajectories.Source: §5 · pp. 6–7, Table 4 · p. 7, Figure 2 · p. 8
Convergence
- Under Framework settings, U reaches 0.626 on average for pooling methods versus 0.286 for Brainwriting at cycle-end synthesis or revision.Both groups begin near 0.36 after independent generation: 0.361 for pooling and 0.362 for Brainwriting.Source: §5 · p. 7
Exploration
- Framework settings show pooling CC reductions of 13.5%, 11.0%, and 10.5% of Stage-1 CC across successive stages, while Brainwriting changes by +16.4%, +2.5%, and −6.1%.Moving from Full to Framework reduces aggregate CC by 26.5% across four strategies, while ER decreases by 3.0% on average.Source: §5 · p. 7
Retention
- Under Full settings, OR declines by 34.6% from Stage 1 to Stage 4 across pooling methods, compared with 11.4% for Brainwriting.Moving from Full to Framework raises the proportion of extension lineages surviving at least three positions by 6.0 percentage points on average.Source: §5 · p. 7
Trade-offs
- Across nine conditions, U–CC correlations average −0.815 by Pearson and −0.825 by Spearman, while CC–SR averages −0.710 by Pearson.Brainwriting’s distributed structure allows information to persist locally without becoming widely shared.Source: §5 · p. 7
6 Discussion
The discussion attributes different information patterns to interaction structures and behavioral constraints. It argues that claim-level information dynamics reveal differences that final outcomes alone do not capture.
Information flow
- Repeated pooling promotes convergence through repeated sharing, while distributed relay encourages exploration along separate paths.Brainwriting’s longer claim lineages indicate that information can persist along a relay path without becoming widely shared.Source: §6 · p. 8
Behavioral constraints
- In limited output space, differentiated views or new details may displace shared and source information, increasing exploration while weakening convergence and retention.The discussion identifies behavioral constraints as shaping content selection.Source: §6 · p. 8
Contribution
- RIVER examines how information is shared, extended, and retained at the claim level, including source information that can reappear.The authors say the framework helps researchers examine information dynamics and understand how interaction settings organize information within groups.Source: §6 · p. 8
7 Conclusion
RIVER analyzes multi-agent information flow through convergence, exploration, and retention. Across the evaluated strategies, interaction structure and additional behavioral constraints correspond to distinct information-dynamic trade-offs.
7 Conclusion
- Pooled interaction promotes convergence but narrows the information space, while distributed relay preserves broader, less shared information.The comparison spans five interaction strategies.Source: §7 · p. 8
- Adding method-specific constraints increases exploration while weakening convergence and retention.This pattern appears in comparisons between Framework and Full settings.Source: §7 · p. 8
- The findings suggest that underlying information-flow mechanisms better explain multi-agent behavior than final outcomes alone.Source: §7 · p. 8
Limitations
The experiments have boundaries in model and persona coverage, as well as in interaction duration. These limits leave generality across agents and sustained interactions unexamined.
Limitations
- The experiments use one generation model and five fixed personas, leaving the effects of other models and persona compositions unexamined.The authors identify further experiments with other models and agent profiles as a way to assess generality.Source: Limitations · p. 9
- The observed patterns’ generality across models and agent profiles remains to be assessed.Source: Limitations · p. 9
- Independent Cycles do not carry discussion history or private memory forward, so the experiments do not capture accumulation across sustained interactions.Source: Limitations · p. 9
Ethical Considerations
The study uses public news and synthetic agent profiles, and generated discussions are not verified reporting. Its human validation reuses existing datasets to assess information sharing and agreement, not individual participants.
Ethical Considerations
- Generated discussions may include unsupported explanations, predictions, or proposals and should not be treated as verified reporting.Source: Ethical Considerations · p. 9
- Human validation reuses existing datasets without recruiting new participants and evaluates information sharing and agreement rather than individual participants.Source: Ethical Considerations · p. 9
- The framework measures expressed information: high convergence does not imply factual correctness, and low convergence does not imply undesirable disagreement.Source: Ethical Considerations · p. 9
A Implementation Details
The implementation compares fixed-agent discussions across interaction strategies and settings, decomposes outputs into claims, and scores their stances and semantic relations. It uses staged prompts and automated semantic judgments to evaluate information dynamics.
A.1 Discussion Schedule
- Each of 40 cases uses five fixed personas across four independent four-stage Cycles, followed by a private Stage 17 summary with no exchange between Cycles.Each Cycle starts from the same news, objective, and persona; agents privately integrate their own final Cycle states at Stage 17.Source: §A.1 · p. 11
A.2 Interaction Strategies
- Free is the control; Full retains each method’s complete constraints, while Framework removes additional constraints but preserves core interaction structure.This comparison applies to NGT, Delphi, Policy Delphi, and Brainwriting.Source: §A.2 · p. 11
A.3 Atomic Decomposition and Stance Scoring
- Public outputs are evaluated at each Stage, while private reasoning is excluded; atomic claims preserve proposition details and receive continuous stance scores from rejection to support.The stance scale runs from 0 for rejection through 0.5 for neutral or uncertain to 1 for strong support; reported attribution is distinguished from endorsing the proposition.Source: §A.3 · p. 11
A.4 Prompt Templates
- Shared generation prompts use Stage-specific instructions, while Delphi and Policy Delphi use separate moderator prompts to compile anonymous feedback between Stages.At later Stages, the original source is not repeated; Stage 17 summaries integrate each agent’s own Cycle-end states.Source: §A.4 · pp. 11–12
A.5 Runtime Configuration
- Generation and Stage-17 summaries use temperature 0.4, moderator calls 0.1, and decomposition and evaluation calls 0.0; other settings remain at API defaults.Only outputs passing parsing and schema validation are stored in the evaluation cache.Source: §A.5 · p. 13
A.6 Semantic Judgments
- Semantic judgments score source support, claim correspondence, and directional preservation, with a 0.75 threshold and a minimum passing score of 0.8.An added substantive proposition slot cannot be ignored when judging correspondence.Source: §A.6 · p. 13, §A.4 · p. 13
B Metric Details and Robustness
The appendix specifies how RIVER tracks claims across stages and aggregates its metrics. It also describes claim clustering and reports the metric trajectories across experimental conditions.
B.1 Evaluation Positions and Propagation Paths
- SR follows adjacent positions along each agent’s path, except in Brainwriting, while SL counts positions within an interaction unit; Cycles have no temporal links.The evaluated paths include each Cycle endpoint to Stage 17, where agents privately summarize their own final states.Source: §B.1 · p. 14
B.2 Claim Clustering
- Claim clustering processes claims with more cross-agent correspondences first and admits a claim only when it corresponds to every cluster member and no member is from the same agent.Among eligible clusters, the procedure favors larger clusters, then higher minimum and mean correspondence; otherwise it creates a singleton, without merging existing clusters.Source: §B.2 · p. 14
B.3 Aggregate Values
- U, CC, ER, OR, and SR are averaged equally across 40 news cases, while SL aggregates extension-lineage counts across 160 case–Cycle units per condition.The five agents within a case are not treated as independent statistical samples; SL is normalized as defined in the Method section.Source: §B.3 · pp. 14–15
B.4 Evaluation Pipeline
- Algorithm 1 constructs information units for each case–Cycle unit using its ordered evaluation positions and tracks assigned claim instances.The passage identifies the algorithm’s inputs and local sets but does not provide the full pipeline steps here.Source: §B.4 · p. 15
C Additional Experimental Results
- Figure 3 displays extension rate across 17 stages, survival rate across 16 transitions, and lineage survival proportions for d = 1, . . . , 5.The panels compare Full and Framework; cycle boundaries are not connected.Source: C Additional Experimental Results · p. 15
C.1 Numerical Values of the Metric Trajectories
- Tables 9–14 report metric means across experimental conditions, with U, CC, ER, and OR covering stages and SR covering evaluated transitions.The tables include four within-Cycle position averages and the Stage-17 summary; Table 14 reports cumulative lineage survival proportions, not mean SL.Source: §C.1 · p. 15, C Additional Experimental Results · p. 15
C.2 Mean Correlations Among the Six Metrics
The appendix describes how metric correlations and human-validation comparisons are calculated across several datasets. It also specifies the construction of the news inputs used in the experiments.
C.2 Mean Correlations Among the Six Metrics
- Table 15 reports mean Pearson and Spearman correlations across nine conditions, using stage-level, transition-aligned, or case-level summaries depending on the metric pair.The coefficients are averaged across setting-specific correlations rather than computed from pooled observations.Source: §C.2 · p. 15, Table 15 · p. 21
D Human Validation Details
- Human-validation comparisons use Gemini 2.5 Flash Lite for DBS-eval and Polis, while DELPHISTAR Unity comes directly from participant ratings.The three datasets provide reference scores of information sharing or agreement based on participants’ expressions.Source: D Human Validation Details · p. 21
D.1 DBS-eval
- DBS-eval validation uses ten topics, treating each topic’s complete human-written summaries as participant outputs and excluding gold SCUs and reference scores from evaluator inputs.The Simple baseline gives one holistic Unity score per topic; the comparison with formal U uses the same reference scores.Source: §D.1 · p. 21
D.2 DELPHISTAR
- DELPHISTAR validation uses 33 Round-3 items, treating valid participant importance ratings as responses and defining agreement through pairwise stance agreement.The reference is the proportion of valid ratings of 6 or 7, with missing ratings excluded from the denominator.Source: §D.2 · pp. 21–22
D.3 Polis
- Polis validation uses 16 derived cases of 12 participants, converting agree votes to retained statements and disagree votes to explicit disagreement expressions.High and Low cases are defined by distinct agreement and vote-coverage thresholds, respectively.Source: §D.3 · p. 22
D.4 Correlation Analysis
- Correlation analysis compares continuous Unity scores with dataset reference scores, using binary High/Low proxies for Polis and continuous references for DBS-eval and DELPHISTAR.The sample sizes are ten topics, 33 items, and 16 derived cases, respectively.Source: §D.4 · p. 22
E News Sources
- The 40 experimental news inputs come from 20 Reuters reports, each used to create one Event-only input and one Event+Seed input.The constructed inputs concisely paraphrase the reports’ factual content.Source: E News Sources · p. 22