From Process Loss to Assembly Bonus: Human-Grounded Diagnosis of Multi-Agent LLM Collaboration

1 Introduction

• The study tests whether LLM deliberation improves solution quality or reproduces human-group process losses such as conformity, premature consensus, and lost minority signals.The motivation is that final accuracy alone cannot distinguish correction of wrong agents from process loss among initially correct agents.Source: §1 · p. 1
• Matched human and LLM chats preserve solo, intermediate, and final answers, using intent-based floor-taking and voluntary exit to track deliberation trajectories.The study begins with Wason-style reasoning and tests whether patterns extend to analogical, abductive, and analytical tasks.Source: §1 · pp. 1–2, Figure 1 · p. 1
• LLM groups show stronger majority-following, lower unique-information realization, and earlier lock-in than humans, despite sharing the assembly-bonus asymmetry.Discussion improves the average member more often than it preserves the best initial member.Source: §1 · p. 2
• Inference-time interventions modestly improve collective outcomes but do not remove the coordination bottleneck, which also appears across additional reasoning domains.The authors caution that similar outcomes do not establish human-like deliberative mechanisms.Source: §1 · p. 2

2.1 Simulation setup

• Human chats average 3.16 participants and use a two-stage solo-to-group protocol; LLM agents first answer solo, then deliberate in visible chat with intermediate updates and voluntary exit.LLM groups contain three agents, with homogeneous or heterogeneous mixes of Qwen3-4B, Qwen3-14B, Qwen3-32B, and gpt-oss-20b; turn length varies across open-exit and forced-stay conditions.Source: §2.1 · p. 2

2.2 Assembly bonus and process loss

• Mean-improvement and best-member measures compare final outcomes with the average and strongest initial scores; recovery and drift track opposite correctness transitions.Recovery captures initially wrong-to-correct movement, while drift captures movement away from an initially correct state.Source: §2.2 · pp. 2–3
• Discussion yields gains over private starting points in only half of cases, while the gap between weak and strong assembly bonus is sharpest for humans.LLMs often have higher strong assembly bonus than humans, whereas human collaboration primarily corrects weaker initial answers.Source: §2.2 · p. 3
• Qwen-14B homogeneous groups combine strong correction of initially wrong agents with frequent movement away from initially correct states, unlike the more conservative human groups.The authors interpret this as a volatile interaction regime rather than stable truth-wins dynamics.Source: §2.2 · p. 3

3 Diversity trade-off

• Model mixing improves average gain and worsens best-member gap primarily through initial-answer diversity; controlling for it leaves no residual composition-diversity effect.The indirect path is significant while the direct path is not.Source: §3 · p. 4, Figure 2 · p. 4
• When no agent starts correct, gain increases with diversity; when a correct member is present, process loss rises and the correct signal is often overwritten.For LLMs, majority-correct starts already show negative mean gain.Source: §3 · p. 4, Table 2 · p. 4
• LLM process loss in minority-correct groups is 46–51%; it changes little when a second member starts correct, then falls to 34.8% at unanimity.Human minority survival drops sharply when all three members start with distinct answers, but cell sizes are too small for the same conclusion about their decision regime.Source: §3 · p. 4

4.1 Majority pressure

• The FJ-inspired model finds stronger peer alignment in LLM updates than human updates, while initial anchoring remains nonzero in both populations.The model relates current card selection to prior selection, peer mean, and initial private commitment.Source: §4.1 · pp. 4–5, Figure 3 · p. 5
• Informational conformity is associated with higher collective gain and BMG, whereas harmful conformity is associated with lower values of both metrics.For LLMs, even one informational follow can move both metrics from negative to positive, while one harmful follow can push both below zero.Source: §4.1 · p. 5, Figure 4 · p. 5

4.2 Unique signal realization and resurfacing

• Humans surface unique cards quickly, while LLM realization plateaus lower; positive-BMG LLM groups can resurface unique cards late in discussion.The lifecycle analysis tracks whether unique cards have ever appeared and whether they remain actively held; its dotted line marks mean first consensus lock-in.Source: §4.2 · p. 5, Figure 5 · p. 6
• When the best initial answer belongs to one agent, minority wins coincide with unique cards remaining actively held throughout discussion.This suggests that minority-correct runs depend not only on mentioning the correct signal but also on its survival to the end.Source: §4.2 · p. 6

Interventions

• The interventions target hidden-profile failures, early lock-in, and confidence-weighted integration through solo-answer disclosure, majority-flip announcements, and calibrated confidence projection.ConfProj uses an oracle upper bound, mapping current task scores to a five-point confidence scale shown to other agents.Source: §5 · p. 6

Results

• All three interventions move Gain, BMG, and SAB upward while PL remains stable; RevSol and ConfProj improve BMG without increasing PL.The gains are modest, partly because the pooled average best-member score starts at .78, leaving a narrow window for frontier improvement.Source: §5 · p. 6, Figure 6 · p. 7
• Wrong agents bid substantially higher than correct agents, and logprob-derived certainty does not remain calibrated once group discussion begins.Logprob-derived certainty is moderately calibrated at the initial solo submission.Source: §5 · p. 7
• Lightweight inference-time interventions nudge LLM group quality in the right direction but do not eliminate the coordination bottleneck.The intervention evaluation tracks Gain, BMG, SAB, and PL because improving the mean while damaging the best available signal would not be a clear improvement.Source: §5 · pp. 6–7

Task scope

• The study applies the collaborative structure to EAL analogy mapping, ACR explanation choice, and AR-LSAT logic problems to test generalization across task types.The tasks differ from Wason in logical form and scoring.Source: §6 · p. 7

Generalization results

• Across ACR, EAL, and AR-LSAT, pooled baseline results were Final .52, Gain +.03, BMG −.04, SAB 6.6%, and PL 14.5%, retaining the mean-over-frontier asymmetry.Collective gain is positive for all three tasks, yet SAB remains below PL.Source: §6 · p. 7

Initial-answer diversity

• Process loss is 2.6–11.4% when agents start from one shared answer, but rises to 37–76% with two distinct answers.As in Wason, initial disagreement amplifies movement in both corrective and destructive directions rather than reliably preserving the best starting position.Source: §6 · p. 7

Harmful conformity

• Harmful conformity is negatively associated with collective gain and BMG across ACR, AR-LSAT, and EAL, with all reported p-values below .001.The partial correlations are ACR (r = −0.10, −0.21), AR-LSAT (r = −0.28, −0.35), and EAL (r = −0.26, −0.33).Source: §6 · p. 7

Quality-minority signals

• When one agent holds the best initial answer, minority win rates are 25.2% for ACR, 34.4% for AR-LSAT, and 48.4% for EAL.These rates indicate that minority answers sometimes survive deliberation but are far from reliably preserved.Source: §6 · p. 7, Figure 7 · p. 7

Interventions across tasks

• RevSol and ConfProj gains transfer to AR-LSAT and ACR, while EAL has limited room for gain with its near-ceiling baseline agreement rate.The authors caution that human-motivated inference-time interventions alone may not be the way forward for robust multi-task improvement.Source: §6 · pp. 7–8

Related work

• Prior work spans multi-agent debate, LLM-based social simulation, and analyses of conformity and other collaborative failure modes; this study connects them by comparing matched interaction affordances and tracking solo-to-final trajectories.The paper also diagnoses process mechanisms and tests interventions motivated by those mechanisms.Source: §7 · p. 8

• LLM groups reproduce the assembly-bonus asymmetry: discussion helps the average member more often than it protects the best initial member, while their process differs from humans.Compared with humans, LLM groups show stronger conformity, earlier consensus lock-in, and lower realization of information available in the group's initial state.Source: §8 · p. 8
• Near-ceiling solo accuracy can leave little room to observe group dynamics, while smaller models make those dynamics observable but complicate mechanism-level human comparisons.In a GPT-5 homogeneous setting, final accuracy reaches .983; differing failure and improvement profiles limit what matched outcome patterns support about human groups.Source: §8 · p. 8
• Inference-time interventions modestly improve collective outcomes, but even an oracle confidence signal does not remove the coordination bottleneck.Initial-answer diversity appears more consequential than model-composition diversity because composition diversity improves average gain primarily by seeding varied starting answers.Source: §8 · p. 8
• Human-AI collaboration faces a design tension: a much stronger AI may dominate deliberation, while a more closely calibrated one may share LLM groups' coordination failures.The discussion identifies preserving high-quality minority signals, reliable uncertainty, and avoiding premature consensus as potential design needs.Source: §8 · p. 8

Limitations

• The intervention study is limited to inference-time facilitator-style methods, leaving training-time and LLM-native strategies open despite possible gains and interpretability trade-offs.The authors cite structured protocols, role-specialized agents, and fine-tuning on collaborative traces as untested alternatives.Source: Limitations · p. 9
• Results may differ under other debate formats or asynchronous interaction, and conclusions about untested models, protocols, and task structures warrant caution.The paper focuses on intellective tasks with objectively correct solutions; transfer to judgmental or subjective tasks remains open because analogous process metrics need task-specific definitions.Source: Limitations · p. 9

A Related Work

• Prior work studies LLM social simulation, group dynamics, confidence and influence, and coordination architectures; this work instead uses process diagnostics and tests interventions grounded in small-group research.The paper's related-work discussion notes that comparative evaluations find debate benefits can sometimes come from ensembling rather than conversation.Source: A Related Work · pp. 13–14

B Simulation Design

• The Wason setup uses five logical rule variants across 20 card configurations, with solo submissions followed by visible chat actions and voluntary exit.Each evaluation sweep contains 100 problem instances; sessions end when all agents leave or the group-phase step budget reaches 30.Source: B Simulation Design · pp. 14–15

C Answer-quality and revision dynamics

• Human fine scores rise mainly near conversation end, while LLM scores oscillate before late improvement and their submissions change more substantially across discussion.The final bins reflect voluntary stopping rather than an externally imposed final round; human JSD is approximately 0.03–0.06 versus approximately 0.08–0.21 for LLM agents.Source: Figure 8 · p. 15, C Answer-quality and revision dynamics · p. 15, Figure 9 · p. 15

D Example Conversation

• Two agents initially submit the correct all-card answer for the biconditional rule, but the group converges on an incorrect reduced selection.The transcript shows the shift to an incorrect answer after agents endorse checking only the vowel and odd-number cards.Source: D Example Conversation · pp. 15–16

E Floor-taking Analysis

• High-quality minority holders bid more aggressively than other agents (2.228 vs. 1.854), but the shift was small relative to model-family differences.Bids were only weakly related to answer advantage over the majority, with slightly higher bids under disagreement.Source: E Floor-taking Analysis · p. 17, Figure 10 · p. 18

F Deliberation Length

• Delaying exit from LR1 to LR2 reduced process loss from 39.1% to 31.9% but increased harmful conformity from 28.1% to 31.8%; LR3 added little change.Weak assembly bonus remained more common than strong assembly bonus, process loss remained substantial, and consensus lock-in stayed near two-thirds across conditions.Source: Table 3 · p. 17, F Deliberation Length · p. 17

G Logprob-based Certainty Proxy

• Pooled submission certainty correlated only modestly with correctness (Spearman ρ=+.099), while initial solo certainty showed a stronger positive association (ρ=0.264).After interaction began, Correct-to-Wrong runs retained relatively high certainty while Wrong-to-Correct runs began lowest and rose; the restricted top-n signal is an imperfect proxy, not calibrated probability.Source: G Logprob-based Certainty Proxy · pp. 18–19, Figure 11 · p. 18, Figure 12 · p. 19

H Social Pressure

• The instructor label did not produce dominance: instructors won 14.5% of floor opportunities, and initially correct instructors’ groups had more negative average gain (−.295 vs. −.096).The authors conclude that a human-interpretable status label does not necessarily produce the expected behavioral effect in multi-turn deliberation.Source: H Social Pressure · p. 20

J Intervention Results

• RevSol and oracle ConfProj improved selected model profiles, but their effects did not consistently reduce process loss or remove the coordination bottleneck.RevSol improved BMG for Qwen-4B and Qwen14B; ConfProj had its clearest gains for Qwen-14B and improved gpt-oss-20b final quality.Source: J Intervention Results · pp. 20–21

I GPT-5 Analysis

• GPT-5-containing groups finished at .98 accuracy with mean gain comparable to humans (+.12), while best-member gain remained −.01 and strong assembly bonus was 0.2%.The authors interpret this profile as ceiling-limited rather than a collaboration failure; process loss was 7.9% and minority wins were 64.5%.Source: I GPT-5 Analysis · p. 20, Table 7 · p. 21
• Adding one GPT-5 agent brought final accuracy near ceiling in most mixtures (0.94–0.97), while the 1×GPT-5 + 2×gpt-oss-20b mix rose from 0.68 to 0.86.Two GPT-5 agents reached 0.98–0.99 final accuracy; homogeneous 3×GPT-5 groups began at 0.98, leaving little room for discussion gains.Source: I GPT-5 Analysis · p. 20, Figure 13 · p. 21

K Analysis of additional reasoning tasks

• With one initially correct agent, final group correctness was 21.6% and process loss reached 89.0%; with two, final correctness rose to 78.4%.Among 477 groups with early lock-in, later discussion hurt 33.5% and helped 30.0%, with mean net change +.040.Source: K Analysis of additional reasoning tasks · p. 21
• Across EAL, ACR, and AR-LSAT, collective gain was positive in most conditions, but strong assembly bonus remained low and process loss persisted.These results support improvement in the average group state without reliable preservation of the best initial member.Source: K Analysis of additional reasoning tasks · p. 21, Table 9 · p. 22
• More distinct initial answers raised gain in EAL and AR-LSAT, whereas ACR disagreement sharply increased process loss compared with near-zero loss under initial agreement.Across tasks, diversity was useful when disagreement exposed helpful alternatives, but could destabilize a correct starting point.Source: K Analysis of additional reasoning tasks · p. 21, Table 10 · p. 22
• All three additional tasks showed substantial informational and harmful conformity; informational-follow rates were especially high in EAL and AR-LSAT.ACR had lower informational-follow rates, consistent with its more compressed answer space and higher initial agreement rate.Source: K Analysis of additional reasoning tasks · p. 21, Table 11 · p. 22
• RevSol and ConfProj transferred partially to ACR and AR-LSAT, where BMG generally improved without large process-loss increases; EAL had less room to improve.The interventions did not remove the coordination bottleneck.Source: K Analysis of additional reasoning tasks · pp. 21–22, Table 12 · p. 23

L Group Size

• Increasing group size from N=3 to N=5 left final fine scores statistically indistinguishable (.789 vs. .798; p=.13), despite more recovery in N=5 groups (38.7% vs. 35.1%).N=5 groups took more steps on average (19.8 vs. 17.6), but nominally higher drift meant the recovery advantage did not yield substantially better aggregate performance.Source: L Group Size · pp. 22–23, Table 13 · p. 23