Source-linked AI summary
From Diagnosis to Redesign: Using Quantitative Ethnography to Improve Multi-Agent LLM Reasoning
Vedant Khatri, Anthony Cusimano, Zachari Swiecki, Zhen Xu, Xiner Liu, Renzhe Yu
TL;DR
Multi-agent LLM systems can produce complex reasoning that is difficult to diagnose from output accuracy alone. This paper applies QE and ENA to five-agent essay-scoring debates, then uses discourse findings to revise prompts and evaluate both scores and discourse. Exact scoring accuracy increased from 27.78% to 40.28%, while redesigned debates became more similar to the rubric-grounded pattern associated with accurate debates.
Problem
Multi-agent LLM systems do not inherently produce coherent, task-aligned reasoning, and output-level accuracy does not reveal the discourse-level causes of failure.
Method
The study uses QE and ENA to model discourse patterns in a five-agent Toulmin-structured automated essay-scoring debate and guide targeted prompt revision.
Results
27.78% to 40.28% exact scoring accuracy improvement followed ENA-informed prompt revision, with inaccurate debates shifting toward the rubric-grounded discourse pattern of accurate debates.
Takeaways & Limitations
QE can support a diagnostic-to-redesign loop that links agent-interaction patterns to system performance and evaluates whether redesigns change both outcomes and discourse.
Takeaways & Limitations
The revised system remained inaccurate on a majority of essays, indicating that rubric-grounding deficits are not the only mechanism driving inaccuracy.
Abstract
from arXiv · showhide
Multi-agent large language model (LLM) systems are designed to improve reasoning by decomposing tasks across multiple agents with specialized functions, but the presence of multiple agents does not inherently guarantee coherent reasoning or outputs that align with task objectives. This paper introduces a quantitative ethnographic (QE) approach for diagnosing and redesigning multi-agent LLM systems based on the discourse produced through agent interactions. We test this approach using automated essay scoring as an example context, applying Epistemic Network Analysis (ENA) to model a five-agent multi-agent debate system and examine differences between debates that produced correct versus incorrect scoring decisions. Results show that, in the initial system, correct scoring decisions were characterized by rubric-grounded justification, agreement, and elaboration. Incorrect scoring decisions, in contrast, were characterized by extended proposition-challenge-response exchanges that were less consistently tied to rubric criteria. We then used the findings to revise the agents' prompts. The revised system improved exact scoring accuracy from 27.78% to 40.28% and shifted the discourse of incorrect debates toward the rubric-grounded pattern of correct ones, making the two nearly indistinguishable. Based on these results, we argue that QE can support a diagnostic-to-redesign loop for AI reasoning by tracing how patterns of agent interaction relate to system performance, informing prompt redesign, and evaluating whether those redesigns change both outcomes and interaction patterns.
1 Introduction
The paper addresses why multi-agent LLM reasoning can fail despite specialized roles by using quantitative ethnography to analyze agent discourse and guide targeted redesign. Automated essay scoring provides a test context in which both accuracy and rubric-grounded justification matter.
- Motivation: Multi-agent LLM systems can outperform single-agent baselines, but complex multi-round reasoning makes task-level failure difficult to diagnose from accuracy alone.Output-level metrics indicate when a system fails without explaining why the agents’ discourse produced the failure.
- Approach: ENA models connections among coded discourse elements to identify reasoning patterns associated with successful and unsuccessful multi-agent outputs.The approach treats meaning as patterns of connection rather than individual-element frequency.
- Test Context: Automated essay scoring tests the approach because a holistic score must be justified in relation to a scoring rubric.The system uses a five-agent Toulmin-structured debate with Claim, Grounds, Warrant, Rebuttal, and Judge roles.
- Contributions: The paper contributes a QE methodology for discourse-level diagnosis, identifies accurate-versus-inaccurate debate structures, and demonstrates a diagnostic-to-redesign loop evaluated through outcomes and discourse change.Although demonstrated on essay scoring, the authors intend the methodological approach to generalize to other deliberative multi-agent LLM systems.
2 Related Work
Prior work shows that specialized multi-agent roles and debate can support reasoning, but systems still produce coordination failures and unjustified inferences. QE and ENA offer a way to analyze interactive discourse structure and use that diagnosis to test redesigns.
- Motivation: LLM-generated rationales may not reflect the actual basis of predictions, motivating designs intended to make reasoning more transparent.
- Multi-Agent Debate: Multi-agent debate uses sequential exchanges in which agents read prior contributions and add argumentative moves before a judge reaches a final solution.Prior frameworks use iterative critique, revision, and judge-managed debate to seek more reliable answers than isolated single-agent generation.
- Role-Separated Designs: Toulmin-based role separation has been used to organize evidence, warrants, rebuttals, and counter-rebuttals in assessment-oriented debate systems.The cited prior system mapped debate stages to separate agents for evaluating students’ critical thinking.
- Open Problems: Role-separated multi-agent systems still exhibit coordination failures, premature consensus, and unjustified inferences, and may not reliably outperform simpler prompting strategies.Some systems can also shift from correct to incorrect answers across debate rounds.
- Analytic Gap: Coding-and-counting approaches aggregate discrete failure modes but do not capture how agent turns connect into larger argumentative structures.This motivates modeling transcripts as networks of argumentative relations, especially when assessment scores require justification.
- QE and ENA: QE connects interpretive discourse coding with quantitative modeling, while ENA represents codes as nodes connected by within-context co-occurrences.The underlying assumption is that meaning is produced through connections among discourse elements, not merely through their frequency.
- QE and ENA: Frequency-based methods can miss interactive, interdependent, and temporal discourse structures that ENA can distinguish even when code distributions are similar.
- Research Gap: ENA has not been widely applied to AI-generated discourse, while multi-agent evaluations often rely on output metrics or selective transcript inspection.The paper treats complete generated transcripts as analyzable network structures for comparing configurations and prompting conditions.
3 Methods
The study combines a Toulmin-structured five-agent essay-scoring system with CACS coding and ENA to compare debate structures, diagnose rubric-grounding failures, and revise prompts. The revised prompts preserve the rubric, sampling parameters, and architecture while targeting discourse-level behavior.
- Data: The study uses ASAP 2.0 argumentative essays scored on a six-point holistic scale with expert-assigned gold-standard scores.The scale ranges from 1, indicating very little or no mastery, to 6, indicating clear and consistent mastery.
- Architecture: Toulmin’s model provides the theoretical frame for the architecture and analysis through claims, grounds, warrants, and rebuttals.The model describes warrants as licensing the inferential move from grounds to claim and rebuttals as identifying conditions under which claims would not hold.
- Architecture: Each essay passes sequentially through five argumentative agents and a final Judge Agent.The five roles correspond to Claim, Grounds, Warrant, Rebuttal, and Judge.
- Debate Process: Every agent receives the essay, the ASAP 2.0 rubric, and the complete prior transcript before producing its contribution.
- Prompt Revision: The revised round changed system-prompt instructions to require explicit evidence-to-rubric mapping without adding the rubric itself.The rubric remained part of every agent’s system prompt in both experimental rounds.
- Model and Sampling: Agents used Qwen 2.5 72B Instruct with temperature 0.3 and a maximum of 400 new tokens per turn.The low temperature was selected to support stable scoring outputs while allowing distinct argumentative contributions.
- Design Contributions: The architecture adds a bidirectional Rebuttal Agent and rubric-level few-shot examples for the Claim Agent.The Rebuttal Agent may argue upward, downward, or agree when no revision is warranted.
- Discourse Coding: CACS codes each agent turn for argumentative functions such as assertions, supports, objections, and responses.The scheme was selected because it captures argumentative moves in multi-party conversational exchanges.
4 Results
ENA distinguished accurate from inaccurate initial debates by their discourse structures, and prompt revision improved exact scoring accuracy while making revised debate networks nearly indistinguishable across accuracy groups.
- 4.1 Accurate Debates Were More Rubric-Grounded (RQ1): Initial accurate and inaccurate debates differed significantly on MR1, with t(37.48) = −4.69, p < .001, d = 1.19.The accurate centroid was MR1 = -0.21, while the inaccurate centroid was MR1 = 0.08.
- 4.1 Accurate Debates Were More Rubric-Grounded (RQ1): Accurate initial debates connected Justification, Agreement, Elaboration, and Assertion, whereas inaccurate debates emphasized Proposition, Challenge, and Response.The former pattern reflected rubric-grounded convergence; the latter reflected unanchored contestation without consistent rubric connections.
- 4.2 ENA-Informed Prompt Revision Improved Exact Accuracy (RQ2): 27.78% initial exact scoring accuracy rose to 40.28% after prompt revision, a relative improvement of approximately 45%.The paired improvement was statistically significant: χ2(1) = 3.24, one-tailed p = .0359.
- 4.2 ENA-Informed Prompt Revision Improved Exact Accuracy (RQ2): Of 52 initially incorrect essays, 17 became correct after revision, while 8 of 20 initially correct essays became incorrect.The net change was 9 essays moving from incorrect to correct.
- 4.2 ENA-Informed Prompt Revision Improved Exact Accuracy (RQ2): Prompt revision increased Justification from 52 to 225 turns and reduced Objection from 57 to 12 and Response from 43 to 5 turns.Agreement also increased from 139 to 159 turns, consistent with more rubric-grounded justification.
- 4.3 Revised Prompts Reduced the Discourse-Level Separation Between Groups (RQ3): Revised accurate and inaccurate debates showed no significant ENA separation, with t(58.30) = −0.03, p = .97, d = 0.01.Both groups had a centroid at V1 = -0.42, compared with an initial-round separation of 0.29.
5 Discussion and Conclusion
The study frames ENA as a diagnostic-to-redesign method for multi-agent LLM reasoning, showing that role separation alone does not ensure rubric-grounded reasoning. Its accuracy and discourse findings support inspecting and revising agent interaction patterns, while the evidence remains bounded by a single essay-scoring setting and model.
- Discussion: Role separation alone did not produce evidence-to-rubric reasoning, because inaccurate initial debates still used proposal-and-challenge cycles without rubric grounding.The revised prompts instead translated this discourse-level failure into an actionable redesign target.
- Discussion: ENA extended quantitative ethnography to LLM-generated discourse as both an analytic tool and evidence source for system redesign.The approach models connections among discourse codes rather than treating agent contributions as isolated frequencies.
- Limitations: The revised system remained inaccurate on a majority of essays, indicating that rubric-grounded discourse was not the only source of remaining errors.The study also lacked a single-agent baseline and an ablation isolating which prompt changes mattered.
- Limitations: The empirical scope was limited to essay scoring with one open-weight model, Qwen 2.5 72B Instruct.The authors call for testing whether the diagnosed discourse structures remain stable across contexts and LLM families.
- Conclusion: The diagnostic-to-redesign loop converted inaccurate essay scores into a rubric-grounded warranting target, revised prompts, and evaluated both performance and discourse change.The authors present this logic as applicable in principle to other deliberative multi-agent systems.
A Appendix: Agent Prompts
The appendix provides the full original and revised agent prompts in a supplementary repository.
- Appendix: Agent Prompts: The full original and revised prompts are available in the supplementary repository.The repository is identified by the paper as https://github.com/AnthonyCusi/ICQE-appendix.