Source-linked AI summary
How AI Coders Discuss, Disagree, and Reach Consensus: Challenges and Opportunities for LLM-Based Qualitative Coding
Jeongyeon Kim, John Mitchell
TL;DR
Qualitative coding is time-consuming even for experts, and the study examines reliability and consensus when AI performs this work. It evaluates structured dialogue among LLM coders and finds that disagreement can signal higher accuracy, while agents lack human-like adaptive responsiveness.
Problem
The study examines what reliability and consensus mean for AI coders performing time-consuming qualitative coding.
Method
The paper develops a structured dialogue pipeline in which LLM coders independently code, debate disagreements, and reconcile their labels.
Results
Disagreement and debate between AI coders can signal higher coding accuracy, while LLM coders mirror basic human reasoning moves but lack adaptive and inquisitive depth.
Takeaways & Limitations
Automated coding systems should use disagreement as a performance signal and adjust agents’ discussion styles to the coding task’s objectives.
Takeaways & Limitations
The AI model was not fine-tuned for qualitative coding tasks or domain-specific data, tempering the study’s conclusions.
Abstract
from arXiv · showhide
The utility of AI in multi-coder qualitative coding has been widely discussed, yet little empirical evidence exists to delineate the contexts in which it performs reliably. We address this gap by quantifying the effectiveness of multi-agent LLM coding across varied qualitative datasets, revealing key contextual and structural factors that mediate coding outcomes. We developed a literature-informed baseline pipeline that enables AI agents to independently code, debate, and reconcile disagreements. Results revealed that coding accuracy depends on factors such as codebook length, qualitative data similarity, and agent disagreement. Notably, intense and unresolved debates between agents led to higher accuracy. Our analysis showed that while LLMs emulate many human discussion behaviors, they lack adaptive responsiveness to context. From these findings, we offer design recommendations for building automated coding systems. Our open-source AI discussion dataset and methodological framework lay the groundwork for advancing the design of AI-mediated automated thematic analysis.
1 Introduction
The paper examines when multi-agent LLMs can reliably support qualitative coding and develops a literature-informed pipeline, evaluation, and design guidance for automated systems.
- The study quantifies multi-agent LLM coding across varied qualitative datasets to identify contextual and structural factors affecting outcomes.The analysis considers dataset characteristics, labeling uncertainty, coder consensus, and related system-design concerns.
- The literature review identifies transparency, reliability, explanations, and actionable optimization guidance as central user needs for automated coding systems.Users need to understand why codes are assigned, how disagreements are resolved, and how reliable outputs are.
- The authors develop a baseline multi-agent pipeline in which AI coders independently code, discuss disagreements, and reconcile outputs across four qualitative datasets.The pipeline models collaborative human coding practices and translates identified user needs into an experimental system.
- Coding accuracy improves with more concise codebooks and more intense disagreement during agent discussions.These findings indicate that codebook structure and discussion dynamics mediate coding effectiveness.
- LLMs reproduce basic interaction behaviors but miss adaptive moves such as clarifying uncertainty and drawing on personal experience.The systems acknowledge perspectives and propose alternatives, yet lack some nuanced behaviors observed in human coders.
- The paper recommends using uncertainty and conflict diagnostically, adapting discussion styles to task ambiguity, and supporting transparent user interaction.The authors also release the baseline pipeline and agents’ discussion dataset as reusable resources.
2 Related Work
Prior work has advanced automated qualitative coding through prompts, reasoning, context, and human-AI collaboration, but evidence remains fragmented across datasets and conditions.
- Studies have improved coding through precise codebooks, debate-informed rules, reasoning, contextual information, and repeated-run consistency checks.These approaches seek greater precision, adaptability, methodological rigor, and robustness.
- Multi-agent and collaborative approaches extend automated coding beyond single-model workflows by incorporating comparison, debate, confidence scoring, or human collaboration.Prior systems include DebateGPT, multi-human-coder workflows, and interactive tools that retain user autonomy.
- Existing research often evaluates automated qualitative coding on individual datasets, leaving limited guidance for scalable and generalizable systems.The paper identifies a gap in comprehensive analysis across varying coding conditions.
- The paper responds to the lack of analytical evidence about how codebook length, data complexity, and other dataset features relate to LLM performance.This gap leaves developers with limited support for optimizing systems beyond trial and error.
- Prior work emphasizes human agency, transparency, ambiguity, and oversight because automated systems can miss context and nuance or encourage echo-chamber perspectives.The paper builds on these concerns by comparing literature-derived user needs with LLM experiment outcomes.
3 Exploratory Literature Review
The exploratory review derives design goals for automated qualitative coding from a screened literature corpus, using an iterative coding process rather than a comprehensive systematic review.
- Table 1 summarizes the design goals for automated qualitative coding systems and the literature supporting their formulation.
- The search targeted AI, LLM, automation, qualitative coding, qualitative analysis, and thematic-analysis terminology.The database query combined AI-related terms with qualitative-coding terms.
- An iterative full-text eligibility evaluation reduced the corpus to 118 papers for analysis.The final corpus excluded work without a primary focus on automated qualitative coding, user expectations, or design considerations.
- The review used an exploratory rather than comprehensive systematic approach because its purpose was to inform baseline-pipeline design goals.One author generated preliminary codes and finalized the codebook through a second analysis round, with independent expert assessment.
3.3 Review Results
The review identifies design goals centered on transparent explanations, uncertainty, calibrated expectations, and understanding how data characteristics affect coding accuracy.
- Together, the goals emphasize transparent outputs, visible uncertainty, calibrated expectations, and actionable understanding of conditions influencing performance.These goals are presented as the basis for optimizing automated qualitative coding systems.
- Automated coding systems should explain how codes are assigned and provide supporting reasons and evidence.The goal is to help users understand AI reasoning rather than receive unexplained outputs.
- Systems should expose uncertainty and interpretive variability by representing confidence and enabling independent coders to discuss disagreements.The pipeline uses two LLM coders and confidence estimates to mirror human coding processes.
- Users should be able to calibrate expectations for coding accuracy through performance signals such as coder disagreement.The review motivates disagreement as a proxy for evaluating automated coding accuracy.
- Systems should help users understand how codebook complexity, language abstractness, data similarity, length, and specialization affect coding reliability.
4 Large Language Model Experiments and Data Analysis
The study evaluates a literature-informed multi-agent LLM coding pipeline across four qualitative domains, measuring accuracy, reliability, dataset factors, and discussion dynamics.
- The experiments sampled five labels and 500 instances per label from education, law, sociology, and medicine datasets.The corpora included learning dialogues, European human-rights cases, interview responses, and medical abstracts.
- The study released its prompt design, coding pipeline, and LLM-generated discussions as open-source materials.
- The pipeline extracts code-relevant excerpts before coding, then has two AI coders independently code, discuss conflicts, derive labeling rules, and recode.Coding prompts required justification and allowed present, absent, or undecidable responses; unresolved cases after three rounds were marked disagreed.
- Accuracy was analyzed against dataset characteristics and labeling dynamics using mixed-effects models, alongside qualitative analysis of AI discussion patterns.Factors included codebook length, semantic similarity, excerpt characteristics, specialization gaps, uncertainty, consensus, and controversy.
- Across datasets and labels, inter-rater reliability exceeded 0.85, while F1 scores ranged from 0.31 to 0.89.The average F1 score was 0.68, with a standard deviation of 0.16.
- Disputed or undecidable cases averaged 0.60 accuracy, while lower confidence and consensus were associated with reduced accuracy.The result highlights a tradeoff between coding coverage and accuracy.
teristic
Coding accuracy varied with codebook and data characteristics, uncertainty, consensus, and controversy. Diverse codebook language, consistent target data, and extended debates were associated with more accurate discussion-prompted coding.
- teristic: Concise codebooks and data were associated with higher accuracy, although this pattern may partly reflect simpler codes requiring less detailed explanations.
- teristic: Codebook and excerpt characteristics showed opposing similarity effects: diverse codebook language and consistent target data improved accuracy, while terminology gaps impaired it.Higher similarity within excerpts helped, whereas codebook similarity and specialization disparities were associated with lower accuracy.
- 4.4.2 Relationship between Coding Accuracy Increase and Labeling Uncertainty: Greater initial ambiguity was associated with larger accuracy improvements from discussion, whereas persistent undecidable labels indicated smaller gains.The dialogue process was described as clarifying uncertainties and building shared understanding.
- 4.4.3 Relationship between Coding Accuracy Increase and Consensus Level of AI Coders: The number of initial conflicts did not affect discussion-related accuracy gains, but fewer conflicts after discussion indicated a positive discussion effect.
- 4.4.2 Relationship between Coding Accuracy Increase and Labeling Uncertainty: Accuracy increased with more discussion turns and was higher in controversial, including unresolved, debates about challenging or borderline cases.Such debates clarified inclusion and exclusion criteria and exposed subtle details about code scope.
- 4.4.2 Relationship between Coding Accuracy Increase and Labeling Uncertainty: Compromising and collaborating discussion modes were negatively related to accuracy, while competing modes showed no significant relationship.The paper attributes the contrast partly to LLM agents lacking human emotional and social dynamics, allowing competitive exchanges to refine reasoning.
- 4.4.2 Relationship between Coding Accuracy Increase and Labeling Uncertainty: Because LLM agents may share architectures and training data, collaborative modes can produce overly convergent answers and reinforce similar opinions.
5 Design Recommendations for Automated Qualitative Coding Systems
The paper recommends designing automated qualitative coding systems around dataset preprocessing, interpretable performance signals, and discussion dynamics rather than rigid thresholds. These recommendations use codebook and excerpt characteristics, uncertainty, disagreement, and agent discussion structure to help users anticipate system performance.
- Scope of Recommendations: The recommendations are conceptual and methodological rather than rigid threshold rules, and the findings provide a foundation for further quantitative research.The paper also reports that quantitative experiment details, including mean, standard deviation, minimum, and maximum values, are available in supplementary materials.
- Design Recommendation for Designers 1: Designers should provide targeted preprocessing guidance, including diagnostic feedback and actionable suggestions, to optimize system performance.Recommendations address dataset properties, codebook construction, excerpt length, and data organization.
- Design Recommendation for Designers 1: Concise, distinct codebooks and appropriately specialized definitions are recommended because codebook characteristics can affect coding accuracy.The paper illustrates recommendations involving codebook length, repetitive definitions, similarity among code descriptions, and specialization relative to the data.
- Design Recommendation for Designers 2: “Undecidable” and conflicting labels should be presented as interpretable indicators of potential accuracy gains from AI agent discussions.The proposed indicators include uncertainty before and after discussion and the persistence of conflicting labels.
- Design Recommendation for Designers 4: Designers may offer discussion modes that foster opposing perspectives, since structured opposition was associated with improved discussion-prompted coding accuracy.The recommendation is to adjust discussion behavior according to the coding task’s objectives rather than impose one fixed style.
- Design Recommendation for Designers 3: Users should evaluate the structure of agent disagreement, because prolonged or unresolved debates may provide more information than conflict resolution alone.The paper links high conversational exchange and controversy with expected accuracy, while treating persistent post-discussion conflict as a possible signal of limited discussion effects.
<Example of Discussion with Competing Mode>
The competing discussion mode shows agents defending and revising interpretations while handling cases that remain difficult to categorize definitively.
- The example contrasts positive-evaluation reasoning with a competing interpretation of whether the excerpt expresses an explicit evaluation.
- Competing interpretations may remain undecidable when the excerpt does not support a definitive present-or-absent label.
- Agents can revise an initial interpretation after reassessing another coder’s reasoning and evidence.
6 Discussion
The discussion finds that multi-agent LLM coding can support structured qualitative analysis, but accuracy and interpretive value depend on data characteristics, disagreement, and debate dynamics.
- Coding accuracy varies with dataset length, semantic similarity, codebook–data specialization gaps, and preprocessing choices.
- Disagreement and debate between AI coders can signal higher coding accuracy and increase interpretive depth.
- Rapid consensus can mask unresolved ambiguity, whereas prolonged exchanges were associated with more nuanced understanding of edge cases.
- LLM agents reproduced stance maintenance, acknowledgment, explicit comparison, evidence reassessment, and alternative proposals found in human discussions.
- Unlike human coders, the agents did not ask clarifying or probing questions and followed a less adaptive discussion pattern.
6.3 Theoretical Implications: Rethinking Reliability and Agreement
The paper argues that AI-era reliability should account for correlated errors, discussion processes, and persistent disagreement rather than relying only on final agreement.
- High agreement between AI coders may not guarantee validity because same-model agents can share correlated errors and biases.
- Reliability measures could incorporate discussion requirements and decision stability before and after debate, not only final-code agreement.
- Human–AI systems may deliberately preserve some disagreements because they can identify areas requiring further human analysis.
- The paper treats ambiguity and disagreement as constitutive features of qualitative knowledge that can become measurable design signals.
- Discussion behavior should be tunable: debate can support ambiguous coding, while strict mode can streamline routine, well-defined tasks.
- AI can surface ambiguity and conflicts, but reflexive interpretation and positional disclosure remain human responsibilities.
7 Conclusion
The paper provides a foundation for understanding multi-agent LLM qualitative coding by linking discussion dynamics and data characteristics to coding outcomes. It recommends treating disagreement and ambiguity as design resources and adapting discussion style to task goals.
- The study combines simulated coder discussions, analysis of data characteristics, and qualitative examination of AI behavior to characterize LLM coding dynamics.
- Disagreement and ambiguity are presented as opportunities for deeper insight rather than merely obstacles to automated coding.
- The design recommendations are concise diverse codebooks, coherent-topic excerpts, uncertainty diagnostics, and task-appropriate discussion styles.
A Comparison of Discursive Moves Between LLM Coders and Humans
Table 7 identifies discursive moves reported in human coding discussions that did not appear in LLM coder discussions.
- Human discursive moves reported in the literature were absent from LLM coder discussions.
- Table 7 compares human discursive moves with their absence in LLM coder discussions.
- The comparison concerns discursive moves used by humans in prior literature rather than coding accuracy.