Source-linked AI summary
A Computational Method for Measuring "Open Codes" in Qualitative Analysis
John Chen, Alexandros Lotsos, Sihan Cheng, Caiyi Wang, Lexie Zhao, Yanjia Zhang, Jessica Hullman, Bruce Sherin, Uri Wilensky, Michael Horn
TL;DR
Inductive coding is exploratory, so ground-truth metrics do not fit its open-ended aims, while manual evaluation can be labor-intensive. This paper merges coder codebooks and measures contributions with four metrics, then tests their behavior across experiments. The metrics show stable top and bottom coder rankings across LLMs and detect coding on irrelevant data through reduced Coverage and Overlap and sharply increased Divergence.
Problem
Inductive coding lacks a single correct result, making ground-truth-based evaluation a mismatch, while manual evaluation can be labor-intensive.
Source: §1 · p. 1, §2.1 · p. 2Method
The method merges individual codebooks into an aggregated code space and measures each coder against it using Coverage, Overlap, Novelty, and Divergence.
Source: §3.1 · p. 3, §3.2 · p. 5Results
Top- and bottom-ranked coders remained consistent across four LLMs, while irrelevant-data coding had 35.6% Coverage, 15.6% Overlap, and 75.7% Divergence.
Source: §5.2.4 · p. 7, §5.3.2 · p. 8Takeaways & Limitations
The metrics provide a way to assess inductive coding contributions without enforcing agreement through a single ground truth.
Source: §7 · p. 9Because the ACS is built from all coders’ outputs, it cannot detect critical codes absent from every individual code space.
Source: §8.3 · p. 10
Abstract
from arXiv · showhide
Qualitative analysis is critical to understanding human datasets in many social science disciplines. A central method in this process is inductive coding, where researchers identify and interpret codes directly from the datasets themselves. Yet, this exploratory approach poses challenges for meeting methodological expectations (such as ``depth'' and ``variation''), especially as researchers increasingly adopt Generative AI (GAI) for support. Ground-truth-based metrics are insufficient because they contradict the exploratory nature of inductive coding, while manual evaluation can be labor-intensive. This paper presents a theory-informed computational method for measuring inductive coding results from humans and GAI. Our method first merges individual codebooks using an LLM-enriched algorithm. It measures each coder's contribution against the merged result using four novel metrics: Coverage, Overlap, Novelty, and Divergence. Through two experiments on a human-coded online conversation dataset, we 1) reveal the merging algorithm's impact on metrics; 2) validate the metrics' stability and robustness across multiple runs and different LLMs; and 3) showcase the metrics' ability to diagnose coding issues, such as excessive or irrelevant (hallucinated) codes. Our work provides a reliable pathway for ensuring methodological rigor in human-AI qualitative analysis.
1 Introduction
Inductive coding is open-ended and lacks a single ground truth, making conventional evaluation a poor fit. The paper introduces a computational approach to assess human and machine coding through codebook merging and four metrics, tested in two experiments.
1 Introduction
- Inductive coding derives concepts from raw data, but its open-endedness makes ground-truth-based evaluation methods such as inter-coder reliability mismatched.The paper frames a theory-informed, computationally operational evaluation method as needed as researchers use GAI for inductive coding.Source: §1 · p. 1
- The proposed method merges codebooks with an LLM-enhanced algorithm and evaluates human and machine coders using four metrics without ground-truth assumptions.Two experiments examine merging effects, metric stability and robustness across runs and LLMs, and diagnosis of excessive or irrelevant codes.Source: §1 · p. 1
2 Related Work
Inductive coding seeks broad discovery rather than a singular correct result, creating evaluation needs that existing approaches meet only partially. The section contrasts those needs with the limitations of ground-truth metrics, structural metrics, and human assessment.
2.1 The Nature and Challenges of Inductive Qualitative Coding
- Inductive coding seeks novel insights and broad coverage of patterns, but is subjective, time-consuming, and does not aim for a singular correct result.It is also prone to ambiguity, making methodological rigor difficult; deductive measures such as inter-rater reliability rely on ground-truth assumptions.Source: §2.1 · pp. 1–2
- Team-based coding compares multiple perspectives, which can yield more insights, mitigate individual biases, and move analysis toward depth, variation, and theoretical saturation.The cited work includes Cascio et al. (2019), Thomas (2006), Corbin and Strauss (1990), and Thomas (2006).Source: §2.1 · p. 2
2.2 Evaluating ML/GAI for Inductive Qualitative Coding
- Existing ML/GAI evaluation includes classification-based approaches that mimic human labels and generation-based approaches that produce codes from data.The section also describes the need for robust evaluation methods that account for inductive coding’s open-ended and exploratory characteristics.Source: §2.2 · p. 2
- Ground-truth metrics quantify precision and recall but presume a single correct answer, while clustering and topic-coherence metrics prioritize homogeneity over conceptual breadth and variation.Clustering and topic-coherence approaches also assess outputs in isolation rather than how coders align, diverge, or collectively represent the dataset.Source: §2.2 · p. 2
- Human-annotated assessments can provide qualitative insight, but are costly, time-consuming, and difficult to scale for large datasets or iterative evaluation.They may also overlook systematic omissions when all coders fail to recognize a key concept.Source: §2.2 · p. 2
3 Computational Metrics
The method evaluates individual code spaces relative to a shared aggregated code space, using a staged semantic-merging algorithm and four complementary metrics. The metrics capture breadth, alignment, distinct contributions, and distributional difference without assuming a ground truth.
3 Computational Metrics
- The framework requires evaluation without ground truth, measures breadth and interpretive balance while discouraging fragmentation or redundancy, and aims for stability and sensitivity to problematic cases.It introduces Coverage, Overlap, Novelty, and Divergence to assess each coder’s contribution to the collective interpretation without requiring human adjudication.Source: §3 · p. 3
3.1 Aggregating Coding Results: Code Spaces (CSP) and Aggregated Code Spaces (ACS)
- The ACS combines coders’ code spaces and consolidates semantically similar codes, recognizing that different labels may represent the same idea.A four-stage algorithm uses semantic similarity and hierarchical clustering; LLMs generate definitions from labels and examples and generate labels and definitions for merged codes.Source: §3.1 · p. 3
3.2 Four Computational Metrics
- Coverage measures a coder’s reach across the ACS, weighting codes by how many coders identified them and coders by their total code counts to limit over-coding inflation.Excessively high Coverage, particularly alongside many codes, may indicate redundancy or flooding.Source: §3.2 · p. 4
- Overlap reflects how much a coder’s codes coincide with or neighbor others’ codes, while Novelty captures unique codes that add conceptual space beyond the team’s coverage.Very high Overlap with low Novelty may indicate redundancy; excessive Novelty with low Overlap may suggest idiosyncratic or hallucinatory coding.Source: §3.2 · pp. 4–5
- Divergence measures the distance between a coder’s code distribution and the ACS distribution using Jensen-Shannon Divergence, which tolerates potential zeros.The metrics are best interpreted together rather than against fixed thresholds; productive coding tends toward moderate Coverage, shared-understanding Overlap, and distinct but relevant Novelty.Source: §3.2 · p. 5
4 Experimental Design
The experiments test how merging stages affect metrics, whether metrics remain stable across runs and LLMs, and whether they detect excessive or irrelevant coding. They apply the methods to human and machine coding of an online conversation dataset and compare algorithm stages and edge-case variants.
4 Experimental Design
- Two experiments address the effects of merging stages, metric stability across probabilistic LLMs, and detection of excessive codes or hallucinations.These correspond to RQ1–RQ3.Source: §4 · p. 5
4.1 Task and Dataset
- The study applies metrics to four human coders and four machine coders coding an online conversation dataset between Physics Lab designers and teacher users.The prompt, dataset, and human coding results were reused from Chen et al.’s study; the data were anonymized and use was consistent with the original intention and IRB approval.Source: §4.1 · p. 5
4.2 Experiment 1: Ablation and Comparison Study
- Experiment 1 evaluates four merging-algorithm conditions: label-only merging, label merging at strict threshold 0.32, LLM-generated definitions plus strict 0.32, and iterative merging using upper 0.55 and lower 0.32 thresholds.Each condition and LLM used was repeated 10 times; metrics were calculated for individual coders and AI or human coder groups.Source: §4.2 · pp. 5–6
4.3 Experiment 2: Measuring Edge Cases
- Experiment 2 replaces the best-performing Item-Level coder with variants that generate excessive codes, code an irrelevant AI-generated conversation, or combine both changes.Each variant is repeated 10 times with the same human and machine coders, using Gemma3 27B because preliminary results found little impact from LLM choice.Source: §4.3 · p. 6
5 Empirical Study
The experiments examine how merging stages and LLM choice affect code counts and coder metrics, whether rankings remain stable, and whether the metrics diagnose simulated coding issues. Results show stable top- and bottom-coder rankings alongside metric shifts across merging conditions and diagnostic differences between excessive and hallucinated coding.
5.1.1 Hypothesis 1: Evaluation condition significantly affects the number of merged codes and computed metrics.
- Each algorithm stage significantly reduced merged-code counts; metrics shifted in Conditions 3 and 4, but not Condition 2.The results report p < 0.001 for the code-count reductions; Table 1 presents OLS coefficients for Conditions 2–4 relative to Condition 1.Source: §5.1.1 · p. 6, Table 1 · p. 6
5.1.2 Hypothesis 2: Evaluation condition has minimal impact on the relative ranking of coder metrics.
- Metric rankings were relatively stable across algorithmic conditions: the top five coders retained their rankings, and other coders’ rankings were similar across label-only and LLM-enriched conditions.Source: §5.1.2 · p. 6
5.2.1 Hypothesis 3: LLM used in the merging process significantly influences metrics and code counts.
- Across Conditions 3 and 4, Gemma3 27B, Qwen QwQ 32B, and GPT 4.1 produced similar metrics and code counts; Gemini-2.5-pro was the substantial deviation.Gemini-2.5-pro produced fewer merged codes, Coverage and Overlap increases of approximately 4% to 6%, and a 4% decrease in Divergence.Source: Figure 2 · p. 7, §5.2.1 · p. 7
5.2.2 Hypothesis 4: Metric outcomes are well-explained by condition, model, and coder identity.
- Condition, merging model, and coder identity together explained over 0.91 of adjusted R2 for every metric.Source: §5.2.2 · p. 7
5.2.3 Hypothesis 5: Repeated measurements under the same condition/model yield low coefficients of variation (CoV).
- Across repeated runs, all coefficients of variation were below 0.1; Divergence had the lowest variability, around 0.01.Condition 4 showed slightly higher variance than Condition 3.Source: §5.2.3 · p. 7
5.2.4 Hypothesis 6: LLMs used in the merging process have little effect on the relative ranking of coders.
- Top- and bottom-ranked coders remained consistent across all four LLMs, while mid-performing coders’ rankings fluctuated, primarily when differences were not statistically significant.Source: §5.2.4 · p. 7
5.3.1 Hypothesis 7: Excessive coding increases coverage, overlap, and novelty with diminishing returns, while divergence remains stable.
- Flooding increased Coverage, Overlap, and Novelty over Baseline, while Divergence remained stable; Novelty showed diminishing returns.The reported Flooding means were Coverage 78.7%, Overlap 57.9%, Novelty 68.1%, and Divergence 67.3%, versus 64.9% for Baseline Divergence.Source: Figure 3 · p. 7, §5.3.1 · p. 8
6 Discussions
The discussion presents four complementary coder metrics and explains how the merging design and LLM choice affect their interpretation. It reports stable rankings and repeated measurements, and describes metric patterns that flag simulated coding problems for further evaluation.
6 Discussions
- Coverage reflects a coder’s breadth and depth against the Aggregated Code Space; Overlap captures alignment, Novelty unique concepts, and Divergence difference in conceptual focus.Source: §6 · p. 8
6.1 The Necessity of the Iterative, LLM-Enriched Merging Algorithm
- Adding LLM-generated definitions at Stage 3 produced the largest metric shift; Stage 4 further consolidated the Aggregated Code Space without significantly changing measurement outcomes.Stage 3 uses definitions informed by underlying data points without directly comparing those data points; Stage 4 is computationally intensive and may suit conceptual-map goals, while Stage 3 may suffice for metric screening.Source: §6.1 · pp. 8–9
6.2 The Impact of LLM on Outcomes
- Metrics were stable across repeated runs and LLM choices, with top- and bottom-coder rankings consistent across models; the discussion notes that smaller open-source models can be used.The authors connect local deployment of such models with better privacy protections and regulation compliance for sensitive human-subject data.Source: §6.2 · p. 9
6.3 Performance of Metrics in Edge Cases
- Flooding variants showed higher Coverage and Novelty with diminishing returns, while Hallucinating variants showed lower Coverage and Overlap and higher Divergence; abnormal metric values can flag cases for further evaluation.Source: §6.3 · p. 9
7 Conclusion
The conclusion presents a computational method for systematically evaluating inductive coding without relying on a single ground truth. It frames Coverage, Overlap, Novelty, and Divergence as a way to quantify coder contributions and support methodological rigor in human-AI analysis.
7 Conclusion
- The method evaluates inductive coding without relying on a single ground truth and measures Coverage, Overlap, Novelty, and Divergence.Source: §7 · p. 9
- The approach shifts evaluation from enforcing agreement toward recognizing the diverse contributions coders bring to collaborative, exploratory analysis.Source: §7 · p. 9
- The paper presents the method as a pathway for qualitative researchers to use AI in inductive analysis while pursuing methodological rigor and transparent human-AI collaboration.Source: §7 · p. 9
8 Limitations
The authors identify limits in the validation data and simulated edge cases, as well as blind spots in collective code aggregation and differences in LLM merging behavior. They call for future work to establish best practices for using the metrics.
8 Limitations
- The study calls for future research to establish best practices and clear guidelines for adopting its metrics.Application and misuse risks are discussed separately in Appendix A.Source: §8 · p. 9
8.1 Limitation on the Dataset and Domain
- Validation used one online-conversation dataset from a specific community, so the metrics’ utility on interviews and open-ended surveys remains untested.Those data types may present challenges not encountered in this study.Source: §8.1 · p. 9
8.2 Limitation on the Simulated Edge Cases
- The metrics identified simulated flooding and hallucination, but their ability to diagnose subtler real-world coding issues still needs validation.Proposed tests include novice or non-expert coders and subtly flawed AI prompts.Source: §8.2 · pp. 9–10
8.3 Limitation on Missing Critical Codes
- Because the Aggregated Code Space combines existing coders’ outputs, it cannot detect critical concepts absent from every individual code space.The authors identify coder diversity and multiple human or model perspectives as ways to mitigate this blind spot.Source: §8.3 · p. 10
8.4 Limited Interpretations on Deviant LLM Behaviors
- Although three of four models behaved similarly, Gemini-2.5-Pro merged more codes, changing raw metric values despite relatively stable rankings.The authors leave open whether this deeper merging is preferable to human researchers.Source: §8.4 · p. 10
A Potential Risks
- The paper identifies improper application or misinterpretation of its metrics as potential risks.Specific risks are discussed in Appendix A.Source: A Potential Risks · p. 13
A.1 Misinterpretation of Metrics
Metric values can be mistaken for definitive quality measures, while the Aggregated Code Space depends on the coding team’s diversity and rigor. Using cloud-based APIs can also expose confidential or personally identifiable data.
A.1 Misinterpretation of Metrics
- Treating metrics as definitive quality scores can discourage junior researchers from developing interpretive skills through comparison, reflection, and consensus-building.The authors recommend using metrics as diagnostic prompts for deeper qualitative inquiry, not as a substitute for it.Source: §A.1 · p. 13
A.2 Dependency on the Coding Team
- If the team shares a bias or misses a critical theme, the Aggregated Code Space inherits that omission, and reliance on metrics alone risks automation bias.The authors encourage diverse teams, multiple LLMs, and engagement with the code-network visualization.Source: §A.2 · p. 13
A.3 Privacy and Data Security
- Using cloud-based APIs to process qualitative data risks exposing confidential or personally identifiable information, so researchers must follow IRB protocols and privacy regulations.The study prioritized local, open-source models to protect sensitive data.Source: §A.3 · p. 13
B.1 Regression Results for Hypothesis 1
The appendix presents regression outputs for Hypothesis 1 on condition effects and coder rankings for Hypothesis 2. The supplied passages specify the analyses and table scopes, but not the regression estimates or ranking outcomes.
B.1 Regression Results for Hypothesis 1
- The five OLS models use Condition 1 as the baseline, dummy-code condition, sum-code coder identity, and use HC3 robust standard errors.Standard errors are unavailable for some parameters because of rank deficiency in constraints.Source: §B.1 · pp. 13–14
- Table 2 reports the regression output for consolidated code count, the outcome used to test whether condition affects the number of consolidated codes.The appendix describes this as Regression 1.Source: Table 2 · p. 13, §B.1 · p. 13
- Tables 3–6 report regression outputs for Coverage, Overlap, Novelty, and Divergence, the code-level metrics tested for condition effects.The appendix describes these as four separate metric models.Source: §B.1 · p. 13, Table 3 · p. 14, Table 4 · p. 14
B.2 Coder Rankings for Hypothesis 2
- Tables 7–10 give rankings for eight coders and two groups per condition across Coverage, Overlap, Novelty, and Divergence.Rel denotes the strength of evidence that a coder performs better than the next one in the ranking chain.Source: §B.2 · p. 14, Table 7 · p. 15, Table 8 · p. 15
- Ranking evidence thresholds come from Tukey’s HSD comparisons following a one-way ANOVA; Conditions 1 and 2 are deterministic because they do not use probabilistic LLM merging.The appendix marks these deterministic conditions with >>>.Source: §B.2 · p. 14
B.3 Regression Results for Hypotheses 3 and 4
The appendix reports regression analyses for Hypotheses 3–5 and coder rankings for Hypothesis 6. It describes the analyses and ranking comparisons, but the supplied passages do not report their results.
B.3 Regression Results for Hypotheses 3 and 4
- The H3–H4 appendix reports OLS regressions testing merging-model effects on metrics and consolidated code counts, and how condition, model, and coder identity explain outcome variation.Condition 3 is the baseline; model and coder identity use Average as the baseline, with HC3 robust standard errors.Source: §B.3 · p. 16, Table 11 · p. 16, Table 12 · p. 16
B.4 Regression Results for Hypothesis 5
- For H5, OLS models predict each metric’s coefficient of variation from condition, merging model, and coder identity across Conditions 3 and 4.The appendix includes regression-output tables for consolidated code count and Coverage, Overlap, Novelty, and Divergence.Source: §B.4 · p. 17, Table 16 · p. 17, Table 17 · p. 17
B.5 Coder Rankings for Hypothesis 6
- For H6, coder averages are ranked separately by metric under four evaluation models in Conditions 3 and 4, with Rel indicating evidence strength between adjacent coders.The ranking tables cover Coverage, Overlap, Novelty, and Divergence.Source: §B.5 · p. 18
- Ranking significance symbols reflect Tukey’s HSD post-hoc comparisons after a separate one-way ANOVA for each condition.The symbols distinguish p-value bands from p ≤ 0.001 through p > 0.05.Source: §B.5 · p. 18