PaperlayerView exact revision on arXiv

Source-linked AI summary

LOGOS: LLM-driven End-to-End Grounded Theory Development and Schema Induction for Qualitative Research

Xinyu Pi, Qisen Yang, Chuong Nguyen

arXiv:2509.24294v2cs.CLcs.HC

TL;DR

Grounded theory can yield insights from qualitative data, but manual coding carries substantial expert costs and existing computational tools remain only partly automated. LOGOS automates codebook induction and reuse, alongside a five-dimensional evaluation metric and train-test protocol. It outperforms coding- and RAG-based workflows across five datasets and reports 80.4% average expert-schema matching on the Multi-agent System Failure corpus.

  • Problem

    Grounded-theory coding and synthesis require substantial expert labor, while existing tools either stop at retrieval and organization or retain expert supervision at critical steps.

    Source: §1 · p. 2

  • Method

    LOGOS combines a relation-aware code graph and iterative refinement to induce a codebook from raw text and reuse it for coding new datapoints, with a five-dimensional metric and train-test evaluation protocol.

    Source: §1 · p. 2

  • Results

    LOGOS surpasses four coding- and RAG-based workflows across five datasets and achieves 80.4% average expert-schema matching on the Multi-agent System Failure corpus across configurations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LOGOS indicates potential to reduce expert labor without discarding theoretical nuance, within the evaluated corpora and schema comparisons.

    Source: §6 · p. 10

    LOGOS models hierarchical semantic relations but does not yet capture richer causal, temporal, or processual relations between codes.

    Source: §A.1 · p. 15

Abstract

from arXiv · show

Grounded theory offers deep insights from qualitative data, but its reliance on expert-intensive manual coding presents a major scalability bottleneck. Existing computational tools either fail on full automation or lack flexible schema construction. We introduce LOGOS, a novel, end-to-end framework that fully automates the grounded theory workflow, transforming raw text into a structured, hierarchical theory. LOGOS integrates LLM-driven coding, semantic clustering, graph reasoning, and a novel iterative refinement process to build highly reusable codebooks. To ensure fair comparison, we also introduce a principled 5-dimensional metric and a train-test split protocol for standardized, unbiased evaluation. Across five diverse corpora, LOGOS consistently outperforms strong baselines and achieves a remarkable average $80.4\%$ alignment with an expert-developed schema on complex datasets. LOGOS demonstrates a potential to democratize and scale qualitative research without sacrificing theoretical nuance.

1 INTRODUCTION

Grounded theory builds explanatory categories from empirical data, but manual coding and interpretation make the process costly, while existing tools stop short of autonomous interpretive synthesis. LOGOS automates codebook induction and reuse, and the paper introduces a train-test evaluation protocol and reports strong benchmark and expert-schema alignment.

1 INTRODUCTION

  • Grounded theory develops concepts and explanations inductively from empirical data, with open, axial, and selective coding moving from datapoint tags toward a coherent framework.The paper uses “grounded theory development” and “schema induction” interchangeably.Source: §1 · p. 1
  • Manual grounded-theory work requires substantial expert effort, while commercial tools support retrieval and organization and academic systems still require supervision at critical interpretive steps.The cited example describes six experts spending more than 20 hours each coding a 200-trace dataset.Source: §1 · p. 2
  • LOGOS combines a relation-aware conceptual graph and iterative refinement to automate open, axial, and selective coding, then reuses its induced codebook to code new and held-out datapoints.Source: §1 · p. 2
  • 80.4% average expert-schema matching on the Multi-agent System Failure corpus, while LOGOS surpasses four coding- and RAG-based workflows across five datasets.The matching rate is reported across various configurations.Source: §1 · p. 2
  • A five-dimensional codebook metric and train-test split protocol aim to support standardized comparison with less bias when expert codebooks are unavailable.The dimensions are reusability, semantic fitness, semantic coverage, parsimony, and consistency.Source: §1 · p. 2

2 RATIONALE AND CONCEPTUALIZATION

The paper formulates grounded-theory development as constructing a codebook from a research question and qualitative corpus. It argues that interpretation is needed to group datapoints by underlying phenomena rather than surface similarity, and evaluates codebooks along five qualitative dimensions.

2 RATIONALE AND CONCEPTUALIZATION

  • Grounded-theory development constructs a codebook from research question Q and corpus D to name recurring phenomena, organize relations, and support a question-focused report.Source: §2 · p. 2
  • A codebook is defined as Σ = (C, R): descriptive codes tag salient datapoint properties, while typed relations organize codes into a semantic hierarchy.The listed relation types include subsumption, equivalence, orthogonality, and none.Source: §2 · pp. 2–3
  • The proposed codebook criteria are reusability, accuracy, comprehensiveness, parsimony, and consistency; traditional LLM metrics such as perplexity are described as unsuitable for this evaluation.Source: §2 · p. 3
  • Naive embedding-based clustering can group datapoints by task domain rather than shared failure reason, so complex grounded-theory work may require interpretation of syntactic, semantic, and pragmatic attributes.The passage notes that this interpretation may require heavy reasoning, advanced language understanding, and systematic expertise external to a datapoint.Source: §2 · p. 3

3 METHOD

LOGOS automates grounded-theory development through inductive codebook construction followed by deductive coding. Its method combines code relationships, iterative refinement, and train–test evaluation with five codebook-quality metrics.

3.1 THE LOGOS FRAMEWORK

  • LOGOS chunks text for open coding, clusters embedded codes, and generates high-level cluster codes as axial coding.The procedure uses Qwen3-32B for code generation and Qwen3-embed-0.6B with K-means for clustering.Source: §3.1 · pp. 3–4
  • Within clusters, LOGOS classifies code-pair relations, infers transitive and equivalence links, then merges mutual codes and folds or drops low-frequency codes.Merged groups retain the highest-scoring code; codes appearing fewer than six times are folded into parent codes when possible, otherwise dropped.Source: §3.1 · pp. 4–5

3.2 ITERATIVE CODEBOOK REFINEMENT MECHANISM

  • Iterative refinement replaces codes with better-matching alternatives to improve reusability, using semantic and graph-based candidates, a pruned pool, LLM revision, and output validation.Candidate generation draws on up to five semantically similar codes and up to five graph-related codes for connected codes.Source: §3.2 · p. 5

3.3 STATISTICAL METRICS FOR CODEBOOK QUALITY

  • LOGOS fits the codebook on training data, then codes test datapoints deductively by retrieving existing codes that an LLM may select but not invent.This train–test protocol evaluates the train-derived codebook on held-out datapoints.Source: §3.3 · p. 5
  • The evaluation measures reusability on test data, LLM-scored descriptive fitness, and LLM-scored coverage of research-question-relevant datapoint aspects.Reusability is the ratio of train-derived codes used at least once in test prediction to all codes in the cleaned codebook.Source: §3.3 · pp. 5–6
  • The remaining measures penalize semantic redundancy and compare train–test code distributions for stability, assuming the splits are independent and identically distributed.A higher stability score indicates that the codebook maintains consistency across data splits.Source: §3.3 · p. 6
  • The five metric scores can be combined as a weighted sum; this work uses equal weights for simplicity.The authors note that semantic adequacy and reusability could instead receive higher weights.Source: §3.3 · p. 6

4 EXPERIMENT AND DISCUSSION

Across five datasets, LOGOS is reported as the top-performing method and reaches 80.4% average alignment with the MAS expert schema. Experiments also examine iterative refinement and compare LOGOS with retrieval systems on schema-induction questions.

4.1 DATASET AND EXPERIMENT SETTING

  • The evaluation covers Ali Abdaal transcripts, podcast transcripts, UIST abstracts, multi-agent failure records, and generated GSM8K math solutions.The MAS experiment uses 230 programming-task datapoints; the Math Failure dataset contains 316 records.Source: §4.1 · pp. 6–7

4.2 EXPERIMENT 1 - STATISTICAL METRICS COMPARISONS

  • LOGOS is consistently top-performing in Table 1; its global reusability is 0.892, versus LLOOM's 0.921, while its fitness is 0.538 and coverage is 0.565.OpenCoding's global reusability, fitness, and coverage are 0.209, 0.641, and 0.672; LLOOM's fitness and coverage are 0.482 and 0.463.Source: §4.2 · p. 7, Table 1 · p. 7, Figure 2 · p. 7

4.3 EXPERIMENT 2 - EFFECTIVENESS OF LOGOS ITERATIVE REFINEMENT

  • Across ten iterations, reusability rises while codebook size and semantic adequacy decline; parsimony remains relatively stable, while consistency decreases.Semantic adequacy here refers to codebook fitness and coverage.Source: §4.3 · pp. 7–8, Figure 3 · p. 8

4.4 EXPERIMENT 3 - CASE STUDY: GLOBAL SENSE-MAKING QUESTION ANSWERING

  • On the Math Failure dataset, GraphRAG and LightRAG fail to capture failure patterns because their extracted graphs omit logical dependencies, procedures, formula choices, and value filling; LOGOS summarizes recurring patterns.The case study asks, “What are LLMs’ failure patterns in mathematical reasoning?”Source: Figure 4 · p. 8, §4.4 · p. 9

4.5 EXPERIMENT 4 - CASE STUDY: ALIGNMENT WITH HUMAN GROUND TRUTH

  • 80.4% average alignment with the MAS expert schema was achieved after five LOGOS iterations, with 88.2% under the best GPT-O3 aggregator configuration.The 286-code hierarchical codebook was aggregated into 30 clusters; a ground-truth code counted as matched if it matched at least one cluster.Source: §4.5 · p. 9, Figure 5 · p. 9

5 RELATED WORK

Prior qualitative-analysis tools range from retrieval aids and expert-centered collaboration to automated thematic-analysis pipelines, while specialized schema-induction work often assumes restricted domains or graph formalisms. The related work positions LOGOS as an automated approach to hierarchical, relational, iterative grounded-theory development.

Computer-Aided Grounded Theory Development

  • Commercial CAQDAS tools such as NVivo and MAXQDA speed retrieval but stop short of interpretive work, while academic systems often retain expert judgment.Cody and PaTAT recommend codes based on human-generated patterns; CoAICoder supports collaboration among human coders.Source: §5 · p. 9
  • LLM-assisted qualitative-analysis systems scaffold collaboration or provide analytical traces, but experts generally remain responsible for distilling implicit meanings from raw data.The passage contrasts these systems with fully automated workflows.Source: §5 · p. 9
  • End-to-end thematic-analysis pipelines aggregate datapoint codes into themes, whereas grounded theory seeks hierarchical, relational, iterative structures that can generalize to future data.LOGOS is described as constructing a hierarchical code graph, refining reusability and parsimony, and reusing the schema deductively on held-out data.Source: §5 · p. 10

Automated Theoretical Induction for Specialized Domains

  • Automated event-schema induction commonly uses knowledge graphs and next-event prediction, while broader induction approaches often rely on domain-, ontology-, or topology-specific assumptions.The related-work section frames LOGOS as a general-purpose alternative to these constraints.Source: §5 · p. 10

6 CONCLUSION

LOGOS combines an end-to-end grounded-theory pipeline with a standardized evaluation protocol and performs well across five corpora. Its current codebook typology captures hierarchical semantic relations, with richer relation types and computational efficiency left for future work.

6 CONCLUSION

  • LOGOS combines semantic clustering, relation-aware graphs, iterative refinement, a five-dimensional codebook metric, and a train/test protocol, and outperforms coding- and RAG-based baselines on five corpora.It attains 88.2% alignment with expert schemas on MAS.Source: §6 · p. 10
  • Current codebooks model hierarchical semantic relations, leaving richer causal and temporal relations and computational-cost optimization for future work.Source: §6 · p. 10

A.1 LIMITATIONS AND ETHICAL CONSIDERATIONS

LOGOS's codebook typology is limited to hierarchical semantic relations, and the pipeline can be computationally costly. The authors also warn that LLM biases and interpretive errors require researcher review and responsible data governance.

A.1 LIMITATIONS AND ETHICAL CONSIDERATIONS

  • The codebook typology models static hierarchical semantics, not causal, temporal, or processual relations between codes.Multiple LLM passes and codebook iterations may also be costly for very large datasets or researchers without substantial compute.Source: §A.1 · p. 15
  • LLM biases, hallucinations, and overgeneralization may distort codebooks; researchers should review and, where needed, edit or contest the generated codes and themes.Researchers remain responsible for consent, privacy, and data governance for input corpora.Source: §A.1 · p. 15

A.2 ON EVALUATING SCHEMA ALIGNMENT

LOGOS captures 88.2% of concepts in the expert schema, with the authors treating alignment as conceptual containment rather than exact replication. The appendix illustrates how expert categories and failure-mode items map to LOGOS clusters, while noting an unused cluster.

A.2 ON EVALUATING SCHEMA ALIGNMENT

  • 88.2% conceptual alignment with the expert-developed schema is reported, with 30 LOGOS clusters compared with 17 expert codes.The authors frame this as recall and coverage: broader, finer-grained clusters can contain expert concepts without replicating them one-for-one.Source: §A.2 · p. 15
  • FC categories map one-to-many onto broadened LOGOS cluster sets, including specification issues and inter-agent misalignment.The listed mappings include FC-1 to C01, C02, C14, and C15, and FC-2 to C06, C07, C08, C09, and C25.Source: §A.2 · p. 16
  • Ground-truth failure-mode items map to clusters for several topics, while conversation reset and information withholding have no assigned cluster.For example, premature termination maps to C10 and C19, while no or incomplete verification maps to C11 and C24.Source: §A.2 · p. 16
  • The listed unused cluster is C17, covering ethical, legal, and normative reasoning gaps.Source: §A.2 · p. 17

A.3 EXTENSION OF HUMAN ALIGNMENT CASE STUDY IN 4.5

The extended human-alignment case study reports 100% matching on Vibe Coding and 80% matching on the Y Combinator dataset.

A.3 EXTENSION OF HUMAN ALIGNMENT CASE STUDY IN 4.5

  • 100% matching on Vibe Coding and 80% matching on the Y Combinator dataset are reported for LOGOS.Source: §A.3 · p. 17

A.3.1 Y COMBINATOR ACCEPTED APPLICATION DATASET RESULT:

The Y Combinator case study uses transcripts from ten accepted startup application videos and presents a codebook spanning pitch, problem, solution, and traction themes. Its mappings assign some expert items to clusters while leaving opening and closing items unmatched.

A.3.1 Y COMBINATOR ACCEPTED APPLICATION DATASET RESULT:

  • The dataset comprises 10 accepted Y Combinator application videos, each 45 seconds to 1 minute 37 seconds long, averaging approximately 237 transcript words.Source: §A.3.1 · p. 17
  • The listed codebook clusters cover value propositions, pitch delivery, problem narratives and validation, competitive positioning, and technical capabilities.Source: §A.3.1 · p. 17
  • Further clusters cover solution positioning, traction, monetization, launch metrics, partnerships, market expansion, founder engagement, audience engagement, momentum, and credibility.Source: §A.3.1 · pp. 17–18
  • The listed ground-truth mappings leave opening and closing items unassigned, while problem definition and validation maps to clusters C03 through C06.Source: §A.3.1 · p. 18

A.3.2 VIBE CODING DATASET RESULT:

The Vibe Coding analysis uses the YouTube portion of the dataset and presents eight clusters spanning AI behavior, collaboration, customization, design, development, evaluation, community, and tooling.

A.3.2 VIBE CODING DATASET RESULT:

  • The analysis uses 33 accessible YouTube videos from the dataset’s 36 YouTube videos and excludes its Twitch streams.It uses transcripts only up to durations reported in the original paper; average transcript length is 3,342 words and video durations range from 2 minutes 35 seconds to 45 minutes 16 seconds.Source: §A.3.2 · p. 18
  • Clusters C1–C4 cover AI behavior and limitations, co-creation, output customization, and AI-driven content, UI/UX, and visual design.Source: §A.3.2 · p. 18
  • Clusters C5–C8 cover development efficiency, AI evaluation and oversight, community and engagement, and integration and interoperability.Source: §A.3.2 · p. 18

A.3.3 LLM-AS-A-JUDGE ALIGNMENT WITH HUMAN

This section reports human–LLM agreement and related LOGOS evaluation details, alongside coded themes and code lists from UIST abstracts and podcast data. It also describes the comparison systems used for coding, clustering, retrieval, and thematic aggregation.

A.3.3 LLM-AS-A-JUDGE ALIGNMENT WITH HUMAN

  • 0.7297 for fitness and 0.6952 for coverage were the Pearson correlations between human and LLM scores on 50 LOGOS examples.Two authors rated Iteration 3 outputs on the MATH Failure dataset from 0–10; Qwen3-32B evaluated the same examples using the question, correct and incorrect answers, and generated code.Source: §A.3.3 · p. 19

A.4 THEMATICLM BASELINE REPORT

  • The supplied A.4 report fragment lists Method, AliAbdaal, Podcast, Abstracts, MAS, and Math Failure.The fragment does not provide baseline results or comparisons.Source: §A.4 · p. 19

A.5 DISTILLED CLASSIFIER RESULTS

  • Larger models significantly improved distilled-classifier performance, while global fine-tuning performed similarly to LoRA fine-tuning.On a 350k-pair Wikipedia corpus, mutual labels were about 10% of the other labels, and oversampling-based balancing was effective in this label-imbalance case.Source: §A.5 · pp. 19–20

A.6.1 PAPER ABSTRACTS DATASET QUESTIONS AND TOP 25 FREQUENT CODES: •

  • The UIST abstract codes for research problem framings emphasize user experience, human-device interaction, input methods, physical-digital interaction, and user-centered evaluation.Other listed themes include novel interaction techniques and empirical or user-study validation.Source: §A.6.1 · pp. 20–21
  • The UIST abstract codes for claimed contributions include real-time interaction, interface design flexibility, empirical validation, and integration of physical and virtual interaction.The lists also include user-study-driven evaluation and interaction frameworks and techniques.Source: §A.6.1 · pp. 21–22
  • The UIST abstract codes for foundational methods include prototype validation, user studies, interaction-technique development, HCI research, system design, and modular design.Additional listed codes cover gestural input, prototyping, real-time physical-digital interaction, and implementation complexity.Source: §A.6.1 · pp. 22–23
  • Paper-abstract Q4 codes emphasize adaptive, real-time, and hybrid physical-digital interaction alongside user-centered and collaborative designs.The listed codes also include task-oriented interaction, context awareness, and integration of physical gestures into digital interaction.Source: §A.6.1 · pp. 23–24
  • Paper-abstract Q5 codes center on implementation and prototyping, with user studies, technical benchmarking, and illustrative case studies also represented.The listed themes include implementation-based validation, practical demonstrations, controlled user studies, and technical benchmarking against existing techniques.Source: §A.6.1 · p. 24

A.6.2 PODCAST DATASET QUESTIONS AND TOP 25 FREQUENT CODES: •

  • Podcast Q1 codes foreground interdisciplinary convergence and synergy, alongside democratization of technical knowledge, skills, and creation.Other listed themes include mentorship, learning through failure, and interplay between technical and creative skills.Source: §A.6.2 · p. 25
  • Podcast Q2 fragments identify long-term societal impact, human-AI collaboration, ethical considerations, and balancing innovation as future-facing themes.The supplied passages provide theme labels rather than a complete Q2 code list.Source: §A.6.2 · p. 26
  • Q2’s frequent codes emphasize interdisciplinary problem-solving, ethical AI, long-term societal impacts, curiosity, and human–AI collaboration.The list also includes technological innovation, societal values, democratized access to technology, and personal passion in professional life.Source: §A.6.2 · p. 26
  • Q3’s frequent codes connect creativity and technical execution with human-centric values, adaptation to changing platforms, and interdisciplinary problem-solving.The list includes both knowledge integration and collaboration as approaches to solving problems.Source: §A.6.2 · p. 27
  • Q3’s codes also foreground AI’s societal implications, public engagement in tech ethics, long-term planning, curiosity, social connection, and natural intelligence as a design reference.Other listed themes include interdisciplinary collaboration, perspective in evaluating challenges, creativity in technical fields, and ethical responsibility in technological development.Source: §A.6.2 · p. 27
  • Q3’s remaining codes include perspective-taking, interdisciplinary approaches to AI, ethical responsibility, human-centered AI design, and technology outpacing societal adaptation.Long-term societal planning and integration of creativity into technical fields also recur in the list.Source: §A.6.2 · p. 27
  • Q4’s frequent codes link mentorship and peer interaction with research-to-application dynamics, personal purpose, technical literacy, creativity, and interdisciplinary exchange.The list also includes academic-industry synergy and mentorship’s role in bridging academic and professional gaps.Source: §A.6.2 · p. 28
  • Q4’s other codes highlight long-term technological vision, intrinsic motivation, expanded creative possibilities, grassroots inclusion, art–engineering interplay, risk-taking, and ethical and societal impacts.The list also names public engagement as a catalyst for scientific relevance and interdisciplinary collaboration in innovation.Source: §A.6.2 · p. 28
  • Podcast codes emphasize interdisciplinary problem-solving and collaboration, education and knowledge sharing, and long-term technological impact and broader access to knowledge.The listed codes also include curiosity-driven exploration, resilience through iterative learning, public understanding of technical ideas, and ethical considerations in AI development.Source: §A.6.2 · p. 29

A.6.3 ALIABDAAL DATASET QUESTIONS AND TOP 25 FREQUENT CODES: •

  • AliAbdaal’s Q1 codes include autonomy, efficient knowledge acquisition, personal growth, fulfillment-based success, intrinsic motivation, and continuous learning.The list also includes creative expression, work-life balance, relationships, and control over one’s digital environment.Source: §A.6.3 · p. 30
  • AliAbdaal’s Q2 codes describe credibility through personal experience, validating audience struggles with shared experiences, and applying theoretical concepts practically.These codes address recurring strategies for constructing authority and credibility across the videos.Source: §A.6.3 · p. 31

A.6.3 ALIABDAAL DATASET QUESTIONS AND TOP 25 FREQUENT CODES

  • Recurring communication codes emphasize personal experience as a credibility anchor, actionable advice, supportive community, and validation of audience struggles.The codes also identify continuous learning, metaphors that simplify complex ideas, and real-world application of theory.Source: §A.6.3 · p. 31
  • Q3 themes frame adversity through learning from failure, intentional time allocation, systemic support, and a process-oriented growth mindset.Other listed themes include iterative improvement through feedback loops and intrinsic motivation as a core driver.Source: §A.6.3 · p. 32
  • Q4 codes combine personal experience with external insights and validation, alongside self-reflection, intrinsic motivation, and success beyond financial gain.The listed codes also include digital tools, long-term commitment, and iterative practice.Source: §A.6.3 · p. 33

A.7.1 LOGOS

  • LOGOS uses deterministic, multi-iteration coding and graph refinement, reusing prior representations and consolidating the final schema into a topologically ordered structure.Its default setup uses two iterations; refinement stops at the configured limit or when the graph stabilizes.Source: §A.7.1 · pp. 35–36

A.7.2 LLOOM

  • LLOOM adapts LOGOS’s iterative pipeline while retaining its distillation, embedding and clustering, and high-level code-generation operations.The implementation drops LLOOM’s quota mechanism in distillation.Source: §A.7.2 · p. 36

A.7.4 LIGHTRAG

  • LightRAG processes 500-token chunks with 50-token overlap and retrieves top-k=20 results per query using Qwen3-32B and Qwen3-embed-0.6B.Its query fallback progressively reduces top-k and token limits.Source: §A.7.4 · p. 36

A.7.5 HICODE

  • HICode generates labels deterministically and applies three rounds of LLM-based hierarchical clustering, merging cluster dictionaries after each round.The final codebook and themes from the training set are applied to the test set.Source: §A.7.5 · p. 36

A.7.6 THEMATIC-LM

  • The Thematic-LM reimplementation aggregates open codes in batches and uses a reviewer to merge labels into a global codebook at cosine similarity threshold 0.65.It omits the original paper’s quotation design and uses one persona producing 15 codes per datapoint; the training-set codebook and themes are applied to the test set.Source: §A.7.6 · p. 37

Related papers

Loading 2509.24294v2…