Source-linked AI summary
Designing an Auditable LLM-Supported Workflow for Qualitative Thematic Analysis
Nadia Jul Jeldtoft, Tariq Yousef
TL;DR
LLM-supported qualitative analysis offers scale but lacks methodological transparency about how qualitative methods become computational procedures. This paper presents an auditable, privacy-preserving two-phase TA workflow combining interpretative LLM inference with deterministic control; exploratory evaluation found comparable code-level coverage and stronger analytical justifications, but fewer and broader themes than human analysis.
Problem
Existing LLM-supported TA approaches provide limited methodological transparency about computational operationalization, workflow design, and preservation of qualitative principles.
Method
The paper derives five design principles and implements them in a privacy-preserving workflow that separates interpretative LLM inference from deterministic procedural control while preserving links to source material.
Results
The workflow produced expert-rated code coverage broadly comparable to human annotations, higher-rated analytical justifications, and fewer and substantially broader themes.
Takeaways & Limitations
The study demonstrates the feasibility of methodologically explicit, traceable, and privacy-preserving computational operationalization of inductive and latent TA.
Takeaways & Limitations
The evaluation is exploratory because it uses a limited sample, one small open-weight model, one full human comparison, and no independent assessment of generated-theme quality.
Abstract
from arXiv · showhide
Large Language Models (LLMs) offer new possibilities for scaling qualitative analysis, but existing applications often provide limited methodological transparency regarding how qualitative methods are translated into computational procedures. This paper presents an auditable and privacy-preserving computational operationalization of inductive and latent Thematic Analysis (TA). This paper first derives five design principles from the methodological requirements of TA and the conditions introduced by LLM-based inference: preserving interpretative context, maintaining traceable relationships between empirical material and analytical outputs, representing analytical constructs and reasoning explicitly, constraining LLM inference to interpretative tasks, and enabling privacy-preserving local deployment. Second, it presents a proof-of-concept for a two-phase workflow that operationalizes these principles by combining interpretative LLM inference with deterministic procedural control to generate codes, analytical justifications, themes, and theme descriptions while preserving explicit links to the source material. Third, it proposes an evaluation framework combining structural comparison with human-led TA and independent expert assessment of analytical quality. The evaluation is conducted on semi-structured Danish interview transcripts. and the results shows that the workflow produces code-level outputs with coverage broadly comparable to human annotations and highly rated analytical justifications, while generating a more compressed thematic structure characterized by fewer and broader themes. The findings demonstrate the feasibility of auditable LLM-supported TA through a modular workflow designed to scale to larger datasets, accommodate different LLMs, and support transfer across research domains, with domain adaptation primarily requiring adjustments to the prompting strategy.
1 Introduction
The paper addresses the limited methodological transparency of LLM-supported Thematic Analysis by presenting an auditable computational operationalization that preserves qualitative principles and analytical traceability.
- LLMs expand possibilities for analyzing large collections of unstructured textual data and are increasingly applied in qualitative research.
- Thematic Analysis requires contextual interpretation, iterative coding, theme development, and explicit links between data, codes, and themes.
- Existing LLM-supported approaches emphasize model capabilities while providing limited accounts of how qualitative methods are translated into computational procedures.
- The paper presents a proof of concept for an auditable workflow that documents analytical operations and links outputs to empirical source material.
- The workflow separates interpretative LLM inference from deterministic computation and is evaluated against expert-generated TA using Danish interview transcripts.
- The paper contributes five design principles, an auditable workflow architecture, and an evaluation framework combining structural comparison with expert assessment.
2 Related work
Prior work shows that LLMs can support thematic-analysis tasks, but key decisions governing computational operationalization, methodological transparency, and data governance remain underspecified.
- Existing LLM-supported TA studies provide limited descriptions of computational architecture, analytical representations, and workflow design decisions.
- Underspecified decisions include analytical-unit representation, contextual management, intermediate-product linkage, and maintenance of analytical traceability.
- Many existing solutions rely on proprietary cloud-based models and API infrastructures, raising challenges for governance and sensitive qualitative data.
- The related work motivates workflow architectures that embed TA’s methodological principles while preserving traceability, transparency, and data governance.
3 Background Theory
The paper frames inductive and latent Thematic Analysis as an interpretative, iterative process in which meaningful codes are developed into broader themes while quality depends on the analytical process and its documentation.
- Thematic Analysis develops themes through iterative interpretation of meaningful patterns, with representations depending on analytical focus and epistemological perspective.
- Inductive and latent TA derives codes and themes from underlying patterns of meaning rather than predefined computational rules.
- The workflow has two phases: assigning codes to interview segments, then constructing broader themes from shared meaning across codes.
- TA quality criteria are organized into analytical process, representation of analytical constructs, traceability of analytical relationships, and analytically valid outputs.
- Process, representation, and traceability guide computational design, whereas analytically valid outputs provide evaluation criteria.
4 Five design requirements for adapting LLMs for TA
The paper derives five computational design requirements for LLM-supported TA: preserve context, traceability, explicit analytical constructs, constrained inference, and privacy-preserving deployment.
- Motivation: LLMs can identify and organize patterns of meaning, but computational design must preserve TA’s methodological characteristics while accommodating probabilistic inference.
- Principle 1: Preserve interpretative context: Analytical units must preserve enough local context for interpretation while remaining small enough to support stable inference within the model’s context window.
- Principle 2: Maintain traceability: Codes, themes, justifications, and descriptions must remain linked so the complete analytical path from themes to original data can be reconstructed.
- Principle 3: Represent analytical constructs and reasoning explicitly: Codes and themes should have clear labels, descriptions, and inspectable rationales grounded in empirical material rather than remaining implicit in model inference.
- Principle 4: Constrain LLM inference: LLM inference should be restricted to analytical tasks requiring contextual judgment because its outputs arise through probabilistic rather than deterministic procedures.
- Principle 5: Enable privacy-preserving deployment: Privacy-preserving deployment requires local inference with open-weight language models so sensitive data are not exposed to external services.
5 Data
The study uses semi-structured Danish interviews to develop and test a computational workflow for interpretative thematic analysis. A stratified, anonymized sample preserves variation in interview structure while separating development from evaluation data.
- Data source: The dataset comes from the FREI project on family life and social change in Denmark between the 1950s and 1970s.The interviews contain rich, context-dependent narratives requiring interpretative analysis.
- Dataset split: 99 transcripts remained after one interview was excluded for poor transcription quality, with 94 assigned to development and 5 to testing.The split was performed at the interview level to separate model development from evaluation.
- Segmentation: Interview transcripts were segmented into analytical units consisting of two consecutive question-answer pairs.This unit balances contextual interpretation against the risk of overly long inputs and unstable or overlapping codes.
6 Methodology
The methodology separates LLM-based interpretation from deterministic procedural control in a two-phase workflow. It generates traceable codes, justifications, themes, and descriptions while supporting local, privacy-preserving deployment.
- Workflow architecture: The workflow assigns interpretative coding and theme construction to the LLM, while programmatic procedures enforce links among data, codes, and themes.This division aligns computational roles with the methodological requirements of thematic analysis.
- Auditability: Generated codes and themes remain explicitly linked to their originating data, with analytical and procedural steps systematically represented.This separation supports computational traceability and reproducibility.
- Phase 1: From text to codes: Phase 1 produces segment-level codes with explicit justifications, using domain-specific prompting, predefined schemas, and constrained output generation.The design accommodates many-to-many relationships between segments and codes.
- Phase 2: From codes to themes: Phase 2 maps multiple codes to themes iteratively, updating the thematic map when an accumulated pool reaches 20 unassigned codes.Each update uses a random subset of 12 codes, while prior themes guide subsequent assignments.
- Phase 2: From codes to themes: Phase 2 outputs include themes, descriptions, code-assignment justifications, and retained unassigned codes within a traceable structured dataframe.An inverse codes-in-themes mapping documents assignments and their analytical reasoning.
- Design implications: The proof-of-concept preserves context sensitivity, data traceability, and coherent code-theme representation while supporting local deployment and domain adaptation through prompting changes.The workflow uses a smaller open-weight model for interpretative tasks and keeps data within controlled environments.
7 Evaluation framework
The evaluation framework treats human analyses as reference points rather than a single correct answer and combines structural comparison with independent expert assessment. It evaluates both analytical organization and output quality while recognizing the study’s exploratory scope.
- Evaluation framework: Because inductive thematic analysis permits multiple valid interpretations, accuracy, semantic similarity, and agreement metrics are insufficient as stand-alone quality measures.The framework therefore combines structural comparison with independent expert assessment.
- Evaluation framework: The framework evaluates computational thematic analysis through structural characteristics and the analytical quality of generated outputs.Structural analysis examines organization into codes and themes, while quality assessment examines meaningfulness and thematic-analysis criteria.
- Annotated datasets: Three experienced researchers independently produced annotations using the same methodological procedure.One researcher annotated the full test set, while two independently coded a smaller subset.
- Annotated datasets: Evaluation Dataset A contains a full thematic analysis of the test set, while Evaluation Dataset B supports focused code-level comparison across multiple annotators.Dataset A includes 291 segments and covers both coding and theme generation; Dataset B is a smaller subset for Phase 1.
- Expert assessment: Expert assessment used randomized, blinded outputs and five-point Likert scales to assess code coverage and justification quality.Code coverage concerns capturing excerpt meaning; justification quality concerns clarity and adequacy of analytical reasoning.
- Implications: The evaluation framework is presented as transferable to future computational operationalizations of qualitative analytical methods.Its scope extends beyond the present workflow’s application.
- Limitations: The analysis is exploratory because the datasets are relatively small and relies primarily on descriptive statistics rather than significance testing.The findings should be interpreted as indicative rather than conclusive, although the framework can be extended to larger datasets.
8 Results
The workflow produces code-level analyses broadly comparable to human annotations, with highly rated justifications, but organizes codes into fewer and broader themes. Structural comparisons also show broadly comparable coding volume and code diversity, alongside a more compressed thematic structure.
- Evaluation approach: The evaluation combines structural comparison with human-led analyses and independent expert assessment of code meaningfulness and justification quality.The two evaluation components characterize organization into codes and themes, then assess whether codes and justifications are analytically meaningful and grounded.
- Code-level structure: The workflow produces broadly comparable numbers of codes per segment, although it generally generates slightly fewer codes overall than human annotators.In one evaluation dataset, the workflow instead averages 2.33 codes per segment versus 1.85 for the human annotator, indicating slightly more inclusive coding there.
- Code-level structure: The LLM generates fewer unique code labels than human annotators, but the relatively small differences suggest broadly comparable code diversity.The result is notable because the LLM processes segments independently without memory of prior coding decisions; the domain-specific prompting strategy is offered as one plausible explanation.
- Thematic structure: The workflow generates fewer but substantially larger themes than the human analysis, producing a more compressed thematic structure.This difference appears in the mean, median, and maximum number of codes per theme.
- Thematic structure: Codes are concentrated in a relatively small number of broad LLM-generated themes, whereas human themes distribute codes more evenly across more medium-sized themes.The authors suggest this concentration may reflect early thematic consolidation, although that relationship was not systematically evaluated.
- Expert assessment: Expert assessment finds code-level outputs comparable to human annotations, while analytical justifications receive consistently higher ratings for the workflow.The LLM has a mean justification score of 3.80 versus 2.73 for human annotators and scores higher in four of five interview segments.
9 Conclusion
The paper operationalizes inductive and latent Thematic Analysis through explicit design principles, separating interpretative LLM inference from deterministic procedural control. Preliminary evaluation supports feasibility but also reveals fewer and broader themes than human analysis, with important evaluation limitations.
- Contribution: Five design principles guide the operationalization: preserving context, traceability, explicit reasoning, constrained inference, and privacy-preserving local deployment.These principles translate methodological requirements of Thematic Analysis into computational design criteria.
- Workflow: Interpretative LLM inference generates codes, themes, and analytical justifications, while deterministic procedures structure, validate, and preserve their relationships to source data.Auditability is produced by the workflow architecture rather than by language-model transparency alone.
- Evaluation: At the coding level, workflow outputs achieved expert-rated coverage broadly comparable to human annotators, while analytical justifications received higher mean ratings.The evaluation provides preliminary support for the feasibility and interpretability of the approach.
- Evaluation: The workflow generated fewer and substantially broader themes than the human analysis, suggesting that sequential Phase 2 processing favors early thematic consolidation.This indicates that computational workflow design actively shapes the resulting analytical structure.
- Limitations: The findings require cautious interpretation because evaluation used a limited sample, one small open-weight model, one human comparison, and no independent assessment of generated-theme quality.Future evaluation should use more diverse datasets, models, annotators, and expert evaluators.