A Framework For Discussing LLMs as Tools for Qualitative Analysis

1 INTRODUCTION

• Researchers disagree about whether, where, and to what extent LLMs should or could support qualitative analysis.Emergent work explores deductive coding and inductive thematic analysis.Source: §1 · p. 1
• The paper proposes a framework for discussing LLM use in qualitative analysis, organized around whether an LLM proposes or refutes a model and whether researchers check its decisions directly.It identifies surfacing counter-examples for human review as a promising space for adoption.Source: §1 · p. 1

2.1 Positivism and Constructivism

• The paper uses consistency and congruence, rather than ontology, epistemology, and methodology, to characterize positivism and constructivism, while noting that this two-paradigm framing is limited.The authors describe the paradigms as a continuum and acknowledge that the broader picture is oversimplified and potentially limiting.Source: §2.1 · pp. 1–2
• Positivism seeks congruent models that correspond one-to-one with external reality, making the rejection of observation-inconsistent models relevant.Once congruence is established, the possibilities of consistent and unknown models become irrelevant to this account.Source: §2.1 · pp. 1–2
• Constructivism denies that congruent models can be established and instead explores consistent models while remaining attentive to the unknown.The unknown includes models not yet tested or conceived, as well as models fundamentally unknowable to humans.Source: §2.1 · p. 2

2.2 LLMs and The Implications of Cognitive Linguistics

• Evidence that language understanding draws on cultural and embodied experience challenges LLM use, but positivists focus on consistency of outcomes rather than how claims are produced.They still regard measuring how consistently LLMs reach congruence with reality as important.Source: §2.2 · p. 2
• Constructivists focus on how an LLM produces claims because evaluating them requires understanding that process; the value of LLM-originated claims remains uncertain.The paper presents positivist and constructivist positions as idealized explanatory characterizations, not individuals’ fixed identities across contexts.Source: §2.2 · p. 2

3 A FRAMEWORK FOR DISCUSSING LLMS FOR QUALITATIVE ANALYSIS

• The framework asks whether an LLM is proposing or refuting a qualitative model and whether a human researcher is directly checking the LLM’s decision-making.The first question highlights an asymmetry between affirming and refuting consistency; the second draws attention to human language understanding and the extent to which LLMs support or replace it.Source: §3 · p. 2

4 DISCUSSION

• Surfacing counter-examples for human review gives positivists and constructivists a shared set of questions, although they may differ on what evidence indicates inconsistency and in which contexts.Both are interested in discarding models inconsistent with observation.Source: §4 · p. 2
• Having a human researcher judge counter-examples leaves the methodological question of whether this LLM use is useful or effective for qualitative analysis.The paper argues that negating a model is ontologically neutral and that no qualitative-analysis tool is epistemologically neutral.Source: §4 · p. 2
• Researchers could analyze very large datasets by refining an initial traditional-method sample with LLM-surfaced counter-examples for human review.The proposed sequence resembles theoretical sampling used to develop and refine emerging categories [4, p. 205].Source: §4 · p. 2
• Whether LLM-surfaced counter-examples improve or worsen qualitative-analysis results remains an open question.The proposed restriction keeps consistency claims with human researchers and gives them an opportunity to revise a model before an LLM-identified observation leads to its rejection.Source: §4 · p. 2
• The authors present counter-example review as a point of compatibility for collaboratively developing tools and practices across philosophical boundaries.This possibility follows from the shared interest in rejecting models inconsistent with observation.Source: §4 · pp. 2–3

• The framework asks whether an LLM is proposing or refuting a qualitative model and whether a human researcher directly checks its decision-making.Source: §5 · p. 3
• Using LLMs to surface counter-examples for human review is identified as a promising use within qualitative analysis.Source: §5 · p. 3
• This use offers overlap among philosophical perspectives, and the authors hope the framework supports cross-paradigm discussion of LLMs as tools for qualitative analysis.Source: §5 · p. 3