PaperlayerView exact revision on arXiv

Source-linked AI summary

SFT-TA: Supervised Fine-Tuned Agents in Multi-Agent LLMs for Automated Inductive Thematic Analysis

Seungjun Yi, Joakim Nguyen, Huimin Xu, Terence Lim, Joseph Skrovan, Mehak Beri, Hitakshi Modi, Andrew Well, Liu Leqi, Mia Markey, Ying Ding

arXiv:2509.17167v1cs.CL

TL;DR

Manual thematic analysis is time-consuming, and LLM-generated themes remain imperfectly aligned with human reference themes. SFT-TA embeds supervised fine-tuned agents into a multi-agent thematic-analysis pipeline. It outperforms the gpt-4o baseline and existing frameworks, although SFT agents alone may underperform.

  • Problem

    Manual thematic analysis limits scalability, while LLM-generated results show limited alignment with human reference themes.

    Source: §1 · p. 2

  • Method

    SFT-TA embeds supervised fine-tuned agents into multi-agent coding and theme-generation stages.

    Source: §1 · p. 2

  • Results

    SFT-TA improves over gpt-4o by 10.3% in Fuzzy Match, 15.3% in Cosine Similarity, and 22.8% in METEOR.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Embedding SFT agents within a multi-agent system is a promising pathway for improving alignment with desired thematic-analysis outputs.

    Source: §1 · p. 2, §4 · p. 8

    The study uses only nine AAOCA clinical transcripts, so transferability to other datasets remains uncertain.

    Source: §4 · p. 8

Abstract

from arXiv · show

Thematic Analysis (TA) is a widely used qualitative method that provides a structured yet flexible framework for identifying and reporting patterns in clinical interview transcripts. However, manual thematic analysis is time-consuming and limits scalability. Recent advances in LLMs offer a pathway to automate thematic analysis, but alignment with human results remains limited. To address these limitations, we propose SFT-TA, an automated thematic analysis framework that embeds supervised fine-tuned (SFT) agents within a multi-agent system. Our framework outperforms existing frameworks and the gpt-4o baseline in alignment with human reference themes. We observed that SFT agents alone may underperform, but achieve better results than the baseline when embedded within a multi-agent system. Our results highlight that embedding SFT agents in specific roles within a multi-agent system is a promising pathway to improve alignment with desired outputs for thematic analysis.

1. Introduction

Manual thematic analysis is resource-intensive, while LLM-generated themes remain limited in alignment with human reference themes. SFT-TA addresses this gap by embedding supervised fine-tuned agents within a multi-agent workflow for coding and theme generation.

  • Manual thematic analysis requires substantial time and effort, limiting timely, scalable, and reproducible theme generation.Source: §1 · p. 2
  • LLMs can automate coding and candidate-theme generation within minutes, but their alignment with manual thematic analysis remains limited.Source: §1 · p. 2
  • The framework embeds SFT agents in coding and theme-generation stages alongside other agent identities to introduce diverse analytic perspectives.Source: §1 · pp. 2–3
  • SFT-TA improves alignment with the gpt-4o baseline by 10.3% in Fuzzy Match, 15.3% in Cosine Similarity, and 22.8% in METEOR.Source: §1 · p. 2
  • SFT agents alone may underperform the baseline, but integration into the multi-agent system yields gains of 10.3% in Fuzzy Match and 22.6% in C.Source: §1 · p. 3
  • The study identifies a dataset-size “golden point” at which SFT performance saturates.Source: §1 · p. 3

2. Related Work

Prior automated thematic-analysis systems commonly focus on individual workflow stages or require human intervention, motivating more scalable multi-agent approaches. Existing multi-agent systems introduce role specialization, collaboration, and supervision to improve thematic analysis.

2. Related Work

  • Most existing frameworks target coding or theme generation, rely on single-agent LLMs, or require manual intervention that constrains scalability.Source: §2 · p. 3

Multi-Agent Systems for Thematic Analysis

3. Method

SFT-TA processes clinical transcripts through independent coding and theme-generation agents, with SFT agents inserted into both stages and iterative refinement afterward. It is evaluated across ablations and baselines using lexical, semantic, and thematic-analysis-specific measures.

3.1. The SFT-TA Framework.

  • SFT-TA distributes transcripts across independent coder and theme-generation agents, aggregates their outputs, and applies three refinement rounds to produce twelve themes.Source: §3.1 · p. 4

3.2. Dataset

  • The dataset contains nine manually condensed transcripts and twelve human-generated reference themes, with quote IDs supporting reproducible evaluation.Source: §3.2 · p. 4

3.3. Supervised Fine-Tuned (SFT) Agents and Dataset Augmentation

  • Fine-tuning uses ten reference themes for training and two for validation, with each training theme expanded into thirty reviewed paraphrases.Source: §3.3 · pp. 4–5
  • An ablation study found performance peaked at approximately 300 SFT datapoints, supporting a dataset-size saturation point.Source: §3.3 · p. 5

3.4. Implementation Details.

  • Prompts reproduce Braun and Clarke’s six phases, specify evaluation criteria, require twelve themes, and assign a qualitative-researcher role.Source: §3.4 · p. 5

3.5. Experiments

  • Ten experimental conditions compare gpt-4o, gpt-5, prior frameworks, vanilla SFT, framework-integrated SFT, and selective placement in coding or theme generation.Source: §3.5 · p. 5

3.6. Evaluation

  • Evaluation combines lexical-overlap, semantic-similarity, and thematic-analysis metrics for credibility/confirmability, dependability, and transferability.Source: §3.6 · pp. 5–6
  • Transferability is measured across 36 train-validation splits, while human evaluation uses four blinded raters and randomized theme sets.Source: §3.6 · p. 6

1. Coverage.

Coverage evaluates whether generated themes collectively captured the breadth of parents’ lived experiences, with one rating assigned per theme set by dataset-familiar raters.

Coverage

  • Coverage assessed whether the 12 generated themes collectively captured the breadth of parents’ lived experiences.Source: §1 · p. 6
  • Table 1 reports evaluation results on traditional and thematic-analysis metrics.Source: Table 1 · p. 7
  • Coverage was scored once per set of themes rather than individually for each theme.Source: §1 · p. 7

2. Actionability.

Actionability assessed whether each theme expressed a clear, specific, meaningful insight relevant to interventions, resources, or future research.

Actionability

  • Actionability assessed whether each theme offered a clear, specific, meaningful insight that could inform interventions, resources, or future research.Source: §2 · p. 7

3. Distinctiveness.

Distinctiveness assessed whether each theme was unique and avoided overlap or redundancy with other themes.

Distinctiveness

  • Distinctiveness assessed whether each theme was unique and avoided overlaps or redundancies.Source: §3 · p. 7

4. Relevance.

Relevance assessed whether themes accurately represented parents’ concerns and needs without conflating them with patient perspectives.

Relevance

  • Relevance assessed whether each theme accurately reflected parents’ concerns and needs without conflating them with patient perspectives.Source: §4 · p. 7
  • Relevance scores were averaged across themes and raters, alongside other per-theme criteria, to represent the theme set.Source: §4 · p. 7

4. Results

SFT-TA achieved the strongest traditional-metric alignment when SFT agents were integrated into both coding and theme generation, while human evaluation showed gains in coverage, actionability, and relevance but a decline in dependability. The section also reports limitations involving metric alignment, dataset scope, and the non-exhaustiveness of human reference themes.

4. Results

  • +10.3% Fuzzy Match, +15.3% Cosine Similarity, and +22.8% METEOR were achieved by the full SFT-TA pipeline, which also attained the strongest dependability score, D = 0.394.The full pipeline outperformed the vanilla gpt-4o baseline, whereas SFT agents alone did not improve performance and sometimes performed worse.Source: §4 · p. 7
  • Dependability decreased from D = .407 for vanilla gpt-4o to D = .394 for full SFT-TA, possibly because additional agents increase complexity or stochasticity.The authors propose examining whether tuning individual-agent parameters can mitigate this degradation.Source: §4 · pp. 7–8
  • Coverage = 5.00, Actionability = 4.28, and Relevance = 4.63 were the highest human-evaluation scores, while Distinctiveness was 4.35 and its underlying cause remained unclear.The reported human-evaluation comparisons use a 1–5 Likert scale relative to the gpt-4o baseline.Source: §4 · p. 8, Table 2 · p. 8
  • Human evaluation results did not consistently align with traditional or thematic-analysis-specific metrics.This limits straightforward interpretation of automated metric performance against human judgments.Source: §4 · p. 8

Appendix A. Criteria for Evaluating Generated Themes

  • The evaluation criteria cover uniqueness, actionability, specificity, condition relevance, segment focus, clarity, plausibility, and overall quality.These criteria were used to assess the quality of generated themes.Source: Appendix A · pp. 11–12

Appendix C. Definition of Win Rate

  • Win Rate measures the proportion of evaluation instances in which a model output is preferred over a baseline, with reliable gains defined as exceeding 0.70 in the stated evaluation design.The analysis uses Wilson 95% confidence intervals, one-sided binomial tests against 50% chance, and Holm–Bonferroni correction.Source: Appendix C · p. 12

Appendix D. Effect of Dataset Size on SFT Performance

Increasing SFT training examples improved win rate, with gains diminishing after approximately 300 examples. The appendix establishes a >0.70 win-rate threshold using confidence intervals, exact binomial testing, multiple-testing correction, and power analysis.

Appendix D. Effect of Dataset Size on SFT Performance

  • Approximately 300 labeled examples were sufficient to reach stable SFT performance, while larger datasets mainly reduced variability rather than improving accuracy.The largest gain occurred between 100 and 300 samples, after which improvements diminished.Source: Appendix D · p. 12, Figure 2 · p. 12
  • Win rate improved steadily with training size and reliably exceeded chance after training surpassed a few hundred examples.Figure 2 marks chance at 0.50, a reliability threshold at 0.70, and a saturation point near N ≈300.Source: Appendix D · p. 12, Figure 2 · p. 12

Appendix E. Statistical Basis for Threshold Determination

  • Wilson 95% confidence intervals were used instead of normal approximations to assess whether observed win rates differed from the 0.50 chance level.Intervals not containing 0.50 were interpreted as evidence of a difference from chance at the 5% level.Source: Appendix E · p. 13
  • Exact one-sided binomial tests evaluated whether observed wins exceeded chance, with Holm–Bonferroni correction controlling familywise error across conditions.The test evaluates the probability of observing at least x wins under a 0.5 null probability.Source: Appendix E · p. 13
  • With n = 100 paired outputs, power analysis indicated that a 0.20 effect was the smallest reliably detectable at α = 0.05 with 95% power.This corresponds to distinguishing a 0.70 observed win rate from the 0.50 null rate.Source: Appendix E · p. 13
  • The study adopted a >0.70 win-rate criterion for reliable gains because it combines corrected statistical significance with practical detectability.The threshold is intended to distinguish robust improvements from chance fluctuations.Source: Appendix E · p. 13

Appendix F. Lexical Overlap Metrics Details

The appendix describes lexical-overlap and semantic-similarity metrics for comparing generated themes with human references. These metrics differ in whether they prioritize surface form, partial lexical correspondence, or meaning beyond wording.

Fuzzy Match

  • Fuzzy Match measures lexical similarity with normalized Levenshtein distance and token-alignment heuristics, allowing partial credit for close wording.A score of 1.0 indicates an exact match, while lower scores reflect greater edit distance.Source: Appendix F · p. 13

Cosine Similarity

  • Cosine Similarity measures semantic closeness between sentence-embedding vectors, capturing alignment when themes are paraphrased or reworded.The metric uses the cosine of the angle between two embedding vectors.Source: Appendix F · pp. 13–14

BLEU

  • BLEU measures precision of overlapping n-grams with a brevity penalty, so it can under-reward semantically aligned paraphrases.Its strict reliance on exact matches distinguishes it from meaning-oriented similarity measures.Source: Appendix F · p. 14

METEOR

  • METEOR incorporates unigram recall, stemming, synonym matching, and fragmentation penalties to provide partial credit for lexically different but related phrases.This makes it more tolerant of synonym substitutions than strict n-gram overlap.Source: Appendix F · p. 14

ROUGE-L

  • ROUGE-L measures the longest common subsequence between reference and candidate texts, capturing shared content and word order.Related phrases with shared sequential structure can therefore receive moderate scores.Source: Appendix F · p. 14

Appendix G. Human Evaluation Guiding Statements

Human raters evaluated themes on a five-point Likert scale using coverage, actionability, distinctiveness, and relevance. The criteria assess comprehensiveness, usefulness, nonredundancy, and fidelity to parents’ lived experiences.

  • Coverage assesses whether themes comprehensively capture parents’ lived experiences and are supported by multiple participants.Source: Appendix G · p. 14
  • Actionability assesses whether a theme provides specific insights that can inform interventions, resources, or research.Source: Appendix G · p. 14
  • Distinctiveness assesses whether a theme remains clearly separate from other themes without major overlap or redundancy.Source: Appendix G · p. 14
  • Relevance assesses whether a theme accurately reflects parents’ lived experiences and concerns related to having a child with AAOCA.Source: Appendix G · p. 14
Loading 2509.17167v1…