PaperlayerView exact revision on arXiv

Source-linked AI summary

QuAD: Quantitative Assessment via Distribution-Alignment for Qualitative Data Quality Enabled by a Content-Aware Multi-Agent System

Wenshuo Zhang, Leixian Shen, Qiqi Zhou, Shuchang Xu, Feiyang Hao, Huamin Qu, Haobo Li, Zhao Chen, Lin-Ping Yuan

arXiv:2610.23153v1cs.AIcs.HC

TL;DR

Qualitative researchers lack a dataset-level way to assess whether collected responses support a chosen analytical lens before investing in full analysis. QuAD compares response-derived structure with metadata-derived distributions and uses a content-aware multi-agent system to score dissimilarity. Controlled experiments and a real-world survey case study support its use as a pre-analysis aid that also prioritizes responses for human interpretation.

  • Problem

    Researchers need to assess whether a qualitative response dataset supports a selected analysis lens before investing effort in full analysis or overlooking patterns and unusual cases.

    Source: §1 · p. 1

  • Method

    QuAD compares response-derived cluster structure with a metadata-derived analytical lens through a four-quadrant assessment and uses multi-agent task-conditioned dissimilarity scoring with iterative instructions and adaptive anchors.

    Source: §1 · p. 2, §3.2.1 · p. 5

  • Results

    Controlled experiments and a human-labeled case study show QuAD complements qualitative coding by assessing readiness and prioritizing responses for human checking.

    Source: §5 · p. 9

  • Takeaways & Limitations

    QuAD offers quantitative fitness-for-purpose signals for examining general patterns and unique insights while surfacing representative or unusual responses for human interpretation.

    Source: §1 · p. 2, §5 · p. 9

    Scalar projection improves efficiency but loses relational detail when dissimilarity is intrinsically multifaceted; pairwise scoring retains beyond-axis relations but has O(N^2) cost and suits smaller corpora or pilot subsets.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Assessing whether qualitative responses collected from semi-structured questions are ready for a particular analysis goal is a persistent challenge. Manual qualitative analysis is interpretive and labor-intensive, while existing automated methods often lack a dataset-level perspective for deciding whether a response set supports a researcher-selected analytical lens. We introduce QuAD, a framework for quantitative pre-analysis assessment of qualitative response datasets. QuAD compares a reconstructed intrinsic cluster distribution reconstructed from responses with an extrinsic, researcher-defined analytical lens derived from metadata. QuAD is structured into a four-quadrant assessment whose metrics are read jointly, from lens-level relations to within-cluster variation, to provide fitness-for-purpose signals for revealing general patterns and unique insights. Because this assessment relies on lens-conditioned inter-response dissimilarity, we propose a content-aware multi-agent system with two-stage instruction generation and adaptive anchors to improve scoring consistency. We validate QuAD through controlled experiments on synthetic data, including robustness and baseline comparisons, and a real-world social survey case study. The case study shows how QuAD quantitatively assesses whether a selected analysis axis is promising and also surfaces representative or unusual responses for human interpretation.

1 Introduction

QuAD addresses the need to assess whether qualitative response datasets support a researcher-selected analysis lens before full interpretation. It compares response-derived structure with metadata-derived distributions and combines that assessment with task-conditioned dissimilarity scoring.

1 Introduction

  • Pre-analysis assessment matters because researchers may otherwise invest in an unsupported analysis lens or overlook cross-group patterns and unusual cases.QuAD is positioned as an assessment aid before full coding or interpretation, not as a replacement for them.Source: §1 · pp. 1–2
  • QuAD quantitatively assesses dataset readiness by comparing empirically derived response structure with a researcher-defined analytical lens, treating the lens as an external hypothesis rather than ground truth.Alignment may indicate support for group-level patterns, while divergence may point to consensus, latent factors, or unusual responses for closer interpretation.Source: §1 · p. 2
  • Its four-quadrant framework jointly reads four complementary metrics to assess utility for general patterns and unique insights at distribution and data-point levels.The assessment uses distributions reconstructed from metadata and inter-response dissimilarities.Source: §1 · p. 2
  • A content-aware multi-agent system computes task-conditioned inter-response dissimilarity using iterative instruction generation and adaptive anchors to support consistency and mitigate scoring drift.The instruction-generation process is domain-specific, while dynamic anchors maintain a stable semantic reference frame.Source: §1 · p. 2
  • The paper validates QuAD and its components through controlled experiments, including robustness checks, and a real-world study using existing thematic labels.The case study concerns a qualitative social survey of student food choices.Source: §1 · p. 2

2 Related Work

Prior approaches assess qualitative dataset quality through interpretive practices or automated support focused on individual responses. QuAD instead provides a dataset-level pre-analysis signal about alignment with a researcher-defined lens.

2 Related Work

  • Manual qualitative quality practices include coding, theme development, intercoder reliability, saturation, and richness assessment, but occur after collection and require substantial human effort.Source: §2 · p. 2
  • Automated NLP and LLM approaches support individual-response assessment or qualitative coding but provide limited dataset-level pre-analysis signals.Source: §2 · p. 2
  • LLM-as-a-Judge can scale evaluation but faces bias, inconsistency, prompt sensitivity, and scoring drift; QuAD uses interactive instruction generation and adaptive anchors to improve robustness.Source: §2 · p. 3

3 Proposed Method

QuAD assesses dataset fitness for a researcher-selected analytical lens by comparing response-derived clusters with a metadata-derived partition. It combines task-conditioned dissimilarity scoring with a four-quadrant profile, read jointly to characterize general patterns and unique insights.

3 Proposed Method

  • QuAD computes task-conditioned inter-response dissimilarities, clusters them, and compares the resulting distribution with a metadata-derived analytical distribution.Its pipeline generates a task-specific instruction, scores responses using adaptive anchors, then clusters the dissimilarity matrix for assessment.Source: §3 · p. 3

3.1.1 Problem Formulation

  • The system seeks an instruction whose evaluator scores align with oracle judgments, while maintaining global consistency in the dissimilarity matrix.The objective is pursued through feedback-guided refinement without a guarantee of convergence or a global optimum.Source: §3.1.1 · p. 3

3.1.2 Iterative Instruction Generation

  • Instruction generation uses automated oracle feedback to develop candidates, followed by human-expert feedback to adapt the selected instruction to the specific task.Natural-language critique acts as a pseudo-gradient for refinement; the expert-guided loop ends when the expert is satisfied and gives no more input.Source: §3.1.2 · pp. 3–4

3.1.3 Adaptive Anchor Manifold Maintenance

  • Adaptive anchor maintenance keeps a bounded set of diverse response-and-score references to provide the evaluator with a stable semantic reference during scoring.Novel candidates are selected by their average dissimilarity to existing anchors; when the set exceeds its limit, the most redundant anchor is pruned.Source: §3.1.3 · p. 4

3.2.1 Problem Formulation

  • QuAD treats dataset quality as fitness for purpose, comparing an external metadata partition with intrinsic clusters across general-pattern and unique-insight goals at distribution and data-point levels.Its four normalized metrics are interpreted jointly rather than as a binary quality label or independent higher-is-better scores.Source: §3.2.1 · p. 5

(1) Demographic and Cluster Distributions

  • GMR and DDR characterize lens alignment and group separation, while DP and DC describe cluster mixing and spread; QuAD also flags representative or unusual responses for human review.DC is interpreted relative to a baseline and compared only across lenses with the same number of groups; response mining is triage, not a substitute for thematic coding.Source: (1) Demographic and Cluster… · pp. 5–6

4 Experiment

The experiments test QuAD on synthetic data and a human-labeled food-choice survey, examining instruction generation, robustness, controlled distribution shifts, and real-data validation.

4.1.1 User Study for Instruction Generation

  • Participants reported lower cognitive load and higher ratings on every satisfaction item with the instruction-generation framework than with manual instruction writing.The study involved 20 postgraduate students and used a custom 10-item satisfaction scale; generated instructions were used in later experiments.Source: §4.1.1 · p. 7, Figure 4 · p. 7
  • Pairwise scores from manually written instructions drift in scale across pairs and correlate near zero with ground-truth dissimilarities, while scalar-projection gains are smaller.Table 1 compares baseline and generated instruction quality using Spearman and Pearson correlations.Source: §4.1.1 · p. 7, Table 1 · p. 7

4.1.2 Robustness Checks

  • The full adaptive-anchor configuration has the highest mean Spearman correlation in both regimes: pairwise 0.743 versus ≤0.726 and scalar 0.776 versus ≤0.775.It ranks first in five of ten model–regime combinations, within the top two in nine, and has the highest worst case, 0.708 versus ≤0.659.Source: §4.1.2 · p. 7
  • Embedding distances reach 0.594–0.621 in the pairwise and 0.614–0.698 in the scalar regime, versus QuAD’s 0.708–0.830 and 0.719–0.827; cross-model correlations remain 0.727–0.733 for small and 0.842–0.878 for large tasks.Across repeated runs, within-model distance correlations are 0.80 for Qwen-max and 0.94 for DeepSeek-v3, cross-model agreement is 0.87, and the largest metric SD is 0.050.Source: §4.1.2 · p. 7
  • Across four clustering algorithms, ARI ranges from 0.592 to 0.675; K-means is strongest overall at ARI/NMI/purity 0.675/0.743/0.850.The comparison holds the synthetic data, QuAD dissimilarity matrix, and K fixed.Source: §4.1.2 · pp. 7–8

4.1.3 Controlled Distribution Shift Experiment

  • As variance increases, GMR, DP, and DDR decrease while DC increases; as mean difference increases, GMR, DP, and DDR increase while DC remains stable.These distinct joint patterns matched the designed distribution changes in both small- and large-scale configurations.Source: §4.1.3 · p. 8, Figure 5 · p. 8

4.2 Case Study: External Validation on Human-Labeled Food-Choice Responses

  • For grade level, QuAD reports GMR = 0.39, DDR = 0.02, DC = 0.90, and DP = 0.43, judging it a weak lens in line with Cramér’s V = 0.130.On the same responses, switching the lens to eating-habit change raises DDR from 0.02 to 0.52 and DP from 0.43 to 0.75.Source: §4.2 · p. 8
  • QuAD’s mean distance rises from 14.61 for same-label pairs to 30.38 for different-label pairs, with label-mismatch AUC 0.749 and top-3 label purity 0.742, above TF–IDF’s 0.551 and 0.650.Three evaluators gave median ratings of 6 for correctness, interpretability, and usefulness.Source: §4.2 · p. 8

5 Conclusion

QuAD provides quantitative pre-analysis fitness-for-purpose signals and response prioritization to complement qualitative coding.

5 Conclusion

  • QuAD combines task-conditioned dissimilarities from a content-aware multi-agent system with a four-quadrant assessment of cluster alignment to a researcher-defined lens and within-cluster variation.The assessment reads its metrics jointly to signal fitness for discovering general patterns and unique insights.Source: §5 · p. 8
  • Controlled experiments and a human-labeled case study show QuAD can assess analysis readiness quantitatively and prioritize responses for human review, complementing rather than replacing qualitative coding.The conclusion frames the framework as a pre-analysis aid.Source: §5 · p. 9

Limitations

QuAD’s scope is bounded by LLM variability and bias, scalar-projection information loss, context-dependent metric interpretation, and limited real-data validation.

Limitations

  • LLM updates, decoding settings, and service-side changes can cause scoring drift; repeated-run and cross-model checks quantify but do not eliminate this variation.QuAD inherits the capabilities and stability of its underlying judge.Source: Limitations · p. 9
  • LLM judgments may privilege standard dialects, dominant cultural framings, verbosity, or particular writing styles, reducing the visibility of culturally specific meanings.The authors recommend inspecting instructions, anchors, representative responses, and outliers, especially for minority-language and culturally situated responses.Source: Limitations · p. 9
  • Scalar projection compresses multidimensional semantic relations onto one task-conditioned axis, losing relational detail when dissimilarity is multifaceted.Pairwise scoring can retain relations beyond one axis when instructions define them, but its O(N^2) cost suits pilot subsets and smaller corpora; synthetic pairwise tasks validate latent-score gaps, not beyond-axis relations.Source: Limitations · p. 9
  • A high score alone is not a complete quality judgment: metric meaning depends on context, theory, and interpretive aims, and may downplay meaningful outliers.The authors recommend reading the four metrics as one profile alongside mined responses; defaults can be adapted, including checks for minority coverage or tail behavior.Source: Limitations · p. 9
  • Real-data validation uses one survey corpus, although controlled experiments use synthetic responses with known latent scores and the case study tests eight metadata lenses and human-code alignment.Additional corpora and lenses would further test generality.Source: Limitations · p. 9

Ethics Statement

The ethics statement reports human-subject protections and flags risks of LLM-assisted qualitative analysis.

Ethics Statement

  • Both human-subject studies had institutional review board approval; participants consented, were compensated, and are reported anonymously.Compensation was USD 8 for the user study and USD 10 for the evaluator study.Source: Ethics Statement · p. 9
  • LLM-assisted analysis may introduce interpretation bias or over-reliance, while sensitive open responses may raise privacy concerns.The authors recommend human validation, anonymization, and transparent reporting of model-assisted decisions.Source: Ethics Statement · p. 9
  • Gemini assisted with grammar, wording, and related-work search, while Cursor and associated generative models assisted with code writing.The authors state that Gemini did not alter meaning or introduce new references or knowledge, and that retrieved papers were reviewed.Source: Ethics Statement · p. 9

A.1 Ablation Details for Adaptive Anchor Updating

The ablation study finds the complete adaptive-anchor configuration to be the most reliable default across tested settings. Metric clarification explains how the empirical normalization factor varies by scoring regime and affects DC.

A.1 Ablation Details for Adaptive Anchor Updating

  • Four 15-interviewee and four 90-interviewee datasets test Full against five variants that each disable one adaptive-anchor component.The variants remove in-context examples, adaptive updating, novelty filtering, example-length limits, or LLM-assigned anchor scores; reported correlations average four randomly selected questions across four domains.Source: §A.1 · p. 11
  • The complete configuration has the highest mean Spearman correlation in both regimes (pairwise 0.743; scalar 0.776) and the highest worst case (0.708) across ten settings.It ranks first in five of ten model–regime combinations and in the top two in nine; the scalar-regime mean margin is small, and these summaries are not significance tests.Source: §A.1 · p. 11, Table 2 · p. 12

A.2 Metric and Quantity Clarification

  • DC uses λ = 1.2 for pairwise scoring and λ = 0.8 for scalar projection; changing λ shifts DC uniformly without reversing its ordering across lenses with the same C.The empirical factor reflects estimation-regime variance, and λ is a normalization convention rather than a validated high-or-low cutoff for DC.Source: §A.2 · p. 12

A.3 Baseline and Clustering Robustness

The appendix reports baseline and clustering checks alongside repeated-run and cross-model reliability tests. Cross-family correlations remain high in the tested settings, while repeated-run results support within-model repeatability but not interchangeability of absolute metric values across models.

A.3 Baseline and Clustering Robustness

  • Cross-family Spearman correlations with ground truth remain 0.727–0.733 at small scale and 0.842–0.878 at large scale in both generator–evaluator directions.The appendix also reports embedding-based dissimilarity baselines and fixed-K clustering sensitivity checks.Source: §A.3 · p. 12

A.4 Repeated-Run and Cross-Model Reliability

  • Within-model distance correlations are 0.80 and 0.94, cross-model agreement is 0.87, and the largest observed metric standard deviation is 0.050.These results indicate moderate-to-high rank agreement and limited metric variation in the tested slice; differing metric levels support repeatability within a model, not cross-model interchangeability of absolute values.Source: §A.4 · p. 12

A.5 Pipeline Component Necessity Checks

The component checks report stronger task-conditioned dissimilarity extraction than generic embeddings and robust reconstruction across tested clustering choices. In the real survey, QuAD’s scores correspond to independent human-coded labels, while the appendix also describes scoring cost, case-study metrics, and the pipeline stages.

A.5 Pipeline Component Necessity Checks

  • Pairwise instruction generation raises Spearman correlation from −0.036 to 0.725 for Qwen-max, 0.599 to 0.830 for GLM, and 0.331 to 0.726 for DeepSeek-v3; no single anchor ablation consistently dominates.The complete anchor configuration remains among the strongest settings across models and inference regimes.Source: §A.5 · p. 13
  • Embedding baselines reach Spearman correlations of 0.594–0.621 in pairwise scoring and 0.614–0.698 under scalar projection, versus QuAD’s 0.708–0.830 and 0.719–0.827.The reported differences indicate that researcher-defined evaluation instructions capture criterion-specific distinctions beyond generic semantic similarity.Source: §A.5 · p. 13, Table 4 · p. 13
  • Across clustering alternatives, ARI ranges from 0.592–0.675, NMI from 0.679–0.743, and purity from 0.786–0.850; the current K-means ensemble scores 0.675/0.743/0.850.The comparison holds K fixed, so it does not test changes in cluster granularity; on the case-study matrix, MDS has stress-1 0.046 and distance correlation r = 0.997.Source: §A.5 · pp. 13–14, Table 6 · p. 13

A.6 Scalability and Deployment Cost

  • Scalar projection reduces the 30-response scoring run from 449,765 to 13,391 tokens and scores all 122 responses with 56,727 tokens at an estimated cost of $0.0060.The reported accounting covers dissimilarity-scoring API calls, not total end-to-end deployment cost.Source: §A.6 · p. 14, Table 8 · p. 14

B.1 Case Study Setup

  • The case study uses 122 food-choice responses, with grade level as the analytical lens and pre-existing eating-habit labels reserved for post-hoc validation; QuAD reports GMR = 0.39, DDR = 0.02, DC = 0.90, and DP = 0.43.All 122 responses enter the grade–label association, while pair and retrieval analyses exclude three unclear responses; scalar scoring is used because N exceeds Nthreshold.Source: §B.1 · p. 15

B.2 Pipeline Walkthrough

  • The walkthrough proceeds from instruction generation and adaptive-anchor dissimilarity scoring to clustering, lens assessment, and prioritization of representative or unusual responses for subsequent coding.The example focuses the instruction on eating-habit-change dissimilarity rather than generic lexical similarity.Source: §B.2 · p. 15

B.3 External Validation against Human-Coded Labels

  • QuAD achieves label-mismatch AUC 0.749 and top-3/top-5 purities 0.742/0.694, exceeding TF–IDF and random retrieval on both AUC and neighborhood-purity measures.Qwen3-Embed has slightly higher neighbor purity, and fine-code agreement is lower (AUC 0.629); the checks support criterion validity on real text, not theme discovery beyond the instruction.Source: §B.3 · p. 16

B.4 Weak versus Strong Lens on the Same Responses

QuAD tests its weak grade-level verdict against a strong eating-habit-change lens on the same responses, then evaluates how readings change as lens informativeness is reduced. Against direct LLM assessment and representation baselines, QuAD yields more differentiated lens verdicts and stronger response-structure agreement with human codes.

B.4 Weak versus Strong Lens on the Same Responses

  • The lens-switch test holds responses and the dissimilarity matrix fixed, but the lenses differ in sample size and group count, so DDR is the most directly comparable metric.The grade-level and eating-habit-change lenses use 122/4 and 119/3 responses/groups, respectively; DC is normalized by C and is not compared across them.Source: §B.4 · p. 16
  • QuAD’s DDR rises from 0.02 for grade level to 0.52 for eating-habit change, while GMR rises from 0.39 to 0.53 and DP from 0.43 to 0.75.The DDR shift agrees with the external criterion: grade level is barely associated with the human code (V = 0.130), while eating-habit change is that code.Source: §B.4 · p. 16
  • As randomized label reassignment increases from p = 0 to p = 1, DDR decreases from 0.52 to 0.02, following the Kruskal–Wallis effect size ε2.DP falls to the majority-group share of 0.63; DC stays constant because clusters and C are fixed and DC does not use lens labels.Source: §B.4 · pp. 16–17

B.5 Direct LLM Assessment and Representation Baselines

  • The direct LLM rated grade level “strong” in all three runs despite V = 0.13, and labeled seven unassociated metadata lenses “strong” in 9 of 21 runs.Ratings stayed within a narrow band of 4–5 for every metadata lens.Source: §B.5 · p. 18
  • QuAD’s representative and unusual picks reached proxy hit rates of 0.81 and 0.82, while direct LLM picks reached 0.63 and 0.47, respectively.The direct LLM’s representative-pick rate was below the random level (0.66).Source: §B.5 · p. 18
  • QuAD reconstructions agreed more with human codes than direct LLM partitions (ARI 0.30/0.11 vs. 0.11/0.08), and pair-level mismatch AUC was 0.749 vs. 0.613.Reordering responses changed most direct-LLM selections (mean top-3 Jaccard 0.12) and partitions (mean pairwise ARI 0.37). QuAD’s reporting stage is deterministic for a fixed distance matrix; this does not establish order-invariant scoring with adaptive anchors.Source: §B.5 · p. 18
  • The direct-LLM comparison does not isolate one component: it sees lens groups and lacks QuAD’s refined rubric, while its proxy triage score is not a substitute for evaluator study.The proxy checks whether picks match the coarse human code, not whether they are worth reading.Source: §B.5 · p. 18

B.6 Code-then-Test Check of Group Separation

The code-then-test check finds that standard tests and DDR agree at the extremes, but the joint QuAD profile adds information about response structure and possible next steps. Independent evaluators found the report understandable and useful for starting coding, while the mined-response examples illustrate representative and unusual eating-change patterns.

B.6 Code-then-Test Check of Group Separation

  • No test detected a grade effect (p ≥0.27; DDR p = 0.38), while all detected the human-code effect (p ≤0.002).KW and PERMANOVA flagged current diet (p = 0.031 and 0.023), but DDR did not reach p < 0.05 for any metadata lens.Source: §B.6 · p. 18, Table 15 · p. 19
  • For grade level, high DC (0.90) alongside low DP (0.43), GMR (0.39), and DDR (0.02) indicates varied responses that mix grades rather than follow them.This profile points toward cross-grade themes and inspecting unusual responses; the human-code profile instead supports comparing groups and reading representative responses first.Source: §B.6 · p. 19

B.7 Independent Evaluator Validation

  • The median rating for interpretability and usefulness for subsequent coding was 6 (range 6–7), while correctness also had a median of 6 (range 4–7).The three-evaluator study provides face-validity evidence, not evidence of effects on final coding outcomes or of new insights from unusual responses.Source: Appendix B.8.3 · p. 20, Table 16 · p. 20, §B.7 · p. 20

B.8.1 Question and Response Examples

  • The appendix presents eating-change responses as examples of the qualitative inputs and response variation summarized by QuAD.The supplied excerpts include fragmented examples, so this point describes their role rather than drawing conclusions from them.Source: §B.8.1 · p. 21

B.8.2 Grade-Level Label Distribution

  • The four thematic labels have similar proportions across grade levels, providing context for the reported weak association..Source: §B.8.2 · p. 21

Appendix B.8.3 excludes such answers.

  • Representative responses capture recurring eating-change patterns, while unusual responses surface less common trajectories for targeted inspection.Non-substantive answers without descriptions of eating habits are excluded, with the next-ranked response shown instead.Source: §B.8.3 · p. 21

C Supporting Iterative Qualitative Data Analysis

QuAD is presented as a repeatable diagnostic checkpoint in iterative qualitative analysis, with its profile and response-mining signals informing researcher inspection and possible revisions. These mappings are reading guides, not automatic decision rules.

C Supporting Iterative Qualitative Data Analysis

  • At pilot and later checkpoints, researchers can repeat QuAD to assess a lens, inspect central or unusual responses, and consider revisions to instructions, lens, sampling plan, or study scope.The pairwise regime is proposed for pilot corpora, while scalar projection supports assessment as the corpus grows larger.Source: C Supporting Iterative Qualitative… · p. 22
  • The four metrics are read together: GMR summarizes lens-cluster alignment, DDR group separation, DC within-cluster dispersion, and DP analytical-group mixing.A separate response-mining ranking identifies central and unusual cases; mappings from signal combinations to next steps guide close inspection rather than automatic decisions.Source: C Supporting Iterative Qualitative… · p. 22, Table 17 · p. 22

D.1 User Study Protocol and Materials

  • The appendix documents the within-subject counterbalancing schedule and materials supporting the controlled user study.It includes the schedule used to mitigate ordering effects.Source: §D.1 · p. 22, §D.1.1 · p. 22

D.1.2 Statistical Significance of User Study Results

The appendix documents supplementary user-study comparisons and materials for evaluating instruction-generation support. It describes the shared interface design and the workload and usability questionnaires used in the study.

D.1.2 Statistical Significance of User Study Results

  • Table 19 provides per-item statistical comparisons corresponding to the usability results summarized in the main-paper figure.The study schedule describes 20 postgraduate participants using QuAD and a same-layout baseline with agent-assisted features disabled.Source: §D.1.2 · p. 22, Table 18 · p. 23, Table 19 · p. 23

D.1.3 Prompt Generation System Interface Overview

  • Both study conditions use the same overall interface layout, while QuAD adds an auto-generated prompt and agent-assisted refinement to the manual baseline workflow.The interface is intended to support generating or refining task-specific dissimilarity instructions, not to serve as a full qualitative-analysis workbench.Source: §D.1.3 · p. 23

D.1.4 Participant Demographics

  • Table 20 reports participant background variables, including domain experience and familiarity ratings.Its caption specifies that ratings use a 1–5 Likert scale.Source: §D.1.4 · p. 23, Table 20 · p. 23

D.1.5 NASA-TLX and System Usability Questionnaire

  • The questionnaires collect NASA-TLX workload ratings and seven-point system-usability ratings covering prompt generation, refinement, confidence, and overall process.The workload instrument uses a 1–20 scale and focuses interpretation mainly on mental demand, temporal demand, effort, and frustration.Source: §D.1.5 · pp. 24–26

D.2 Synthetic Data Generation

This section describes QuAD’s controlled synthetic data generation using responses conditioned on defined communities, scores, and personas across four domains. Its finance examples represent community profiles, scored candidate personas, and simulated banking-service responses that contrast an experienced wealth manager with a novice.

D.2 Synthetic Data Generation

  • The generation system creates ground-truth-controlled response sets for Finance, Post-operative Medicine, HCI, and Additive Manufacturing.The questionnaires for these domains are used to produce experimental datasets.Source: §D.2 · p. 27

D.2.2 Questionnaires

  • The questionnaires cover Finance, post-treatment health, HCI, and additive manufacturing, with questions spanning domain knowledge, experiences, concerns, and future choices.Examples include banking and saving, medication-related health changes, AI and interface use, and 3D-printing materials and societal implications.Source: §D.2 · p. 27, §D.2.2 · pp. 27–30

D.2.1 Controllable Synthetic Data Generation System

  • A four-stage LLM-agent pipeline defines communities, samples dimension scores, instantiates consistent personas, and simulates questionnaire responses conditioned on personas and scores.Community-specific score distributions control attribute levels and diversity; scores are prioritized to keep responses aligned with the assigned ground truth.Source: §D.2.1 · p. 27

D.3 Synthetic Data Example on Finance

  • Finance examples illustrate the pipeline’s community-definition, candidate-instantiation, and response-simulation outputs for reference.The examples clarify what the synthetic inputs and outputs look like in practice.Source: §D.3 · p. 30

D.3.1 Community Example

  • The community example gives a named group a natural-language profile that constrains subsequent persona or candidate generation.The example profile describes a medium financial practitioner’s background, financial exposure, behaviors, and planning priorities.Source: §D.3.1 · p. 30

D.3.2 Candidate Example

  • The candidate example pairs a persona with assigned ground-truth scores used as labels in controlled evaluation.It contrasts a high-scoring financial expert with a low-scoring financial novice.Source: §D.3.2 · pp. 30–31

D.3.3 Response Example

  • The response example consists of free-text answers conditioned on a candidate persona and pre-assigned scores, which serve as QuAD inputs for dissimilarity extraction and controlled evaluation.The example links simulated qualitative responses to the controlled experiment pipeline.Source: §D.3.3 · p. 32
  • The experienced manager describes strategic and transactional banking uses, while the novice focuses on basic accounts and app-based banking.The manager discusses services including international transfers, custodial banking, and liquidity solutions; the novice says busy delivery shifts make online essentials more practical.Source: §D.3.3 · pp. 34–35
  • The manager identifies inflation, opportunity, and credit risks, whereas the novice recognizes possible bank failure but is unsure how deposit insurance works.The novice also says they lack confidence explaining these risks.Source: §D.3.3 · pp. 34–36
  • The examples assign the manager financial-literacy and financial-behavior scores of 80 and 81, compared with 30 and 35 for the novice.The manager's other listed score is financial attitude 90; the novice's is 26.Source: §D.3.3 · pp. 33, 35, 36

E Background and Reproducibility Materials

This appendix positions QuAD as a pre-coding assessment alongside qualitative interpretation and documents its distribution-comparison pipeline, instruction refinement, adaptive anchors, and prompts. It also provides reproducibility examples for synthetic-data generation and case-study scoring, including score-grounded response generation and a rubric for changes in eating habits.

E.1 Extended Background

  • Traditional qualitative analysis emphasizes interpretive coding and theory building, while much computational and LLM work focuses on individual responses or coding rather than dataset-level lens readiness.The appendix frames QuAD as a pre-analysis signal for deciding whether a researcher-selected axis is supported before committing to manual coding.Source: §E.1 · p. 36
  • QuAD complements traditional analysis by assessing lens readiness before coding, while its joint profile provides more than a group-difference verdict.A direct end-to-end LLM verdict is order-sensitive and does not reproduce QuAD’s verdict; standard tests on scalar scores agree with DDR but do not describe group mixing, within-cluster variation, or pairwise-regime results.Source: §E.1 · pp. 36–37

E.2 Detailed Algorithms for QuAD Framework

  • The appendix consolidates implementation algorithms, mapping the end-to-end workflow and detailing instruction refinement and anchor maintenance.The algorithms correspond to the workflow in Fig. 1 and Sec. 3.2, the refinement procedure in Sec. 3.1.2, and anchor maintenance in Sec. 3.1.3.Source: §E.2 · p. 38

E.2.1 Overall QuAD Framework

  • The full pipeline compares metadata-defined groups with reconstructed response clusters, computes a four-metric profile, and mines archetypal and outlier responses.It generates an adapted instruction, extracts dissimilarities, embeds and clusters responses, aligns clusters to metadata groups using IoU matching, then calculates GMR, DDR, DP, and DC.Source: §E.2.1 · p. 38

E.2.2 Iterative Instruction Generation

  • QuAD’s instruction refinement uses automated discovery followed by human adaptation, with corrected scores and critiques serving as feedback for prompt updates.Automated iteration continues until an evaluator reaches a high threshold, after which a human selects and adapts the instruction until satisfied.Source: §E.2.2 · pp. 38–39

E.2.3 Adaptive Anchor Manifold Maintenance

  • Adaptive anchors provide context for LLM dissimilarity calls and are maintained by adding diverse responses and removing redundant anchors when the set exceeds its limit.Small datasets use pairwise scores with both presentation orders symmetrized; large datasets use scalar response scores to construct pairwise distances.Source: §E.2.3 · pp. 39–40

E.3 Prompt Templates

  • The prompt-template section lists the prompts used in QuAD.Source: §E.3 · p. 41

E.3.1 Automated Instruction Discovery

  • The selected synthetic-proficiency instruction asks the judge to estimate respondent proficiency and return their difference, which is used as the dissimilarity score.Its estimated gap and banded guide are monotone, so Spearman correlations are unaffected by which the judge follows, while Pearson correlations and distance metrics use the returned-score scale.Source: §E.3.1 · p. 43
  • The proficiency rubric scores respondent gaps from 0–100 and directs evaluators to judge demonstrated content rather than communication style or confidence.It asks for separate 0–100 proficiency estimates, the absolute gap as the final difference score, and brief evidence-based justifications.Source: §E.3.1 · pp. 44–45

E.3.2 Interactive Instruction Generation

  • Interactive prompt generation first adds domain-specific semantic guidance while preserving the base prompt structure, then makes only minimal changes in response to explicit user feedback.The initial prompt covers domain expression, proficiency signals, underlying mechanisms, and condition-specific comparisons; the sequential prompt preserves aspects not addressed by feedback.Source: §E.3.2 · pp. 45–47

E.3.3 Data Generation

  • Data-generation prompts define exactly three high-, medium-, and low-proficiency communities with dimension-specific score ranges, but medical communities are based on recovery outcomes rather than expertise.Medical respondents are assigned excellent, moderate, or poor recovery profiles while sharing typical patient-level medical knowledge.Source: §E.3.3 · p. 48
  • Persona prompts use assigned scores to create distinct, coherent profiles whose backgrounds justify their proficiency and reflect their community characteristics.They require score-driven background creation, occupational and individual coherence, and consistency with the specified community.Source: §E.3.3 · pp. 49–50
  • Candidate profiles combine realistic career and learning histories with community-consistent diversity, while response simulation asks agents to embody each candidate’s identity.The prompt also directs responses to reflect background traits such as education, experience, interests, and location.Source: §E.3.3 · pp. 50–51
  • Synthetic respondents must answer according to their assigned proficiency scores, even when those scores conflict with their community or background descriptions.The prompt maps score ranges from 0–15 through 96–100 to response-level descriptions and treats the scores as ground truth.Source: §E.3.3 · pp. 51–52

E.3.4 Case-Study Prompt

  • The case-study prompt assigns a 0–100 score to eating-habit change based on behavioral evidence, prioritizing specific actions, contextual factors, and change rather than current state.It treats 50 as no significant change, with scores rising for healthier habits and falling for worsening habits; the rubric further grades improvement and decline by severity.Source: §E.3.4 · pp. 52–53
  • The scoring guide maps score bands from 90–100 significant improvement to 0–19 severe decline, with intermediate bands keyed to the strength of behavioral evidence.The prompt requests a condition name, score, brief evidence-based justification, and behavior-change level.Source: §E.3.4 · pp. 53–54

Related papers

Loading 2610.23153v1…