PaperlayerView exact revision on arXiv

Source-linked AI summary

CollabCoder: A Lower-barrier, Rigorous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Models

Jie Gao, Yuchen Guo, Gionnieve Lim, Tianqin Zhang, Zheng Zhang, Toby Jia-Jun Li, Simon Tangi Perrault

arXiv:2304.07366v4cs.HC

TL;DR

Rigorous CQA is valuable but costly and difficult to coordinate, especially when existing tools do not support a fluid, comprehensive workflow. CollabCoder integrates LLM assistance across independent coding, discussion, and codebook formation, and a 16-participant evaluation found user-friendly collaborative support and advantages over existing platforms. The paper also identifies scope and human-AI interaction limitations for future work.

  • Problem

    Existing CQA tools and workflows make rigorous adherence difficult because they lack centralized transitions between stages and impose coordination demands.

    Source: §1 · p. 2, §4.2 · p. 6

  • Method

    CollabCoder is a one-stop inductive CQA workflow that uses LLMs for code suggestions, shared discussion support, consensus-building, and code-group formation.

    Source: §1 · p. 2, §5.1.3 · p. 9

  • Results

    A 16-participant evaluation found CollabCoder user-friendly and effective for facilitating discussions, consensus-building, and codebook creation, with reported preference over Atlas.ti Web.

    Source: §1 · p. 3, §10 · p. 19

  • Takeaways & Limitations

    LLMs can support multiple CQA stages when their capabilities are balanced with user autonomy and collaborative decision-making.

    Source: §1 · p. 3, §10 · p. 19

    The current system assumes semantically independent data units, singular semantics per unit, and two-person coding teams.

    Source: §9 · p. 19

Abstract

from arXiv · show

Collaborative Qualitative Analysis (CQA) can enhance qualitative analysis rigor and depth by incorporating varied viewpoints. Nevertheless, ensuring a rigorous CQA procedure itself can be both demanding and costly. To lower this bar, we take a theoretical perspective to design the CollabCoder workflow, that integrates Large Language Models (LLMs) into key inductive CQA stages: independent open coding, iterative discussions, and final codebook creation. In the open coding phase, CollabCoder offers AI-generated code suggestions and records decision-making data. During discussions, it promotes mutual understanding by sharing this data within the coding team and using quantitative metrics to identify coding (dis)agreements, aiding in consensus-building. In the code grouping stage, CollabCoder provides primary code group suggestions, lightening the cognitive load of finalizing the codebook. A 16-user evaluation confirmed the effectiveness of CollabCoder, demonstrating its advantages over existing software and providing empirical insights into the role of LLMs in the CQA practice.

1 INTRODUCTION

CollabCoder addresses the cost, complexity, and coordination barriers that make rigorous CQA difficult by integrating LLM assistance across an end-to-end inductive workflow. A 16-user evaluation found strong usability and support for independent coding, disagreement resolution, shared understanding, and collaborative efficiency.

Motivation

  • Rigorous CQA requires independent coding and iterative discussion, but conventional tools impose substantial coordination work and can create barriers for less-experienced researchers.These limitations can reduce interactive discussion, coding rigor, and interpretive depth.Source: §1 · p. 2

Approach

  • CollabCoder integrates LLMs into independent open coding, code-decision discussions, and code-group formation within a unified CQA workflow.It combines automated code suggestions, shared decision data and agreement metrics, and initial code-group suggestions.Source: §1 · pp. 2–3

Evaluation

  • 75%+ of participants agreed or strongly agreed that CollabCoder was easy to use and quick to learn, particularly for beginners.The evaluation also reported support for coding independence, understanding disagreements, resolving disagreements, and understanding team agreement.Source: §1 · p. 3
  • CollabCoder supported collaborative efficiency by enabling code pairs to be resolved in a single dialogue, whereas Atlas.ti Web typically discussed only a few codes.The workflow was designed to reduce the need for multiple discussion rounds while supporting shared understanding.Source: §1 · p. 3

Contributions

  • The paper contributes design guidelines, an LLM-integrated inductive CQA system, and user-feedback-based insights into coding and collaboration.The contributions also examine how LLMs can support different stages of the CQA process.Source: §1 · p. 3

2 BACKGROUND OF QUALITATIVE ANALYSIS

Qualitative analysis transforms unstructured data into interpretive insights, while CQA uses independent coding and discussion to build rigorous consensus. CollabCoder adapts this tradition by emphasizing consensus-oriented, inductive analysis in which codes and the codebook evolve during analysis.

Foundations

  • Qualitative analysis interprets interviews, focus groups, observations, and other unstructured data to identify meaningful aspects of a situation or phenomenon.Grounded Theory and Thematic Analysis are common strategies for this work.Source: §2 · p. 3
  • Grounded Theory coding progresses from initial coding of data fragments to focused coding that refines codes through continuous comparison across data and existing codes.Thematic analysis instead identifies, analyzes, and explains recurring themes.Source: §2 · p. 3

CQA Framework

  • A six-step CQA methodology moves from planning and open or axial coding through codebook development, pilot testing, final coding, and codebook and theme finalization.The process embeds iterative refinement before the final codebook is established.Source: §2 · p. 3
  • Consensus coding has each coder independently analyze the same data and discuss disagreements, making it more rigorous but more time-consuming than split coding.Split coding is quicker but depends more heavily on preliminary clarity and predefined coding conventions.Source: §2 · p. 3

CollabCoder Positioning

  • CollabCoder emphasizes consensus coding and inductive analysis, supporting thorough discussion while allowing codes and the codebook to evolve during analysis.Its overarching aim is to lower the effort required to maintain rigor and reliability.Source: §2 · pp. 3–4

Key Concepts

  • In this paper, agreement is reached through discussions that scrutinize divergent viewpoints, while independence protects initial coding choices from external influence.The codebook is a hierarchical structure of code categories or themes, and intercoder reliability numerically measures coder agreement.Source: §2 · p. 4

3 RELATED WORK

Existing CQA tools address code comparison, disagreement identification, and selected AI-assisted tasks, but generally lack a comprehensive workflow spanning rigorous independent coding through codebook development. CollabCoder targets this gap by integrating LLM assistance across the full inductive CQA process.

3.1 Existing Tools for CQA

  • Existing CQA tools help coders compare codes, identify disagreements, inspect ambiguity, calculate agreement, and support consensus-building.Examples include Aeonium, Code Wizard, and disagreement-ranking approaches.Source: §3.1 · p. 4
  • Commercial platforms support shared coding and data exchange, but persistent visibility of codes can prevent independent open coding and encourage a less rigorous workflow.Atlas.ti Web provides real-time collaboration, while traditional tools often require exporting and importing coded documents.Source: §3.1 · p. 4
  • CollabCoder extends prior tools with a one-stop workflow that preserves coder independence, carries outputs between stages, and supports discussion based on agreements and disagreements.Its design is grounded in Grounded Theory and Thematic Analysis.Source: §3.1 · p. 4

3.2 AI-assisted (C)QA Systems

  • Prior AI-assisted qualitative-analysis work has focused mainly on individual coding, while emphasizing that AI’s timing, task allocation, and role require deliberate design.Suggested roles include preliminary analysis and generating codes from prior coding work.Source: §3.2 · p. 4
  • Earlier systems apply supervised learning, program synthesis, visualization, or shared coding histories to extend coding, reference human patterns, and accelerate shared understanding.These approaches informed CollabCoder’s consideration of AI throughout the CQA process.Source: §3.2 · pp. 4–5

3.3 Using LLMs in Qualitative Analysis

  • LLM studies report human-like theme generation and viable deductive coding with expert codebooks, while prompt design and coding setup can substantially affect performance.The cited work found codebook-centered designs outperforming example-centered designs in one comparison.Source: §3.3 · p. 5
  • A comprehensive end-to-end workflow that lowers adherence barriers and integrates LLMs across independent coding, iterative discussion, and codebook development remains largely unexplored.CollabCoder is proposed to address this gap while examining associated benefits and concerns.Source: §3.3 · p. 5

4 DESIGN GOALS

CollabCoder’s design goals seek to make rigorous CQA easier to follow by integrating standardized workflow stages, coder independence, synchronized data management, and LLM assistance. These goals were derived from literature, platform examination, and preliminary expert interviews, while recognizing trade-offs around coding-unit flexibility and segmented data.

4.1 Method

  • The design process combined a semi-systematic literature review, examination of prevalent CQA platforms, and preliminary interviews with five experienced qualitative researchers.Platform examination contributed DG3 and DG8, while interviews refined DG1 and DG4.Source: §4.1 · pp. 5–6

4.2 Results for Design Goals

  • CollabCoder targets a lower-barrier CQA workflow by connecting key stages so outputs transition directly into subsequent inputs.The design responds to existing workflows that are demanding, fragmented, and discouraging of standardized CQA adherence.Source: §4 · p. 5, §4.2 · p. 6
  • CollabCoder preserves independent open coding before enabling shared discussion and mutual code access at later CQA stages.This separation is intended to prevent premature exposure to others’ viewpoints while supporting later collaboration.Source: §4.2 · p. 6
  • A web-based platform balances coding independence with centralized synchronization, reducing the need for exporting, importing, and sharing documents manually.The goal is to retain independent coding while making team data management more convenient.Source: §4.2 · p. 6
  • Predefined coding units align coders’ interpretation levels for comparison and IRR calculation, but can constrain flexibility when meanings span multiple segments or require multiple codes.Expert feedback identified the workflow as an idealized scenario for cases involving overlapping codes or distributed meanings.Source: §4.2 · pp. 6–7
  • LLM assistance is designed to remain on demand, supporting coder autonomy while helping surface overlooked codes, resolve disagreements, and generate code groups.The design also seeks balanced participation and more efficient discussions by respecting each coder’s prior decisions.Source: §4.2 · p. 7

5 COLLABCODER SYSTEM

CollabCoder’s finalized system implements the proposed CQA workflow as a complete workflow and directs readers to Figure 3 for its detailed organization.

5 COLLABCODER SYSTEM

  • The finalized CollabCoder system embodies the designed CQA workflow.The authors refer readers to Figure 3 for the workflow’s organization.Source: §5 · p. 7
  • The system’s workflow is presented as an integrated sequence rather than as isolated features.This follows the authors’ statement that the system and workflow were finalized together.Source: §5 · p. 7
  • Figure 3 serves as the reference point for understanding the finalized workflow.The section explicitly directs readers to that figure.Source: §5 · p. 7

5.1 CollabCoder Workflow & Usage Scenario

The usage scenario organizes CollabCoder into independent coding, collaborative code merging, and code-group formation, beginning with shared project setup and consistent data units. GPT supports suggestions and decisions, while coders retain responsibility for discussion and finalization.

5.1 CollabCoder Workflow & Usage Scenario

  • CollabCoder divides the workflow into independent open coding, code merging and discussion, and final code-group formation.The process begins after the lead coder imports data, defines coding granularity, and invites collaborators.Source: §5.1 · p. 7, Figure 3 · p. 8

5.1.1 Phase 1: Independent Open Coding.

  • During Phase 1, coders work in separate workspaces without seeing one another’s codes, using personal codes, GPT recommendations, coding-history suggestions, keyword supports, and certainty ratings.Certainty ranges from 1 to 5, and each coder maintains an individual codebook.Source: §5.1.1 · p. 8

5.1.2 Phase 2: Code Merging and Discussion.

  • After both coders complete coding, CollabCoder presents their code pairs together, calculates similarity scores and IRR, and sorts pairs to prioritize discussion.The metrics include Cohen’s Kappa and Agreement Rate.Source: §5.1.2 · p. 8
  • Coders resolve differences by comparing raw data, code supports, and certainty information, with GPT offering three potential decisions when expression-level consensus is difficult.Low-similarity pairs require additional scrutiny of the information underlying divergent interpretations.Source: §5.1.2 · p. 8
  • Once decisions are finalized, coders can replace the original codes and organize the decisions into manually edited or GPT-generated code groups.The resulting groups support subsequent reporting of findings.Source: §5.1.2 · pp. 8–9, §5.1.3 · p. 9

5.2 Key Features

CollabCoder’s key features operationalize the three CQA phases through separate and shared interfaces, synchronized web-based workspaces, decision-support records, agreement metrics, and LLM assistance. Coders can request suggestions while retaining control over coding decisions and code-group editing.

5.2.1 Three-phase Interfaces.

  • Three interfaces support independent open coding, collaborative merging and discussion, and code-group generation as distinct CQA stages.The interfaces are an Editing Interface, Comparison Interface, and Code Group Interface.Source: §5.2.1 · p. 9

5.2.2 Individual Workspace vs. Shared Workspace.

  • Individual and shared workspaces preserve coding independence initially, then expose progress and coding information for integrated team discussion.The shared workspace becomes available after both participants complete their individual coding.Source: §5.2.2 · p. 9, §5.2.3 · p. 9

5.2.3 Web-based Platform.

  • The web-based platform synchronizes project data among collaborators without requiring downloads, imports, or additional exchanges after individual coding.This design aims to balance centralized management with independent coding.Source: §5.2.3 · p. 9, §5.2.4 · pp. 9–10

5.2.4 Consistent Data Units for All Users.

  • CollabCoder standardizes coding units such as sentences or paragraphs so team members interpret and compare equivalent data segments.The units are collaboratively determined before data import or coding begins.Source: §5.2.4 · pp. 9–10

5.2.5 LLMs-generated Coding Suggestions Once the User Requests.

  • On-demand GPT suggestions complement user-generated codes, including recommendations and the three most relevant codes from each coder’s current codebook.A brief delay gives users time to formulate their own codes first.Source: §5.2.5 · p. 10, Figure 5 · p. 10

5.2.6 A Shared Workspace for Deeper Discussion.

  • Shared discussion is supported by recorded keywords, certainty scores, side-by-side coding information, and similarity scores ranging from 0 to 1.Similarity scores can be calculated for all data units in 3–10 seconds and sorted to highlight stronger agreements.Source: §5.2.6 · p. 10

5.2.7 LLMs as a Group Recommender System.

  • For difficult code pairs, the LLM-based group recommender proposes three code decisions based on raw data, codes, keyword supports, and certainty scores.Users can select and customize the suggestions before reaching a final decision.Source: §5.2.7 · p. 10

5.2.8 Formation of LLMs-based Code Groups.

  • The Code Group interface lets teams inspect source data, edit decisions, and manually or automatically organize decisions into adjustable preliminary groups.Users can create, rename, delete, drag, remove, and modify groups after GPT assistance.Source: §5.2.8 · pp. 10–11

5.3 Prompts Design

CollabCoder uses GPT-3.5-turbo as a helpful qualitative analysis assistant across its CQA phases, with prompts tailored to different coding tasks. Its code-group interface supports editing, AI-assisted grouping, and saving final groups.

5.3 Prompts Design

  • Across CollabCoder’s three phases, GPT provides code, code-decision, and primary code-group suggestions to aid researchers in developing codes for subsequent qualitative-analysis stages.Prompts distinguish descriptive codes for raw data from relevant codes derived from coding history, with temperature set at 0.7.Source: §5.3 · p. 11
  • The code-group interface compiles unique codes, supports manual or AI-generated grouping, and lets users edit, rearrange, save, and update groups.Users can access original data while editing codes and drag code decisions into named groups.Source: Figure 7 · p. 12

5.4 System Implementation

CollabCoder implements synchronized individual and shared coding interfaces with backend support for GPT calls, data storage, preprocessing, and agreement analysis. Its semantic-similarity and IRR features quantify coding agreement to support comparison and discussion.

5.4.1 Web Application.

  • The web application auto-saves coding changes, synchronizes collaborators through a backend database, and restricts access to others’ codes until editing is complete.Separate Edit, Compare, and Codebook interfaces manage individual codes, shared decisions, and code groups.Source: §5.4.1 · p. 12
  • Raw CSV and TXT data are segmented into sentence- or paragraph-level data units during preprocessing.Sentence delimiters and double newlines determine the segmentation boundaries.Source: §5.4.2 · p. 12

5.4.2 Data Pre-processing.

  • Cohen’s Kappa measures inter-rater reliability from chance-adjusted agreement, while Agreement Rate reports the proportion of code pairs exceeding the similarity threshold.Agreement Rate ranges from 0 to 1, and similarity above 0.8 indicates agreement between coders.Source: §5.4.3 · p. 12

5.4.3 Semantic Similarity and IRR.

  • Semantic textual similarity compares each pair of coders’ codes for a data unit, with scores near 1 indicating agreement and scores near 0 indicating disagreement.The implementation uses the sentencetransformers package to calculate pairwise semantic similarity.Source: §5.4.3 · p. 13

6 USER EVALUATION

The evaluation used a within-subject comparison of CollabCoder and Atlas.ti Web with 16 participants coding two review datasets. Participants independently coded, discussed and finalized decisions, completed questionnaires, and participated in interviews that were thematically analyzed.

6 USER EVALUATION

  • The within-subject study involved 16 participants in eight pairs who used CollabCoder and Atlas.ti Web on two qualitative coding tasks.The study was designed to evaluate CollabCoder against an existing collaborative qualitative-analysis platform.Source: §6 · p. 13

6.1 Participants and Ethics

  • Participants varied substantially in experience, including experts, intermediate users, beginners, and participants without qualitative-analysis experience.The sample included 2 experts, 3 intermediate participants, 4 beginners, and 7 participants with no experience.Source: §6.1 · p. 13

6.2 Datasets

  • The tasks used 15 Business and 15 History book reviews selected to avoid domain-specific knowledge and support iterative theme development.Reviews were filtered to 400–700 characters and cleaned of odd symbols.Source: §6.2 · p. 13

6.3 Conditions

  • Atlas.ti Web provided collaborative code management, whereas CollabCoder was evaluated as the full-featured comparison platform, with presentation order counterbalanced by a Latin-square design.Counterbalancing reduced systematic effects from platform or material order.Source: §6.3 · p. 13

6.4 Procedure

  • Each two- to three-hour session included questionnaires, novice training, two platform-specific coding sessions, a post-study survey, and a semi-structured interview.Participants first coded independently, then discussed codes, finalized decisions, formed thematic groups, and completed a Likert-scale survey.Source: §6.4 · p. 13, §6.4.1 · p. 13, §6.4.2 · p. 13

6.4.3 Data Recording.

  • Interview transcripts and observation notes were analyzed thematically through familiarization, initial coding, grouping, discrepancy resolution, and quote extraction.The analysis followed Braun and Clarke’s methodology.Source: §6.4.4 · p. 13

7 RESULTS

CollabCoder supported independent coding, disagreement identification, shared understanding, and AI-assisted code grouping, and participants generally rated it more effective than Atlas.ti Web. However, users also reported risks from excessive AI or similarity reliance and uneven usefulness across features and datasets.

7.1.1 Key Findings (KF) on features that support CQA.

  • CollabCoder’s independent workspace and GPT suggestions reduced perceived cognitive burden, especially for beginners, while preserving coding independence.Seven of 16 participants valued GPT assistance, which provided references and reduced the need to generate codes from scratch.Source: §7.1.1 · pp. 13–14
  • Users preferred reading raw data before AI suggestions because suggestions shown first could influence interpretation and introduce coding bias.Participants described AI suggestions as confirmation of their understanding rather than a substitute for reading the source text.Source: §7.1.1 · p. 14
  • Predefined data units, documented decisions, progress tracking, and side-by-side comparison improved contextual understanding, disagreement identification, and discussion preparation.These features helped coders understand how collaborators’ codes arose and reduced effort spent revisiting source data and decision processes.Source: §7.1.1 · p. 14

7.3 RQ3. How can the design of CollabCoder be improved?

  • Similarity ranking could encourage coders to maximize agreement at the expense of more comprehensive interpretation, while detailed GPT summaries were sometimes unsuitable for users’ needs.Participants also questioned the usefulness of certainty, relevant-code, and individual-codebook functions across tasks and datasets.Source: §7.1.1 · p. 14, §7.3 · pp. 16–17

7.1.1 Key Findings (KF) on features that support CQA.

  • GPT-generated primary code groups gave participants a top-down reference structure that reduced the mental effort of organizing many codes compared with Atlas.ti Web’s manual bottom-up approach.Users could edit and rearrange AI-generated categories rather than create all groups from scratch.Source: §7.1.1 · p. 15

7.1.2 Key Findings (KF) on collaboration behaviors with CollabCoder supports.

  • Observed collaboration patterns included follower-leader, amicable cooperation, and swift but less cautious behavior across the two platform conditions.CollabCoder’s equal workload and documented decisions supported participation, while some pairs relied heavily on GPT to finish quickly.Source: §7.1.2 · pp. 15–16

7.2.1 Post-study questionnaire.

  • All p< .05, participants rated CollabCoder significantly better than Atlas.ti Web for code generation, final code groups, disagreement handling, agreement understanding, and understanding others’ thoughts.Participants also rated CollabCoder quicker to learn: M=4 versus M=3.1 for Atlas.ti Web, t(15)=-3.05, p<.01.Source: §7.2.1 · p. 16, Figure 8 · p. 16

7.2.2 Log data analysis.

  • IRR was significantly higher with CollabCoder (M≈1) than Atlas.ti Web (M=0.06, SD=0.40), alongside more discussed codes: 15 versus 4.5 on average.CollabCoder discussions lasted about 24 minutes versus 11 minutes for Atlas.ti Web, t(15)=-3.22, p=.017.Source: §7.2.2 · p. 16, Table 3 · p. 17

8 DISCUSSION AND DESIGN IMPLICATIONS

The discussion presents CollabCoder as a theory-aligned, end-to-end workflow that lowers the effort of rigorous CQA while using LLMs to assist coding, discussion, and code grouping. It also emphasizes preserving coder autonomy, improving suggestion relevance, and treating GPT as a mediator or facilitator rather than a replacement for human judgment.

8.1 Facilitating Rigorous, Lower-barrier CQA Process through Workflow Design Aligned with Theories

  • CollabCoder makes rigorous CQA more practical by integrating theory-aligned stages and seamless data transitions into one workflow.The design supports transparent, standardized, and manageable team coordination for both experienced practitioners and novices.Source: §8.1 · pp. 17–18

8.2.1 Utilizing LLMs to Reduce Cognitive Burden.

  • CollabCoder supports independent coding by using GPT as an on-demand suggestion provider for cognitively demanding interpretation tasks.Suggestions are intended to reduce mental burden without removing coders’ responsibility for developing codes.Source: §8.2.1 · p. 18

8.2.2 Improving LLMs’ Suggestions Quality.

  • LLM suggestions should incorporate researchers’ questions or intended coding direction because generic prompts may not match descriptive, in-vivo, sentiment, or opinion-focused coding.The discussion proposes research-question-aware or customizable prompts to improve contextual relevance.Source: §8.2.2 · p. 18

8.2.3 LLMs should Remain a Helper.

  • GPT assistance must preserve coder autonomy because beginners and experts may rely on suggestions when uncertain, while some users fear imposed codes and weakened interpretive ownership.The authors propose mixed-initiative designs that vary automation and may provide suggestions only for difficult cases or upon request.Source: §8.2.3 · p. 18

8.3.1 LLMs as a “Mediator” in Group Decision-Making.

  • In coding discussions, GPT can act as an impartial mediator that helps harmonize labor distribution and opinion expression when consensus is difficult to reach.The design frames GPT as a group recommendation system rather than an autonomous decision-maker, while substantive discussion remains necessary.Source: §8.3.1 · pp. 18–19

8.3.2 LLMs as “Facilitator” in Streamlining Primary Code Grouping.

  • GPT-generated primary code groups reduce the need to start from scratch, enabling coders to refine a top-down structure and compare their ideas with broader suggestions.Participants regarded this approach as more efficient and less cognitively demanding than manually merging and grouping all primary codes.Source: §8.3.2 · p. 19
  • Code-grouping assistance can weaken collaboration when time pressure leads one coder to complete grouping or when GPT dominates the grouping perspective.Suggested safeguards include having coders create initial groupings first and requesting LLM input afterward or for large datasets.Source: §8.3.2 · p. 19

9 LIMITATIONS AND FUTURE WORK

The authors identify scope and process limitations in CollabCoder and propose future work to support more complex data, user interaction with suggestions, and adaptive assistance.

Limitations

  • CollabCoder assumes semantically independent units, two-person teams, and single-semantic data units, although real coding may assign multiple codes to one unit.Future work should address disagreements arising from multiple codes on the same data unit.Source: §9 · p. 19
  • The study used predefined data units and did not examine splitting complex interview data into semantic units.Future research could use GPT to support segmentation and automate importing those units.Source: §9 · p. 19
  • The authors did not investigate how users select and edit GPT suggestions or how to balance autonomy with appropriate reliance.Future systems could generate suggestions from users’ coding patterns rather than directly providing them for repeated analysis of large datasets.Source: §9 · p. 19

10 CONCLUSION

CollabCoder integrates key CQA stages into a one-stop workflow intended to lower the burden of rigorous practice. In a 16-participant evaluation, users preferred it to existing platforms, while the study also identified challenges for future human-AI and human-human collaboration.

Conclusion

  • CollabCoder integrates key CQA stages into a one-stop workflow intended to lower the burden of following a strict CQA procedure.The system supports collaboration across coding, discussion, consensus-building, and codebook creation.Source: §10 · p. 19
  • In an evaluation with 16 participants, users preferred CollabCoder to Atlas.ti Web because of its user-friendly design and collaboration-oriented AI assistance.The evaluation also indicated support for discussions, consensus-building, and codebook creation.Source: §10 · p. 19
  • The study’s analysis of human-AI and human-human interactions identifies challenges and insights for future qualitative-analysis tool design and research.Source: §10 · p. 19
Loading 2304.07366v4…