PaperlayerView exact revision on arXiv

Source-linked AI summary

Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM

Michelle S. Lam, Janice Teoh, James Landay, Jeffrey Heer, Michael S. Bernstein

arXiv:2404.12259v1cs.HCcs.AI

TL;DR

Topic models and clustering often produce low-level text signals that require substantial interpretation, while theory-driven analysis calls for high-level concepts. LLooM uses LLMs to iteratively generate criteria-defined concepts and offers a mixed-initiative Workbench; across four scenarios, its concepts covered more examples on average than cluster-based topic models.

  • Problem

    Topic modeling and clustering often yield low-level signals that require analysts to interpret and validate, rather than high-level concepts suited to theory-driven analysis.

    Source: §1 · p. 2

  • Method

    LLooM uses iterative LLM-assisted concept generation and scoring to produce high-level concepts defined by explicit inclusion criteria, with a Workbench for interactive analysis.

    Source: §1 · p. 3, §3.1 · p. 5

  • Results

    Across four scenarios, LLooM concepts covered 93% of examples on average, compared with 77.7% coverage for cluster-based topic models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LLooM improves topic quality and coverage while providing steerability and interpretability for analysis of unstructured text.

    Source: §7 · p. 18

    The analysis scenarios used one LLooM iteration and approximately 200 sampled examples each, so concepts may differ on significantly larger datasets.

    Source: §4.5 · p. 12

Abstract

from arXiv · show

Data analysts have long sought to turn unstructured text data into meaningful concepts. Though common, topic modeling and clustering focus on lower-level keywords and require significant interpretative work. We introduce concept induction, a computational process that instead produces high-level concepts, defined by explicit inclusion criteria, from unstructured text. For a dataset of toxic online comments, where a state-of-the-art BERTopic model outputs "women, power, female," concept induction produces high-level concepts such as "Criticism of traditional gender roles" and "Dismissal of women's concerns." We present LLooM, a concept induction algorithm that leverages large language models to iteratively synthesize sampled text and propose human-interpretable concepts of increasing generality. We then instantiate LLooM in a mixed-initiative text analysis tool, enabling analysts to shift their attention from interpreting topics to engaging in theory-driven analysis. Through technical evaluations and four analysis scenarios ranging from literature review to content moderation, we find that LLooM's concepts improve upon the prior art of topic models in terms of quality and data coverage. In expert case studies, LLooM helped researchers to uncover new insights even from familiar datasets, for example by suggesting a previously unnoticed concept of attacks on out-party stances in a political social media dataset.

1 INTRODUCTION

Topic models produce low-level signals that require analysts to interpret and validate, while theory-driven analysis calls for high-level concepts with explicit inclusion criteria. LLooM iteratively generates and applies such concepts, and its Workbench lets analysts explore data through them.

1 INTRODUCTION

  • Topic models group text by low-level signals such as keywords, leaving analysts substantial work to interpret, validate, and reason about the resulting topics.The section contrasts those outputs with high-level concepts defined by explicit inclusion criteria.Source: §1 · p. 2
  • Concept induction seeks human-interpretable concepts defined by explicit criteria, rather than topic labels such as “women, power, female.”Examples include “Criticism of traditional gender roles” and “Dismissal of women’s concerns.”Source: §1 · p. 2
  • LLooM uses LLMs to synthesize sampled text into concepts, then repeats the process with lower-level concepts as input to produce broader concepts.The explicit criteria can also be applied to new data and used to identify gaps in the concept set.Source: §1 · p. 2
  • The LLooM Workbench visualizes data through interpretable concepts and lets analysts inspect concept traces and seed the algorithm to steer its attention.A concept can be unrolled into lower-level subconcepts, each paired with criteria and representative examples.Source: §1 · p. 2
  • Across four scenarios, LLooM concepts covered 93% of examples on average, compared with 77.7% coverage for cluster-based topic models.LLooM also provided an average of 2.0 times the number of high-quality topics.Source: §1 · p. 2
  • In expert case studies, researchers used the Workbench to steer concepts and pursue theory-driven explorations of familiar datasets.Examples included refining a policy-related concept and adding a concept for social distrust.Source: §1 · p. 3

2 RELATED WORK

The related work situates LLooM among topic modeling, qualitative coding, and mixed-initiative analysis tools. It positions LLooM as combining iterative concept development with explicit criteria and bidirectional interaction.

2 RELATED WORK

  • LLooM draws on prior work in topic modeling and unsupervised clustering, qualitative analysis, and mixed-initiative data analysis tools.These strands provide context for a concept-centered approach to understanding and interacting with data.Source: §2 · p. 3

2.1 Topic Modeling and Clustering: Automated Concept Development

  • Topic models and embedding-based clusters can produce incoherent or surface-level topics because their signals may not align with human judgments of semantic similarity.LLooM addresses this gap by supporting extraction of interpretable, high-level concepts from unstructured text.Source: §2.1 · pp. 3–4

2.2 Qualitative Analysis: Manual Concept Development

  • LLooM draws on qualitative coding’s iterative interpretation, thematic synthesis, application back to data, and refinement toward broader insights.Prior qualitative analysis methods involve substantial manual review and interpretation.Source: §2.2 · p. 4
  • Unlike systems focused on human-led inductive coding or applying existing codebooks, LLooM explores AI-initiated concept generation to extend analysts’ possible directions.Its concepts pair human-understandable descriptions with explicit inclusion criteria, span broad and nuanced attributes, and serve as both interpretive outputs and inputs for authoring concepts.Source: §2.2 · p. 4, §2.3 · p. 4

2.3 AI-Assisted Data Analysis: Mixed-Initiative Concept Development

  • LLooM’s mixed-initiative approach builds on tools that support visualization, iteration, and user-understandable abstractions for data analysis.It differs by making concepts bidirectional: outputs for interpreting data and inputs for authoring concepts and investigating questions.Source: §2.3 · p. 4

3 LLOOM: CONCEPT INDUCTION USING LARGE LANGUAGE MODELS

Concept induction turns unstructured text into high-level concepts defined by explicit criteria. LLooM uses iterative LLM-based generation and scoring, supported by operators that distill and group text before concepts are applied back to data.

3.1 The LLooM Algorithm

  • Concept induction produces emergent, high-level concepts with explicit criteria; LLooM iteratively generates and scores concepts to bridge low-level text signals to human-interpretable descriptions.The algorithm can generalize from lower-level concepts to broader ones by repeating the process.Source: §3 · p. 4, §3.1 · p. 5

3.1.4 Algorithm Limitations.

  • LLooM’s limitations include generic direct synthesis, context-window constraints, parameter defaults not robustly tested across datasets, missing verification, and run-to-run variability.The authors note that variability may be undesirable when analyses must be replicable or alignment must be robust and consistent.Source: §3.1 · p. 5, §3.1.4 · p. 8

3.1.1 Concept Generation.

  • Synthesize proposes concepts from grouped examples, including names, inclusion criteria, and representative examples; it identifies salient subsets rather than labeling whole clusters.An optional seed steers Distill or Synthesize toward an analyst-selected aspect of the data.Source: §3.1.1 · pp. 5–7
  • Distill filters text to exact quotes and summarizes it into bullets; Cluster pools and groups the bullets using embeddings and HDBSCAN.Distill addresses context-window limits, while clustering groups related items before synthesis.Source: §3.1 · p. 5, §3.1.1 · pp. 6–7

3.1.2 Concept Scoring.

  • Score applies each concept’s criteria to dataset examples, producing scores that can be thresholded into labels; Loop sends uncovered or only generically covered examples into another generation round.The next round adds its concepts to the full set.Source: §3.1.2 · p. 7

3.1.3 Implementation Details.

  • The implementation uses GPT-3.5 for most operators, GPT-4 for Synthesize, and averages $1.44, 848,323 tokens, and 13.7 minutes per run across scenarios.Scoring accounts on average for 79.9% of total cost and 58.4% of total time.Source: §3.1.3 · p. 7

3.2.1 Workbench Components.

  • The Workbench displays concepts interactively; its matrix compares prevalence across user-defined slices, while concept and slice detail views expose matching examples and concept criteria.Users can also inspect subconcepts, representative examples, and prevalence histograms in the concept detail view.Source: §3.2 · p. 8, §3.2.1 · pp. 8–9

3.2.2 Workbench Actions.

  • Workbench users can add or edit concepts and rescore them, or merge related concepts and split concepts that are too general.Merging generates a combined name and criteria; splitting generates subconcepts.Source: §3.2.2 · p. 9

4 LLOOM SCENARIOS

Across three analysis scenarios, LLooM surfaced interpretable concepts that often went beyond keyword-based topics and left fewer examples uncategorized than BERTopic. The scenarios used a single LLooM iteration and samples of about 200 examples, limiting representativeness and preventing global-coverage comparisons against manual ground truth.

4 LLOOM SCENARIOS

  • The scenarios spanned content moderation, partisan animosity, HCI literature analysis, and anticipated AI consequences, using BERTopic as the qualitative baseline.LLooM runs sampled up to 200 items and generated at most 20 final concepts to prioritize interactive completion times.Source: §4 · p. 9, §4.1 · p. 9, §4.1.1 · p. 9

4.1.2 Baseline qualitative analysis.

  • Researchers manually labeled BERTopic topics, then conservatively grouped overlapping LLooM concepts before comparing shared meaning.Grouping included synonymous concepts and cases where one concept was a subset of another.Source: §4.1.2 · pp. 9–10

4.2 Scenario 1: Developing Moderation Policies for Toxic Content

  • In the toxic-content scenario, 44.4% of examples were uncategorized by BERTopic versus 9.5% by LLooM, whose concepts also captured attitudes and interpretations beyond surface-level keywords.The methods overlapped on 50% of BERTopic results and 40% of LLooM results; LLooM’s concepts included content, style, and tone.Source: §4.2.1 · p. 10, Figure 5 · p. 10

4.3 Scenario 2: Mitigating Partisan Animosity on Social Media

  • For partisan-animosity posts, 26.2% were uncategorized by BERTopic versus 2.5% by LLooM, which also surfaced novel political issues and user behaviors.LLooM covered 87.5% of BERTopic results, while BERTopic covered 50% of LLooM results.Source: §4.3.1 · p. 11, Figure 7 · p. 12

4.4 Scenario 3: Analyzing UIST Paper Abstracts

  • For UIST abstracts, 18.6% of examples were uncategorized by BERTopic versus 6.7% by LLooM, whose concepts included higher-level methods and evaluation strategies.LLooM covered 83.3% of BERTopic results, and BERTopic covered 62.5% of LLooM results.Source: §4.4.1 · pp. 11–12, Figure 8 · p. 12

4.5 Scenario Limitations

  • The scenario runs used one LLooM iteration and samples of approximately 200 examples, risking concepts that are not representative of larger datasets.The authors note that induced criteria can still be applied to larger datasets to assess generalizability and coverage.Source: §4.5 · p. 12
  • Without manual ground-truth annotations, the scenarios cannot establish global concept coverage or alignment with analysts’ concepts, and results may differ on other dataset types.Source: §4.5 · p. 12

5 TECHNICAL EVALUATIONS

Technical evaluations compare LLooM with BERTopic and GPT-4 variants on annotated real-world and controlled synthetic datasets. LLooM particularly improves coverage of specific concepts while maintaining strong coverage of generic concepts.

5.1.2 Method.

  • The benchmark used annotated Wikipedia articles and Congressional bill summaries, with sampled documents for GPT-4 baselines; Wiki mappings may have appeared in pretraining, unlike the less likely-seen Bills data.Source: §5.1 · p. 13, §5.1.2 · p. 13, §5.1.3 · p. 13

5.1.1 Metric.

  • Coverage measures the proportion of human-annotated ground-truth topics matched by generated concepts, with automated matching checked against manual matches in sampled trials.The automated measure had a mean absolute error of 0.07 in the verification sample.Source: §5.1.1 · p. 13

5.1.4 Results.

  • On Bills, LLooM coverage was 0.74 versus 0.56 for GPT-4 Turbo; on Wiki, LLooM coverage was 0.81, comparable to GPT-4 at 0.83 and GPT-4 Turbo at 0.82.BERTopic coverage was 0.29 on Bills and 0.63 on Wiki; the authors caution that Wiki results may be inflated by pretraining exposure.Source: Figure 9 · p. 13, §5.1.4 · pp. 13–14
  • Across benchmark datasets, LLooM produced concepts that both matched ground-truth topics and described distinct characteristics within topic areas.Examples included “Community Development” within education and “Character Design” within video games.Source: §5.1.4 · p. 14

5.2.1 Dataset generation.

  • The synthetic evaluation varied document length and within-document concept prevalence while holding ground-truth concepts and their across-document prevalence constant.It used ten generic concepts, each with four specific concepts, and generated documents with 5 or 10 sentences and 20% or 40% concept prevalence.Source: §5.2 · p. 14, §5.2.1 · pp. 14–15

5.2.3 Results.

  • Specific concept coverage was 0.71 for LLooM versus 0.55 for GPT-4 Turbo; Generic concept coverage was 0.98 for both LLooM and GPT-4 Turbo.These trends held across document lengths and concept prevalence levels; Specific coverage was highest for LLooM, while Generic coverage did not differ significantly from GPT-4 Turbo.Source: Figure 11 · p. 15, §5.2.3 · p. 15, Figure 12 · p. 16

5.3 Concept Classification

  • LLooM’s Score operator achieved inter-rater reliability near human annotators’ and accuracy of 0.91 with precision of 0.70 on subjective concepts.Reported reliability was κ=0.63 and κ=0.645 for LLooM, compared with κ=0.64 for human annotators.Source: §5.3 · p. 16

6 EXPERT CASE STUDIES

In exploratory case studies with researchers familiar with their datasets, LLooM concepts were rated more favorably than BERTopic topics and supported analysts’ own explorations. Analysts used concepts to uncover unexpected patterns, test hypotheses, and consider theory-driven analyses.

6 EXPERT CASE STUDIES

  • The case studies were exploratory probes with two experienced analysts who had previously analyzed the same datasets.Each one-hour session included BERTopic and LLooM analysis tasks and a concluding interview.Source: §6 · p. 16
  • 90% of Expert 1’s LLooM concepts were helpful (vs. 62.5% of BERTopic topics); Expert 2 rated 100% of LLooM concepts helpful (vs. 66.7%).Experts also consistently rated a higher proportion of LLooM concepts helpful, interpretable, and unique.Source: §6.1 · p. 17, Figure 13 · p. 17, §6.2 · p. 18

6.1.1 BERTopic Analysis Process—Making sense of vague and over-

  • The analyst struggled to distinguish overlapping BERTopic topics, while LLooM let them inspect data through concepts without first interpreting each label.The analyst spent time trying to explain keywords and compare similar topics, but could not identify a meaningful difference.Source: §6.1.1 · p. 17, §6.1.2 · p. 17

6.1 Expert 1: Mitigating Partisan Animosity on Social Media

  • The first analyst found an unanticipated “Political Party Positions” concept that surfaced attacks on out-party stances in high-animosity posts.They also used LLooM to explore policy-related crises and create a “Social Distrust” concept; those examples had mid-to-high, but not the highest, animosity scores.Source: §6.1.2 · p. 17

6.1.3 Interview Takeaways.

  • The analyst described BERTopic as useful for visualizing broad patterns, but LLooM as better for navigating concepts and examples and potentially supporting early qualitative coding.They also cautioned that an initial categorization could affect their judgment.Source: §6.1.3 · p. 17

6.2.1 BERTopic Analysis Process—Dealing with incoherent and overly-

  • The second analyst struggled to interpret or label generic and incoherent BERTopic topics, including three topics (25%) they could not manually label.Earlier LDA use on this dataset had required time-consuming iterations to remove stopwords and optimize clusters.Source: §6.2.1 · p. 18

6.2.2 LLooM Analysis Process—Leveraging concepts to explore hy-

  • LLooM let the second analyst compare research trends over time, author an “AI” concept, and find AI-related papers that did not explicitly use the term.The analyst reported a steady rise in AI-related papers across decades and noted that an example mentioning “object recognition” might be hard to anticipate through keywords alone.Source: §6.2.2 · p. 18

6.2.3 Interview Takeaways.

  • The second analyst valued LLooM for enabling hypothesis-driven analysis that had been difficult with existing tools, while noting a need for reliability indicators and greater visibility into its process.They particularly wanted quantitative measures to help calibrate trust in the outputs.Source: §6.2.3 · p. 18

7 DISCUSSION

The discussion frames LLooM as a way to center text analysis on interpretable concepts, linking data-driven patterns to analysts’ theories. It also identifies limitations involving LLM reliability, closed-source models, cost, and possible effects on analyst agency.

7.1.1 Redesigning data analysis abstractions to support theory-

  • LLooM orients analysis around human-understandable concepts in natural language, letting analysts steer attention toward analytic goals and apply their theories to emergent patterns.The discussion describes this as a shift from only enacting an analysis process to also reflecting on it.Source: §7.1.1 · p. 18

7.1.2 Introducing automation to aid reflection on analysis processes.

  • Automation can distill data, group relevant items, synthesize trends into concepts, and apply concepts to categorize data, while helping analysts notice gaps and explore alternative interpretations.Analysts can also work in a closed loop when they want to inject their own perspectives.Source: §7.1.2 · pp. 18–19
  • The implementation combines Distill, Cluster, and Synthesize operators in one architecture, while other operator arrangements and expert tasks remain open design possibilities.The authors propose dynamically rearranging operators for different analyst goals and developing operators for other domain experts.Source: §7.1.3 · p. 19

7.2.1 Uncertain LLM behaviors: risks of uneven cross-domain perfor-

  • LLMs lack reliability and performance guarantees, and performance may decline in specialized domains such as law, medicine, or technical fields.Human review can help catch failures by tracing concepts to source examples and reviewing scores and rationales, but cannot eliminate foundational errors.Source: §7.2.1 · p. 19

7.2.2 Drawbacks of closed-source LLMs: cost and lack of trans-

  • Using closed-source GPT models limits transparency into training data and model design, and API model changes can impede reproducibility.The authors chose these models because they were state-of-the-art and other tested models could not reliably perform central synthesis operations.Source: §7.2.2 · p. 19
  • External API charges and token limits make LLooM costly to run at extremely large scales, particularly for concept scoring.The authors suggest distilled models trained on a smaller set of LLM-labeled examples or open-source LLMs as possible ways to reduce cost and inference time.Source: §7.2.2 · p. 19

7.2 Limitations and Future Work

  • Overreliance on generated concepts or scores may bias analysts or cause them to miss patterns outside the concepts, while LLM values and biases may shape outputs.The authors call for tools that help users calibrate trust, verify outputs, inspect results, and retain control over follow-up analyses.Source: §7.2.3 · p. 19

8 CONCLUSION

The conclusion presents LLooM as a method for turning unstructured text into high-level concepts with explicit inclusion criteria, aimed at making theory-driven analysis more interpretable. It reports that LLooM’s concepts exceed topic models in quality and can support analysts in exploring data through actionable concepts.

8 CONCLUSION

  • LLooM performs concept induction by iteratively sampling unstructured text and synthesizing high-level concepts of increasing generality.The concepts are human-interpretable descriptions with explicit inclusion criteria that can be applied back to data.Source: §8 · p. 19
  • The conclusion reports that LLooM concepts exceed topic models in quality and support analysts in leading theory-driven analyses of unstructured text.It presents concepts as interpretable and actionable ways to read data patterns and apply theories back to the data.Source: §8 · p. 19

B ADDITIONAL METHODS

The expert case study recruited two analysts who had previously conducted research using unstructured text datasets. Participants completed BERTopic and LLooM analysis tasks on their familiar data, then annotated the outputs and discussed their experience.

B.1 Expert Case Study: Study Design

  • The study recruited two analysts with prior publications based on publicly shareable unstructured-text datasets, selected for their expertise and familiarity with the data.Both participants were new to LLooM Workbench before the session.Source: §B.1 · pp. 22–23
  • Participants explored BERTopic topics and LLooM concepts, then annotated outputs for helpfulness, interpretability, and uniqueness.The LLooM task also asked them to edit or add a concept.Source: §B.1 · p. 23
  • Each remote session allotted 15 minutes to analysis with each tool, followed by interviews and an LLooM tutorial.The session also included setup and consent and a concluding interview.Source: §B.1 · p. 23

C ADDITIONAL RESULTS

In the NeurIPS broader-impact statements, LLooM generated 14 unique concepts, while BERTopic generated two topics. The concepts include specific anticipated impacts and proposed solutions.

C ADDITIONAL RESULTS

  • The analysis explores how LLooM could help examine downstream consequences, ethical issues, and potential mitigations discussed by AI researchers.It uses NeurIPS broader-impact statements, introduced as a window into researchers’ ethical thought processes.Source: §C.1 · p. 23
  • LLooM generated 14 unique concepts, compared with two BERTopic topics for the NeurIPS statements.Examples include “Adversarial Attacks and Defenses,” “Privacy Concerns,” and “Energy Conservation.”Source: §C.1.1 · p. 23
  • LLooM concepts included specific anticipated impacts, including energy conservation, privacy concerns, and adversarial attacks.They also included proposed solutions such as adversarial defenses and verification.Source: §C.1.1 · p. 23

C.1.1 Results.

For NeurIPS broader-impact statements, LLooM produced concepts that extended beyond BERTopic’s topics, which were often generic. Its concepts captured specific positive and negative impacts as well as proposed solutions.

C.1.1 Results.

  • LLooM identified specific positive impacts, including energy conservation and improved training techniques, and negative impacts such as privacy concerns and adversarial attacks.It also captured proposed solutions, including adversarial defenses and the importance of verification.Source: §C.1.1 · pp. 23–24
  • BERTopic’s topics were generic enough to apply to nearly all impact statements, so they did not help break the data into emergent trends.LLooM also produced some generic concepts, such as “Societal Impact.”Source: §C.1.1 · p. 23
  • 100% of BERTopic results overlapped with LLooM, while only 14.3% of LLooM results overlapped with BERTopic.The authors describe the additional LLooM concepts as novel contributions.Source: §C.1.1 · p. 24
  • BERTopic assigned 93.3% of examples to a vague catch-all topic and left none uncategorized; LLooM left 9.3% of examples uncategorized.Source: §C.1.1 · p. 24

C.2 Concept Classification Evaluation

The evaluation compares LLooM’s automated concept scores with human annotations on sampled documents. Concepts were sampled across prevalence levels, and the selected examples were used to assess alignment with human judgment.

C.2 Concept Classification Evaluation

  • The evaluation compares LLooM’s automated concept scores with human annotations on documents for sampled concepts.The stated aim is to assess alignment with human judgment.Source: §C.2 · p. 24
  • For each scenario dataset, the researchers sampled one concept from each prevalence quartile, selecting four concepts per dataset.Prevalence is the proportion of documents LLooM classified as matching a concept.Source: §C.2.1 · p. 24
  • For each selected concept, the researchers built a balanced 100-document set with 50 LLooM-positive and 50 negative documents, except when rare concepts had fewer than 50 positives.In that case, the remaining documents were randomly sampled negatives.Source: §C.2.1 · p. 24

C.2.1 Method.

The evaluation reports mean classification metrics for LLooM’s concept scoring and notes wide variation across datasets and concepts. Human annotators assessed examples using each concept’s name and inclusion criteria.

C.2.1 Method.

  • The NeurIPS sample includes concepts framed as questions about verification, new frameworks, potential benefits and risks, and wide application space.Source: §C.2.1 · p. 24
  • Two team members independently annotated the four sampled Partisan Animosity concepts, while one rater annotated concepts for the other three datasets.Each Partisan Animosity rater annotated 400 documents.Source: §C.2.1 · p. 24
  • Annotators judged document matches using each concept’s name and inclusion criteria, with response options from “strongly agree” to “strongly disagree.”The options matched those given to GPT-4 in LLooM’s Synthesize operator prompt.Source: §C.2.1 · p. 24
  • Mean accuracy was 0.91, precision 0.70, recall 0.59, and F1 score 0.59 for LLooM’s concept classification.Performance varied widely across datasets and concepts; Table 1 reports per-dataset means and standard deviations.Source: §C.2.2 · p. 24, Table 1 · p. 24

C.2.2 Results.

LLooM’s concept annotations achieved inter-rater reliability comparable to human annotators, while disagreements reflected subjective judgments and differences in expertise. The technical evaluation also presents sampled concept-generation outputs for Wiki, Bills, and synthetic datasets.

C.2.2 Results.

  • LLooM achieved high classification accuracy across four datasets, but performance varied across datasets and concepts.The per-concept metrics include accuracy, recall, precision, and F1-score.Source: Figure 15 · p. 25, §C.2.2 · p. 25
  • False positives often reflected LLooM applying a broader threshold to borderline examples, where concept judgments could reasonably differ.For example, annotators disagreed over whether a political post implicitly advocated for a cause.Source: §C.2.2 · pp. 24–25
  • Cohen’s κ was 0.64 between human annotators, 0.63 between LLooM and A1, and 0.645 between LLooM and A2, indicating comparable agreement.Agreement was moderate to high across the four concepts.Source: §C.2.2 · p. 24, Table 2 · p. 25, Figure 16 · p. 25
  • False negatives often involved examples requiring deeper expertise or nuance, such as recognizing Head Start references as advocacy.The authors identify this as a possible failure mode for LLMs underlying LLooM’s Score operator.Source: §C.2.2 · p. 25

C.3 Technical Evaluation: Concept Generation Outputs

  • The output examples sample ground-truth topics and matching generated concepts from LLooM, BERTopic, GPT-4, and GPT-4-Turbo across Wiki, Bills, and synthetic datasets.The figures show samples for three topics per dataset, drawn from across trials.Source: §C.3 · p. 26, Figure 17 · p. 26

C.4 Technical Evaluation: Synthetic Dataset Concepts

The synthetic dataset is built from generic concepts paired with more specific concepts, spanning political, social, and policy topics. Figure 19 presents sample results for LLooM and baseline methods.

C.4 Technical Evaluation: Synthetic Dataset Concepts

  • The synthetic dataset uses 10 generic concepts and 40 specific concepts.The listed topics include election campaigns, government policies, political parties, and human rights.Source: §C.4 · p. 26
  • Its concept hierarchy includes immigration, economy, healthcare, and environment topics alongside more specific subtopics.Examples include border control policies, unemployment, mental health, and renewable energy.Source: §C.4 · p. 26
  • Foreign policy and gun control are also represented by specific concepts such as trade agreements, diplomatic relations, background checks, and assault weapons bans.Source: §C.4 · p. 26
  • Figure 19 shows sample synthetic-dataset results for LLooM and baseline methods.Source: Figure 19 · p. 28

Related papers

Loading 2404.12259v1…