PaperlayerView exact revision on arXiv

Source-linked AI summary

EviMap: Evidence-Grounded Hierarchical Topic Maps for Exploring Unlabeled Corpora

Zhiyin Tan, Changxu Duan

arXiv:2609.06664v1cs.IRcs.CLcs.HC

TL;DR

Unlabeled free-text collections are difficult to explore at scale before useful labels or queries are known, and existing approaches trade scale against verifiability. EviMap builds a phrase-grounded hierarchy with interactive evidence inspection; across six dashboards, traceability checks found no missing document references or span-text mismatches.

  • Problem

    Exploring unlabeled free-text at scale requires discovering themes before labels or queries exist, while current approaches trade scale against verifiability.

    Source: §1 · pp. 1–2

  • Method

    EviMap builds a three-level hierarchy bottom-up from evidence phrases, using embedding clusters to narrow LLM semantic grouping and linking nodes to source spans.

    Source: §1 · p. 2, §3 · pp. 2–3

  • Results

    3,290,920 topic-evidence links and 4,357,621 character-offset spans across six dashboards had no missing document references or span-text mismatches.

    Source: §6 · p. 4

  • Takeaways & Limitations

    Across six heterogeneous corpora, the dashboard supports map browsing, evidence inspection, topic intersection, and long-tail auditing while treating labels as auditable claims rather than a gold taxonomy.

    Source: §8 · p. 4

    Model-generated extraction context may omit important aspects, small-topic folding may obscure minority concerns, and LLM grouping and labeling remain fallible.

    Source: §8 · p. 4

Abstract

from arXiv · show

Research teams and organizations often explore unfamiliar free-text collections, from survey comments and reviews to reports and domain documents, before labels, queries or coding schemes exist. At this stage, the first thematic map shapes what users notice, prioritize and carry into downstream analysis, so it should be trusted only insofar as it can be verified. Existing options force a trade-off between scale and verifiability. Qualitative coding preserves evidence but is slow. Search presupposes a query. Clustering and topic models scale but produce labels users must interpret. One-shot large language model (LLM) summaries are fluent yet difficult to reproduce or audit. We present EviMap, an interactive system providing researchers and practitioners with an auditable thematic overview of such corpora. Guided by model-generated context describing the corpus and hypothesized stakeholder concerns, EviMap extracts within-document evidence phrases and organizes them, rather than whole documents, into a three-level map of aspects, groups and fine-grained topics. Embedding-based clustering narrows the search space for finer semantic judgments by the LLM. Each node traces back to supporting phrase spans, so documents link to topics through evidence they contain and users can audit labels against the original text. Users can start from a top-level corpus map, drill into topics, inspect highlighted evidence in original documents, and combine two topics to find documents discussing both. We demonstrate this workflow across six heterogeneous corpora spanning 2,108 to 101,699 documents, with a comparison against flat and hierarchical LLM baselines. By grounding every label in verbatim source spans, EviMap makes a topic map not just readable, but verifiable. Code, demo video, and interactive dashboard are available at https://github.com/zhiyintan/EviMap.

1 Introduction

Early corpus understanding requires discovering themes before labels, queries, or codes exist, while keeping the evidence behind those themes verifiable. EviMap responds with a phrase-grounded hierarchy and interactive exploration.

1 Introduction

  • Unlabeled free-text collections expose unexpected concerns and minority viewpoints, but exhaustive reading becomes impractical at scale and analysts must discover concepts before labels or queries.One cited health-services project faced over half a million comments across 320,500 surveys.Source: §1 · p. 1
  • Manual coding preserves context but does not scale; search requires a query, while topic models and one-shot LLM summaries leave labels difficult to interpret or verify.The cited discussion also notes topic-model instability and weak correlation between automated coherence scores and human judgment for modern models.Source: §1 · pp. 1–2
  • EviMap links documents to topics through evidence they contain, grounding every node in verbatim spans and avoiding document-level assignments that may include documents that never raised a topic.Documents can appear under multiple nodes, preserving heterogeneous detail.Source: §1 · pp. 1–2
  • EviMap induces a hierarchy bottom-up from evidence phrases, combining embedding-based clustering with LLM semantic judgments without imposing a predefined taxonomy.Source: §1 · p. 2
  • The dashboard supports map browsing, evidence inspection, topic-intersection retrieval, and long-tail auditing across six heterogeneous corpora.Source: §1 · p. 2

2 Related Work

Related work spans exploratory sensemaking and qualitative tools, topic models and visual analytics, and LLM-based topic or taxonomy systems. EviMap distinguishes itself by making a phrase-grounded, browsable hierarchy its primary artifact.

2 Related Work

  • Qualitative tools organize codes and excerpts but still require user-curated coding schemes, while topic-model visual analytics systems inspect outputs that EviMap instead induces from inspectable evidence.The related work frames EviMap as targeting discovery before precise queries and candidate coding schemes.Source: §2 · p. 2
  • EviMap clusters extracted evidence phrases rather than documents, grounds leaf topics in highlighted sub-sentence spans, and aggregates repeated grouping decisions through co-association voting.TopicGPT [30] generates natural-language topics with supporting quotations and can be extended hierarchically; EviMap instead makes the browsable three-level map primary.Source: §2 · p. 2
  • Table 1 compares nearby systems using demo-facing criteria, with Map, Support, and Stabilizer denoting a persistent map, exposed evidence, and an explicit mechanism against unstable grouping.Source: §2 · p. 2, Figure 1 · p. 2

3 System Architecture

EviMap converts heterogeneous corpora into a phrase-level evidence index, then repeatedly groups phrases and topics to build a three-level map. Model-generated context guides extraction, while categories are induced from the extracted evidence.

3 System Architecture

  • The system normalizes records, uses a sampled corpus profile and hypothesized stakeholder concerns to guide phrase extraction, and builds categories from evidence rather than a preset list.Each extracted phrase is indexed by document occurrence, deduplicated, assigned IDF, embedded, and stored with character offsets for highlighting in context.Source: §3 · pp. 2–3
  • Fine-grained embedding clusters narrow the context for LLM semantic grouping; multi-round random-subset voting links phrase pairs only when their co-association exceeds a threshold.This is intended to reduce the effect of poor coarse clusters and resist over-aggressive merging from spurious links.Source: §3 · p. 3
  • The grouping operation first forms fine-grained topics, then mid-level groups and top-level aspects; small topics are folded into related topics or retained in a visible Other bin.Labels name concrete objects, actions, or relations, and prompts use abstract rules without in-domain examples to support transfer across corpora.Source: §3 · p. 3

4 The Interactive Dashboard

The dashboard lets users inspect the induced map and the source evidence behind candidate labels without editing the map. Users can also combine topics to retrieve documents discussing both.

4 The Interactive Dashboard

  • Users expand aspects into groups and topics, inspect support counts and representative phrases, and view highlighted evidence in its original document context.The dashboard is for inspecting and challenging the candidate map, not committing rename, merge, or split operations.Source: §4 · p. 3
  • Combining two topics retrieves documents that discuss both, while the Other bin, support counts, representative phrases, and highlighted evidence help users audit uncertain labels.Source: §4 · p. 3

5 Demonstration Scenario

The demonstration shows how users browse a topic hierarchy, inspect its source evidence, and retrieve documents that discuss two selected topics.

5 Demonstration Scenario

  • Drilling through three job-posting levels reaches Bilingual/Multilingual Skill, whose card reports 383 supporting documents and representative bilingual phrases.The path starts at Interpersonal & Service Skills and passes through Language Concepts & Skills.Source: §5 · p. 3
  • The document pane highlights current-topic evidence in orange and other-topic evidence in gray, and clicking a gray span jumps to that topic.Source: §5 · p. 4
  • Combining Bilingual/Multilingual Skill with Customer Success retrieves job postings discussing both, while the peer-review example shows the prompts and grouping rules are not job-specific.The demonstration also exposes the Other bin and run summary so attendees can inspect unresolved long-tail topics.Source: §5 · p. 4

6 Demonstration Corpora

The evaluation covers six heterogeneous corpora, validating stored evidence traceability and comparing topic soundness with flat and hierarchical LLM baselines.

6 Demonstration Corpora

  • Across six dashboards, 3,290,920 topic-evidence links and 4,357,621 character-offset spans had no missing document references or span-text mismatches; this validates traceability, not topic quality.Source: §6 · p. 4
  • On job postings, EviMap produced 14 induced aspects plus Other, 350 groups, and 3,245 leaf nodes, compared with 24 flat TopicGPT-style topics and 8 top-level plus 32 child TnT-LLM-style categories.Source: §6 · p. 4
  • Across six corpora, EviMap leaf topics averaged 4.53/4.33 (4.30/4.04 with Other bins) for evidence coherence/label soundness, versus TopicGPT-style 3.60/3.50 and TnT-LLM-style 3.96/3.79.Source: §6 · p. 4
  • EviMap led on coherence in five corpora and label soundness in four, while TopicGPT-style led on hotel reviews; the diagnostic measures semantic soundness, not topic accuracy.Source: §6 · p. 4

7 System Realization

EviMap is a reproducible, configurable pipeline whose submitted build uses deterministic LLM calls and a multilingual encoder, with extraction dominating runtime.

7 System Realization

  • The reproducible pipeline uses deterministic decoding through a provider-agnostic API, DeepSeek-V4-Flash for LLM calls, and a swappable multilingual sentence encoder.The submitted encoder is paraphrase-multilingual-MiniLM-L12-v2; DeepSeek-V4-Flash was chosen for its low per-call cost across many short prompts.Source: §7 · p. 4
  • For 101,699 documents, a complete build took about 18 hours on 4 × H100 GPUs, with approximately 1.1 × 10^5 LLM calls and 4.5 × 10^8 tokens; extraction dominates and runtime is approximately linear in document count.Submitted dashboards use five voting rounds, a 0.7 minimum co-association threshold for the first hierarchy layer, three levels, and K≤20 defaults.Source: §7 · p. 4

8 Conclusion

EviMap presents an auditable three-level map for early exploration of unlabeled corpora, while acknowledging limitations in extraction, grouping, and evaluation.

8 Conclusion

  • EviMap builds a three-level map bottom-up from evidence phrases, linking documents to topics only through contained evidence and tracing every node to source spans.Source: §8 · p. 4
  • Across six corpora, the dashboard supports map browsing, evidence inspection, topic intersection, and long-tail auditing, presenting generated labels as auditable claims rather than a gold taxonomy.Source: §8 · p. 4
  • Model-generated extraction context may omit important aspects, small-topic folding may obscure minority concerns, and LLM grouping and labeling remain fallible.Gold hierarchies for automatic evaluation are also missing, though sectioned corpora may provide partial proxies.Source: §8 · p. 4
  • Future work will add user-specified aspects and reintegrate Other material, measure extraction recall, parameter sensitivity, and run-to-run variation, and broaden validation with user studies and multilingual evaluation.Source: §8 · p. 4

GenAI Usage Disclosure

EviMap uses LLMs for core system tasks, while a generative AI assistant was used only for manuscript grammar and style improvements; the authors take responsibility for the research and final text.

GenAI Usage Disclosure

  • EviMap uses LLMs for phrase extraction, semantic cleanup, naming, and multi-round voting; a generative AI assistant helped with grammar and style, while the authors conceived, verified, and approved the research and final text.The authors take full responsibility for the content, including reference accuracy.Source: GenAI Usage Disclosure · p. 5
Loading 2609.06664v1…