PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to structure data from user-generated thematic corpora

Elishay Avram, Oren Glickman, Elad Yom-Tov

arXiv:2610.01463v1cs.IR

TL;DR

The paper addresses the challenge of extracting structured information when relevant attributes are implicit, domain-dependent, and unknown in advance. It proposes an iterative LLM framework that discovers and consolidates attributes without a predefined ontology, then assigns types and extracts values. On health-related Reddit communities, discovered attributes achieved 61% agreement with human-identified attributes, structural typing achieved 82% accuracy, and value extraction reached an F1 score of 80%.

  • Problem

    Relevant attributes in thematic corpora are often implicit, domain-dependent, and unknown in advance, while existing approaches may rely on predefined schemas.

    Source: §1 · p. 1, §2.1.2 · p. 2

  • Method

    An iterative LLM framework induces and consolidates attributes without a predefined ontology, assigns structural types, and supports value extraction.

    Source: §1 · p. 1, §3.1.3 · p. 4

  • Results

    61% agreement with human-identified attributes, 82% structural-type accuracy, and an F1 score of 80% for value extraction were reported.

    Source: §6 · p. 7

  • Takeaways & Limitations

    The framework supports automated schema discovery and value extraction without human intervention or predefined schemas.

    Source: §6 · pp. 7–8

    Evaluation on a specific collection of thematic corpora leaves generalizability across additional domains to be investigated.

    Source: §6 · p. 8

Abstract

from arXiv · show

Thematic corpora, such as social media communities, contain unstructured text describing data that could be made structured. These include, for example, personal attributes, behaviors, and experiences mentioned in social media data. Extracting structured data is challenging as relevant attributes are often implicit, domain-dependent, and unknown in advance. We propose a fully automated, iterative framework for discovering and extracting domain-specific attribute schemas without a predefined ontology. Using large language models (LLMs), the framework induces candidate attributes, sequentially consolidates semantically overlapping attributes, and assigns a structural type. These enable creating an ontology and populating it with values from the corpus. The framework also enables the use of smaller LLMs for value extraction with estimable accuracy loss compared to large LLMs. We evaluate the framework on 5 health-related Reddit communities. Discovered attributes achieved 61% agreement with human-identified attributes, close to the 62% agreement between independent annotators. In most cases, the algorithm converges to a stable attribute set in fewer than 10 iterations. Structural type assignment achieves 82% accuracy, and value extraction reaches an F1 score of 0.8 compared to human annotations. Across four LLM families, smaller instruction-tuned models show statistically significant improvements in extraction performance with model scale when evaluated against a high-capacity reference LLM, supporting informed accuracy-cost trade-offs. These results show that attributes comparable to those identified by humans can be discovered automatically, enabling the creation of high-quality structured datasets economically and at scale. By removing the need for predefined ontologies, iterative model-driven schema induction offers a practical and scalable foundation for mining thematic corpora.

1 Introduction

The introduction frames schema-free extraction as a response to attributes that are implicit, domain-dependent, and unknown in advance. It proposes an iterative LLM framework and evaluates its discovery, typing, and extraction on five health-related Reddit communities.

1 Introduction

  • Attributes in thematic corpora are often unknown beforehand, domain-dependent, and implicit, while predefined schemas demand expert effort and may not generalize.These constraints leave existing approaches limited to narrow attribute sets or substantial human adaptation.Source: §1 · p. 1
  • LLM extraction also risks redundant or unstable attributes, and repeated use of large models can be computationally costly.The paper motivates schema discovery that is robust, cost-aware, and adaptable across domains.Source: §1 · p. 1
  • The proposed framework iteratively induces and consolidates domain-specific attributes without a predefined ontology, then assigns structural types for value extraction.The resulting types support consistent, automated extraction across document collections.Source: §1 · p. 1
  • Evaluation covers five health-related Reddit communities and examines schema stability, consolidation, typing accuracy, and model scale.The authors report that stable schema discovery can be achieved with small LLMs for scalable, cost-efficient mining.Source: §1 · p. 1

2 Related Work

Related work distinguishes discovering a corpus-specific attribute schema from extracting values once a schema exists. Prior approaches address clustering, entity extraction, and structured generation, but the paper identifies schema induction for thematic corpora as an unresolved need.

2.1 Information Extraction from Unstructured Text

  • Information extraction converts unstructured language into structured representations; attribute discovery identifies meaningful corpus attributes when prior specification is limited.A predefined schema can bypass discovery, but this work assumes none exists.Source: §2.1.1 · p. 2, §2.1 · p. 2

2.1.1 Automatic Attributes Discovery.

  • Prior discovery methods use representation learning, clustering, and LLM feedback to find latent groupings, but do not define and consolidate thematic-corpus attributes without predefined structure.The cited examples include methods by Zou et al. [25] and Fan et al..Source: §2.1.1 · p. 2

2.1.2 Information Extraction.

  • NER and relation extraction identify entities and links, while text-to-table and related LLM systems generate structured representations, generally by populating predefined structures.The paper argues that corpus-driven, semantically nuanced attributes may require adaptation beyond these extraction targets; jointly inducing schemas and extracting values can operationalize datasets without predefined ontologies.Source: §2.1.2 · p. 2

2.2 Social Media as a Medical Data Source

  • Health information on Reddit has been extracted ad hoc, including through regular expressions or LLMs applied to predefined schemas; the paper frames unknown attributes as the central challenge.Examples include diet information, epilepsy, and structured extraction from r/AskDocs.Source: §2.2 · pp. 2–3

3 Model

The model induces corpus-specific attributes from documents, iteratively ranks and consolidates them, then assigns types and supports cost-aware value extraction. Its output is a document-by-attribute table of extracted values.

3 Model

  • The method transforms a thematic corpus into a table whose rows are documents and columns are attributes induced from the corpus.Cells contain document-level values, potentially aggregated from multiple mentions, without requiring span-level annotations.Source: §3 · p. 3

3.1 Iterative Attribute Discovery

  • Attribute discovery repeatedly refines candidate attributes from a random corpus sample until the set converges or reaches a preset iteration limit.Each iteration’s candidate set initializes the next iteration.Source: §3.1 · p. 3

3.1.1 Extraction of Information and Attribute Definitions from doc-

  • For each sampled document, an LLM extracts information and pairs each textual value with an unconstrained, free-form attribute name.Names and values can vary across documents in language, granularity, or perspective; supporting evidence is retained for leading attributes after the zero-shot first iteration.Source: §3.1.1 · p. 3

3.1.2 Attribute Aggregation and Ranking.

  • Attribute frequencies are accumulated across documents and used to rank prevalent attributes for subsequent consolidation.Figure 2 illustrates document-level counts, corpus aggregation, and consolidation across the example corpus.Source: §3.1.2 · pp. 3–4, Figure 2 · p. 4, §3.1.3 · p. 4

3.1.3 Semantic Attribute Consolidation.

  • Four LLM-based consolidation strategies compare or group attributes: pairwise comparison, global grouping, incremental consolidation, and bottom-up merging.Their stated LLM-call complexities are O(n^2), O(1), O(n), and O(n), respectively.Source: §3.1.3 · p. 4
  • The authors test the four strategies on one corpus, then apply the most accurate strategy to other corpora to reduce corpus-specific overfitting.The top-k attribute set is fed back with evidence and iteration continues until it is unchanged between consecutive iterations or the maximum is reached.Source: §3.1.3 · pp. 4–5

3.2 Attribute Type Assignment

  • Each discovered attribute receives an LLM-assigned type: exact value, list, or free text.The types distinguish atomic values, multiple discrete values, and open-ended text, supporting consistent extraction across documents.Source: §3.2 · p. 5

3.3 Cost-Aware Model Selection for Large-Scale Extraction

  • Attribute-level comparisons with a high-capacity reference LLM estimate how closely smaller models reproduce its extraction quality.Users can set accuracy thresholds to route attributes to the smallest qualifying model, reserving larger models for attributes needing higher precision.Source: §3.3 · p. 5

4 Experimental Settings

The experiments use five health-related Reddit communities as separate thematic corpora. They evaluate extraction with type-sensitive metrics, compare discovered attributes with human proposals, and create human annotations for type and value evaluation.

4 Experimental Settings

  • Each Reddit community is treated as an independent thematic corpus, with posts and comments as documents; experiments cover five health-related subreddits.The communities span fertility, diabetes, hypertension, steroid use, and general medical advice.Source: §4 · p. 5

4.1 Evaluation Protocol

  • Evaluation is attribute-level and matched to each assigned type: exact match for exact values, set-based F1 for lists, and LLM-rated semantic similarity for free text.Free-text similarity is scored from 0 to 10 and normalized to [0, 1].Source: §4.1 · pp. 5–6

4.2 Large Language Models

  • The primary model is ChatGPT-4o, and smaller-model tests use instruction-tuned Llama, Qwen, and Mistral models with identical prompts and no task-specific fine-tuning.ChatGPT-4o uses temperature 0 and a maximum generation length of 2048 tokens; its exact parameter count is undisclosed.Source: §4.2 · p. 6

4.3 Comparison to Human Performance

  • Two people independently proposed the ten most common categories after reading 20 documents from each subreddit, for comparison with algorithmic discovery.Source: §4.3.1 · p. 6

4.3.1 Attribute Discovery.

  • Two human annotators labeled the structural type of each of the ten discovered attributes to establish a reference for type-assignment evaluation.Source: §4.3.2 · p. 6

4.3.2 Attribute Type.

  • For value-extraction evaluation, annotators labeled 50 randomly sampled documents per attribute using the discovered attributes and types.Each document received reference values for all attributes associated with its subreddit.Source: §4.3.3 · p. 6

5 Results

Human annotators showed substantial agreement, and the discovered attribute sets overlapped with their suggestions. Across communities, the attributes reflected each subreddit’s thematic focus.

5.1 Inter-Annotator Agreement

  • Human annotators proposed categories averaging 62% similarity to one another.Source: §5.1.1 · p. 6

5.1.1 Attribute Sets Agreement.

  • Across domains, inter-annotator agreement averaged 0.71, reaching 1.00 for exact value attributes, 0.78 for lists, and 0.45 for free text.Source: §5.1.2 · p. 6

5.2 Most Popular Attributes Across Thematic Corpora

  • Discovered attributes reflected subreddit focus, from clinical measurements and medications to fitness goals, reproductive attributes, and diverse symptoms and demographics.The final attribute sets achieved 61% agreement with human-identified attributes.Source: §5.2 · p. 6

5.3 Semantic Consolidation Strategy

The authors selected Incremental consolidation using a held-out Reddit corpus, then applied it across the other communities.

5.3 Semantic Consolidation Strategy

  • Strategy selection used r/LoseIt, a held-out corpus excluded from downstream evaluation to avoid biasing the strategy toward the main experimental corpora.Source: §5.3 · p. 6
  • Incremental consolidation scored F1 0.32, versus Bottom-up merging at 0.13, Global grouping at 0.08, and Pairwise comparison at 0.08.Incremental achieved precision 0.25 and recall 0.45.Source: §5.3 · p. 6
  • The authors selected Incremental consolidation and applied it to the other subreddits based on its consistent relative advantage.Source: §5.3 · p. 6

5.4 Convergence Behavior Across Domains

The attribute-discovery process stabilized at different rates by domain; three evaluated communities stabilized before the ten-iteration limit, while two reached it.

5.4 Convergence Behavior Across Domains

  • Convergence is measured as the point when the k most popular attributes remain unchanged, or when the maximum of 10 iterations is reached.Source: §5.4 · pp. 6–7
  • 4 iterations sufficed for Steroids, 5 for Hypertension, and 8 for AskDocs; TTC and Diabetes reached the 10-iteration maximum without earlier stabilization.Source: §5.4 · p. 7
  • The held-out r/LoseIt corpus converged after 3 iterations, adding to evidence that stabilization often occurs within a small number of iterations.The text notes domain-specific variation.Source: §5.4 · p. 7

5.5 Attribute Type Assignment Accuracy

Automatic attribute type assignment achieved 82% accuracy against manual annotations across the evaluated attributes.

5.5 Attribute Type Assignment Accuracy

  • 82% accuracy was achieved across the full set of 50 evaluated attributes.Source: §5.5 · p. 7
  • Model predictions were compared with manual annotations across all discovered attributes.Source: §5.5 · p. 7
  • The authors state that inferred attribute structure supports downstream value extraction without manually defined rules or domain-specific heuristics.Source: §5.5 · p. 7

5.6 Model-Scale Results

Extraction accuracy rises with model size, while attribute type, model family, and subreddit are also associated with accuracy. The analysis reports variability across attributes and models, and its regression findings are similar under three plausible ChatGPT-4o size assumptions.

5.6 Model-Scale Results

  • Figure 3 displays attribute-level F1 variability across the ten final TTC attributes and evaluated LLMs of increasing size.Source: §5.6 · p. 7
  • Figure 4 shows mean accuracy increasing with model size, alongside variability among model families.Source: §5.6 · p. 7
  • R²=0.67; model scale is positively correlated with accuracy (β=0.107, p<0.001), with similar regression results under 200B, 1T, and 1.8T ChatGPT-4o assumptions.The paper reports the 1T estimate as the representative approximation.Source: §5.6 · p. 7
  • List-based (β=0.057, p=0.003) and number-based attributes (β=0.306, p<0.001) are associated with higher accuracy than free-text attributes.Source: §5.6 · p. 7
  • Mistral (β=0.171, p=0.002) and Qwen (β=0.136, p=0.015) are associated with higher accuracy than GPT, while Llama is not significantly different from GPT.Source: §5.6 · p. 7
  • TTC accuracy is significantly higher (β=0.266, p<0.001), while the remaining subreddit coefficients are not significantly different from the baseline.Source: §5.6 · p. 7

6 Discussion

The discussion presents ontology-free schema induction as a way to address attributes that are implicit, domain-dependent, and unknown in advance. Results indicate agreement with human-identified attributes, accurate typing and value extraction, while the authors note that broader-domain generalizability remains to be studied.

6 Discussion

  • The framework discovers and consolidates schemas without a predefined ontology, then assigns structural types for automated value extraction.Source: §6 · pp. 7–8
  • Discovered attributes achieved 61% agreement with human-identified attributes, versus 62% agreement between independent annotators; structural typing reached 82% accuracy and value extraction reached 80% F1.Source: §6 · p. 7
  • The Figure 3 heatmap covers attribute-level F1 for ten popular discovered attributes across LLMs ordered by parameter count.Source: Figure 3 · p. 8
  • Figure 4 presents mean accuracy across category types by model size, with individual models as markers and a dotted log-linear regression.Source: Figure 4 · p. 8
  • Comparisons with GPT-4o let practitioners set an accuracy threshold in advance when considering smaller models for extraction.Source: §6 · p. 8
  • Extraction accuracy grows with model size; Mistral and Qwen outperform a theoretically similarly sized GPT model, while Llama shows no significant difference.Source: §6 · p. 8
  • The evaluation used a specific collection of thematic corpora, so additional domains are needed to characterize generalizability.Source: §6 · p. 8
  • The authors plan to apply the framework to the entire Reddit corpus to structure it for research.Source: §6 · p. 8

7 Ethical Considerations

The ethical discussion warns that structuring sensitive user-generated information can increase privacy risks and that discovered attributes are bounded by potential bias and corpus-level scope.

7 Ethical Considerations

  • Combining extracted attributes could facilitate profiling or expose sensitive characteristics users did not intend to reveal.Source: §7 · p. 8
  • Use safeguards for sensitive data, including data minimization and access restrictions, and do not use structured outputs to identify, profile, or decide about individuals.Source: §7 · p. 8
  • Discovered attributes may reflect community and language-model biases and should not be treated as representative of broader populations.Source: §7 · p. 9
  • Evaluation is aggregate, the running example uses synthetic posts, and the framework is framed for corpus-level research rather than individual inference or decisions.Source: §7 · p. 9
Loading 2610.01463v1…