PaperlayerView exact revision on arXiv

Source-linked AI summary

LentEx: Generalizable Latent Entity Extraction via Synthetic Data and Instruction-Tuned LLMs

Umesh Bodhwani, Yuan Ling, Cibi Chakravarthy Senthilkumar, Shujing Dong, Yarong Feng, Hongfei Li, Ayush Goyal

arXiv:2609.04511v1cs.CL

TL;DR

LEE lacks flexible methods for inferring implicit entities, while LLM applications to this task remain largely unexplored. LentEx addresses this gap with template-based synthetic data and instruction fine-tuning of smaller LLMs, achieving strong generalization and performance across clustering and retrieval tasks. The paper also identifies reduced Recall in retrieval filtering as a limitation requiring further refinement.

  • Problem

    LEE requires inferring contextually implicit entities, but prior classification and topic-modeling approaches have limited flexibility, interpretability, and granularity, while LLM use for LEE remains largely unexplored.

  • Method

    LentEx combines template-based synthetic data generation with instruction fine-tuning to train smaller, efficient LLMs for latent entity extraction.

  • Results

    LentEx consistently outperforms embedding- and prompting-based baselines across most datasets, achieving an average v-measure of 59.54 versus 58.46 for NV-Embed-v2.

  • Takeaways & Limitations

    LentEx generalizes across domains and tasks, including 11 MTEB clustering benchmarks and specialized legal and biomedical retrieval datasets.

  • Takeaways & Limitations

    Retrieval-augmented filtering shows an observed drop in Recall, motivating adaptive filtering and context-aware retrieval strategies.

Abstract

from arXiv · show

Latent entity extraction (LEE) tackles the challenge of identifying implicit, contextually inferred entities within free text-an area where traditional entity extraction methods fall short. In this paper, we introduce LentEx, a novel framework for latent entity extraction that leverages synthetic data generation and instruction fine-tuning to optimize smaller, efficient large language models (LLMs). Latent entities, which are often abstract and thematic, are crucial for applications such as retrieval-augmented generation (RAG), customer persona analysis, and knowledge graph enrichment. LentEx addresses the scarcity of labeled datasets by employing a template-based approach to generate diverse, contextually rich synthetic data, ensuring high variability and alignment with real-world distributions. To our knowledge, LentEx is the first to systematically approach LEE through the lens of LLMs. LentEx demonstrates significant performance improvements across multiple tasks, notably surpassing state-of-the-art models on the MTEB Clustering Benchmark. Furthermore, our methodology enables robust generalization to unseen domains, making LentEx highly applicable in real-world NLP tasks, including RAG and clustering, thereby establishing a new paradigm for latent entity understanding and extraction in natural language processing.

I. INTRODUCTION

LEE identifies implicit, abstract entities from context rather than explicit mentions, addressing limitations of traditional extraction and earlier LEE approaches. LentEx introduces synthetic-data generation and instruction fine-tuning to support domain-agnostic extraction with smaller LLMs.

  • LEE infers abstract or thematic entities from contextual cues instead of extracting only explicitly mentioned names, locations, or organizations.
  • Classification-based LEE depends on predefined labels and may require extensive retraining as new categories emerge.
  • Topic modeling often produces keyword clusters that lack interpretability, actionable specificity, and context-sensitive granularity.
  • LLM applications to LEE remain largely unexplored despite broad success in natural-language processing.
  • LentEx combines diverse synthetic training data with self-instruction fine-tuning to extract latent entities across domains using smaller, efficient LLMs.
  • The framework formalizes domain-agnostic LEE, generates synthetic data without manually curated seed data, and reports stronger downstream performance than embedding- and generation-based methods.

II. RELATED WORK

Earlier approaches use classification or topic modeling but struggle with evolving entity categories, interpretability, and fine-grained entity extraction. LentEx instead targets open-ended prediction through flexible synthetic-data-based learning.

  • Topic models identify thematic patterns through embedding spaces and keyword post-processing, but their clusters often lack actionable entity-level specificity.
  • Classification methods rely on fixed labels, limiting flexibility as new entity categories or out-of-distribution data appear.
  • LentEx enables open-ended latent-entity extraction without predefined labels.
  • The flexible framework supports dynamic entity prediction for out-of-distribution data and shifting real-world datasets without retraining.
  • Synthetic-data generation offers large, diverse training datasets at lower cost than manual annotation, while Self-Instruct produces labeled examples from limited seed data.

III. LENTEX

LentEx formulates LEE as identifying entities implied by text but absent from its explicit content. Its formalization represents extraction as a function of input text, entity type, text type, and optional labels.

  • A. Problem Definition: LEE targets entities implied by context but not explicitly mentioned in the input text.
  • A. Problem Definition: Synthetic data covers 100 domains, which are clustered into broader domains according to similarity.
  • A. Problem Definition: The extraction function maps input text, latent-entity type, text type, and optional predefined labels to an identified latent entity.
  • A. Problem Definition: The formulation supports both unsupervised uses such as topic modeling and clustering and supervised tasks such as classification.

B. Method Overview

LentEx addresses labeled-data scarcity through template-based domain generation, synthetic input-output creation, filtering, and instruction fine-tuning with a smaller LLM. The process is designed to produce broad domain coverage and varied synthetic distributions.

  • Method Overview: LentEx combines synthetic data generation and instruction fine-tuning to extract latent entities across multiple domains with smaller LLMs.
  • Method Overview: Template prompts and few-shot Claude 3 Sonnet generation create 100 unique text-type and latent-entity-type domain combinations without manually curated seed data.
  • Method Overview: The synthetic-data distributions track input-token length, domain counts with corresponding unique entities, and dataset size across Rouge-L values.
  • Method Overview: Previously created domains are randomly sampled for new combinations, and duplicate combinations are filtered out.

2) Synthetic Input-Output Instance Generation:

LentEx generates diverse synthetic text–latent-entity pairs through iterative domain combinations and filters them to encourage contextual inference. Instruction templates are varied to reduce overfitting.

  • 10,000 synthetic samples are generated across domains through 10 iterations per domain, pairing generated texts with context-inferred latent entity labels.Diversity is assessed with Rouge-L similarity, with over 80% of instances scoring below 0.5.
  • Instances combine instruction templates, domain descriptions, latent entity types, and input texts, with the inferred entity as the target label.Examples with Rouge-L similarity above 0.5 are filtered out so entities are not explicitly mentioned in the input.
  • Template prompts define multiple workflow stages, including synthetic generation and paraphrased instructions for instruction fine-tuning.
  • Paraphrased templates introduce variation in sequence structure, spacing, and line breaks to mitigate overfitting.

2) Model Selection and Training:

LentEx is evaluated as a latent entity extraction framework using synthetic reference labels and semantic similarity, after fine-tuning an efficient instruction model.

  • Mistral-7B-Instruct-v0.2 is fine-tuned with LoRA for efficient training on four NVIDIA A10G GPUs.Training uses one epoch, batch size 8, learning rate 0.0001, rank 16, and LoRA alpha 32.
  • The experiments assess LentEx for latent entity extraction and downstream clustering and retrieval-augmented generation applications.
  • The evaluation compares predicted latent entities with synthetic reference labels using mean semantic similarity from all-mpnet-base-v2.The evaluation samples 50 synthetic instances per domain, totaling 5,000 instances.
  • LentEx achieves higher semantic similarity with reference labels than Mistral-7B-Instruct-v0.2 and Claude-3-Haiku.

B. Clustering with Latent Entity Representations

LentEx represents latent entities for clustering across diverse MTEB domains and text granularities, using v-measure to evaluate cluster quality. It outperforms the listed embedding-based and prompting-based baselines across most datasets.

  • LentEx consistently outperforms embedding-based and prompting-based baselines across most of the 11 MTEB clustering datasets.The datasets cover paragraph-to-paragraph and sentence-to-sentence tasks across scientific, social-media, and other domains.
  • Clustering embeds inferred latent entities and applies mini-batch k-means with k equal to the number of labels, evaluated using v-measure.V-measure is the harmonic mean of homogeneity and completeness when β = 1.
  • V-measure balances homogeneity and completeness through β.
  • 59.54 is LentEx’s highest average v-measure, exceeding NV-Embed-v2 (58.46) and bge-en-icl (57.89).
  • Paragraph-level gains include arxiv-p2p at 61.42 versus 56.46 and reddit-p2p at 79.59 versus 74.94.

C. Improving Retrieval in RAG with Latent Entity Filtering

LentEx is integrated into RAG by extracting latent entities as metadata and filtering documents before retrieval. It is evaluated on legal and biomedical datasets alongside automated response ratings and dataset-size analysis.

  • The RAG evaluation covers COLIEE legal queries and BioASQ biomedical questions with domain-specific latent entity types.COLIEE contains 100 queries and 375 supporting documents; BioASQ contains 1,130 questions and 4,719 supporting documents.
  • RAG+LentEx is compared with standard RAG, prompting-based extractors, NER-based filtering, and other RAG variants.
  • LentEx extracts paragraph-level entities, stores them as metadata, and filters documents by query–metadata alignment before retrieval.The method uses dynamic queries because filtering strength and heterogeneous entity types affect relevance.
  • Figure 5 measures mean semantic similarity on a test dataset to show the effect of dataset size on model performance.

D. LLM as an Evaluator

The evaluator categorizes model outputs as precise, relevant-but-generic, or irrelevant/invalid, using shuffled output order to reduce evaluation bias. LentEx achieves 98.4% of outputs in the first two categories.

  • The evaluator assigns outputs to three categories: precise and correct, relevant but generic, or irrelevant and invalid.Outputs are assessed with Claude-3.5-Sonnet, and their order is shuffled for each evaluation.
  • 98.4% of LentEx outputs are classified as either precise or relevant, but generic.This result is reported as evidence of the robustness of LentEx’s fine-tuning approach.

A. Impact of Dataset Size on Model Fine-Tuning

Fine-tuning performance improves with larger datasets up to 6000 samples, after which it plateaus. The synthetic data pipeline also produces mostly aligned and correct examples, with paired generation outperforming independent label generation.

  • A. Impact of Dataset Size on Model Fine-Tuning: 6000 samples mark the point up to which fine-tuned model performance consistently improves before plateauing.The result is based on evaluation using progressively larger training-data subsets and a held-out test set.
  • B. Effect of Synthetic Data Generation on Data Quality: The evaluation covers a stratified sample of 2000 synthetic examples.
  • B. Effect of Synthetic Data Generation on Data Quality: 91% correctness occurs when labels are generated independently after input creation, compared with paired input-output generation.The comparison supports generating inputs and labels within the same model call.
  • LentEx generalizes from synthetic training data to diverse real-world datasets spanning clustering, retrieval, scientific, social, news, legal, and biomedical domains.The paper reports robust generalization across 11 MTEB clustering benchmarks and specialized retrieval datasets.

VII. ERROR ANALYSIS

LentEx errors arise when latent entities are ambiguous or incompletely inferred, harming clustering and retrieval. The paper identifies adaptive entity selection and domain-specific fine-tuning as future remedies while positioning downstream applications as validated use cases.

  • VII. ERROR ANALYSIS: Ambiguous entity assignments can cause LentEx to select a suboptimal latent entity and misclassify clusters.This issue is more pronounced in Twenty Newsgroups and StackExchange, where latent entity types are generic.
  • VII. ERROR ANALYSIS: Over-restrictive retrieval filtering can exclude relevant documents when query entities are incompletely extracted.The paper highlights COLIEE, where cases may involve intersecting legal categories such as contract and tort law.
  • VII. ERROR ANALYSIS: Domain-specific fine-tuning and adaptive entity selection are proposed to address clustering and retrieval errors.
  • VIII. CONCLUSION AND FUTURE WORK: LentEx is evaluated for downstream clustering and retrieval-augmented generation, where the paper reports enhanced retrieved-result quality and relevance.
  • VIII. CONCLUSION AND FUTURE WORK: Future work includes refining RAG filtering, exploring self-supervised, contrastive, or reinforcement learning, and applying LentEx to temporal reasoning and multi-hop inference.The paper also identifies knowledge-graph, search, recommendation, and bias-related directions.
Loading 2609.04511v1…