PaperlayerView exact revision on arXiv

Source-linked AI summary

A Tale of LLMs and Induced Small Proxies: Scalable Small Language Models for Knowledge Mining

Sipeng Zhang, Longfei Yun, Zilong Wang, Letian Peng, Jingbo Shang

arXiv:2510.01427v4cs.AI

TL;DR

Knowledge mining extracts user-directed structure from massive corpora, but LLMs are costly at scale and small models have limited reasoning and adaptability. Falconer assigns planning and supervision to LLMs and execution to lightweight proxies; it closely matches state-of-the-art LLM accuracy while reducing inference cost and accelerating processing.

  • Problem

    LLMs can interpret instructions but are prohibitively expensive at scale, while small models are efficient but have limited reasoning and adaptability for knowledge mining.

    Source: §1 · pp. 1–2

  • Method

    Falconer uses LLMs to plan pipelines and generate supervision, then trains a unified lightweight proxy to execute classification and extraction operations.

    Source: §1 · p. 2

  • Results

    Up to 90% lower inference cost and over 20x faster large-scale processing accompany accuracy close to state-of-the-art LLMs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Falconer’s reported accuracy, cost, and speed make large-scale knowledge mining practical at scale.

    Source: §1 · p. 2

Abstract

from arXiv · show

At the core of Deep Research is knowledge mining, the task of extracting structured information from massive unstructured text in response to user instructions. Large language models (LLMs) excel at interpreting such instructions but are prohibitively expensive to deploy at scale, while traditional pipelines of classifiers and extractors remain efficient yet brittle and unable to generalize to new tasks. We introduce Falconer, a collaborative framework that combines the agentic reasoning of LLMs with lightweight proxy models for scalable knowledge mining. In Falconer, LLMs act as planners, decomposing user instructions into executable pipelines, and as annotators, generating supervision to train small proxies. The framework unifies classification and extraction into two atomic operations, get label and get span, enabling a single instruction-following model to replace multiple task-specific components. To evaluate the consistency between proxy models incubated by Falconer and annotations provided by humans and large models, we construct new benchmarks covering both planning and end-to-end execution. Experiments show that Falconer closely matches state-of-the-art LLMs in instruction-following accuracy while reducing inference cost by up to 90% and accelerating large-scale knowledge mining by more than 20x, offering an efficient and scalable foundation for Deep Research.

1 Introduction

Falconer combines LLM planning and supervision with lightweight proxies that execute knowledge-mining pipelines. It is reported to closely match state-of-the-art LLM accuracy while reducing cost and accelerating large-scale processing.

1 Introduction

  • LLMs offer strong instruction-following but are costly to scale, while small models are efficient yet weak in reasoning and adaptability for real-world workloads.Knowledge mining may span millions of records, making high throughput and low cost first-order requirements.Source: §1 · pp. 1–2
  • Falconer uses LLMs as planners to translate task descriptions into workflow code and as annotators to generate supervision for lightweight proxies.After training, proxies replace LLMs as primary executors for large corpora.Source: §1 · p. 2
  • Falconer composes get_label for classification and get_span for extraction into workflows, including complex tasks such as relation extraction, entity linking, and multi-hop queries.SLM-based proxies execute the atomic operations, while LLM planners compose them into pipelines.Source: §1 · p. 2
  • The unified instruction-following proxy supports both classification and extraction, replacing separate task-specific components; Cuckoo uses next-token extraction pretraining for this role.The paper reports that Cuckoo outperforms LLM-based extractors more than 20x larger.Source: §1 · p. 2
  • Falconer closely matches state-of-the-art LLM accuracy, reducing inference cost by up to 90% and accelerating large-scale processing by over 20x.New benchmarks assess planning quality and end-to-end execution.Source: §1 · pp. 2–3

2 Related Works

Related work covers information extraction tasks, LLM agents’ reasoning and tool use, and agentic retrieval approaches.

Information Extraction

  • Information extraction ranges from entity, relation, and event extraction to more difficult abstract entity extraction.The passage describes IE as extracting spans in response to a user requirement, question, or instruction.Source: §2 · p. 3

LLM Agents

  • LLM-agent research uses language models for downstream tasks, including environment interaction, external tools, and accumulated experiential knowledge.ReAct is described as alternating reasoning with external operations such as information retrieval.Source: §2 · p. 3

LLM Agents for Retrieval

  • LLM retrieval agents use pretraining, reranking, or prompting, while existing approaches largely rely on heuristic prompts or few-shot examples for retrieval strategy guidance.Domain-specific examples include EHRAgent’s structured tool-use planning and interactive coding mechanism.Source: §2 · p. 3

3 Method

Falconer separates instruction planning, supervision generation, and proxy execution to make knowledge-mining workflows adaptable and efficient. Its method combines explicit task pipelines and a unified classification/extraction model with corpus-grounded training and a compact executor.

3.1 Preliminary

  • Large language models show strong instruction-following and downstream-task generalization, but scale poorly on knowledge-mining workloads involving large data volumes; traditional systems improve efficiency yet require rigid, manually built task-specific pipelines.Source: §3.1.1 · p. 3

3.1.1 Background.

  • For span extraction, BIO-style token classifiers predict a tag for each input token, whereas autoregressive models generate output tokens drawn from the input sequence.The generated-token count is K, and the generated tokens correspond to extracted spans.Source: §3.1.2 · p. 3

3.1.2 Notations.

  • In the notation for autoregressive extraction, K counts generated tokens, which are selected from the input sequence as extracted spans.Source: §3.1.2 · p. 3

3.2 Falconer

  • Falconer takes a user prompt and corpus, then uses a planner, supervision generator, and compact proxy metamodel to produce structured outputs through an automated pipeline.The paper describes this pipeline as reducing cost and increasing speed relative to direct LLM execution while maintaining strong performance.Source: §3.2 · p. 3

3.3 Unified TC/IE Proxy

  • Cuckoo’s instruction-following model supports both information extraction and classification, with classification framed as a natural-language-inference prompt over candidate labels.The model is fine-tuned on classification datasets to produce the metamodel used in the experiments.Source: §3.3 · p. 3

3.4 Planning

  • GPT-4.1 plans well across diverse tasks but struggles with multi-step tasks; in-context learning improves accuracy, while Falconer codifies subtasks and dependencies into transparent, deterministic workflows.The planner decomposes tasks into get_label and get_span operations; complex tasks require intermediate execution results rather than one fixed string.Source: Table 1 · p. 3, §3.4 · p. 4

3.5 Generator

  • Using about 5% of the corpus, the generator obtains LLM annotations on sampled examples and fine-tunes the metamodel, achieving performance comparable to or better than state-of-the-art LLMs.The approach labels naturally occurring examples rather than fabricating inputs; high-quality supervision is essential and may also come from curated human annotations.Source: §3.5 · pp. 4–5

3.6 Metamodel: Lightweight Yet Capable Proxy

  • Cuckoo uses up to 20× fewer FLOPs and 1000× less memory than GPT-class models while maintaining competitive accuracy on instruction-following and span-extraction benchmarks.The paper presents it as an efficient proxy for large-scale knowledge mining.Source: §3.6 · p. 5

4 Experiments

The experiments assess metamodel alignment with human annotations on labeled datasets and with GPT-4o on unlabeled corpora. They examine adaptation across task types, including basic, query-based, and multi-entity tasks.

4.1 Labeled Dataset

  • With 2,048 training samples, the metamodel outperforms GPT-4o across all task categories, and performance improves consistently as sample size increases.The rate of improvement depends on annotation quality: high-quality annotations lead to faster saturation, while suboptimal annotations yield more gradual gains.Source: §4.1 · pp. 5–6

4.2 Unlabeled Dataset Evaluation

  • Across three unlabeled corpora, the evaluation tests proxy consistency with GPT-4o on basic, query-based, and multi-label or multi-entity tasks.The corpora are TED Talk Summary, Steam Game Description, and Text Message.Source: §4.2 · p. 6
  • On basic tasks, fine-tuned metamodels achieve consistently high agreement with the large model.The benchmark includes classification, entity and relation extraction, and composite tasks; pairwise relation extraction specifies one participating entity.Source: §4.2 · p. 6
  • For query-based tasks, fine-tuning raises F1 from 0.23 to 0.56 on a medicine-focused retrieval and extraction prompt.The untuned metamodel is weakest in this category; fine-tuning uses only a small fraction of the annotated corpus.Source: §4.2 · p. 6
  • On multi-entity tasks, the adapted metamodel is competitive with, and sometimes surpasses, multi-turn prompting augmented with human annotations.The planner decomposes multi-label queries into sequential classification subtasks whose outputs are combined with Boolean logic.Source: §4.2 · p. 6

5 Analysis

The analysis compares executor models, examines sequential adaptation and model consolidation, and investigates a proposed arising ability. Results cover generalization, performance retention, and adaptation to new tasks.

5.1 Ablation on MetaModel

  • Autoregressive small-model executors show little or no improvement after continual pretraining on NER data, while RoBERTa/NTE-based small models generalize better on downstream tasks.The authors suggest autoregressive models are not effective at extracting knowledge from NER-style supervision and emphasize task-specialized designs.Source: §5.1 · p. 6

5.2 Continual Integration Analysis

  • After sequential fine-tuning across five tasks, metamodels retain capabilities comparable to models fine-tuned directly from the base model.The evaluation also reports that this continual integration alleviates deployment overhead when adapting to diverse tasks.Source: §5.2 · pp. 6–7, Figure 4 · p. 7
  • A unified metamodel performs nearly indistinguishably from separate classification and extraction metamodels across task types, while reducing deployment overhead.Figure 5 compares performance with varying numbers of metamodels across task types.Source: §5.2 · p. 7, Figure 5 · p. 7
  • With equal-size datasets and equal training epochs, pretraining yields significantly faster convergence when the metamodel is adapted to new tasks.The comparison evaluates performance without classification-dataset pretraining against the pretrained metamodel.Source: §5.2 · p. 7, Figure 5 · p. 7

5.3 Case Study: Arising Abilities of model

  • Manual analysis attributes 74% of the improvement on Biology tasks to arising abilities, defined as correcting outputs despite inaccurate annotation guidance.The paper reports GPT-4o’s average F1 on Biology tasks as 0.27, roughly matching the metamodel’s zero-shot performance.Source: §5.3 · p. 7
  • Fine-tuning on annotations with randomized span starts but preserved endings yields performance nearly identical to training on original annotations; RoBERTa-large drops substantially.The authors interpret this controlled result as evidence that arising ability is driven almost entirely by positional supervision.Source: §5.3 · pp. 7–8

6 Conclusion

The conclusion presents a framework that decomposes knowledge-mining tasks and uses a unified model. The classification appendix describes reformulating classification as natural-language inference using token-level supervision.

6 Conclusion

  • The framework decomposes knowledge-mining tasks into subtasks and uses a unified model, with users specifying a task prompt and output format.The conclusion reports up to 90% lower inference costs and 20x higher inference speed.Source: §6 · p. 8

A Cuckoo For Text classification

  • For classification, token-level supervision is used to reformulate the task as a natural-language inference problem.The model predicts whether an input sentence entails a candidate label.Source: A Cuckoo For Text classification · p. 9

B Generating Fine-tuning Samples

The classification sampling procedure uses metamodel scores to construct a fine-tuning set from a corpus. It selects high-scoring and low-scoring examples as positive and negative samples.

B Generating Fine-tuning Samples

  • For classification fine-tuning, the procedure ranks corpus examples by metamodel score for a label, then selects the top and bottom N as positive and negative samples.The training set is the union of those two groups.Source: B Generating Fine-tuning Samples · p. 9

C Continual Pretraining on autoregressive models

Table 4 reports results on NER datasets with ground-truth labels.

  • Table 4 covers NER datasets with ground-truth labels.Source: Table 4 · p. 9

D Sample Task

The sample tasks combine selecting texts by topic with extracting requested information, including speakers, locations, institutions, and countries.

  • One prompt selects finance-focused talks and asks for their lecturer.Source: D Sample Task · p. 9
  • Other prompts request locations generally or countries from talks about gender stereotypes.Source: D Sample Task · p. 9
  • Health-related examples ask for institutions in mental-health talks or lecturers in talks about health and the brain.Source: D Sample Task · p. 9

E Sample Planning Code

The sample planning code filters a corpus with a finance label, then extracts lecturer spans from the retained texts and returns them with their source text.

F Human Proposed Task on Unlabeled Datasets

The task examples for the TED description dataset span topic selection, information extraction, and combined or constrained instructions.

Loading 2510.01427v4…