PaperlayerView exact revision on arXiv

Source-linked AI summary

PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognition

Jinghui Lu, Ziwei Yang, Yanjie Wang, Xuejing Liu, Brian Mac Namee, Can Huang

arXiv:2402.04838v5cs.CLcs.AI

TL;DR

Autoregressive NER generation can produce long sequences of label-mention pairs, contributing to LLM inference latency. PaDeLLM-NER decodes pairs in parallel, and experiments report faster inference while maintaining prediction quality. The authors also identify limitations in mention counting, location preservation, and duplicate removal.

  • Problem

    Autoregressively generating all NER label-mention pairs can lengthen output sequences and increase inference latency.

    Source: §1 · p. 2

  • Method

    PaDeLLM-NER predicts mention counts per label, then decodes indexed label-mention pairs in parallel and aggregates them.

    Source: §3.2 · p. 5

  • Results

    1.76 to 10.22 times faster than conventional autoregressive methods, with average sequence length around 13% of theirs, while maintaining or improving prediction quality across settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework reduces inference time without sacrificing prediction accuracy, according to the reported experiments.

    Source: §8 · p. 10

    Reformulating label-mention pairs loses location information needed for downstream editing, and aggressive de-duplication may remove mentions valid under different labels.

    Source: §7 · p. 9

Abstract

from arXiv · show

In this study, we aim to reduce generation latency for Named Entity Recognition (NER) with Large Language Models (LLMs). The main cause of high latency in LLMs is the sequential decoding process, which autoregressively generates all labels and mentions for NER, significantly increase the sequence length. To this end, we introduce Parallel Decoding in LLM for NE} (PaDeLLM-NER), a approach that integrates seamlessly into existing generative model frameworks without necessitating additional modules or architectural modifications. PaDeLLM-NER allows for the simultaneous decoding of all mentions, thereby reducing generation latency. Experiments reveal that PaDeLLM-NER significantly increases inference speed that is 1.76 to 10.22 times faster than the autoregressive approach for both English and Chinese. Simultaneously it maintains the quality of predictions as evidenced by the performance that is on par with the state-of-the-art across various datasets.

1 Introduction

Generative LLM-based NER produces structured label-mention pairs, but autoregressive output formats can lengthen generation and increase latency. PaDeLLM-NER parallelizes pair generation to reduce latency while maintaining prediction quality.

1 Introduction

  • Autoregressive augmented-language outputs duplicate input text, while structured annotation still generates label-mention pairs sequentially, lengthening output when pairs are numerous.The paper identifies lengthy sequence generation as a main source of high LLM latency.Source: §1 · p. 2
  • PaDeLLM-NER predicts a label’s mention count and indexed mentions in separate sequences, then aggregates the results and resolves cross-label duplicates by prediction probability.The approach is designed to shorten generated sequences and enable parallel decoding of label-mention pairs.Source: §1 · p. 2
  • PaDeLLM-NER maintains or surpasses prediction quality across flat and nested English and Chinese NER in zero-shot and supervised settings.The reported performance is on par with state-of-the-art results on many datasets.Source: §1 · p. 2
  • 1.76 to 10.22 times faster than previous approaches, with average sequence length around 13% of conventional autoregressive methods.The reported speedup accompanies shorter sequences from decoupled label-mention generation.Source: §1 · p. 2
  • The approach is complementary to LLM.int8() and speculative sampling and can be integrated with those inference acceleration methods.The authors describe it as a pioneering approach to accelerating NER inference in LLMs through parallel decoding of label-mention pairs.Source: §1 · p. 2

2 Related Work

Related work covers generative models for NER and methods for accelerating LLM inference. This paper emphasizes supervised NER inference speedup without changing model architecture or adding extra modules.

2.1 Generative Models for NER

  • NER has shifted from token-level classification with tailored prediction heads [1] [2] toward seq2seq generation of text-based inputs and outputs [3] [4] [5] [6] [7].Recent LLM-based work also explores zero-shot and few-shot learning, including in-context learning, self-consistency, and learning programming [29] [30] [33] [34].Source: §2.1 · p. 3
  • Unlike prior work emphasizing training-free few-shot performance, this paper focuses on fully supervised NER and inference speed.Source: §2.1 · p. 3

2.2 Inference Speedup in LLMs

  • Prior LLM speedup approaches include training-free extra modules, earlier-layer exits, and alternative training or inference mechanisms [22] [23] [39] [40] [41] [42].This work instead targets NER speedup without architectural changes or extra modules.Source: §2.2 · p. 3

3 Method

PaDeLLM-NER reframes instruction tuning so each sequence handles one label and its mentions, then uses two-step inference to decode pairs in parallel. Aggregation resolves duplicate mentions across labels using prediction probabilities.

3.1 Reframing of Instruction Tuning

  • Reformulation splits each text into label-specific sequences that predict a mention count followed by indexed mentions; labels with no mentions target the end-of-sequence token.The number of new training examples depends on each example’s ground-truth labels and mentions.Source: §3.1 · p. 4
  • The reformulation shortens generated length per sequence by focusing the model on one label-mention pair at a time.Training uses standard instruction tuning with cross-entropy loss beginning at the mention-count tokens; the authors retain this procedure after finding that ignoring mention-index text has negligible impact.Source: §3.1 · pp. 4–5

3.2 Inference of Label-Mention Pairs

  • Inference first predicts mention counts per label, then appends an index template for each predicted mention so all label-mention pairs can be decoded in parallel.A label with no predicted mentions skips mention decoding and produces an end-of-sequence token.Source: §3.2 · p. 5
  • When GPU resources are limited, batch inference on one GPU supports parallel decoding; with sufficient resources, count and mention inference can run on separate GPUs.The authors note that this strategy is especially effective when inputs arrive in a streaming manner.Source: §3.2 · p. 5

3.3 Removal of Duplicate Mentions

  • Independent pair generation can repeat a mention under different labels because pairs cannot attend to preceding pairs.The paper illustrates this with “Italy” predicted under both LOC and MISC.Source: §3.3 · p. 5
  • For a mention appearing under multiple labels, the method retains the instance with the highest prediction probability, computed from its mention-token probabilities.Token probabilities can be computed during generation, so duplicate identification adds no cost.Source: §3.3 · p. 5

4 Experiments

The experiments assess inference latency and prediction quality across zero-shot and supervised NER benchmarks. PaDeLLM-NER reduces latency against autoregressive methods while retaining strong prediction quality, with results varying across datasets and baselines.

4.1 Setup

  • Evaluation covers zero-shot CrossNER and MIT after training on Pile-NER, plus supervised English and Chinese datasets spanning flat and nested NER.The supervised benchmarks include general-domain and industrial-domain datasets.Source: §4.1 · p. 6
  • Latency comparisons use AutoRegAug and AutoRegStruct with the same pretrained LLMs, while prediction-quality comparisons include zero-shot and supervised SOTA baselines.The English and Chinese studies use Llama2-7b and Baichuan2-7b, respectively.Source: §4.1 · p. 6
  • Quality is measured with micro F-score; latency is averaged across three runs on the same NVIDIA A100, with each PaDeLLM example's latency set by its slowest sequence.The study also reports tokenized average sequence length to show sequence-length reduction.Source: §4.1 · pp. 6–7

4.2 Main Results

  • 791.55 ms: PaDeLLMMulti reduces mean latency versus AutoRegStruct; speedups reach 10.22x, while single-GPU PaDeLLMBatch is 1.76x to 4.73x faster than baselines.Speedup varies across datasets and baseline methods.Source: Table 2 · p. 7, §4.2 · pp. 7–8, Figure 3 · p. 7
  • 61.68: PaDeLLM-NER's average zero-shot score ranks second behind GNER-LLaMA-7B's 66.05.The comparison covers domains including AI, Literature, Music, Politics, Science, Movie, and Restaurant.Source: Table 3 · p. 7, §4.2 · p. 8
  • PaDeLLM-NER outperforms AutoReg in mean F-score across English and Chinese supervised datasets, while encoder-based methods surpass LLM-based methods in the supervised setting.PaDeLLM-NER demonstrates SOTA performance on some task-specific datasets, including YouKu.Source: Table 4 · p. 8, §4.2 · p. 8, Table 5 · p. 9

5 Speedup Analysis

Batch inference provides less acceleration than multi-GPU inference, while PaDeLLM-NER is described as compatible with multiple generative architectures without architectural changes or added modules.

5 Speedup Analysis

  • Batch inference speeds up less than multi-GPU inference; prior work found LLM batching slower under identical conditions, likely due to GPU memory-bandwidth limits [19], [64].The comparison is between batch inference and single-sequence inference under identical conditions.Source: §5 · p. 9
  • PaDeLLM-NER is self-contained and can integrate with decoder-only models, RWKV, and multimodal LLMs without architectural changes or additional data or modules.The paper names Key Information Extraction as a possible multimodal task application.Source: §5 · p. 9
  • The authors plan to explore prompt-engineering integration with off-the-shelf ChatGPT and Claude-2 without further training.Source: §5 · p. 9

6 Data Contamination Concerns

The authors note a possibility that pretraining included parts of their evaluation sets, which could affect results. They consider this risk unlikely to significantly affect comparisons because baselines use the same base LLM.

6 Data Contamination Concerns

  • The LLMs may have encountered parts of the evaluation sets during pretraining.The authors describe this as a possibility, not a confirmed exposure.Source: §6 · p. 9
  • Such exposure could potentially affect the experimental results.Source: §6 · p. 9
  • Because the proposed method and baselines use the same base LLM, the authors consider significant effects on their comparisons unlikely.Source: §6 · p. 9

7 Limitations

The method trades a larger training set for lower inference latency, while facing challenges with mention counting, preserving mention locations, and batch decoding.

7 Limitations

  • Training examples increase from one to m*n, where m is the label count and n the mention count.The authors consider this trade-off justifiable given low latency as an LLM bottleneck.Source: §7 · p. 9
  • Accurately counting mentions remains a challenge for LLMs.Source: §7 · p. 9
  • Reformulating label-mention pairs loses location information, hindering downstream editing.Source: §7 · p. 9
  • During batch decoding, sequences that reach the <eos> token early continue predicting.The authors identify KV-cache reuse and an early-quit mechanism as possible future improvements.Source: §7 · p. 10

8 Conclusion

PaDeLLM-NER batch-decodes all label-mention pairs in parallel. The authors report inference 1.76 to 10.22 times faster without sacrificing prediction accuracy.

8 Conclusion

  • Inference is 1.76 to 10.22 times faster without sacrificing prediction accuracy.Source: §8 · p. 10
  • PaDeLLM-NER is a parallel decoding framework for NER within LLMs.Source: §8 · p. 10
  • The framework enables batch parallel decoding of all label-mention pairs.Source: §8 · p. 10

A Ablation study

The ablations report that PaDeLLM-NER outperforms the one-step variant in both speed and prediction quality, and that de-duplication improves results across tested configurations. Omitting loss on mention markers has minimal impact on prediction quality; zero-shot comparisons also favor PaDeLLM over AutoRegAug.

A Ablation study

B Dataset Statistics

The appendix documents the evaluated datasets and associated processing, and provides reformulated English and Chinese examples.

B Dataset Statistics

C Reformulation Examples

D Implementation Details

The implementation uses a shared training configuration, while the sequence-length analysis reports substantially shorter outputs for PaDeLLM-NER.

D Implementation Details

E Sequence Length Reduction

F Error analysis

The section analyzes sampled ACE2005 errors and reports mixed model-scaling results, alongside disclosures about limitations and experimental reporting.

F Error analysis

  • Manual review of 50 ACE2005 errors distinguishes Count Mismatch, Index Inaccuracy, and Ground Truth Errors.Source: F Error analysis · p. 21
  • About 56.8% of sampled ACE2005 errors were Count Mismatch; overcounts can repeat or fabricate entities, while undercounts can omit entities.Source: F Error analysis · p. 21
  • The authors report that 19.3% of predictions were actually correct and suggest ground-truth inaccuracies contribute to elevated false positives.Source: F Error analysis · p. 21

G Model Scaling Up

  • At 13B, results improve on CoNLL2003 and GENIA but worsen slightly on ACE2005; the GENIA improvement is approximately 1.18%.Source: G Model Scaling Up · p. 21

4. Experimental Result Reproducibility

  • The authors report that implementation and training details appear in Section 4.1 and Appendix D, supporting reproducibility.Source: §4 · p. 23

5. Open access to data and code

  • The authors report releasing the model and code, with the URL masked for anonymous review.Source: §5 · p. 24

2. Limitations

  • The paper acknowledges the absence of traditional statistical tests and points to Section 7 and supplementary failure cases for its limitations.Source: §2 · p. 22, §7 · p. 24
Loading 2402.04838v5…