PaperlayerView exact revision on arXiv

Source-linked AI summary

Empower Sequence Labeling with Task-Aware Neural Language Model

Liyuan Liu, Jingbo Shang, Frank F. Xu, Xiang Ren, Huan Gui, Jian Peng, Jiawei Han

arXiv:1709.04109v4cs.CLcs.LG

TL;DR

Sequence labeling models face limited annotations, while existing knowledge-transfer approaches may require additional supervision or retain irrelevant information. LM-LSTM-CRF combines pre-trained word embeddings with a character-level neural language model and task-aware co-training, using highway layers to mediate the tasks. Across three benchmark tasks, it achieves state-of-the-art effectiveness with a smaller, more efficient model; on CoNLL03 NER it reaches 91.71±0.10 F1 in about six hours on one GPU.

  • Problem

    Sequence labeling models can be difficult to train with limited annotations, while related-task transfer may require unavailable additional supervision and pre-trained language models may contain irrelevant knowledge.

  • Method

    LM-LSTM-CRF combines pre-trained word embeddings with a character-level neural language model, guides it with sequence labeling through co-training, and uses highway layers to mediate discordant tasks.

  • Results

    State-of-the-art performance is achieved on three benchmark datasets, with a smaller network and better efficiency; CoNLL03 NER reaches 91.71±0.10 F1.

  • Takeaways & Limitations

    Task-specific knowledge extracted from self-contained order information enables effective sequence labeling without additional annotation while reducing model size and training cost.

  • Takeaways & Limitations

    Naive co-training can hurt sequence labeling when the language-model and labeling tasks are discordant, motivating highway-layer mediation.

Abstract

from arXiv · show

Linguistic sequence labeling is a general modeling approach that encompasses a variety of problems, such as part-of-speech tagging and named entity recognition. Recent advances in neural networks (NNs) make it possible to build reliable models without handcrafted features. However, in many cases, it is hard to obtain sufficient annotations to train these models. In this study, we develop a novel neural framework to extract abundant knowledge hidden in raw texts to empower the sequence labeling task. Besides word-level knowledge contained in pre-trained word embeddings, character-aware neural language models are incorporated to extract character-level knowledge. Transfer learning techniques are further adopted to mediate different components and guide the language model towards the key knowledge. Comparing to previous methods, these task-specific knowledge allows us to adopt a more concise model and conduct more efficient training. Different from most transfer learning methods, the proposed framework does not rely on any additional supervision. It extracts knowledge from self-contained order information of training sequences. Extensive experiments on benchmark datasets demonstrate the effectiveness of leveraging character-level knowledge and the efficiency of co-training. For example, on the CoNLL03 NER task, model training completes in about 6 hours on a single GPU, reaching F1 score of 91.71$\pm$0.10 without using any extra annotation.

Introduction

Sequence labeling supports core language-understanding tasks, but conventional feature-dependent models and annotation-hungry neural models are difficult to transfer or train in low-resource settings. LM-LSTM-CRF addresses this by combining word- and character-level knowledge from raw text, using task-aware co-training for efficient sequence labeling.

  • Sequence labeling underlies POS tagging, noun phrase chunking, and NER, supporting downstream applications including relation extraction and entity linking.
  • Traditional HMM and CRF methods rely heavily on handcrafted features and language-specific resources, limiting transfer to new tasks or domains.
  • Neural models automatically extract features, but their many parameters and relatively small labeled corpora can make annotations insufficient for training complex models.
  • Transfer and multi-task learning incorporate extra knowledge, yet related-task supervision may be unavailable for low-resource languages or specialized domains.
  • Pre-trained language models provide character-level knowledge but can include task-irrelevant information, requiring larger models, external corpora, and longer training.
  • LM-LSTM-CRF combines word- and character-level knowledge, guides a neural language model toward task-specific information, and achieves significant gains with efficient training.

LM-LSTM-CRF Framework

LM-LSTM-CRF combines character-level language modeling with word-level sequence labeling, using shared representations and task-specific transformations to transfer useful knowledge efficiently.

  • Multi-task Learning Strategy: LM-LSTM-CRF shares character-level representations between a language model and sequence labeling model under a multi-task learning strategy.The language model and sequence labeling model use the same character-level layer.
  • Multi-task Learning Strategy: Highway units transform character-level outputs into separate semantic spaces so each task can select task-specific features.This avoids forcing both tasks to share the entire feature space.
  • Character-level Layer: The character-level component uses forward and backward LSTMs to extract information from both directions of the character sequence.The two directional outputs are processed and aligned differently before transfer to downstream components.
  • Word-level Layer: Word-level processing fine-tunes pre-trained word embeddings and uses a bi-LSTM to capture bidirectional word context.Character-derived features are concatenated with word embeddings before entering the word-level bi-LSTM.
  • CRF for Sequence Labeling: A CRF layer models dependencies among labels and jointly decodes label sequences to avoid illegal annotations.Training minimizes negative log likelihood, while decoding seeks the highest-likelihood sequence using the Viterbi algorithm.
  • Neural Language Model: The language model predicts words from character sequences in both forward and reverse directions and contributes to a joint training objective.The joint objective combines the sequence-labeling objective with forward and reverse language-model terms, weighted by λ.

Experiments

Experiments evaluate LM-LSTM-CRF on benchmark datasets for NER, chunking, and POS tagging, using dataset-specific annotation schemes and standard or held-out development splits.

  • Datasets: LM-LSTM-CRF is evaluated on CoNLL 2003 NER, CoNLL 2000 chunking, and the WSJ portion of Penn Treebank POS tagging.The three datasets cover named entities, syntactic chunks, and 45 POS tags, respectively.
  • Datasets: CoNLL03 NER contains four entity types—PER, LOC, ORG, and MISC—and is divided into training, development, and test sets.
  • Datasets: CoNLL00 chunking defines eleven syntactic chunk types plus Other and uses 1000 sampled training sentences as a held-out development set.
  • Datasets: WSJ uses the standard split: sections 0–18 for training, 19–21 for development, and 22–24 for testing.Each word is categorized into one of 45 POS tags.

Network Training

Network training reuses prior hyperparameter settings for fair comparison while expanding the LSTM state size, and evaluates against sequence-labeling and supervised multitask baselines.

  • Training setup: Initialization, optimization, and related hyperparameters are borrowed from Ma and Hovy (2016), except for the LSTM state size.The authors report limited parameter tuning for fair comparison.
  • Training setup: The LSTM hidden state size is increased from 200 to 300 because additional knowledge allows training a larger network.
  • Optimization: Training uses mini-batch stochastic gradient descent with momentum, batch size 10, momentum 0.9, dropout 0.5, and gradient clipping at 5.0.The learning rate decays with η_t = η_0/(1 + ρt), where ρ = 0.05.
  • Evaluation protocol: Early stopping on the development set keeps the model training restricted to the training set, unlike baselines that incorporate development data.
  • Baselines: Baselines include sequence-labeling-only models and methods jointly trained with other supervised tasks.

Performance Comparison

LM-LSTM-CRF outperforms most sequence-labeling baselines across NER, POS tagging, and chunking, while maintaining comparatively efficient training without additional resources. Its task-aware co-training also achieves strong performance with a concise model.

  • LM-LSTM-CRF significantly outperforms all baselines without additional resources and surpasses most baselines using extra resources.It is only slightly worse than TagLM (index 4).
  • NER: 91.71±0.10 F1 is achieved by LM-LSTM-CRF on CoNLL03 NER, compared with 91.62±0.23 for the smaller TagLM variant.The comparison supports reducing redundancy through task-specific guidance.
  • POS Tagging: LM-LSTM-CRF outperforms all baselines on WSJ POS tagging, with higher F1 scores and smaller variances than competing models on NER and POS.The improvements over LSTM-CRF and CNN-LSTM-CRF are less obvious on POS tagging than on CoNLL03 NER.
  • Chunking: LM-LSTM-CRF outperforms all baselines without extra resources and most baselines trained with extra resources on CoNLL00 chunking.The smaller corpus is associated with greater variance in the results.
  • Efficiency: LM-LSTM-CRF has the best reported performance and roughly the same efficiency as competing methods despite taking twice the training time of LSTM-CNN-CRF⋆.The language-model component adds only a small number of parameters in two highway units and a soft-max layer.
  • Efficiency: LM-LSTM-CRF achieves similar performance to externally pre-trained language-model systems without relying on extensive external corpora, using a more concise model and efficient training.The framework is intended to extract task-specific knowledge through its language model.

Analysis

Additional experiments show that LM-LSTM-CRF benefits from larger hidden states, effective co-training, and highway layers that mediate language-model and sequence-labeling objectives.

  • Model size: LM-LSTM-CRF F1 continues increasing as hidden-state size grows, unlike the comparison models’ observed peaks or drops.LSTM-CNN-CRF peaks at state size 200, while LSTM-CRF drops at state size 200.
  • Ablation analysis: LM-LSTM-CRF NH performs worse than LM-LSTM-CRF NL, supporting the effectiveness of the proposed co-training strategy.The authors attribute this pattern to weak relatedness between NER and the language-model task.
  • Ablation analysis: Highway layers are necessary because they mediate the discrepant semantic spaces of language modeling and sequence labeling.The comparison without highway units underperforms the variant retaining them.

Related Work

Related sequence-labeling systems evolved from handcrafted-feature models toward end-to-end neural architectures that incorporate character structure and label dependencies.

  • Sequence Labeling: Traditional sequence-labeling methods included CRFs, HMMs, and maximum entropy classifiers, often using handcrafted features.These approaches represent earlier work in sequence labeling before the end-to-end neural models discussed here.
  • Component analysis: Table 10 examines the effects of the language model and highway components.The table is presented as an analysis of these two components.
  • Sequence Labeling: End-to-end systems such as BiLSTM-CNN, LSTM-CRF, and LSTM-CNN-CRF incorporate character-level structure and improve over pure word-level models.LSTM-CNN-CRF is identified as a state-of-the-art method in the cited NER and POS-tagging work.
  • Sequence Labeling: CRF layers capture dependencies among labels in neural sequence-labeling architectures.The cited related work reports that CRF layers have been demonstrated effective for modeling label dependencies.

Conclusion

The paper proposes LM-LSTM-CRF to extract character-level knowledge from sequence order and uses highway layers to address discordant co-training tasks, achieving state-of-the-art results with improved efficiency.

  • Conclusion: LM-LSTM-CRF extracts character-level knowledge from self-contained order information through a language model.The framework is designed to leverage this knowledge for sequence labeling.
  • Conclusion: Highway layers address discordance in naive co-training between the language model and sequence-labeling task.The resulting task-specific knowledge supports a more concise model and better efficiency without loss of effectiveness.
  • Conclusion: The framework achieves state-of-the-art performance on three benchmark datasets while improving efficiency.The conclusion identifies this as the principal reported outcome.
Loading 1709.04109v4…