PaperlayerView exact revision on arXiv

Source-linked AI summary

N-LTP: An Open-source Neural Language Technology Platform for Chinese

Wanxiang Che, Yunlong Feng, Libo Qin, Ting Liu

arXiv:2009.11616v4cs.CL

TL;DR

Chinese NLP lacks comprehensive high-performance tooling across fundamental tasks, while existing independent models overlook shared knowledge. N-LTP addresses this gap with an open-source six-task platform using shared multi-task modeling, distillation, and user-facing tools, achieving state-of-the-art or competitive performance and high speed.

  • Problem

    Chinese NLP has relatively few high-performance, high-efficiency toolkits supporting comprehensive fundamental analysis, while independent task models ignore shared knowledge and scale poorly in memory.

  • Method

    N-LTP uses a shared multi-task framework for six Chinese NLP tasks, adds teacher-to-student knowledge distillation, and provides APIs and visualization tools.

  • Results

    N-LTP achieves state-of-the-art or competitive performance on six Chinese NLP tasks and has high speed.

  • Takeaways & Limitations

    N-LTP provides an open-source Chinese toolkit intended to facilitate Chinese NLP research through comprehensive task coverage and accessible use.

Abstract

from arXiv · show

We introduce \texttt{N-LTP}, an open-source neural language technology platform supporting six fundamental Chinese NLP tasks: {lexical analysis} (Chinese word segmentation, part-of-speech tagging, and named entity recognition), {syntactic parsing} (dependency parsing), and {semantic parsing} (semantic dependency parsing and semantic role labeling). Unlike the existing state-of-the-art toolkits, such as \texttt{Stanza}, that adopt an independent model for each task, \texttt{N-LTP} adopts the multi-task framework by using a shared pre-trained model, which has the advantage of capturing the shared knowledge across relevant Chinese tasks. In addition, a knowledge distillation method \cite{DBLP:journals/corr/abs-1907-04829} where the single-task model teaches the multi-task model is further introduced to encourage the multi-task model to surpass its single-task teacher. Finally, we provide a collection of easy-to-use APIs and a visualization tool to make users to use and view the processing results more easily and directly. To the best of our knowledge, this is the first toolkit to support six Chinese NLP fundamental tasks. Source code, documentation, and pre-trained models are available at \url{https://github.com/HIT-SCIR/ltp}.

1 Introduction

N-LTP addresses limited high-performance Chinese NLP tooling with an open-source platform covering six fundamental tasks. It combines shared multi-task modeling, distillation, and user-facing tools to provide comprehensive and efficient processing.

  • Motivation: Chinese NLP has fewer high-performance, high-efficiency toolkits despite growing demand for processing fundamental tasks.The targeted tasks include word segmentation, POS tagging, NER, dependency parsing, semantic dependency parsing, and semantic role labeling.
  • Motivation: Independent task models omit shared knowledge and require more memory as task count increases, complicating mobile deployment.N-LTP instead uses a shared encoder in a multi-task framework to reduce memory and improve speed.
  • Contributions: N-LTP is an open-source neural Chinese NLP platform supporting six fundamental tasks across lexical, syntactic, and semantic analysis.It is described as the first neural Chinese toolkit supporting all six tasks.
  • Contributions: Single-task teachers distill knowledge into the multi-task model to help it surpass its teachers.The distillation method is introduced alongside multi-task learning to enhance subtask performance.
  • Contributions: N-LTP provides extensibility through configurable BERT-like pretrained models and open-sourced task-training configurations.Users can add a new pretrained model by changing configuration settings.
  • Contributions: N-LTP offers easy-to-use APIs, visualization, and multiple-language bindings for directly using and viewing processing results.The visualization tool supports direct inspection of results, while bindings include C++, Python, Java, and Rust.
  • Evaluation: Across six Chinese NLP tasks, N-LTP achieves state-of-the-art or competitive performance at each task.The platform is evaluated on all six tasks and is reported to outperform Stanza by a large margin on CWS, POS, NER, and DEP.

2 Design and Architecture

N-LTP uses a shared pre-trained encoder with task-specific decoders to jointly process Chinese NLP tasks. Its architecture combines multi-task learning with knowledge distillation to encourage the joint model to surpass single-task teachers.

  • Architecture: N-LTP shares one encoder across tasks and uses separate decoders optimized simultaneously in a joint learning scheme.The shared encoder is intended to leverage knowledge common to the tasks.
  • Shared Encoder: The shared encoder uses the ELECTRA pre-trained model to capture shared knowledge across six Chinese tasks.
  • Shared Encoder: ELECTRA converts an input sequence with [CLS] and [SEP] tokens into corresponding hidden representations.The constructed sequence is [CLS], the input tokens, and [SEP].
  • Chinese Word Segmentation: Chinese word segmentation is treated as character-based sequence labeling, with a linear decoder classifying each character from the hidden representations.The decoder uses trainable parameters W_CWS and b_CWS.

2.3 POS Tagging

N-LTP formulates part-of-speech tagging as sequence labeling over the shared hidden representations. It predicts the POS label sequence using the representations associated with the input characters or words.

  • POS Tagging: POS tagging is treated as a sequence labeling task in N-LTP.
  • POS Tagging: The model takes the hidden representation sequence H as input and outputs the corresponding POS sequence labels.
  • POS Tagging: For each input position, y_i denotes the POS label probability distribution, while h_i is the first sub-token representation of word s_i.

2.4 Named Entity Recognition

N-LTP treats named entity recognition as identifying entity spans and assigning entity classes. It uses an Adapted-Transformer before linear classification to incorporate direction- and distance-aware characteristics.

  • Named Entity Recognition: Named entity recognition finds entity starts and ends and assigns classes such as people, locations, and organizations.
  • Named Entity Recognition: N-LTP applies an Adapted-Transformer to hidden representations to model direction- and distance-aware characteristics.
  • Named Entity Recognition: The Adapted-Transformer produces updated representations ˆH for the sequence.
  • Named Entity Recognition: A linear decoder then classifies a label for each word using the updated representations.

2.5 Dependency Parsing

N-LTP performs dependency parsing with a deep biaffine neural parser and the Eisner algorithm. The scoring process is extended from dependency arcs to labeled dependencies.

  • Dependency Parsing: N-LTP implements a deep biaffine neural dependency parser together with the Eisner algorithm to obtain dependency parses.
  • Dependency Parsing: The parser computes a score for each dependency arc i↶j.
  • Knowledge Distillation: The distillation overview increases λ linearly from 0 to 1 over the training curriculum.
  • Dependency Parsing: The same scoring process is extended from a one-dimensional score vector to L dimensions for dependency labels.L denotes the total number of dependency labels.

2.6 Semantic Dependency Parsing

Semantic dependency parsing identifies semantically related word pairs and assigns predefined relations. N-LTP uses a biaffine module for this task, while its broader toolkit also includes semantic role labeling.

  • 2.6 Semantic Dependency Parsing: Semantic dependency parsing determines which word pairs are semantically related and assigns each pair a predefined semantic relation.
  • 2.6 Semantic Dependency Parsing: N-LTP performs semantic dependency parsing with a biaffine module following Dozat and Manning (2017).
  • 2.7 Semantic Role Labeling: The toolkit also supports semantic role labeling using a deep biaffine network with a CRF-based decoder.

2.8 Knowledge Distillation

Because multi-task learning may not benefit every task, N-LTP uses knowledge distillation to guide the shared model with separately trained task-specific teachers.

  • 2.8 Knowledge Distillation: With many tasks, it can be difficult to ensure that every task benefits from multi-task learning.
  • 2.8 Knowledge Distillation: N-LTP trains each task as a teacher model before distilling its knowledge into the multi-task model.
  • 2.8 Knowledge Distillation: The multi-task model learns from single-task teachers while simultaneously learning from gold-standard labels.
  • 2.8 Knowledge Distillation: Teacher annealing uses task-sampling probabilities proportional to |Dτ|^0.75 so large datasets do not dominate training.

3 Usage

N-LTP is designed for simple integration through JSON-configured models, one-line usage, multilingual bindings, and an interactive visualization demo for inspecting annotations.

  • 3 Usage: N-LTP configurations can be initialized from JSON files, allowing users to load models or process sentences with one line of code.
  • 3 Usage: The toolkit provides bindings for C++, Python, Java, and Rust.
  • 3 Usage: Its APIs support downloading models, annotating sentences with customized models, and predicting all annotations.
  • 3 Usage: An interactive web demo runs the pipeline and visualizes processing results for a Chinese sentence.

4 Experiments

Experiments evaluate N-LTP across six Chinese tasks against Stanza and prior single-task systems, showing strong task performance alongside speed and memory advantages from joint modeling.

  • 4 Experiments: N-LTP is evaluated on six Chinese tasks and compared with Stanza using the same datasets adopted by Stanza.
  • 4.2 Results: N-LTP outperforms Stanza by a large margin on CWS, POS, NER, and DEP.
  • 4.2 Results: Multi-task learning outperforms independently trained models, consistent with using shared knowledge across tasks.
  • 4.3.1 Speedup and Memory Reduction: N-LTP jointly trained with distillation achieves a x4.3 speedup over separate modeling, while separately trained N-LTP achieves x1.7 over Stanza.
  • 4.3.1 Speedup and Memory Reduction: Despite performing six tasks, jointly trained N-LTP requires half the memory of Stanza, which performs four tasks.
  • 4.2 Results: N-LTP achieves the best performance on five of six tasks but underperforms the best baseline on SDP.

5 Conclusion

N-LTP is presented as an open-source Chinese language technology platform supporting six fundamental Chinese NLP tasks. The authors report state-of-the-art or competitive performance and high speed, with the goal of facilitating Chinese NLP research.

  • N-LTP is an open-source platform supporting six fundamental Chinese NLP tasks.The six tasks span lexical analysis, syntactic parsing, and semantic parsing.
  • N-LTP obtains state-of-the-art or competitive performance across its supported tasks.
  • N-LTP has high speed and is intended to facilitate Chinese NLP research.
Loading 2009.11616v4…