Source-linked AI summary
NCRF++: An Open-source Neural Sequence Labeling Toolkit
Jie Yang, Yue Zhang
TL;DR
Neural sequence-labeling research has fewer flexible toolkits than statistical CRF sequence labeling, despite the success of neural models. NCRF++ provides a configurable, feature-flexible PyTorch framework with CRF inference and batch GPU acceleration, and experiments show comparable or state-of-the-art performance for models built with it. Its reported results include an efficient running speed, though CNN-based word representations use a smaller learning rate of 0.005.
Problem
Neural sequence labeling has limited toolkit support compared with statistical CRF systems that facilitate flexible feature extraction, training, decoding, and model extension.
Method
NCRF++ uses configurable character, word, and inference layers to build neural sequence-labeling models with handcrafted or automatic features and CRF inference.
Results
Models built with NCRF++ achieve comparable reported performance and state-of-the-art results across the evaluated sequence-labeling experiments, with efficient running speed.
Takeaways & Limitations
NCRF++ facilitates rapid construction, reproduction, and refinement of neural sequence-labeling models through configuration, flexible features, GPU-accelerated batches, and n-best decoding.
Takeaways & Limitations
CNN-based word sequence representations use a smaller learning rate of 0.005.
Abstract
from arXiv · showhide
This paper describes NCRF++, a toolkit for neural sequence labeling. NCRF++ is designed for quick implementation of different neural sequence labeling models with a CRF inference layer. It provides users with an inference for building the custom model structure through configuration file with flexible neural feature design and utilization. Built on PyTorch, the core operations are calculated in batch, making the toolkit efficient with the acceleration of GPU. It also includes the implementations of most state-of-the-art neural sequence labeling models such as LSTM-CRF, facilitating reproducing and refinement on those methods.
1 Introduction
NCRF++ addresses the limited availability of flexible neural sequence-labeling toolkits with a configurable PyTorch framework. It supports feature-rich model design, reproduces state-of-the-art architectures, and provides efficient batch-based computation and n-best decoding.
- NCRF++ is a PyTorch-based neural sequence-labeling toolkit designed for general tasks with effective and efficient neural models.
- Users can build neural models through configuration files without writing code, including a CNN character encoder, word LSTM-CRF framework, POS features, and capitalization features.The illustrated configuration requires fewer than 10 lines.
- NCRF++ supports handcrafted features through distributed lookup-table representations and integrates CNN and LSTM character encoders for reproducing recent models.Feature embeddings may be randomly initialized or loaded from pretrained embeddings.
- Implementations of several state-of-the-art neural models achieve performance comparable to reported results in the literature.
- Batch calculation enables GPU acceleration, while an extended Viterbi algorithm decodes n-best label sequences with probabilities.
2 NCRF++ Architecture
NCRF++ organizes sequence labeling into configurable character, word, and inference layers. Its modular architecture supports multiple encoders, handcrafted features, softmax or CRF outputs, and user-defined extensions through configuration and plug-in layers.
- The architecture comprises character-sequence, word-sequence, and inference layers, combining word, character, and handcrafted feature representations.Word representations concatenate word embeddings, character encodings, and configured handcrafted neural features.
- Character-sequence encoding supports RNN, GRU, LSTM, and CNN options selected through the configuration file.
- The word-sequence layer accepts combined word representations, supports RNN and CNN extractors, and can stack layers for deeper feature extraction.Bidirectional RNNs capture left and right contextual information.
- The inference layer maps sequence representations to label scores and supports either independent softmax probabilities or whole-sequence CRF scoring.
- Configuration interfaces control network structure, data paths, training settings, hyperparameters, feature use, and decoding options.Supported settings include learning rate, hidden dimensions, nbest size, batch size, dropout, and feature embedding size.
- User-defined modules can replace components in all three layers while remaining integrated with the rest of the system.
3 Evaluation
The evaluation tests NCRF++ across NER, chunking, and POS tagging, examining model representations, feature effects, n-best decoding, and batch-size speed. Results show competitive accuracy, strong n-best coverage, and efficient GPU-accelerated execution.
- Evaluation settings: Experiments evaluate NCRF++ on NER, chunking, and POS tagging using CoNLL benchmarks and varied character and word representations.The study uses CoNLL 2003 for NER, CoNLL 2000 for chunking, and the POS data and split from Ma and Hovy (2016).
- Model representations: LSTM-based word representations consistently outperform CNN-based word representations, while character information substantially improves performance.Character LSTM and CNN provide similar improvements, and the comparison is framed as evidence that LSTM better captures global features.
- Model performance: NCRF++ implementations achieve comparable results to reported state-of-the-art models, with better NER and chunking performance but slightly lower POS tagging accuracy.The experiments use nearly identical hyperparameters across tasks, which the authors associate with implementation robustness.
- N-best decoding: 97.47% oracle entity F1 at n = 10 rises from a 91.35% baseline, while token accuracy increases from 98.00% to 99.39%.The authors interpret the n-best outputs as providing broad coverage of gold entities and labels for successor tasks.
- Batch-size speed: Over 2000 sentences/second decoding and 1000 sentences/second training demonstrate efficient execution, with decoding saturating at batch size 100 while training speed continues increasing.The speed measurements use the NER dataset and an Nvidia GTX 1080 GPU.
4 Conclusion
NCRF++ provides an open-source CRF-based toolkit with configurable neural representation layers and flexible feature support. Its experiments report state-of-the-art performance with efficient running speed.
- Toolkit design: NCRF++ combines a CRF architecture with configurable neural representation layers for custom neural sequence labeling models.Users configure model structures through a configuration file.
- Toolkit capabilities: The toolkit supports handcrafted and automatically extracted features and can generate n-best label sequences instead of only the best sequence.These capabilities extend feature utilization and decoding output options.
- Overall evaluation: Experiments report state-of-the-art results from models built in NCRF++ together with efficient running speed.The conclusion summarizes both predictive performance and execution efficiency.