Source-linked AI summary
Design Challenges and Misconceptions in Neural Sequence Labeling
Jie Yang, Shuailong Liang, Yue Zhang
TL;DR
Neural sequence-labeling results are difficult to reproduce and compare because published studies use inconsistent settings and sometimes report contradictory conclusions. The paper reproduces twelve architectures in a unified framework across NER, chunking, and POS tagging, finding that character information helps, word-level LSTMs usually outperform CNNs, and CRF benefits depend on the task.
Problem
Published neural sequence-labeling results are difficult to reproduce and compare because studies use inconsistent experimental settings and report contradictory conclusions.
Method
The paper reproduces twelve neural sequence-labeling architectures in a unified framework and evaluates them on NER, chunking, and POS tagging.
Results
Character information improves performance, word-level LSTMs usually outperform CNNs, and CRF inference helps on NER and chunking but not POS tagging.
Takeaways & Limitations
Controlled comparisons show that character representations, word encoders, and inference layers should be assessed jointly rather than compared through isolated published scores.
Takeaways & Limitations
The analysis fine-tunes learning rates for several optimizers and reports the best results, while result analysis uses models with median test-set performance.
Abstract
from arXiv · showhide
We investigate the design challenges of constructing effective and efficient neural sequence labeling systems, by reproducing twelve neural sequence labeling models, which include most of the state-of-the-art structures, and conduct a systematic model comparison on three benchmarks (i.e. NER, Chunking, and POS tagging). Misconceptions and inconsistent conclusions in existing literature are examined and clarified under statistical experiments. In the comparison and analysis process, we reach several practical conclusions which can be useful to practitioners.
1 Introduction
The paper identifies reproducibility and fairness problems in neural sequence-labeling research, driven by inconsistent experimental settings and contradictory reports. It addresses these concerns by comparing twelve architectures across NER, chunking, and POS tagging in a unified framework.
- Research challenges: Published neural sequence-labeling results can be difficult to reproduce, even when source code is publicly available.Prior reports also reach contradictory conclusions, including about which optimizer performs best on NER.
- Research challenges: Literature comparisons vary in datasets, data splits, development-set usage, dataset size, and preprocessing, complicating fair evaluation.The paper lists six inconsistent configuration choices across existing studies.
- Approach: The study evaluates twelve neural sequence-labeling architectures on CoNLL 2003 NER, CoNLL 2000 chunking, and PTB POS tagging.The models are organized around character representations, word representations, and inference layers.
- Approach: A unified framework reproduces existing models with comparable or better results, supporting the practicability and reliability of systematic comparison.The framework is intended to provide fair comparisons under consistent conditions.
2 Related Work
Prior work established neural architectures for sequence labeling using CNNs, LSTMs, character information, and CRF output layers. More recent comparison studies examine variability across runs and configurations, motivating this paper’s unified end-to-end evaluation.
- Neural sequence-labeling architectures: Early neural sequence-labeling systems combined pretrained word embeddings, CNN-based sequence modeling, handcrafted features, and CRF decoding.Later work added character CNNs and deeper dilated CNNs, while LSTM-based models introduced recurrent sequence modeling.
- Neural sequence-labeling architectures: BiLSTM-CRF architectures were extended with character-level representations and became prominent structures for sequence labeling.The related work traces extensions across several influential neural models.
- Comparison studies: Reimers and Gurevych compared word-based LSTM systems across tasks using score distributions over multiple runs and investigated hyperparameters and configurations.Their study differs from this work in its use of handcrafted word features and a BiLSTM-based system.
- Comparison studies: This paper instead compares end-to-end neural architectures under unified settings across a broader set of design choices.The stated contrast concerns model inputs, performance scope, and the architectures examined.
3 Neural Sequence Labeling Models
The framework decomposes neural sequence labeling into character representation, word representation, and inference layers. It compares CNN and LSTM encoders at character and word levels, alongside softmax and CRF decoding.
- Overall architecture: The framework contains character sequence representation, word sequence representation, and inference layers.Character and word representations are subsequently passed to an inference layer for label assignment.
- Character sequence representations: Character-level representations can be learned without hand-engineered features, focusing the comparison on neural character sequence encoders.The paper contrasts this approach with feature-based lookup representations.
- Character sequence representations: Character representations are built either with a one-layer CNN using max-pooling or a bidirectional LSTM over each word’s character sequence.The character LSTM concatenates final forward and backward states; the selected structure supports parallel computation.
- Word sequence representations: Word sequence information is modeled with either LSTM encoders for global context or CNNs for local features and parallel computation.The word CNN uses stacked convolutional layers, while the bidirectional word LSTM concatenates forward and backward hidden states.
- Inference layer: The inference layer assigns labels using either independent softmax decoding or CRF decoding over the extracted word representations.The paper examines both inference choices across NER, chunking, and POS tagging.
4 Experiments
The experiments compare twelve neural sequence-labeling models across NER, chunking, and POS tagging while varying architecture, optimization, embeddings, tagging, and decoding factors. Results show broad reproducibility, task-dependent inference effects, and accuracy–speed trade-offs.
- Experimental design: The study evaluates character and word encoders, inference layers, embeddings, tag schemes, optimizers, decoding speed, and OOV performance across three benchmarks.The benchmarks are NER, chunking, and POS tagging, with repeated experiments under shared evaluation procedures.
- Reproducibility: Most published results are reproducible, with the implementations achieving comparable or better state-of-the-art performance under the same hyperparameters across tasks.The authors report no task-specific hyperparameter fine-tuning, using this shared setup to assess generalization.
- Network settings: Character information significantly improves sequence labeling (p < 0.01), whereas character LSTM and CNN differences are generally small and statistically insignificant.Character LSTM and CNN are comparable across frameworks and tasks, with task- and configuration-specific exceptions.
- Network settings: Word-based LSTM models significantly outperform word-based CNN models in most cases (p < 0.01), indicating the importance of global word context.The comparison spans the three benchmarks and contrasts word LSTM and CNN structures with and without CRF inference.
- Network settings: CRF inference consistently outperforms softmax on NER and chunking, while softmax is slightly better for POS tagging without a significant difference (p > 0.2).The results associate label-dependency modeling with gains on NER and chunking but not with a clear POS advantage.
- External factors: Pretrained embeddings and BIOES tagging improve NER, while SGD achieves the best optimizer results despite slower convergence.GloVe 100-dimension embeddings outperform SENNA and random initialization, and SGD significantly outperforms the other tested optimizers (p < 0.01).
- Efficiency: Character CNN preserves decoding speed better than character LSTM while improving accuracy, and word CNN decodes faster than word LSTM with close accuracies.CRF inference also severely limits parallel decoding, creating a practical efficiency trade-off.
- OOV analysis: Character LSTM and CNN representations improve OOV performance most, supporting their role in disambiguating out-of-vocabulary words.Character LSTM gives the best in-vocabulary scores, while character LSTM and CNN are comparable on OOV cases.
5 Conclusion
The unified comparison isolates how character and word representations, inference layers, tagging schemes, embeddings, and optimizers affect neural sequence labeling. Results show several design choices are task- or efficiency-dependent rather than universally superior.
- Character information improves performance, particularly for disambiguating OOV words.
- Character-level LSTM and CNN provide comparable improvements, while CNN is more efficient.
- Word-level LSTM encoders generally outperform CNN encoders but require longer decoding time.
- CRF inference is effective on NER and chunking but offers no advantage on POS tagging.
- BIOES tags outperform BIO, while pretrained GloVe 100d embeddings and SGD significantly outperform their competitors.
- Result analysis uses models with median test-set performance.