Source-linked AI summary
Attending to Characters in Neural Sequence Labeling Models
Marek Rei, Gamal K. O. Crichton, Sampo Pyysalo
TL;DR
Word-embedding sequence labelers have limited representations for unseen and rare words and do not share morpheme-level information efficiently. The paper adds character-level representations and an attention mechanism that dynamically selects between character and word information. Character extensions improved performance on every benchmark, and attention outperformed concatenation with fewer parameters.
Problem
Word embeddings provide useful similarity representations but handle previously unseen and rare words poorly and encode shared morpheme information inefficiently.
Method
The paper extends a task-independent sequence labeling model with character-level representations and attention-based gating between character and word representations.
Results
Character-based components improved performance on all 8 benchmarks, while the attention-based model outperformed concatenation in every evaluation with fewer parameters.
Takeaways & Limitations
Character and morpheme-level information is useful for general-purpose sequence labeling, and dynamically controlling its use can improve results with fewer parameters.
Takeaways & Limitations
The concatenation-based approach assumes word-level and character-level components learn somewhat disjoint information.
Abstract
from arXiv · showhide
Sequence labeling architectures use word embeddings for capturing similarity, but suffer when handling previously unseen or rare words. We investigate character-level extensions to such models and propose a novel architecture for combining alternative word representations. By using an attention mechanism, the model is able to dynamically decide how much information to use from a word- or character-level component. We evaluated different architectures on a range of sequence labeling datasets, and character-level extensions were found to improve performance on every benchmark. In addition, the proposed attention-based architecture delivered the best results even with a smaller number of trainable parameters.
1 Introduction
Sequence labeling models based on word embeddings are broadly useful but struggle with unseen and rare words. The paper introduces character-level extensions, including attention-based combination, and finds improvements across all evaluated benchmarks.
- Motivation: Sequence labeling supports tasks including NER, POS tagging, and shallow parsing, making accurate and efficient models useful for downstream applications.Neural systems can use surface text alone while achieving competitive results.
- Motivation: Word embeddings struggle with out-of-vocabulary words, rare words, and vocabulary-wide sharing of morpheme-level information.Unseen words fall back to a generic OOV representation, while infrequent-word embeddings may be low quality.
- Approach: The paper extends a task-independent sequence labeling architecture with character-level components that infer representations for unseen words and share morpheme-level regularities.It also introduces an objective that trains character representations to mimic current word embeddings.
- Results: On 8 datasets spanning NER, POS tagging, chunking, and learner-text error detection, character-based components improved performance on every benchmark.The attention-based architecture achieved the best results across all evaluations while requiring fewer parameters.
2 Bidirectional LSTM for sequence labeling
The baseline sequence labeler maps tokens to embeddings, processes them with bidirectional LSTMs and a narrow hidden layer, and predicts labels using either softmax or CRF.
- Architecture: The model maps each token sequence to word embeddings and processes them with forward and backward LSTMs to create context-specific representations.The two directional representations are concatenated for each token.
- Architecture: A narrow hidden layer sits above the LSTM representations to detect higher-level feature combinations while encouraging generalisable patterns.Its size is intentionally kept small.
- Output layer: The network produces label predictions with either a softmax layer or a conditional random field.Softmax assigns a normalized distribution independently over possible labels for each word.
- Output layer: Softmax training minimizes categorical crossentropy, equivalent to minimizing the negative log-probability of the correct labels.The output probability is defined over the possible labels for each word.
- Output layer: The CRF conditions each prediction on the previously predicted label using label-transition weights, then finds the highest-scoring sequence with Viterbi decoding.Training maximizes the correct sequence score while minimizing scores for alternatives.
3 Character-level sequence labeling
Character-level models complement word embeddings by capturing surface and morphological regularities, especially for unseen words. The paper compares concatenating character and word representations with a character architecture integrated into sequence labeling.
- Motivation: Character-level models capture surface and morphological similarities that word embeddings treat as unrelated atomic words.Operating over individual characters can exploit regularities shared across different word forms.
- Character representation: Character embeddings are processed with a bidirectional LSTM to construct a representation for each word.The final hidden vectors are concatenated and passed through a separate nonlinear layer.
- Combined representations: The model maintains both a word-level embedding and a character-built representation for each input word.The word representation is denoted xt, while the character representation is m(t).
- Concatenation architecture: The concatenation architecture supplies word- and character-level vectors separately to the sequence labeler.This design assumes the two components learn somewhat disjoint information.
4 Attention over character features
The attention architecture combines word and character representations through learned, feature-specific weighting rather than simple concatenation. It is designed to align shared semantic information while retaining word-specific exceptions and OOV fallback behavior.
- Attention architecture: The attention model lets word and character components learn aligned semantic features before combining them.The architecture is explicitly constructed to produce the same representations at both levels.
- Weighted combination: A two-layer network predicts weights for a weighted sum of the word embedding and character representation.The vectors are added rather than concatenated.
- Feature-level gating: Feature-specific gates dynamically control how much information comes from character features versus the word embedding.The gate uses the logistic function, with values between 0 and 1, and has the same dimensions as the two representations.
- Training constraint: The character component is trained toward word embeddings using cosine similarity only for non-OOV words.A control term sets this cost component to zero for OOV tokens, excluding the shared generic OOV embedding.
- Complementary information: Word embeddings retain word-specific information and exceptions, while character features provide shared regularities.The optimization is directed toward the character component rather than forcing word embeddings toward character representations.
5 Datasets
The evaluation covers eight sequence-labeling datasets spanning chunking, named entity recognition, part-of-speech tagging, and learner-text error detection. The datasets include both general-domain and biomedical text.
- Evaluation scope: The study evaluates its models on 8 datasets covering NER, POS tagging, chunking, and learner-text error detection.The collection is intended to test the task-independent sequence-labeling models across multiple tasks and domains.
- General-domain datasets: CoNLL00 is a chunking benchmark based on Wall Street Journal sections from the Penn Treebank.Sections 15–18 are used for training and Section 20 for testing, with part of the training data held out for development.
- General-domain datasets: CoNLL03 provides an English language-independent NER dataset drawn from Reuters news stories.The corpus was created for a shared task on language-independent named entity recognition.
- General-domain datasets: PTB-POS contains Wall Street Journal text annotated with 36 main part-of-speech tags plus 12 additional tags.The additional tags cover items such as punctuation.
- Learner-text dataset: FCEPUBLIC is converted into binary token-level error detection using learner essays and examiner corrections.Each token is labeled correct or incorrect in context.
- Biomedical datasets: The biomedical datasets cover gene, chemical, drug, and multiple biomedical entity types, plus biomedical part-of-speech annotations.They include BC2GM, CHEMDNER, JNLPBA, and GENIA-POS.
6 Experiment settings
Experiments use standardized preprocessing, shared model settings, established task-specific evaluation measures, and repeated random seeds. The setup prioritizes controlled comparison across datasets rather than dataset-specific hyperparameter optimization.
- Preprocessing: Words occurring only once are replaced by the generic OOV token for word embeddings but remain available to character components.All digits are also replaced with the character 0.
- Model settings: The models use 200 hidden units per direction, a 50-unit narrow hidden layer, and character representations matching word-embedding dimensionality.A CRF output layer is used for all experiments.
- Attention analysis: Attention values are visualized for restructuring, an in-vocabulary word, and bankrupting, an OOV word, on PTB-POS.Darker blue denotes higher weights for the character-level representation.
- Evaluation procedure: Each model configuration is trained with 10 random seeds, and reported results are averaged across runs.This reduces the influence of randomness from model initialization.
- Evaluation measures: Evaluation uses established measures: token accuracy for POS tasks, F0.5 for FCEPUBLIC, official BC2GM scoring, and mention-level micro-F1 elsewhere.The metric varies by dataset according to prior work.
7 Results
Character-based components improved sequence-labeling performance across all eight benchmarks. The attention-based model performed best while using fewer parameters than concatenation.
- Character-based components improve performance on every benchmark.
- The attention-based character model outperforms concatenation on all evaluations.Its attention mechanism dynamically varies character-level information across words.
- 97.27% on PTB-POS compares with 97.55% by Huang et al. (2015), while 72.70% on JNLPBA compares with 72.55% by Zhou and Su (2004).
- The attention-based architecture achieves improved results with fewer parameters than concatenation.Concatenation enlarges word representations and increases the bidirectional LSTM’s parameter count.
8 Related work
Prior work developed task-independent neural sequence-labeling models and explored recurrent, convolutional, and character-level architectures across NLP applications.
- Earlier neural tagging models applied convolutional or recurrent architectures across POS tagging, chunking, NER, semantic role labeling, opinion mining, and error detection.
- The paper combines design choices from earlier sequence-labeling work and evaluates two alternative character-level architectures.
- Character-level models were applied to tasks including machine translation and language modeling.
9 Conclusion
The paper addresses weaknesses of word-level representations by adding character-level modeling and attention-based combination. Across eight datasets, character information improved performance, with attention outperforming concatenation using fewer parameters.
- Word-level models poorly represent rare words, cannot model unseen words, and do not share morpheme information across the vocabulary.
- Character-level components construct word representations from individual characters alongside a word-level bidirectional LSTM.
- Attention dynamically chooses how much information to use from character representations and word embeddings.
- Across eight datasets and tasks, character information improved every benchmark, while attention outperformed concatenation with fewer parameters.