Source-linked AI summary
Neural Word Segmentation Learning for Chinese
Deng Cai, Hai Zhao
TL;DR
Chinese word segmentation methods have been limited by character-level labeling, fixed context windows, and restricted history modeling. This paper introduces direct structured segmentation with character-derived word representations and sentence-level scoring, achieving better results than recent neural segmenters and performance comparable to previous state-of-the-art systems, while retaining a strict four-character limitation that can reduce performance on longer-word data.
Problem
Existing Chinese word segmentation approaches use fixed context windows and simple tag interactions, which restrict modeling of diverse distances and complete segmentation history.
Method
The framework directly scores candidate segmentations using character-derived distributed word representations, sentence-level likelihood evaluation, decoding, and max-margin training.
Results
The models outperform previous neural network segmenters and are comparable to previous state-of-the-art systems.
Takeaways & Limitations
Direct structured learning offers a neural alternative that models complete segmentation history without relying on feature engineering or fixed context windows.
Takeaways & Limitations
The four-character limitation is overly strict and causes additional performance loss on datasets with larger average word lengths, such as MSR.
Abstract
from arXiv · showhide
Most previous approaches to Chinese word segmentation formalize this problem as a character-based sequence labeling task where only contextual information within fixed sized local windows and simple interactions between adjacent tags can be captured. In this paper, we propose a novel neural framework which thoroughly eliminates context windows and can utilize complete segmentation history. Our model employs a gated combination neural network over characters to produce distributed representations of word candidates, which are then given to a long short-term memory (LSTM) language scoring model. Experiments on the benchmark datasets show that without the help of feature engineering as most existing approaches, our models achieve competitive or better performances with previous state-of-the-art methods.
1 Introduction
Chinese word segmentation is needed because Chinese text lacks explicit word delimiters, but prevailing character-labeling methods rely on handcrafted features, fixed context windows, and limited tagging history. The paper instead directly learns segmentation using distributed word-candidate representations and complete segmentation history.
- Chinese word segmentation is a preliminary processing step because Chinese is written without explicit word delimiters.
- Earlier sequence-labeling methods depend heavily on handcrafted features and score characters using fixed local context and limited tagging history.These methods typically model only interactions between adjacent tags and the previous tagging history.
- Fixed context windows restrict modeling flexibility, while tag transitions cannot represent the complete influence of previous segmentation decisions.The paper also identifies word-level information as a missing granularity in prior approaches.
- The proposed framework directly evaluates segmented sentences rather than assigning tags to individual characters, then searches for the highest-scoring segmentation.Word candidates are represented with distributed vectors, enabling more discriminative features to be captured.
- The framework builds representations across character, word, and sentence levels, beginning with character embeddings and deriving word-candidate representations from their characters.
- The model eliminates context windows and captures the complete history of segmentation decisions and input characters.
2 Overview
The paper formulates Chinese word segmentation as scoring possible segmentations of an input character sequence, with scores sensitive to the complete contents of partially segmented sentences. Its neural scoring model, decoder, and max-margin training procedure jointly identify the highest-scoring segmentation.
- Problem formulation: Chinese word segmentation maps an input character sequence x to a word sequence y selected from the possible segmentations GEN(x).The output is defined as the segmentation with the highest score among candidate word sequences.
- Problem formulation: The scoring function is sensitive to the complete contents of a partially segmented sentence, unlike previous approaches.
- Model and decoding: The neural network scoring model evaluates the likelihood of a segmented sentence, and a decoder finds the sentence with the highest score.
- Training: Max-margin training compares the decoder output with the golden segmentation using their structured difference.
- Model and decoding: The architecture combines input characters, learned representations of word candidates, and predictions for subsequent word candidates to score individual candidates.Figure 2 identifies ci as an input character, yj as a learned word-candidate representation, pk as the prediction for the next candidate, and u as the trainable scoring parameter vector.
3 Neural Network Scoring Model
The scoring model represents candidate words from characters, scores their individual plausibility and links to prior segmentation history, and combines these signals into sentence scores. A gated combination network captures character composition, while an LSTM preserves complete segmentation history for sequence-level discrimination.
- Scoring framework: Candidate word vectors are scored for both the candidate’s own word likelihood and its compatibility with the previous segmentation history.The sentence score combines word scores and link scores after each candidate updates the scoring system’s history.
- Character representation: Character embeddings are used instead of immediate word embeddings because rare and out-of-vocabulary words are poorly estimated, while internal character structure helps judge candidate words.
- Gated combination neural network: The gated combination neural network uses reset gates to select character information and update gates to determine what to preserve when composing a word vector.The mechanism is designed to capture character and character-interaction characteristics in an efficient word representation.
- Scoring framework: The sentence score sums individual word scores and link scores, with the link score computed from the predicted and actual candidate vectors.
- Link score: The model’s LSTM chains word candidates left to right, producing predictions from hidden states that encode the entire segmentation history.This gives the model sequence-level discrimination rather than only local optimization.
4 Decoding
Because complete segmentation history makes exhaustive scoring exponential, the decoder uses beam search with dynamic-programming structure to retain promising partial segmentations. Its runtime is bounded by maximum word length, beam size, and sentence length.
- Decoding challenge: Complete-history scoring makes enumerating every segmented sentence impractical because the number of candidate segmentations grows exponentially with character-sequence length.
- Beam search: Beam search maintains approximate top segmentations while extending partial hypotheses with candidate words.Each partial segmentation carries its score, hidden state, and memory cell state.
- Beam search: A partial segmentation of the first i characters is split at position j into a prefix hypothesis and the remaining candidate word.
- Complexity: Beam search takes total time w × k × n for a sentence of n characters, where w is maximum word length and k is beam size.
5 Training
The model is trained with a structured max-margin objective that penalizes incorrect segmentations according to their incorrectly segmented characters, with regularization and subgradient optimization.
- Max-margin training: The structured margin loss compares a predicted segmentation with the correct segmentation for each training character sequence.
- Max-margin training: The margin loss is proportional to the number of incorrectly segmented characters through a fixed discount parameter.The sequence length and discount parameter determine the loss calculation.
- Objective: The regularized objective maximizes over candidate segmentations the sentence score plus margin loss minus the correct segmentation score.The sentence score is the model score defined for segmented sentences.
- Optimization: Because the hinge-loss objective is nondifferentiable, training uses subgradients and diagonal AdaGrad with minibatches.
6 Experiments
Experiments evaluate preprocessing, hyperparameters, scoring components, model architecture, and comparisons with prior systems on PKU and MSR. The proposed model benefits from GCNN and complete segmentation history, while maximum word length affects MSR performance.
- Experimental setup: PKU and MSR are evaluated using standard Bakeoff precision, recall, and F1-score after replacing continuous English characters and digits with a unique token.
- Hyper-parameters: Beam size 4 is sufficient for the best PKU performance, balancing accuracy and efficiency.
- Limitations: The four-character limit is acknowledged as especially strict for MSR because that corpus has a longer average word length, causing additional performance loss.
- Model analysis: 95.5% F1-score with GCNN exceeds 94.0% for the simplified model, whose best result is 94.7% with a much larger character embedding.
- Model analysis: The link-score-only model performs much better than the word-score-only model and approaches the joint model, highlighting the value of complete segmentation history.
- Comparison with prior systems: Under closed-set comparisons, the model achieves state-of-the-art performance on PKU and a competitive MSR result, while outperforming prior neural network methods under matched settings.
- Model analysis: Allowing six-character words improves MSR F1-score by 0.3%, although longer maximum word lengths require more training time.
7 Related Work
Related work largely treats Chinese word segmentation as character-level sequence labeling or uses word-based models with handcrafted features. These approaches generally rely on local context windows, while some alternatives incorporate word-level information but remain window-limited.
- Neural network models: Most neural Chinese word segmentation methods score tags on individual characters using context information.
- Other techniques: The released code is available through the cited project repository.
- Alternatives to sequence labeling: Word-based perceptron and incremental models can use word-level features but rely heavily on massive handcrafted features.
- Alternatives to sequence labeling: Ma and Hinrichs model segmentation as configuration-action matching but still use context information within limited-sized windows.
- Other techniques: Other state-of-the-art techniques include semi-supervised learning, global information, and joint models that could further benefit the proposed approach.
8 Conclusion
The paper presents a neural framework that directly learns Chinese word segmentation as structured scoring over candidate words and segmented sentences. It reports better results than recent neural segmenters and performance comparable to prior state-of-the-art systems, while leaving room for improvement.
- The framework combines word representation construction, sentence-level likelihood evaluation, and an efficient search algorithm for the best segmentation.
- It reformulates word segmentation as a direct structured learning procedure using distributed representations.
- The system performs better than recent neural network segmenters but remains comparable to previous state-of-the-art systems.
arXi
Chinese word segmentation is needed because Chinese and other East Asian languages lack explicit word delimiters. Existing approaches commonly use character-level sequence labeling, but fixed windows, limited tag transitions, and underused word-level information constrain the modeling of segmentation context.
- Chinese lacks explicit word delimiters, making word segmentation a preliminary step for many natural-language-processing tasks.
- Since Xue (2003), most Chinese word-segmentation methods have formalized the task as sequence labeling with character-position tags.
- Adjacent tag transitions cannot model the complete influence of historical segmentation decisions.
- Fixed-size context windows restrict modeling flexibility across diverse information distances.
- Sequence-labeling schemes leave word-level information unemployed, while word-based methods rely heavily on handcrafted features and cannot fully model word interactions.
- A valid segmentation should produce a natural, complete, coherent, and smooth word sequence from the input characters.
Formalization
The framework scores candidate segmentations as word sequences while combining character, word, and sentence-level information. It avoids sliding windows and Markov assumptions, retains complete segmentation history, and uses beam search for approximate inference.
- Formalization: The formal task maps an input character sequence x to the highest-scoring segmentation y* among GEN(x).
- Framework: The scoring framework represents an output sentence as a sequence of word candidates and evaluates each candidate using word and history-link scores.
- Benefits: The model directly represents segmentation structure and covers character, word, and sentence processing levels.
- Benefits: The scoring state uses complete historical information, including prior characters and segmentation decisions.
- Benefits: The framework uses no sliding context window and makes no Markov assumption.
- Beam Search: Beam search iteratively approximates the k-best segmentations of prefixes using a specified beam size and maximum word length.
- Beam Search: Because possible segmentations grow exponentially with sequence length, inference splits each segmentation into its last word and preceding subsegmentation.
Beam Size
Experiments examine beam size, scoring strategies, prior-method comparisons, and performance issues associated with longer words. The analyses identify a balance between accuracy and efficiency and highlight the importance of link scoring.
- Different beam sizes are evaluated on the PKU dataset.
- The beam-size analysis identifies a good balance between accuracy and efficiency.
- Different model performances are compared on the PKU dataset.
- Different scoring strategies, including word score, link score, and both, are compared on the PKU dataset.
- Link score plays a critical role in gaining performance improvement, and link-only scoring performs close to the joint model.
- Prior-method comparisons distinguish results obtained from released implementations and results using external dictionaries or corpora.
- Long words longer than 4 characters account for 0.19% of the PKU test set and 1.07% of the MSR test set.
- Words longer than 6 characters account for 0.42% of the MSR test set, while longer words involve less training data and more GCNN parameters.