Source-linked AI summary
Neural Word Segmentation with Rich Pretraining
Jie Yang, Yue Zhang, Fei Dong
TL;DR
Neural segmentation had benefited from pretraining on raw text, but richer external information used in statistical segmentation had not been systematically investigated for neural models. The paper pretrains a modular segmentor’s shared character-context network on external-source classification tasks. Rich pretraining reduces relative error by 15.4%, and the model is highly competitive on six benchmarks.
Problem
Rich external resources had not been systematically investigated for neural word segmentation, despite their use in statistical segmentation.
Source: §1 · pp. 1–2Method
The paper uses a modular beam-search segmentor and multitask pretraining to share character-context parameters across external-source classification tasks.
Source: §1 · p. 2Results
15.4% relative error reduction; the model gives results highly competitive to the best systems on six benchmarks.
Source: §6 · p. 9Takeaways & Limitations
The results show that rich external-resource pretraining can improve neural word segmentation across six benchmarks.
Source: §6 · p. 9The study did not investigate lexicons because differing OOV coverage across training and test data makes comparisons difficult unless methods use a single shared lexicon.
Source: §5.3 · p. 9
Abstract
from arXiv · showhide
Neural word segmentation research has benefited from large-scale raw texts by leveraging them for pretraining character and word embeddings. On the other hand, statistical segmentation research has exploited richer sources of external information, such as punctuation, automatic segmentation and POS. We investigate the effectiveness of a range of external training sources for neural word segmentation by building a modular segmentation model, pretraining the most important submodule using rich external sources. Results show that such pretraining significantly improves the model, leading to accuracies competitive to the best methods on six benchmarks.
1 Introduction
Neural segmentation had reached accuracies comparable to the best statistical models, while richer external information had not been systematically investigated for neural segmentation. This work pretrains a modular segmentor’s five-character context network with external sources and reports competitive benchmark results.
1 Introduction
- Earlier neural segmentation work used character, character-bigram, and word representations, as well as MLP or LSTM context encoders and structured inference methods.The passages cite CRFs for character labeling and structural beam search for word-based segmentors.Source: §1 · p. 1
- The study addresses the gap that rich external information used in statistical segmentation had not been systematically investigated for neural segmentation.Examples of such resources in prior statistical work include punctuation, automatic segmentation, POS, and differing segmentation standards.Source: §1 · pp. 1–2
- The model uses a modular globally optimized beam-search framework, pretraining its five-character context network through auxiliary classification tasks that share network parameters.The pretrained network initializes the corresponding module in the segmentor.Source: §1 · p. 2
- The method outperforms the best statistical and neural segmentation models on six benchmarks and reports the best results on five datasets across different domains and genres.The paper describes this as consistent across the six benchmarks.Source: §1 · p. 2
2 Related Work
Earlier statistical segmentation work used semi-supervised raw-text clues and heterogeneous annotations, while neural work had used embedding pretraining and varied context and inference architectures. This paper distinguishes its neural integration of external knowledge through shared parameters.
2 Related Work
- Prior statistical approaches included CRF character labeling, max-margin structured models with word features, and semi-supervised methods for both character- and word-based models.The paper situates its work within recent neural word segmentation.Source: §2 · p. 2
- Statistical segmentation work used raw-text clues and heterogeneous annotations, including POS and segmentation under different standards.Raw-text methods explored mutual information, punctuation, and self-predictions.Source: §2 · p. 2
- Unlike Sun and Xu (2011), which integrated external knowledge as features in a statistical model, this work integrates it through shared neural-network parameters.The two approaches report a similar degree of error reduction using external data.Source: §2 · p. 2
- The model uses word context on top of character context but finds word LSTM unnecessary, enabling a simpler, modular design with a pretrainable five-character window.The comparison is with Zhang et al. (2016b) and Cai and Zhao (2016).Source: §2 · pp. 2–3
3 Model
The segmentor incrementally scores transitions over recognized words, a partial word, and incoming characters. Its representation network combines character and word context, while pretraining shares a five-character window representation across external-data classification tasks.
3 Model
- At each step, the model appends the next character to the partial word or separates it to begin a new word, using states consisting of recognized words, a partial word, and remaining characters.The initial state has no recognized words and an empty partial word; processing ends when no characters remain and the partial word is empty.Source: §3 · p. 3
- A neural network scores each state by summing incremental decision scores, allowing the global structural model to disambiguate sequences of interdependent transitions.The scoring network derives representations for recognized words, the partial word, and incoming characters before scoring the next action.Source: §3 · p. 3
3.1 Representation Learning
- Character context is represented with a five-character window or a bidirectional LSTM; the window supplies local context that may be combined with LSTM context.Character embeddings and character-bigram embeddings are considered, while the partial word uses its first and last characters and length; word context can use a two-word window or a unidirectional LSTM.Source: §3.1 · pp. 3–4
3.2 Pretraining
- Pretraining uses punctuation prediction and automatically segmented text to classify B/M/E/S labels, plus heterogeneous labeled treebanks for the same segmentation labels and POS prediction for the word containing the center character.These tasks pretrain the five-character window network before its parameters initialize the corresponding segmentor module.Source: §3.2 · pp. 4–5
- Multitask pretraining shares the character-window representation across classification tasks, sampling punctuation, automatic-segmentation, heterogeneous, and POS sources at a 10/1/1/1 ratio.For each punctuation-corpus sentence, only the two characters before and after the punctuation are used as input instances.Source: §3.2 · p. 5
4 Decoding and Training
The segmentor uses global transition-based learning with beam search, maintaining the best partial hypotheses as it decodes. Training applies max-margin updates when the gold sequence leaves the beam or, if it remains, at sentence end.
4 Decoding and Training
- At each step, beam search expands all agenda hypotheses and retains the B highest-scored resulting hypotheses for the next step.The agenda starts with only the start state; the decoding strategy follows Zhang and Clark (2011).Source: §4 · p. 5
- When the gold action sequence falls out of the agenda, training uses the current best hypothesis as a negative example and the gold sequence as a positive example.This triggers a max-margin update.Source: §4 · pp. 5–6
- The loss uses the number of incorrect local decisions in the competing hypothesis, while η controls the score margin.Source: §4 · p. 6
- If the gold hypothesis remains in the agenda through the full sentence, a final update contrasts the highest-scored non-gold hypothesis with gold using the same loss.The early-update strategy is attributed to Collins and Roark (2004).Source: §4 · p. 6
- Adagrad optimizes the parameters, while L2 regularization and input dropout are used to reduce overfitting.Character and character-bigram embeddings are fine-tuned, but word embeddings are not.Source: §4 · p. 6
5 Experiments
The experiments test contextual representations, structured search, and rich pretraining on CTB6 and additional Chinese segmentation datasets. Pretraining improves development performance, and the final model achieves strong results across datasets, with an exception on MSR.
5.1 Experimental Settings
- CTB6, SIGHAN 2005, and Weibo were used to evaluate the model, with standard splits for the additional test datasets and external corpora for pretraining.Pretraining sources included automatically segmented Chinese Gigaword and a five-month People’s Daily corpus for heterogeneous segmentation and POS data.Source: §5.1 · p. 6
5.2 Development Experiments
- Development experiments examined context representations, network configurations, and pretraining methods.Source: §5.2 · p. 6
5.2.1 Context Representations
- Without character information, F-score was 84.62%; a five-character window reached 95.41%, while a 2-word context reached 95.86% and word LSTM added no improvement.Combining character window and LSTM representations improved results further; removing character or character-bigram embeddings lowered F-score to 95.20% or 94.27%, respectively.Source: Table 4 · p. 7, §5.2.1 · p. 7, Table 5 · p. 7
5.2.2 Stuctured Learning and Inference
- Larger beams resolve more global structural ambiguities, but gains shrink as beam size increases; beam size 8 was selected to trade off speed and accuracy.Source: §5.2.2 · p. 7, Figure 4 · p. 8
5.2.3 Pretraining Results
- On development data, punctuation pretraining raised F-score from 95.86% to 96.25% (9.4% relative error reduction), and multitask integration of all sources reached 96.48% (15.0% reduction).Automatic segmentation, heterogeneous segmentation, and POS pretraining individually reached 96.26%, 96.27%, and 96.22%, respectively.Source: §5.2.3 · pp. 7–8, Table 6 · p. 8
5.2.4 Comparision with Zhang et al. (2016b)
- ZPar-only automatic segmentation yielded 96.02%, below outputs matched between ZPar and a CRF segmentor; unlike Zhang et al. (2016b), this model fine-tunes character embeddings.Figure 5 plots F1 by sentence length.Source: §5.2.4 · p. 8, Figure 5 · p. 8
5.3 Final Results
- The model achieved the best reported F-score on CTB6 and best accuracies on all evaluated corpora except MSR, where it trailed Zhang et al. (2016b) by 0.2%.On CTB6, multitask pretraining raised F-score from 95.44% to 96.21% (16.9% relative error reduction). Lexicons were not investigated because differing OOV coverage complicates fair comparisons without a shared lexicon.Source: §5.3 · pp. 8–9, Table 7 · p. 9, Table 8 · p. 9
6 Conclusion
The paper uses multi-task pretraining on rich external resources in a globally optimized beam-search model that leverages character and word contexts. Rich pretraining reduces relative error by 15.4%, with results highly competitive on six benchmarks.
6 Conclusion
- 15.4% relative error reduction from rich pretraining, with results highly competitive to the best systems on six benchmarks.The model uses a globally optimized beam-search framework leveraging character and word contexts.Source: §6 · p. 9
- The model uses globally optimized beam search and leverages both character and word contexts.Source: §6 · p. 9
- Each type of external resource serves as an auxiliary classification task in neural multi-task learning to pretrain shared character-context parameters.Source: §6 · p. 9