Source-linked AI summary
Adversarial Multi-Criteria Learning for Chinese Word Segmentation
Xinchi Chen, Zhan Shi, Xipeng Qiu, Xuanjing Huang
TL;DR
Chinese word segmentation corpora use heterogeneous criteria, while existing neural models do not directly handle these incompatibilities simultaneously. The paper introduces adversarial multi-criteria learning with shared and private representations, and experiments across eight corpora report significant improvements over single-criterion methods. The study also identifies a scope boundary: shared features can hurt relatively large datasets such as AS.
Problem
Heterogeneous segmentation criteria make CWS corpora incompatible, while most neural models cannot handle them simultaneously.
Method
The paper treats each criterion as a task and combines shared criterion-invariant layers, private criterion-specific layers, and adversarial training.
Results
Significant improvements over single-criterion methods are achieved across eight CWS corpora with different segmentation criteria.
Takeaways & Limitations
Shared information across heterogeneous criteria can improve CWS performance, while traditional Chinese can benefit from simplified-Chinese knowledge.
Takeaways & Limitations
Shared features may hurt relatively large datasets such as AS because they can bias toward other datasets and introduce noisy features.
Abstract
from arXiv · showhide
Different linguistic perspectives causes many diverse segmentation criteria for Chinese word segmentation (CWS). Most existing methods focus on improve the performance for each single criterion. However, it is interesting to exploit these different criteria and mining their common underlying knowledge. In this paper, we propose adversarial multi-criteria learning for CWS by integrating shared knowledge from multiple heterogeneous segmentation criteria. Experiments on eight corpora with heterogeneous segmentation criteria show that the performance of each corpus obtains a significant improvement, compared to single-criterion learning. Source codes of this paper are available on Github.
1 Introduction
Chinese word segmentation is important but requires expensive annotation, and heterogeneous corpora use incompatible criteria. The paper proposes adversarial multi-criteria learning to share criteria-invariant knowledge across these resources.
- Chinese word segmentation is an important NLP task whose supervised methods rely on expensive large-scale annotated corpora.
- Different segmentation criteria make major CWS corpora incompatible, limiting direct reuse of their annotations.The introduction illustrates this incompatibility with PKU and CTB segmentations of the same sentence.
- The paper proposes adversarial multi-criteria learning that integrates shared knowledge from multiple heterogeneous segmentation criteria.
- The approach treats each segmentation criterion as a task, combining shared criteria-invariant layers with private criteria-specific layers.
- Experiments use eight CWS corpora spanning five simplified-Chinese and three traditional-Chinese datasets.
2 General Neural Model for Chinese Word Segmentation
The general neural CWS model represents characters, extracts bidirectional sequence features, and infers B/M/E/S tags with a CRF layer. Its architecture combines embedding, neural feature, and structured tag-inference components.
- CWS is formulated as character-based sequence labeling with B, M, E, and S tags marking word boundaries.
- The general neural architecture contains character embeddings, neural feature layers, and a tag-inference layer.
- 2.1 Embedding layer: The embedding layer maps each character x_i to a distributed vector e_xi using an embedding matrix.
- 2.2 Feature layers: Bi-LSTM feature layers process the sequence in forward and backward directions to incorporate information from both sides.
- 2.3 Inference Layer: A first-order linear-chain CRF infers tags from feature scores, modeling interactions between successive labels.
3 Multi-Criteria Learning for Chinese Word Segmentation
The paper treats heterogeneous CWS criteria as related tasks and uses shared-private architectures to combine criteria-invariant and criterion-specific information. Three models differ in how information flows between shared, private, and CRF task layers.
- Heterogeneous segmentation criteria are treated as related tasks because shared information can improve each criterion simultaneously.
- Three shared-private models separate feature extraction into criterion-invariant shared layers and criterion-specific private layers while sharing embeddings.
- Model-I: Model-I places the private and shared layers in parallel before computing the CRF score function.
- Model-II: Model-II stacks the private layer after the shared layer before computing the CRF score function.
- Model-III: Model-III stacks the private layer after the shared layer and also sends shared-layer outputs directly to the CRF layer.
4 Incorporating Adversarial Training for Shared Layer
The adversarial strategy trains the shared layer to represent features that are common across heterogeneous criteria. A criterion discriminator predicts the annotation criterion from averaged shared features, while shared-layer training maximizes prediction entropy.
- Shared-private separation alone does not guarantee that shared and criterion-specific features remain confined to their intended spaces.
- A criterion discriminator recognizes the annotation criterion from shared features, while adversarial training discourages criterion-specific information in the shared space.
- The discriminator averages shared hidden states and applies a linear transformation followed by softmax to obtain probabilities over criteria.
- The discriminator maximizes cross-entropy against the true criterion, whereas shared-parameter training maximizes the entropy of the predicted criterion distribution.
- The method uses an entropy term instead of the negative cross-entropy used by Ganin et al.
5 Training
Training combines the CWS task objective with the adversarial objective and alternates optimization of the task predictor and criterion discriminator. After adversarial pretraining, the shared parameters are fixed while the task objective is optimized to convergence.
- The task and adversarial objectives are combined with λ controlling the interaction between their loss terms.
- The two discriminative classifiers are optimized alternately using minibatch Adam updates.
- Adversarial training first runs for 2400 epochs, with each epoch using eight batches from different corpora.
- After adversarial training, only the CWS objective is optimized, with shared parameters fixed until convergence using early stopping.
6 Experiments
Experiments evaluate the proposed models on eight heterogeneous CWS datasets, comparing baseline, multi-criteria, and adversarial variants. Multi-criteria learning improves results overall, while adversarial training further boosts performance and helps correct criterion-specific errors.
- Experimental setup: The evaluation uses eight CWS datasets with different segmentation criteria, including five simplified-Chinese and three traditional-Chinese corpora.A shuffled 10% of each training set is used for development.
- Overall results: Multi-criteria learning improves the averaged F-measure by 0.75% over the Bi-LSTM baseline of 94.14%, with only MSRA dropping slightly.The proposed models exploit information shared across heterogeneous segmentation criteria.
- Overall results: Adversarial training further boosts performance across datasets, with Model-I slightly outperforming Model-II and Model-III.The strategy aims to keep shared-layer features criterion-invariant; its additional boost is reported as not significant.
- Speed: After about 1000 epochs, Model-I without adversarial training becomes stable and convergent on all eight development sets.The learning curves show gradual progress across datasets.
- Speed: The models process 441.38 sentences per second on average, while adversarial training increases training time from about 10 to about 16 hours.The proposed and baseline models have nearly the same computational complexity and decoding efficiency.
- Error analysis: Adversarial training corrects a large proportion of personal-name segmentation mistakes caused by heterogeneous segmentation granularities.The paper illustrates these corrections with examples from PKU and MSRA test sentences.
7 Knowledge Transfer
The paper tests whether shared features transfer across segmentation domains. Features learned from simplified Chinese help traditional Chinese, and features learned from formal texts improve micro-blog segmentation.
- Simplified Chinese to Traditional Chinese: Shared features trained on five simplified-Chinese datasets improve average F-measure on three traditional-Chinese datasets by 0.41%, from 93.78% to 94.19%.The shared parameters are trained on simplified-Chinese data and then fixed for the traditional-Chinese datasets.
- Simplified Chinese to Traditional Chinese: The traditional-Chinese transfer setting can hurt the relatively large AS dataset because shared parameters may bias toward other datasets.AS contains 5.4M training tokens.
- Formal texts to informal texts: On the NLPCC 2016 micro-blog dataset, transfer from eight formal-text datasets raises F-measure by 0.30%, from 93.94% to 94.24%.The OOV recall rate also increases by 3.97%.
8 Related Works
Prior work exploits heterogeneous annotation data through stacking, multi-task, or coupled sequence-labeling models, but often relies on shallow classifiers. This paper instead uses shared-private neural architectures and adversarial training across eight heterogeneous corpora.
- Prior approaches: Earlier methods use stacking, multi-task learning, or coupled sequence labeling to exploit heterogeneous annotations across NLP tasks.These approaches include unidirectional and structured information-sharing models.
- Prior approaches: Many earlier approaches rely on shallow classifiers, making shared feature-space design difficult.The paper positions deep neural networks as a way to share information through hidden layers.
- This paper: The proposed work differs by combining three shared-private architectures, adversarially learned criterion-invariant features, and eight heterogeneous CWS corpora.The shared-private design preserves both shared and criterion-specific information.
9 Conclusions & Future Works
The paper concludes that adversarial multi-criteria learning can exploit shared knowledge across heterogeneous CWS criteria and significantly improve performance over single-criterion methods.
- Conclusion: The three shared-private models effectively extract shared information and achieve significant improvements over single-criterion methods.The conclusion frames the contribution as adversarial multi-criteria learning for CWS.