PaperlayerView exact revision on arXiv

Source-linked AI summary

A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction

Shamil Chollampatt, Hwee Tou Ng

arXiv:1801.08831v1cs.CL

TL;DR

The paper targets automatic correction of grammatical, spelling, and collocation errors, where prior neural approaches lagged behind state-of-the-art SMT. It develops a multilayer convolutional encoder-decoder with attention, pretrained embeddings, ensembling, and rescoring, and reports new state-of-the-art GEC performance over prior neural and SMT systems.

  • Problem

    Prior neural GEC systems lagged substantially behind state-of-the-art SMT, while SMT had limitations in accessing global context and generalizing from discrete word and phrase representations.

  • Method

    The paper uses a multilayer fully convolutional encoder-decoder with attention, BPE tokens, pretrained embeddings, model ensembling, and language-model and edit-feature rescoring.

  • Results

    The approach achieves state-of-the-art GEC performance and reaches 54.13 F0.5 with web-scale language-model rescoring, exceeding the previous published 53.14 F0.5 result.

  • Takeaways & Limitations

    The analysis reports that convolutional models capture local context more effectively than recurrent models, while hierarchical convolutions and attention capture wider context.

Abstract

from arXiv · show

We improve automatic correction of grammatical, orthographic, and collocation errors in text using a multilayer convolutional encoder-decoder neural network. The network is initialized with embeddings that make use of character N-gram information to better suit this task. When evaluated on common benchmark test data sets (CoNLL-2014 and JFLEG), our model substantially outperforms all prior neural approaches on this task as well as strong statistical machine translation-based systems with neural and task-specific features trained on the same data. Our analysis shows the superiority of convolutional neural networks over recurrent neural networks such as long short-term memory (LSTM) networks in capturing the local context via attention, and thereby improving the coverage in correcting grammatical errors. By ensembling multiple models, and incorporating an N-gram language model and edit features via rescoring, our novel method becomes the first neural approach to outperform the current state-of-the-art statistical machine translation-based approach, both in terms of grammaticality and fluency.

Introduction

The paper addresses grammatical error correction beyond grammar alone, proposing a convolutional encoder-decoder approach to overcome limitations of prior neural and SMT systems.

  • GEC corrects grammatical, spelling, and collocation errors, addressing the growing need for improved authoring tools for non-native English writers.
  • Prior neural encoder-decoder systems performed substantially below state-of-the-art SMT approaches, which also struggled with global context and discrete word representations.
  • The proposed model uses a multilayer fully convolutional encoder-decoder with attention to capture local context and interactions between distant words.
  • BPE preprocessing, pretrained embeddings from larger English corpora, an N-gram language model, and edit features strengthen the model and its rescoring stage.
  • The paper reports state-of-the-art GEC performance and compares attention mechanisms and error-type strengths against recurrent models and SMT.

Related Work

Related work established SMT as the leading GEC framework while earlier neural systems used recurrent or hybrid architectures with varied data, language-model, and editing features.

  • SMT became state of the art by correcting varied and complex error patterns and incorporating large error-corrected parallel corpora.
  • Earlier neural GEC systems commonly used recurrent encoder-decoder architectures, including bidirectional RNNs, character-level recurrent models, and word-level bidirectional LSTMs.
  • Prior systems differed in training resources and feature tuning, including non-public CLC data, grid-search rescorers, and edit-operation tags.
  • 39.97 F0.5 was achieved by a character-level recurrent model on CoNLL-2014, improving to 40.56 after supervised edit and pretrained embedding features.
  • 45.15 F0.5 was achieved by a hybrid word-character model after adding a web-scale language model during rescoring.

A Multilayer Convolutional Encoder-Decoder Neural Network

The model uses a seven-layer convolutional encoder and decoder with attention to transform erroneous source text into corrected output. It combines BPE-based and character N-gram-informed embeddings, beam-search ensembling, and feature-based rescoring.

  • Architecture: Seven encoder and seven decoder layers form the multilayer convolutional architecture, with Figure 1 detailing one layer of each.The encoder and decoder are made up of L layers each, and the caption specifies seven layers in the described model.
  • Encoder: Source tokens combine word and position embeddings before entering convolutional encoder layers with residual connections.The source representation is linearly mapped into the encoder dimension, while position embeddings are added to word embeddings.
  • Encoder: Each encoder layer applies width-three convolutions, gated linear units, and padding to preserve one output vector per source token.The first encoder layer uses 2h filters of dimension 3 × h, followed by GLU nonlinearities.
  • Decoder: Decoder layers apply convolutions and nonlinearities to previous decoder states, while layer-specific attention uses encoder outputs and source embeddings.Adding source embeddings to the attended encoder representations helps retain information about source tokens.
  • Representations: Character N-gram-informed pre-trained embeddings and BPE preprocessing support rare-word handling in the parallel training corpus.Word embeddings represent words as bags of character N-grams, while rare words are split into BPE sub-word units.
  • Decoding: Beam search produces correction hypotheses, and averaging predictions from differently initialized models performs decoding-time ensembling.The highest-scoring beam candidate becomes the correction hypothesis.
  • Rescoring: A MERT-trained log-linear rescorer combines model scores with edit-operation and Common Crawl 5-gram language-model features.The rescoring features include token substitutions, deletions, insertions, a 5-gram score, and hypothesis length.

Experimental Setup

The study evaluates grammatical error correction on CoNLL-2014 and fluency on JFLEG, comparing the models with prior neural and state-of-the-art SMT systems. It also describes the training setup, model initialization, and rescoring features.

  • CoNLL-2014 evaluation uses the official test set and MaxMatch F0.5 scoring, with ablations conducted on CoNLL-2013.
  • JFLEG evaluation measures output fluency with GLEU and also calculates F0.5 after automatically extracting error annotations.JFLEG contains fluency-based rewrites by native writers.
  • The models use 500-dimensional source and target embeddings, 30K BPE-token vocabularies, and fastText pre-training on Wikipedia.Pre-training uses one skip-gram pass with a window size of 5.
  • Training applies 0.2 dropout across embeddings, convolution layers, and decoder output, with validation after each epoch, early stopping, and F0.5-based model selection.Models are trained on three Titan X GPUs with batch size 32 per GPU.
  • The systems are compared with prior neural approaches and two state-of-the-art SMT systems using task-specific features and large language models.One SMT system additionally uses an adapted neural network joint model and character-level spelling correction.

Experiments and Results

Experiments show that multilayer convolutional models achieve strong grammaticality and fluency, with performance further improved by ensembling, rescoring, and fastText initialization. Architectural analysis attributes MLConv’s advantage over BiLSTM partly to broader context capture and attention over surrounding words.

  • Grammaticality: 50.70 F0.5 is achieved on CoNLL-2014 with fastText initialization, a four-model ensemble, and edit-operation rescoring.This configuration uses pre-trained fastText embeddings before decoding and rescoring.
  • Grammaticality: 54.13 F0.5 with web-scale language-model rescoring exceeds the previous published 53.14 F0.5 result.The improvement is statistically significant at p < 0.001.
  • Fluency: On JFLEG, edit-operation rescoring outperforms the comparison system on both development and test datasets and metrics, while the full method achieves the best reported scores.The reported full method adds the web-scale language model and spell checker, and does not require tuning on the JFLEG development set.
  • Encoder and Decoder Architecture: MLConv outperforms SLConv and BiLSTM on CoNLL-2013 by capturing a 21-token surrounding context through seven convolutional layers with filter width 3.The comparison excludes pre-trained word embeddings.
  • Encoder and Decoder Architecture: BiLSTM has higher precision but lower recall than MLConv, whose attention assigns more probability to surrounding context and favors corrections over copying source words.The authors observed this attention pattern across the examples they examined.
  • Initialization with Pre-trained Embeddings: fastText initialization is selected because character sequences provide morphology-sensitive word representations and perform well empirically on CoNLL-2013.The study compares random, word2vec, and fastText initialization.

Analysis and Discussion

Error-type analysis on CoNLL-2014 shows that the ensembled, rescored model surpasses the SMT state of the art on several common error types, especially subject–verb agreement. The analysis attributes this strength to convolutional access to broad source and target context, while noting complementary RNN precision and future language-model extensions.

  • Error-type performance: The ensembled model with rescoring performs competitively on preposition errors and outperforms the SOTA system on noun-number, determiner, and subject-verb agreement errors.These comparisons use ERRANT on four common error types in CoNLL-2014, evaluated with F0.5.
  • Error-type performance: A single MLConvembed model without rescoring already exceeds the SOTA SMT system on subject-verb agreement errors.The comparison is based on CoNLL-2014 error-type performance.
  • Context and architecture: The model’s subject-verb agreement advantage is linked to global attention over the source and multiple decoder convolution layers providing longer target context.The cited analysis contrasts this context access with the difficulty SMT systems have when subjects and verbs are far apart.
  • Context and architecture: The convolutional encoder-decoder captures context more effectively and achieves superior results, although RNNs can provide higher precision.The paper suggests investigating combinations of convolutional and recurrent approaches.
  • Future work: Web-scale language-model integration during beam search and fusion of neural language models remain future-work directions.The paper notes that improved language modeling has previously improved GEC performance considerably.

Conclusion

The paper concludes that multilayer convolutional encoder-decoder models substantially improve grammatical error correction over prior neural approaches. Pretrained embeddings, language-model and edit-feature rescoring, model ensembling, and evaluation on CoNLL-2014 and JFLEG together close the earlier gap with statistical systems.

  • Conclusion: The multilayer convolutional encoder-decoder achieves significant improvements over all previous encoder-decoder neural approaches.The conclusion presents this as a central outcome for grammatical error correction.
  • Conclusion: Large English corpora support pretrained word embeddings and a language model used to rescore candidate corrections.Edit-operation features are also incorporated during rescoring.
  • Conclusion: Ensembling multiple neural models and rescoring improve performance on both CoNLL-2014 and JFLEG while significantly outperforming leading SMT-based systems.The conclusion states that the method closes the previously large performance gap between neural and statistical approaches.
Loading 1801.08831v1…