Compositional Sequence Labeling Models for Error Detection in Learner Writing

• Error detection is treated independently from correction because correction-based evaluation assumes systems can propose a correction for every error, and correction quality may not match detection quality.Source: §1 · p. 1
• Common n-gram and context-pattern approaches face data sparsity, treat words independently, and rely on fixed context sizes that limit context to a small window.Source: §1 · p. 1
• The paper introduces neural models for error detection, compares compositional structures, and proposes a framework that labels each token using the full sentence.The model labels tokens as correct or incorrect in context.Source: §1 · p. 1
• Integration with a publicly deployed self-assessment system led to performance comparable to human annotators.Source: §1 · p. 2

• Earlier detection studies often focused on specific errors, including prepositions, articles, verb forms, and adjective-noun pairs.Source: §2 · p. 2
• General detection systems existed, but shared-task systems were often error-type specific and the strongest approach favored article and preposition errors.Source: §2 · p. 2
• Correction approaches included phrase-based machine translation and classifiers using native-language correction priors.Source: §2 · p. 2
• Detection can use a wider range of tagging and classification models, and feedback locating errors may be pedagogically preferable to correcting every error.Source: §2 · p. 2

3 Sequence Labeling Architectures

• Each architecture maps token embeddings to context-dependent representations, then uses a softmax layer to predict a label distribution for every token.The framework labels each token as correct or incorrect in context.Source: §3 · p. 2, Figure 1 · p. 3
• Convolutional networks use a fixed context window to detect n-gram patterns, while deep convolutional networks add a layer to capture more complex higher-order features.The convolution uses surrounding tokens with their ordering preserved.Source: §3 · pp. 2–3
• Elman-type recurrent networks pass information recursively through the sentence, learning which patterns to disregard or pass forward instead of using a fixed context window.The recurrent hidden representation depends on the current token embedding and the previous hidden vector.Source: §3 · p. 3
• Bidirectional recurrent networks build separate representations from left and right context and concatenate them, letting predictions use context on both sides of the target token.Deep bidirectional RNNs feed one bidirectional layer’s representations into another to learn higher-level features through multiple recurrent passes.Source: §3 · p. 3
• LSTMs use input and forget gates to update an internal cell state, and their linear update makes them less susceptible to vanishing gradients over time.The paper applies this architecture bidirectionally and also experiments with two consecutive bidirectional LSTM layers.Source: §3 · p. 4
• The comparison also includes a CRF trained with unigram, bigram, and trigram features in a seven-word window, with each label conditioned on the previous label.The window covers three words before and after the target.Source: §3 · p. 4

Experimental setup

• Evaluation uses the FCE-public test set of 2,720 sentences, with 30,953 sentences left for training and 2,222 held out for development.The texts are learner responses to exam prompts and are manually annotated using a taxonomy of 77 error types.Source: §4 · p. 4

Task formulation

• Tokens inside annotated error spans are labeled incorrect; for missing-word errors, the token immediately after the gap receives the error label.This convention treats that token as incorrect in context because another token should have preceded it.Source: §4 · p. 4

Evaluation

• F0.5 is the main evaluation measure, weighting precision twice as heavily as recall because accurate feedback is often more important than coverage.The study also reports raw counts of predicted and correct tokens; correction-dependent measures are not directly applicable to detection.Source: §4 · pp. 4–5

Implementation

• The experiments use 300-dimensional pretrained Word2Vec embeddings and select the model with the best development-set F0.5 for final evaluation.Inputs are lowercased, rare tokens map to unk, and optimization uses ADAM with batch size 64.Source: §4 · p. 5

Architecture comparison

• The CRF has the lowest F0.5 among the neural models, combining high precision on frequent error sequences with insufficient recall.Conditioning on the previous label provides little help when there are only two labels and errors are relatively sparse.Source: §5 · p. 5
• The convolutional network achieves the second-highest test result and outperforms the Bi-RNN despite using a fixed window of seven context words.The Bi-RNN’s distant context signal can diminish, which may help explain its highest precision but lower overall performance.Source: §5 · p. 5
• The bidirectional LSTM achieves the highest overall F0.5 by using full-sentence context and a composition function designed to capture longer-distance dependencies.Its cell updates and gates help select information for hidden representations and error-detection output.Source: §5 · p. 5

Depth comparison

• Deeper networks give comparable or slightly lower results across the composition architectures.The authors contrast this with prior Bi-RNN gains on opinion mining, where longer semantic target sequences may benefit from additional layers.Source: §5 · pp. 5–6

Data sources

• The training-data experiments incrementally add corpora to the Bi-LSTM, with FCE-public, NUCLE, IELTS, FCE, CPE, and CAE represented in the dataset descriptions.The IELTS subset contains essays from all proficiency levels; the additional FCE texts come from the Cambridge Learner Corpus.Source: §6 · p. 6
• The additional CPE and CAE corpora comprise essays from proficient and advanced examination levels, respectively.They contain 210,678 and 219,953 sentences, respectively.Source: §6 · p. 6

Training-data results

• Adding NUCLE does not improve performance over FCE-public alone, which the authors attribute to differences in domain and writing style.FCE contains young intermediate learners responding to letters, emails, and reviews; NUCLE is mostly argumentative essays by advanced adult learners.Source: §6 · p. 6
• IELTS and FCE each increase F0.5 by roughly 10%, while additional CPE and CAE data yields only minor further improvements.The authors associate IELTS and FCE gains with their coverage of average errors across proficiency levels.Source: §6 · pp. 6–7
• The performance–data-size trade-off is optimal at around 8 million tokens, after the FCE dataset is added.Figure 2 plots F0.5 on the public FCE test set against total training tokens.Source: Figure 2 · p. 6, §6 · p. 7

Human agreement

• Human annotators average roughly 50% F0.5 when evaluated against one another on the two test annotations.The authors note that trained annotators can disagree on both the number and location of errors.Source: §7 · p. 7

Shared-task comparisons

• CAMB, CUUI, and AMU preserve their relative ranking in error detection; CAMB has the highest recall and F0.5 despite lower precision.AMU has slightly higher precision than CUUI.Source: §7 · p. 7
• Susanto et al.’s system ranks third and fourth on the two annotations by F0.5, despite outperforming shared-task participants on error correction.Its high-precision detections cover a small number of errors.Source: §7 · p. 7

Training-data condition

• With FCE-public alone, the Bi-LSTM performs poorly; the authors report that it labels many tokens as errors when training data is small and from a different domain.They interpret this behavior as learning correct constructions and classifying unseen structures as errors rather than merely memorizing error sequences.Source: §7 · p. 7

Shared-task results

• The Bi-LSTM trained on all datasets achieves the highest F0.5 on both CoNLL-14 test annotations, with an absolute improvement of 3% over the previous best.It is a single model for all error types, unlike the competing systems’ combinations of models or error-specific classifiers; it was also trained on more data than other contestants.Source: §7 · p. 8

8 Essay Scoring

• The extrinsic evaluation added each essay’s average token-correctness probability to the existing SAT ranking perceptron features and tested assigned-score prediction.The SAT model used FCE-public for training and evaluation on correctly predicting assigned essay scores.Source: §8 · p. 8
• 64% relative error reduction on Pearson’s correlation; with all available data, the combined system also outperformed human annotators on Spearman’s correlation.Table 4 reports Pearson’s correlation r and Spearman’s correlation ρ on the public FCE test set for automated essay scoring.Source: Table 4 · p. 8, §8 · p. 8
• Around 1% absolute improvement, corresponding to 20–31% relative error reduction with respect to human performance, came from combining the Bi-LSTM with existing SAT features using FCE-public training only.The existing SAT system already performed very close to human assessors.Source: §8 · p. 8, Table 4 · p. 8

9 Conclusions

• The proposed token-level framework uses bidirectional LSTMs and multilayer composition to classify tokens with full-sentence context; LSTM self-modulation supports longer-distance dependencies.The architecture combines token-level embeddings, bidirectional LSTM context representations, and a multi-layer architecture for learning more complex features.Source: §9 · p. 8
• The largest training-data benefit came from 8 million tokens spanning varied proficiency levels, while adding still more high-proficiency learner data yielded only marginal further improvements.The paper identifies automatically generated training data as a direction for future investigation.Source: §9 · pp. 8–9
• The neural sequence-tagging model specialized for error detection outperformed all other participating systems, while the best correction system differed from the best detection system.The best correction system was P1+P2+S1+S2, while CAMB was the best shared-task detection system.Source: §9 · p. 9
• Adding error-detection probabilities to an essay scorer further improved its near-human performance; with a larger training set, the scorer exceeded human-level performance.The model was evaluated by incorporating its probabilities as features in an essay-scoring model.Source: §9 · p. 9