PaperlayerView exact revision on arXiv

Source-linked AI summary

SkipFlow: Incorporating Neural Coherence Features for End-to-End Automatic Text Scoring

Yi Tay, Minh C. Phan, Luu Anh Tuan, Siu Cheung Hui

arXiv:1711.04981v1cs.AIcs.CL

TL;DR

ATS feature engineering can be labor-intensive, and the paper targets neural models’ limited treatment of essay flow and coherence. It introduces SkipFlow, which compares LSTM hidden-state snapshots to create auxiliary neural coherence features. On ASAP, it reports state-of-the-art performance, including improvements over LSTM and feature-engineering baselines.

  • Problem

    Handcrafted ATS features can demand substantial human involvement and laborious implementation, while the paper targets limited modeling of essay flow and coherence.

    Source: Introduction · p. 1

  • Method

    SkipFlow compares multiple LSTM hidden-state snapshots with a neural tensor layer and aggregates semantic-relatedness features as auxiliary inputs for prediction.

    Source: Introduction · pp. 1–2

  • Results

    On ASAP, SKIPFLOW LSTM achieves state-of-the-art performance, with about 10% increase over a baseline LSTM and 6% improvement over feature engineering baselines.

    Source: Our Contributions · p. 2, Conclusion · p. 8

  • Takeaways & Limitations

    SkipFlow’s additional parameters provide auxiliary memory and implicit access to intermediate states, which the paper presents as protection against vanishing gradients.

    Source: Our Contributions · p. 2, Introduction · p. 2

Abstract

from arXiv · show

Deep learning has demonstrated tremendous potential for Automatic Text Scoring (ATS) tasks. In this paper, we describe a new neural architecture that enhances vanilla neural network models with auxiliary neural coherence features. Our new method proposes a new \textsc{SkipFlow} mechanism that models relationships between snapshots of the hidden representations of a long short-term memory (LSTM) network as it reads. Subsequently, the semantic relationships between multiple snapshots are used as auxiliary features for prediction. This has two main benefits. Firstly, essays are typically long sequences and therefore the memorization capability of the LSTM network may be insufficient. Implicit access to multiple snapshots can alleviate this problem by acting as a protection against vanishing gradients. The parameters of the \textsc{SkipFlow} mechanism also acts as an auxiliary memory. Secondly, modeling relationships between multiple positions allows our model to learn features that represent and approximate textual coherence. In our model, we call this \textit{neural coherence} features. Overall, we present a unified deep learning architecture that generates neural coherence features as it reads in an end-to-end fashion. Our approach demonstrates state-of-the-art performance on the benchmark ASAP dataset, outperforming not only feature engineering baselines but also other deep learning models.

Introduction

The paper introduces an end-to-end ATS architecture that models relationships between essay positions to capture coherence and support prediction. It targets both labor-intensive feature engineering and limits in neural models’ treatment of long essays and temporal flow.

Introduction

  • Handcrafted ATS features can require substantial human involvement and laborious implementation, while existing neural models do not effectively capture relationships between multiple essay positions over time.Long essays also push the limits of LSTM memorization.Source: Introduction · p. 1
  • The model compares distant LSTM hidden-state snapshots with a neural tensor layer, aggregates semantic-relatedness features, and uses them as auxiliary prediction features.This combines sentence-level compositional reading and semantic matching in an end-to-end framework.Source: Introduction · pp. 1–2
  • The auxiliary features aim to represent logical and semantic flow, adding a measure of semantic similarity beyond the base LSTM’s compositionality.Semantic relationships across sentences are used as indicators of writing flow and textual coherence.Source: Introduction · pp. 1–2
  • SkipFlow’s additional parameters provide implicit access to intermediate states, serving as auxiliary memory and protection against vanishing gradients.The paper also reports better performance with a smaller LSTM parameterization.Source: Our Contributions · p. 2, Introduction · p. 2
  • SKIPFLOW LSTM achieves state-of-the-art performance on ASAP, with about 10% increase over a baseline LSTM and 6% improvement over feature engineering baselines.It also outperforms multi-layered and attention-based LSTMs in the same experimental configuration.Source: Our Contributions · p. 2

Related Work

Prior ATS work relied heavily on handcrafted features, while neural approaches learned features end-to-end. This paper situates its neural coherence features alongside earlier coherence measures and semantic-matching architectures.

Related Work

  • Handcrafted ATS pipelines used domain-specific lexical, syntactic, and semantic features, with custom feature design and implementation often laborious.Examples include length, grammar, readability, and coherence features.Source: Related Work · p. 2
  • Neural ATS models such as RNNs and CNNs offered competitive results without feature engineering, but the paper identifies temporal coherence as an unmodeled dimension.Prior coherence features included semantic similarity between sentences and lexical chaining (Higgins and Burstein 2007; Higgins et al. 2004; Chen and He 2013; Somasundaran, Burstein, and Chodorow 2014).Source: Related Work · p. 2
  • Unlike traditional coherence features, the paper’s neural coherence features are incorporated into an end-to-end unified framework.Earlier approaches measured sentence similarity or semantic similarity between nouns and proper nouns.Source: Related Work · p. 2
  • Semantic matching architectures used tensor layers for vector-pair similarity, while SkipFlow performs matching repeatedly as it reads and uses the resulting auxiliary features for prediction.Related tensor methods include CNTN and NTN-LSTM (Tay et al. 2017), which focus on matching two sentences, often with two networks.Source: Related Work · p. 3
  • SkipFlow’s auxiliary parameters are also framed as memory for easing the LSTM’s burden; unlike attention’s weighted global feature, its learned auxiliary features are used for prediction.The paper describes the mechanism as exposing hidden states to deeper layers, providing additional protection against vanishing gradients.Source: Related Work · p. 3

Our SKIPFLOW LSTM Model

SKIPFLOW uses an LSTM to represent an essay, then models relationships between selected hidden-state snapshots as neural coherence features. It combines these features with the essay representation for score prediction.

Embedding Layer

  • Essays are padded to a fixed maximum length, converted into low-dimensional word vectors, and read by an LSTM whose hidden outputs are temporally mean-pooled.The LSTM output at each time step represents the essay at that position.Source: Embedding Layer · p. 3, Long Short-Term Memory (LSTM) · p. 3

Skipping and Relevance Width

  • SKIPFLOW pairs sequential LSTM outputs separated by relevance width δ, starting at position i = 3 and looping to the essay’s beginning if δ exceeds its maximum length.Fixed-width matching limits preprocessing and avoids maintaining essay-specific indices for batch-wise training.Source: Skipping and Relevance Width · p. 3

Neural Tensor Layer

  • A shared neural tensor layer composes each pair of hidden vectors into a similarity score in [0, 1], with multiple tensor slices modeling different aspects of their relationship.The tensor layer’s parameters are shared across output pairs.Source: Neural Tensor Layer · pp. 3–4, Figure 1 · p. 4

Fully-connected Hidden Layer

  • The model concatenates the tensor-derived scores into a neural coherence feature vector and joins it with the temporally mean-pooled essay representation.The combined vector is passed through a fully connected hidden layer.Source: Fully-connected Hidden Layer · p. 4

Linear Layer with Sigmoid

Experimental Evaluation

The evaluation compares SKIPFLOW variants with neural and feature-engineering baselines on ASAP using prompt-specific QWK. The reported results include performance against published models and runtime and parameter comparisons.

Experimental Evaluation

  • The ASAP evaluation uses five-fold cross-validation with 60/20/20 train, development, and test splits; QWK measures agreement between raters.ASAP contains eight prompts, and scores are rescaled to prompt-specific ranges for QWK evaluation.Source: Dataset · p. 4, Experimental Setup · p. 5, Evaluation Metric · p. 5
  • The tested SKIPFLOW configurations include bilinear and tensor compositions, with tensor slices and relevance width tuned across specified values.The tensor model tests 2, 4, 6, or 8 slices and both models test relevance widths δ of 20, 50, or 100.Source: Our Models · p. 5
  • Mean pooling is more effective than using the last output; among LSTM variants, four-layer ML-LSTM performs best, while bidirectionality yields no significant improvement.ATT-LSTM performs notably better than the base LSTM.Source: Experimental Results · p. 6
  • Almost 10% higher QWK than baseline LSTM (Mean), SKIPFLOW LSTM (Tensor) outperforms the bilinear variant, while SKIPFLOW CNN does not improve CNN performance.The authors suggest the CNN’s width of 3 might be insufficient for fixed-width matching.Source: Experimental Results · p. 6
  • The compared models marked † use d = 300; SKIPFLOW LSTM (Tensor) significantly outperforms LSTM† (d = 300) by 5%, while higher dimensionality improves performance over d = 50.Source: Experimental Results · p. 6
  • About 6% over EASE, SKIPFLOW LSTM (Tensor) outperforms the best feature-engineering baseline on ASAP; Table 2 reports the compared models’ results.The comparison is reported as about 6%.Source: Table 2 · p. 6, Experimental Results · p. 7
  • SKIPFLOW remains competitive with a 20-model CNN + LSTM ensemble as a single model, and its published-results comparison is reported in Table 3.The evaluation compares single neural models rather than ensemble approaches.Source: Experimental Results · p. 6, Comparison against Published Results · p. 7, Table 3 · p. 7
  • SKIPFLOW adds only 1–2 seconds over baseline LSTM runtime and outperforms LSTM† and four-layer ML-LSTM in QWK, memory footprint, and runtime.The runtime and parameter-size comparison is for prompt 1; models are d = 50 unless stated otherwise.Source: Runtime and Memory · p. 7, Table 4 · p. 7

Effect of Hyperparameters

Performance depends on both tensor-slice count and relevance width. The reported settings favor k around 4–6 and δ from 50 to 100 with k = 4, with exceptions across prompts.

Effect of Hyperparameters

Effect of Tensor Slices on Performance

Effect of Relevance Width δ on Performance

Conclusion

SKIPFLOW LSTM uses parameterized tensor compositions to model relationships between essay positions and generate neural coherence features for scoring. It outperforms the cited LSTM and feature-engineering baselines, as well as multi-layered and attentional LSTMs.

Conclusion

  • Prompt 1 is omitted from the visualization because its much higher score distorts the graphs.Source: Conclusion · p. 7
  • Approximately 10% improvement over a baseline LSTM and 6% over feature-engineering baselines; the model also outperforms multi-layered and attentional LSTMs.Source: Conclusion · p. 8
  • SKIPFLOW LSTM uses parameterized tensor compositions to model relationships between essay positions, generating neural coherence features that support predictions.Source: Conclusion · pp. 7–8
Loading 1711.04981v1…