PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated essay scoring with string kernels and word embeddings

Mădălina Cozma, Andrei M. Butnaru, Radu Tudor Ionescu

arXiv:1804.07954v2cs.CL

TL;DR

Automatic essay scoring must account for linguistic features ranging from character patterns to semantics. This paper combines string kernels with bag-of-super-word-embeddings and reports the best performance on ASAP in both in-domain and cross-domain settings, surpassing recent deep-learning approaches.

  • Problem

    AES requires capturing grammar, semantics, pragmatics, and discourse, while string kernels alone do not cover aspects such as semantics and discourse.

    Source: §1 · p. 1

  • Method

    The approach combines character n-gram string kernels with BOSWE, a semantic word-embedding representation, and trains using summed kernel matrices.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    String kernels alone and in combination with word embeddings attain the best AES performance in both in-domain and cross-domain comparisons, outperforming recent deep-learning approaches.

    Source: §4 · p. 5

  • Takeaways & Limitations

    The reported comparisons support shallow string-kernel approaches, alone or combined with word embeddings, as top-performing methods on the evaluated AES task.

    Source: §4 · p. 5

    The dual-space approach with the histogram intersection kernel prevents error analysis using discriminant features weighted by ν-SVR because the dual weights cannot be transformed into primal weights.

    Source: §3 · p. 5

Abstract

from arXiv · show

In this work, we present an approach based on combining string kernels and word embeddings for automatic essay scoring. String kernels capture the similarity among strings based on counting common character n-grams, which are a low-level yet powerful type of feature, demonstrating state-of-the-art results in various text classification tasks such as Arabic dialect identification or native language identification. To our best knowledge, we are the first to apply string kernels to automatically score essays. We are also the first to combine them with a high-level semantic feature representation, namely the bag-of-super-word-embeddings. We report the best performance on the Automated Student Assessment Prize data set, in both in-domain and cross-domain settings, surpassing recent state-of-the-art deep learning approaches.

1 Introduction

AES assigns grades to educational essays automatically, but is challenging because it involves grammar, semantics, pragmatics, and discourse. The paper combines character n-gram string kernels with semantic word embeddings and reports better performance than recent state-of-the-art approaches on ASAP in both in-domain and cross-domain settings.

1 Introduction

  • AES is challenging because it requires handling grammar, semantics, pragmatics, and discourse, while traditional systems typically relied on handcrafted features and recent deep-learning methods achieved better performance.Source: §1 · p. 1
  • The proposed approach combines character n-gram string kernels with high-level semantic word-embedding features, reporting better AES performance than recent state-of-the-art approaches on ASAP in both evaluation settings.The authors describe this as the first use of string kernels for AES and the first successful combination of string kernels and word embeddings.Source: §1 · p. 1
  • String kernels alone do not cover AES aspects such as semantics and discourse, motivating their combination with bag-of-super-word-embeddings (BOSWE).Source: §1 · p. 1

2 Method

The method combines character n-gram string-kernel similarities with a semantic document representation built from clustered word embeddings. It sums their kernel matrices and trains ν-SVR for essay-score regression.

String kernels

  • The intersection string kernel measures text similarity through shared character n-grams, using the minimum occurrence count for each n-gram across the two strings.AES is treated as regression and trained with ν-SVR.Source: §2 · p. 2

Bag-of-super-word-embeddings

  • BOSWE adapts bag-of-visual-words to text by clustering word embeddings into semantic super-word vectors and representing each document with their occurrence counts.The model uses a histogram intersection kernel and ν-SVR, matching the string-kernel approach.Source: §2 · p. 2

Model fusion

  • The HISK and BOSWE kernel matrices are summed before learning, equivalent to concatenating their feature representations and expanding the linear-pattern search space.The combined kernel is used with ν-SVR to find a regression function.Source: §2 · pp. 2–3

Data set

  • Table 1 describes essay counts and score ranges for each of the eight prompts in the ASAP data set.Source: Table 1 · p. 3

3 Experiments

On ASAP, the combined string-kernel and BOSWE approach performs strongly in both in-domain and cross-domain scoring, though the combination does not always beat string kernels alone in cross-domain settings with few target essays. The experiments use training data only and compare against prior systems under established in-domain and cross-domain evaluation procedures.

Evaluation procedure

  • In-domain evaluation uses 5-fold cross-validation repeated 10 times, while cross-domain evaluation uses source→target pairs 1→2, 3→4, 5→6, and 7→8 with target samples nt = {10, 25, 50, 100}, plus nt = 0.Only the ASAP training data are used because the official competition test data are not public.Source: §3 · p. 3

Baselines / Implementation choices

  • The experiments compare with handcrafted-feature and deep-feature systems; HISK uses character n-grams from 1 to 15, and BOSWE is combined with HISK by summing their kernel matrices.The combined kernel is trained with dual ν-SVR; BOSWE uses 500 clusters and L1 normalization.Source: §3 · pp. 3–4

In-domain results / Cross-domain results

  • 0.785 average QWK for HISK and BOSWE is more than 2% above the best-performing state-of-the-art averages; the combination is best on 7 of 8 prompts and improves every cross-domain pair over both comparators.HISK alone reaches 0.780 overall, above the reported inter-human agreement of 0.754.Source: §3 · p. 4, Table 2 · p. 4, Table 3 · p. 5

Cross-domain results

  • In cross-domain scoring, the combined model does not always outperform string kernels alone when the target-domain training sample is 25 essays or fewer.For pair 5→6 with nt = 0, the improvement over Phandi et al. (2015) is more than 54%, from 0.187 to 0.728.Source: §3 · p. 4

Discussion

  • Averaging word vectors and their component-wise standard deviations produced 600 features but only around 0.251 overall QWK, so the authors chose BOSWE instead.This preliminary result is not included in the paper’s main experiments.Source: §3 · pp. 4–5
  • The authors could not analyze highly weighted discriminant features because HISK has no explicit embedding map, and propose the presence-bits kernel as a future alternative for that analysis.Source: §3 · p. 5

4 Conclusion

The paper combines string kernels and word embeddings for automatic essay scoring and evaluates the approach on ASAP in in-domain and cross-domain settings. The comparative studies report strong performance from string kernels alone and in combination, including better results than recent deep-learning approaches.

  • The approach combines low-level string-kernel features with high-level semantic word-embedding features for automatic essay scoring.Source: §4 · p. 5
  • Across the in-domain and cross-domain comparisons, string kernels alone and combined with word embeddings attain the best reported AES performance.Source: §4 · p. 5
  • The authors report better results than recent deep-learning approaches using a shallow approach.Source: §4 · p. 5

Related papers

Loading 1804.07954v2…