PaperlayerView exact revision on arXiv

Source-linked AI summary

Extraction of Salient Sentences from Labelled Documents

Misha Denil, Alban Demiraj, Nando de Freitas

arXiv:1412.6815v2cs.CLcs.IRcs.LG

TL;DR

Sentence extraction lacks a gold standard when correct extractions are unlabelled, motivating scalable evaluation. The paper introduces a hierarchical ConvNet and visualization-based extraction, and reports that it retains more task-relevant information than baseline methods, even with very few sentences.

  • Problem

    Comparing sentence-extraction systems is difficult when there are no labels for correct extractions to provide a gold standard.

    Source: §4 · p. 5

  • Method

    A hierarchical ConvNet creates sentence-level representations for visualization-based extraction, while a reference classifier compares full-document and extracted-document performance to gauge retained information.

    Source: §1 · p. 1, §4 · p. 5, §6 · p. 8

  • Results

    The ConvNet extracts more task-relevant information than baseline methods even with very few sentences; with two review sentences, reference-model test accuracy drops by less than 1.5%.

    Source: §6 · p. 8, §5.2 · p. 8

  • Takeaways & Limitations

    Comparing reference-classifier performance on full documents and extracted portions provides a scalable measure of task-relevant information preserved by extraction.

    Source: §6 · p. 8

Abstract

from arXiv · show

We present a hierarchical convolutional document model with an architecture designed to support introspection of the document structure. Using this model, we show how to use visualisation techniques from the computer vision literature to identify and extract topic-relevant sentences. We also introduce a new scalable evaluation technique for automatic sentence extraction systems that avoids the need for time consuming human annotation of validation data.

1 Introduction

The paper introduces a hierarchical ConvNet for extracting task-relevant sentences from labelled documents and a scalable way to compare extraction systems using document labels.

1 Introduction

  • The hierarchical ConvNet supports document-structure introspection, enabling visualization techniques to identify and extract relevant sentences from labelled documents.The paper demonstrates extraction of sentiment-relevant sentences from movie reviews.Source: §1 · p. 1
  • A scalable evaluation method compares a classifier’s accuracy on full documents with its accuracy on extracted portions to gauge retained task-relevant information.The approach uses labelled documents to train a simple document classifier.Source: §1 · pp. 1–2
  • Sentence extraction matters for summaries used in internet search results, review sites, and news front pages.The paper presents its work as a neural-network application to extraction and a scalable way to compare solutions.Source: §1 · p. 2

2 Model description

The model processes documents through sentence-level and document-level convolutional stages, with an intermediate representation for each sentence that makes sentence-specific relevance measurable.

2 Model description

  • Each model level transforms an embedding matrix: word embeddings become sentence embeddings, which the document-level ConvNet combines into a document embedding for classification.The model is trained by backpropagation through both levels, with tied sentence-level filter banks.Source: §2 · p. 2, Figure 1 · p. 3

2.1 Embedding matrix

  • At the sentence level, each word’s embedding is placed in a matrix column; sentence embeddings are assembled similarly into the document-level input matrix.Word embeddings are model parameters optimized by backpropagation.Source: §2.1 · pp. 2–3

2.2 Convolution

  • One-dimensional convolutions run across words at the sentence level and across sentences at the document level, producing latent representations through feature maps.The model uses wide convolutions so filter weights reach words or sentences at the edges.Source: §2.2 · pp. 3–4

2.3 k-max pooling

  • Because sentence and document lengths vary, k-max pooling keeps the k largest values in each row to produce fixed-size outputs, padding short inputs with zeros.Convolution handles variable-width inputs, but fixed-width outputs are useful at interfaces to fully connected or softmax layers.Source: §2.3 · p. 4

2.4 Full model

  • Intermediate representations for individual sentences let the model measure each sentence’s relevance contribution, supporting the extraction application.The sentence-processing ConvNet shares weights across sentences, while the document-level ConvNet has separate weights.Source: §2 · p. 2, §2.4 · p. 4
  • The model’s intermediate sentence representation is essential for extraction; convolution over the whole document would not separate individual sentence contributions.The sentence level uses one tied-weight pipeline across sentences; the document level uses the same types of convolution and pooling primitives without shared weights.Source: §2.4 · p. 4

3 Sentence extraction through visualisation

The extraction method assigns saliency scores to words and sentences using backpropagation-based visualisation. It ranks sentences by saliency and selects the top k for extraction.

3 Sentence extraction through visualisation

  • An inverted prediction label is used as the loss target to induce the greatest loss before backpropagating saliency.A forward pass produces the document prediction, which is inverted to form the pseudo-label.Source: §3 · pp. 4–5
  • Word saliency is estimated from the magnitude of the loss gradient with respect to each word, computed through a first-order Taylor approximation.The gradient magnitude indicates which words need the least change to affect the score most.Source: §3 · pp. 4–5
  • Backpropagation through the embedding layer yields derivatives with respect to word indices, while sentence-level embeddings are shared through an implicitly defined dictionary.At the document level, sentence embeddings and backpropagation messages provide the derivative used to assess each sentence.Source: §3 · p. 5
  • Sentence saliency scores rank sentences within a document, and the method extracts the k highest-ranked sentences.The scores are used to create a document saliency map and select a fixed number of sentences.Source: §3 · pp. 4–5

4 Scalable evaluation

Summarisation metrics cannot readily evaluate extractions because the paper lacks labels for correct sentence selections. The proposed scalable alternative compares a reference classifier's accuracy on full documents and extracted portions.

4 Scalable evaluation

  • Because there is no gold standard for correct extractions, standard summarisation metrics cannot directly evaluate extraction quality.The paper notes that labels for correct extractions are unavailable.Source: §4 · p. 5
  • The evaluation trains a reference model on full documents and compares its performance on full documents with its performance on extracted sentences.Table 1 covers Proportion and Fixed selection settings and includes ConvNet, Word2Vec, random, Full, and First+Last conditions.Source: §4 · pp. 5–6, Table 1 · p. 6
  • The method treats retained reference-model accuracy as an intuitive indication that extracted sentences preserve task-relevant information.Irrelevant selections are expected to confuse the reference model and reduce its accuracy.Source: §4 · p. 6

5 Experiments

On labelled IMDB movie reviews, the experiments compare ConvNet sentence extraction with several baselines and assess both retained sentiment information and the selected sentences’ relevance.

5.1 Data preparation

  • The experiment uses the 25,000 labelled IMDB reviews in the training set and 25,000 labelled reviews in the test set, each labelled positive or negative.The source dataset also contains 50,000 unlabelled reviews, which the experiments do not use.Source: §5.1 · p. 6

5.2 Review Summarisation via Sentence extraction

  • Figure 2 shows coloured ConvNet-selected sentences among the full review text; the examples favor ConvNet over first-and-last selection, using up to 20% of sentences.Source: Figure 2 · p. 7
  • The ConvNet extraction is compared with a shallow word2vec-based neural model, uniform-random sentence scoring, and first-and-last sentence selection.Source: §5.2 · p. 8
  • With only two sentences per review, reference-model test accuracy drops by less than 1.5%, and ConvNet extraction beats every baseline in all cases.The evaluation uses a Naïve Bayes reference model trained on the full training set.Source: §5.2 · pp. 6, 8
  • The model learns to ignore background such as viewing-location descriptions and plot summaries that do not express an opinion on the movie.Source: §5.2 · p. 8

6 Conclusion

The conclusion presents an introspectable document ConvNet for extracting task-specific salient sentences and a scalable classifier-based evaluation method. In comparisons with several baselines, the ConvNet preserves more task-relevant information even when few sentences are extracted.

6 Conclusion

  • The hierarchical ConvNet is designed to support introspection of document structure, and visualization techniques identify and extract task-specific salient sentences.The paper demonstrates the approach by extracting sentiment-relevant sentences from movie reviews.Source: §6 · p. 8
  • A scalable evaluation method estimates preserved task-relevant information by comparing a reference classifier’s performance on full documents and extracted-sentence documents.Source: §6 · p. 8
  • The ConvNet extracts more task-relevant information than baseline methods, even when the number of extracted sentences is very small.Source: §6 · p. 8

Related papers

Loading 1412.6815v2…