AttSum: Joint Learning of Focusing and Summarization with Neural Attention

1 Introduction

• Reference summaries trade off query relevance and sentence saliency, so isolated rankers cannot learn either set of weights well from that supervision.Salient content may be excluded if it does not answer the query, while relevant but unrepresentative content may also be excluded.Source: §1 · p. 1
• Surface similarity can rank query keywords above the actual need: sentences matching “drug legalization” may fail to describe its benefits.The paper notes that query relevance is not equivalent to similarity and that surface features can inadequately measure it.Source: §1 · pp. 1–2
• AttSum jointly learns query relevance and sentence saliency with neural attention, using learned relevance weights to pool sentence embeddings into a query-conditioned document representation.The authors describe this attention as simulating human attentive reading and state that the system learns features automatically rather than relying on hand-crafted features.Source: §1 · p. 2
• On DUC 2005–2007, AttSum outperforms widely used systems relying on rich hand-crafted features, and its high-relevance sentences focus on query-relevant content.The paper reports this qualitative finding alongside its benchmark comparison.Source: §1 · p. 2

2 Query-Focused Sentence Ranking

• AttSum uses attention to focus sentence ranking on query-relevant content, combining learned sentence and document-cluster representations.Its framework comprises CNN, pooling, and ranking layers.Source: §2 · pp. 2–3

2.1 CNN Layer

• CNNs project sentences and queries into distributed embeddings by convolving over word windows and applying max-over-time pooling.The pooled features serve as the sentence and query embeddings.Source: §2.1 · p. 3

2.2 Pooling Layer

• The pooling layer weights sentence embeddings by learned query relevance to form a document-cluster embedding biased toward relevant and representative sentences.The relevance score uses a sigmoid over sentence–query embedding interactions; relevance is not treated as similarity.Source: §2.2 · pp. 3–4

2.3 Ranking Layer

• The ranking layer scores sentences by cosine similarity to the query-conditioned document embedding and trains pairwise rankings using high-ROUGE-2 sentences as positives.The stated training procedure uses diagonal AdaGrad with mini-batches.Source: §2.3 · p. 4

3 Sentence Selection

• Greedy selection discards sentences shorter than 8 words, then adds top-ranked sentences only when they contribute sufficiently new bigrams.The new-bigram ratio cutoff is 0.5.Source: §3 · p. 4
• The selection procedure sorts remaining sentences by ranking score and iteratively dequeues them for possible inclusion.A sentence is non-redundant when it contains significantly new bigrams compared with the current summary.Source: §3 · p. 4
• The greedy procedure targets summaries that are both informative and non-redundant, using a selection strategy similar to MMR.Source: §3 · p. 4

4.1 Dataset

• DUC 2005–2007 comprise news-document clusters with four NIST reference summaries each; summaries are capped at 250 words and evaluated using three-fold cross-validation.Each fold trains on two years and tests on the remaining year.Source: §4.1 · pp. 4–5

4.2 Model Setting

• Pretrained 50-dimensional word2vec embeddings remain fixed; CNN window h=2 and dimension l=50, with pairwise-ranking margin 0.5.The word embeddings were trained on a large English news corpus; the model setting also specifies learning rate 0.1 and batch size 100.Source: §4.2 · p. 5

4.3 Evaluation Metric

• ROUGE-2 is the main evaluation metric, and ROUGE-2 scores also rank sentences during pairwise-ranking training.ROUGE measures overlap between peer and reference summaries; ROUGE-1 results are also reported.Source: §4.3 · p. 5

4.4 Baselines

• Comparators include LEAD, QUERY SIM, graph-based MultiMR, feature-based SVR, neural DocEmb, and ISOLATION, which removes attention and combines embedding saliency with TF-IDF relevance.MultiMR is unsupervised, while SVR is supervised; DocEmb is supplemented with query information for document representation.Source: §4.4 · pp. 5–6

4.5 Summarization Performance

• AttSum improves over ISOLATION, LEAD, QUERY SIM, and DocEmb, and generally surpasses MultiMR and SVR, except on DUC 2005 ROUGE-2; overfitting is a possible explanation.The authors attribute the DUC 2005 exception in part to fewer training examples under three-fold cross-validation, which impedes embedding learning.Source: Table 2 · p. 6, §4.5 · p. 6

4.6 Query Relevance Performance

• AttSum's highly ranked sentences usually address query needs, unlike TF-IDF matches that often repeat query keywords; SVR's query-dependent features have extremely small weights, and removing them drops performance 1%.The query-relevance analysis also notes that AttSum may focus on only part of a multi-sentence query.Source: §4.6 · pp. 6–7
• For multi-sentence queries, AttSum may focus on only part of the query; such complex queries are rare in DUC, so the authors do not design a special model for them.The authors note that a single query embedding may augment biases in the reference summaries.Source: §4.6 · p. 7

5.1 Extractive Summarization

• Extractive summarization includes greedy MMR balancing saliency and redundancy, ILP formulations, graph-based ranking, and learning-based classifiers such as CRF, SVR, and Logistic Regression.The cited ILP work is described as finding an optimal solution; graph-based models represent sentence relationships.Source: §5.1 · p. 8
• Query-focused systems often extend generic summarizers with query-dependent features, including TF-IDF cosine similarity, WordNet similarity, and word co-occurrence, which can reward similarity without meeting query needs.The passage cites TF-IDF cosine similarity, WordNet similarity (Ouyang et al., 2011), and word co-occurrence (Prasad Pingali and Varma, 2007).Source: §5.1 · p. 8

5.2 Deep Learning in Summarization

• Deep-learning approaches include neural ranking features, distributed sentence representations, sequence labeling, and sum-pooled embeddings, alongside LSTM compression and attention-based one-sentence generation.The cited approaches include sentence ranking features (Cao et al., 2015a; Cao et al., 2015b), CNN sentence representations, and recurrent sequence labeling (Cheng and Lapata, 2016).Source: §5.2 · p. 8

• AttSum jointly handles saliency and relevance ranking, learning distributed representations for sentences and document clusters while using attention to model query-guided attentive reading.Source: §6 · p. 8
• On DUC query-focused datasets, AttSum achieves competitive performance without hand-crafted features, and sentences identified as query-focused are observed to meet query needs.Source: §6 · p. 8
• The authors identify extending the system to abstractive summarization as future work, by integrating a neural language model after document embeddings.Source: §6 · p. 8