PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Opinion Summarization as Copycat-Review Generation

Arthur Bražinskas, Mirella Lapata, Ivan Titov

arXiv:1911.02247v2cs.CLcs.IRcs.LGstat.ML

TL;DR

Opinion summarization has often been extractive, while supervised approaches rely on expensive summary annotations. This paper introduces COPYCAT, an unsupervised hierarchical latent-variable model for generating abstractive consensus summaries; the authors report favorable comparisons with competitors and better alignment with input reviews in human evaluation.

  • Problem

    Opinion summarization has largely been extractive, and supervised summarization depends on expensive annotations that are especially burdensome across domains.

    Source: §1 · pp. 1–2

  • Method

    COPYCAT uses a hierarchical variational autoencoder with product- and review-level latent representations to generate abstractive consensus summaries without summaries in training.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    The model compares favorably with competitors, especially MeanSum, and human evaluation finds its summaries better aligned with input reviews.

    Source: §1 · p. 2, §8 · p. 8

  • Takeaways & Limitations

    The generated consensus summaries are presented as useful for quick decisions and for getting an overall impression of a product or business.

    Source: §1 · p. 2

    The analysis frames mean-latent-code grounding as a hypothesis about reflecting review content, not as a stated guarantee.

    Source: §6 · p. 8

Abstract

from arXiv · show

Opinion summarization is the task of automatically creating summaries that reflect subjective information expressed in multiple documents, such as product reviews. While the majority of previous work has focused on the extractive setting, i.e., selecting fragments from input reviews to produce a summary, we let the model generate novel sentences and hence produce abstractive summaries. Recent progress in summarization has seen the development of supervised models which rely on large quantities of document-summary pairs. Since such training data is expensive to acquire, we instead consider the unsupervised setting, in other words, we do not use any summaries in training. We define a generative model for a review collection which capitalizes on the intuition that when generating a new review given a set of other reviews of a product, we should be able to control the "amount of novelty" going into the new review or, equivalently, vary the extent to which it deviates from the input. At test time, when generating summaries, we force the novelty to be minimal, and produce a text reflecting consensus opinions. We capture this intuition by defining a hierarchical variational autoencoder model. Both individual reviews and the products they correspond to are associated with stochastic latent codes, and the review generator ("decoder") has direct access to the text of input reviews through the pointer-generator mechanism. Experiments on Amazon and Yelp datasets, show that setting at test time the review's latent code to its mean, allows the model to produce fluent and coherent summaries reflecting common opinions.

1 Introduction

The paper introduces unsupervised abstractive opinion summarization to generate consensus summaries without summary-labeled training data. COPYCAT controls novelty through hierarchical latent representations, and the authors report outperforming the prior method in automatic and human evaluations.

1 Introduction

  • Opinion summarization needs costly annotations across domains, while prior approaches primarily extract and copy input fragments rather than generate abstractive summaries.The authors target consensus opinions, which they argue can help readers make quick decisions and get an overall feel for a product or business.Source: §1 · pp. 1–2
  • COPYCAT models review collections with controllable novelty, then uses mean latent representations at test time to generate consensus-oriented summaries.Product-level codes represent shared semantics, while individual-review codes encode review-specific content.Source: §1 · p. 2
  • A pointer-generator gives the review decoder direct access to input text, encouraging specific details rather than generic summaries.The authors report that performance drops substantially without this conditioning.Source: §1 · p. 2
  • The approach substantially outperforms the previous unsupervised multi-document method in automatic metrics and human evaluation.The paper evaluates on Amazon product reviews and Yelp business reviews.Source: §1 · p. 2
  • The paper also provides a dataset of abstractive summaries for Amazon products.Source: §1 · p. 2

2 Model and Estimation

COPYCAT models review collections with product-level and review-level latent representations, and lets the decoder consult other reviews directly. It trains the model by maximizing a variational lower bound and uses encoder and decoder networks to represent the latent codes and generate text.

2.1 Overview of the Generative Model

  • COPYCAT assigns each product a group code c for shared semantics and each review a code z_i for its individual semantics.The decoder uses z_i to produce review text.Source: §2.1 · p. 3
  • Direct access to the other reviews lets the decoder reuse fine-grained information, including named entities, that could otherwise be lost through the latent bottleneck.The model conditions on all other reviews in the group without imposing an order; the authors report that both this conditioning and hierarchical modeling benefit performance in ablations.Source: §2.1 · p. 3

2.2 Model Estimation

  • The model is estimated by maximizing a variational lower bound with a reconstruction term and KL-divergence regularizers for the inferred latent distributions.The bound is optimized over generative and inference-network parameters; Gaussian assumptions permit closed-form KL terms, and the reparameterization trick computes reconstruction-term gradients.Source: §2.2 · p. 4

2.3.1 Text Representations

  • A pointer-generator decoder attends to other reviews and combines vocabulary generation with copying, helping preserve details and generate rare tokens.The architecture uses Gaussian distributions and separate linear projections to compute means and diagonal log-covariances for the latent codes.Source: §2.3.4 · pp. 4–5, §2.3.1 · p. 4

3 Summary Generation

Given a review collection, the model generates a summary intended to reflect common information. It uses the means of the review and group latent codes rather than sampling them.

3 Summary Generation

  • Summary generation uses the review and group latent-code means rather than sampling, to reflect common information in the input reviews.The group-code mean is predicted by the inference network from the review collection.Source: §3 · p. 5

4 Experimental Setup

The experiments use Yelp business reviews and Amazon product reviews, whose distinct content calls for different summarization choices. Evaluation combines human-created summaries with ROUGE and comparison systems ranging from learned models to simple review-selection baselines.

4.1 Datasets

  • Yelp reviews contain personal and irrelevant details to abstract away, whereas Amazon reviews include specific, decision-useful information to preserve.The experiments use business reviews from Yelp and product reviews from Amazon.Source: §4.1 · p. 5
  • Yelp references summarize eight input reviews, while Amazon references were collected for 60 products with eight reviews each; 28 products were used for development and 32 for testing.Three summaries were collected per Amazon product; the Yelp references were previously released.Source: §4.1 · p. 5

4.2 Experimental Details

  • The models use GRUs for sequential encoding and decoding, shared randomly initialized embeddings, Adam, and cyclical annealing to address posterior collapse; ROUGE scores use F1.Source: §4.2 · p. 5

4.3 Baseline Models

  • The comparison includes Opinosis, which selects review words, LexRank, which selects graph-central sentences, and the unsupervised abstractive model MeanSum.LexRank weights sentence similarity using tf-idf and can rank centrality with PageRank.Source: §4.3 · p. 6
  • Additional baselines are a vanilla text VAE, a review chosen by ROUGE-L centrality within its group, a random review, and leading sentences from group reviews.Source: §4.3 · p. 6
  • An oracle review provides an upper-bound comparison by selecting the group review with the highest ROUGE-L against reference summaries.Source: §4.3 · p. 6

5 Evaluation Results

Copycat obtains the highest ROUGE scores on both Yelp and Amazon, and human evaluations show strong results alongside dataset-specific trade-offs. The content-support evaluation also finds better information preservation than MeanSum.

5.1 Automatic Evaluation

  • Copycat yields the highest ROUGE scores on both Yelp and Amazon, with large gains over vanilla VAE and substantial gains over MeanSum.The authors conjecture that vanilla VAE struggles to represent varied categories under a single prior; Copycat instead gives each group its own prior and access to other reviews during decoding.Source: §5.1 · p. 6

5.2 Human Evaluation

  • On Yelp, Copycat scores higher than other models on most human-evaluation criteria, including overall quality, with differences significant at p < 0.01.Its fluency difference from gold summaries is not statistically significant.Source: §5.2 · p. 7
  • On Amazon, Copycat outperforms other methods in fluency, coherence, and non-redundancy, but trails LexRank on opinion consensus, and LexRank is slightly preferred overall.All pairwise differences between Copycat and comparison systems are significant at p < 0.05.Source: §5.2 · p. 7
  • LexRank’s coverage and overall preference differ by dataset: it has better coverage but is not preferred overall on Yelp, while on Amazon its coverage is on par with Copycat and it is preferred overall.The authors suspect breadth of exact details matters more on Amazon; LexRank summaries tend to be about 20 tokens longer.Source: §5.2 · p. 7
  • The content-support study finds Copycat better at preserving information than MeanSum.The study asked workers to rate whether summary sentences were fully, partially, or not supported by the reviews.Source: §5.2 · p. 7, Table 7 · p. 8

6 Analysis

The analysis tests whether mean latent codes yield more grounded summaries than sampling and examines the roles of model components and copied words.

6 Analysis

  • Removing z caused the largest ROUGE drop and wordier, more review-like summaries; removing attention produced generic summaries, while removing c caused the smallest ROUGE-L drop.The ablations were retrained on Amazon reviews.Source: §6 · p. 8
  • Sampling latent variables produced summaries that were wordier, less fluent, and less aligned to the input than using mean values.This tests the hypothesis that mean values produce more grounded summaries, while sampling yields novel, potentially irrelevant details.Source: §6 · p. 8
  • During summary generation, the full model copies around 3–4 tokens per summary, often product-type words, brands, and names.Source: §6 · p. 8

7 Related Work

Prior work includes weakly supervised and unsupervised extractive methods, while this paper’s closest conceptual comparison is MeanSum, which uses a different latent-state formulation.

7 Related Work

  • Angelidis and Lapata (2018) use weak supervision for segment sentiment and a small sample of gold summaries for aspect labels before constructing summaries heuristically.Opinosis uses redundancy and part-of-speech tags but cannot generate novel words and phrases or coherent long summaries.Source: §7 · p. 8
  • LexRank selects representative sentences using a graph, while Isonuma et al. (2019) summarize single reviews using latent discourse trees.Source: §7 · p. 8
  • MeanSum treats a summary as a discrete latent state of an autoencoder, whereas this work models review collections hierarchically with continuous latent codes.Source: §7 · p. 8

8 Conclusions

The paper presents an end-to-end abstractive opinion summarizer trained without summaries. Its model compares favorably with competitors, and human evaluations find its summaries better aligned with input reviews.

8 Conclusions

  • The paper presents an end-to-end abstractive opinion summarizer trained on a large review collection without summaries.Source: §8 · p. 8
  • The model compares favorably with competitors, especially the only other unsupervised abstractive multi-review summarization system, and human evaluations find better alignment with input reviews.Source: §8 · p. 8

A.1 Derivation of the Lower Bound

The derivation simplifies notation by abbreviating two variational distributions.

A.1 Derivation of the Lower Bound

  • The derivation abbreviates qφ(c|r1:N) as qφ(c|·) and qφ(z|ri, c) as qφ(z|i).Source: §A.1 · p. 11

A.2 Dataset Pre-Processing

The appendix specifies dataset filtering and model settings, then documents human evaluation procedures and qualitative analyses of latent-code behavior and repetition. It reports that mean-based z summaries were more fluent and contained fewer unsupported details than sampled summaries.

A.2 Dataset Pre-Processing

  • Businesses and products needed at least 10 reviews; review lengths were limited to 20–70 words, groups above the 90th percentile were removed, and training groups contained eight reviews.The selected Amazon categories were Electronics; Clothing, Shoes and Jewelry; Home and Kitchen; and Health and Personal Care.Source: §A.2 · p. 11

A.3 Hyperparameters

  • The implementation used 600-dimensional GRUs and latent variables, shared 200-dimensional embeddings, and length-normalized beam search of size five with latent-code means for summary decoding.It used Adam, cycling annealing for the z- and c-related KL terms, and a 50,000-word vocabulary with a 30,000-word extended vocabulary.Source: §A.3 · pp. 11–12

A.4 Human Evaluation Setup

  • Human-evaluation workers completed both tasks in combined HITs, reread reviews before each task, and qualified through approval, experience, location, and test requirements.Requirements included a 98% approval rate, 1,000 or more HITs, location in the USA, UK, or Canada, and a maximum score on the qualification test.Source: §A.4 · p. 12

A.5 Full Human Evaluation Instructions

  • The human-evaluation rubric asked workers to assess fluency, coherence, non-redundancy, opinion consensus, and overall best and worst summaries.Consensus meant reflecting common review opinions, while the rubric defined coherence as a well-organized summary rather than a heap of related information.Source: §A.5 · p. 12

A.6 Amazon Summaries Creation

  • For Amazon references, three workers summarized eight reviews for each of 15 products per category, aiming to preserve common opinions and sentiment while writing fluent, coherent summaries in their own words.Workers could use exact review wording but were told not to copy more than five consecutive words; they were asked to iterate and reread reviews as needed.Source: §A.6 · pp. 12–13

A.7 Latent Codes Analysis

  • Mean-based z summaries were more fluent and contained fewer partially or wholly unsupported details, while different z samples expressed different review details about the same product.The analysis fixed c at its posterior mean and compared sampled and mean assignments to z.Source: §A.7 · p. 13, Table 9 · p. 14

A.8 Repetitions

  • Low z-related KL and beam search increased repetitions in reconstructed reviews and summaries as the decoder relied more on learned local statistics.When the KL divergence vanishes, the decoder essentially becomes an unconditional language model, for which beam search was associated with repetitions.Source: §A.8 · p. 13, §A.6 · p. 13

Related papers

Loading 1911.02247v2…