Source-linked AI summary
Fine-Grained Spoiler Detection from Large-Scale Review Corpora
Mengting Wan, Rishabh Misra, Ndapa Nakashole, Julian McAuley
TL;DR
Fine-grained spoiler annotation is difficult to collect comprehensively because few users apply sentence-level tags, while prior work left sentence dependencies and user/item effects understudied. The paper builds a large Goodreads dataset, analyzes spoiler-language patterns, and develops SpoilerNet; the complete model substantially outperforms baselines on both evaluated datasets. The analysis also identifies error patterns that constrain current detection quality.
Problem
Fine-grained spoiler detection lacks comprehensive annotations and has not studied sentence dependencies, user/item spoiler bias, or item-contextual sentence semantics.
Method
The paper constructs a large Goodreads dataset, analyzes spoiler-language patterns, and develops SpoilerNet as a domain-specific neural sentence-classification model.
Results
The complete SpoilerNet model consistently and substantially outperforms baselines on both the Goodreads and TV Tropes datasets.
Takeaways & Limitations
Spoiler sentences tend to be book-specific, clustered later in reviews, and distributed differently across books and users, motivating item- and user-aware modeling.
Takeaways & Limitations
SpoilerNet can be distracted by revelatory terms and surrounding spoiler sentences, while self-reported spoiler labels can be subjective and inconsistent.
Abstract
from arXiv · showhide
This paper presents computational approaches for automatically detecting critical plot twists in reviews of media products. First, we created a large-scale book review dataset that includes fine-grained spoiler annotations at the sentence-level, as well as book and (anonymized) user information. Second, we carefully analyzed this dataset, and found that: spoiler language tends to be book-specific; spoiler distributions vary greatly across books and review authors; and spoiler sentences tend to jointly appear in the latter part of reviews. Third, inspired by these findings, we developed an end-to-end neural network architecture to detect spoiler sentences in review corpora. Quantitative and qualitative results demonstrate that the proposed method substantially outperforms existing baselines.
1 Introduction
The paper addresses the limited scalability and coverage of fine-grained spoiler annotation by introducing a large Goodreads review resource and SpoilerNet for automatic sentence-level detection. It frames the task around domain-specific language patterns and dependencies among review sentences.
- Motivation: Few users apply sentence-level spoiler tags, making comprehensive fine-grained annotation difficult to obtain manually.Crowdsourcing offers a complementary approach but may be difficult to scale.
- Prior work: Prior spoiler-detection work used topic, lexical, verb, named-entity, genre, or external metadata features with standard classifiers.Earlier research had not studied sentence dependency, user/item spoiler bias, or sentence semantics across item contexts.
- Approach: The task is formulated as sentence classification while modeling domain-specific language patterns.The paper places spoiler detection within neural sentence/document classification while focusing on patterns specific to the review domain.
- Contributions: The paper introduces a large-scale Goodreads dataset with sentence-specific self-reported spoiler tags and proposes the SpoilerNet model.The dataset is intended to support supervised models, while SpoilerNet is motivated by preliminary Goodreads analysis.
2 The Goodreads Book Review Dataset
The Goodreads dataset contains large-scale sentence-level spoiler annotations and supports analyses of spoiler position, span structure, book specificity, and user/item variation. These analyses show that spoiler sentences are later, more clustered, more book-specific, and unevenly distributed across users and books.
- Dataset scale: 1,378,033 reviews span 25,475 books and 18,892 users, with 17,672,655 sentences and 3.22% labeled as spoilers.The dataset provides fine-grained spoiler annotations at a substantially larger scale than previously available.
- Spoiler distribution: Spoiler sentences tend to appear later in reviews and form longer consecutive spans than randomly sampled sentences.The random-position benchmark uses an expected average position of 0.5, while the span benchmark compares real spoilers with randomly sampled sentences.
- Item specificity: Spoiler sentences are likely to be more book-specific than non-spoiler sentences.For Harry Potter #7, the ten highest-ranked terms refer to character or author names and important plot points.
- User and item variation: Spoiler-label distributions differ substantially across users and books, indicating distinct spoiler tendencies across both groups.These differences motivate explicit calibration of user and item effects in subsequent model design.
3 The Proposed Approach: SpoilerNet
SpoilerNet frames spoiler detection as sentence-level binary classification and extends hierarchical attention modeling with item-specific language features, sequential sentence context, and item/user biases.
- Task formulation: SpoilerNet predicts whether each review sentence contains spoilers using a binary classification formulation.The target label is ys = 1 for spoilers and ys = 0 otherwise.
- Architecture: SpoilerNet extends hierarchical attention networks with sentence-sequence modeling, item-specific word representations, and item/user bias terms.These components address domain-specific language patterns and disparities in spoiler distributions.
- Input layer: Item-specificity features concatenate shared word embeddings with DF-IIF features to distinguish word meanings across book contexts.The paper illustrates that “Green” may denote a character name in one context and a color in another.
- Hierarchical encoding: Bidirectional GRUs encode words and review sentences, while word attention emphasizes revelatory terms such as “kill” and “die.”Sentence-level encoding models sequential dependencies among sentences within each review.
- Output and training: The output layer combines sentence representations with learnable item and user biases, and training uses a weighted loss controlled by η.The biases model item- and user-level effects not explained by the language model; η balances positive and negative labels.
4 Experiments
Experiments evaluate SpoilerNet on Goodreads and TV Tropes against classical and neural baselines using metrics designed to address label imbalance and self-reporting subjectivity. The complete model consistently outperforms baselines, with sentence dependency and pretrained embeddings especially important in specific settings.
- Datasets: The experiments use Goodreads and TV Tropes, with Goodreads labels highly imbalanced and TV Tropes labels approximately balanced.Goodreads uses η = 0.05 after validation, while TV Tropes uses η = 1.
- Baselines and ablations: Baselines include SVM, SVM-BOW, CNN, and HAN, with ablations testing item-specificity, item/user biases, word attention, pretrained embeddings, and sentence encoding.The study adds individual components to baselines and removes modules from HAN to assess their effectiveness.
- Evaluation: AUC is the primary metric because self-reported spoiler labels may be subjective, while TV Tropes accuracy supports comparison with prior work.For Goodreads, document-level ranking AUC also evaluates language modeling independently of item/user reporting bias.
- Results: SpoilerNet consistently and substantially outperforms baselines on both Goodreads and TV Tropes.On TV Tropes, its accuracy exceeds the strongest prior result without external genre information, 0.723, but remains below the genre-enhanced result of 0.756.
- Ablation findings: Removing the sentence encoder causes a substantial Goodreads performance drop, while pretrained word embeddings are particularly important on TV Tropes.The authors suggest that insufficient initialization can make the model overfit the smaller TV Tropes dataset.
5 Error Analysis
The error analysis identifies three recurring failure modes: revelatory terms, nearby spoiler sentences, and inconsistent self-reported spoiler standards. These cases expose where SpoilerNet’s lexical and contextual signals can mislead its predictions.
- Case Studies: The case studies compare review text, ground-truth spoiler tags, and SpoilerNet’s predicted spoiler probabilities across three popular books.The examples cover Murder on the Orient Express, The Fault in Our Stars, and The Hunger Games.
- Distracted by Revelatory Terms: SpoilerNet is frequently distracted by revelatory terms such as “murder” and “killed,” producing false-positive spoiler predictions.The authors suggest emphasizing difficult negative sentences containing such terms during training.
- Distracted by Surrounding Sentences: The model can mistake a nonspoiler sentence for a spoiler when it follows a spoiler and contains an item-specific revelatory term.In the example, the third sentence mentions the character “Hazel,” illustrating the need for finer-grained dependency modeling.
- Inconsistent Standards of Spoiler Tags: Self-reported spoiler labels can be controversial, making some examples difficult to justify even for human annotators.The authors propose ranking sentences within the same review as a possible alternative to binary classification.
6 Conclusions and Future Work
The paper closes by identifying future uses for its dataset, analysis, and model, including spoiler-aware generation, richer language modeling, and transfer to sparsely annotated corpora.
- Future Work: The dataset and analysis could support rewriting spoiler content in a non-spoiler way as a language-generation task.The authors also suggest incorporating syntax information into spoiler language models.
- Future Work: The Goodreads dataset could serve as a spoiler source corpus for transferring learned models and knowledge to corpora with limited annotations.The paper gives spoiler detection in tweets as an example target.