SummScreen: A Dataset for Abstractive Screenplay Summarization

1 Introduction

• SUMMSCREEN pairs TV-series transcripts with episode recaps, addressing the scarcity of abstractive summarization datasets for narrative text centered on entities and dialogue.Source: §1 · p. 1
• Plot events are often implied in dialogue, while character development and humor distract from plot; multiple subplots require integrating utterances across the transcript.Some details may also be absent because scripts are grounded by audiovisual accompaniment.Source: §1 · p. 2
• The paper proposes entity-centric metrics for overlap in recap characters and in co-occurring character pairs.Source: §1 · p. 2
• Oracle extractive approaches outperform all benchmarked models on every automatic metric, suggesting current methods do not fully exploit transcripts.Source: §1 · p. 2
• Human evaluations find non-oracle hybrid models competitive with oracle counterparts on faithful plot events; qualitative analysis finds generic neural summaries and unfaithful details.Hybrid models may benefit from better content selection.Source: §1 · p. 2

2 Related Work

• Prior work includes extractive screenplay summarization, crime-drama analysis, and summarization datasets for dialogue and meetings.Source: §2 · p. 2
• Other long-form summarization work covers books, scientific articles, multiple news articles, opinionated text, patents, TV-show stories, and novel chapters.Source: §2 · p. 2
• TV-show resources have also been adapted for question answering, speaker identification, sarcasm and emotion detection, relation extraction, and story generation.Source: §2 · p. 2

3.1 Dataset Construction

• SUMMSCREEN pairs TV transcripts with corresponding recaps and filters for over 85% character overlap and more than 100 speaker-labeled utterances.Source: §3 · p. 2, §3.1 · p. 3
• SUMMSCREEN’s four-gram overlap with inputs is comparable to XSum despite longer summaries, suggesting the dataset favors abstractive approaches.Source: §3.1 · p. 3
• FD and TMS differ in genres, transcript length, recap coverage, and writing style, so the paper treats them as separate datasets.FD covers more genres and has longer transcripts; TMS recaps seek to cover more detail, while FD writing styles are more diverse.Source: §3.1 · pp. 3–4

3.2 Dataset Comparison

• SUMMSCREEN combines longer inputs, more speakers, many episodes per show, and enough instances to train and evaluate neural methods.Source: §3.2 · p. 4
• Pretrained models perform worse on SUMMSCREEN than on MediaSum according to automatic metrics, although MediaSum is the closest comparison dataset.MediaSum summaries are twenty times shorter than SUMMSCREEN summaries.Source: §3.2 · p. 4

3.3 Dataset Challenges

• Plot events and character sentiments are often implied through dialogue, requiring context and knowledge about the characters to interpret utterances.Source: §3.3 · pp. 4–5
• Models must filter out character-development and humor lines while connecting parallel plot threads that can be separated by hundreds of transcript lines.The paper also notes that more than half of the transcript characters do not appear in recaps.Source: §3.3 · pp. 5–6

4 Approaches

• The benchmark covers nearest-neighbor extractive, neural abstractive, and hybrid methods; extractive approaches are not expected to be useful because transcript sentence meaning is context-dependent.Nearest-neighbor extractive results are included mainly to characterize the dataset.Source: §4 · p. 6

4.1 Neural Models

• Neural models encode transcript lines with Longformer and expose line-ending [EOS] vectors to a transformer decoder using global attention.Source: §4.1 · p. 6

4.2 Nearest Neighbor Models

• Nearest-neighbor baselines retrieve transcript lines or recaps using BM25 or ROUGE, with recap-to-transcript retrieval serving as an oracle extractive estimate.Transcript-to-transcript retrieval uses training transcripts and their recaps; recap-to-recap retrieval also evaluates similarity using entity-centric scores.Source: §4.2 · p. 6

4.3 Hybrid Models

• Hybrid models select salient transcript content with recap-to-transcript neighbors or BM25, then generate with BART-large.For recap-based selection, each recap sentence retrieves three transcript lines plus neighboring context, alongside a whole-recap retrieval; a learned selector predicts lines for the non-oracle baseline. The recap-based selector relies on gold recaps and is an approximate upper bound with powerful selectors.Source: §4.3 · p. 6

5.1 Setup

• Generic metrics average BLEU, ROUGE-1, ROUGE-2, and ROUGE-L, treating each word equally.The paper describes the average as generally showing semantic similarity between generations and references.Source: §5.1 · p. 7
• Bag-of-character metrics measure overlap between characters in generated and reference recaps, while bag-of-character-relation metrics compare cooccurring character pairs.The relation metric treats characters appearing in the same sentence as related; it does not account for direction or cooccurrence frequency.Source: §5.1 · p. 7

5.2 Results

• Nearest-neighbor models perform strongly, with NNM-r2t showing the best performance in general.The authors say this suggests room to improve neural models’ ability to extract useful transcript information.Source: §5.2 · p. 7
• Neural models are strong on generic semantic matching but relatively weak on entity metrics compared with non-oracle baselines.Source: §5.2 · p. 7
• Hybrid models outperform the neural model on character mentions and relations, and an oracle content selector improves both semantic matching and entity-related metrics.The authors identify better content selection as a future research direction.Source: §5.2 · p. 7

7 Effect of Combining FD and TMS

• Training on both FD and TMS helps FD but hurts TMS compared with training on the in-domain dataset alone.The authors hypothesize that FD’s smaller training set explains its gain, while TMS’s larger training set and resource differences may explain its decline.Source: §7 · p. 7, Table 6 · p. 8

7.1 Human Evaluation

• In human evaluation, the non-oracle hybrid is competitive with the oracle hybrid on predicate match but less close on character relations.The evaluation used 120 randomly selected FD test instances and asked separate questions about predicate matching and character-relation similarity.Source: §7.1 · pp. 7–8

7.2 Generation Samples

• Neural generations share topics with references and often include their character names, but lack the hybrids’ specificity.The authors hypothesize that difficulty modeling long-form text causes this pattern.Source: §7.2 · pp. 8–9
• The non-oracle hybrid includes transcript details absent from the reference, yet can omit main characters and introduce incorrect facts.The oracle hybrid is described as most faithful among the four sampled models, but it too produces incorrect facts.Source: §7.2 · p. 9

8 Conclusion

• SUMMSCREEN pairs TV show transcripts with recaps and introduces metrics for character overlap and character-pair overlap.The paper benchmarks neural and nearest-neighbor models, finding that an oracle extractive summarizer performs best on automatic metrics; human and qualitative analyses also identify content-selection and factuality challenges.Source: §8 · p. 9

A Hyperparameters

• The reported neural-model setup uses a maximum encoder sequence length of 14336 and a decoder length of 1024.It also specifies a one-layer encoder and 12-layer decoder with 1024 hidden units unless otherwise noted, and beam search with trigram blocking and beam size 5 at test time.Source: A Hyperparameters · p. 13
• The neural content selector uses a three-layer Longformer encoder followed by a two-layer feedforward network, and selects at least 10% of transcript lines.Its reported test F1 scores are 19.0 on FD, 19.2 on anonymized FD, 41.5 on TMS, and 40.1 on anonymized TMS.Source: A Hyperparameters · p. 13

B Anonymized SUMMSCREEN

• The anonymized dataset replaces character names with random IDs that are reassigned across episodes.The authors use this design to counter the possibility that models focus on a limited set of characters and their typical behaviors rather than transcript actions.Source: B Anonymized SUMMSCREEN · p. 13

• Anonymization leaves recap-transcript similarities largely unchanged but makes their correlations much weaker, especially for entities.Source: C Results for the Anonymized Datasets · p. 13
• The excerpt pairs dialogue about a board game with a recap describing characters’ differing enthusiasm for playing.Source: C Results for the Anonymized Datasets · p. 13
• The anonymized example replaces character names with entity IDs that are permuted across episodes.Source: Figure 4

D Effect of Anonymization

• Anonymized models perform better on rare entities, suggesting anonymization helps model rare entities.Entities were selected by ranking their frequencies within each TV show’s training set.Source: D Effect of Anonymization · p. 13
• The models perform equally in the “all” setting, indicating anonymization also makes learning common entities harder.Source: D Effect of Anonymization · p. 13

E Effect of Copy Mechanism

• The copy mechanism helps tremendously on anonymized ForeverDreaming but gives mixed results on non-anonymized data.These results use 6-layer decoders with 512 hidden units and are not directly comparable to other results.Source: E Effect of Copy Mechanism · p. 13, Table 11 · p. 14
• On anonymized data, scarce training for character-ID embeddings makes copying useful by reducing the required supervision.On non-anonymized data, character names provide more information for learning name embeddings.Source: E Effect of Copy Mechanism · p. 14

F Effect of Combining FD and TMS

• Training on TMS + FD helps FD but hurts TMS; anonymized ForeverDreaming also benefits greatly from additional training data.Source: Table 12 · p. 14, F Effect of Combining FD and TMS · p. 14