Cut the CARP: Fishing for zero-shot story evaluation

1 Introduction

• Story-evaluation methods such as ROUGE can assess stories poorly, while other approaches rely on costly human evaluation.This makes evaluation difficult to integrate into end-to-end pipelines.Source: §1 · pp. 1–2

1.1 Representations of Knowledge in Computational Narratology

• End-to-end representations can be expressive but may go off-topic, while language models can produce self-contradictory or nonsensical stories.The text argues that these weaknesses motivate more robust grounded language modeling.Source: §1.1 · pp. 2–3

1.2 Contrastive Learning with Transformers

• Contrastive learning trains encoders to match aligned pairs and distinguish them from incorrect targets in the batch.The paper relates this approach to CLIP, which aligns images with their associated captions.Source: §1.2 · p. 3

1.3 Our Contribution

• The paper presents CARP as an automated story-evaluation model for end-to-end pipelines and introduces the Story-Critique dataset for narratology and language-modeling research.It also identifies contrastive learning as an effective tool for learning.Source: §1.3 · p. 3

2 The Story-Critique Dataset

• The dataset was created because the authors could not find textual datasets with inline critiques, while its use must address three privacy concerns.Story authors objected to public release of full text and identification; hosts objected to becoming publicly known, so access is restricted and hosts retain control.Source: §2 · pp. 3–4

2.1 Dataset Structure

• The dataset contains more than 80,000 stories and 1,378,696 passage-critique pairs, with critiques linked to specific passages.Passages are not indexed by story ID, and entries are anonymized while critique type and word count are retained.Source: §2.1 · p. 4

2.2 Preprocessing

• Preprocessing splits stories at inline critiques, masks quoted story text, and removes proper nouns to limit data leakage and protect author privacy.The authors identify quote matching and overlapping proper nouns as sources of leakage.Source: §2.2 · p. 4

2.2.1 Quote Masking

• Quote masking uses a Longest Common Subsequence of at least four words, replacing each matching quote instance with [quote].Words are compared after capitalization and punctuation are removed; the original review’s capitalization and punctuation are retained.Source: §2.2.1 · p. 4

2.2.2 Anonymization

• A BERT named-entity recognizer and neuralcoref identify proper nouns, which are replaced by generic numbered names before training.Manual spot-checks of 200 replacements found no missed names.Source: §2.2.2 · p. 4

2.2.3 Final Processing

• Pairs with text under eight characters were removed (75,000 pairs, or 5.5%); remaining passages were truncated or padded to 512 tokens, and hosts determine if or when data are released.Texts are truncated at the token level, independently of sentence boundaries.Source: §2.2.3 · pp. 4–5

2.3 Analysis of Dataset Quality

• SC negative responses average twice the length of positive ones, while WP negative responses are only slightly longer; Figure 2 breaks lengths down by sentiment.The SC 75th percentile for negative responses nearly doubles as well.Source: §2.3 · p. 5, Figure 2 · p. 5
• Most negative SC responses received near-zero toxicity and insult scores from Detoxify, whose measures the authors use to identify potentially non-constructive critiques.The analysis compares SC with Writing Prompts, which the authors judge the closest existing dataset.Source: §2.3 · p. 5

3 Model Architecture and Training

• CARP uses separate text encoders for passages and critiques, projects their embeddings into a 2048-dimensional space, and scores their cosine similarity with a learned temperature.The temperature is clamped between ln(1/100) and ln(100) to stabilize training.Source: §3 · p. 6
• The three RoBERTa-based models are CARP-Tiny (58M parameters), CARP-Base (252M), and CARP-Large (715M), each with two encoder branches.Their total sizes are approximately twice those of the corresponding RoBERTa models.Source: §3 · p. 6
• Chunked loss computation accumulates gradients while reducing memory use, allowing larger contrastive batches; training used a batch size of 2048.Both modalities are first embedded without gradients, then embedded in chunks for loss calculation.Source: §3 · p. 6
• Validation loss and accuracy were periodically measured for all three models on 1,000 held-out samples, as plotted in Figure 3.Table 2 describes architecture performance after 1,400 training steps but the supplied text gives no cell values.Source: §3 · p. 6, Table 2 · p. 6, Figure 3 · p. 7
• At inference, Pegasus paraphrases each classifier four ways, and CARP averages the story-to-classifier cosine similarities as an ensemble score.Source: §3 · p. 6

4 Evaluation

• CARP is compared with three GPT-J baselines to assess which models human readers’ preferences more closely.The baselines include a fine-tuned seq2seq model and two zero-shot prompt-engineered models.Source: §4 · p. 7, §4.1 · p. 7

4.1 GPT-J Baseline Models

• The fine-tuned baseline generates critiques from passage–critique pairs, while two zero-shot baselines use prompts that mimic training data or pose a multiple-choice quiz.The multiple-choice baseline scores each class using negative log likelihood per byte of a hand-written review, then normalizes the scores.Source: §4.1 · p. 7

4.2 Experimental Design

• Participants selected from nine candidate reviews per story; votes were aggregated and normalized into a human review distribution.Participants were encouraged to select at least two reviews and selected 3.2 on average.Source: §4.2 · p. 8
• Models scored each candidate review for a story, using perplexity for prompted models and cosine similarity for CARP as proxies for negative log likelihood.Human and model score distributions were normalized by subtracting their minimum value and applying softmax.Source: §4.2 · p. 8

4.3 Human Evaluation

• Human evaluation involved 200 participants rating seven stories, with three stories assigned to each participant.The stories were collected from colleagues unfamiliar with CARP because the model’s transformer encoders could not handle sufficiently long stories.Source: §4.3 · p. 8

5 Results

• CARP’s mean performance improves as model size increases; CARP-L reaches cosine similarity of 0.9 in some cases.The authors hypothesize that performance gains from increased scale may continue, but frame this as a hypothesis.Source: §5 · p. 8
• The fine-tuned GPT-J model performs worst among the autoregressive models, and none approach the human baseline on any task.The paper leaves open whether autoregressive models are suitable for zero-shot story evaluation.Source: §5 · p. 8
• A similar evaluation was infeasible for STORIUM because it lacks a human-interpretable metric; Scarecrow’s nine preset evaluations limit its general use.The authors describe Scarecrow as automated and human-interpretable, but limited to nine predetermined evaluations.Source: §5 · p. 8
• CARP correlates more strongly with human evaluation than zero-shot methods and costs a fraction of human evaluation.Figure 4 compares CARP-L, CARP-B, and CARP-T with prompt-engineered and fine-tuned baselines using cosine similarity to the human distribution and per-story KL divergence.Source: §5 · p. 8, Figure 4 · p. 9

6 Conclusions and Future Work

• The paper presents both CARP and the Story-Critique dataset.The authors believe CARP may contribute to story evaluation models that avoid expensive annotations and benefit from scale.Source: §6 · p. 9
• Future work includes extending CARP to larger story contexts and expanding the dataset to include local and global critiques.The authors suggest multi-hop critique as a possible approach to accumulating critiques across hops.Source: §6 · p. 9

• Shahbuland Matiana implemented and trained CARP and worked on scaling; Leo Gao trained the GPT-J baseline models.Source: A Author Contributions · p. 12
• JR Smith created and cleaned the dataset, while Ryan Teehan handled data cleaning and obfuscation.Source: A Author Contributions · p. 12
• Louis Castricato conducted evaluation, designed prompt engineering, and oversaw the project; Stella Biderman advised experiment design and analysis, negotiated data use, and led writing.Spencer Frazier edited the paper and provided ethical oversight and assistance with dataset origination.Source: A Author Contributions · p. 12

B CARP Prompting

• Prompt engineering permits rapid prompt iteration without curating datasets or training models, and the paper describes several CARP-based zero-shot classifiers.Source: B CARP Prompting · p. 13
• Emoji or emoticon use can classify sentiment, with positive sentimental reviews serving as a signal that a passage has no major issues.Source: B CARP Prompting · p. 13
• A critique beginning with “...” or a [quote] token can act as a proxy for wording or grammatical errors.Source: B CARP Prompting · p. 13
• “Show, don’t tell” can classify whether a passage contains enough action, but it often requires global information and should be used sparingly.Source: B CARP Prompting · p. 13

C Baseline Model Prompting

• The GPT-J-6B prompt-engineered baseline uses both an open-ended prompt and a multiple-choice prompt.Source: C Baseline Model Prompting · p. 13

C.1 Sequence-to-Sequence Prompting

• The sequence-to-sequence prompt presents example story-and-critique pairs before placeholders for a target story and critique.Source: §C.1 · p. 13

C.2 Multiple-Choice Prompting

• The multiple-choice prompt asks the model to read four stories and answer classification questions, with the first story serving as an example.Source: §C.2 · p. 13
• One example story is labeled “a bit too short” among nine critique choices.Source: §C.2 · p. 13
• Another example labels a short story about cookies as “boring” among the available choices.Source: §C.2 · p. 14
• The final multiple-choice item leaves the target story and its answer blank after listing the nine choices.Source: §C.2 · p. 14

D Sequence-to-Sequence Generated Critiques

• Generated prompts were produced using a sequence-to-sequence model fine-tuned on the Story-Critique dataset, with top-p/top-k sampling configured at k=50 and p=0.9.Source: D Sequence-to-Sequence Generated… · p. 14
• Some generated critiques express confusion, including “I’m not sure what this means” and “I don’t understand this sentence.”Source: D Sequence-to-Sequence Generated… · p. 15
• The displayed story examples include a shift from a sports car to a motorcycle and a winter setting that later describes summer heat and bugs.Source: D Sequence-to-Sequence Generated… · p. 15
• Shown critique examples include suggestions to cut material, including one that says, “I think you can cut this.”Source: D Sequence-to-Sequence Generated… · p. 15
• Other critique examples suggest removing “very” and correcting “summers day” to “summer’s day,” while also flagging redundant description.Source: D Sequence-to-Sequence Generated… · p. 15