PaperlayerView exact revision on arXiv

Source-linked AI summary

What Breaks When Facts Become Stories? The Distortions Source-Agreement Metrics Miss When Models Retell Real Events

Kwangsuk Park, JiWoong Yang, Vaniushkina Kseniia

arXiv:2610.39500v1cs.AI

TL;DR

Agreement-based evaluation cannot distinguish warranted departures from distortions in reconstruction, where departure may be intended. The authors assess summaries and stories of 2,000 news articles across event axes and find that accountability distortion rises with story freedom, while judgments of its legitimacy remain contested.

  • Problem

    Existing factuality and overlap evaluation treats source agreement as success, although reconstruction may warrant departures and therefore needs a way to distinguish them from misinformation.

    Source: §1 · p. 2

  • Method

    The authors apply a judgeable warrant criterion to reconstructions of news articles and complement model judgments with rule-based indicators across event axes.

    Source: §2 · pp. 3–4

  • Results

    Accountability distortion rose six to sevenfold in stories versus summaries, and two in three cases with determinable direction lightened the perpetrator’s responsibility.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Judges and authors agreed more on whether distortion occurred than on whether it was warranted, so ReTold-2k preserves judgments unaggregated.

    Source: §7 · pp. 8–9

    Rerunning the procedure may not yield directly comparable figures because absolute levels shift with the judge models employed.

    Source: §6 · p. 8

Abstract

from arXiv · show

Language models are now widely used to turn recorded facts into media for public consumption, yet evaluation of such output still assumes agreement with the source as its measure of success. In reconstruction, where departing from the source can itself be the goal, this criterion measures the wrong construct: applied as is, it counts intended departures as errors; withheld, it lets real distortion through. We had three model families rewrite 2,000 news articles as summaries and as stories, measured what changed on every axis of the five Ws and how, and asked of each detected distortion whether the assigned perspective warranted it, which set a third of them aside. Time and manner were not lost but replaced, while place and causation lost the expressions that carry them. Accountability distortion behaved otherwise. Barely present in summaries, it rose up to sevenfold in stories, and of those whose direction could be determined, two in three lightened the perpetrator's responsibility. Such distortions survive claim-level fact checking and leave the retelling grammatically cleaner than its source. Where our judges disagreed, three authors relabeling the same cases converged on whether a distortion had occurred but not on whether it was warranted, a boundary contested rather than merely hard to detect. To support work on detecting and reducing them, we release ReTold-2k, a reconstructable corpus with the scoring protocol and per-judge baselines kept unaggregated.(https://github.com/Retold2K/retold2k).

1 Introduction

Source agreement alone cannot evaluate reconstructions in which departure may be intended: evaluation must distinguish warranted perspective-based changes from distortion. The paper introduces that distinction, measures changes across event axes, and finds that accountability distortion rises in stories.

1 Introduction

  • Agreement-based evaluation counts intended departures as errors when applied unchanged, yet can miss real distortion when withheld.The paper frames the needed criterion as whether each departure is warranted by the assigned perspective.Source: §1 · pp. 1–2
  • The study rewrites roughly 2,000 news articles as source-bound summaries and less constrained stories, judging changes across the five Ws and how.Narrators span perpetrators, victims, and outside observers; the assigned perspective warrants only changes within focalization.Source: §1 · p. 2
  • Accountability distortion was rare in summaries but rose up to sevenfold in stories, and two in three direction-determinable cases lightened perpetrator responsibility.Judges and authors converged on whether distortion occurred but disagreed over whether it was warranted.Source: §1 · p. 2
  • ReTold-2k releases the judging criterion, measurement procedure, per-judge baselines, and contested cases with three authors’ labels.The paper diagnoses evaluation methods rather than presenting a catalogue of model faults.Source: §1 · p. 2

2 Experimental Setup

The study pairs news articles with source-bound summaries and less constrained stories, then evaluates judged and rule-extracted changes. Its protocol checks whether detected changes are warranted by the assigned narrator.

2 Experimental Setup

  • The dataset contains 2,000 English news articles stratified across seven regions and four topic-severity levels.News provides verifiable source texts and explicitly reported event responsibility.Source: §2 · p. 2
  • Figure 1 distinguishes judged changes across six types from the remaining three axes measured by rule-based extractors.Only detected changes judged unwarranted receive target and direction labels.Source: Figure 1 · p. 3
  • All three model families summarized every article at a 0.55 compression ratio; paired comparisons matched each story with a summary from the same family, yielding 1,923 paired articles.The story pathway yielded 1,926 outputs from 2,000 requests.Source: §2 · p. 3
  • Judges assess six distortion types, determine whether each detected change is warranted, and record target and direction when accountability is implicated.A leave-one-generator-out protocol assigns outputs to the two model families that did not generate them.Source: §2 · p. 3
  • Table 1 names the six distortion types used to assess whether core event relations and other content change in reconstruction.The success conditions include retaining harmful agents, preserving cause-effect links, and attributing contested claims.Source: §2 · p. 3, Table 1 · p. 4
  • Rule-based extractors track anchors such as temporal expressions, place entities, causal connectives, and attribution cues without model judgment.They report anchor density, source-anchor retention, and output-anchor novelty, but cannot determine legitimacy.Source: §2 · pp. 3–4

3 Each axis breaks in its own way

The axis-level measurements show that reconstruction changes differ by type: some content is replaced, some thinned, and accountability distortion varies with task freedom. The task contrast persists across event-severity strata.

3 Each axis breaks in its own way

  • Accountability was the exception to the broader axis pattern: Vanishing Actor and Causal Fading were nearly absent in summaries but rose six to sevenfold in stories.Other axes were refilled or thinned rather than showing the same task dependence.Source: §3 · p. 4

3.1 Time and manner are replaced, place and causation thinned

  • Time and manner retained density while losing source anchors and gaining novel ones, whereas place density fell without a compensating rise in novelty.Causation and attribution also fell together; these rule-based measures track thinning but cannot judge legitimacy and may miss paraphrase.Source: §3.1 · p. 4

3.2 Freedom shifts the kind of distortion, not its amount

  • Distortion types common to both tasks did not rise overall with story freedom; instead, stories admitted accountability types, and 587 of 661 task-discordant articles were story-only.Role Reversal did not occur in summaries, and one third of detections were judged warranted and excluded from Figure 2 rates.Source: §3.2 · p. 4, Figure 2 · p. 5

3.3 Task freedom, not event gravity, drives the gap

  • The story-versus-summary accountability contrast held across all four severity strata, including neutral events with sparse accountability structure.The figure reports unwarranted detections by distortion type; its panels use different vertical scales.Source: §3.3 · p. 4, Figure 2 · p. 5

4 Accountability distortion is unseen by the measures in use and tilted toward the perpetrator

Accountability distortions can leave stories grammatically cleaner and pass claim-level fact checking, while still removing or shifting responsibility. Detected distortions often lighten the perpetrator’s responsibility.

4 Accountability distortion is unseen by the measures in use and tilted toward the perpetrator

  • Grammatical indicators scored stories above their sources, and claim-level fact checking found no contradiction, despite detected distortions that lightened responsibility.The agent may be missing even when the retelling contains no claim that conflicts with the article.Source: §4 · p. 5

4.1 Accountability distortion leaves the grammar cleaner

  • Stories had fewer passives, yet unwarranted Vanishing Actor was detected six times as often as in summaries.The agent can disappear even when every clause remains active and has a subject.Source: §4.1 · p. 5

4.2 Fact checking passes accountability distortion

  • Of 562 contradiction-free stories, 34.0% carried detected accountability distortion, including 27 with two judges in agreement.Vanishing Actor and Causal Fading work through deletion or severance, so they need not create a sentence that contradicts the source.Source: §4.2 · pp. 5–6

4.3 Accountability distortion tilts toward the perpetrator

  • In two of every three cases with a determinable direction, accountability distortion lightened the perpetrator’s responsibility.The pattern was observed across narrator groups, though close-to-victim and responsibility-assigning voices did not clear the majority threshold.Source: §4.3 · p. 6

5 The boundary divides judges and people alike

Judges and authors divide most sharply over whether responsibility-related changes are warranted, not simply whether distortion occurred. The released judgments preserve these differences rather than collapsing them into a single label.

5.1 The judges do not divide everywhere

  • On accountability judgments implicating the perpetrator, pairwise disagreement reached 65.9%, versus 2.7% when the perpetrator was not implicated.For surface distortion, perpetrator involvement made no difference; disagreement exceeded a third throughout.Source: §5.1 · p. 6

5.2 People divide in the same place

  • Three authors independently relabeled 50 cases on which judges had divided; unanimity was 40% for disagreements about whether distortion occurred, versus 10% for disagreements about its legitimacy.Source: §5.2 · p. 6
  • The authors’ legitimacy judgments showed below-chance Fleiss’ κ of −0.07, with one author mostly permitting and another mostly rejecting the same cases.The authors’ spread in strictness recurred among the model judges.Source: §5.2 · p. 7

5.3 We do not aggregate

  • The corpus keeps judgments unaggregated because eight of the fifty author-relabeled cases had no majority, and disagreement itself is informative.Figure 3 reports mitigation rates with 95% intervals by narrator group; narrator and article effects are not separated because each article has a single narrator.Source: §5.3 · p. 7, Figure 3 · p. 7

6 ReTold-2k

ReTold-2k preserves contested judgments and the materials needed to inspect or rerun their evaluation. Its benchmark consists of disputed cases, with judgments kept separately rather than reduced to a consensus label.

ReTold-2k

  • The release keeps each judge’s detection and legitimacy verdict separate, alongside accountability targets, direction, and verbatim source and output passages when applicable.For each item, the two judge records remain side by side.Source: §6 · p. 7
  • ReTold-2k’s benchmark is 911 contested judgments, with the three authors’ individual labels attached to the 50 cases they relabeled.The authors drew those 50 from cases where the model judges divided; they were not a random sample.Source: §6 · p. 7, Table 6 · p. 8
  • The release excludes article text and two other summary compression ratios, which the paper says its claims do not rely on.Shipping unused runs could confuse readers about the reported figures.Source: Table 7 · p. 8
  • Because the benchmark cases have no settled answer, the paper proposes reporting agreement against each judge separately and showing the spread across judges.Matching a majority-vote gold loses its meaning for this subset.Source: §6 · pp. 7–8
  • Rerun figures are not directly comparable when different judge models are employed, since their absolute levels shift.The per-judge labels and prompts are released in full; rule-based indicators remain comparable over time.Source: §6 · p. 8

7 Conclusion

Source agreement alone cannot evaluate reconstruction: it misses consequential distortions while treating warranted departures as errors. Accountability distortions rise with task latitude, and judgments about whether departures are warranted remain contested.

7 Conclusion

  • Six to sevenfold higher in stories than summaries, accountability distortion often lightened perpetrators’ responsibility, escaped current measures, and appeared in one-third of fact-check-passing stories.Across the full set, grammatical indicators scored stories above their sources; verification should track how loose the task is.Source: §7 · p. 8
  • Model judges and human authors agreed that distortions occurred but disagreed about whether they were warranted, especially when perpetrator responsibility was at stake.The corpus preserves these judgments separately because aggregating them would erase the contested boundary.Source: §7 · pp. 8–9
  • ReTold-2k keeps judgments unaggregated, judge by judge, rather than resolving contested judgments into a single score.A measure returning one number would decide in advance what counts as an error.Source: §7 · p. 9

A Data and setup

The study pairs news articles with generated summaries and narrator-based stories, then evaluates departures against perspective-specific rules. ReTold-2k retains the two judging families’ records side by side.

A Data and setup

  • The corpus setup stratifies articles across seven regions, four topic-severity levels, and narrator classes.Region and severity are assigned per article, with one narrator assigned to each article.Source: Table 8 · p. 10, A Data and setup · p. 10

B Generation

  • Summaries were generated at a 0.55 compression ratio without an assigned narrator, while story prompts requested roughly 200 words and produced 194-word stories versus 304-word paired summaries.Source: B Generation · p. 10

C Judging

  • Judges separately assessed whether a departure occurred and whether the assigned perspective warranted it; summary departures were always unwarranted.For stories, only departures remaining after warranted changes were set aside contributed to the reported comparison.Source: C Judging · pp. 10–11

D Record format

  • ReTold-2k stores each item as side-by-side records from two judging families instead of resolving their judgments into one verdict.The example shows both judges detecting the same distortion as unwarranted while assigning different targets and directions.Source: D Record format · p. 11

E Contradiction-rate audit, by stratum

The audit applies separate detection and legitimacy judgments, finding accountability distortions even among stories that do not contradict their sources. Regional differences do not explain the contrast, while declined requests are excluded from detection rates.

E Contradiction-rate audit, by stratum

  • 24.9% of contradiction-free neutral stories carried detected accountability distortion, compared with 34.0% of contradiction-free stories in the graded strata.The neutral-stratum audit found 209 of 481 stories contradiction-free; the blind spot persisted despite lower absolute levels.Source: E Contradiction-rate audit, by stratum · p. 11
  • Judges first detect each departure, then assess whether the assigned perspective warrants it; responsibility-structure changes are never licensed, and summary departures are unwarranted.The prompt permits story changes in style, voice, order, emotional coloring, and plausible omissions or compression.Source: E Contradiction-rate audit, by stratum · p. 12
  • The audit codes agency removal, relation inversion, attribution stripping, and causal dilution as distinct distortions.These respectively cover disappearing agents, changed perpetrator-victim roles, removed claim attribution, and weakened cause-effect links.Source: E Contradiction-rate audit, by stratum · p. 12
  • Selective omission and information injection are separate codes, while paraphrase and lexical generalization are not flagged.The injection rule excludes narrator inner states, feelings, and generic scene color, but not new factual claims about the event or blame.Source: E Contradiction-rate audit, by stratum · p. 12
  • In the sample record, both judges detected unwarranted causal fading but assigned different targets and directions.The records retain the judges’ evidence fields verbatim.Source: Table 9 · p. 13

F Regional breakdown

  • Regional differences in retention and novelty were small relative to severity-stratum differences and had no consistent direction.The paper therefore claims only that region does not account for the contrast.Source: F Regional breakdown · p. 13

G Declined requests

  • 74 of 2,000 storification requests were declined, leaving no output to judge and excluding them from detection rates.Refusal categories were assigned from surface cues and were not checked by the authors.Source: G Declined requests · p. 13

H Direction by individual narrator

Perpetrator-targeting distortions generally mitigate responsibility, but the direction varies across narrator roles. The narrator-level table reports rates and confidence intervals for that comparison.

H Direction by individual narrator

I Significance tests

The summary–story difference in unwarranted accountability distortions is statistically significant overall and across all topic-severity levels. Judge disagreement is concentrated in cases where the perpetrator is implicated, and judge-specific story-to-summary ratios are at least 3.4 across nine cells.

I Significance tests

  • p < 10−99 overall, the paired summary–story difference in unwarranted accountability distortions is significant; topic-severity tests range from p < 10−15 to 10−33.The reported tests are p < 10−15 for neutral events, 10−29 for low, 10−33 for medium, and 10−27 for high severity.Source: I Significance tests · p. 14
  • Judge disagreement is 58.8% when the perpetrator is implicated versus 3.1% when not, significant under a χ2 test at p < 10−200.Source: I Significance tests · p. 14

J Exact figures for Figure 2

  • At least 3.4, each judge’s story-to-summary ratio for unwarranted detections across all nine task cells, using that judge’s own summary baseline.Table 11 reports mean ± s.d. over three judges; the ± reflects differences in judge strictness, not run-to-run variation.Source: Table 11 · p. 14
Loading 2610.39500v1…