PaperlayerView exact revision on arXiv

Source-linked AI summary

AI-Enabled Human Memory Manipulation: Misleading AI-Generated Summaries Distort Human Memory

Mattea Sim, Yael Eiger, Tadayoshi Kohno

arXiv:2609.28820v1cs.CY

TL;DR

AI summaries are used in high-stakes settings, but human oversight assumes people can catch their mistakes. This paper analyzes video summaries and tests misleading summaries’ effects on memory, finding frequent errors and memory distortion.

Abstract

from arXiv · show

AI-generated summaries are increasingly used in high-stakes settings, like policing, despite considerable evidence that AI often generates misleading or inaccurate information. This research asked: do errors in AI-generated summaries distort human memory? To answer this question, we adopted two methodological approaches. First, we conducted an analysis of AI summary output, prompting large language models to generate summaries of videos. This analysis quantified how often AI summaries contain errors and the categories of these errors, revealing the kinds of misleading information that may distort human memory. Second, we conducted a human-subjects experiment to test the impact of misleading information in AI-generated summaries on human memory. Participants were first exposed to an event via watching a video of a car-pedestrian accident, and later read an AI-generated summary describing the video that either contained misleading or accurate information. Participants' memory for the original event was assessed in a memory recognition test. In the AI analysis, we found a high frequency of mistakes in AI summaries, and particularly frequent omissions of critical details. For instance, the majority of summaries omitted the most central event of the video, a critical error which is likely to be impactful. We also observed a strong effect of AI misinformation on human memory. People who read a misleading AI summary were significantly less likely to accurately recall the original event, compared to people who read an accurate AI summary. These findings have implications for how AI should be used in critical settings. Though "humans-in-the-loop" are often expected to correct for AI's mistakes, our work suggests human memory can instead be distorted by these mistakes. AI has the potential to generate misinformation, even absent any adversarial intent, which can meaningfully impact human memory.

Introduction

The introduction asks whether errors in AI-generated summaries can distort human memory, challenging the assumption that human oversight will reliably catch AI mistakes. It reports frequent summary errors and an AI-enabled misinformation effect, with implications for high-stakes human–AI collaboration.

Introduction

  • Because AI summaries are being adopted in high-stakes settings where people are expected to check them, the findings raise concern that AI mistakes can instead distort human memory.The paper frames both AI output and human memory as fallible in critical settings.Source: Introduction · pp. 1–2
  • The researchers prompted large language models to summarize two brief, straightforward videos used in classic memory research, then analyzed error frequency and categories.The analysis was limited in video range but designed to examine AI errors in depth.Source: Introduction · pp. 1–2
  • In the human-subjects experiment, participants watched an event, read an AI-generated summary with accurate or misleading information, and then took a memory test.The researchers also varied whether participants were told the summary was AI-generated or human-written and measured their trust in AI.Source: Introduction · p. 2
  • People who read misleading AI-summary information had distorted memories, regardless of whether they believed the summary was AI- or human-written or how much they trusted or used AI.The summaries were identical across authorship-belief conditions and were generated by ChatGPT.Source: Introduction · p. 2
  • AI-generated summaries showed frequent errors across categories, with critical-detail omissions appearing frequently in each summary analyzed.The introduction identifies omissions as errors that can likely degrade human memory.Source: Introduction · p. 2

Background and Related Work

Prior work shows misleading information can reshape and sustain false memories, while AI systems and summaries can make consequential errors. This study connects these literatures by examining memory-relevant AI summary misinformation and whether source beliefs matter.

Misinformation and False Memories

  • People’s recollection of events can be altered after exposure to inaccurate or misleading information; in a forced-choice task, misleading post-event information made participants more likely to misremember a stop versus yield sign (Loftus, Miller, and Burns 1978).A crash description using “smashed” versus “hit” can alter later memory for car speed.Source: Misinformation and False Memories · p. 2
  • Misinformation effects can persist despite warnings or instructions to disregard post-event information, making implanted false memories difficult to correct (Eakin, Schreiber, and Sergent-Marshall 2003; Lindsay 1990; cf. Highhouse and Bottrill 1995).Source: Misinformation and False Memories · p. 2

AI Misinformation and Its Impacts on People

  • LLMs can hallucinate, perpetuate bias, omit relevant information, and make inaccurate claims, while AI summaries show ordering errors, omissions, and baseless inferences (Rawte et al. 2023; Xu, Jain, and Kankanhalli 2024; Oukelmoun et al. 2025; Ramprasad, Ferracane, and Lipton 2024).Such errors also appear in medical summaries, where omissions and hallucinations could have clinical impact.Source: AI Misinformation and Its Impacts on… · pp. 2–3
  • Prior video-summary research documents hallucinations and omissions; this study is the first to categorize AI-summary errors in ways relevant to human memory effects (Rawal et al. 2025).The authors distinguish this contribution from large-scale analyses of model behavior, already studied by other researchers (Li, Im, and Fazli 2025; Patel et al. 2026; Rawal et al. 2025; Wang et al. 2024).Source: AI Misinformation and Its Impacts on… · p. 3
  • AI exposure can affect decisions, biases, beliefs, and attitudes; people also overlook nearly 40% of errors in ChatGPT answers and can be influenced by misleading chatbots or AI-edited images (Wilson et al. 2025; Sim et al. 2025; Kabir et al. 2024; Pataranutaporn et al. 2025a, 2025b).AI-generated persuasive messages may be perceived as stronger than human-generated ones.Source: AI Misinformation and Its Impacts on… · p. 3
  • Because people may view AI as error-prone yet also reliable, objective, or preferable, the study explores whether susceptibility to misleading information depends on whether its supposed source is AI or human (Kabir et al. 2024).Source: AI Misinformation and Its Impacts on… · p. 3

Methodology

The methodology combined an analysis of errors in AI-generated video summaries with a human-subjects experiment testing whether misleading summaries distort memory.

Methodology

  • The study paired analysis of AI-summary errors with a human experiment testing their effect on memory.The AI analysis quantified and categorized errors, while the experiment tested an AI-enabled misinformation effect.Source: Methodology · p. 3

AI Summary Analysis Procedure

  • Researchers tested ChatGPT-5.5 and Gemini 2.5 Flash-Lite on two 25-second accident videos, prompting each model five times per video to produce 20 summaries.The videos were also used in the human study; repeated runs enabled examination of error consistency across outputs.Source: AI Summary Analysis Procedure · pp. 3–4

Videos.

  • The two animated videos depicted a red car turning right at an intersection, colliding with a pedestrian, and the driver and pedestrian meeting afterward; they differed by stop versus yield sign.The videos were created in a recent replication of a classic memory paradigm (Wiechert et al. 2026; Loftus, Miller, and Burns 1978).Source: Videos · p. 4

Prompting the Models.

  • A single prompt asked models for neutral, factual summaries of at least 300 words, including all important events and avoiding speculation or new facts.Preliminary tests informed the prompt; these instructions were intended to give models their best chance of producing accurate summaries.Source: Prompting the Models · p. 4

Coding Analysis.

  • Two researchers independently coded each summary for omissions, inaccurate details, and hallucinated additions, then resolved disagreements by reviewing the videos.The error scheme distinguished failures to mention 19 central details, inaccuracies about those details, non-critical inaccuracies, and additions not present in the event.Source: Coding Analysis · p. 4

Human Memory Study Procedure

  • Participants watched a car-pedestrian accident video, read a summary that varied whether the car stopped at a stop or yield sign, and completed a recognition test.The conceptual replication and extension also examined whether believing the summary was AI- or human-generated influenced memory (Loftus, Miller, and Burns 1978; Wiechert et al. 2026).Source: Human Memory Study Procedure · pp. 4–5

Part 1: Encoding the Original Event.

Participants watched an accident video, later read a summary with manipulated source and traffic-sign information, and then completed a memory recognition test. The study used randomized sign videos and recruited U.S. adults through Prolific.

Part 1: Encoding the Original Event.

  • Participants watched one of two 25-second animated accident videos, which differed in whether the car approached a stop or yield sign, and answered a car-color attention check.The sign was counterbalanced across videos; only participants who correctly identified the car as red were invited to the study’s second part.Source: Part 1: Encoding the Original Event · p. 5

Part 2a: Post-Event Summary.

  • Researchers repeatedly re-prompted ChatGPT to select a higher-quality summary, a process they noted might not represent typical summarization use.They considered the selected summary a valid ChatGPT output, while acknowledging that they were trying to ensure it included the accident’s central details.Source: Part 2a: Post-Event Summary · p. 5
  • Participants were randomly assigned to an AI or human summary label and to a summary whose stop- or yield-sign description was consistent or misleading.The 21-sentence summary was generated by ChatGPT and otherwise kept consistent across participants; minor edits addressed differences between the two videos.Source: Part 2a: Post-Event Summary · p. 5

Part 2b: Remembering the Event.

  • Participants completed a 12-item, two-choice recognition task, including a critical question asking whether the car stopped at a stop sign or yield sign.Ten questions were taken from prior work (Wiechert et al. 2026).Source: Part 2b: Remembering the Event · p. 6

Participants.

  • Of 360 U.S.-based adults recruited for part 1, 328 participants were included in the analyses after attention-check exclusions.The four conditions included 80–83 participants each.Source: Participants · p. 6

Results

AI-generated video summaries frequently contained errors, especially omissions of central events, and misleading summaries reduced participants’ memory accuracy. Susceptibility to misinformation did not significantly depend on whether a summary was labeled AI-written or human-written, participants’ trust in AI, or their AI-use frequency.

AI Summary Errors Are Frequent and Irregular.

AI Summaries Omit Critical Details.

  • Summaries omitted 51.6% of central details on average; 95% omitted the collision, and 90% omitted details of the pedestrian lying down and standing up afterward.The omitted-detail percentages are reported across prompts and models.Source: AI Summaries Omit Critical Details · p. 6, Table 3 · p. 7

AI Summaries Misconstrue Critical and Peripheral Details.

  • 65% of summaries misdescribed at least one central detail; central inaccuracies averaged 1.15 per summary (6.1% of central details), while peripheral inaccuracies averaged 2.05 per summary.Central errors included inaccurate interactions, movements, and event sequences; Gemini appeared more prone to them than ChatGPT.Source: AI Summaries Misconstrue Critical and… · p. 7

AI Summaries Hallucinate and Directly Contradict or Alter Central Details.

  • 60% of summaries included hallucinations or additions, averaging 1.6 per summary, including invented actors or movements and contradictions about whether the pedestrian entered the roadway.The contradiction concerned a key sequence leading to the car-pedestrian accident.Source: AI Summaries Hallucinate and Directly… · p. 7

A Misleading AI Summary Distorts Human Memory.

  • 83.6% of participants who read a consistent summary answered the critical question correctly, compared with 44.8% who read a misleading summary.The relationship between information condition and correct answers was significant (χ2 = 53.937, p < .001).Source: A Misleading AI Summary Distorts Human… · p. 7, Figure 1 · p. 8

People are Similarly Susceptible to AI and Human Misinformation Regardless of Their Trust in AI.

  • Correct-answer rates did not significantly differ by whether summaries were labeled AI-written or human-written, and susceptibility was not significantly associated with AI trust or chatbot-use frequency.The misinformation effect was observed in both label conditions; neither AI trust nor reported AI-use frequency significantly interacted with the experimental conditions.Source: People are Similarly Susceptible to AI… · p. 8, Figure 1 · p. 8

11 non-critical questions.

Ratings showed a labeling effect on perceived clarity: participants judged identical summaries as clearer when labeled human rather than AI. Ratings of readability did not differ significantly, and Information Condition and its interaction with Summary Condition had no significant effects.

11 non-critical questions.

  • There were no significant effects of Information Condition or interactions between Summary Condition and Information Condition on ratings of clarity, readability, or enjoyment.Source: §11 · p. 8
  • Perceived clarity was higher for the human-labeled summary (M = 9.01, SD = 1.07) than the AI-labeled summary (M = 8.62, SD = 1.47), although the summaries were identical.The effect of Summary Condition was significant, F(1, 326) = 7.800, p = .006; t(326) = −2.792, p = .006, d = −0.31, 95%CI[−0.53, −0.09].Source: §11 · p. 8
  • The human-labeled summary was descriptively more readable (M = 9.16, SD = 1.14) than the AI-labeled summary (M = 8.97, SD = 1.34), but the difference was not significant (p = .159, d = −0.16).Source: §11 · p. 9

Discussion

The authors conclude that errors in AI-generated summaries can distort human memory, including when people expect humans to catch AI mistakes. The findings raise concerns about AI use in high-stakes settings and motivate further research on error types, real-world effects, and possible interventions.

Discussion

  • Misleading AI-summary information distorted memory, and susceptibility did not depend on whether participants believed the source was AI or human, or on their trust or familiarity with AI.The authors caution that the trust and familiarity findings were statistically non-significant, so they should be interpreted cautiously.Source: Discussion · p. 9

AI Summary Errors in Human Memory Context

  • AI summaries frequently erred, most commonly by omitting central details; misleading inaccuracies and hallucinated additions may also affect memory.Omission errors connect to findings that people recall critical details less accurately after receiving no information than after accurate information; erroneous additions may create false memories.Source: AI Summary Errors in Human Memory… · p. 9

Implications for AI Summaries in Practice

  • Human-in-the-loop oversight may not prevent harm: subtle summary errors can distort memory in high-stakes settings like policing, even without malicious intent.The authors argue that AI can generate misleading information without an adversary, raising risks for human cognition.Source: Implications for AI Summaries in… · pp. 9–10

Open Questions and Limitations

  • The AI-output analysis was limited to two simple accident videos, one prompt, and a small number of summaries, constraining generalizability despite errors from both ChatGPT and Gemini.The authors call for research using more realistic videos and summaries.Source: Open Questions and Limitations · p. 10
  • The controlled experiment used one misinformation type and identical ChatGPT summaries, limiting conclusions about other sources, media, errors, and real-world settings.Future research could test audio-transcript summaries, more realistic high-stakes contexts, and other error types such as omissions.Source: Open Questions and Limitations · p. 10
  • Future work could test discrepancy detection or warnings to reduce misinformation effects, but warnings may fail, prompting the question of whether AI should be used in critical contexts.The authors note that noticing discrepancies can reduce memory effects, while prior findings show warnings are not always effective.Source: Open Questions and Limitations · pp. 10–11

Conclusion

  • The authors conclude that AI-summary errors can distort memory, leaving people tasked with correcting AI mistakes vulnerable to false memories in high-stakes settings.Source: Conclusion · p. 11

Ethical Statement

The study received institutional ethics approval, participants were debriefed after learning its full purpose, and exploratory screening excluded videos with graphic content.

Ethical Statement

  • IRBs at all authors’ institutions approved the study; participants learned its full purpose and were debriefed afterward, and exploratory screening excluded graphic videos.The researchers initially explored videos without restrictions but excluded graphic material, such as real police body-camera footage.Source: Ethical Statement · p. 11

Researcher Positionality Statement

The authors describe a cross-disciplinary team whose backgrounds shape their approach to the research questions.

Researcher Positionality Statement

  • The research team’s backgrounds span AI ethics, computer security and privacy, and social psychology.The authors state that these backgrounds influence how they approach the research questions.Source: Researcher Positionality Statement · p. 11

Appendix

The appendix provides the human-subjects study materials, further details on AI summary generation, and supplementary results.

  • The appendix includes full human-subjects study materials, additional information about AI summary generation, and supplementary results.Source: Appendix · p. 13

Human-Subjects Materials and Measures

This section presents the full instructions, materials, and measures shown to participants in the human-subjects study.

Study Part 1 … Additional AI Summary Analysis Results

This block describes a two-part eyewitness-memory study in which participants watched an accident video, later read assigned summaries, and completed a recognition test, alongside a tally of error categories in AI summaries.

Instructions.

  • In part 1, participants watched an animated car-pedestrian accident as eyewitnesses and were instructed to watch closely because they would answer questions later.The study instructions noted that no audio was needed and invited eligible participants to a separately compensated second part.Source: Instructions · pp. 13–14

Additional AI Summary Analysis Results

Attention Check. / Study Part 2

  • Twenty-four hours after part 1, participants who identified the main car as red and reported never having seen the video were invited to part 2.Part 2 remained available for up to 24 hours or until enough participants completed it.Source: Attention Check · p. 14, Study Part 2 · p. 14

Summary Conditions.

  • Participants were randomly assigned to an AI- or human-labeled summary and to a stop- or yield-sign version, making sign information consistent or misleading depending on their video.Each summary sentence appeared on a separate page with a one-second minimum display time.Source: Summary Conditions · p. 14, Video Summary · p. 14

Video Summary.

  • The summary described a red car turning at an intersection, striking a pedestrian, stopping, and the driver and pedestrian meeting beside the car.The presented version varied the traffic sign between stop and yield.Source: Video Summary · pp. 14–15

Attention Check and Filler Questions.

Memory Recognition Task.

  • In the recognition task, participants answered questions about the original video, including the traffic sign, car color, collision details, and scene features.Question and response-option order was randomized; the correct sign answer depended on the video assigned.Source: Memory Recognition Task · p. 15

Summary Generation for Human Study

  • Researchers selected a relatively accurate ChatGPT summary of the yield-sign video, edited details that varied between videos, and restored selected AI-generated details.Restored details included the pedestrian’s yellow shirt and the red stop sign for participants assigned the stop-sign video.Source: Summary Generation for Human Study · p. 15
Loading 2609.28820v1…