PaperlayerView exact revision on arXiv

Source-linked AI summary

"Nothing to See Here'': Unintended Disclosure through Revision Traces of LLM Deliverables

Yage Zhang, Yukun Jiang, Yang Zhang

arXiv:2609.35408v1cs.CRcs.AI

TL;DR

LLM assistants can remove or replace an item yet reveal it in their account of the edit, allowing a recipient to recover withdrawn content. The paper analyzes public conversations and introduces RevLeakBench, a controlled benchmark with conversation and agent tracks. Across six models, about half of revocation deliverables state the edit, and readers recover the withdrawn item from about 13% of deliverables.

  • Problem

    An LLM may remove a withdrawn item from intended content but reveal it in its account of the edit, allowing a recipient to recover it.

    Source: §1 · p. 1

  • Method

    The authors analyze three public conversation corpora and introduce RevLeakBench, a controlled benchmark of 100 tasks across five scenarios in conversation and agent tracks.

    Source: §1 · p. 2

  • Results

    Across six models, about half of revocation deliverables state the edit, and readers recover the withdrawn item from about 13% of deliverables.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The compared methods substantially reduce recovery, with different tradeoffs in delivery and required-content retention.

    Source: §1 · p. 2, §6 · p. 7

    RevLeakBench covers five scenarios, six models, and conversation and agent settings, but not multilingual, long-horizon, or executable-code settings.

    Source: §6 · p. 8

Abstract

from arXiv · show

Large language model (LLM) assistants increasingly help users draft content for third-party recipients. During private drafting, the user or the model may introduce an item and later remove or replace it. The model may remove the item from the intended content but reveal it again when stating the edit. We call such statements revision traces. For example, after a user removes the password before sharing a configuration file, the model may delete it but leave a comment saying, "Removed the password 'No****4!' as requested." A third-party recipient who sees only the delivered file can therefore recover the withdrawn password from the comment. In an in-the-wild analysis of three public conversation corpora, we identify 26,753 revision requests, of which 2,363 (8.8%) leave revision traces. We study them in greater depth under controlled conditions by introducing RevLeakBench, a benchmark of 100 tasks across five scenarios with a conversation track and an agent track. We measure trace occurrence, withdrawn-item recovery, trace position, and required-content retention. Across six models, about half of the deliverables in both tracks state the edit after a revocation, and a reader that sees only the deliverable can recover the withdrawn item from about 13% of them. Telling the model that its entire reply will be forwarded to the recipient still leaves revision traces in 36.4% of the deliverables. We compare prompt defenses and a delivery boundary, and propose an output-side filter that sharply reduces recovery with little loss of required content. We believe our work can benefit efforts to understand and mitigate unintended disclosure in LLM interactions.

1 Introduction

Revision traces occur when a model removes or replaces an item but discloses it again while describing the edit. This paper measures that disclosure in public conversations and controlled tasks, then evaluates ways to reduce it.

1 Introduction

  • A model can replace a password with [removed] yet repeat it in a comment, allowing the recipient to recover it from the delivered code.The recipient sees the deliverable, not the private drafting conversation that produced it.Source: §1 · pp. 1–2
  • Prior privacy, redaction, and deletion evaluations assess information remaining in or inferable from final outputs, but do not separate retained content from revision-trace disclosure.The paper asks how often deliverables state a revision in the wild and, under controlled conditions, how much traces reveal and which defenses reduce disclosure while retaining needed content.Source: §1 · p. 2
  • 2,363 of 26,753 revised replies (8.8%) leave traces, and RevLeakBench finds that readers recover withdrawn items from about 13% of deliverables across its two tracks.Explicit instructions to forward the entire reply still leave traces in 36.4% of deliverables; the body-only boundary and output-side filter improve end-to-end rates in both tracks.Source: §1 · p. 2
  • The paper contributes an in-the-wild analysis, RevLeakBench, and comparisons of prompt defenses, delivery boundaries, and an output-side filter.The benchmark pairs revisions with the same final requirements without revision history and measures traces, withdrawn-item recovery, and required-content retention.Source: §1 · p. 2

2 Background and Related Work

The related work spans privacy and deletion evaluations, document revision histories, and reasoning traces. This paper distinguishes its setting by studying model-written accounts of removals or replacements in recipient-facing deliverables.

2 Background and Related Work

  • Existing privacy and deletion work evaluates inappropriate disclosure, removal, redaction, or recovery, while revision-history studies examine information exposed by document artifacts and reasoning traces.The paper’s setting differs because the model itself writes an account of a removal or replacement into the recipient-facing deliverable.Source: §2 · pp. 2–3

3 Revision Traces in the Wild

The in-the-wild analysis screens three public conversation corpora for revision requests and traces. It finds traces in 8.8% of identified revision events, with rates varying by revision type, and documents a trace reaching a public repository artifact.

3 Revision Traces in the Wild

  • The analysis covers WildChat, ShareChat, and DevGPT, including developer conversations linked to artifacts in public GitHub commits, pull requests, and issues.The screening selects change-request turns with a keyword rule, then checks conversation history and the turn to determine whether a revision occurred.Source: §3 · p. 3
  • 2,363 of 26,753 revised replies (8.8%) contain traces, with rates of 8.6–11.3% across corpora; in WildChat, traces occur in 36.8% of revocations versus 5.4% of replacements.The screening check identifies revision requests and classifies whether they withdraw, replace, correct, reformat, or extend an earlier draft.Source: §3 · p. 3
  • A DevGPT assistant’s edit explanation reached public repository history after the developer pasted the conversation into a commit message.The delivered code replaced onclick attributes, while the assistant’s comment stated that it had removed them.Source: §3 · p. 4
  • Because traces often appear outside the reply body, the study treats the full reply as the deliverable and reports the body separately.A recipient who receives the full reply sees both the body and the material outside it.Source: §3 · p. 4

4 RevLeakBench

RevLeakBench extends the public-corpus analysis with controlled tasks that measure trace occurrence, disclosure, and retention. Its design spans five scenarios, two tracks, and paired conditions that separate revision history from a shared final requirement.

4 RevLeakBench

  • The benchmark studies controlled disclosure and defenses because public conversations do not annotate withdrawn items and cannot measure how much traces reveal.RevLeakBench contains 100 tasks across five scenarios, with 20 tasks per scenario, adapted from existing benchmarks.Source: §4 · p. 4, §4.1 · p. 4

4.1 Overview

  • Each task has conversation and agent tracks and tests revocation, replacement, or model-added removal, with the withdrawn item required to stay out of the deliverable.The conditions are motivated by revision patterns observed in the in-the-wild analysis.Source: §4.1 · p. 4

4.2 Tasks

  • Across the 200 candidate withdrawn items, 6 were security credentials, 72 personal-information items, 77 organizational-information items, and 45 were content the user chose not to deliver.The five scenarios are adapted from AgentCIBench, DRBench [2], RedacBench, DiscoveryBench and InfiAgent-DABench [18] [28], AgentDAM and PrivacyAlign [47] [59].Source: §4.2 · p. 4

4.3 Conditions

  • Each revocation is paired with a static-exclusion control and each replacement with a replacement control, holding final requirements constant while removing revision history.Conversation-track models generate the final turn, whereas agent-track models work with files in a workspace and deliver their first reply without a tool call.Source: §4.3 · p. 4

5 Experiments

The experiments measure revision traces and recovery across controlled tasks, compare revocation with matched controls, and test defenses. Traces are common, and readers recover withdrawn items from a notable share of deliverables.

5.1 Setup

  • Across six models, the setup evaluates withdrawn-item inferability and required-content preservation in conversation and agent deliverables.Presence uses source-benchmark checks where available; utility uses RedacBench’s checkPropositions.Source: §5.1 · p. 4

5.2 Revision Traces

  • Trace rates and recovery differ by scenario: revocation and model-added removal reach 53.4–56.5% full-reply trace rates, while replacement is 31.8%.Model-added removal has lower recovery, at 6.0–6.5%, than revocation, at 11.9–15.2%.Source: §5.2 · p. 5
  • Full-reply traces occur in 56.9% of conversation-track and 50.3% of agent-track deliverables, while body rates are 4.6% and 13.8%.Most traces sit around the body, so the difference between scopes captures what forwarding the reply unedited adds.Source: §5.2 · p. 5, Table 2 · p. 5
  • Readers recover withdrawn items from 13.7% of conversation-track and 13.4% of agent-track deliverables; deleting the trace sentence reduces recoveries from 303 of 1,239 deliverables to 24.Deleting an unrelated sentence of comparable length leaves 293 recoveries.Source: §5.2 · p. 6

5.3 Matched Controls

  • Revocation raises trace rates over static exclusion in most scenarios, with increases of 19–33 percentage points in enterprise briefs, coordination messages, and software artifacts.Turn-matched controls show added turns alone do not raise trace rates; late exclusion raises them, but revocation raises them further.Source: §5.3 · p. 6

5.4 Defenses

  • The output-side filter reduces recovery from 13.7% to 1.3% in conversation and from 13.4% to 3.1% in agent deliverables, with small paired utility changes.The structured delivery boundary nearly eliminates conversation-track disclosure among successful deliveries, but 17.8% of planned requests produce no valid deliverable.Source: §5.4 · p. 7, Table 4 · p. 7
  • A forwarding notice lowers full-reply trace rates from 59.8% to 36.4%; a deliverable-only instruction lowers them from 59.7% to 27.5%.The deliverable-only instruction reduces recovery from 15.3% to 4.7%, but traces remain inside the body.Source: §5.4 · p. 7, Table 5 · p. 7

6 Discussion and Limitations

The discussion finds that revision-trace risk depends on revision history, task wording, and delivery format, and describes evaluation and deployment boundaries for the study and its filter.

Discussion

  • Matched controls indicate that revocation increases withdrawn-item recovery beyond the effect of an exclusion stated only in the final turn.Moving an exclusion to the final turn also increases trace frequency.Source: §6 · p. 7
  • Revision-trace behavior varies with task and exclusion wording: reduced wording changes policy-constrained documents most, while other scenarios lack the same consistent pattern.Trace rates should be interpreted alongside the instructions, policy language, and source labels shown to the model.Source: §6 · pp. 7–8

Limitations

  • The public-corpus 8.8% rate applies to revision events identified in publicly released conversations by a keyword-based screening pipeline.The controlled study complements this with matched conditions that fix the withdrawn item and final requirements.Source: §6 · p. 8
  • Several measurements use automatic judges validated against human annotations and alternative judges; utility measures required-content retention, while defenses also report delivery coverage and end-to-end success.Source: §6 · p. 8
  • The filter’s results characterize mitigation only when recorded aliases and required-content annotations are available; deployment would require obtaining them from the interaction or another component.Source: §6 · p. 8

7 Conclusion

The paper studies disclosure through revision traces in LLM deliverables and introduces RevLeakBench as a controlled setting for examining their occurrence, disclosure, and mitigation. It aims to support deeper understanding and more effective mitigations for unintended disclosure in LLM interactions.

Conclusion

  • The paper studies unintended disclosure through revision traces in LLM deliverables.Source: §7 · p. 8
  • RevLeakBench provides a controlled setting to study when revision traces occur, what they reveal, and how they can be reduced.Source: §7 · p. 8
  • The authors hope the work supports deeper understanding of unintended disclosure and the development of more effective mitigations for LLM interactions.Source: §7 · p. 8

Ethics Statement

The study derives controlled tasks from published benchmarks and uses public conversations under their release terms, with measures to limit unnecessary exposure of sensitive details and user identities.

Ethics Statement

  • Controlled tasks derive from published benchmarks, with withdrawn items annotated by the source benchmark or selected from its source material.Source: Ethics Statement · p. 8
  • Reader evaluations expose only the recipient-visible deliverable, without access to the private drafting conversation or the ability to query the assistant.Source: Ethics Statement · p. 8
  • The public-conversation analysis uses released conversations, short necessary excerpts, and paraphrasing or redaction to reduce re-identification risk; it does not identify users or link identities across sources.Source: Ethics Statement · p. 8

Reproducibility Statement

The reproducibility materials describe the benchmark artifacts, public examples, task construction, generation setup, and evaluation procedures, including scenario-specific checks.

Reproducibility Statement

  • Supplementary materials include the 100 tasks, rendered conversations and agent materials, matched conditions, checks, runner, defenses, and filter; evaluation code is also available.Source: Reproducibility Statement · p. 8

A In-the-Wild Examples

  • In-the-wild examples include traces in revision explanations and deliverables, with two DevGPT cases showing traces copied or linked into public repository artifacts.Source: A In-the-Wild Examples · p. 11

B Scenario Categories

  • RevLeakBench maps its scenarios to existing task and writing taxonomies, while excluding taxonomy categories such as editing or critique of user text, literature, education, and marketing writing.Source: B Scenario Categories · p. 11

C Task Construction by Scenario

  • Tasks draw on benchmark materials across five scenarios, with scenario-specific selection and construction of withdrawn items, required findings, and final requirements.Task review checks comparable support for candidate items, avoids repetition, and verifies that required content remains achievable after withdrawal.Source: C Task Construction by Scenario · pp. 11–12

D Construction Details

  • Matched conditions fix the task, withdrawn item, and final requirement; conversation revisions use fixed acknowledgements, while agent tasks add a workspace instruction and use a bounded tool-running setup.The runner retains tool-use traces and counts runs reaching its step or request limit without a deliverable as generation failures.Source: D Construction Details · pp. 12–13

E Evaluation Details

  • Evaluation computes rates over non-empty, nontruncated deliverables and uses task-cluster bootstrap intervals; defense success additionally requires delivery, required-content retention, and no reader recovery.Source: E Evaluation Details · pp. 13–14

F Presence and Utility Checks by Scenario

  • Presence checks assess whether the withdrawn item remains inferable anywhere in the deliverable, while trace detection separately identifies revision statements using removal expressions and recorded aliases.Source: E Evaluation Details · p. 13

G Measurement Validation

Human annotation and detector checks validate trace measurement while identifying specific boundary errors. Examples show traces in multiple reply positions, including cases where recipients recover withdrawn items.

G Measurement Validation

  • Annotators agree on trace presence at κ = 0.94, disclosure level at κ = 0.93, and position at κ = 0.90 across 375 sampled deliverables.The weighted trace rate is 34.3%; most traces occur only in the preface or afterword (26.0%), while 7.1% occur inside the body.Source: G Measurement Validation · p. 14
  • GLM-5.3-Flash reaches precision 0.89, recall 0.91, and κ = 0.84 against the gold standard; the scanner reaches deliverable-level precision 0.91 and recall 0.63.The scanner’s identifying-level classification reaches precision 0.90 and recall 0.45, and it often misses short confirmations and passive statements.Source: G Measurement Validation · pp. 14–15
  • The LLM check’s largest disagreement is in policy-constrained documents, where general statements about omitted categories can be misclassified as traces.Treating unverifiable positive judgments as no trace changes pooled defense effects by at most 1.2 points without changing their direction.Source: G Measurement Validation · p. 15

H Trace Position Examples

  • Traces appear in all five reply positions, and some body traces name withdrawn references, findings, or sentences that recipients can recover.Figure 6 illustrates the reply-part split used to classify positions.Source: Figure 6 · p. 14, H Trace Position Examples · p. 15

I Revocation Example

  • In the coordination-message example, GLM-5.2 leaves the appointment out but identifies it in a trace, while GPT-5.5 gives only a non-identifying explanation.Both models satisfy required content, and readers recover the appointment only from GLM-5.2’s message.Source: I Revocation Example · p. 16

J Per-Model Results

  • Table 12 reports scanner rates and trace levels, while Tables 13 and 14 break those results down by model for the conversation and agent tracks.Table 15 reports the same measures under reduced wording.Source: J Per-Model Results · p. 16

K Reader Recovery

The reader task tests whether the deliverable alone reveals a withdrawn item, and trace deletion sharply reduces observed recovery. Recovery varies by scenario and trace wording, while some cases remain recoverable because the item appears elsewhere in the body.

K Reader Recovery

  • The reader sees only the deliverable, and recovery means it identifies the recorded withdrawn item; “unknown” counts as unsuccessful.The reader reaches precision 0.88, recall 0.70, and κ = 0.76 against human consensus, with misses often identifying only the item’s category.Source: K Reader Recovery · p. 16
  • Recovery is highest in coordination messages, where identifying traces often repeat the item’s name, and lower where traces name only a category or change.Enterprise briefs fall between these patterns because traces may identify the withdrawn metric without revealing its numerical value.Source: K Reader Recovery · p. 16
  • 303 of 1,239 deliverables yield recovery before trace deletion, compared with 24 afterward; trace deletion reduces recovery by −23.0 percentage points [−28.6, −17.5].The paired difference excludes zero in every scenario and track, while unrelated sentence deletion leaves 293 recoveries.Source: K Reader Recovery · p. 19
  • Five agent-track policy-constrained deliverables become recoverable after trace deletion because the body still contains the withdrawn item.Trace deletion removes recovery in 16 paired deliverables in this main exception.Source: K Reader Recovery · p. 19

L Additional Results

Additional analyses examine information type, agent tool use, and controls that alter instructions or delivery boundaries. Several interventions reduce traces or recovery, but their effects and utility tradeoffs vary across settings.

L Additional Results

  • Recovery and verbatim repetition vary by information type, while body-only filtering raises marginal utility despite a −1.4-point paired utility change in the agent track.Among identifying traces, withdrawn personal information is repeated verbatim in 92.0%, organizational information in 43.0%, and undelivered content in 14.0%; body-only filtering excludes 81 baseline outputs with 3.7% utility.Source: L Additional Results · p. 19

M.1 Direct Condition and Withdrawn Item

  • Withdrawing A and B produces nearly identical pooled LLM trace rates: 53.4% and 53.8%.Scanner rates are 48.8% and 46.9%, respectively.Source: §M.1 · p. 19

M.2 Agent Tool Use

  • Agents read at least one workspace file in 84.0–99.0% of revocation deliverables across scenarios, but search for the withdrawn item after revocation in at most 2.0%.Among runs that read a file, trace rates remain within six percentage points of the overall rate in every scenario.Source: §M.2 · p. 19

M.3 Forwarding Notice

  • Across 1,588 matched pairs, the forwarding notice lowers full-reply trace rate from 59.8% to 36.4%, with declines in all ten scenario-track cells.The comparison uses four models across 100 tasks, both tracks, and both revocation conditions.Source: §M.3 · p. 19, Table 29 · p. 24

M.4 Deliverable-Only Instruction

  • The deliverable-only instruction lowers recovery from 15.3% to 4.7%, a −10.6-point paired change, while utility changes by −1.5 points.The utility change has a 95% interval of [−3.8,+0.8]; Table 17 reports paired end-to-end differences against the matched no-defense baseline.Source: Table 17 · p. 20, §M.4 · p. 24

M.5 Turn-Matched and Late-Exclusion Controls

  • The turn-matched control changes trace rate by −2.4 points, while moving exclusion to the final turn adds 15.5 points and revocation adds a further 15.9 points.Revocation places 82.1% of trace sentences in the preface and repeats the earlier instruction in 12.3%, versus 39.3–44.6% for the other conditions.Source: §M.5 · p. 24, Table 32 · p. 25

M.6 Replacement Control

  • Against the original replacement control, replacement leaves 11.9 points more traces; against the matched control, it leaves 5.3 points fewer.The scanner gives +12.5 points where the LLM check gives −5.3, with disagreement concentrated in enterprise briefs.Source: §M.6 · pp. 24–25

M.7 Task Wording

The task-wording analysis reduces extra framing while preserving the underlying writing task and withdrawn items. Its largest change occurs in policy-constrained documents; effects in other tested scenarios are not consistent.

M.7 Task Wording

  • In policy-constrained documents, reduced wording lowers trace rate from 66.2% to 26.1% and raises utility from 80.0% to 88.6%.Many original-wording traces explicitly cite policy or authorization, while none do under reduced wording; enterprise briefs and data analysis reports show no consistent reduction.Source: §M.7 · p. 25, Table 34 · p. 26

N Task Wording Variants

  • Reduced versions remove additional task framing while keeping source documents, required content, and withdrawn items identical to the original versions.For policy-constrained documents, the reduced version also withholds policy excerpts from the model and uses them only for evaluation.Source: N Task Wording Variants · p. 25

Related papers

Loading 2609.35408v1…