PaperlayerView exact revision on arXiv

Source-linked AI summary

Unmasking the Secret Code: Using Canaries to Show How Masked Context Controls Memorization in SFT

Avi Schwarzschild, Om Thakkar, Sahra Ghalebikesabi, Congzheng Song, Vinnie Monaco

arXiv:2610.20656v1cs.CR

TL;DR

Canary-based measurements of training-data extraction may depend on how secrets are presented in training and at evaluation. Through controlled fine-tuning experiments, the paper finds that visibility, timing, and canary population alter measured memorization, while synthetic-canary findings do not directly calibrate risk for natural private records.

  • Problem

    Canary audits provide controlled tests of training-data extraction, but the inference from canary recovery to vulnerability of other training records remains uncertain.

    Source: §I · p. 1

  • Method

    The authors vary secret visibility and encoding, training placement, and canary-set composition in controlled fine-tuning experiments spanning language and vision-language models.

    Source: §IV · p. 5, §V · p. 6, C. Predictive Context Changes… · p. 9

  • Results

    Repeatedly producing an input-supplied secret can coexist with little recall after its cue is removed, whereas transforming or withholding the cue can yield stronger memorization signals.

    Source: §VI · p. 10

  • Takeaways & Limitations

    Canary measurements depend on training and evaluation choices, and the authors motivate testing whether audit conclusions hold across those choices before extending them to other records.

    Source: §VI · p. 10

    The evidence concerns synthetic secrets in selected fine-tuning tasks, models, and recipes; it does not directly calibrate natural-record extraction risk or establish generalizable absolute risk rates.

    Source: C. Limitations · p. 11

Abstract

from arXiv · show

Canaries, or synthetic training records containing known secrets, are widely used to assess whether language models reveal information from their training data. But how much do these measurements depend on the canaries themselves? We study this question through controlled fine-tuning experiments with language and vision-language models, spanning synthetic tasks and tasks where training improves useful performance. Whether a secret is provided in the training input strongly changes what the model learns; for example, models can reproduce a secret in the input yet fail to recall it when it is removed, even after extensive repetition during training. When training instead requires predicting the secret from the surrounding context, later recall can become reliable. Repetition alone is therefore insufficient to characterize memorization. We also find that changing when a canary appears, or which other canaries accompany it, can substantially change measured memorization without increasing that canary's repetition count. These results show that canary measurements depend on how secrets are presented during training and tested afterward. Audits should test whether their conclusions hold across these choices before extending them to other training records.

I. INTRODUCTION

Canary audits offer controlled tests of training-data extraction, but their results depend on how secrets are represented, when canaries appear, and which other canaries accompany them. The paper connects these design choices to prior work on leakage, memorization, and complementary audit measures.

II. RELATED WORK

  • Canary audits provide controlled recovery tests, while prior studies examine leakage across model updates, synthetic text, and recovery of injected canaries [1].The inference from canary recovery to vulnerability of other training records remains uncertain; related work also examines visual-secret retention.Source: §I · pp. 1–2, §II · p. 2

I. INTRODUCTION

  • A secret supplied verbatim in the input can be reproduced during training yet show little recall when removed, while encoded or absent cues can yield strong memorization signals.The paper calls this design axis predictive context: how much the input cue reveals about the secret.Source: §I · p. 2
  • Canary dosage, training placement, and the surrounding canary population all affect measured memorization; late low-dose canaries can exceed higher-dose canaries placed earlier.Adding canaries can also increase unintended memorization, complicating claims about single samples and generalization to real records.Source: §I · p. 2
  • Low extraction risk under one audit protocol is insufficient evidence of low risk under another, so the authors recommend varying secret visibility, timing, and population.They also recommend reporting exposure alongside exact recovery under sampling.Source: §I · p. 2

II. RELATED WORK

  • Exposure measures a secret’s rank within a candidate set [7], unlike conditional prompt probability, membership inference, counterfactual memorization, or adversarial prompt compression.The paper motivates reporting rank-based exposure and exact recovery separately.Source: §II · pp. 2–3

III. PROBLEM SETUP AND EXPERIMENTAL DESIGN

The experimental design studies supervised fine-tuning in which input context conditions target prediction but is not itself scored by the loss. Canary inputs vary whether the secret is visible, redacted, or encoded, and evaluation separates recovery from ranking-based exposure and tests alternate prompts.

A. Training Setup and Canaries

  • SFT computes loss on target positions while input positions remain available as conditioning information for target predictions and parameter updates.The setup focuses on fine-tuning already capable pretrained models using chat-template samples.Source: A. Training Setup and Canaries · p. 3
  • Each canary pairs a record context with an assigned synthetic secret in the target, while its input cue may show, redact, or transform that secret.The email example places sender and address context in the input and the rewritten email with assigned code in the target.Source: A. Training Setup and Canaries · p. 3, Figure 2 · p. 3

B. Evaluation

  • Exposure ranks the true secret among candidates using base-2 logarithms, but rank does not guarantee that greedy decoding recovers the code.Text experiments exhaust all 10,000 four-digit candidates, and exact recovery is measured separately.Source: B. Evaluation · pp. 3–4
  • Counterfactual prompts vary whether the code is redacted, supplied as trained, or replaced, while utility checks compare fine-tuned models with pretrained and no-canary controls.These comparisons distinguish recall from decoding using the supplied context and assess task performance and canary-associated changes.Source: B. Evaluation · p. 4

C. Canary construction and protocol dimensions

The paper varies secret visibility, canary dose and placement, and surrounding canary population to test how protocol choices shape memorization measurements. In synthetic email rewriting, these choices substantially change exposure and recovery, with context inferability identified as the dominant factor.

C. Canary construction and protocol dimensions

  • The protocol varies whether training hides, directly supplies, or transforms the code, and whether evaluation hides or replaces it to distinguish recovery from copying or decoding.Dose and placement are also manipulated; dose is the number of training appearances, while placement is the canary's position in the training sequence.Source: C. Canary construction and protocol… · p. 4

D. Experimental Settings and Controls

  • Experiments span a controlled synthetic email-rewriting task and utility-backed text and GUI-grounding tasks with language and vision-language model families.The paper describes the email task as a toy setting and names TAT-QA, Banking77, and GUI grounding as utility-backed settings.Source: D. Experimental Settings and Controls · p. 4, §IV · p. 4

IV. UNINTENDED MEMORIZATION IN SYNTHETIC EMAIL REWRITING

  • Low-dose canaries show higher exposure in mixed-dose than single-dose runs, so adding other canaries changes measurements even when focal dose is fixed.The comparison also differs in total canary appearances.Source: §IV · p. 5, Figure 3 · p. 5
  • At fixed dose, later placement raises exposure; dose also raises exposure until saturation in independently retrained runs.For Qwen3-1.7B, eight appearances yield median exposure 4, while dose 16 saturates the metric and often permits greedy verbatim extraction; moving dose-eight canaries from the first to last 20% raises median exposure from 3.26 to 6.80.Source: §IV · p. 5, Figure 4 · p. 5
  • With the code visible during training, replacing it at evaluation elicits the replacement, but hiding it leaves ground-truth-secret exposure at baseline across tested doses.This distinguishes copying from recall when the secret is absent at evaluation.Source: §IV · p. 5
  • As training cues become less directly informative, outcomes shift from no detectable signal with visible codes to detectable exposure with transformed cues and low-dose exact recovery with redacted codes.The tested transformations include binary and chained-fruit encodings; evaluation hides the code in each condition.Source: §IV · pp. 5–6, Figure 5 · p. 6
  • Across four Qwen3 model sizes, the section's representative findings are reported as extremely consistent.The authors also point to Appendix A for the full breadth of experiments.Source: §IV · p. 6

V. SECRET RECOVERY DURING LLM AND VLM FINE-TUNING

The section tests whether canary-audit findings extend to language- and vision-language-model fine-tuning on tasks where fine-tuning improves useful performance. It evaluates secret recovery with the code hidden and varies canary context and populations.

V. SECRET RECOVERY DURING LLM AND VLM FINE-TUNING

  • The experiments vary canary population, timing, spacing, and predictive context while fine-tuning on Banking77 and GUI grounding tasks.The VLM study uses desktop grounding examples and typing-task canaries with assigned alphanumeric codes.Source: §V · pp. 6–7
  • Fine-tuning improves GPT-OSS-20B Banking77 accuracy from 56.10% to 91.30%–93.86% across retained checkpoints, and improves grounding accuracy for all four VLMs on both benchmarks.Qwen2.5-VL-7B improves from 37.45% to 48.56% on OSWorld-G and from 11.95% to 30.44% on ScreenSpot-Pro.Source: §V · p. 6
  • For visual canaries, visible and redacted training screenshots retain the same instruction and supervised target, while hidden-code evaluation masks the source region.Canary identities, codes, and presentation positions are matched across models within each experimental condition.Source: §V · p. 7, Figure 6 · p. 7
  • Secret recovery is measured with the code hidden at evaluation, using rank-based exposure and exact recovery under greedy decoding.For VLMs, the reference is ranked among 19,999 sampled decoys; the exposure ceiling reflects rank one in that sampled bank, not the full code space.Source: §V · p. 7

A. Canaries Interact with One Another

Additional high-dose canaries can raise exposure of focal canaries even when focal identities, codes, doses, and positions are held fixed. The pattern is also observed in task-based fine-tuning.

A. Canaries Interact with One Another

  • The VLM comparison holds focal identities, codes, doses, and positions fixed while adding canaries at doses 8 and 16.It tests whether the higher exposure seen in synthetic email rewriting also occurs when models learn a background task.Source: A. Canaries Interact with One Another · pp. 7–8
  • At one appearance, Qwen2.5-VL-7B focal-canary median exposure rises from 4.11 to 6.99 with additional high-dose canaries; at two, it rises from 9.97 to the 14.29 ceiling.For GPT-OSS-20B Banking77 dose-four canaries, pooled median exposure rises from 3.36 in single-dose training to the 13.29 ceiling in mixed-dose training.Source: A. Canaries Interact with One Another · p. 8

B. Placement and Spacing Change Exposure

At fixed dosage, exposure varies with when canaries appear and how their appearances are spaced. Predictive context also changes recovery: repeatedly producing a visible secret can coexist with little recall once it is hidden.

B. Placement and Spacing Change Exposure

  • Under a constant learning rate, Banking77 median exposure rises from 3.68 for early placement to 8.74 for middle and 13.29 for late placement; cosine decay yields 8.76, 8.16, and 1.32, respectively.The different schedules show that learning-rate size also shapes timing effects.Source: B. Placement and Spacing Change Exposure · p. 8
  • For Qwen2.5-VL-7B under cosine decay, exposure decreases as the final-presentation window shifts from about 50% to 95% of training, but the direction reverses under a constant schedule.At each final-presentation window, a 125-update gap yields higher pooled median exposure than a 16-update gap; the authors suggest the higher learning rate at the earlier presentation may help explain this.Source: B. Placement and Spacing Change Exposure · p. 8

C. Predictive Context Changes Memorization

  • Visible-code training leaves median exposure below 1.2 after 64 Banking77 appearances, while redacted-code training reaches 13.29 by 16 appearances.On GUI grounding, visible-code exposure is 1.39 after 128 appearances, while redacted-code training reaches 14.29 by four appearances; the visible model copies the original code when its source remains visible.Source: C. Predictive Context Changes… · p. 9, Figure 9 · p. 9
  • When a different code is substituted at the highest doses, visible-code-trained models copy the replacement code in 100% of Banking77 cases and 91.2% of GUI-grounding cases.Repeatedly producing a supervised secret can therefore coexist with little ability to recover it after its source is removed.Source: C. Predictive Context Changes… · p. 9
  • Binary visual encoding yields intermediate exposure between visible-code and redacted-code training through 16 appearances, approaching saturation at higher doses.Because binary encoding fully specifies the target, its intermediate exposure indicates that representation of predictive information can affect memorization.Source: C. Predictive Context Changes… · pp. 9–10

VI. DISCUSSION AND CONCLUSION

Canary measurements depend on training prediction context, evaluation information, repetition, placement, spacing, and surrounding canaries. The discussion probes transformations and identifies open questions about generalizing canary findings and understanding timing effects.

VI. DISCUSSION AND CONCLUSION

  • The experiments show that prediction context, evaluation information, dose, timing, spacing, and canary population can all change measured memorization.The effects extend from controlled synthetic tasks to LLM and VLM fine-tuning settings where benchmark performance improves.Source: §VI · p. 10

A. Compositional Transformations of Visual Cues

  • At width eight with a +4 cyclic shift, median exposure reaches 8.76; combining reversal and shifting raises exposure above either operation alone.Across reversal widths, exposure increases from no shift to +1 and +4; the selected width-four, +4 condition falls between matched copy and redacted endpoints in all four models.Source: Figure 10 · p. 10, A. Compositional Transformations of… · p. 10
  • The transformation comparisons do not isolate a single measure of decoding difficulty, and future work could test other transformation families and compositions.The authors also propose varying base checkpoints to separate contextual information availability from a model's ability to use it.Source: A. Compositional Transformations of… · pp. 10–11

B. Open questions

  • Whether deliberately optimized canaries represent natural records remains open; comparing them with constrained canaries and natural records could test transfer.The leap from canary recovery to conclusions about non-canary data relies on this transfer question.Source: B. Open questions · p. 11
  • Which optimization, training-phase, and background-replacement factors drive timing and cohort effects remains unresolved.The authors note that phases such as reinforcement learning after pretraining require independent controls and large-scale experimentation.Source: B. Open questions · p. 11

C. Limitations

The paper limits its claims to trends and comparisons from synthetic secrets in selected tasks, models, and training recipes, and notes that audit readouts depend on evaluation and training choices. It recommends testing plausible alternatives while distinguishing copying, cue following, and retrieval.

C. Limitations

  • The experiments do not calibrate absolute memorization risk for natural private records, and synthetic-code findings may not transfer to unstructured sensitive data.The authors also caution that abstract facts about a person may behave differently from codes, Social Security numbers, or addresses.Source: C. Limitations · p. 11
  • Recovery tests cover only the queries and decoding procedures used here, so failure under these tests does not rule out recovery through other methods.The authors note that adversarial prompts could enable more extraction in some cases.Source: C. Limitations · p. 11

D. Practical consequences for canary audits

  • Audits should state what is supplied during training and extraction and how canaries are placed, then test plausible alternatives and report exposure separately from exact recovery.The authors also recommend distinguishing copying, cue following, and retrieval, and making optimization settings, appearance timing, and evaluation delay explicit.Source: D. Practical consequences for canary… · p. 11

ETHICAL CONSIDERATIONS

  • The work uses synthetic secrets and established public benchmarks; its measurements describe the studied settings, not privacy guarantees for natural training records.The authors note that knowledge of conditions associated with greater memorization could also inform attempts to extract training information.Source: ETHICAL CONSIDERATIONS · p. 12

APPENDIX A ADDITIONAL SYNTHETIC EXPERIMENTS

This appendix adds training and evaluation details, expands synthetic-email results across models and conditions, and tests questions about ordering, auxiliary canaries, and transfer to unseen encoded values. Across these experiments, exposure and recovery vary with dose, representation, timing, and the surrounding population.

A. Training and Evaluation Details

  • The email experiments use a fixed 66,176-row sequence, with 64 focal canaries; cue comparisons retain decimal targets while varying redacted, visible-decimal, binary, or chained-fruit inputs.The chained-fruit transformation is invertible, but its rule is not supplied in the evaluation prompt; exposure ranks among 10,000 decimal candidates and greedy recovery is evaluated separately.Source: A. Training and Evaluation Details · p. 15, B. Cue Construction Evaluation · p. 15

C. Expanded Results for Experiments in the Main Body

  • Across four Qwen3 sizes, pooled median exposure rises with dose toward the ranking ceiling; with codes hidden at evaluation, redacted training yields the highest medians at doses 16, 32, and 64.Visible-decimal training stays near baseline, while binary and chained-fruit training are intermediate; encoded-cue exposure does not rise monotonically with dose.Source: C. Expanded Results for Experiments in… · pp. 15–16, Figure 11 · p. 16, Figure 12 · p. 17
  • At doses 16, 32, and 64, binary-trained exposure is lower for Qwen3-14B than for Qwen3-1.7B and Qwen3-4B, and binary-versus-fruit ordering differs by model.Dose-2 and dose-4 redacted canaries also have higher pooled median exposure in mixed-dose than single-dose runs; the comparison changes neighboring doses and total appearances.Source: C. Expanded Results for Experiments in… · p. 16, Figure 13 · p. 18, Figure 15 · p. 19

1) Broader Representation and Dose Results:

D. Additional Experiments on Synthetic Email Data

  • Globally shuffled and conventionally shuffled background orderings have closely overlapping training-loss curves; held-out loss is 0.003580 and 0.003073, respectively.This one-seed, no-canary control addresses background-task learning, not canary memorization.Source: D. Additional Experiments on Synthetic… · p. 17, Figure 17 · p. 20
  • Moving an auxiliary cohort from after to before focal canaries raises exposure for 86% of focal canaries, with a median paired increase of 4.77.In a single-seed control, one-shot auxiliary examples yield median focal exposure 1.16, versus 1.91 without auxiliaries and 7.21 with the repeated cohort before the focal cohort.Source: D. Additional Experiments on Synthetic… · pp. 17–18, Figure 18 · p. 20, Figure 19 · p. 21
  • After dose-64 fine-tuning, both models recover all 64 training codes from familiar encoded cues, but recover none of 256 unseen binary codes and only four or five unseen chained-fruit codes.They recover most or all of the same new values when supplied as decimal digits; digit-level accuracy is stronger for leading than final digits, without establishing how that partial generalization arises.Source: D. Additional Experiments on Synthetic… · p. 22, Figure 20 · p. 23, Figure 21 · p. 23
  • At binary doses 128, 256, and 512, neither model recovers or ranks first any training code when the cue is redacted, although both recover all 64 with corresponding binary cues.These are single-seed diagnostics; at dose 512, focal canaries occupy 32,768 of 66,176 training rows.Source: D. Additional Experiments on Synthetic… · p. 22

APPENDIX B ADDITIONAL UTILITY-BACKED TEXT EXPERIMENTS

The appendix extends utility-backed text experiments to TAT-QA and gives fuller Banking77 results, examining canary conditions while checking performance on the intended task.

APPENDIX B ADDITIONAL UTILITY-BACKED TEXT EXPERIMENTS

  • The appendix examines dosage, placement, predictive context, and surrounding canary populations in TAT-QA and Banking77 while checking whether fine-tuning still improves task performance.It also reports results separately for repetitions to show consistency across runs.Source: Appendix B · p. 24
  • The protocol overview notes that canary appearances replace background rows within a fixed training budget, and pooled observations are not necessarily distinct identities or independent training replicates.Table III reports pre- and post-fine-tuning text-task accuracy, using strict answer-and-scale accuracy for TAT-QA and intent accuracy for Banking77.Source: Table II · p. 24, Table III · p. 24

A. Qwen3 on TAT-QA

Across TAT-QA and Banking77, the experiments examine how dose, context, placement, and canary populations shape exposure while fine-tuning improves task performance. Repetition, visible cues, placement, and mixed-dose populations produce distinct exposure patterns, and the TAT-QA results recur across repetitions.

A. Qwen3 on TAT-QA

  • On TAT-QA, Qwen3-14B pooled median exposure rises from 2.46 at dose 16 to the 13.29 ceiling at dose 32, while dose-64 references rank first for every canary in all three runs.Fine-tuning without canaries raises Qwen3-14B strict accuracy from 21.8% to 70.4%; accuracy across TAT-QA dose and context comparisons remains 68.2%–71.4%.Source: A. Qwen3 on TAT-QA · pp. 24–25, Figure 25 · p. 27
  • With visible references during Qwen3-14B training at dose 512, models copy supplied and substituted references, yet hidden-reference median exposure is 0.87.The results separate copying a visible reference from recovery when that reference is hidden.Source: A. Qwen3 on TAT-QA · p. 25
  • On TAT-QA, late placement raises exposure for 87 of 96 canaries, with a median paired increase of 3.12; mixed-dose dose-4 exposure rises from 1.07 to 13.29.The mixed-dose comparison also changes neighboring canary doses and total appearances, so it does not isolate an interaction.Source: A. Qwen3 on TAT-QA · p. 25, Figure 23 · p. 26

B. GPT-OSS-20B on Banking77

  • On Banking77, held-out accuracy rises from 56.1% to 93.1% after ordinary-example fine-tuning, while constant-rate canary runs retain higher accuracy than the base model.Pooled median exposure rises from 3.95 at dose 4 to 8.90 at dose 8 and the 13.29 ceiling at dose 16.Source: B. GPT-OSS-20B on Banking77 · p. 26
  • On Banking77, dose-8 exposure increases from 5.17 with early placement to 8.93 with middle placement and 13.29 with late placement; at dose 16, early placement yields 9.29 while uniform and late placement reach 13.29.Visible-code training copies original and substituted codes, but hidden-code median exposure is 0.99 at dose 4 and 1.18 at dose 64.Source: B. GPT-OSS-20B on Banking77 · p. 26
  • On Banking77, pooled median dose-4 exposure rises from 3.95 in single-dose training to 13.29 in mixed-dose training.Mixed-dose runs also increase total canary appearances, so the comparison does not isolate an interaction between canaries.Source: B. GPT-OSS-20B on Banking77 · p. 26, Figure 24 · p. 27

C. TAT-QA replication

  • Across three Qwen3-14B repetitions, the dosage threshold, low exposure after visible-code training, late-placement increase, and mixed-dose increase recur.The plots pool variation across canaries rather than uncertainty intervals over training runs; task accuracy averages for placement and mixed-dose comparisons range from 69.8% to 70.8%.Source: C. TAT-QA replication · p. 27

1) Broader TAT-QA Results Across Models and Seeds:

Expanded results extend the TAT-QA representation and dose grid and compare Banking77 dose, placement, and learning-rate schedules. Across these settings, the reported measurements vary with representation and placement, not just repetition count.

1) Broader TAT-QA Results Across Models and Seeds:

D. Banking77 replication

  • Banking77 median exposure reaches the 13.29 ceiling at dose 16 with uniform or late placement, and rises from early to late placement at dose 8 in every repetition.Mixed-dose exposure reaches the ceiling at all four assigned doses, though companion canaries and total appearances remain confounded.Source: D. Banking77 replication · p. 34, Figure 33 · p. 34, Table V · p. 34

1) Broader Banking77 Results Across Joint Seeds:

APPENDIX C ADDITIONAL VISION-LANGUAGE EXPERIMENTS

This appendix adds vision-language training and evaluation details, expanded experiments, and encoding comparisons. It specifies the grounding corpora, matched canary conditions, and utility-evaluation procedures used in these experiments.

APPENDIX C ADDITIONAL VISION-LANGUAGE EXPERIMENTS

  • The appendix supplements the main vision-language experiments with training and evaluation details, expanded results, and matched binary and dictionary encoding experiments.The encoding experiments examine additional questions about predictive visual cues.Source: Appendix C · p. 38

1) GUI grounding training data:

  • Three models use a fixed 10,000-example grounding corpus sampled across Linux, macOS, and Windows, while Qwen2.5-VL-32B uses a curated 10,000-example desktop corpus.The three-model corpus contains 3,400 Linux, 3,300 macOS, and 3,300 Windows training examples.Source: 1) GUI grounding training data · p. 38

2) Training setup:

  • Each run makes one pass through 10,000 examples in 313 updates, with canary presentations replacing grounding examples and the vision encoder frozen.Three seeds are used per setting; models retain native image processing and loss normalization, so cross-model comparisons include those differences.Source: 2) Training setup · p. 38

3) Experimental allocations:

  • Dose/context experiments match canaries and presentation positions across visible and redacted conditions, while timing and interaction experiments use matched identities and placements.Timing varies presentation windows and gaps; interaction adds higher-dose canaries in place of 336 grounding examples under a fixed 10,000-example budget.Source: 3) Experimental allocations · pp. 38–39

4) Utility evaluation:

  • Utility is evaluated on OSWorld-G and ScreenSpot-Pro with model-specific output formats, and predictions count as correct when their point falls inside the target annotation.Unparsable responses count as incorrect.Source: 4) Utility evaluation · p. 39

1) Canary construction:

  • Canaries use screenshots with visible, redacted, or evaluation-only substituted source codes; only the designated source region changes between visible and redacted versions.The fixed Windows screenshot pool is screened for duplicates and overlap, although screenshots may share an application or interface style.Source: 1) Canary construction · p. 39

2) Canary evaluation:

The appendix defines source-masked candidate ranking and exact-code recovery measures, then reports dose-sweep results across vision-language models. Redacted training raises exposure and recovery, while visible-code training supports copying but not masked recall.

2) Canary evaluation:

  • Exposure ranks the reference code among 20,000 candidates, whereas greedy recovery checks whether generation returns the requested code exactly.Exposure is log2(20,000/rank), with a maximum of approximately 14.29; generation also distinguishes the displayed replacement from the original reference on substituted probes.Source: 2) Canary evaluation · pp. 39–40

3) Full example prompts:

  • The canary example instructs the model to type the visible reference code into a focused search field, with a literal computer-use action as the target.The instruction stays unchanged when the source code is redacted.Source: 3) Full example prompts · p. 40, Figure 41 · p. 40

1) Grounding utility across models:

  • Qwen2.5-VL-7B’s ScreenSpot-Pro parse rate rises from 58.25% to 99.60%, so its reported gains include output-format learning as well as grounding.Unparsable responses count as errors.Source: 1) Grounding utility across models · p. 40

2) Exposure and utility across the dose sweep:

  • Across four models, source-masked median exposure is higher at D128 than D1, yet exact-code recovery with the source pane absent is 0/216 at every dose.At D128, all four models recover nearly all codes from intact training cues, but none recovers new visible or substituted codes on the probes.Source: 2) Exposure and utility across the… · p. 42, Table VII · p. 42, Figure 43 · p. 43
  • Through D128 under redacted training, mean grounding accuracy falls by at most 2.61 percentage points on OSWorld-G and 1.95 on ScreenSpot-Pro versus grounding-only fine-tuning.At D128, canary rows make up 46.08% of the corpus, so the comparisons combine repetition with a changed training mixture.Source: 2) Exposure and utility across the… · pp. 42–43, Figure 43 · p. 43

3) Ranking and exact code recovery:

The results distinguish candidate ranking from exact generation, and show that presentation timing and the surrounding canary population also shift exposure. Encoded visual cues yield intermediate exposure but do not ensure exact recovery or general decoding.

3) Ranking and exact code recovery:

  • At D8, 28.2–99.5% of observations rank first across models, but only 0–63.9% yield the exact code; at D16, Qwen2.5-VL-32B ranks first in 199/216 cases but generates it in 73/216.At D32, every redacted-training observation ranks first, with exact recovery ranging from 96.3% to 100%; at D128 every model recovers 216/216.Source: 3) Ranking and exact code recovery · p. 43, Figure 44 · p. 44

4) Placement and Recency Across Models and Schedules:

5) Canary Interactions Across Models:

  • Adding higher-dose canaries generally raises focal-canary exposure, although Qwen2.5-VL-32B’s D1 median falls from 2.43 to 2.03; exact recovery is unchanged at D1 and D2.The comparison combines added canaries with displaced background examples and does not isolate a facilitation mechanism.Source: 5) Canary Interactions Across Models · p. 47, Figure 47 · p. 47

6) Encoded visual cues and compositional transformations:

  • The composed reversal-and-shift conditions produce no exact masked-source recovery or correct substituted-source target recovery in any of the four models.With the original source intact, exact recovery is 44/216 for Qwen2.5-VL-7B and 7/216 for InternVL3-8B, and zero for the other two models.Source: 6) Encoded visual cues and… · pp. 47–48, Table VIII · p. 48

D. Additional Experiments and Analyses

Across encoded-cue experiments, exposure and exact recovery depend on the encoding and evaluation cue: direct fruit supports more recovery than binary or chained fruit, while chained fruit has higher median exposure than direct fruit. The appendix examples show how canary cues and target responses are constructed across task formats.

1) Binary and dictionary encodings:

  • The matched D4 setup compares ASCII binary, direct fruit, and chained fruit using the same canaries and seeds, with copy and redacted conditions as endpoints.The dictionary encodings use eight source units and a shared character-to-fruit mapping; chained encoding depends on the preceding character.Source: 1) Binary and dictionary encodings · p. 48
  • Chained fruit has higher median exposure than direct fruit in all four models, and all encoded conditions fall between matched copy and redacted endpoints.These exposure relationships also hold for two models when code values and mode labels are hidden but the source-pane layout remains visible.Source: Table IX · p. 49, 1) Binary and dictionary encodings · p. 49, 2) Summary Across Encoded-Cue… · p. 49
  • Direct fruit supports the highest intact-cue exact code recovery among encoded conditions, while chained fruit and binary recover much less.Direct fruit also supports recovery from familiar donor cues and new codes in some models; the other encodings show little or no corresponding recovery in those probes.Source: 1) Binary and dictionary encodings · p. 49

2) Summary Across Encoded-Cue Experiments:

  • Combining block reversal and cyclic shifts yields higher median exposure than either transformation alone in the Qwen2.5-VL-7B transformation grid.The selected composition has median exposure between copy and redacted endpoints in all four models, but none of its four conditions yields exact masked recovery.Source: 2) Summary Across Encoded-Cue… · p. 49

APPENDIX D COMPLETE TEXT CANARY EXAMPLES

  • The appendix examples include hidden-code synthetic emails, TAT-QA requests with a reference added to an unchanged answer, and Banking77 responses with a customer name and request reference.Other examples show binary and chained-fruit input cues paired with a comma-separated decimal target code.Source: Figure 48 · p. 50, Figure 49 · p. 50, Figure 50 · p. 50

Related papers

Loading 2610.20656v1…