PaperlayerView exact revision on arXiv

Source-linked AI summary

SpectralCanary: Embarrassingly Simple Low-Frequency Traces for T2I Dataset-Use Auditing

Sy-Tuyen Ho, Anirudh Satheesh, Soham Chawathe, Huy Nghiem, Yongkyun Lee, Andrew Adams Schoen, Furong Huang

arXiv:2610.31659v1cs.CVcs.LG

TL;DR

Fine-tuning can absorb a collection’s visual characteristics without reproducing its images, making dataset-use auditing difficult. SpectralCanary blends a fixed canary into image low frequencies before release and detects its trace in suspect-model outputs. Across the reported settings, it achieves strong auditing performance while preserving image fidelity and generative utility.

  • Problem

    Models can absorb collection-level visual characteristics without reproducing individual images, leaving rights holders with little evidence that their work was used.

    Source: §1 · p. 1

  • Method

    SpectralCanary deterministically blends a fixed canary’s low-frequency content into collection images and uses a lightweight verifier to test suspect-model generations.

    Source: §1 · p. 2

  • Results

    91.1% average TPR at 1% FPR and 94.8% fixed-threshold model-level accuracy using 30 generations were achieved across eight Pokemon settings.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Across five visual domains, low-frequency traces transfer while protected-image fidelity and generative utility remain high, supporting low-frequency visual signals as carriers for collection-level T2I audit traces.

    Source: §1 · p. 3, §6 · p. 11

    The adaptive evaluation assumes the canary and verifier remain private and does not establish cryptographic or white-box robustness.

    Source: §5 · p. 10

Abstract

from arXiv · show

Auditing whether a text-to-image (T2I) model was fine-tuned on a particular image collection is difficult: a model can absorb collection-level visual characteristics without reproducing training images. Proactive dataset-use auditing plants a shared signal in a dataset before release, then later checks whether that signal appears in a suspect model's generated outputs. Existing approaches often trade off verifiability, fidelity, robustness, and deployability. We introduce $\textbf{SpectralCanary}$, which uses a smooth low-pass mask to blend a fixed canary image into every collection image while leaving captions unchanged and requiring neither auxiliary generation nor per-image optimization. A lightweight verifier detects the transferred signal in suspect-model outputs. Across eight tests on the Pokemon dataset and three realistic data transformations, spanning two architectures families, SpectralCanary achieves the highest average per-generation TPR@$1\%$ FPR ($91.1\%$) and fixed threshold model-level accuracy using 30 generations ($94.8\%$) among the evaluated methods. Across five datasets spanning artistic, medical, satellite, and fashion domains with SD3.5, SpectralCanary averages $92.4\%$ per-generation TPR@$1\%$ FPR while preserving the released images (SSIM $0.989$, LPIPS $0.022$) with a near-zero mean change FID (mean $\Delta$FID $0.05$) across datasets. Matched-energy ablations on Pokemon and ROCOv2 support our design choice, low-frequency placement transfers more reliably compared to mid/high-frequency placement. Under the evaluated adaptive attacks, the most effective removal requires aggressive modification and incurs more than $25$ $\Delta$FID points. These results identify low-frequency visual signals as effective carriers for collection-level T2I audit traces.

1. Introduction

Proactive dataset-use auditing seeks black-box evidence that a model used a collection, even when it reproduces no training image. SpectralCanary adds a shared low-frequency trace before release and tests suspect-model outputs for it.

1. Introduction

  • Dataset-use auditing matters because models can absorb collection-level visual characteristics without reproducing individual images, leaving rights holders with little evidence of use.Practical auditing can support accountability and licensing compliance while images remain available for legitimate use.Source: §1 · p. 1
  • Prior approaches trade off collection-level verifiability, fidelity, robustness, and deployability, motivating a subtle, transferable, caption-independent, inexpensive trace.Membership and similarity tests rely on sample-specific evidence; pixel-level signals may attenuate or fail to transfer, while explicit canaries can be conspicuous or filterable.Source: §1 · p. 2
  • SpectralCanary blends a fixed canary’s low-frequency Fourier content into collection images, leaves captions unchanged, and uses a lightweight verifier to detect the trace in generated outputs.The deterministic transform requires neither auxiliary generation nor per-image optimization.Source: §1 · pp. 2–3
  • 91.1% average TPR at 1% FPR and 94.8% fixed-threshold model-level accuracy using 30 generations were achieved across eight Pokemon settings.Across five visual domains with SD3.5, it achieved 92.4% TPR at 1% FPR, SSIM 0.989, LPIPS 0.022, and mean ΔFID 0.05.Source: §1 · p. 3
  • The evaluation spans two architecture families, five datasets, three direct baselines, three data transformations, and measures discrimination, model-level auditing, fidelity, utility, robustness, and cost.The contribution also includes matched-band, matched-energy frequency ablations and an adaptive-attack boundary assessment.Source: §1 · p. 3

2. SpectralCanary

SpectralCanary protects selected collection images by moving their low-frequency spectra toward a shared canary target. A verifier trained on original–protected pairs then scores black-box model generations against clean reference scores.

2. SpectralCanary

  • The protection algorithm and collection are public, but the defender retains originals, canary, and verifier; auditing provides statistical evidence, not cryptographic attribution or white-box robustness.If an attacker learns the canary spectrum, blending strength, and bandwidth, the pre-export spectrum is exactly invertible when α < 1, though clipping and quantization prevent exact recovery from released images.Source: §2 · pp. 3, 5
  • Low frequencies encode broad structure important to T2I generation, and matched studies report higher detection rates for low-frequency placement than for mid/high-frequency placement.The design uses a smooth mask and small blending strength to control image changes because strong low-frequency modification is visible.Source: §2 · p. 4
  • For selected images, a smooth radial low-pass mask weights a Fourier blend between each image spectrum and the fixed canary spectrum; the inverse transform is clipped and quantized for export.The mask is strongest at the spectral center and decays with radial distance; smaller bandwidth concentrates weighting around the lowest frequencies.Source: §2 · p. 4
  • The shared canary term shifts the collection mean toward its spectrum while contracting variation in low-pass-weighted frequencies.The deterministic construction requires neither an auxiliary generative model nor iterative per-image optimization.Source: §2 · p. 4
  • The verifier learns from selected protected images as positives and their original counterparts as negatives; paired images share semantic content, making the transform their only systematic label-aligned difference.Training uses no model generations, and transfer to unseen generations is evaluated separately.Source: §2 · p. 5
  • A one-sided two-sample Kolmogorov–Smirnov test compares suspect-generation verifier scores with clean-data reference scores and rejects at level γ.Experiments use matched clean-data models; deployment calibrates reference scores on representative independent models.Source: §2 · p. 5

3. Related Work

Related work distinguishes reactive tests of already-trained models from proactive methods that alter a collection before release. SpectralCanary is positioned as a collection-specific statistical audit rather than a poisoning objective.

3. Related Work

  • Reactive membership-inference and similarity tests can lose evidence when fine-tuning transfers collection-level characteristics without retaining identifiable information about individual samples.Proactive traceability instead modifies a collection before release while aiming to preserve customization utility and enable later statistical auditing.Source: §3 · p. 5
  • Pixel-level traces preserve appearance but can be attenuated by augmentation, regeneration, or backbone changes; explicit canaries can transfer strongly but may be conspicuous and rely on removable caption triggers.The paper compares methods from both proactive families.Source: §3 · p. 5
  • Poisoning seeks chosen behavior, whereas auditing requires a collection-specific signal detectable under ordinary trigger-free queries with controlled false positives.Persistent behavior alone does not establish use of a particular collection.Source: §3 · p. 5

4. Experiments

Across model architectures, training-set changes, and image domains, the experiments assess SpectralCanary’s discrimination, fidelity, runtime, and resistance to trace removal. The results show strong detection alongside low image and model-quality costs, with low-frequency placement outperforming matched alternatives.

4.1 Experimental Setup and How to Read the Results

  • The evaluation covers five image domains, two Stable Diffusion architectures, and Pokemon tests with augmentation, image regeneration, and recaptioning.The main experiments use LoRA, with full-parameter fine-tuning tests reported in the appendix.Source: §4.1 · p. 6

4.2 Per-Generation Discrimination Survives Changes to Models and Training Data

  • Across eight Pokemon settings, SpectralCanary has the strongest average per-generation discrimination, averaging 91.1% TPR at 1% FPR versus 73.2% for the next-best method.It also achieves 94.8% average model-level accuracy.Source: §4.2 · pp. 6–7

4.3 The Same Setting Works Across Five Image Types

  • 92.4% average per-generation TPR at 1% FPR across five datasets accompanies average LPIPS of 0.022 and mean ΔFID of 0.05.The strongest and hardest datasets are Sketch-Scene at 100% TPR with LPIPS 0.002 and RSICD at 78.9% TPR with LPIPS 0.048.Source: §4.3 · pp. 7–8, Figure 4 · p. 8

4.4 Protection Is Fast as Well as Effective

  • 0.39 seconds per image on CPU accompanies the highest average per-generation TPR and the lowest paired-image LPIPS among methods modifying every image.DIAGNOSIS is faster but averages 32.2% TPR; the timing comparison is hardware-dependent.Source: Figure 5 · p. 8, §4.4 · pp. 8–9

4.5 Low Frequencies Carry the Trace More Reliably

  • 16.9 percentage points on Pokemon and 88.9 points on ROCOv2 separate low-frequency from high-frequency placement in per-generation TPR.The bands have equal Fourier-coefficient counts and equal total Fourier-domain change; mid-frequency placement is also weaker on both datasets.Source: Figure 6 · p. 9, §4.5 · p. 9

4.6 Removal Attacks Damage Quality Before Erasing the Trace

  • More than 25 additional FID points are required under the strongest evaluated attack to drive TPR toward the clean-training baseline.The three evaluated attacks also produce large LPIPS changes; the removal–quality boundary is not a guarantee against every possible attack.Source: §4.6 · p. 9, Figure 7 · p. 10

5. Limitations and Responsible Use

The evaluation leaves important boundaries around model and data coverage, adaptive-attack assumptions, and the evidence needed for deployment decisions. Responsible use also requires authorization and safeguards because traces can be inserted without downstream users’ knowledge.

Scope

  • The evaluation covers two architecture families, five datasets, and LoRA and full-parameter fine-tuning, but not every model or mixtures with larger unprotected corpora.Main experiments also use one fixed natural canary image; broader testing across independently chosen canaries and key reuse is needed.Source: §5 · p. 10

Adaptive threat model

  • Adaptive-attack results assume the canary and verifier remain private and do not establish cryptographic or white-box robustness.If the attacker learns V, α, and c, Eq. 6 permits exact inversion before clipping or quantization.Source: §5 · p. 10

Evaluation and deployment

  • Image-fidelity and utility metrics capture only part of quality, and deployment requires broader clean-model calibration; a positive audit is statistical evidence, not standalone proof of unauthorized use.Human perceptual studies and task-specific utility measurements would be needed to support stronger claims about invisibility or harmlessness.Source: §5 · p. 10

Responsible use

  • Trace insertion requires authorization, appropriate disclosure, respect for dataset licenses, and safeguards against unsupported attribution.For legal or reputational decisions, the audit should be combined with independent evidence.Source: §5 · p. 10

6. Conclusion

The conclusion presents SpectralCanary as a deterministic, inexpensive proactive audit trace whose low-frequency placement transfers across the evaluated settings. The appendix outlines supporting benchmark, metric, mechanism, baseline, and implementation material.

6. Conclusion

  • Matched-energy experiments favor low-frequency placement over mid/high-frequency placement, while evaluated sanitization attacks substantially reduce detection only at considerable image-quality or generative-utility cost.The conclusion reports strongest average per-generation discrimination and reliable KS-based multi-query auditing across eight Pokemon model–transformation settings, with results also generalizing across five visual domains.Source: §6 · p. 11

Appendix Roadmap

  • The appendix roadmap groups evidence into primary protocols and benchmarks, additional training and auditing studies, mechanism and robustness analyses, baseline analyses, and theoretical and related-work foundations.It points readers to appendices covering spectral location, canary structure, adaptive attacks, EnTruth, and Silent Branding.Source: Appendix Roadmap · pp. 14–15

A.1 Auditing Metrics

  • Per-generation AUC compares verifier-score separation between clean- and protected-data generations, while TPR at a target FPR uses the feasible empirical threshold with the largest FPR not exceeding the target.The protocol reports TPR at 5% and 1% FPR.Source: §A.1 · p. 15
  • KS-based multi-query auditing compares 30 suspect scores with a fixed clean reference set of 300 scores and reports rejection rates from 100,000 resampling trials.The rejection-rate estimates are averaged across three runs and reported with standard deviations.Source: §A.1 · p. 15

A.2 Implementation Details

  • The implementation trains clean and protected LoRAs under specified SD1.5 and SD3.5 settings and scores generations from matched prompt lists independently.The protocol uses 1,000 generations per model for SD1.5 and 500 for SD3.5.Source: §A.2 · pp. 15–16
  • Protection uses a fixed RGB canary at stored image resolution, with default δ=1, α=0.02, and c=0.20; a separate ResNet-50 verifier is trained without T2I generations.The verifier uses an 80/20 source-index split so each original and protected counterpart remains in the same partition.Source: §A.2 · p. 16

B. Full Cross-Domain Results

The cross-domain section points to a table comparing SD3.5 results across datasets, with paired training-image fidelity deterministic for a fixed protected dataset.

B. Full Cross-Domain Results

  • Table 1 compares SD3.5 across datasets; EnTruth injects 0.5% auxiliary samples and has no directly comparable paired-LPIPS value.The accompanying results summary identifies the table as covering generative utility and image fidelity.Source: Table 1 · p. 17, B. Full Cross-Domain Results · p. 17

C. Full Pokemon Benchmark Results

The Pokemon benchmark section expands per-generation discrimination results by architecture and false-positive-rate target.

C. Full Pokemon Benchmark Results

D. Protection-Hyperparameter Ablations

The ablations vary blending strength, protected fraction, and Gaussian bandwidth. Stronger blending and a larger protected fraction improve discrimination, while the bandwidth sweep saturates near the default operating point.

D. Protection-Hyperparameter Ablations

E. Dataset Protection Cost

SpectralCanary’s trace becomes detectable early in fine-tuning, while dataset protection takes 0.39 seconds per image on CPU. The protection pipelines differ, so operation counts are not directly comparable.

E. Dataset Protection Cost

  • 0.39 seconds per image is the reported CPU protection time for SpectralCanary.The comparison measures end-to-end time on identical hardware because the methods use different protection pipelines.Source: E. Dataset Protection Cost · p. 19

F. The Trace Appears Early in Fine-Tuning

  • After five epochs, AUC reaches 0.956 and TPR at 1% FPR reaches 0.730; performance approaches saturation by epochs 10–15 and remains similar through epoch 80.Each checkpoint uses 100 matched Pokemon generations per condition with the same captions.Source: Figure 10 · p. 19, F. The Trace Appears Early in… · p. 19

G. Full Fine-Tuning Evaluation

The supplementary evaluations test full-parameter tuning, model-level decisions, query budgets, spectral placement, and adaptive sanitization. They report robust detection in several settings, with low-frequency placement strongest in matched-energy comparisons and aggressive removal constrained by image fidelity.

G. Full Fine-Tuning Evaluation

  • Across four settings, full-parameter fine-tuning yields average AUC of 95.2% and average TPR at 1% FPR of 83.8%; training-time augmentation is most challenging, with TPR at 1% FPR of 61.9%.The main paper evaluates LoRA fine-tuning; this evaluation tests full-parameter fine-tuning of SD1.5 on Pokemon.Source: G. Full Fine-Tuning Evaluation · pp. 19–20

H. Model-Level Verification

  • Model-level verification calibrates a method- and architecture-specific threshold once, holds it fixed across transformations, and classifies each test model using 30 queries.The held-out evaluation includes 320 test models.Source: H. Model-Level Verification · p. 20

I. Generations from Models Fine-Tuned on Protected Data; N. Additional Protected-Image Examples; O. EnTruth Injected-Sample Visualization

J. Multi-Query Performance at Different Query Budgets

K. Effect of Spectral Location on Trace Transfer

  • With matched band size and perturbation energy, low-band coating is strongest, mid-frequency coating is markedly weaker, and high-frequency coating degrades substantially on ROCOv2.The comparison averages results across three canaries on Pokemon and ROCOv2 with SD1.5.Source: K. Effect of Spectral Location on… · p. 22

L. Effect of the Canary’s Spatial Structure

  • The fixed natural canary yields the highest discrimination, particularly at 1% FPR, while both phase-randomized controls retain substantial discrimination.The controls approximately preserve spectral magnitude and pixel-value distribution while randomizing spatial organization.Source: Figure 13 · p. 23, L. Effect of the Canary’s Spatial… · pp. 23–24

M. Additional Adaptive Spectral-Sanitization Settings

  • Collection centering cancels the shared canary but also removes genuine spectral mean; full donor replacement and direct low-frequency removal are omitted for severe fidelity loss.Full donor replacement also transfers substantial coarse structure from the donor.Source: M. Additional Adaptive… · p. 25

P. EnTruth Ablation: Soft- and Hard-Trigger Auditing

Q. Repurposing T2I Data Poisoning for Dataset-Use Auditing

The section compares repurposed semantic branding with SpectralCanary and outlines a Gaussian analysis of trace learnability under diffusion and flow-matching objectives. The analysis uses explicit spectral assumptions and distinguishes normalized mean shift from perceptual visibility.

Q. Repurposing T2I Data Poisoning for Dataset-Use Auditing

  • Under the Pokemon protocol, Silent Branding [12] transfers less consistently than SpectralCanary and depends on semantically appropriate regions and auxiliary logo models; this does not rule out all semantic insertion methods.The evaluated branding pipeline is a weaker dataset-use auditing baseline in this protocol.Source: Q. Repurposing T2I Data Poisoning for… · p. 26

R. Theoretical Analysis

  • The theoretical analysis uses an idealized Gaussian setup for SD1.5-style noise prediction and SD3.5-style linear flow matching, separating variance contraction from the shared population mean shift.Source: R. Theoretical Analysis · p. 26

R.1 Assumptions and Setup

  • The analysis assumes Gaussian, frequency-independent clean-image coefficients and decreasing spectral power beyond a small cutoff; its normalized trace energy is not a model of human perceptual visibility.b(f) measures squared mean shift in units of natural coefficient variance, and B2 is its total normalized energy.Source: §R.1 · p. 27

R.2 Learnability of the Low-Frequency Trace

The analysis separates the low-frequency trace’s mean-shift contribution from the exact coating’s variance-contraction contribution, then relates trace learnability to frequency and training objective. It supports low-frequency carriers while qualifying how the Gaussian-scale ablation bears on optimal mask choice.

R.2 Learnability of the Low-Frequency Trace

  • The clean-data KL of the mean-shift surrogate, B^2/2, is a lower bound for the exact coating, which adds a nonnegative variance-contraction term.The two contributions are additive because the cross term vanishes for zero-mean V.Source: §R.2 · pp. 28, 30
  • Low-frequency carriers persist above the noise floor longer because spectral power decreases with frequency, although the mean-shift contribution is not necessarily greatest at the very lowest frequencies.For diffusion noise prediction, the mean-shift component is governed by w_t(f)(1 − w_t(f)); linear flow matching has a different objective-specific weight.Source: §R.2 · p. 28
  • For either training objective, the mean-shift lower-bound component is maximized by allocating trace energy to frequencies with the largest objective-specific weight G_o(f).For diffusion noise prediction, the weight approaches zero when spectral power is very small or very large, subject to the stated timestep-distribution condition.Source: §R.2 · p. 29
  • The Gaussian-scale ablation does not establish an interior optimum: it changes total trace energy and operates in pixel space while training uses learned latent spaces.The three tested scales are described as a qualitative comparison, not fixed-budget empirical evidence for the theorem’s allocation result.Source: §R.2 · p. 31

S. Extended Related Work

  • SpectralCanary distributes a weak shared visual component across original images, unlike pixel-level traces or visibly out-of-distribution repeated-template canaries.It leaves captions unchanged, unlike trigger-dependent approaches that can be vulnerable to caption cleaning.Source: S. Extended Related Work · pp. 31–32

Related papers

Loading 2610.31659v1…