PaperlayerView exact revision on arXiv

Source-linked AI summary

CopyShield: A Cross-Level Benchmark of Copyright Defenses in LLMs

Maryam Alshehyari, Dushyant Singh Chauhan, Samuele Poppi, Martin Takac, Salem Lahlou, Nils Lukas

arXiv:2609.01161v1cs.LG

TL;DR

Copyright defenses for LLMs are difficult to compare because prior evaluations use incompatible protocols and may miss non-literal leakage. CopyShield benchmarks three defenses at output, behavioral, and representation levels under a shared evaluation, finding distinct trade-offs and model-dependent DPO degeneracy.

  • Problem

    Prior copyright-defense studies lacked a common benchmark jointly comparing intervention levels while calibrating non-literal leakage and evaluating utility under identical conditions.

  • Method

    CopyShield evaluates contrastive decoding, DPO, and activation intervention across two model families, five public-domain books, query types, leakage criteria, utility, and degeneracy.

  • Results

    Intervention level is associated with distinct compliance-utility trade-offs: output-level suppression plateaus, DPO can induce paraphrase-loop degeneracy, and activation intervention reduces non-literal flags through broad blocking.

  • Takeaways & Limitations

    CopyShield supplies cross-level reference baselines and identifies targeted non-literal suppression as an open challenge.

  • Takeaways & Limitations

    The benchmark uses public-domain literary works, so transfer to copyrighted literary evaluation requires future verification when ethically and legally feasible.

Abstract

from arXiv · show

Large language models can reproduce memorized text verbatim, yet copyright defenses are usually evaluated under incompatible protocols. We introduce CopyShield, a controlled benchmark comparing three representative defenses at distinct intervention levels: contrastive decoding (output), Direct Preference Optimization (behavioral), and activation intervention (representation). We evaluate CopyShield on two model families, LLaMA-3.1-8B and Mistral-7B-v0.3, using controlled memorization over five public-domain books and a shared protocol measuring literal leakage, calibrated non-literal leakage, utility, and degeneracy. Across these methods, intervention level is associated with distinct compliance-utility trade-offs. On LLaMA-3.1-8B, contrastive decoding remains near-degeneracy-free (0-2%) but reaches a literal-suppression floor at NV-Recall 0.192-0.203. DPO nearly eliminates literal leakage (0.263 to 0.002) but induces paraphrase-loop degeneracy in 58% of QA outputs, with no utility gain over the SFT baseline. Activation intervention attains the lowest non-literal flagging rate (1/200) by blocking 84% of non-literal queries before generation. Human evaluation confirms that DPO has low coherence, whereas activation lowers perceived copyright risk through broad refusal. On Mistral-7B-v0.3, the output- and representation-level patterns persist, while DPO degeneracy falls to 10-14%, showing that its severity is model-dependent. Together, CopyShield provides cross-level reference baselines and identifies targeted non-literal suppression as an open challenge. The code is available at https://github.com/spotai-mbzuai/CopyShield.git.

1 Introduction

CopyShield addresses the lack of a common protocol for comparing copyright defenses across intervention levels. It benchmarks output-, behavioral-, and representation-level methods while jointly evaluating literal and non-literal leakage, utility, and operational trade-offs.

  • Research gap: Prior benchmarks used different datasets, metrics, and conditions, limiting direct comparison of copyright-defense trade-offs.Existing work also omitted either preference-based or representation-level defenses, and often lacked calibrated non-literal thresholds or utility evaluation.
  • Motivation: Non-literal reproduction matters because semantically equivalent paraphrases can pose copyright risk even when verbatim copying is suppressed.CopyShield therefore treats calibrated non-literal leakage as a separate evaluation target rather than relying only on surface overlap.
  • Benchmark: CopyShield compares contrastive decoding, DPO, and activation intervention at output, behavioral, and representation levels, respectively.The benchmark holds the memorized model and evaluation protocol common so differences can be associated with intervention level and characteristic failure modes.
  • Evaluation scope: The benchmark spans LLaMA-3.1-8B and Mistral-7B-v0.3, five public-domain books, three query types, calibrated non-literal thresholds, and multi-dimensional leakage and utility metrics.Metrics include literal leakage, calibrated embedding-similarity flagging, QA scores, refusal rate, and degeneracy rate.
  • Findings: CopyShield identifies an output-level suppression floor, behavioral-level paraphrase-loop degeneracy, and representation-level over-blocking as distinct compliance-utility trade-offs.The output- and representation-level patterns persist across both model families, whereas DPO degeneracy varies across models.
  • Novelty: CopyShield integrates leakage forms, calibrated thresholds, utility evaluation, multi-book testing, and multi-method comparison under identical conditions.This common protocol is presented as the benchmark’s distinction from prior evaluation frameworks.

3 Problem Formulation

The formulation represents copyright defense as wrapping a memorized model with an algorithm that must suppress protected-corpus leakage while preserving legitimate question answering. CopyShield compares three defenses by where they intervene in generation and defines literal, non-literal, and utility criteria for evaluation.

  • 3 Problem Formulation: The problem consists of a memorized model M, protected corpus D, user query Q, and defense algorithm A.Experiments use LLaMA-3.1-8B fine-tuned on five public-domain books, with those books forming D.
  • 3 Problem Formulation: Without defense, R ∼ M(Q); with defense, R ∼A(M, Q, D) must avoid reproducing D while retaining legitimate-query utility.The paper treats leakage suppression and useful answering as simultaneous objectives.
  • 3 Problem Formulation: Literal leakage is declared when NV-Recall(R, D) > 0.5, with NV-Recall counting verbatim-matched words through a merge-and-filter pipeline.The 0.5 threshold is a severity operating point interpreted as more than half of the reference being verbatim copied.
  • 3 Problem Formulation: Non-literal leakage is declared when response-reference cosine similarity exceeds a threshold calibrated at α = 0.001 over neutral text.The calibration is intended to distinguish semantic overlap with D from baseline lexical similarity.
  • 3 Problem Formulation: Literal queries request continuation of a 150-token book prefix, whereas non-literal queries ask narrative questions whose semantically reproducing answers should be suppressed.These query types test different forms of protected-content reproduction.
  • 3 Problem Formulation: QA queries ask factual questions whose brief answers should remain permitted, requiring the defense to distinguish them from literal and non-literal requests.The formulation makes this distinction central to preserving utility.
  • 4 Methods: Contrastive decoding changes output logits, DPO changes behavioral parameters, and activation intervention changes hidden-state representations.All three share the same fine-tuned baseline M and use knowledge of D at inference.
  • 4.1 Method 1: Contrastive Decoding: Contrastive decoding subtracts copyright-specialist logits from model logits, with λ controlling suppression strength and an adaptive plausibility constraint limiting degeneration.The method modifies logits without altering M’s parameters and can be reversed by removing the specialist model.

M Q R

CopyShield compares defenses that intervene at different stages of generation: output logits, behavioral parameters, and hidden-state representations. These methods respectively suppress shared high-probability tokens, learn response preferences, or refuse before generation.

  • Output-level intervention: Contrastive decoding subtracts specialist-model logits from base-model logits at each token step to suppress tokens both models score highly.It modifies output distributions without changing the model parameters.
  • Behavioral-level intervention: DPO fine-tunes model parameters on preference pairs that favor non-infringing responses over copyrighted ones while retaining QA examples.Literal and non-literal pairs prefer refusals, whereas QA pairs preserve factual answering.
  • Representation-level intervention: Activation intervention uses a hidden-state classifier to detect copyright-reproducing intent and trigger refusal before output generation.The intervention is representation-level and does not modify the underlying model.
  • Representation-level intervention: Layer 20 is selected for activation intervention, where the classifier achieves AUC = 0.936, TPR = 87%, and FPR = 11.3% at threshold θ = 0.5.At inference, responses are replaced with a fixed refusal when predicted copyright probability exceeds θ = 0.5.

5 Experimental Setup

The experimental setup uses controlled memorization, non-overlapping evaluation data, three query types, calibrated semantic-leakage detection, and shared utility and degeneracy metrics. The design separates protected and neutral corpora while keeping training, calibration, and evaluation conditions controlled.

  • Corpora and pipeline: The protected corpus contains five cleaned public-domain novels totaling ≈907,979 tokens, and it is reused across memorization and defense stages.The books are Pride and Prejudice, Frankenstein, Dracula, Moby-Dick, and The Adventures of Sherlock Holmes.
  • Corpora and pipeline: A separate neutral corpus contains five non-overlapping public-domain books totaling ≈589,432 tokens for threshold calibration and classifier negatives.Its books include Alice in Wonderland, Crime and Punishment, The Great Gatsby, Romeo and Juliet, and Wuthering Heights.
  • Defense data: DPO training uses 800 preference triples: 200 literal, 200 non-literal, and 400 QA examples scored with NV-Recall or calibrated embedding similarity.Synthetic refusals are preferred for literal and non-literal prompts, while QA pairs preserve factual responses.
  • Defense data: Activation-classifier training uses 800 balanced inputs spanning literal positives, non-literal positives, neutral passages, and QA negatives.Neutral passages are length-matched to literal positives to prevent input length from serving as a shortcut.
  • Evaluation sets: Evaluation uses 200 samples each for literal, non-literal, and QA queries, with references of 200, 382, and 49 tokens on average, respectively.Literal and non-literal queries must be blocked, while QA queries must remain permitted.
  • Calibration and metrics: The non-literal threshold is calibrated from cross-corpus embedding similarities at α = 0.001, yielding 0.625 with 100% TPR and ≤0.4% FPR on calibration data.The threshold is a reproducible screening rule for unusually high semantic overlap, not a legal determination.
  • Calibration and metrics: Utility is measured by uQA = (correctness + completeness + coherence)/3 on a 1–5 scale, while degeneracy uses 4-gram uniqueness and repeated-sentence criteria.All methods use greedy decoding with max_new_tokens=200, and absolute utility scores are reported directly.

6 Results

Across the shared evaluation matrix, every defense reduces leakage relative to SFT, but intervention level determines distinct trade-offs among literal and non-literal suppression, utility, coherence, and degeneracy.

  • Literal leakage: NV-Recall falls from 0.263 under SFT to 0.002 with DPO, eliminating all high-leak samples and perfect copies.Activation reaches 0.029, while contrastive decoding plateaus at 0.192–0.203 as λ increases.
  • Non-literal leakage: 1/200 non-literal samples is flagged by activation intervention, but this reflects blocking 168/200 queries before generation rather than fine-grained discrimination.Among unblocked queries, the flag rate is 1/32 (3.1%).
  • Utility and degeneracy: 58% of LLaMA-3.1-8B QA outputs degenerate under DPO, compared with 8% for SFT, while contrastive decoding remains at 0–2%.DPO’s paraphrase-loop behavior coexists with utility of 1.37, nearly identical to SFT’s 1.38.
  • Utility and degeneracy: Activation intervention has the highest QA utility at 1.51, whereas contrastive decoding scores 1.25 at λ=2.0 and 1.19 at λ=8.0.Activation’s utility comes with a 16% false-positive QA refusal rate.
  • Human validation: Human evaluation assigns DPO the lowest non-literal coherence score, 2.0, alongside 55% paraphrase-loop degeneracy.The human evaluation captures a failure mode that the AI judge’s partial-credit utility score does not reveal.
  • Cross-model results: On Mistral-7B-v0.3, output- and representation-level patterns persist, while DPO degeneracy falls to 10% and utility reaches 1.87.The reported DPO degeneracy difference shows model-dependent severity for the behavioral defense.

7 Discussion

CopyShield shows that intervention level shapes both the form and magnitude of copyright compliance–utility trade-offs. Output-, behavioral-, and representation-level defenses each reduce leakage through distinct mechanisms, with targeted non-literal suppression remaining unresolved.

  • Intervention level shapes both the form and magnitude of compliance–utility trade-offs across output-, behavioral-, and representation-level defenses.The benchmark holds the model, corpus, query distribution, and calibrated thresholds fixed for like-for-like comparison.
  • Output-level defense: Contrastive decoding remains near-degeneracy-free but reaches a literal-suppression floor at NV-Recall 0.192 by capping logit subtraction.The adaptive plausibility constraint limits suppression to preserve fluent generation.
  • Behavioral-level defense: 58% of LLaMA-3.1-8B QA outputs exhibit DPO paraphrase-loop degeneracy, with no utility advantage over the SFT baseline.On Mistral-7B, the same recipe degenerates on only 10% of QA outputs, indicating model-dependent severity.
  • Representation-level defense: Activation intervention attains a 1/200 non-literal flagging rate by blocking 84% of non-literal queries before generation.Its detector identifies book-content intent in hidden states rather than separating narrative discussion from verbatim copying.
  • Open challenge: None of the three defenses delivers targeted non-literal suppression, leaving semantic reproduction as an open challenge.Proposed directions include finer-grained representation classifiers and training signals that explicitly penalize semantic similarity.
  • Deployment implications: Defense selection reduces to choosing among characterized trade-off profiles rather than selecting a universally dominant method.Contrastive decoding favors reversible, near-degeneracy-free operation; activation intervention favors deployments prioritizing non-literal leakage.

8 Conclusion

CopyShield benchmarks three copyright defenses at distinct intervention levels under a shared protocol across LLaMA-3.1-8B and Mistral-7B-v0.3. The methods show distinct, model-dependent compliance–utility trade-offs, while targeted non-literal suppression remains the main open challenge.

  • CopyShield compares contrastive decoding, DPO, and activation intervention at output, behavioral, and representation levels.The benchmark uses a shared evaluation protocol across two model families.
  • Contrastive decoding remains near-degeneracy-free but reaches a literal-suppression floor, whereas DPO strongly suppresses literal leakage with severe LLaMA-3.1-8B paraphrase-loop degeneracy.DPO degeneracy is substantially lower on Mistral-7B-v0.3, while activation intervention achieves low non-literal flagging through broad pre-generation blocking.
  • The trade-offs are associated with both intervention level and model configuration, and no single defense dominates across all evaluation dimensions.
  • Targeted non-literal suppression remains the main open challenge for future defenses.Suggested directions include selective representation-level detection, semantic-reproduction objectives, and hybrid defenses.

Limitations

CopyShield’s conclusions are bounded by its controlled corpus, non-adversarial threat model, limited model and seed coverage, single AI judge, and unablated DPO preference-data design.

  • Corpus: The benchmark uses five public-domain novels as a controlled proxy for copyrighted content because reproducible evaluation requires freely includable text.Transfer to copyrighted literary works requires future verification when ethically and legally feasible.
  • Threat model: Queries reflect plausible user behavior rather than adversarial prompt-engineered extraction, leaving deliberate extraction attacks for future work.
  • Model coverage: Evaluation covers LLaMA-3.1-8B and Mistral-7B-v0.3, while broader coverage of larger scales and additional model families remains an extension.The qualitative level-dependent pattern holds across both families, but DPO degeneracy severity is model-dependent.
  • Evaluation: Utility and quality scores rely on a single AI judge, so multi-judge averaging or targeted human evaluation would provide complementary signal.
  • DPO training: DPO’s task-aware QA preference data is a design choice whose isolated contribution was not ablated or quantified causally.The preference set contains 400 task-aware QA preference pairs.
  • Seed coverage: LLaMA-3 experiments use one random seed, while Mistral experiments use two, so behavior near the paraphrase-loop boundary may vary across seeds.Broader replications should report seed variance for all quantities.

Reproducibility

The study provides reproducibility resources and documents its data, compute, and evaluation safeguards. It uses public-domain texts, releases the research artifacts, and acknowledges seed and evaluator limitations.

  • Compute and models: The experiments use Llama-3.1-8B, Llama-3.2-1B, and Mistral-7B-v0.3, with training performed on a single NVIDIA A100 80GB GPU.Approximate training times were 12 hours for SFT, 8 hours for the specialist model, and 2 hours for DPO.
  • Experimental coverage: The main LLaMA-3 experiments use one random seed, while Mistral-7B additionally reports a second seed.The main seed is 42; the additional Mistral seed is 7.
  • Data governance: The protected corpus consists entirely of public-domain Project Gutenberg texts, and no copyrighted material is included or released.The corpus requires no licensing.
  • Released artifacts: The repository releases code, fine-tuning scripts, the evaluation harness, calibration artifacts, trained classifiers, and evaluation sets.
  • Evaluation caveat: The utility scores come from Claude Sonnet, a commercial AI judge whose rubric interpretations may not generalize to human evaluation.

A GRPO Experiment Details

The GRPO experiment trains a behavioral defense with reward functions tailored to literal leakage, non-literal similarity, and QA utility. Training plateaus early, with refusal learning, objective conflict, and low response diversity identified as contributing factors.

  • GRPO setup: GRPO generates four completions per prompt at temperature 0.7 and normalizes rewards within each group into advantage estimates.The KL coefficient is β = 0.05.
  • Reward design: Literal rewards combine NV-Recall, Phase-1 LCS, and ROUGE-L, while non-literal rewards use calibrated embedding similarity and QA rewards use clipped reference-answer similarity.Refusals receive task-specific rewards, including −3.0 for QA.
  • Training behavior: Reward shows no statistically significant improvement over 1500 steps, with R2 = 0.007 and p = 0.32.The reported explanations are early refusal saturation, conflicting suppression and QA objectives, and low within-group variance.
  • Data partitioning: The protected corpus is reused across training, calibration, and evaluation through mutually non-overlapping splits intended to prevent leakage.
  • Preference data: A representative DPO preference pair selects a refusal with a summary over a rejected verbatim copy.
  • Representation defense: A logistic-regression classifier is trained on last-token hidden states across six layers, with layer 20 selected for the representation-level defense.Layer 20 has the highest AUC-ROC and lowest QA false-positive rate among the candidates.

D Training Configurations

The supplementary configurations document training settings, layer selection, per-book leakage, and utility or degeneracy measurements. Results show book-dependent suppression and distinct evaluation behavior across methods.

  • Training configurations: Appendix tables provide full hyperparameters for the SFT baseline, specialist model, and DPO adapter, all trained on a single GPU.
  • Layer selection: Layer 20 is selected using classifier AUC-ROC and false-positive rate across candidate layers.
  • Per-book leakage: Pride and Prejudice has the highest residual leakage under contrastive decoding at all λ values, with NV-Recall > 0.44.
  • Per-book leakage: DPO reduces per-book NV-Recall to near-zero, while activation intervention rates range from 75.0% for Dracula to 97.5% for Frankenstein.The variation suggests distinct activation patterns at layer 20 across books.
  • Non-literal evaluation: Activation intervention blocks all 50 non-literal evaluation queries, so its non-literal utility reflects refusal behavior.
  • Evaluation measures: Tables report utility sub-dimensions for QA and non-literal tracks, alongside degenerate and QA refusal rates by method.

F.1 Robustness Analyses

Robustness analyses test threshold and embedding-model sensitivity and add a second Mistral seed. The reported method ordering and level-dependent pattern remain stable, while DPO degeneracy is milder on Mistral.

  • Threshold sensitivity: Re-thresholding NV-Recall across 0.3, 0.4, 0.5, 0.6, and 0.7 leaves method ordering unchanged on both model families.The 0.5 threshold is an interpretable severity point, not a statistically calibrated cutoff.
  • Embedding sensitivity: A second sentence encoder yields per-sample score correlations of r = 0.76–0.94 and agrees on 90–100% of binary high-similarity flags.Activation intervention remains the lowest-flagging method across the tested models and Mistral seeds.
  • Seed robustness: The second Mistral-7B-v0.3 seed preserves the level-dependent pattern, while DPO QA degeneracy remains mild at 14%.
  • Evaluation protocol: The robustness analyses evaluate leakage, utility, and degeneracy on the same 200-item sets using Claude Sonnet.
Loading 2609.01161v1…