PaperlayerView exact revision on arXiv

Source-linked AI summary

SEW: Style-Encoded Watermarking of LLM-Generated Code

Soohan Lim, Hyundong Jin, Yo-Sub Han

arXiv:2609.39414v1cs.CR

TL;DR

Code watermarking must support provenance tracking while distinguishing marked from unmarked code without compromising behavior, and existing approaches face trade-offs in generation or completed-code watermarking. SEW applies structure-conditioned style transformations with calibrated, context-aware detection. Across CodeContests, it improves mean TPR@FPR5% by 12.44% over the strongest baseline per setting and shows a 0.94% mean relative decrease under four code-editing attacks.

  • Problem

    Code watermarking must distinguish marked from unmarked generated code while preserving program behavior, but generation-time changes can trade detectability against correctness and completed-code choices can recur predictably.

    Source: §1 · p. 1

  • Method

    SEW watermarks completed code using secret-key and structure-conditioned style rules, human-code style-preference calibration, and aggregation across structurally matching locations.

    Source: §1 · pp. 1–2

  • Results

    Across CodeContests, SEW achieves a 12.44% mean relative TPR@FPR5% improvement over each setting’s strongest baseline, with a 0.94% mean relative decrease under four non-LLM code-editing attacks.

    Source: §6 · p. 9

  • Takeaways & Limitations

    SEW supports watermarking completed code without model training or decoding access while preserving Vanilla pass@1 on the evaluated settings.

    Source: §6 · p. 9

    Very short programs may provide too few eligible style sites for reliable detection, and incomplete or structurally invalid code can limit watermark coverage.

    Source: H LIMITATIONS · p. 18

Abstract

from arXiv · show

Code watermarking supports provenance tracking for code generated by LLMs. Modifying token selection to embed watermarks as an LLM generates code can create a trade-off between detectability and functional correctness. Other methods instead watermark completed code using predefined transformations or trained neural models. Recurring patterns can make watermark choices predictable across programs, while treating patterns common in unwatermarked code as watermark evidence can cause false detections. We therefore introduce SEW, which embeds and detects watermarks in already generated code through three components: (i) code style rules collected from style guides and transformation rules, with style choices determined by a secret key and each program's structural context; (ii) style-preference calibration, which evaluates watermark evidence using style probabilities estimated from human-written code; and (iii) context-aware style aggregation, which combines evidence from structurally matching locations assigned the same code style choice, preventing repeated applications of that choice from inflating watermark evidence. On CodeContests across three LLMs and three programming languages, SEW achieves a mean relative improvement of 12.44% in TPR@FPR5% over the baselines and is robust to four non-LLM code-editing attacks, with only a 0.94% mean relative decrease. Our code is available at https://github.com/suhanmen/SEW.

1 INTRODUCTION

Code watermarking must support provenance detection without compromising program behavior, but generation-time and completed-code methods face distinct trade-offs. SEW uses structure-conditioned style choices with calibrated, context-aware detection; on CodeContests it achieves high detection while preserving pass@1.

1 INTRODUCTION

  • Generation-time token changes can trade detectability against functional correctness, while recurring completed-code choices may become predictable across programs.Existing completed-code approaches use fixed transformations or learned transformation selectors.Source: §1 · p. 1
  • SEW selects between syntax or formatting variants using a secret key and structural context, then calibrates style evidence and aggregates matching locations to avoid overweighting repeated choices.Calibration accounts for patterns that also occur without watermarking.Source: §1 · pp. 1–2
  • Table 1 compares approaches on model-free detection, resistance to pattern inference, and preservation of matched-condition pass@1.Model-free detection requires neither the generating LLM nor a separately trained extraction model.Source: §1 · p. 2, Table 1 · p. 2
  • 99.14% mean TPR@FPR5% on CodeContests, with pass@1 preserved across all nine model–language settings.Figure 1 compares detection and pass@1 across three LLMs and three programming languages.Source: Figure 1 · p. 2, §1 · p. 2

2 RELATED WORK

Prior code-watermarking work embeds signals during generation or transforms completed programs, while code-style research covers conventions, attribution, transfer, and semantics-preserving transformations. SEW applies context-appropriate syntax and formatting alternatives to completed code.

2 RELATED WORK

  • Generation-time code watermarking methods such as SWEET and STONE require decoding access and restrict where embedding occurs.SWEET excludes low-entropy positions; STONE restricts embedding to non-syntactic tokens.Source: §2 · p. 3
  • Completed-code watermarking spans trained neural embedding and extraction methods and ACW's predefined transformations, whose recurring style choices can be inferred from collected programs.SEW instead uses structure-conditioned, secret-key-guided transformations without decoding access or neural training.Source: §2 · p. 3
  • Code-style research includes conventions, authorship attribution, style transfer, LLM-generated-code inconsistencies, and semantics-preserving transformations; SEW targets context-appropriate syntax and formatting alternatives.The pipeline overview is identified in Figure 2.Source: §2 · p. 3, Figure 2 · p. 3

3 METHOD

SEW watermarks completed code by assigning style choices from a secret key and structural context, then tests for watermark evidence using calibrated natural style preferences and context-aware aggregation.

3 METHOD

  • SEW embeds watermarks in completed code and detects them without access to the generation model, using structure-aware style choices and calibrated, context-aware evidence.Matching structural locations receive consistent style choices; detection accounts for natural style preferences and repeated applications.Source: §3 · pp. 3–4

3.1 STYLE RULE LIBRARY

  • The rule library defines binary style alternatives with shared applicability checks, while calibration estimates natural variant frequencies from separate human-written code.The catalogs contain 29 Python, 22 Java, and 19 C++ rules; calibration reduces evidence for naturally common matches and affects detection, not embedding.Source: §3.1 · p. 4

3.2 WATERMARK EMBEDDING

  • Embedding selects eligible sites using semantic and structural checks, then derives a target style bit from the secret key, site identifier, and program context.Structurally matching sites in the same context receive the same target; the same rule can receive different targets across programs.Source: §3.2 · pp. 4–5

3.3 WATERMARK DETECTION

  • Detection groups matching sites by structural identifier and program context so each shared style choice contributes one observation, then tests agreement against calibrated natural-style probabilities.It also tests syntax-rule grades separately and combines the two tests with a Bonferroni correction.Source: §3.3 · p. 5

4 EXPERIMENTAL SETUP

The evaluation uses CodeContests solutions across three languages and three open-weight LLMs, measuring watermark detection and functional correctness against token-level and post-hoc baselines.

4.1 DATASETS AND EVALUATION METRICS

  • CodeContests validation and test splits provide Python, Java, and C++ solutions; evaluation measures TPR@FPR5%, AUROC, and pass@1.Problems whose Vanilla output did not produce an evaluable program within the generation token budget were excluded; evaluation sets are shared across methods within each model–language setting.Source: §4.1 · p. 6

4.2 EVALUATED MODELS AND BASELINES

  • Three open-weight LLMs generate nine model–language settings, and SEW is compared with five token-level and three post-hoc baselines.ACW is evaluated only in Python; Table 4 reports TPR@FPR5% under five code-editing attacks.Source: §4.2 · p. 6, Table 4 · p. 7

5 RESULTS AND ANALYSIS

Across the evaluated settings, SEW combines high detection with preserved functional correctness, and retains detection under code-editing attacks. It also resists rule inference, while targeted flipping and ablations reveal how its aggregation and calibration components behave.

5.1 MAIN RESULTS

  • 99.14% mean TPR@FPR5%, a 12.44% mean relative improvement over each setting’s strongest baseline, and highest AUROC in all nine settings; pass@1 matched Vanilla.TPR@FPR5% ranged from 96.60% to 100%.Source: §5.1 · p. 7

5.2 ROBUSTNESS TO CODE-EDITING ATTACKS

  • 98.21% mean TPR@FPR5% under four non-LLM attacks, with a 0.94% mean relative decrease; comment removal and identifier renaming left detection unchanged in all nine settings.Formatting retained 89.70%–99.51% TPR@FPR5%.Source: §5.2 · p. 7
  • Under LLM rewriting, SEW averaged 74.79% TPR@FPR5% and remained highest in all nine settings.Rewriting that changes siteID(s) or program context can change the recomputed target.Source: §5.2 · pp. 7–8

5.3 ROBUSTNESS TO TARGETED STYLE FLIPPING

  • At 50% and 75% targeted flipping, SEW retained 82.53%–95.40% and 53.19%–70.67% TPR@FPR5%, respectively, across the three languages.Repeated sites sharing a structural identifier and context are grouped into grades and represented by a majority observation.Source: §5.3 · p. 8, Figure 3 · p. 8

5.4 RESISTANCE TO RULE INFERENCE

  • At N = 10 observed programs, rule-inference accuracy was 19.91% for SEW versus 85.97%–99.67% for baselines; SEW’s targets depend on site and program context.As N increases, more site–context combinations become known, but unseen combinations limit prediction.Source: §5.4 · p. 8, Table 5 · p. 9

5.5 ABLATION STUDY

  • Combining calibration with grade-level detection achieved the highest attacked and flipping TPR@FPR5% in every language (97.18%–99.00% and 77.49%–88.00%), while matching grade-level detection alone without attacks.Without attacks, calibration raised T@5 from 89.19% to 97.84% for gemma-4 on Java and from 90.13% to 99.57% for gpt-oss on C++; the combined configuration improved over Base in seven of nine settings.Source: §5.5 · p. 9

6 CONCLUSION

SEW watermarks completed LLM-generated code without model training or decoding access, using keyed, context-dependent styles and calibrated, aggregated evidence. The experiments used CodeContests and human-posted LeetCode solutions to estimate natural style probabilities.

6 CONCLUSION

  • Across three LLMs and three languages on CodeContests, SEW achieved 99.14% mean TPR@FPR5% and a 12.44% mean relative gain over the strongest baseline per setting while matching Vanilla pass@1.Four non-LLM editing attacks caused a 0.94% mean relative TPR@FPR5% decrease; Appendix H notes limited evidence in short code and reduced coverage for incomplete or structurally invalid code.Source: §6 · p. 9

A IMPLEMENTATION DETAILS

  • The experiments used greedy-decoded Vanilla outputs capped at 8,192 new tokens, and evaluated correctness by requiring passage of all public, private, and generated test cases.SEW and post-hoc baselines were applied to the same Vanilla outputs.Source: A IMPLEMENTATION DETAILS · p. 13
  • Style-preference calibration used user-posted LeetCode solutions: 11,834 Python, 10,643 Java, and 11,594 C++ files, excluding CodeContests evaluation solutions.SrcMarker and RoSeMary were trained on separate code datasets, not CodeContests.Source: A IMPLEMENTATION DETAILS · p. 13

B DATA CONSTRUCTION

The evaluation filters CodeContests outputs to focus on watermarking completed code, while retaining failures and abstentions in detection scoring. Natural style probabilities are estimated from a separate LeetCode corpus and reused during detection.

B DATA CONSTRUCTION

  • CodeContests problems were retained only when a canonical human solution existed in the target language, and outputs that exhausted the 8,192-token budget without completing a code block were excluded.This focuses the evaluation on completed-code watermarking; detecting signals without generated code falls outside its scope.Source: B DATA CONSTRUCTION · p. 13, Table 7 · p. 13
  • Filtering did not depend on compilation, test success, or watermarking success; embedding failures and abstentions in retained sets received a detection score of zero.The same Vanilla-based filtering criterion and retained problem set were used for all methods within each setting.Source: B DATA CONSTRUCTION · p. 13

C STYLE-PREFERENCE CALIBRATION COST

  • Natural variant probabilities were estimated from 10,643–11,834 LeetCode files per language in 23.2–24.7 seconds, then reused during detection.Calibration applies the fixed rule library and applicability checks without learning or modifying rules.Source: C STYLE-PREFERENCE CALIBRATION COST · p. 14, Table 8 · p. 14

D AUROC ANALYSIS UNDER CODE-EDITING ATTACKS

SEW retains high AUROC under the evaluated non-LLM code edits, but LLM rewriting causes larger losses and can change reconstructed watermark evidence. In rule inference, SEW’s context-dependent targets transfer less readily than recurring baseline choices.

D AUROC ANALYSIS UNDER CODE-EDITING ATTACKS

  • Under four non-LLM edits, SEW achieved AUROC of 96.26–100 across 36 model–language–attack settings; identifier renaming left AUROC unchanged in all nine settings.Comment removal changed AUROC by at most 0.01, while formatting retained AUROC of 96.79–99.76.Source: D AUROC ANALYSIS UNDER CODE-EDITING… · p. 14, Table 9 · p. 15
  • LLM rewriting yielded SEW AUROC of 80.95–99.56, exceeding post-hoc baselines in every setting and ranking highest among all methods in seven of nine.Changes to siteID(s) or program context can change the recomputed target, so preserved local style alone does not ensure unchanged evidence.Source: D AUROC ANALYSIS UNDER CODE-EDITING… · p. 14
  • AUROC and TPR@FPR5% can diverge: under Qwen3.5 C++ rewriting, STA-1 had higher AUROC than SEW, 90.43 versus 80.95, but lower TPR@FPR5%, 64.00% versus 66.00%.AUROC summarizes ranking across thresholds; TPR@FPR5% measures detection at the evaluated operating point.Source: D AUROC ANALYSIS UNDER CODE-EDITING… · p. 14

E RESISTANCE TO RULE INFERENCE

  • Across nine model–language settings, SEW had the lowest rule-inference accuracy; at N = 10, baselines reached 85.70%–99.77% versus 2.37%–26.85% for SEW.At N = 100, baseline accuracy remained 85.53%–99.95%, while SEW ranged from 17.53% to 53.14% across languages.Source: E RESISTANCE TO RULE INFERENCE · p. 15, Table 10 · p. 15, Table 11 · p. 16
  • SEW targets depend on the secret key, siteID(s), and program context, so decision transfer requires matching site–context combinations rather than merely recurring rules.The attack measures decision transfer under a fixed key, not secret-key recovery or unrestricted target prediction.Source: E RESISTANCE TO RULE INFERENCE · p. 15

F ROBUSTNESS TO THE CHOICE OF SECRET KEY

Across ten sampled keys, SEW retained similar detection performance across evaluated model–language settings. The results include both low key-to-key variation and a consistently lower-performing setting.

F ROBUSTNESS TO THE CHOICE OF SECRET KEY

  • Five of nine settings had 0.00 TPR@FPR5% standard deviation across keys; the other four ranged from 0.17 to 0.81.The largest variation was for gpt-oss on Python: standard deviations of 0.81 for TPR@FPR5% and 0.51 for AUROC, with means of 99.23% and 99.53, respectively.Source: F ROBUSTNESS TO THE CHOICE OF SECRET KEY · p. 16, Table 13 · p. 17
  • Qwen3.5 on Java had the lowest mean TPR@FPR5% at 96.60% and lowest mean AUROC at 97.43, with standard deviations of 0.00 and 0.07.The lower performance was consistent across sampled keys rather than driven by an outlying key.Source: F ROBUSTNESS TO THE CHOICE OF SECRET KEY · p. 16, Table 13 · p. 17
  • Changing the key changes HMAC outputs and can reassign style targets, yet detection performance remained similar across the ten sampled keys.The calibration table remains fixed across keys, while context-aware aggregation prevents repeated occurrences of one target decision from dominating evidence.Source: F ROBUSTNESS TO THE CHOICE OF SECRET KEY · p. 16
  • 99.24% mean TPR@FPR5% and 99.45 mean AUROC across nine settings and ten keys show strong aggregate detection across sampled keys.These setting-level results summarize 90 key–setting evaluations.Source: F ROBUSTNESS TO THE CHOICE OF SECRET KEY · p. 16, Table 13 · p. 17

G STRUCTURAL ANALYSIS OF ROBUSTNESS TO CODE-EDITING ATTACKS

SEW retained the highest TPR@FPR5% in all nine evaluated settings under LLM rewriting, while structural evidence locations largely survived four non-LLM attacks. The analysis links robustness to evidence reconstructed from structural tuples rather than surface tokens, names, or formatting.

G STRUCTURAL ANALYSIS OF ROBUSTNESS TO CODE-EDITING ATTACKS

  • For Qwen3.5 on Python, comment removal lowered TPR@FPR5% from 91.43% to 17.86% for KGW and from 86.43% to 12.14% for SWEET.Identifier renaming reduced SrcMarker and RoSeMary TPR@FPR5% to 7.10%–58.67% and 4.59%–56.67%, respectively; formatting reduced ACW to 0.00%–1.53%.Source: G STRUCTURAL ANALYSIS OF ROBUSTNESS TO… · p. 17
  • 52.55%–99.02% TPR@FPR5% under LLM rewriting left SEW highest in all nine settings.LLM rewriting regenerates tokens, can rename identifiers, and normalizes formatting; the strongest token-level baseline in each setting reached at most 64.00%.Source: G STRUCTURAL ANALYSIS OF ROBUSTNESS TO… · p. 17
  • SEW reconstructs targets from the secret key, structural identifier siteID(s), and program context χ(C), which exclude identifier names, line numbers, and comments.For Qwen3.5, χ(C) changed in 20.95%–22.86% of programs under rewriting, causing targets in affected programs to be recomputed from a different HMAC input.Source: G STRUCTURAL ANALYSIS OF ROBUSTNESS TO… · p. 17
  • Under LLM rewriting, program context remained unchanged in 77.14%–100% of programs and 70.77%–99.76% of grades survived.In Python, formatting normalization toward PEP 8 reduced the share of surviving grades that remained target-agreeing to 63.9%; despite such variation, SEW remained highest in all nine settings.Source: G STRUCTURAL ANALYSIS OF ROBUSTNESS TO… · p. 17
  • Across four non-LLM attacks, χ(C) was preserved in 98.47%–100% of programs and 97.19%–100% of grades survived across model–language settings.Comment removal changed 26%–37% of tokens and identifier renaming changed 10%–13%, while preserving nearly all structural evidence locations.Source: G STRUCTURAL ANALYSIS OF ROBUSTNESS TO… · p. 17, Table 14 · p. 18

H LIMITATIONS

SEW’s coverage depends on having sufficient eligible style evidence and completed, structurally valid code. Its style rules also have rule-specific applicability conditions that determine which sites can be embedded and detected.

H LIMITATIONS

  • Short programs may have too few eligible style sites for reliable detection, and matching sites contribute only one grade-level observation.Even functionally correct programs can therefore provide insufficient watermark evidence.Source: H LIMITATIONS · p. 18
  • SEW requires completed, parseable code; incomplete or structurally invalid outputs can prevent site identification and style transformations, limiting coverage.These input defects precede embedding, so detection on supported inputs differs from coverage across all generated outputs.Source: H LIMITATIONS · p. 18

I CODE STYLE RULES AND APPLICABILITY CONDITIONS

  • A pattern is eligible only when it meets rule-specific type, side-effect, and structural conditions, which embedding and detection share.For example, comparison-direction changes require side-effect-free operands, and Python augmented-assignment conversion requires a numeric target.Source: I CODE STYLE RULES AND APPLICABILITY… · p. 18
  • The style library has 29 Python, 22 Java, and 19 C++ rules across expression, declaration, control-flow, and formatting categories.Formatting rules modify whitespace and layout; the other three categories contain syntax rules.Source: I CODE STYLE RULES AND APPLICABILITY… · p. 18, Table 15 · p. 19
Loading 2609.39414v1…