PaperlayerView exact revision on arXiv

Source-linked AI summary

Grammar-Guided Code Watermarking with Green Temperature

Hyundong Jin, Hyeseon An, Soohan Lim, Yo-Sub Han

arXiv:2610.05323v1cs.CRcs.AI

TL;DR

Code watermarking seeks to make generated code attributable, but token-probability changes can affect syntax or behavior, and existing methods do not partition directly over grammar-admissible continuations. GTCW combines grammar-guided keyed partitions with preference-aware green-temperature reweighting. Across five models and five benchmarks, it improves detection while keeping mean Pass@1 close to unwatermarked generation.

  • Problem

    Existing code watermarks mitigate probability-distortion risks with entropy-based or syntax-aware selection, but do not form partitions directly over the tokens admitted by the current grammar state.

    Source: §1 · p. 2

  • Method

    GTCW partitions grammar-admissible tokens into keyed green and red sets, then reweights green tokens by model preference at eligible positions.

    Source: §1 · p. 2

  • Results

    73.61% mean AUROC versus 67.83% for the strongest baseline, and 59.18% mean Pass@1 versus 59.58% for unwatermarked generation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across five models and five benchmarks, GTCW strengthens detection while largely preserving functional correctness.

    Source: §5 · p. 9

    Extensive LLM rewriting is the most disruptive tested transformation; GTCW AUROC falls to 51.4% and T@5 to 6.8%.

    Source: §4.5 · p. 9

Abstract

from arXiv · show

Large language model watermarking embeds detectable statistical signals during decoding, but the resulting changes to token probabilities can degrade generation quality. This trade-off is particularly important for code, where small changes in token selection can break syntax or alter program behavior. Existing code watermarking methods mitigate this risk through entropy-based insertion or syntax-aware token selection, but they do not directly construct the watermark over the set of continuations admitted by the current grammar state. We propose Grammar-Guided Code Watermarking with Green Temperature (GTCW), which integrates grammar-constrained decoding with probability-aware watermarking. At each decoding step, GTCW restricts the candidate set to grammar-admissible tokens and partitions this support into keyed green and red subsets. At eligible high-entropy positions, green temperature reweights the green tokens according to the model's relative preferences, strengthening the watermark signal while retaining the grammar constraint. Across five models and five benchmarks spanning four programming languages, GTCW achieves a mean AUROC of 73.61%, compared with 67.83% for the strongest baseline, while maintaining a mean Pass@1 of 59.18% versus 59.58% for unwatermarked generation. Our implementation is available at https://github.com/hyundong98/GTCW .

1 INTRODUCTION

Code watermarking must support provenance attribution while limiting token-probability changes that can harm syntax or behavior. GTCW constructs keyed partitions over grammar-admissible tokens and reweights green tokens by model preference, achieving stronger detection with nearly unchanged Pass@1.

1 INTRODUCTION

  • Existing methods reduce code-watermarking risks with entropy-based insertion or syntax-aware selection, but do not partition tokens directly over the current grammar-admissible support.Uniform additive green-list bias can also promote low-probability tokens, potentially shifting mass toward undesirable continuations.Source: §1 · p. 2
  • GTCW combines grammar-constrained candidate selection with green-temperature reweighting that emphasizes model-preferred green tokens at eligible high-entropy positions.Detection replays the grammar and keyed partitioning procedure without stored generation traces.Source: §1 · p. 2
  • 73.61% mean AUROC versus 67.83% for the strongest baseline, with mean Pass@1 of 59.18% versus 59.58% for unwatermarked generation.At TPR@5%FPR, GTCW reaches 41.86% versus 24.50% for the strongest baseline.Source: Figure 1 · p. 1, §1 · p. 2

2 RELATED WORK

Prior watermarking work includes general decoding-time schemes, code-specific carrier constraints, and grammar-constrained generation. GTCW differs by forming keyed partitions over tokens admissible in the current grammar state.

LLM watermarking

  • General decoding-time watermarks include keyed green-list biasing, fixed-partition unigram schemes, and unbiased or distortion-reducing sampling approaches.These methods apply to pretrained models, but do not account for grammar-restricted token choices in code.Source: §2 · p. 2

Code watermarking

  • Code-watermarking methods target structural constraints through high-entropy scoring, syntax-critical-token avoidance, type prediction, or AST-aware signals.GTCW instead uses current grammar-admissible tokens without auxiliary training.Source: §2 · pp. 2–3

Grammar-constrained generation

  • Grammar-constrained decoding restricts generation to tokens consistent with a target grammar, while GTCW forms its keyed green partition only over the permitted tokens.The related grammar systems discussed include SynCode and XGrammar.Source: §2 · p. 3

3 GRAMMAR-GUIDED CODE WATERMARKING WITH GREEN TEMPERATURE

GTCW addresses vocabulary-level partition mismatch and distortion from uniform green-token bias. It constructs keyed partitions over grammar-admissible tokens, reweights green candidates by relative preference, and replays generation states for detection.

3.1 CONDITIONAL GREEN-RATE DISTORTION

  • The effective green fraction can deviate from its nominal value because only tokens in the current grammar-admissible set can continue the program.Repeated local contexts can reproduce partitions, so deviations may recur rather than average out across decoding steps.Source: §3.1 · p. 3

3.2 GRAMMAR-GUIDED PARTITIONING

  • GTCW constructs green and red sets directly over the admissible support, using its signature and recent-token context to seed a ranking.The selected green-set size stays close to target γ, both sets remain nonempty, and no watermark is inserted when at most one token is admissible.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 GREEN TEMPERATURE

  • Green temperature adjusts green-token logits by relative model preference, sharpening preferred candidates instead of uniformly boosting every green token.Insertion is limited to eligible uncertain positions; the grammar constraint remains active when the carrier gate suppresses insertion during long comment or whitespace runs in C++ and Java.Source: §3.3 · pp. 4–5

3.4 DETECTION

  • Detection replays the original model, grammar, and key to reconstruct admissible sets and green partitions, then scores eligible green-token hits against their expected fractions.If parsing fails or no eligible position exists, replay stops at the admissible prefix or yields no finite detection score, respectively.Source: §3.4 · p. 5

4 EXPERIMENTS

Across five models and five benchmarks, GTCW improves watermark detection while keeping mean Pass@1 close to unwatermarked generation. Ablations attribute complementary roles to grammar guidance and green temperature, and transformation tests show that detection varies with the type of code change.

4.1 EXPERIMENTAL SETTINGS

  • The evaluation covers five instruction-tuned models, five benchmarks, and Python, C++, Java, and Go, measuring detection and execution-based functional correctness.Results are averaged over three random seeds with equal weight across model-benchmark configurations.Source: §4.1 · p. 5

4.2 MAIN RESULTS

  • 73.61% mean AUROC and 41.86% mean TPR@5%FPR exceed strongest baselines by 5.78 and 17.36 points, while mean Pass@1 is 59.18% versus Base’s 59.58%.Detection gains vary across models; GTCW has the highest TPR@5%FPR in every reported setting, while C++ shows the largest mean Pass@1 decrease among benchmarks.Source: Table 1 · p. 6, §4.2 · pp. 6–7

4.3 ABLATION STUDY

  • Grammar guidance primarily protects functional correctness, while green temperature supplies the larger gain in watermark detectability.Removing grammar guidance lowers mean Pass@1 from 58.16% to 55.93% while mean AUROC changes from 83.51% to 83.38%; replacing green temperature with additive bias lowers mean AUROC from 83.51% to 74.46%.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 GREEN TEMPERATURE SENSITIVITY

  • 84.25% mean AUROC at Tg = 0.3 falls to 79.58% at Tg = 0.7, while mean TPR@5%FPR drops from 57.7% to 45.5%.Across the displayed range, mean Pass@1 varies from 57.63% to 58.16%; the authors use Tg = 0.5 as a balanced operating point.Source: §4.4 · p. 8, Table 3 · p. 8

4.5 DETECTION AFTER CODE TRANSFORMATIONS

  • GTCW has the highest AUC and T@5 after comment removal and across all three control-flow settings, but formatting lowers its AUROC to 67.8%.Under formatting, it still has the highest T@5 at 27.5%; under identifier renaming, it leads T@5 at 25% and 50% strength and ranks second at 75% and 100%.Source: §4.5 · p. 8
  • Across 13 transformation settings, GTCW has the highest average TPR@5%FPR at 27.2%, while its 70.1% average AUROC ranks second to Unigram.The evaluation includes identifier renaming, formatting, comment removal, dead-code insertion, control-flow transformations, and LLM rewriting; clean outputs serve as a reference.Source: §4.5 · p. 8, Table 4 · p. 9
  • Under dead-code insertion, GTCW ranks second in both AUC and T@5 at all three strengths, while Unigram reaches 85.5% AUC and 42.7% T@5 at five insertions.The authors caution that inserted tokens can change the detector statistic, so this reflects behavior on the transformed sequence rather than preservation of original token-level evidence.Source: §4.5 · p. 9
  • LLM rewriting is the most disruptive tested transformation: GTCW’s AUROC falls to 51.4% and its T@5 to 6.8%.Rewriting changes token identities, local contexts, and grammar states throughout the completion, disrupting keyed partitions reconstructed during replay.Source: §4.5 · p. 9

5 CONCLUSION

GTCW combines grammar-admissible keyed partitioning with preference-aware green-temperature reweighting; results indicate stronger detection while largely preserving functional correctness.

5 CONCLUSION

  • Mean AUROC reaches 73.61% and TPR@5%FPR reaches 41.86%, improving over the strongest corresponding baselines by 5.78 and 17.36 percentage points, respectively.Mean Pass@1 is 59.18%, about 0.4 points below the unwatermarked reference; GTCW has the highest TPR@5%FPR in all 25 configurations.Source: §5 · p. 9
  • Grammar guidance improves mean Pass@1 by 2.23 points with little aggregate AUROC change, while green temperature improves mean AUROC by 9.05 points over additive bias with grammar retained.The conclusion attributes quality protection primarily to grammar guidance and the stronger detectable signal primarily to green temperature.Source: §5 · p. 9
  • Detection remains competitive under local code transformations, but LLM rewriting substantially weakens the signal; large-rewrite robustness and runtime overhead remain future-work targets.The implementation's current runtime overhead is also identified as a direction for improvement.Source: §5 · p. 9

A DETAILED ALGORITHM

  • A keyed hash of the grammar-support signature and recent token context seeds a deterministic ranking that assigns admissible tokens to green and red sets.The implementation uses HMAC-SHA256 for the keyed hash and SplitMix64 for token ranking; generation and replay share the procedure.Source: A DETAILED ALGORITHM · p. 12

A.1 GENERATION

  • Generation samples from grammar-admissible finite-support tokens, applying green-temperature reweighting only at eligible positions above the original-model entropy threshold.The entropy gate is evaluated before grammar masking and watermark reweighting, and the grammar constraint remains active even when watermark insertion is disabled.Source: §A.1 · p. 12

A.2 DETECTION BY REPLAY

  • Replay detection reconstructs grammar states, entropy gates, and keyed partitions from the completed code, accumulating evidence without stored generation traces.Replay stops on inadmissible tokens or parsing failure while retaining evidence already accumulated over the admissible prefix.Source: §A.2 · p. 13

B.1 IMPLEMENTATION AND DECODING SETTINGS

The appendix specifies shared decoding and baseline settings, evaluation data and metrics, tested models, and the grammars used to determine admissible continuations.

B.1 IMPLEMENTATION AND DECODING SETTINGS

  • All methods generate one sampled completion per task and seed under shared decoding settings, while GTCW uses γ = 0.5, τ = 0.9 nats, Tg = 0.5, and w = 1.Baselines use fixed settings across models and benchmarks; only GTCW applies grammar constraints in the main comparison, with ablations removing grammar guidance or replacing green temperature.Source: §B.1 · pp. 13–14

B.2 DATASETS, PROMPTS, AND METRICS

  • Evaluation uses HumanEval+ and MBPP+ plus multilingual HumanEvalPack subsets, with execution-based correctness tests and code-continuation prompts.The benchmark suites include 164 HumanEval+ tasks, 378 MBPP+ tasks, and 164 HumanEvalPack tasks for each of C++, Java, and Go.Source: §B.2 · p. 14
  • Detection performance is reported using AUROC and TPR@5%FPR, with negative examples consisting of unwatermarked completions generated using the same model, task, decoding configuration, and seed as corresponding watermarked outputs.This controls for differences in model family, prompting, and task content; reported detection results characterize separation from matched unwatermarked model outputs, not human-written code.Source: §B.2 · p. 14

B.3 MODELS

  • Five instruction-tuned checkpoints span several model families, including Qwen2.5-Coder at 3B and 7B, Qwen3.5 9B, Llama3.1 8B, and Gemma4 12B.Sensitivity, ablation, and transformation experiments use Qwen2.5-Coder (3B) as their fixed source model.Source: §B.3 · p. 15, §B.4 · p. 15

B.4 GRAMMARS

  • Grammar admissibility is computed with Python, C++, Java, and Go grammars, but C++ grammar admissibility does not guarantee compilation or functional correctness.Execution tests assess those properties separately; parser-side adapters provide context for benchmark prefixes that are not complete translation units.Source: §B.4 · p. 15

B.5 CODE TRANSFORMATION EVALUATION PROTOCOL

The transformation protocol tests watermark detection after independently modifying generated continuations, using matched clean completions as negatives and common evaluation procedures across methods.

B.5 CODE TRANSFORMATION EVALUATION PROTOCOL

  • Six transformation families with 13 settings are applied to Qwen2.5-Coder (3B) continuations across five benchmarks and three seeds, preserving each original code prefix.The transformations are applied independently to clean completions, and all methods use the same transformation implementations and strength settings.Source: §B.5 · p. 16
  • The protocol covers identifier renaming, formatting and comment removal, dead-code insertion, control-flow edits, and LLM rewriting.Renaming varies the share of eligible local variables; dead-code and control-flow settings request 1, 3, or 5 insertions or edits, while LLM rewriting requests behavior-preserving structural changes.Source: §B.5 · p. 16
  • Transformed outputs must pass Tree-sitter parsing or revert to the original completion, and functional tests are not rerun after transformation.Successful partial transformations are retained, and unchanged and failed transformations remain in the reported metrics.Source: §B.5 · p. 16
  • Detection scores are recomputed on transformed continuations, and results average five benchmarks and three seeds with equal weight across the 13 settings.The AVG excludes the clean reference; transformation metrics are measured relative to matched clean Base completions and may reflect transformation-induced score shifts.Source: §B.5 · pp. 16–17

B.6 EXPERIMENTAL ENVIRONMENT

The appendix records the experimental software and hardware environment and previews additional model, correctness, runtime, and qualitative analyses.

B.6 EXPERIMENTAL ENVIRONMENT

  • Experiments used a 24-core AMD Ryzen Threadripper 9960X, approximately 96 GiB NVIDIA RTX PRO 6000 Blackwell Max-Q GPUs, and a specified Linux, CUDA, and software stack.Each model process used a single GPU.Source: §B.6 · p. 17

C ADDITIONAL EXPERIMENTAL RESULTS

  • Additional analyses report model-benchmark breakdowns, including Qwen3.5 (9B) and Gemma4 (12B), and examine detection on functionally correct outputs and wall-clock runtime.The section states that the main detection trends remain consistent beyond models highlighted in the main text.Source: C ADDITIONAL EXPERIMENTAL RESULTS · p. 17
  • Qualitative code examples illustrate how watermark methods distribute signals and how GTCW combines grammar admissibility, carrier selection, and green-temperature reweighting.Source: C ADDITIONAL EXPERIMENTAL RESULTS · p. 17

C.1 FULL MAIN RESULTS

Additional results on Qwen3.5 (9B) and Gemma4 (12B) retain GTCW’s strongest low-FPR detection trend, while correctness effects vary by model. Qwen3.5 leads on both detection measures across all five benchmarks, whereas Gemma4’s AUROC advantage is less uniform.

C.1 FULL MAIN RESULTS

  • On Qwen3.5, GTCW achieves the highest AUC and T@5 on all five benchmarks, with T@5 gains of 47.8 versus 31.7 on C++ and 52.2 versus 22.2 on Go.The comparators are the strongest baselines, respectively.Source: §C.1 · p. 17
  • On Gemma4, GTCW achieves the highest T@5 on all five benchmarks, while its AUROC is competitive rather than uniformly best on C++, Java, and Go.Source: §C.1 · p. 17
  • Qwen3.5 shows some P@1 degradation on HumanEvalPack languages, whereas Gemma4 remains close to Base on C++ and Java without consistent losses elsewhere.Together, these results extend the main-text detectability-correctness pattern to two additional open-weight models.Source: §C.1 · p. 17

C.2 DETECTION ON FUNCTIONALLY CORRECT OUTPUTS

The section evaluates detection among functionally correct outputs and the joint rate of outputs that are both correct and detected. GTCW leads on both measures, but some outputs receive no finite score, and overall detection does not fully represent watermark retention in correct code.

C.2 DETECTION ON FUNCTIONALLY CORRECT OUTPUTS

  • 28.01% of functionally correct outputs are detected by GTCW, compared with 14.50% for SWEET; the joint correct-and-detected rate is 15.80% versus 8.16%.Q measures detection among correct outputs, while J measures outputs that are both correct and detected.Source: §C.2 · p. 18, Table 6 · p. 19
  • Overall TPR@5%FPR of 41.86% exceeds GTCW’s conditional detection rate, and the same pattern holds for all baselines.Detection on incorrect completions contributes little to provenance of usable programs.Source: §C.2 · p. 18
  • 74.07% of GTCW outputs have finite scores, similar to SWEET’s 74.42%.Unscorable outputs remain in the denominators of Q and J; no entropy-eligible position can prevent a finite score, and GTCW also relies on successful grammar replay.Source: §C.2 · p. 19, Table 6 · p. 19

C.3 GENERATION WALL-CLOCK TIME

The timing evaluation averages completion time, generated-token counts, and per-completion throughput across model-benchmark-seed settings. GTCW is substantially slower than Base and other watermarking methods in the reported setup, which the authors attribute to unoptimized runtime integration.

C.3 GENERATION WALL-CLOCK TIME

  • Timing includes model generation and online watermarking, but excludes model loading, offline grammar precompilation, replay detection, and functional evaluation.Averages are formed by first averaging within each model-benchmark-seed setting, then averaging settings equally; token counts use raw generated tokens before cleanup.Source: §C.3 · p. 19
  • 11.50 seconds per completion for GTCW compares with 4.28 seconds for Base and 4.40–4.62 seconds for other watermarking methods.The reported mean throughput is 17.24 tokens per second for GTCW, compared with 63.07–64.93 for the other methods.Source: §C.3 · p. 19, Table 7 · p. 19
  • GTCW’s current implementation introduces substantial online overhead and is not yet optimized for runtime.The authors identify better integration of model decoding and grammar-state processing as a possible way to reduce the cost.Source: §C.3 · p. 19

C.4 GENERATED CODE EXAMPLES

The examples compare where watermark evidence is placed in functionally correct completions. GTCW scores only positions meeting grammar-admissibility, carrier, and entropy conditions, strengthening evidence at selected positions rather than watermarking more positions.

C.4 GENERATED CODE EXAMPLES

  • Unigram, KGW, and STA-1 watermark broadly across identifiers, syntax, and comments, while STONE’s scored signal in this example appears substantially in comments.Source: §C.4 · pp. 19–20
  • SWEET’s concise completion has only three scored positions, all green, whereas GTCW illustrates strong signal from selectively chosen positions.All displayed completions pass the augmented tests and exceed their corresponding TPR@5%FPR detection thresholds.Source: §C.4 · p. 20, Figure 3 · p. 20
  • GTCW scores ten positions in the displayed completion, nine green, while its remaining scored red token shows that it biases rather than forces green choices.The carrier condition combines grammar admissibility with carrier and entropy gates.Source: §C.4 · p. 20, Figure 3 · p. 20
Loading 2610.05323v1…