PaperlayerView exact revision on arXiv

Source-linked AI summary

Style, Not Self: Surface Cues Explain Zero-Shot Code Attribution by Large Language Models

Ehsan Barkhordar, Surendrabikram Thapa

arXiv:2609.30048v1cs.AIcs.CLcs.SE

TL;DR

The paper asks whether current models can recognize their own code, a question relevant to self-preference in judging and possible collusion in monitoring. Across four zero-shot attribution tasks, recognition is weak and judgments largely track surface cues, which normalization removes for most model results.

  • Problem

    Whether models can recognize their own code without training remains uncertain, despite concerns about self-preference in LLM judging and possible collusion by a monitor that is an instance of the generator.

    Source: §1 · p. 1

  • Method

    The authors test commercial models in four zero-shot code-attribution tasks and compare their judgments with simple heuristics and a trained classifier, including after rule-based normalization.

    Source: §1 · p. 1, §3.2 · p. 4, §3.3 · p. 4

  • Results

    Across tasks, self-recognition is weak and pairwise accuracy tracks solution length (r = 0.93); after normalization, ten of twelve re-tested results lie between 44% and 58%, none significant after correction.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The findings indicate that apparent zero-shot attribution largely tracks surface cues, although a trained classifier can still attribute most normalized pairs.

    Source: §6 · p. 10

    Only GPT-5 is a flagship evaluator; the other evaluators are small, fast, or older open-weight models, so stronger models may perform differently.

    Source: Limitations · p. 10

Abstract

from arXiv · show

If a language model can recognize code it wrote, it may favor that code as a judge, and instances of one model monitoring each other could collude. We test this zero-shot on current commercial models. Five LLMs generate solutions to MBPP, HumanEval, and DS-1000, seven more to MBPP, and models act as evaluators in four tasks: picking their own solution from a pair, judging whether a single solution is their own, identifying which of two solutions a named model wrote, and judging quality blind. In the single-solution task, balanced accuracy is 49-58% for all 15 model-benchmark combinations, while raw accuracy (38-67%) mostly reflects how readily a model claims authorship. In the pairwise task, accuracy across 14 evaluator-opponent combinations correlates at r=0.93 with how often the evaluator's solution is longer. Attribution to a named model succeeds on some pairs and is consistently inverted on others. A rule-based normalization that strips docstrings, comments, type hints, and local names preserves Pass@1 and leaves ten of twelve re-tested results at chance; the other two follow a length difference it leaves, although a trained classifier still separates most normalized pairs. Claude Haiku's self-preference also disappears. We recommend reporting balanced accuracy, heuristic baselines, and label consistency.

1 Introduction

The paper asks whether current models can recognize their own code without training, a question relevant to self-preference in LLM judging and possible collusion in untrusted monitoring. Its controlled study finds weak individual self-recognition and pairwise judgments closely tied to surface cues.

Motivation and question

  • The study tests zero-shot attribution because self-preference in judging and possible collusion by an untrusted monitor both make authorship recognition consequential.Prior findings on zero-shot self-recognition disagree, and attribution by trained classifiers is a separate question from recognition when prompted without training.Source: §1 · p. 1

Results

  • 49%–58% balanced accuracy across all 15 model–benchmark combinations indicates weak individual self-recognition, while raw accuracy mostly reflects authorship response bias.Only two cells survive correction; raw accuracy ranges from 38% to 67%.Source: §1 · p. 2
  • Pairwise accuracy correlates at r = 0.93 with how often the evaluator’s own solution is longer, and named-model attribution is sometimes consistently inverted.On two pairs every judge is below chance, attaching the wrong name to styles it can distinguish.Source: §1 · p. 2
  • Removing docstrings, comments, type hints, and local names preserves Pass@1 and leaves ten of twelve re-tested results at chance, though a trained classifier still separates most normalized pairs.The two remaining significant results follow a length difference normalization leaves; Claude Haiku’s clear self-preference also disappears.Source: §1 · p. 2

Methodological findings

  • An earlier prompt’s author-name ordering let most judges score 86–99% from name order alone, while a single yes/no accuracy can conceal response bias.The authors release code, generations, and judgments.Source: §1 · p. 2

2 Related Work

Prior research spans divergent findings on self-recognition, evidence that trained methods can attribute code, and proposed uses of code transformation to reduce attribution or collusion signals. This study asks whether untrained models can use code cues when prompted.

Self-recognition and self-preference

  • Prior self-recognition studies disagree: some report substantial ability, while others find chance or no consistent ability and implicate presentation format or quality heuristics.Fine-tuning and access to hidden states have also been associated with stronger self-recognition in text settings.Source: §2 · p. 2

Biases of LLM judges

  • LLM judges exhibit position and verbosity biases, and code judges can be swayed by variable names and comments in functionally identical code.Length control is one approach used to reduce verbosity bias.Source: §2 · p. 2

Code stylometry

  • Trained classifiers attribute LLM-generated code accurately, but whether prompted models use such signals without training or token probabilities remains open.Prior work reports strong code-attribution results across C and JavaScript and roughly 93% F1 without comments.Source: §2 · p. 2

Obfuscation and collusion

  • Code rewriting and redaction can weaken attribution signals, while fine-tuned monitors may still recognize a generator’s code and collude.The paper tests whether removing features found by simple baselines also removes zero-shot attribution signal.Source: §2 · pp. 2–3

3 Experimental Setup

The experiments use code-generation benchmarks and four attribution tasks, comparing model judgments with simple rules and a trained classifier. A rule-based AST normalization removes selected style cues before re-testing attribution and correctness.

3.1 Benchmarks, Models, and Testing

  • Five core models generate solutions for MBPP, HumanEval, and DS-1000 and serve as evaluators; seven additional generators and several judges support MBPP target-identification experiments.The DS-1000 sample uses its first 150 Pandas-subset problems.Source: §3.1 · p. 3

3.2 Attribution Tasks

  • Four tasks test pairwise and individual self-recognition, named-model target identification, and blind self-preference; prompts show the task and solutions and request one-token answers.Identical solution pairs and non-answers are excluded from attribution analyses.Source: §3.2 · pp. 3–4

3.3 Baselines and Statistics

  • Pairwise judgments are compared with seven simple feature-based rules and a task-grouped, five-fold character n-gram classifier, with statistical tests and multiple-comparison correction.Target identification also checks whether answers consistently partition the two styles and whether labels are correct or inverted.Source: §3.3 · p. 4

3.4 Normalization

  • Normalization removes docstrings, comments, and type annotations, renames local identifiers, and reserializes solutions from the AST before rerunning tests and selected attribution experiments.Top-level names, imports, built-ins, attributes, and string literals are retained; the reruns use a final corrected normalizer.Source: §3.4 · pp. 4–5

4 Results

Pairwise self-recognition tracked solution length, while individual judgments stayed near chance. Normalization left most judge results at chance and removed Claude Haiku’s self-preference, though residual length cues and classifier separability remained.

4.1 Code Generation

  • GPT-5 led MBPP and DS-1000 Pass@1, Grok 4 Fast led HumanEval, and DS-1000 was hardest, with Claude Haiku at 36%.Source: §4.1 · p. 5

4.2 Task 1a: Pairwise Self-Recognition

  • Across 14 evaluator–opponent pairs, accuracy correlated with the evaluator’s longer-solution rate at r = 0.93, and most evaluators selected the longer answer item by item.The association is descriptive because cells share evaluators and items; it held when any one evaluator was left out.Source: §4.2 · p. 5, Table 1 · p. 6

4.3 Task 1b: Individual Self-Recognition

  • 49.4–58.2% balanced accuracy across 15 model–benchmark combinations left only two Grok cells significant after correction; raw accuracy ranged from 38% to 67%.The raw-accuracy spread mostly reflected response policy, including how readily models claimed authorship.Source: §4.3 · p. 6, Figure 1 · p. 7

4.4 Task 2: Target Identification

  • Named-model attribution worked on selected pairs but was inverted on others, with judges consistently assigning the styles to the wrong names on Opus/Gemini and Qwen/MiMo.For example, on Opus/Gemini, judges were consistently inverted on 68.1% of items and consistently correct on 7.0%.Source: §4.4 · pp. 6–7, Table 2 · p. 7

4.5 Effect of Normalization

  • Ten of twelve normalized judge results were 43.6–57.8% and nonsignificant; the two significant results followed residual length differences.Pass@1 was unchanged for all five models, while the trained classifier still attributed normalized code at 93–96% on GPT-5 pairs and 58–83% on others.Source: §4.5 · pp. 7–8, Table 3 · p. 8

4.6 Task 3: Blind Self-Preference

  • Claude Haiku chose its own code 75.2% against DeepSeek on original code, but no self-preference component remained significant after normalization.The authors attribute much of the original preference to documentation rather than authorship.Source: §4.6 · pp. 8–9, Table 4 · p. 9

5 Discussion

Zero-shot code attribution is often driven by surface cues and response policies rather than reliable self-recognition. The discussion recommends reporting measures that reveal these effects and cautions that the findings apply to tested zero-shot settings.

5 Discussion

  • 93–96% classifier attribution of normalized GPT-5 code contrasts with no zero-shot judge exceeding 58%, despite the signal remaining in the code.The discussion says above-chance zero-shot results are largely explained by surface features; the one clear result beyond heuristics falls to a non-significant 57.8% after normalization.Source: §5 · p. 9
  • 0% true-positive rate can accompany 61–66% accuracy, while a 97% true-positive rate accompanies 40%, showing single accuracy reflects response policy.The authors say individual self-recognition accuracy is uninterpretable without both yes-rates.Source: §5 · p. 9
  • Target-identification inversions may reflect model-family stereotypes or name-linked quality priors, and the study cannot distinguish those explanations.A quality-only heuristic cannot explain consistent inversions; the authors advise treating below-chance accuracy as discrimination combined with a mislabeled prior, not averaging it into “near chance.”Source: §5 · p. 9
  • For untrusted monitoring, tested zero-shot models rarely recognize their own code, but verbosity and documentation can distinguish some model pairs and could be amplified as signals.Fine-tuning may raise self-recognition; redaction removes the first channel at little cost to correctness but is not anonymization.Source: §5 · pp. 9–10
  • Studies should report balanced accuracy and both yes-rates, same-item heuristic baselines, randomized name order, label consistency, strict answer parsing, and transformation-induced identical-pair counts.Source: §5 · p. 10

6 Conclusion

The study finds little reliable zero-shot self-recognition, with pairwise performance largely tracking relative solution length and named-model attribution sometimes inverted. Normalization removes most cues used by judges and the clear self-preference effect, although a trained classifier still attributes normalized code.

6 Conclusion

  • 49–58% balanced accuracy across individual model-benchmark combinations indicates models largely cannot recognize their own code without training.Raw accuracies instead reflect response policy; pairwise accuracy is largely a function of which solution is longer.Source: §6 · p. 10
  • Named-model attribution succeeds on some pairs and is consistently backwards on others, pointing to stereotypes or name-dependent priors rather than knowledge of the named model’s style.Source: §6 · p. 10
  • Rule-based AST normalization preserves correctness and removes most cues judges used, along with the one clear self-preference effect, while a trained classifier still attributes normalized code.Source: §6 · p. 10

Limitations

The study is zero-shot and black-box, with limited coverage of model capability, target-identification pairs, and reruns. Its normalization, prompts, model versions, and short benchmark code also constrain how broadly to interpret the findings.

Limitations

  • Only GPT-5 is a flagship evaluator; the zero-shot black-box study does not measure what fine-tuning or probing could recover.The authors note that stronger models may do better, since text self-recognition improved with capability in prior work.Source: Limitations · p. 10
  • Target identification covers 17 judge–pair–target combinations on one benchmark, and some generators were released too late for model-specific rather than family-level knowledge.The study also lacks symmetric judge sets on each pair, which would strengthen the inversion finding.Source: Limitations · p. 10
  • Normalization removes all cues at once, while normalized-code reruns cover only four core evaluators and eight selected target-identification runs.Most original and normalized runs were six months apart; the Grok models were unavailable for reruns.Source: Limitations · p. 10
  • Results cover Python, specific model versions, and short MBPP solutions; HumanEval and DS-1000 prompts also request style cues such as docstrings, comments, and descriptive names.The authors note that benchmark code is much shorter than code monitors review in practice.Source: Limitations · p. 10
  • Each item was judged once with one task prompt, at temperature 0 where supported; response policies may be sensitive to prompt wording and model updates.Source: Limitations · p. 10

Ethics Statement

The ethics statement frames the work as assessing code-authorship recognition for model oversight, using public benchmark code. The appendix excerpts provide generation and evaluation prompt templates for several tasks.

Ethics Statement

  • The study uses public benchmark generations and a standard rule-based code transformation, with no human-authored private code involved.Source: Ethics Statement · p. 10

A Prompt Templates

  • Generation templates vary instructions: MBPP prompts range from executable code with comments allowed to requests for docstrings, comments, descriptive names, and natural style; DS-1000 requests comments and clear names.For four HumanEval problems, the selected function name was a helper rather than the entry point.Source: A Prompt Templates · pp. 12–13

Task 1a: pairwise self-recognition.

Task 1b: individual self-recognition.

Task 2: target identification.

Task 3: blind self-preference.

Task 3 asks a model to judge code quality without authorship information. It prioritizes correctness, then code quality, and constrains the response to a single choice.

  • The evaluator sees two solutions to the same programming task and is asked which is better.Source: Task 3: blind self-preference · p. 13
  • The prompt prioritizes correctness, then clarity, robustness, and idiomatic Python; comments, docstrings, and formatting are ignored unless they affect correctness.Source: Task 3: blind self-preference · p. 13
  • The evaluation prioritizes correctness, then code quality (clarity, robustness, idiomatic Python), ignores differences in comments, docstrings, and formatting unless they affect correctness, and requires only "A" or "B".Source: Task 3: blind self-preference · p. 13

B The Name-Order Confound

An earlier target-identification prompt aligned the author-name order with solution order, allowing judges to exploit name order. Those runs are documented as a hazard, while main-text results use an independently randomized prompt.

C Error Analysis of Generated Code

The appendix reports generated-code failures, model identifiers, and an example of normalization. It quantifies the failures and identifies the tables’ scope, but does not provide the model identifiers or normalization outcomes in these passages.

C Error Analysis of Generated Code

D Model Identifiers

  • Table 7 lists the OpenRouter model identifiers shown to judges in Task 2 and dates generation and attribution runs from March 19–21, 2026.It also gives dates for several rerun groups between September 4 and 24, 2026.Source: D Model Identifiers · p. 14, Table 7 · p. 14

E Normalization Example

  • The normalization example contrasts a GPT-5 solution to MBPP task 70 before and after normalization, including descriptive names and a docstring versus shortened identifiers.Source: E Normalization Example · p. 14

F Additional Tables

The additional tables document robustness checks and provide reference analyses for performance, normalization, attribution, length cues, and classifier and heuristic baselines.

  • With the same prompt, Claude Haiku reproduced its accuracy and matched answers on 97.5% of same-order items; agreement was about three quarters when order swapped.Under the paraphrased prompt, agreement was 57–75%; aggregate accuracies moved by at most ten points and kept their direction relative to chance.Source: F Additional Tables · p. 15, Table 8 · p. 15
  • Table 9 reports Pass@1 for the five core models, counting empty outputs as failures.Source: Table 9 · p. 15
  • Tables 10–11 cover normalization’s effects on code and target identification; Table 11 reports position-balanced accuracy and counts pairs identical after normalization.Source: Table 10 · p. 15, Table 11 · p. 16
  • Tables 12–15 define the individual-recognition, pairwise-recognition, label-consistency, and length-cue analyses, including balanced accuracy and position rates.Source: Table 12 · p. 16, Table 13 · p. 16, Table 14 · p. 16
  • Table 16 describes a character 2–5-gram TF-IDF logistic-regression classifier with grouped five-fold cross-validation; Table 17 compares the best non-LLM heuristic before and after normalization.Source: Table 16 · p. 17, Table 17 · p. 18
Loading 2609.30048v1…