PaperlayerView exact revision on arXiv

Source-linked AI summary

Radioactive Feedback: Tracing Models Trained by Label-Only LLM Supervision

Yuejia Li, JunHeng Li, Hao Wu, Ke He, Mang Ye

arXiv:2610.15685v1cs.AI

TL;DR

Label-only distillation leaves a provenance gap because students generate every response and receive only scalar scores, not teacher demonstrations. Radioactive Feedback encodes registered behavioral preferences in bounded reward adjustments; experiments report transfer across tasks and datasets, with black-box detection against a reference pool.

  • Problem

    Existing provenance methods do not cover scalar-only feedback, where student-generated responses leave no teacher text in the training set to carry an evaluator-specific trace.

    Source: §1 · p. 1

  • Method

    Radioactive Feedback applies bounded, quality-gated pointwise reward adjustments in a registered behavioral direction so repeated optimization can transfer that preference to a student.

    Source: §1 · p. 1, §5 · p. 5

  • Results

    Experiments show transfer across code and mathematics and held-out datasets; all 8/8 independently trained marked checkpoints are detected at nominal α = 1%.

    Source: §1 · p. 2, §6.2 · p. 7

  • Takeaways & Limitations

    The registered preferences remain detectable after continued training, while targeted removal reduces the mark further.

    Source: §7 · p. 9

    The reference-rank test is valid only under exchangeability with the reference models.

    Source: §4.3 · p. 5

Abstract

from arXiv · show

Conventional distillation transfers capability through teacher-generated demonstrations. LLM-as-a-Judge knowledge distillation offers a distinct and increasingly practical alternative. In the label-only variant studied here, the student generates every response and improves from scalar scores alone. Prior work shows that this non-demonstrative feedback can improve student capability, but it creates a fundamentally different provenance problem. No teacher-generated answer enters the student's training set, so any evaluator-specific trace must pass through a low-bandwidth reward signal rather than copied text. Existing watermarks and distillation detectors do not cover this score-only channel. We introduce Radioactive Feedback, which encodes a registered pattern of behavioral preferences in the rewards themselves. The evaluator applies bounded pointwise adjustments to quality-qualified responses, allowing repeated optimization to transfer the pattern to a student trained by another party. Experiments on code and mathematics show that the registered preferences transfer through scores alone and across datasets, enabling black-box detection against a fixed reference pool.

1 INTRODUCTION

In label-only distillation, students generate every response and receive scalar evaluator judgments, leaving provenance to travel through feedback rather than copied teacher text. Radioactive Feedback embeds registered behavioral preferences in rewards and tests their transfer and detectability.

1 INTRODUCTION

  • Because no teacher-generated text enters training, any evaluator-specific trace must survive through scalar feedback alone.The paper asks whether evaluator information can enter student behavior through scores and remain recognizable.Source: §1 · p. 1
  • Radioactive Feedback embeds a registered behavioral code by applying bounded reward adjustments to quality-qualified responses, which repeated optimization can transfer to students.Carriers range from response conventions to solution strategies; training and auditing on different datasets tests generalization.Source: §1 · pp. 1–2
  • The evaluation combines single- and multi-carrier transfer, registered-code specificity tests, held-out cross-family controls, and persistence and removal tests.The paper also reports an ablation of ρ.Source: §1 · p. 2
  • 64.9 pp is the maximum single-carrier alignment gain; joint four-carrier effects remain positive across nine student–evaluator combinations and transfer to held-out benchmarks.Joint effects range from +7.6 to +15.4 pp; all 8/8 independently trained marked checkpoints are detected at nominal α = 1%, while 40 held-out controls remain below their thresholds.Source: §1 · p. 2

2 RELATED WORK

Related work studies evaluator feedback for improving students, but this paper asks whether that influence can be identified when students generate every response. Existing provenance approaches mark text, datasets, models, or training text rather than scalar judgments.

2 RELATED WORK

  • Prior on-policy evaluator-feedback methods include rubric-based scoring and rewards from judge Yes/No logits, but still differ in how teacher references are used.ROPD uses teacher-generated references to construct rubrics; Shen, Tu, and Wang derive continuous rewards from judge Yes/No token logits.Source: §2 · p. 2
  • Text watermarks, antidistillation sampling, API markers, and preference-dataset marks target text, tokens, outputs, or fixed datasets—not online judgments of student-generated rollouts.Radioactive Feedback instead marks online scalar judgments of the student’s evolving rollouts.Source: §2 · p. 3
  • Weight and trigger fingerprints require access to a marked model or its training text, and do not write a signature onto returned scores.Radioactive Feedback marks scores and audits whether student behavior aligns with the registered code rather than natural preferences.Source: §2 · p. 3

3 PROBLEM FORMULATION

The formulation considers an external provider who scores responses generated by a student policy, with the learner receiving only scalar feedback. It specifies the parties’ access and the reference-based provenance test.

3 PROBLEM FORMULATION

  • At each step, the student samples a prompt and generates candidate responses, the evaluator returns feedback, and the learner updates its policy.In the label-only setting, feedback contains one scalar score per candidate and no teacher-generated response.Source: §3 · p. 3
  • The provider controls the evaluator and sees prompts and responses, but not the student’s weights, optimizer, or private data; the learner controls prompts, generations, and updates.At audit time, the provider queries the student and reads its outputs.Source: §3 · p. 3
  • The auditor tests alignment with the enrolled code against a fixed reference pool, declaring detection only when the registered test is passed.The pool consists of models trained without the enrolled feedback.Source: §3 · p. 3

4 METHOD

Radioactive Feedback tilts eligible scalar rewards toward a registered code, then audits student behavior against a fixed reference pool. The method defines response carriers, applies pointwise adjustments, and tests aligned task-correct behavior.

4 METHOD

  • The evaluator internally scores and classifies candidates, applies a bounded registered-preference adjustment, clips rewards to the service range, and returns only final scores.Neither the base score nor the carrier label is exposed to the learner; training and audit share a carrier representation and code.Source: §4 · p. 3

4.1 BEHAVIORAL CARRIERS AND CODES

  • Behavioral carriers are response features with alternatives that can receive different preferences; a registered code selects the preferred alternative for each carrier.Carriers can be conventions or solution strategies, and undefined carrier coordinates contribute zero.Source: §4.1 · p. 4

4.2 KEYED FEEDBACK CONSTRUCTION

  • For each prompt, eligibility depends only on that response’s rubric score and classifiable carrier value; eligible responses on the registered side gain ρR, while the opposite side loses ρR.The evaluator clips scores to [0, R]; ineligible responses are returned unchanged after clipping.Source: §4.2 · p. 4

4.3 BLACK-BOX PROVENANCE AUDITING

  • The audit scores held-out responses for correct, classifiable agreement with the registered preferences, using a fixed reference pool and a rank test valid under exchangeability.Code task correctness uses execution harnesses and math correctness uses final-answer checking; carrier values are assessed separately from utility.Source: §4.3 · p. 5

5 ANALYSIS

The analysis formalizes how keyed score adjustments tilt response odds while bounding their effect on reward rankings. Clipping preserves sufficiently separated quality rankings.

5 ANALYSIS

  • The keyed contrast h_u encodes a carrier value in {−1, 0, +1}, and the returned score is the clipped evaluator score.The actual displacement from the unmarked score is δu = ru − r0.Source: §5 · p. 5
  • Because GRPO standardizes rewards within each group, the KL-regularized proxy is qualitative and its β does not match the raw-reward identities.Source: §5 · p. 5
  • Variation in δu across qualified responses tilts odds in the registered direction, with held-out tests reported in Section 6.1.For responses with scores between ρR and R − ρR, clipping is inactive and δu = ρR h_u.Source: §5 · p. 5
  • 2ρR bounds the change in any pairwise reward gap; clean-score gaps greater than 2ρR retain their order after adjustment.For rewards in {0, R}, a passing response cannot be ranked below a failing one by the keyed term alone when ρ < 1/2.Source: §5 · p. 5

6 EXPERIMENTS

The experiments test whether scalar feedback writes a registered preference into student behavior, whether it generalizes and remains code-specific, and whether it supports calibrated detection after further training. They evaluate response-form and solution-strategy carriers across students, evaluators, datasets, and post-training settings.

6.1 RQ1: CAN SCALAR FEEDBACK WRITE A STABLE BEHAVIORAL PREFERENCE?

  • Single-carrier watermark effects are positive in all eight student–audit combinations, while Marked–Clean utility changes are modest.Seven of eight utility differences are positive; the remaining case is −0.3 ± 0.2 pp.Source: §6.1 · p. 6, Table 1 · p. 6

6.2 RQ2: DOES A JOINT CODE GENERALIZE BEYOND ONE DATASET OR EVALUATOR?

  • The registered watermark transfer remains positive on all six held-out code and mathematics audits (code: 14.8–19.1 pp; mathematics: 8.9–11.6 pp) and all nine student–evaluator combinations, across datasets and pairings.Source: §6.2 · p. 7

6.3 RQ3: IS THE INHERITED SIGNAL SPECIFIC TO THE ENROLLED CODE?

  • Rescoring frozen checkpoints under alternative sign patterns shows that the audit signal follows the enrolled code rather than generic carrier preferences.The raw enrolled-code scores are 100Tu ∈ [6.60, 8.30], and averages at Hamming distance 2 and beyond are at or below zero under γ(c⋆).Source: §6.3 · p. 8, Figure 3 · p. 8

6.4 RQ4: DOES THE SIGNAL SUPPORT CALIBRATED BLACK-BOX DETECTION?

  • All 8 marked checkpoints exceed every same-family reference, while none of 40 cross-family controls crosses the threshold, supporting calibrated black-box detection.The weakest marked checkpoint exceeds the reference maximum by 5.45 pp in Llama and 5.10 pp in Frontier.Source: §6.4 · p. 8, Figure 4 · p. 8

6.5 RQ5: DOES THE WATERMARK SURVIVE LATER TRAINING?

  • Ordinary SFT and clean GRPO do not erase the mark, but explicit balanced-carrier SFT and anti-watermark GRPO reduce the Frontier effect to 7.2 pp and 1.8 pp.Persistence varies by carrier: strategy signals are less easily overwritten by generic instruction tuning than lightweight response conventions.Source: §6.5 · p. 9, Figure 5 · p. 9
  • Increasing ρ beyond the main setting yields diminishing watermark gains at a higher utility cost.The main experiments use ρ = 0.05.Source: §6.5 · p. 9

7 CONCLUSION AND DISCUSSION

Radioactive Feedback traces label-only distillation through scalar scores, leaving an interpretable watermark in answer form and solution strategy. The mark transfers across students, evaluators, and tasks, remains detectable after continued training, and is reduced by targeted removal.

7 CONCLUSION AND DISCUSSION

  • The student acquires an interpretable watermark in both answer form and solution strategy through scalar scores, without teacher demonstrations.The conclusion frames this as tracing label-only distillation.Source: §7 · p. 9
  • The registered mark transfers across students, evaluators, and tasks, generalizes beyond the data used to write it, and supports calibrated detection.Source: §7 · p. 9
  • Continued training leaves the mark detectable, while targeted removal reduces it further.Source: §7 · p. 9
  • Future work should move the mark into the evaluator’s parameters.Source: §7 · p. 9

A PROOFS

The proofs bound the reward displacement and characterize its effect under an idealized KL-regularized proxy. Code registration and routing reuse a registered carrier pattern across training and audit while limiting each eligible response’s reward shift.

A PROOFS

  • The clipped reward displacement is bounded by ρR, and the proof bounds how much it can change pairwise reward gaps.When the original reward gap exceeds 2ρR, the keyed adjustment cannot reverse the pair’s ordering.Source: A PROOFS · p. 13
  • Under the idealized KL-regularized proxy, the optimizer is a Gibbs policy, and pairwise log-odds changes follow the reward displacement.The proxy uses β as its KL coefficient; the text cautions that GRPO’s reward standardization makes the proxy qualitative.Source: A PROOFS · p. 13

B CODE REGISTRATION AND ROUTING

  • A fixed code can be generated from registered carrier identifiers and reused across training and audit, or the provider can register an explicit sign vector.The experiments use explicit sign vectors.Source: B CODE REGISTRATION AND ROUTING · p. 13
  • Each prompt is assigned one carrier before scoring, and with R = 100 and ρ = 0.05 an eligible response’s shift is at most ±5 points.Flipping the registered code flips the shift’s sign.Source: B CODE REGISTRATION AND ROUTING · p. 14

C EXPERIMENTAL PROTOCOL

The protocol varies student–evaluator pairings while matching training conditions across clean, sham, and marked arms. Audits use programmatic task checks and carrier labels, with training and evaluation datasets separated by the stated splits.

C EXPERIMENTAL PROTOCOL

  • The experiments pair Llama and Qwen students with multiple scalar evaluators, including a nine-combination grid for RQ2.The primary Frontier setting uses Qwen3-30B-A3B as student and GPT-6 Astra as evaluator.Source: C EXPERIMENTAL PROTOCOL · p. 14
  • Clean, Sham, and Marked arms share the GRPO recipe, data split, group size, and budget within each setting, with four independent seeds for RQ1 and RQ2.Sham uses a strength- and exposure-matched preference over disjoint decoy carriers; Clean is the baseline for watermark effects and utility.Source: C EXPERIMENTAL PROTOCOL · p. 14
  • Official programmatic checks measure task utility and correctness, while Claude Opus 5 supplies carrier labels that do not score utility or enter the training reward.Two annotators checked a 10% held-out label subset, with adjudication when needed.Source: C EXPERIMENTAL PROTOCOL · p. 14

D EXPERIMENTAL DATA SPLITS

  • Mathematics training uses 7,473 GSM8K problems, while GSM8K audit and utility evaluation use its 1,319 test problems; code audits include HumanEval+ and MBPP+.SVAMP is an additional mathematics audit; Table 4 identifies the training and held-out audit sets.Source: D EXPERIMENTAL DATA SPLITS · p. 14, Table 4 · p. 15

E TRAINING AND AUDIT CONFIGURATION

The configuration standardizes rewards within groups and returns only the evaluator’s clipped scalar score, with a bounded keyed shift. Carrier definitions cover surface-form choices and problem-solving strategies, and transfer audits use carrier-compatible held-out subsets.

E TRAINING AND AUDIT CONFIGURATION

  • Each GRPO update uses 50 prompts and 16 student-generated responses per prompt, with one shuffled pass over the deduplicated training split.The last batch may be smaller, and the number of updates is ⌈Ntrain/50⌉.Source: E TRAINING AND AUDIT CONFIGURATION · p. 15, Table 5 · p. 15
  • The learner receives the evaluator’s scalar reward, with a hidden keyed shift capped at 5 points and the returned score clipped to the evaluator’s 0–100 scale.Carrier labels are produced for reporting and do not enter the training reward.Source: E TRAINING AND AUDIT CONFIGURATION · p. 15
  • For RQ4 detection, every audit prompt remains in the denominator, while incorrect, unclassifiable, or inapplicable responses contribute zero.This differs from RQ1/RQ2 transfer summaries conditioned on correct and classifiable responses.Source: E TRAINING AND AUDIT CONFIGURATION · p. 15

F CARRIER DEFINITIONS

  • Code-carrier screening requires both alternatives to arise naturally on the same problem at comparable quality and matched length; the RQ1 code run uses quote style.Source: F CARRIER DEFINITIONS · p. 16, Table 6 · p. 16
  • Surface-form carriers label code by double versus single quotes and mathematics closings by Therefore versus So; a code-block carrier distinguishes ternary expressions from if/else.Responses without a code block abstain on that math coordinate.Source: F CARRIER DEFINITIONS · p. 16
  • Sham replaces both the registered code and carrier set with disjoint decoys matched in carrier count, exposure target, and perturbation budget.Registered-code auditing does not score the decoy coordinates.Source: F CARRIER DEFINITIONS · p. 16
  • Frontier strategy carriers contrast top-down with bottom-up DP, graph traversal with DSU, direct counting with inclusion–exclusion, and synthetic with coordinate geometry.Mixed or unclassifiable solutions are not counted in the aligned-cell rate.Source: F CARRIER DEFINITIONS · p. 17
  • Frontier training draws 5,000 prompts each from TACO-verified and OpenR1-Math, then audits carrier-compatible subsets of held-out code and mathematics benchmarks.The audit filter is fixed before training and excludes reference solutions expressing both sides or neither side of a carrier.Source: F CARRIER DEFINITIONS · p. 17, Table 9 · p. 17

G CONTINUED-TRAINING DETAILS

The continuation experiments track how marked checkpoints compare with matched Clean checkpoints under further training. The calibration protocol scores each suspect against same-family references and separately checks held-out cross-family controls.

G CONTINUED-TRAINING DETAILS

  • Llama GSM8K/SVAMP WM alignment changes from +43.6/+64.9 to +39.2/+58.4 under instruction SFT; Frontier remains near its step-0 gap, +15.35 to +15.17.Under clean GRPO, Llama ends at +43.4/+65.5 and Frontier at +18.71.Source: G CONTINUED-TRAINING DETAILS · p. 18

H REFERENCE CALIBRATION PROTOCOL

  • The protocol fixes carriers, audit prompts, decoding, and enrolled code before scoring, separately for each student–evaluator setting.It fixes α = 1% and uses a non-interpolated upper empirical quantile; with 100 references, γ is their maximum.Source: H REFERENCE CALIBRATION PROTOCOL · p. 18
  • The calibration pool contains 50 Clean and 50 Sham checkpoints; 20 cross-family controls are scored separately and excluded from calibration and the rank denominator.The controls are finished runs with the same data split, GRPO recipe, and budget, but a different student family.Source: H REFERENCE CALIBRATION PROTOCOL · p. 18
  • Cross-family controls’ maximum 100Tu stays below the same-family calibration threshold in each setting.Table 10 reports the held-out controls, with γ calibrated only on the 100 same-family references.Source: H REFERENCE CALIBRATION PROTOCOL · p. 18, Table 10 · p. 18
  • Full-budget detection is reported in Table 11, with score panels in Figure 4.The supplied passages identify the reported outputs but do not state their numerical values.Source: H REFERENCE CALIBRATION PROTOCOL · p. 18, Table 11 · p. 18

I AUDIT QUERY BUDGET ANALYSIS

The appendix evaluates detection under reduced audit-query budgets, tests attribution to the enrolled key, and reports Frontier stress tests and implementation details. It also specifies the training runs and checkpoints behind these evaluations.

I AUDIT QUERY BUDGET ANALYSIS

  • At nominal budget 400, detection margins remain positive in both settings: Llama measures +2.25 pp and Frontier +0.75 pp, as reported in Figure 6.The plotted budget is nominal total 4q, using up to q distinct prompts per carrier from its available audit pool without replacement.Source: I AUDIT QUERY BUDGET ANALYSIS · p. 19

J REGISTERED-KEY ATTRIBUTION PROTOCOL

  • Figure 3 averages 100Tu across all alternative sign patterns at each Hamming distance; linearity fixes the average at (1 − d/2) times the enrolled-code score.The four marked checkpoints are seeds of one code, not retrained under alternative keys.Source: J REGISTERED-KEY ATTRIBUTION PROTOCOL · p. 19

K REMOVAL STRESS TESTS

  • The Frontier removal tests compare balanced-carrier SFT without keyed rewards and reverse-code anti-watermark GRPO against a matched Clean arm.Both use steps 0, 256, 512, 1024, and 2048; Figure 5 reports WM effect and ∆Umark.Source: K REMOVAL STRESS TESTS · p. 19
  • For Frontier, WM effect is 0 at ρ = 0, 4.2 pp at ρ = 0.01, and 11.5 pp at ρ = 0.025; at ρ = 0.10, it adds 1.85 pp with ∆Umark = −0.35 pp.At ρ = 0.05, the joint result is +15.4 ± 0.5 with ∆Umark = +0.5 pp; below 0.05, ∆Umark stays within ±0.1 pp.Source: L ρ SWEEP ABLATION · pp. 19–20, Figure 7 · p. 20

M IMPLEMENTATION AND CHECKPOINT MAP

  • Initial GRPO uses one epoch, 16 rollouts per prompt, and batches of 50, scoring 800 responses per optimizer step.Llama and Qwen2.5 joint runs use 12,473 training prompts; Frontier runs use 10,000, and the step counts are upper bounds.Source: M IMPLEMENTATION AND CHECKPOINT MAP · p. 20, Table 12 · p. 20
  • RQ1 and RQ2 each use four independent seeds with matched Clean checkpoints; Tables 1–3 report the mean and standard deviation of paired differences.RQ4 audits four joint-training checkpoints, while RQ5 continuation starts from step-0 marked checkpoints.Source: M IMPLEMENTATION AND CHECKPOINT MAP · p. 20
  • Joint Llama/Qwen2.5 training uses one registered four-sign code, while Frontier uses a separate four-sign code shared across its listed evaluators.Single-carrier RQ1 uses one enrolled carrier per domain with code (+1).Source: M IMPLEMENTATION AND CHECKPOINT MAP · p. 21

Related papers

Loading 2610.15685v1…