OP-Print: Tracing Membership Imprints in On-Policy Self-Distillation

1 Introduction

• On-policy self-distillation removes the external teacher by using the student’s outputs with additional context or demonstrations as supervision.The model encounters coupled standard and privileged views of the same example, a design that may produce a stronger, more distinctive training imprint.Source: §1 · p. 1
• This study investigates whether membership inference can identify samples used to train on-policy self-distillation models under gray-box and black-box threat models.The paper frames this as an underexamined privacy risk of the teacher-free design.Source: §1 · p. 1
• OP-Print uses privileged teacher-student residual discrepancy along the same rollout for gray-box attacks and restart-based output-basin stability as a surrogate for hidden correction in black-box attacks.The framework targets the same hidden supervision imprint at two levels of adversary observability.Source: §1 · p. 2
• Across models and distillation schemes, baseline attacks remain essentially at random guessing even under gray-box access, while OP-Print is effective in both threat models.On Qwen3-8B in the black-box setting, it improves AUC by 7.7% and TPR@5% FPR by 26.5% over the strongest baseline.Source: §1 · p. 2
• The supervision imprint is sparse and structured, concentrating on uncertainty-sensitive tokens rather than being globally visible or primarily aligned with decoding position.This makes gray-box inference feasible but nontrivial, while black-box inference is harder because the adversary sees only final outputs.Source: §1 · pp. 1–2

On-Policy Self-Distillation

• On-policy distillation trains students on their own policy-sampled trajectories, and self-distillation uses the same model as teacher and student under asymmetric conditioning.OPSD gives the teacher privileged information while the student sees the original input, minimizing token-level divergence on student rollouts.Source: §2 · p. 2

Challenges

• Figure 2 reports only mild divergence variation across token positions, greater divergence with student uncertainty, and localized rather than uniform divergence in the reliability map.These observations characterize the supervision imprint as sparse and structured, concentrated on difficult reasoning tokens.Source: Figure 2 · p. 3

Membership Inference Attack

• Membership-inference attacks determine whether a sample was used to train a target model; metric-based attacks use output statistics, while shadow-model attacks train auxiliary models.This work focuses on metric-based methods because shadow-model attacks are computationally expensive and often impractical for LLMs.Source: §2 · p. 3
• Prior LLM membership-inference work covers pretraining and post-training data, and this paper focuses on post-training-stage attacks.The passage notes that developers often construct task-specific datasets for post-training, introducing additional privacy risks.Source: §2 · p. 3

Challenges

• The supervision imprint is sparse and structured, concentrated on difficult reasoning tokens rather than decoding position or the entire rollout.The analysis links this structure to the difficulty of black-box inference, where the informative correction signal is hidden.Source: §3 · pp. 3–4

Gray-box Attack: Privileged Residual Aggregation

• Gray-box OP-Print estimates the privileged correction by comparing student-conditioned and privileged teacher-conditioned behavior along the same rollout.A larger privileged gap indicates more disagreement; members are expected to have smaller average correction because it has been internalized.Source: §3 · p. 4

Black-box Attack: Restart-based Basin Stability

• In black-box access, the hidden correction cannot be estimated directly, so OP-Print attacks its observable behavioral trace in the output basin.The method interprets a higher black-box score as a narrower, more stable output basin under prefix restarts.Source: §3 · p. 5
• Black-box OP-Print truncates sampled generations at multiple prefix ratios, restarts decoding from each prefix, and measures similarity among the resulting continuations.The score averages restart-text stability across full generations and prefix ratios; higher stability is treated as evidence of membership.Source: §3 · p. 5

4 Experiments

• The evaluation samples 1,000 OpenThoughts mathematical reasoning examples, split evenly into 500 members and 500 non-members, and fine-tunes Qwen3-1.7B, 4B, and 8B.The attacks are tested on generalized-JSD and tinker-style objectives; gray-box results use chunk size 64 and K = 32, while black-box results use |P| = 10.Source: §4 · pp. 5–6, §4.1 · p. 7

4.1 Main Results

• OP-Print-G-Max raises gray-box AUC over the strongest non-OP-Print baseline from 0.517 to 0.594 on Qwen3-1.7B, 0.517 to 0.611 on Qwen3-4B, and 0.529 to 0.587 on Qwen3-8B.Its TPR@5%FPR scores are 0.110, 0.127, and 0.102, respectively.Source: §4.1 · p. 7
• Likelihood- and entropy-based baselines remain near random guessing, with AUCs around 0.49–0.52, while OP-Print-G and OP-Print-G-Max perform substantially better.The consistent advantage of OP-Print-G-Max suggests membership information is concentrated at the most discrepant positions.Source: §4.1 · p. 7
• OP-Print-B achieves the best overall black-box AUC and accuracy across model scales, reaching 0.575 AUC and 0.573 accuracy on Qwen3-4B.Some simple rollout heuristics have higher TPR@5%FPR on individual models, but their gains are less consistent.Source: §4.1 · p. 7
• OP-Print remains effective under the tinker-style objective: OP-Print-B is strongest overall in AUC and accuracy across scales, while gray-box variants outperform likelihood-based baselines.OP-Print-G-Max achieves the best AUC on Qwen3-1.7B and Qwen3-8B; simple rollout heuristics sometimes lead in TPR@5%FPR on individual models, but less consistently.Source: §4.1 · pp. 7–8, Table 2 · p. 8

4.2 Ablation Study

• Performance improves with up to 32 rollouts, chunk size 64 is best or near-best, more prefix ratios improve black-box metrics gradually, and additional restarts yield diminishing returns at higher cost.The default restart setting provides a favorable cost-performance trade-off; Figure 5 summarizes these ablations.Source: Figure 5 · p. 7, §4.2 · p. 8

• The identified correction imprint is sparse and concentrated on uncertainty-sensitive tokens.Source: §5 · p. 8
• OP-Print captures privileged teacher–student discrepancies in gray-box attacks or their observable restart-based behavioral traces in black-box attacks.Source: §5 · p. 8
• Experiments across models and distillation schemes show that OP-Print consistently outperforms existing works.Source: §5 · p. 8

• Black-box attacks require repeated restart generation, adding inference cost; the paper leaves defense design against correction-imprint leakage to future work.Source: Limitaions · p. 9

A Training Details

• Models are trained on the OpenThoughts mathematical reasoning subset, partitioned into disjoint train and held-out halves with seed 42, then sampled to form 500-member and 500-non-member pools.Source: A Training Details · p. 13
• All target-model runs use a fixed teacher corresponding to the initial policy with privileged information, while the student is updated during training.Source: A Training Details · p. 13
• Training uses on-policy generation with maximum completion length 1024, and the distillation objective is computed on generated continuations while prompt tokens are masked from loss.Runs use 8 NVIDIA A6000 GPUs, bfloat16, gradient checkpointing, stochastic sampling, and colocated vLLM.Source: A Training Details · p. 13
• Standard OPSD uses token-level generalized Jensen–Shannon divergence, while the Tinker-style variant uses a sampled-token policy-improvement objective based on teacher–student log-probability differences.Both apply the teacher signal to the student’s own on-policy generations.Source: A Training Details · p. 13

B Additional Analysis of the Black-box Score

• The appendix explains why restart-based output stability can serve as a surrogate for the hidden correction imprint.It presents this as intuition and lightweight theoretical support, not a fully general identifiability theorem.Source: B Additional Analysis of the Black-box… · p. 13
• The analysis also explains why repeated restarts reduce estimation variance.Source: B Additional Analysis of the Black-box… · p. 13
• A more concentrated output basin leads to a larger black-box score.Source: B Additional Analysis of the Black-box… · p. 13

B.1 Restart-text Stability as a Surrogate for the Hidden Correction Imprint

• Latent correction strength represents how strongly privileged correction has been internalized at a prefix, and is not directly observable.The assumed link is that this strength affects the local basin concentration of restarted continuations.Source: §B.1 · p. 13
• Population restart-text stability is expected pairwise similarity between two independent continuations restarted from the same prefix-conditioned distribution.The black-box attack assumes this stability is non-decreasing in latent correction strength.Source: §B.1 · p. 13
• Across prefix ratios, the population black-box score averages a monotone transform of correction strength; if members have greater strength on average, stability preserves membership ordering in expectation.The member-strength relationship is posited, and the order-preserving conclusion follows under the non-decreasing-transform assumption.Source: §B.1 · p. 14
• The empirical black-box attack estimates the population-level surrogate using finitely many full generations and restarts.It can therefore be viewed as a Monte Carlo estimator.Source: §B.1 · p. 14

B.2 Why Increasing the Number of Restarts Reduces Estimation Variance

• Pairwise restart-text stability is an unbiased estimator of population stability, and its second-order U-statistic variance decreases at O(R^-1), more slowly than O(R^-2).The variance result assumes a mildly bounded similarity kernel.Source: §B.2 · p. 15
• Averaging restart estimators across sampled generations and prefix ratios further reduces final-score variance by roughly a factor of M|P| when estimates are weakly dependent or approximately independent.Source: §B.2 · p. 15
• Increasing the restart count R reduces variance in each prefix-level stability estimate.Source: §B.2 · p. 15
• Increasing M or |P| further stabilizes the final score, and the ablation trend shows diminishing returns as more restarts improve stability.Source: §B.2 · p. 15

B.3 A Simple Two-State Toy Example

• In the two-basin model, expected pairwise similarity is lowest when restart trajectories split evenly and highest when nearly all fall into one basin.The model assumes within-basin texts are more similar than cross-basin texts.Source: §B.3 · p. 15
• Under the proposed membership connection, stronger internalized correction concentrates member restarts in one basin, yielding higher stability and making members easier to attack.The membership relationship is stated as an assumption in this toy interpretation.Source: §B.3 · p. 16

C Extended Related Work

• Prior self-distillation work uses privileged contexts or reference solutions, while the privacy implications of that setting remain insufficiently understood.The related work also describes other supervision and learning objectives, including semantic verification and uncertainty-guided expert selection.Source: C Extended Related Work · p. 16
• Existing membership attacks use token probabilities, sequence likelihoods, calibrated uncertainty, and distributional-confounder controls; OP-Print instead identifies localized uncertainty-sensitive correction signals.Source: C Extended Related Work · p. 16
• For autoregressive generation, OP-Print measures restart-based output stability as a black-box surrogate, complementing its gray-box aggregation of privileged teacher-student discrepancies.Source: C Extended Related Work · p. 16