Policy Alignment: New Signals for Membership Auditing in On-Policy Distillation

• Because OPD prompts may be private, sensitive, and valuable, prompt-level membership auditing can help protect confidential information and training-resource ownership (Shokri et al. 2017; Liu et al. 2026).Source: §1 · p. 1
• Likelihood-based audits require predefined responses or confidence unavailable for OPD trajectories, while checkpoint drift measures change magnitude but not teacher direction (Puerto et al. 2025; Liu et al. 2026).Source: §1 · p. 1
• Member prompts directly drive teacher-aligned updates that reduce teacher loss on those prompts, whereas non-members receive weaker, indirect effects through cross-prompt generalization.Source: §1 · pp. 1–2
• Teacher Alignment Gain (TAG) estimates alignment without training gradients using a two-point zeroth-order approximation of teacher-loss change before and after distillation on the same student-generated trajectories.Source: §1 · p. 2
• PAMA combines TAG with Zlib, Min-K, student drift, and teacher-aligned entropy gain signals for auditing under different teacher-access settings.Source: §1 · p. 2
• Across six datasets and three teacher–student model families, PAMA improves AUC by 0.101–0.146 over SOTA baselines (14.6–20.6% relative) under Full-distribution and Top-K access.Source: §1 · p. 2

On-Policy Distillation

• OPD trains a student to match a fixed teacher on student-generated responses, using a representative reverse-KL objective averaged over sampled response tokens.The student’s pre- and post-distillation policies and the teacher are distinguished, and each token-level KL term is conditioned on the same prompt and response prefix. (Agarwal et al. 2024)Source: On-Policy Distillation · p. 2

Membership Auditing

• Text-level likelihood and related attacks score an observed response, not whether its prompt was used for OPD; DIBA’s policy-shift signal transfers, but its other signals need unavailable training information.Prior language-model attacks use likelihood, compressed length, or least-likely tokens (Duan et al. 2024; Carlini et al. 2021; Shi et al. 2023; Zhang et al. 2025; Mattern et al. 2023; Xie et al. 2024; Fu et al. 2023). DIBA audits reinforcement-learning data through checkpoint changes (Liu et al. 2026).Source: Membership Auditing · p. 2

Threat Model

• The post-hoc auditor predicts whether a candidate prompt was used in a completed OPD run, without access to its membership label at prediction time.The hypotheses are H1: x ∈ DOPD and H0: x /∈ DOPD.Source: Threat Model · p. 2
• The auditor can query and sample from pre- and post-distillation students, but cannot inspect parameters, gradients, or OPD training information.A disjoint labeled reference set of members and matched non-members supports feature standardization and predictor fitting (Shokri et al. 2017; Puerto et al. 2025; Liu et al. 2026). Teacher access is either full next-token distributions or only the top K tokens and their probabilities.Source: Threat Model · pp. 2–3

Audit Contexts

• Audit scores use sampled response prefixes as shared contexts so each student and teacher distribution is evaluated at the same inputs.Contexts are sampled from the pre- or post-distillation student policy and reused for the evaluations within each score.Source: Audit Contexts · p. 3

Student Drift

• Student Drift (SD) measures the magnitude of policy change on the candidate prompt, without indicating whether that change follows the teacher-directed objective.SD compares pre- and post-distillation policies using post-student contexts.Source: Student Drift · p. 3

Teacher-Directed Progress

• Across Qwen, InternLM, and Gemma, member prompts had higher teacher-directed projection values than matched non-members, with p < 0.001 for every comparison.The comparison uses the completed OPD update in LoRA parameter space on shared pre-student contexts.Source: Figure 2 · p. 3, Teacher-Directed Progress · p. 4
• A member’s own training gradient adds a non-negative projection component, while cross-prompt effects can be positive or negative for both members and non-members.The own-gradient term exists only for members; when cross-prompt effects are comparable, this additional term makes larger member projections more likely, without ordering every member–non-member pair.Source: Teacher-Directed Progress · p. 3
• TAG estimates teacher-directed progress from endpoint evaluations along the checkpoint path, while TAEG separately measures reduction in the teacher–student uncertainty gap.Positive TAG indicates the post-distillation student is closer to the teacher on fixed pre-student contexts; positive TAEG indicates closer uncertainty alignment.Source: Teacher-Directed Progress · p. 4

Access-Aware Signal Computation

• For limited teacher access, PAMA adapts teacher-referenced statistics using top-K probabilities with residual mass or a surrogate teacher, while SD remains unchanged.The top-K representation groups hidden probability mass into a residual bucket and provides a corresponding entropy feature.Source: Access-Aware Signal Computation · p. 4

Evidence Aggregation

• The audit combines selected standardized features with checkpoint-calibrated confidence controls and fits a logistic regressor on each audit-training fold.It reports repeated stratified five-fold out-of-fold scores; calibrated Zlib and Min-K compare pre- and post-distillation checkpoints.Source: Evidence Aggregation · p. 4

Experiment Setup

• Experiments cover six task datasets, three instruction-tuned teacher–student families, and likelihood, Zlib, Min-K, and DIBA auditing baselines.The smaller model is the student, the larger is the fixed teacher; comparisons use checkpoint-calibrated Zlib and Min-K, and only DIBA’s policy-shift component. Auditing reports ROC-AUC and low-FPR TPR, with TPR@0.1%FPR emphasized; the stated protocol samples 1,000 members and 1,000 non-members, uses temperature 0.8, R=10 responses, and K=10 for Top-K access.Source: Experiment Setup · pp. 4–5

Main Auditing Results

• PAMA achieves the strongest overall performance across all six MATH access settings and maintains a clear advantage in the low-FPR region.Top-K matches or exceeds full-distribution access for Qwen and InternLM, while Gemma shows a small decrease; the passage attributes the pattern to informative high-probability tokens and possible noise from the low-probability tail.Source: Figure 3 · p. 5, Main Auditing Results · p. 5
• OPD improves MATH accuracy for Qwen from 65.3% to 84.3%, InternLM from 43.2% to 67.7%, and Gemma from 52.1% to 78.9%.These gains establish the premise that OPD transfers teacher knowledge to the student before auditing is evaluated.Source: Main Auditing Results · p. 5

Further Understanding

• TAG agrees with the gradient-based audit projection on 88% of prompts, and its auditing performs better with the true OPD teacher than a surrogate reference.Across Qwen tasks, PAMA has above-chance AUC on MMLU-Pro, BigCodeBench, and MulDimIF; the preferred access setting varies by task, and AUC changes do not always track low-FPR TPR.Source: Figure 4 · p. 6, Further Understanding · p. 6, Table 1 · p. 6
• Adding SD, TAG, and TAEG cumulatively improves PAMA, with TAG producing the largest gain and the complete signal combination performing best.SD alone is generally modest; TAEG adds complementary predictive-uncertainty information. PAMA also remains effective across forward-KL and Jensen–Shannon OPD objectives, with Top-K yielding higher AUC for both, though low-FPR results vary by objective.Source: Table 2 · p. 6, Further Understanding · p. 6, Table 3 · p. 6
• Auditability weakens as cross-prompt generalization rises, while performance largely saturates by K=10 and response-count gains diminish near R=10.High generalization approaches random guessing; additional teacher probabilities at K=15 add little, and responses beyond R=10 change performance little while increasing query and generation costs.Source: Figure 5 · p. 7, Figure 6 · p. 7, Figure 7 · p. 7
• Without the original teacher, a compatible surrogate still provides useful alignment evidence on Gemma and Qwen, but reduces PAMA’s advantage over policy-drift auditing.The surrogate shares the student’s model family and generation, tokenizer, vocabulary, and prompt format, but has different parameters from the OPD teacher; quality depends on how closely its output distribution matches the unavailable teacher.Source: Figure 8 · p. 7, Further Understanding · p. 7

5 Conclusion

• PAMA consistently improves auditing performance over existing likelihood- and drift-based methods across datasets, model families, OPD objectives, and teacher-access settings.The conclusion reports these results without specifying numerical values.Source: §5 · p. 7
• PAMA combines Student Drift, Teacher Alignment Gain, Teacher-Aligned Entropy Gain, and calibrated confidence signals, using only forward evaluations of pre- and post-distillation students.It avoids needing access to training gradients or trajectories.Source: §5 · p. 7
• High-probability teacher outputs contain most useful alignment information, while strong cross-prompt generalization can weaken membership distinguishability.Source: §5 · p. 7

A Implementation Details of OPD Variant Algorithms

• In the default OPD setting, reverse KL is evaluated token by token using student-generated responses and shared student–teacher contexts.For each member prompt, the student samples a response, and both distributions are evaluated at each position under the same prompt and preceding response tokens.Source: A Implementation Details of OPD… · p. 9
• Forward KL reverses the distributions and emphasizes teacher-high-probability tokens, penalizing the student for failing to cover those regions.The expectation is taken under the teacher distribution.Source: A Implementation Details of OPD… · p. 9
• Jensen–Shannon divergence symmetrically constrains discrepancy between the student and teacher distributions, unlike the asymmetric KL objectives.The appendix introduces a Jensen–Shannon token-level loss for this variant.Source: A Implementation Details of OPD… · p. 9
• Each variant’s final OPD loss averages its corresponding token-level loss across all positions in the sampled response.The token-level loss is instantiated as reverse KL, forward KL, or Jensen–Shannon divergence.Source: A Implementation Details of OPD… · p. 9
• All three OPD variants use the same 1,000 member prompts and training setup, differing only in their distribution-alignment loss.Training lasts 10 epochs with LoRA (rank 32, scaling α = 64) and a learning rate of 1 × 10−4; students generate responses with temperature 0.8, top-p 0.95, and at most 128 new tokens.Source: A Implementation Details of OPD… · p. 9

Feature construction

• For each prompt, the five audit signals use 10 sampled responses per signal, with pre-distillation student, post-distillation student, and teacher evaluated on shared response contexts where applicable.Responses are sampled from Spre for Zlib, Min-K, TAG, and TAEG, and independently from Spost for SD.Source: B Implementation Details of Logistic… · p. 9

Training Procedure

• The labeled reference set has 1,000 member and 1,000 matched non-member prompts; repeated stratified five-fold cross-validation trains on 1,600 samples and holds out 400 per fold.Feature standardization is calculated using only the training fold and applied with those same statistics to held-out samples, preventing leakage.Source: B Implementation Details of Logistic… · p. 9
• The audit model is an ℓ2-regularized logistic regressor trained with binary cross-entropy, C = 1, L-BFGS, an intercept, balanced class weights, and up to 5,000 iterations.Because each training split has equal numbers of members and non-members, balanced weighting assigns identical weights to both classes.Source: B Implementation Details of Logistic… · p. 9

Prediction Procedure

• Across five repetitions with seeds 42, 1042, 2042, 3042, and 4042, each prompt obtains five repeated out-of-fold probability estimates, each generated by a model not trained on that prompt.Source: B Implementation Details of Logistic… · p. 10
• ROC-AUC and TPR at specified FPR operating points are computed from averaged scores, without applying a fixed classification threshold.Source: B Implementation Details of Logistic… · p. 10