PaperlayerView exact revision on arXiv

Source-linked AI summary

LaRA: Layer-wise Representation Analysis for Detecting Data Contamination in RL Post-Training

Minju Gwak, Minseo Kwak, Dongseok Lee, Guijin Son, Alan Ritter, Jaehyung Kim

arXiv:2605.29888v1cs.LGcs.AI

TL;DR

Data contamination in RL post-training is underexplored, and output-level signals can be unreliable because RL shapes behavior through trajectory-level rewards rather than token likelihoods. LaRA detects contamination through layer-wise representation geometry under controlled perturbations, and its aggregated detection protocol outperforms output-level baselines across RL-trained models.

  • Problem

    Data contamination in RL post-training is underexplored even though contaminated samples can undermine generalization and evaluation reliability.

  • Method

    LaRA extracts layer-wise representations under controlled semantic perturbations and aggregates RSM, DC, and RSI deviations into a contamination-detection protocol.

  • Results

    LaRA consistently outperforms output-level baselines across RL-trained models, achieving up to +9.6% AUC improvement and 3.5× higher TPR@FPR=5%.

  • Takeaways & Limitations

    Contamination in RL-trained LLMs is strongly reflected in internal representation geometry, making representation-level signals useful for auditing.

  • Takeaways & Limitations

    LaRA requires semantic perturbation generation, intermediate-layer extraction, and multiple geometric statistics, making inference more expensive than output-only approaches.

Abstract

from arXiv · show

Reinforcement learning (RL) post-training has shown to improve reasoning in large language models (LLMs). However, there has been little exploration on the problem of data contamination in RL post-training, potentially undermining generalization and evaluation reliability of the training process itself. Existing detection methods primarily rely on output-level signals such as likelihood or entropy, which become unreliable for RL-trained models since RL shapes behavior through trajectory-level rewards rather than token likelihoods. We propose LaRA, a layer-wise representation analysis framework for detecting contamination in RL post-trained LLMs. LaRA introduces three complementary metrics, measuring perturbation sensitivity, directional collapse, and local representation rigidity under controlled perturbations. We find that contamination produces progressive geometric deviations across layers, including amplified perturbation sensitivity, stronger directional collapse, and enhanced local rigidity. Based on our findings, we also develop a contamination detection protocol that aggregates representation-level deviations across layers and metrics. Experiments on RL-trained reasoning models show that our protocol outperforms existing output-level baselines for contamination detection.

1 Introduction

Data contamination in RL post-training is underexplored, while output-level detection can be unreliable because RL optimizes trajectory rewards rather than token likelihoods. LaRA addresses this gap by analyzing layer-wise representation geometry under controlled perturbations and achieves stronger detection than output-level baselines.

  • RL post-training contamination can cause reward-driven overfitting and implicit memorization, undermining generalization and evaluation reliability.
  • Output-level signals are unreliable for RL-trained models because poor calibration and trajectory-level reward optimization weaken their alignment with the training objective.
  • LaRA probes abnormal layer-wise representation responses by comparing original inputs with semantically similar variants and versions lacking shared key information.
  • RSM measures perturbation sensitivity, DC measures directional concentration, and RSI measures local invariance across semantically similar variants.
  • Contaminated samples show greater information-removal sensitivity, stronger directional collapse, and higher local representational rigidity than nontrained samples.
  • Up to +9.6% AUC improvement and 3.5× higher TPR@FPR=5% are achieved over the strongest prior output-level method.

2 Related Work

Prior contamination-detection work largely uses output statistics developed for pre-training or supervised fine-tuning, while representation-dynamics research studies internal states and semantic attributes. LaRA extends this representation-level perspective to layer-wise contamination analysis.

  • Prior contamination methods mainly target pre-training or SFT and characterize memorization through higher token likelihoods or lower entropy.
  • Existing RL contamination detectors extend output-level analysis to reasoning trajectories using entropy or behavioral divergence across generation stages.
  • Representation-dynamics research analyzes internal states across layers to characterize properties emerging during post-training.
  • LaRA analyzes hidden representations from original, perturbed, and paraphrased inputs using RSM, DC, and RSI.

3 LaRA: Layer-wise Representation Analysis to Detect RL Contamination

LaRA frames RL-post-training contamination detection as membership inference and analyzes layer-wise representation responses to controlled semantic and information-removal perturbations. It uses complementary geometric metrics to distinguish member from non-member samples across layers.

  • Problem formulation: LaRA formulates contamination detection as membership inference, distinguishing training-set members from non-members using layer-wise representation signals.The analysis asks whether member and non-member samples behave differently in internal representations.
  • Experimental design: The evaluation uses released RL checkpoints and a controlled continued-RL corpus to track contamination signals in two stages.The benchmark covers checkpoint-based detection and signal evolution during additional RL training.
  • Metrics: RSM measures perturbation-induced representation shifts after importance-based blanking, with larger standardized shifts indicating greater sensitivity to information removal.Representations are extracted layer-wise from original and blanked questions, and shift magnitudes are compared with semantic neighbors.
  • Metrics: DC measures cosine alignment between an original perturbation direction and the average direction across semantically similar questions.Higher DC indicates stronger concentration along a shared low-dimensional direction; lower values indicate more distributed dynamics.
  • Metrics: RSI measures local representation variability across paraphrastic variants while preserving blank positions, standardizing the original question against semantic neighbors.Higher RSI indicates greater local variability relative to similar questions, whereas lower values indicate more locally stable behavior.
  • Observed geometry: Across layers, contaminated samples show larger RSM shifts than clean samples, with sharp deviations around layers 7–9 and distinct directional concentration dynamics.The figure compares contaminated and clean samples using RSM, DC, and RSI under input perturbations.

4 Contamination Detection Protocol

The protocol detects contamination as layer-aware deviations from clean representation geometry, combining robustly standardized and metric-aligned signals across layers. This aggregation produces a single score reflecting overall deviation from the clean profile.

  • Protocol rationale: LaRA characterizes contamination through deviations from clean geometric profiles across multiple metrics and layers, rather than isolated layer-wise statistics.The protocol tracks perturbation sensitivity, directional concentration dynamics, and local variability under controlled perturbations.
  • Protocol rationale: The protocol uses RSM, DC, and RSI across the probed transformer layers, whose raw values span several orders of magnitude.These metrics measure distinct contamination-associated geometric behaviors.
  • Step 1: Clean-reference Robust Standardization: Each metric-layer value is first compressed with a sign-preserving transformation to control heavy-tailed values while leaving values near zero unchanged.This preserves relative deviation magnitudes before robust standardization.
  • Step 1: Clean-reference Robust Standardization: Clean validation samples provide robust reference centers and scales, preventing extreme contaminated samples from inflating the reference scale and weakening detection.The scale uses median absolute deviation with the standard 1.4826 factor.
  • Step 2: Metric-specific Anomaly Alignment: Metric-specific alignment preserves directional DC deviations and aligns RSM and RSI deviations with their contamination-associated geometric patterns.For RSI, the alignment reverses the standardized deviation sign.
  • Step 3: Layer-wise Aggregation: Aligned deviations are aggregated across metrics and layers into a per-sample SLaRA score, with larger values indicating stronger overall deviation from the clean geometric profile.Because contributions share a robust z-scale, abnormalities from different layers and metrics can be compared and combined.

5 Experiments

Experiments evaluate representation-based contamination detection across RL models, checkpoints, metric combinations, score mixtures, perturbations, and error cases. The proposed signals generally outperform output-level baselines and remain robust across several design choices, but false negatives and false positives remain.

  • 5.1 Main Results: SLaRA consistently achieves strong and stable detection across RL model families and training checkpoints.On LIMR, it reaches AUC 0.80 and TPR@FPR=5% 0.46 at initial checkpoints.
  • 5.1 Main Results: The combined SC + SLaRA score reaches AUC 0.73 and TPR@FPR=5% 0.31 on Eurus at initialization, improving to 0.79 and 0.38 across checkpoints.These values are reported as (AUC, TPR@FPR=5%).
  • 5.1 Main Results: PPL can obtain relatively high AUC, but its TPR@FPR=5% is lower and less stable than SLaRA and SC + SLaRA.The proposed methods capture hidden-representation inconsistencies rather than relying only on token-likelihood differences.
  • 5.1 Main Results: Combining RSM, DC, and RSI gives the best overall ablation performance, while removing any component degrades results and DC alone varies more across epochs.RSM and RSI are individually weaker but improve generalization under RL post-training when combined.
  • 5.2 Additional Analyses: The shared default β = 0.65 remains competitive despite model-dependent optima, while no single mixture weight is universally optimal.Eurus peaks at β = 0.65 for AUC, whereas LIMR favors SLaRA alone and OLMO has its highest AUC at β = 0.
  • 5.2 Additional Analyses: Detection is robust across perturbation counts and types, although metric-specific peaks differ and all perturbation variants achieve AUC values between 0.56 and 0.69.For perturbation count, AUC peaks at k = 3 with 0.77, while TPR@FPR=5% peaks at k = 4 with 0.32.
  • 5.2 Additional Analyses: Error analysis identifies both missed members and falsely detected non-members caused by atypical representation geometry.A missed member has low RSM, DC, and RSI scores, while a false positive is primarily driven by unusually high DC.

6 Conclusion

LaRA detects contamination in RL-trained models through perturbation-induced representation geometry across layers. Its aggregated representation-level signals complement output-level methods and improve detection performance.

  • LaRA analyzes perturbation-induced representation geometry across layers to detect contamination in RL-trained models.
  • The framework aggregates geometric deviations across multiple metrics and layers rather than relying on isolated layer-wise statistics.
  • Experiments across various RL-trained models show that representation-level signals complement output-level methods and improve contamination detection performance.
  • The results indicate that contamination is strongly reflected in internal representation geometry, supporting representation-level auditing.

Limitations

LaRA requires more computation than lightweight output-level approaches and does not reliably separate every contaminated sample. The relationship between RL post-training dynamics and representation-level memorization also remains only partially understood.

  • LaRA incurs additional inference overhead by generating perturbed variants, extracting layer-wise hidden states, and aggregating geometric statistics.
  • Detection remains imperfect for challenging examples whose representation geometry overlaps closely with the clean distribution.
  • Some memorized samples may lack sufficiently distinctive internal signatures for reliable separation.
  • The precise relationship between RL post-training dynamics and representation-level memorization behavior remains only partially understood.

Broader Impact and Ethical Implications

The work aims to improve transparency and reliability in evaluating RL-trained LLMs through representation-level contamination detection. It also identifies potential privacy and data-governance concerns from adapting such methods to sensitive or proprietary data.

  • Representation-level contamination detection can help assess benchmark integrity and reduce hidden memorization effects in RL-trained LLM evaluation.
  • Stronger contamination auditing may improve the trustworthiness of reported reasoning capabilities.
  • Contamination detection methods may potentially be adapted for membership inference or unauthorized dataset auditing.
  • Applying these methods to sensitive or proprietary data could raise privacy or data-governance concerns.

E.1 Performance Evaluation of Trained Models on Member vs. Non-member

RL post-training produces measurable behavioral changes and a consistent preference for member samples over non-member samples. This member–non-member gap persists across checkpoints, even when overall performance changes little.

  • Overall Pass@5 improves from 15.0% in the base model to 18.3% after RL post-training for Eurus-2-7B-PRIME.
  • Member samples consistently achieve higher Pass@5 than non-member samples across checkpoints, with the gap increasing during training.
  • The results confirm non-trivial optimization and measurable behavioral changes, alongside a consistent preference toward member samples.

F Additional Results on Figure Geometry

Across RL-trained models, contamination is associated with layer-wise geometric deviations, although the dominant metric and separation pattern vary by model and training stage. These signals remain detectable across layer windows and support robust aggregation across depth.

  • Additional model geometry: Contaminated samples show elevated RSM and altered DC and RSI profiles, but the specific geometric pattern varies across RL-trained models.Eurus exhibits strong RSM separation and lower RSI, whereas Olmo shows weaker RSM separation but differences in DC and RSI.
  • Eurus-2-7B-PRIME: Across Eurus checkpoints, RSM remains stable at d ≈ 0.27–0.35 while DC becomes increasingly negative with RL training.The reported separation is measured across early, mid, and late layer windows.
  • LIMR: In LIMR, DC provides the strongest separation, with consistently negative effects of d ≈−0.85 to −1.25 across layer groups.Mid and late layers show the strongest separation, which is slightly amplified during later RL checkpoints.
  • Layer-window analysis: Detection performance remains largely stable across early, middle, late, and all-layer windows, with LIMR achieving approximately 0.8 AUC across windows.LIMR also has the highest TPR, while EURUS is moderate and OLMO weaker; no single layer region dominates.
  • Ablations: The ablation results indicate that SLaRA is relatively robust, including on the additional Eurus model.The cited ablation comparison covers Eurus across RL epochs.
  • Training evolution: RL training progressively increases the separation between clean and contaminated representation trajectories across RSM, DC, and RSI in Eurus.RSM shows particularly strong divergence beginning in early-to-middle layers.

H Justifications of the Scaling Factor Metric in Contamination Detection Protocol

The protocol uses the factor 1.4826 to calibrate MAD as a Gaussian-consistent scale estimate. This preserves standardized-score interpretation while improving robustness to atypical clean-reference samples and heavy-tailed representation metrics.

  • Scaling-factor rationale: The factor 1.4826 = 1/Φ^-1(0.75) makes MAD a consistent estimator of standard deviation under a Gaussian reference.For X ∼ N(µ, σ^2), multiplying MAD by this factor recovers σ asymptotically.
  • Protocol interpretation: Using the scaled MAD keeps standardized deviations and SLaRA scores on ordinary standard-deviation units without retuning downstream thresholds.The resulting scores retain an approximate Gaussian z-score interpretation.
  • Robustness: MAD has a 50% breakdown point, making it more stable than standard deviation when the clean-reference pool contains atypical samples.This robustness is relevant because representation-geometry metrics can remain heavy-tailed.

I.1 Metrics

The evaluation combines threshold-independent overall discrimination with performance at a strict false-positive operating point, alongside output- and behavior-based contamination baselines. These metrics frame both global separability and practical low-false-positive detection.

  • Evaluation metrics: ROC-AUC measures member–non-member separability across all decision thresholds and is threshold-independent.It provides a global measure of detection quality.
  • Evaluation metrics: TPR@FPR=5% measures correctly identified members when false positives are fixed at 5%.This operating point reflects the cost of incorrectly flagging clean samples.
  • Baselines: The comparison includes Recall, CDD, and Min-K% Prob among six contamination-detection baselines.These baselines probe regeneration, perturbation-based prediction discrepancies, and likelihood-based signals, respectively.
Loading 2605.29888v1…