PaperlayerView exact revision on arXiv

Source-linked AI summary

GradSentry: Gradient Spectral Entropy for Backdoor Sample Filtering in Large Language Model Fine-Tuning

Haodong Zhao, Tianyi Xu, Tianhang Zhao, Zhuosheng Zhang, Gongshen Liu

arXiv:2605.26574v1cs.CR

TL;DR

Untrusted fine-tuning data can implant backdoors, while clustering-based filters may require sufficient samples and struggle at extreme poison ratios. GradSentry instead scores individual per-sample gradient spectra using spectral entropy, finding higher entropy for poisoned samples and robustly filtering them across datasets, attacks, and poison ratios.

  • Problem

    Clustering-based backdoor sample filters require sufficient data and can fail when poison ratios are extreme.

  • Method

    GradSentry filters samples by computing spectral entropy from each sample’s gradient spectrum instead of constructing pairwise similarities or clustering samples.

  • Results

    GradSentry reduces ASR to 0.00% in all 16 evaluated dataset-and-attack settings while preserving clean-task utility.

  • Takeaways & Limitations

    Higher spectral entropy in poisoned samples enables robust backdoor-data removal across datasets, attack types, poison ratios, and low-data regimes.

  • Takeaways & Limitations

    GradSentry’s per-sample gradient computation may be memory-intensive for very large batch sizes, and its applicability beyond SFT requires further investigation.

Abstract

from arXiv · show

Fine-tuning Large Language Models with untrusted data exposes models to backdoor attacks, where poisoned samples cause targeted misbehavior. Existing sample-filtering defenses rely on clustering, which requires sufficient data and can fail at extreme poison ratios. We propose GradSentry ({Grad}ient {Sentry}), a backdoor sample filtering method based on the spectral entropy of per-sample gradients. Our key finding is that poisoned samples produce gradients with higher spectral entropy compared to clean samples. GradSentry captures output-altering backdoor signatures using per-sample gradient spectra, avoiding pairwise sample comparisons and clustering during feature construction. Importantly, our method is training-agnostic: it works for both parameter-efficient fine-tuning methods like LoRA and full-parameter tuning, as the gradient analysis operates independently of which parameters are being updated during training. GradSentry requires no clustering, operates effectively across all poison ratios (1%--90%), and introduces minimal computational overhead (20-50ms per sample for 7B model). Evaluation on four QA datasets and four attack types demonstrates the effectiveness of spectral entropy for backdoor detection. Code is available at https://github.com/dongdongzhaoUP/GradSentry.

1 Introduction

GradSentry addresses limitations of clustering-based defenses by filtering poisoned samples from individual gradient spectra. It identifies higher spectral entropy as a poisoning signal and reports robust performance across datasets, poison types, and settings.

  • Motivation: Existing filtering methods can require sufficient samples, fail at extreme poison ratios, and incur pairwise-comparison or clustering costs.These limitations arise from their reliance on relational analysis among samples.
  • Method: GradSentry detects poisoned samples using the spectral entropy of each sample’s gradient rather than pairwise similarities or clustering.Poisoned samples tend to have more uniformly distributed singular values and therefore higher spectral entropy than clean samples.
  • Advantages: GradSentry is clustering-free, provides an interpretable continuous measure of gradient dispersion, and scales linearly with sample volume.Its computation uses truncated SVD on a subsampled gradient matrix.
  • Contributions: Spectral entropy of per-sample gradients is identified as an effective signal for poisoned-sample filtering in LLM fine-tuning.The method detects poisoned samples through the intrinsic spectral structure of individual gradients.
  • Evaluation: Experiments across multiple datasets, poison types, and settings show strong robustness while preserving utility.The supplied contribution statement reports this cross-setting outcome without specifying a numerical aggregate.

2 Related Work

The related work covers insertion-based and stealthy backdoor attacks, along with input-, model-, and data-level defenses. It positions GradSentry against data-level methods that commonly depend on relational analysis and sufficient data density.

  • Backdoor Attacks: Backdoor attacks train models to behave normally on clean inputs but produce attacker-specified outputs when triggers are present.Examples include rare-token, fixed-sentence, semantic-preserving, syntactic, style-based, and composite triggers.
  • Backdoor Defenses: Defense mechanisms include input-level trigger detection, model-level post-training repair, and data-level filtering before or during training.The paper places its approach within data-level defense.
  • Prior Methods: Existing data-level defenses include activation analysis, robust statistics, internal-state analysis, and clustering of learned representations.The cited methods include Spectral Signatures, Activation Clustering, SPECTRE, DEMON, and CUBE.
  • Limitations: Many existing defenses target vision or classification tasks rather than language-model fine-tuning.This limits the stated task coverage of those methods.
  • Limitations: Relational defenses depend on data quantity and feature-space density, especially when clean and poisoned groups are highly imbalanced.Their reliance on comparing or clustering samples creates this dependency.

3 Method

GradSentry detects poisoned samples before fine-tuning by analyzing the spectral entropy of per-sample gradients, independently of whether training updates use full parameters or PEFT. It scores gradient spectra individually and selects a filtering threshold from their entropy distribution.

  • Training-Agnostic Detection: GradSentry identifies poisoned samples before training using per-sample gradients, supporting both full-parameter updates and PEFT methods such as LoRA.The detection procedure is designed to remain independent of the training configuration.
  • Gradient Spectral Entropy: Backdoor samples spread gradient signal across multiple directions, producing greater spectral entropy than clean samples.Clean gradients primarily align with dominant directions, whereas backdoor gradients encode both normal task behavior and trigger-response mappings.
  • Gradient Extraction: For each sample, the method computes a loss gradient for a target module, typically the final output projection layer, then subsamples its matrix for efficiency.The default target is lm_head, and the gradient matrix is subsampled to its top 1/8 rows and columns.
  • Gradient Spectral Entropy: SVD decomposes each gradient matrix into singular directions and values, with truncated randomized SVD using k = 16 singular values by default.The singular values quantify how gradient magnitude is distributed across orthogonal directions.
  • Gradient Spectral Entropy: Normalized spectral entropy approaches 0 for concentrated gradients and 1 for gradients whose singular values are uniformly distributed.Normalization enables comparison across different gradient scales.
  • Threshold-Based Filtering: GradSentry uses Gaussian KDE to estimate the entropy density and sets the decision threshold at the valley between peaks near 0 and 1.If bimodality is absent, it falls back to an empirical threshold, 0.7 by default.

4 Experiments

Experiments evaluate GradSentry across four QA datasets, four attack types, tuning settings, target modules, poison ratios, and sample volumes. Results show strong backdoor removal, clean-utility preservation, and robustness where clustering-based defenses are limited.

  • End-to-End Results: GradSentry reduces ASR to 0.00% in all 16 dataset-and-attack settings while achieving the best ACC in 8/16 settings.The method preserves clean utility better than defenses whose ACC substantially degrades.
  • Sample Identification: 100.00% Recall is achieved in every sample-identification setting, although GraCeFul attains higher F1 in several cases while missing poisoned samples on three datasets.CUBE also has high recall but lower F1, consistent with more false positives and reduced ACC.
  • Entropy Analysis: GradSentry identifies poisoned samples through high-entropy gradient spectra, with poisoned samples concentrated at higher entropy than clean samples across datasets and attacks.WebQA shows greater clean-poison overlap and lower F1, but poisoned samples remain in the high-entropy tail and are removed.
  • Target Module Selection: The lm_head.weight target module achieves 100.00% Recall and 99.80% F1 with the automatic threshold, while effectiveness varies across other modules.The authors associate this choice with gradients in the final projection layer, where output manipulation is directly reflected.
  • Robustness: GradSentry maintains 100.00% Recall from 1% to 90% poison ratios, with average F1 of 82.38% at ratios no more than 5% and 98.82% at ratios at least 50%.It also remains effective with limited sample volumes, whereas clustering methods cannot operate in the smallest setting and become unstable with fewer samples.

5 Conclusion

GradSentry detects backdoor samples during LLM fine-tuning by analyzing per-sample gradient spectral entropy rather than relying on pairwise comparisons or clustering. Its entropy-based thresholding supports robust detection across datasets, attack types, poison ratios, and low-data regimes while preserving clean-task utility.

  • GradSentry analyzes each per-sample gradient’s singular-value distribution and uses a dataset-level entropy threshold to detect backdoor samples.The method avoids high-dimensional pairwise comparison and clustering during feature construction.
  • Poisoned samples exhibit higher spectral entropy than clean samples, enabling GradSentry to remove backdoor data while preserving clean-task utility.

Limitations

The evaluation is constrained by per-sample gradient memory requirements, a focus on supervised fine-tuning, and the need to access training data during filtering.

  • GradSentry may be memory-intensive for very large batch sizes because it requires computing per-sample gradients.
  • The experiments focus on supervised fine-tuning, so applicability to other training methods such as pretraining requires further investigation.
  • The method assumes access to training data at filter time, limiting its applicability to post-hoc model analysis.

Ethical Considerations

The paper frames backdoor detection as a defensive response to the attack surface created by supervised fine-tuning with potentially compromised data. Its gradient analysis is motivated by output-altering attacks and relies on concentrated gradient spectra.

  • The lm_head gradient is a sum of token-level outer products whose effective rank is governed by hidden-state and output-error geometry.
  • The first 16 singular values capture nearly all lm_head gradient energy, supporting the default truncated-SVD choice k = 16.
  • The experiments use Llama-2-7B with LoRA adapters of rank r = 4 and also evaluate full-parameter tuning.

C Dataset Details

The experiments use multiple QA datasets and specified trigger types, while the complexity analysis compares gradient-based and representation-based filtering as data volume and feature dimensions change. GradSentry computes independent entropy scores with truncated SVD and avoids global clustering operations.

  • Dataset Details: Four QA datasets from different domains are used to provide a comprehensive assessment across varied knowledge sources.
  • Attack Settings: The attacks include rare-token BadNets, sentence-trigger AddSent, component-specific CBA triggers, and Bible-style StyleBkd transfers.
  • Attack Settings: All attacks target the output ", and click ⟨malicious_url⟩for more information".
  • Complexity Analysis: GradSentry computes each sample’s spectral score independently using per-sample gradient extraction and truncated SVD on a subsampled gradient matrix.
  • Complexity Analysis: O(N) additional memory is required because the method stores only scalar entropy scores, while thresholding over N scores is negligible relative to gradient extraction.
  • Complexity Analysis: GradSentry scales linearly with data volume and avoids high-dimensional pairwise similarity computation or clustering, unlike methods whose costs grow with N and feature dimensions.

F Performance under Full-Parameter Fine-Tuning

GradSentry remains effective under full-parameter fine-tuning and across different models, datasets, and target modules. Its entropy separation is training-agnostic, while output-proximal modules provide the most reliable detection.

  • Full-parameter tuning: 100.00% recall and 0.00% ASR are achieved across all datasets and attack types under full-parameter fine-tuning.F1 is near-perfect on FreebaseQA, CoQA, and NQ, while WebQA has lower F1 because its clean and poisoned entropy distributions overlap more.
  • Cross-setting generalization: Poisoned samples consistently occupy higher-entropy regions than clean samples across tuning strategies and model architectures.The separation persists under full-parameter updates and across six additional LLMs evaluated with LoRA on FreebaseQA.
  • Threshold selection: Adaptive thresholds vary by model and dataset, so a universal fixed entropy threshold is suboptimal.Observed thresholds are around 0.70 for Qwen2.5-7B and Mistral, around 0.80 for Vicuna and Pythia-6.9B, and around 0.754–0.755 for GPT-J-6B.
  • Target module selection: lm_head.weight is the most reliable target module, reaching 100.00% recall, 99.80% F1 automatically, and 99.90% optimal F1.Intermediate modules are less stable, while late-layer modules can perform well but depend on layer position and module type; LoRA adapter modules are generally less effective.

I Performance on Clean-Only Datasets

GradSentry preserves benign data when no poisoned samples are present, complementing its poisoned-sample filtering results. Its clean-data retention is strongest on FreebaseQA, CoQA, and NQ, but weaker on WebQA.

  • Clean-only evaluation: 97.17% average clean sample identification accuracy is achieved across all four clean-only datasets.Higher accuracy indicates fewer false positives and better preservation of benign training data.
  • Dataset-specific behavior: More than 99% of clean samples are retained on FreebaseQA, CoQA, and NQ.WebQA is more challenging because its clean entropy distribution is broader and overlaps more with high-entropy regions.
  • Practical implication: The clean-only evaluation shows that GradSentry is conservative when no attack is present.This complements poisoned-data experiments by assessing false positives when every training sample is clean.

J.3 Experimental Results

Against an adaptive Gradient Dilution Attack, GradSentry maintains perfect recall across datasets and dilution ratios. The results attribute this robustness to persistent output-side spectral signatures and adaptive thresholding.

  • Adaptive attack results: 100% recall is achieved across all datasets and dilution ratios λ ∈ {0.5, 0.7}.The adaptive attack completely fails to evade detection at a 10% poison rate despite full knowledge of the detection mechanism.
  • Why dilution fails: The malicious output suffix creates a spectral pattern that dominates the gradient structure and cannot be masked by added semantic input content.This explains why context augmentation does not remove the detectable output-side anomaly.
  • Why dilution fails: The trigger-to-malicious-output mapping remains unchanged and continues producing anomalous gradient directions.Preserving backdoor functionality therefore preserves the gradient signature targeted by GradSentry.
  • Why dilution fails: KDE-based thresholding adapts to shifted entropy distributions while preserving bimodal separation between clean and poisoned samples.The results support gradient entropy as an intrinsic signal of the attack mechanism rather than an artifact of naive implementation.
Loading 2605.26574v1…