PaperlayerView exact revision on arXiv

Source-linked AI summary

A Reproducible, License-Aware Distillation Recipe for CPUDeployable Safety Classification

Edson Rodrigues da Cruz Filho, Paulo Ricardo Ferreira Neves, Paulo Henrique Eleuterio Falsetti, João Vitor Pavan, Ian Degaspari, Henrique Vieira Laturrague, Patrick Vieira Laturrague, Guilherme Nielsen Dias, Marccello Wilson Perez Berto, Gustavo Voltani Von Atzingen

arXiv:2608.21570v1cs.AIcs.LG

TL;DR

Commodity-CPU deployment makes existing multi-billion-parameter GPU-oriented safety guards impractical. This paper develops an auditable, license-aware distillation recipe and finds that much smaller students match teachers on adversarial text, reduce false alarms on harmless prompts, and run quickly on CPU, without surpassing the distilled guards.

  • Problem

    Existing multi-billion-parameter GPU-oriented safety guards are impractical for settings requiring local safety classification on commodity CPU hardware.

  • Method

    A strong open guard labels public-source data into seven safety categories, and lexical, shallow, encoder, and generative students distill its signal under license-partitioned, auditable training and independent evaluation.

  • Results

    Distilled students match teachers on adversarial text, reduce harmless-prompt false alarms to 3.8% versus 4.8% for the 8-billion-parameter teacher, and most classify in under 28 ms per CPU request.

  • Takeaways & Limitations

    The recipe transfers teacher behavior to much smaller CPU-runnable models, with per-class rebalancing as the only decisive ingredient and the encoder offering the best quality-latency trade-off.

  • Takeaways & Limitations

    The teacher sets the ceiling because its blind spots propagate, while teachers larger than 8 billion parameters were excluded by hardware limits.

Abstract

from arXiv · show

Deploying a safety layer for large language models on commodity hardware is constrained by the guards available to do it: current open guard models hold between 1 and 9 billion parameters, are oriented toward the graphics processing unit, and answer in seconds per request on a central processing unit. This paper presents a reproducible, license-aware knowledge-distillation recipe addressing that constraint. A strong open guard labels a corpus of roughly 97,000 prompts, drawn from 24 public datasets, into seven safety categories aligned to a public hazard taxonomy, and a fleet of small students spanning lexical, shallow, encoder and generative architectures is trained to reproduce that signal. The corpus is partitioned at the license boundary, so that a deployable and a research model differ only in their training data and the cost of that restriction becomes measurable. Every model is scored against an independent gold benchmark of 6,361 rows over four slices, labeled apart from the teacher and including a slice of harmless prompts that makes over-defense measurable. The distilled students match the teachers on adversarial text within overlapping confidence intervals and reduce false alarms on harmless prompts, the smallest generative student reaching 3.8% against 4.8% for the 8-billion-parameter teacher, while the encoder classifies in roughly 24 ms per request on CPU. Per-class rebalancing is the only decisive ingredient of the recipe. No superiority over the distilled guards is claimed; on the clean reference slice they remain ahead.

1 Introduction

Open guard models provide strong safety classification but are costly on commodity CPUs, motivating a reproducible distillation recipe for smaller CPU-deployable students. The study evaluates auditable training choices, independent performance, and latency rather than claiming state-of-the-art superiority.

  • Current open guards contain 1–9 billion parameters, target GPUs, and take seconds per request on CPUs.
  • Commodity-CPU deployments need comparable-quality safety classifiers with substantially lower per-request latency.
  • The paper distills Llama Guard 3 8B labels into lexical, shallow, encoder, and generative students using a seven-category AILuminate-aligned taxonomy.
  • The evaluation uses an independent gold benchmark and reports distillation gaps under identical conditions, including adversarial performance, over-defense, and teacher comparisons.
  • The paper presents a reproducible resource and benchmark contribution, not a claim that students surpass Llama Guard 3.

2 Related Work

Prior work spans large general-purpose guards, standardized safety resources, compression methods, and narrow CPU detectors. This paper combines those strands into a license-aware fleet of small students positioned between GPU-bound guards and single-purpose detectors.

  • Guard models: Open guard models cover broad hazard taxonomies but use divergent schemes and typically require 1–9 billion parameters.
  • Safety resources: AILuminate and established adversarial, diagnostic, red-teaming, and preference datasets provide public resources for taxonomy alignment, evaluation, and training.
  • Compression and lightweight architectures: Knowledge distillation transfers a strong teacher’s outputs to smaller students, including fast encoder architectures and parameter-efficient generative models.
  • Lightweight and distilled detectors: Narrow detectors such as toxicity and personally identifiable information classifiers are CPU-cheap but cover only part of a multi-hazard policy.
  • Where this work differs: The proposed combination is reproducible, license-aware, independently evaluated, CPU-deployable, and measured for quality–latency trade-offs.
  • Where this work differs: The work is framed as a resource and benchmark contribution rather than a state-of-the-art claim.

3 Taxonomy

The classifier uses seven categories derived from MLCommons AILuminate v1.0’s 12 hazards, grouping or extending hazards to retain training support and consistent benchmarking. Its output is multi-label, while several areas remain outside scope.

  • Taxonomy design: The seven categories C1–C7 derive from the 12-hazard AILuminate taxonomy and expand back to hazard codes for benchmarking.
  • Taxonomy design: Categories merge hazards when data are sparse or detection features overlap, including physical harm, targeted attacks, and information leakage.
  • Taxonomy design: C5 and C6 extend public hazards to include confidential corporate disclosure and broader advice-vectored harms.
  • Output format: Independent sigmoid activations and per-category thresholds produce safe all-zero outputs or unsafe verdicts listing triggered category codes and scores.
  • Scope boundaries: Prompt injection and jailbreak detection, elections and disinformation, and code-interpreter abuse are outside the v0 scope.

4 Data Pipeline

The data pipeline normalizes public sources, partitions rows at license boundaries, prevents train–evaluation leakage, and reserves benign prompts for measuring false alarms. Commercial-only and research recipes then differ only in training data.

  • Normalization: Twenty-five public datasets are normalized into a common six-column schema, with 24 entering the training corpus.
  • License partitioning: Each prompt is assigned to a commercial, noncommercial, or evaluation bucket according to its license and role.
  • License partitioning: The commercial bucket supports deployable models, while the noncommercial bucket is restricted to research.
  • Corpus composition: The corpus composition is reported after normalization and license partitioning.
  • Training recipes: The deployable recipe uses commercial data alone, whereas the research recipe uses commercial and noncommercial data with shared teacher, hyperparameters, and hold-out.
  • Leakage control: Exact normalized-prompt hashes remove training copies that collide with evaluation rows, prioritizing evaluation over training data.
  • Benign evaluation: A fixed 600-prompt benign evaluation set is reserved to measure false alarms because the standard gold benchmarks contain only unsafe prompts.

5 Distillation and Student Fleet

The paper distills labels from a selected guard into students spanning four architectural families, using auditable overrides and class rebalancing to reproduce a seven-category target.

  • Teacher and supervision: The benchmark was excluded from teacher labeling, so reported scores measure agreement with ground truth rather than teacher agreement.This preserves a defined distillation gap under independent evaluation.
  • Student fleet: The students share a seven-dimensional multi-label output, with the all-zeros vector representing a safe verdict rather than a separate safe class.The generative student's text verdict is decoded into the same vector for common scoring.
  • Student fleet: The fleet spans lexical, shallow neural, encoder, and generative architectures, providing models from a term-frequency baseline through a small language model.Table 3 summarizes this common-target student fleet.
  • Training: Neural students use square-root positive-class weighting capped at 12 as their sole rebalancing mechanism.The square root dampens correction, while the cap limits domination by the scarcest category.
  • Deployment form: The generative student was also converted to an 8-bit quantized artifact for CPU-oriented deployment.The adapter was merged into base weights before conversion through llama.cpp.

6 Evaluation Protocol

The evaluation uses one fixed protocol across models: an independently labeled, four-slice gold benchmark, common metrics and threshold, and bootstrap intervals that limit unsupported rankings.

  • Common protocol: Every model is evaluated on the same gold rows, metrics, and decision threshold, preventing model-specific operating-point tuning.Uniform treatment makes comparisons across models and parameter scales interpretable.
  • Evaluation slices: The benchmark is divided into four reproducible slices based on each row's source, including a balanced reference slice and a benign slice.The reference slice supports comparison, while the benign slice measures over-defense.
  • Decision rule: The fixed threshold is 0.5 for every category and model, avoiding threshold tuning that could improve recall while inflating false alarms.The development slice contains no harmless rows, so it cannot expose that over-defense cost.
  • Scoring set: 6,361 scorable rows form the denominator after 441 of 6,802 gold rows are excluded because their labels map to no taxonomy category.The benchmark's gold labels are effectively single-label, with no row carrying two categories.
  • Metrics: Macro-F1 is the primary metric, computed as the unweighted mean of seven per-category F1 scores alongside pooled and binary-blocking views.The binary confusion matrix treats any category firing above threshold as a block, and false-positive rate is measured on 600 harmless rows.
  • Uncertainty: 95% percentile bootstrap intervals are computed from 1,000 fixed-seed resamples, and overlapping intervals preclude asserting an ordering.Macro-F1 variance is especially influenced by sparse categories.
  • Scope boundary: Gold-label mapping fidelity to independent human judgment is unmeasured, so all scores may share systematic mapping error.Such an error would affect teacher and students together without changing their relative ordering.

7 Results

Across the gold benchmark, teachers retain an advantage on the clean headline slice, while students match them on adversarial text and can reduce benign false alarms. The recipe’s decisive ingredient is per-class rebalancing, although no global student superiority is supported.

  • Macro-F1 by slice: 0.556 macro-F1 goes to the 8-billion-parameter teacher on the headline slice, versus 0.529 for the best student, with partially overlapping confidence intervals.The headline slice is the clean reference comparison.
  • Macro-F1 by slice: 0.618 macro-F1 for DistilBERT versus 0.598 for the smaller teacher on the adversarial stress slice falls within overlapping confidence intervals.The defensible interpretation is that the encoder matches the teachers on adversarial text rather than beats them.
  • Over-defense: 0.038 benign false-positive rate for the generative student is below 0.048 for the larger teacher and 0.115 for the smaller teacher.The generative student’s interval, [.023,.053], lies entirely below the smaller teacher’s rate.
  • Per-category behavior: 0.63–0.78 F1 is achieved for physical harm and sexual content, while sensitive information is near 0.40 and harmful advice and illicit acts are 0.29–0.42.The weak-category pattern holds across teachers and students, across architectures and two orders of magnitude in scale.
  • The block-or-pass decision: 0.708 is the generative student’s harmful-prompt block rate, versus 0.736 for its teacher, while the encoder reaches 0.771 at a higher false-positive rate.Unsafe precision exceeds 0.98 for all three models, but every model lets roughly a quarter of harmful prompts through.
  • Per-category behavior: The dominant confusion for every model is a safe prediction on missed harmful prompts, with hate and defamation missed most often, followed by illicit acts.Because gold labels are single-label, the confusion-matrix diagonal represents per-class recall.
  • Ablations: 0.171 macro-F1 is lost when per-class positive weighting is removed, alongside a 0.227 drop in unsafe recall.The apparent 0.077 false-positive-rate improvement reflects defaulting to safe rather than better classification.
  • Ablations: 0.005 headline macro-F1 is gained by adding noncommercial data, while the false-positive rate worsens by 0.027; excluding it therefore costs almost nothing in headline quality.These ablation directions and magnitudes come from a single seed on the cheapest architecture and are not load-bearing.

8 Latency on CPU

The small student fleet runs interactively on commodity CPUs, with latency and quality revealing the encoder as the strongest practical trade-off. Measurements use single-request inference under desktop and constrained two-thread profiles.

  • Measurement setup: Median CPU latency was measured per request on warm models over 120 stratified benchmark prompts under desktop and two-thread profiles.The host was an AMD Ryzen 5 5600X with 16 GB memory, and each configuration used three timed passes.
  • Fleet latency: Every model except the generative student answers in under 28 ms at the median, including under two virtual cores.The generative student reaches roughly 128 ms on the desktop profile and 179 ms when constrained.
  • Thread profiles: Capping small models at two virtual cores changes latency little, while the generative student's latency rises by roughly 40%.Light models are limited more by memory access and framework overhead than arithmetic throughput; only the generative student uses extra cores substantially.
  • Quality–latency trade-off: Figure 1 plots median CPU latency against macro-F1 on the headline slice for both thread profiles.The logarithmic latency axis places runtime and quality on a shared trade-off surface.
  • Quality–latency trade-off: 0.618 adversarial macro-F1 for DistilBERT exceeds 0.568 for the generative student, while the encoder answers roughly five times faster.The generative student's Figure 1 quality is 0.467 because its reported latency uses an 8-bit quantized model, unlike its 0.503 full-precision Table 5 value.

9 Discussion and Limitations

The discussion identifies practical benefits from distillation and license-aware training, while showing that taxonomy design, supervision sparsity, mapping fidelity, and comparison scope constrain interpretation.

  • License boundary: 0.005 headline macro-F1 is gained by adding noncommercial data, while harmless-prompt false-positive rate worsens by 0.027.The commercial-only and expanded recipes are otherwise identical, making the license restriction directly measurable.
  • Over-defense: 0.038 benign false-positive rate for the smallest generative student is below 0.048 and 0.115 for the two teachers.The separation from the smaller teacher lies outside the confidence intervals, and the paper relates this pattern to per-class rebalancing on a corpus dominated by harmless prompts.
  • Taxonomy failure modes: Harmful-advice F1 remains roughly 0.20 with recall near 0.14 for both a 4-million-parameter network and the 8-billion-parameter teacher.The shared collapse is associated with confusion between safe advice, illicit acts, and advice-vectored harm.
  • Taxonomy failure modes: The proposed revision defines harmful advice as the standard's specialized-advice hazard and assigns advice to commit an act to that act's category.The paper also identifies bidirectional confusion between violent and nonviolent crime as a milder boundary problem.
  • Error structure: Most harmful-text errors are silent passes into the safe class rather than wrong-category blocks, making sensitivity the main residual-error lever.The benign slice quantifies the associated over-defense cost.
  • Scope limits: The teacher ceiling is bounded by visible blind spots and a hardware-limited shootout that excluded candidate guards larger than 8 billion parameters.Whether a larger teacher would yield a better student remains untested.
  • Scope limits: No head-to-head comparison with other open guards was possible because their taxonomies do not map cleanly onto the paper's seven categories.The students are therefore compared with their distilled teachers and a lexical floor, not the full open-guard landscape.
  • Supervision limits: The multi-label interface is not a demonstrated capability because unsafe training and gold examples with multiple categories are extremely scarce or absent.Only 0.05% of unsafe commercial labels are multi-category, the noncommercial bucket has none, and the gold benchmark has none; independent category relabeling is required.

10 Conclusion

The paper concludes that an auditable distillation recipe can produce CPU-deployable safety classifiers, with independent evaluation and measurable licensing costs. Its strongest practical gains are low latency and reduced over-defense, not superiority over the teacher guards.

  • Conclusion: A strong open guard labels data from 24 public datasets into seven standard-aligned categories, which small students reproduce and an independent 6,361-row benchmark evaluates.The benchmark spans four slices, including harmless prompts, and is unseen by the teacher.
  • Conclusion: Distillation transfers teacher signal to models 16 to roughly 2,000 times smaller, matching teachers on adversarial text within overlapping confidence intervals.The students do not beat the teachers on that slice.
  • Conclusion: 3.8% harmless-prompt false-positive rate for the smallest generative student is below 4.8% and 11.5% for the two teachers.The confidence intervals sustain this separation.
  • Conclusion: Under 28 ms median CPU latency is achieved by every student except the generative model, making the encoder the quality–latency optimum.Per-class rebalancing was the only decisive recipe ingredient.
  • Conclusion: The license boundary is nearly free and improves over-defense, while harmful-advice quality collapses across model scales because of the taxonomy rather than capacity.The paper presents these as findings that matter beyond its original objectives.
  • Positioning: The work claims reproducibility and benchmark value rather than state-of-the-art performance or superiority over the distilled guards.The guards remain ahead on the clean reference slice.
  • Future directions: Future work calls for taxonomy revision, targeted sparse-category data, independent multi-label relabeling, and licensed multilingual resources.The proposed taxonomy change separates specialized advice from advice associated with a specific act.

A Appendix A: Reproducibility and Licensing

The appendix makes reproducibility and licensing operational: public sources rebuild the corpus and derived results, while source-license buckets determine deployability. The release preserves these boundaries and documents what remains unvalidated.

  • Reproducibility: Every public dataset is rebuilt through per-source normalization scripts, and derived tables are regenerable without rerunning models from materialized predictions.No raw third-party data is redistributed.
  • Licensing: Each row carries a license-derived bucket, with deployable models using only commercial data and noncommercial models reported as research artifacts.The two recipes are kept separate rather than merged into one best number.
  • Licensing: The evaluation bucket is never used for training.Table A1 records source licenses and bucket assignments.
  • Scope: Brazilian-Portuguese resources are restricted, so the paper reports a monolingual boundary rather than a multilingual claim.Those resources are identified as needed for future multilingual evaluation.
  • Release: The labeled corpus is released with the same four splits, while the noncommercial split remains separate to preserve the license boundary.This allows training and evaluation partitions to be inspected without rerunning the pipeline.
  • Release limits: Mapping fidelity was not measured, and no noncommercially trained model is offered beyond research use.The reproducibility claim covers how the mapping was applied, not its correctness against independent human judgment.
Loading 2608.21570v1…