PaperlayerView exact revision on arXiv

Source-linked AI summary

ADTGuard: Adaptive Depth-Time Safeguard for Streaming Inference in Large Language Models

Chuming Wei, Churui Yang, Zihan Zhang, Ruien Li, Wusheng Zhang, Yan Zhang, Guangwen Yang

arXiv:2610.36574v1cs.AI

TL;DR

Streaming moderation can expose users to harmful content before a completed response is checked, while fixed-layer readouts and response-level labels leave prefix risk insufficiently specified. ADTGuard adapts layer selection to each prefix, models safety evidence across depth and time, and refines token targets from response-level supervision. Across evaluated benchmarks, it matches or surpasses state-of-the-art safeguards with 16M trainable parameters and competitive per-token latency.

  • Problem

    Streaming moderation may evaluate a response only after harmful text has reached users, while fixed-layer readouts and response-level labels leave the location of harmful onset unspecified.

    Source: §1 · pp. 1–2

  • Method

    ADTGuard uses a prefix-conditioned Causal Depth Router and Depth-Time Safety Dynamics, with Risk-Onset Target Refinement deriving token-level targets from response-level labels.

    Source: §1 · p. 2

  • Results

    Across evaluated safety benchmarks, ADTGuard matches or surpasses state-of-the-art safeguards with 16M trainable parameters and competitive per-token latency.

    Source: §5 · p. 11

  • Takeaways & Limitations

    ADTGuard provides a lightweight streaming safeguard that checks each token before release while keeping the protected LLM frozen.

    Source: §5 · p. 11

Abstract

from arXiv · show

Large Language Models (LLMs) can generate fluent responses to diverse user requests, yet streaming may expose users to harmful content before generation is complete. Existing guardrails face two limitations in this setting. First, post-hoc moderation at response or chunk boundaries delays intervention, risking harmful exposure when content is released before verification. Second, even with token-level detection, coarse response-level supervision does not specify when harmful content begins, while fixed-layer readouts may miss relevant safety signals at other depths. To address these limitations, we introduce ADTGuard, which checks each token before release, progressively refines token-level training targets, and adapts layer selection to the current prefix. ADTGuard models risk across depth and time through Depth-Time Safety Dynamics (DTSD), which integrates safety evidence across layers at each token and uses it to guide risk accumulation over successive tokens. To adapt the choice of observation layers to the current prefix, we propose the Causal Depth Router (CDR). With only response-level labels available, we further develop Risk-Onset Target Refinement (RTR) to progressively construct token-level training targets for both components. Through extensive experiments across multiple safety benchmarks, we demonstrate that ADTGuard consistently achieves strong streaming detection performance, matching or surpassing state-of-the-art safeguards while requiring only 16M trainable parameters and maintaining competitive per-token latency.

1 Introduction

Streaming safety requires risk assessment before harmful text is exposed, but fixed-layer readouts and response-level labels leave important aspects of prefix risk unaddressed. ADTGuard combines prefix-aware layer selection, depth-time risk modeling, and progressively refined token targets.

1 Introduction

  • Post-hoc response checks can occur after unsafe text reaches users, motivating safeguards that assess risk as generation unfolds.The introduction frames timely intervention before harmful content is exposed as a requirement for streaming safeguards.Source: §1 · p. 1
  • Token-level detectors may still rely on fixed layers, despite safety information varying across model depth.This motivates adapting where internal safety evidence is read as the response prefix evolves.Source: §1 · p. 2
  • Response-level labels do not identify harmful onset, while broadcasting labels or imposing shared temporal priors can misrepresent prefix risk.Manual token annotations are costly and difficult to scale.Source: §1 · p. 2
  • ADTGuard uses a prefix-conditioned Causal Depth Router to select model layers and Depth-Time Safety Dynamics to integrate evidence across depth and response tokens.The underlying LLM remains frozen; the routed representations are modeled with selective state-space modeling (Gu & Dao, 2024).Source: §1 · p. 2
  • Risk-Onset Target Refinement progressively derives token-level targets from response-level labels without requiring token-onset annotations.It uses cached detached risk trajectories and response-equal loss aggregation.Source: §1 · p. 2
  • Across multiple safety benchmarks, ADTGuard reports strong streaming detection while retaining a lightweight trainable footprint and competitive per-token latency.The introduction compares it with representative response-level, representation-based, and streaming safeguards.Source: §1 · p. 2

2 Related Work

Prior work develops streaming detection, adaptive depth processing, and token-level supervision, but the cited approaches leave open how to adapt safety readouts to prefixes and derive temporally differentiated targets directly from response labels.

2.1 Streaming Safety Detection Based on Internal Representations

  • Representation-based safety monitors use internal signals at token level, but their observation schemes are typically fixed rather than adapted to the evolving response prefix.Prior work also identifies variation in safety information across model depth.Source: §2.1 · p. 3

2.2 Adaptive Depth and Sequential Modeling

  • Adaptive-depth methods vary computation or readouts, while safety methods model depth or temporal risk; the related approaches generally do not jointly adapt safety readout and temporal risk modeling.The cited adaptive-depth methods target general inference or classification, and prior safety methods typically treat depth and time separately.Source: §2.2 · p. 3

2.3 Training Strategies for Token-Level Safety Detection

  • Existing token-level supervision strategies use temporal constraints, annotations, or rollout procedures, leaving open how to derive differentiated token targets directly from response-level labels.The cited approaches include temporal aggregation, class prototypes, sentence-level judgments, and progressively refined targets.Source: §2.3 · p. 3

3 Methodology

ADTGuard turns read-only hidden states from a frozen LLM into causal prefix-risk predictions. Its method combines prefix-aware layer routing, depth-time dynamics, refined token targets, and response-balanced training with cached streaming inference.

3 Methodology

  • ADTGuard evaluates each generated response prefix using read-only hidden states from a frozen LLM.The pipeline addresses fixed-layer observation and unspecified token-level risk through layer selection, depth-time modeling, and target refinement.Source: §3 · p. 4

3.1 Causal Depth Router

  • CDR initializes prompt context once, then uses causal Router states to score model layers and select a normalized Top-K set for each response position.The backbone still executes all layers; CDR determines which K hidden states the safeguard reads.Source: §3.1 · pp. 4–5

3.2 Depth-Time Safety Dynamics

  • DTSD uses DSD to integrate routed evidence in depth order and TSD to track risk across response positions, with depth evidence conditioning temporal evolution.Its selective state-space branches model depth and time; the depth branch processes selected layers in order and initializes anew for each response token.Source: §3.2 · pp. 5–6

3.3 Risk-Onset Target Refinement

  • RTR derives progressively refined token targets from detached risk trajectories cached from the preceding epoch, without a separate teacher or additional inference pass.Its target is the weighted forward-KL optimum between response-level supervision and a fixed model-derived onset profile.Source: §3.3 · p. 6

3.4 Training Objective and Streaming Inference

  • Training averages binary cross-entropy within each response and across the batch for equal response weighting, and penalizes decreases across adjacent valid-token risks.Streaming inference reuses the LLM and Router KV caches and recurrent TSD state; each prediction depends only on the prompt and available response prefix.Source: §3.4 · p. 7

4 Experiments

Experiments compare ADTGuard with representation-based and dedicated safeguards across safety benchmarks, then assess component contributions, latency, and hyperparameter choices. Results show strong detection performance, with measured token-level overhead and sensitivity analyses identifying practical settings.

4.1 Experimental Setup

  • The evaluation used five safety benchmarks and compared dedicated moderation models with representation-based safeguards on shared backbones.Unsafe-class F1 was reported as Response F1 at the final response token and Streaming F1 when any evaluated prefix triggered detection.Source: §4.1 · pp. 7–8

4.2 Main Results

  • ADTGuard achieved the best Streaming F1 in 14 of 15 same-backbone settings and the best Response F1 in 14 of 15; it led dedicated guards on four of five datasets.On Qwen3-8B, Streaming F1 was 93.57 on SGB-S-Eval and 85.87 on SGB-WildGuard, gains of 5.24 and 25.75 points over the strongest representation-based baselines; the largest dedicated-guard gain was 5.05 points on SGB-WildGuard.Source: §4.2 · p. 8, Table 1 · p. 8, Table 2 · p. 9

4.3 Ablation Study

  • Adding CDR with DTSD raised Response F1 from 88.06 to 92.83 and Streaming F1 from 86.09 to 87.51; RTR then raised Streaming F1 to 93.58.Mono slightly raised Response F1 to 93.67 while leaving Streaming F1 essentially unchanged.Source: §4.3 · p. 9, Table 3 · p. 9

4.4 Latency Evaluation

  • With a 1,000-token prompt and 1,024 output tokens, ADTGuard added 2.33 ms per output token, increasing request time from 19.32 s to 21.71 s (+12.36%).Across prompt lengths of 128–2,048 tokens, added cost remained 2.31–2.35 ms/token.Source: §4.4 · p. 10

4.5 Hyperparameter Analysis

  • Routing budget k = 4 gave the best overall performance among k = 1–9; larger budgets added inference cost without further gains.The study used SGB-S-Eval with Qwen3-8B Base.Source: §4.5 · p. 10, Table 4 · p. 10
  • Across three gradual RTR schedules, Response F1 ranged from 93.51 to 93.67 and Streaming F1 from 93.41 to 93.58; hard response-level targets yielded Streaming F1 of 87.68.Immediately switching to model-derived targets yielded Streaming F1 of 93.10.Source: Table 4 · p. 10, §4.5 · p. 11

5 Conclusion

ADTGuard is a lightweight safeguard that checks each token before release while keeping the protected LLM frozen. It combines adaptive internal observation with progressively refined token-level supervision, and performs strongly across evaluated benchmarks with competitive latency.

5 Conclusion

  • ADTGuard checks each token before release while keeping the protected LLM frozen.The system jointly adapts internal evidence used for detection and supervision used to learn token-level risk.Source: §5 · p. 11
  • Across evaluated safety benchmarks, ADTGuard matches or surpasses state-of-the-art safeguards with 16M trainable parameters and competitive per-token latency.The method uses response-level labels without requiring additional token annotations or rollout generation.Source: §5 · p. 11
  • CDR selects observation layers conditioned on the evolving prefix, while DTSD integrates depth-wise evidence to guide temporal risk accumulation.RTR progressively constructs token-level training targets from predicted risk trajectories under response-level supervision.Source: §5 · p. 11

A.1 Dataset Descriptions

The appendix describes five safety benchmarks with response-level safety labels, distinguishing regenerated benchmark responses from datasets using their original responses. It also reports dataset split statistics after label filtering.

A.1 Dataset Descriptions

  • SGB-S-Eval uses deterministic Qwen3-8B responses to S-Eval prompts and contains 9,000 training and 1,000 test examples.Its response-level safety labels follow the S-Eval creators’ evaluation procedure.Source: §A.1 · p. 16
  • SGB-WildGuard regenerates Qwen3-8B responses to WildGuard prompts and annotates response safety with Kimi-K2, with 37,934 training and 1,725 test examples.Unlike public WildGuard, this benchmark does not reuse the original responses.Source: §A.1 · p. 16
  • AEGIS2.0 covers diverse benign, harmful, and adversarial interactions, while BeaverTails-30k provides about 30K question–answer pairs across 14 potentially harmful-content categories.Both are used with binary response-safety labels.Source: §A.1 · p. 16
  • Public WildGuard uses original released responses; after excluding 16 test examples without valid response-safety labels, 1,709 test examples remain.Safe and unsafe counts are reported separately for train and test splits after dataset-level label filtering.Source: §A.1 · p. 16, Table 5 · p. 17, §A.2 · p. 17

A.2 Data Statistics and Splits

  • Table 5 summarizes dataset sizes and class distributions after dataset-level label filtering, with train–test partitions fixed across backbone settings.Source: §A.2 · p. 17, Table 5 · p. 17

B Implementation Details

The implementation appendix supplements the experimental setup with implementation settings and protocol exceptions. Its configuration table lists additional architecture and training parameters, while the routing budget, refinement schedule, and default optimization setup follow the main text.

B Implementation Details

  • The implementation section supplements Section 4.1 with implementation settings and protocol exceptions for the results in Tables 1 and 2.Source: B Implementation Details · p. 17

B.1 ADTGuard Configuration

  • The routing budget, refinement schedule, and default optimization setup are specified in the main text rather than repeated in the appendix passage.Source: §B.1 · p. 17
  • Table 6 lists additional ADTGuard architecture and training parameters.Source: Table 6 · p. 17, §B.1 · p. 17

B.2 Baseline Implementations

The section specifies how baseline safeguards are adapted and trained for comparison, including representation-based detectors and dedicated moderation models.

Training configurations

  • The baseline training configurations are summarized in Table 7, where paired SIREN values correspond to its linear-probe and MLP stages.Kelp uses a global batch size of 32; its runner processes full formatted inputs within the backbone context limit rather than applying a fixed 4,096-token truncation.Source: Table 7 · p. 18, §B.2 · p. 18

Dedicated moderation baselines

  • The dedicated generative moderation baselines produce safe/unsafe verdicts from native prompt–response templates, while Qwen3Guard-Stream retains its pretrained risk head and alarms on Unsafe or Controversial predictions.SentGuard instead uses an instruction-based answer format and checks cumulative prefixes every ten SGB response tokens or cumulative sentence prefixes on public datasets.Source: §B.2 · p. 18

Kelp

  • Kelp uses a separately trained detection head for each backbone–benchmark pair, with the backbone frozen and temporal regularization retained.The reported hidden-state entries are 32, 32, and 25 for Qwen3-8B, Llama-3.1-8B, and Gemma-7B, respectively.Source: §B.2 · pp. 18–19

SIREN

  • SIREN selects neurons by probe-weight magnitude until cumulative importance reaches 90%, then weights and concatenates features across layers for its MLP.Layerwise probes use validation macro-F1; the MLP is selected through 32 Optuna trials with five-fold cross-validation.Source: §B.2 · p. 19
  • SIREN’s in-domain variant trains separate detectors per backbone–benchmark pair, while its official-data variant trains per backbone on a seven-source pool and evaluates across five benchmarks without benchmark-specific retraining.Both variants use probe-derived neuron selection, layer weights, and an MLP; the official-data label describes the data recipe, not an unmodified released classifier.Source: §B.2 · p. 19

B.3 Evaluation Protocols

The section defines evaluation protocols for response-level and streaming detection, then presents training-mode evidence of adaptive layer routing and progressively refined token targets. These visualizations illustrate learned patterns, not ground-truth harmful-onset recovery.

B.3 Evaluation Protocols

  • Streaming F1 counts a response unsafe if any evaluated token or prefix triggers an alarm, whereas Response F1 uses the final response decision; this metric does not measure how early intervention occurs.ADTGuard uses the same Response and Streaming criteria as Kelp for direct comparison; SIREN’s prefix-based adapter preserves its mean-pooled representation and is not its original evaluation procedure.Source: §B.3 · pp. 19–20
  • Offline Streaming F1 replays stored responses token by token, while online deployment withholds each new token until its check passes and terminates generation at the first alarm.The triggering token is withheld, but the reported offline metric only records whether risk was detected during a response.Source: §B.3 · p. 20

C.1 Layer Routing

  • In the recorded epoch-5 run, layers 20–22 supply 73.45% of selected readout slots, while layer selections change between adjacent positions at average rates of 51.26% for safe and 40.51% for unsafe responses.The unsafe example exhibits 18 distinct Top-4 combinations over 376 response tokens; these training-mode traces include dropout and parameter updates, so they do not isolate the causes of routing changes.Source: §C.1 · p. 21, Figure 2 · p. 22

C.2 Target Refinement

  • RTR lowers the mean target over one example’s opening quarter from 1.000 in epoch one to 0.064 in epoch five, while all 825 logged safe-example trajectories retain zero targets.Targets incorporate preceding-epoch risk trajectories rather than a predefined harmful-onset boundary; the visualization demonstrates learned temporal differentiation, not recovery of an independently annotated onset.Source: §C.2 · p. 21

C.3 Latency Evaluation

The latency evaluation measures end-to-end overhead with every token check active under a fixed Qwen3-8B guard configuration. Under the tested settings, monitoring adds approximately 2.3 ms per generated token.

Experimental setup

  • Measurements use a fixed fifth-epoch Qwen3-8B Base guard with routing budget k = 4, batch size 1, and matched base/guard runs on each GPU.The four independent A800 GPU measurements use monitoring-only mode, so alarms do not change generated sequences; latency includes risk evaluation and decision overhead rather than isolated guard execution time.Source: §C.3 · pp. 22–24

Latency results

  • 2.3320 ms/token is the added latency for a 1,000-token prompt and 1,024 output tokens; its 95% confidence interval is [2.3235, 2.3410] ms/token.In that setting, total request time rises from 19.3229 s to 21.7108 s, a 12.36% overhead.Source: §C.3 · p. 24
  • 2.3103–2.3477 ms/token is the added cost across prompt lengths of 128–2,048 tokens at a fixed 1,024-token output length.The evaluation reports a stable amortized monitoring cost across the evaluated sequence lengths.Source: §C.3 · p. 24
  • In the primary 1,000-input/1,024-output setting, first-token time rises from 103.549 to 127.183 ms and TPOT rises from 18.787 to 21.098 ms.Measurements keep token-level checks active at every decoding step, including the final-token check.Source: §C.3 · p. 24

C.4 Case Study

Two selected Qwen3-8B examples illustrate token-level monitoring preserving a benign sensitive-topic discussion and interrupting further euphemistic ridicule. The appendix also establishes an optimization interpretation for RTR’s target interpolation.

C.4 Case Study

  • The two case-study examples illustrate behavior but do not estimate its prevalence, and the recorded risk scores assess prefixes rather than isolated tokens.The reference lengths and maximum risks come from monitoring without termination, while released lengths come from separate intervention runs.Source: §C.4 · pp. 24, 27, Table 9 · p. 25
  • The benign discussion example receives a maximum risk score of 0.0036, and all 424 tokens are released unchanged despite sensitive vocabulary.The two cases were purposively selected to illustrate successful behavior, not estimate its prevalence.Source: §C.4 · p. 24
  • For the euphemistic-ridicule response, the first alarm occurs at token 105, so intervention releases tokens 1–104 and withholds the remaining 132 reference tokens.The first four suggestions were already released; token 105 is a Markdown delimiter, not an annotated harmful onset, so the case shows curtailed continuation rather than zero harmful exposure or onset identification.Source: Figure 5 · p. 27, §C.4 · p. 28

D Theoretical Proof

  • For a fixed model-derived onset profile, RTR’s refined target is exactly the minimizer of a weighted forward-KL objective combining the response-level label and model-derived token-level supervision.This is the stated theoretical justification for RTR’s target interpolation.Source: D Theoretical Proof · p. 28

D.1 Proof of the Forward-KL Characterization

The proof characterizes RTR’s refined target as the unique forward-KL optimum, conditioned on a fixed model-derived onset profile. It does not require ground-truth token-level harmful-onset annotations.

D.1 Proof of the Forward-KL Characterization

  • The onset profile is constructed from the detached risk trajectory cached during epoch e −1 and treated as fixed supervision during epoch e.For benign responses, RTR uses a fixed zero target.Source: §D.1 · p. 28
  • For 0 < α < 1, the objective is strictly convex and uniquely minimized at q⋆ = α.The boundary cases α = 0 and α = 1 have unique minima at q = 0 and q = 1, respectively.Source: §D.1 · p. 29
  • Conditioned on a fixed model-derived onset profile, RTR’s refined target is the unique forward-KL optimum between the response-level label and historical token-level supervision.The characterization does not assume access to ground-truth token-level harmful-onset annotations.Source: §D.1 · p. 29

Related papers

Loading 2610.36574v1…