AEGIS: Audio Endogenous Guarding via Internal Signals Against Large Audio-Language Model Jailbreaks

1. INTRODUCTION

• Audio jailbreaks include semantic and multilingual obfuscation, paralinguistic variation, compositional audio, and adversarial waveform perturbations.Existing defenses address different generation stages, but do not locate where safety alignment breaks down when a jailbreak succeeds.Source: §1 · p. 1
• Successful jailbreaks leave risk information increasingly decodable in middle and late layers while refusal probability remains near zero, a discrepancy termed the risk-to-refusal gap.The pattern is reported for Qwen2-Audio and observed similarly across five other models.Source: §1 · p. 1
• AEGIS uses a probe-selected mid-layer risk score to selectively activate later LoRA-based safety adapters, strengthening refusal for high-risk inputs while limiting intervention on benign inputs.The risk gate reads the hidden representation and produces a continuous risk score.Source: §1 · p. 1
• Across six LALMs and three heterogeneous benchmarks, AEGIS reduces the average unsafe rate from 17.9% to 0.4% in-domain, with marginally increased over-refusal on benign inputs.The evaluation also reports transfer of the learned risk signal and intervention to jailbreak distributions excluded from training.Source: §1 · pp. 1–2, Figure 1 · p. 1
• The paper frames its contributions as identifying the gap, introducing selective late-layer intervention, and evaluating safety across heterogeneous benchmarks and held-out settings.It reports limited over-refusal alongside the safety improvements.Source: §1 · p. 2

2. RELATED WORK

• Audio safety studies cover semantic and compositional manipulations, acoustic and paralinguistic variation, and adversarial waveform perturbations, while evaluating harmful behavior and benign-input over-refusal.The cited evaluation sets include AdvBench and JBB-Behaviors for harmful behavior and XSTest for over-refusal.Source: §2 · p. 2
• Existing defenses range from safety prompts and pre-generation guards to safety training and inference-time intervention, but the translation of retained risk into refusal in LALMs remains less understood.The paper investigates this relationship through layer-wise probing and selective safety intervention.Source: §2 · p. 2

3. LAYER-WISE RISK-TO-REFUSAL GAP

• The analysis distinguishes whether a successful jailbreak erases risk information internally or leaves it encoded without triggering refusal.It traces layer-wise risk decodability and refusal tendency to separate these possibilities.Source: §3 · p. 2
• Under undefended successful jailbreaks, risk-related information becomes increasingly decodable in middle and deep layers even as refusal tendency remains near zero.The paper calls this mismatch the risk-to-refusal gap and uses it to motivate selective later-layer intervention.Source: §3 · p. 2

4. AEGIS: DETECT-THEN-INTERVENE DEFENSE

• Table 1 covers unsafe rates across six LALMs and three jailbreak benchmarks, comparing in-domain training with LOBO training that excludes the target benchmark.Parentheses indicate changes from the undefended model.Source: Table 1 · p. 3
• AEGIS keeps the base LALM frozen while jointly training a mid-layer risk gate and late-layer LoRA safety adapters.The gate’s selected layer is chosen separately for each model using layer-wise probing.Source: §4 · pp. 2–3
• The gate maps the final prompt-token representation to a continuous score estimating unsafe-input risk and controlling downstream intervention.The score is produced by an MLP using a sigmoid function.Source: §4 · p. 3
• For inputs above a risk threshold, the gate scales late-layer adapter updates, with intervention strength increasing until a predefined refusal criterion is met.The gate score scales low-rank residual updates, minimizing their effect on low-risk inputs.Source: §4 · p. 3
• Training combines response-token language-model loss with binary cross-entropy risk supervision and an L1 sparsity penalty that discourages unnecessary gate activation.Prompt tokens are masked from the language-model loss.Source: §4 · p. 3

5.1. Experimental Setup

• Evaluation spans six LALMs and three heterogeneous benchmarks, with LOBO holding out one benchmark; defense comparisons use Qwen2-Audio.The benchmarks cover signal-level perturbations, semantic obfuscation, multilingual attacks, and compositional jailbreaks in multi-speaker dialogue.Source: §5.1 · pp. 3–4

5.3. Comparison with Representative Defenses

• AEGIS is the only method below 1% unsafe rate across all three benchmarks; SARSteer remains at 26.9% on SACRED, while AEGIS raises XSTest over-refusal by 5.3 percentage points.The comparison uses Qwen2-Audio under the same evaluation setting.Source: Table 3 · p. 4, §5.3 · p. 4

5.2. Overall Safety, Utility, and Design Effect

• 17.9% to 0.4% average unsafe rate in-domain; LOBO average unsafe rate is 2.6%, with XSTest over-refusal rising 6.3 percentage points on average.The in-domain result averages across 18 model-benchmark pairs; LOBO holds the target benchmark entirely out of training.Source: §5.2 · p. 4
• Full and Always-on increase XSTest over-refusal by 22.1 and 21.9 percentage points, respectively, compared with 5.3 points for AEGIS.Full applies safety LoRA across decoder layers; Always-on removes the risk gate and intervenes on every input.Source: §5.2 · p. 4, Table 2 · p. 4

5.4. Closing the Risk-to-Refusal Gap

• After intervention, late-layer refusal tendency rises sharply, reconnecting decodable risk information with refusal and narrowing the risk-to-refusal gap.The comparison is between AEGIS’s post-intervention tendency and the near-zero refusal tendency without it.Source: §5.4 · p. 4

• The risk-to-refusal gap is the persistence of decodable risk information alongside weak refusal tendency under successful audio jailbreaks.Source: §6 · p. 4
• AEGIS selectively converts retained mid-layer risk signals into late-layer safety interventions and improves safety in in-domain and LOBO settings while limiting over-refusal.Source: §6 · p. 4
• The analysis uses proxy measures, requires access to model internals, and does not cover the full range of audio jailbreaks or adaptive attacks against the defense.Source: §6 · p. 4

• The study uses existing public benchmarks and collects no new human-subject data.Source: §7 · p. 5
• No ethical approval was required.Source: §7 · p. 5