PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Undertow: How Ordinary Typos Break Probes

Elad David, Max Fomin, Amit LeVi

arXiv:2609.15994v1cs.CLcs.LG

TL;DR

Activation-based probes can be brittle to ordinary typing noise even when user intent and surface behavior are preserved. The paper characterizes the localized rotation-and-decay geometry and evaluates aggregation and a KV-cache-forked downstream readout. The fork closes 95% of the single-position probe gap, leaving a −0.6pp residual, while the activation geometry transfers qualitatively across three model families.

  • Problem

    Ordinary typing variation largely preserves user intent and behavior, but its effect on the hidden states read by activation-based probes is insufficiently understood.

  • Method

    The paper characterizes activation rotation and spatial decay, then evaluates multi-position aggregation and a KV-cache-forked suffix that reads downstream of perturbations.

  • Results

    95%: the KV-cache fork reduces the −12pp single-position fragility to a −0.6pp residual, outperforming the −3.7pp augmentation-training residual.

  • Takeaways & Limitations

    Probe robustness can be restored by relocating a single-position readout downstream, exploiting the perturbation’s rapid spatial decay.

  • Takeaways & Limitations

    Activation-level effects replicate across three model families, but the multi-architecture probe sweep is conducted on a single model and all experiments use one layer per model.

Abstract

from arXiv · show

LLMs handle ordinary typing variation fluently: a typo or missing punctuation leaves both user intent and the model's response substantively unchanged. Yet probes that detect malicious prompts by reading the model's hidden states tell a different story: the same edit rotates the readout vector by 43--56 at the perturbed token, decaying below 15% within ~10 downstream tokens. Stacking ~3 common typos per message cuts a single-position prompt-injection probe's TPR@FPR$=1% by 12.0pp, a gap recalibration alone cannot close. Multi-position aggregation cures localized perturbations (<= 0.5 loss) but only attenuates distributed ones, where even attention- and max-based aggregators still drop ~3.8pp. For single-position probes, we introduce a KV-cache fork: a short fixed suffix appended after the user message lets the probe read a few tokens downstream of the perturbation, exploiting its rapid spatial decay. This closes 95% of the gap (-0.6pp residual) -- an order of magnitude better than perturbation-augmented training (-3.7pp). The rotation-and-decay geometry replicates on Llama-3.1-8B, Qwen3-8B, and Gemma-4-E4B; probe evaluation is on Llama-3.1-8B. Code: https://github.com/eladd-ai/latent-undertow

1. Introduction

The paper shows that ordinary typing noise can sharply disrupt hidden-state prompt-injection probes despite preserving user intent and model behavior. It characterizes this localized activation shift and evaluates aggregation and downstream-readout remedies.

  • 1. Introduction: 43–56°: ordinary typing edits cause a large localized activation rotation that decays rapidly downstream.The perturbation is sharp at the edited token but spatially local.
  • 1. Introduction: 12.0pp: stacking common typing errors lowers single-position probe TPR@FPR=1%, and threshold recalibration alone cannot recover it.The stress test combines about three ordinary character edits per sample.
  • 1. Introduction: Multi-position aggregation loses ≤0.5pp on localized perturbations but still drops approximately 3.8pp under distributed noise.Attention- and max-based aggregators attenuate rather than eliminate distributed-perturbation damage.
  • 1. Introduction: The KV-cache-forked suffix closes 95% of single-position fragility, leaving a −0.6pp residual versus −3.7pp for perturbation-augmented training.The suffix places the probe readout downstream, where the activation shift has decayed.

2. Background and Related Work

Prior work established activation probes for monitoring model internals and prompt-injection risk, while robustness research focused more on outputs or targeted perturbations. This paper extends the question to ordinary, non-adversarial typing variation and introduces a post-user mechanism for the activation pipeline.

  • 2. Background and Related Work: Activation probes recover task-relevant internal properties and can detect unsafe intent and prompt injections at inference time.The related work frames probing as a practical monitoring tool.
  • 2. Background and Related Work: Multi-position pooling architectures provide the evaluation taxonomy, while post-user prompt engineering motivates a user-transparent KV-cache-forked defense.The defense transfers a post-user mechanism from behavioral robustness to activation robustness.
  • 2. Background and Related Work: Output-level studies report sensitivity to surface variation, even though non-adversarial typing noise largely preserves the LLM’s reading of user intent.This contrast motivates asking what activation-based probes actually read out.
  • 2. Background and Related Work: Earlier probe-robustness work used targeted character perturbations on harmful keywords, whereas this paper studies ordinary, non-targeted typing variation.The extension removes adversarial intent from the perturbation source.

3. Framework

The framework studies intent-preserving character edits across localized and distributed spatial regimes, separating probe fragility from changes in the model’s interpretation or response. It defines perturbation families and tests how their structure affects activations and behavior.

  • 3.1. Definitions: Intent preservation means a human interprets clean and perturbed inputs as expressing the same intent, whereas behavior preservation means the LLM’s response is unchanged.The perturbations are intent-preserving by construction, but heavy perturbation does not guarantee behavioral preservation.
  • 3. Framework: The framework distinguishes localized perturbations clustered near the readout from distributed perturbations spread across the input.This spatial taxonomy organizes the later robustness comparisons.
  • 3.2. Perturbation Taxonomy: The perturbation families include adjacent-key typos, punctuation changes, capitalization edits, omissions, and a stacked-typing bundle.The bundle is the localized stress test; distributed stress uses every-second-word adjacent-key typos affecting approximately 50% of content tokens.
  • 3.2. Perturbation Taxonomy: Adjacent-key typos enable spatial-decay analysis across arbitrary positions, while terminal punctuation substitutions are end-constrained and can produce larger rotations.The study uses the former for mechanistic isolation and the latter as a severe single-perturbation case.
  • 3.3. Behavioral Invariance Premise: The behavioral-invariance premise is supported by broad evaluation: intent preservation holds in 94.6% of pairs, with functionally unchanged responses in the localized regime.Under distributed noise, intent remains preserved but a malicious example bypasses safety alignment, showing a separate alignment brittleness.

4. Experimental Setup

Experiments evaluate malicious-versus-benign user-intent probes across three instruction-tuned model families, with activation characterization spanning multiple depth checkpoints. Probe-level quantitative evaluation is conducted on Llama-3.1-8B using clean training and complementary in-distribution and out-of-distribution protocols.

  • 4. Experimental Setup: The study uses Llama-3.1-8B-Instruct, Qwen3-8B, and Gemma-4-E4B to test whether the activation mechanism transfers across model families.All three reproduce the large on-site rotation and rapid spatial decay qualitatively, although magnitudes differ.
  • 4. Experimental Setup: 43–56°: activation characterization samples four depth checkpoints per model across 100 OpenOrca prompts with adjacent-key typos at up to four positions.The effect sizes are large relative to per-prompt variance.
  • 4. Experimental Setup: Probe-level multi-architecture quantitative outcomes are limited to Llama-3.1-8B, while cross-model evidence primarily establishes qualitative mechanism transfer.The experiments use one layer per model, with Llama-3.1-8B layer 31 as the probe readout.
  • 4. Experimental Setup: Probe classification separates benign requests from malicious inputs including prompt injections, jailbreaks, unsafe requests, and scams.The probe monitors the user turn before the model responds.
  • 4. Experimental Setup: The corpus combines 29 public datasets into a tractable training set of N=168,440 while preventing any source from dominating.Benign domains and five malicious attack families are represented.
  • 4. Experimental Setup: Clean AUC is 99.80% ± 0.01% under 5-fold stratified cross-validation, complemented by Leave-One-Dataset-Out evaluation for distribution shift.Unless a defense changes the procedure, probes train on clean user-EOT activations and face perturbations at inference time.

5. Perturbation Effects on LLM Activations

Ordinary typos cause large, primarily directional activation rotations at the affected token, but their influence decays rapidly downstream. The resulting spatial locality varies with perturbation placement and interaction, exposing single-position readouts to localized failures.

  • 5.1. On-Site Impact: 43°–56° on-site rotations occur across models and layers, with approximately unchanged activation norms and no systematic depth trend.Measurements use adjacent-key typos that preserve BPE token count, isolating a single-token swap rather than word re-segmentation.
  • 5.2. Spatial Structure of the Perturbation Effect: Below 15% of the on-site angular signal remains by approximately 10 downstream tokens, with the decay replicated in Qwen3-8B and Gemma-4-E4B.The profile is consistent across typo positions and continues to decay slowly afterward.
  • 5.2. Spatial Structure of the Perturbation Effect: Spatial locality makes single-position readouts vulnerable, while multi-position aggregation dilutes the localized perturbation signal.The activation effect is concentrated near the typo and rapidly attenuated downstream.
  • 5.2. Spatial Structure of the Perturbation Effect: At d=10, mid-sequence typos produce approximately 3°–5° EOT angles, whereas terminal substitutions bypass attenuation and can reach 26.4°.The 26.4° question→/ case equals 76% of the between-prompt baseline and is identified as the most operationally dangerous single perturbation tested.
  • 5.2. Spatial Structure of the Perturbation Effect: Relative decay is similar despite roughly 2.4× different on-site magnitudes for terminal punctuation substitutions.The question→. and question→/ perturbations have on-site magnitudes of 24.7° and 59.9°, respectively, while their relative-decay curves overlap closely.
  • 5.3. Perturbation Type Survey: Two closely spaced typos produce an approximately 1.6× elevated combined signal, while wider spacing delays overlap until the second typo’s footprint is reached.The combined signal is compared with the single-typo-A baseline across inter-typo distances of approximately 2, 15, and 25 tokens.

6. Consequence for Activation-Based Probes

Under ordinary stacked typing errors, single-position activation probes lose substantial detection performance, while multi-position readouts are robust to localized but not uniformly distributed perturbations. The results indicate that readout location and aggregation strategy, rather than classifier capacity alone, determine robustness.

  • 6.1. Single-Position Probe Fragility: 3.33 character edits per sample: the stacked-typing bundle combines punctuation, adjacent-key, and capitalization errors when their structural prerequisites are met.Coverage is 99.996%, with 1–5 edits per sample; the specific combination is not claimed to match a measured user distribution.
  • 6.1. Single-Position Probe Fragility: 12.0 percentage points: TPR@FPR=1% falls from 97.4% to 85.4% under stacked typing, and threshold recalibration cannot restore the clean operating point.The AUC drop is only −0.55pp, but class-asymmetric score shifts push malicious samples below the fixed-FPR threshold more readily.
  • 6.2. Multi-Architecture Comparison: ≤0.5pp loss: every full-sequence multi-position probe is nearly immune to the localized stacked-typing bundle, versus ∼12pp for the single-position probe.Per-architecture score-shift quantiles also show reduced per-sample shift magnitude, not merely preserved ranking.
  • 6.1. Single-Position Probe Fragility: 12.5±0.8pp TPR@FPR=1% loss: an MLP at the same user-EOT position matches Linear’s 12.0 ± 0.5pp drop, so higher capacity adds no robustness.This isolates readout location, rather than classifier capacity, as the relevant variable.
  • 6.2. Multi-Architecture Comparison: −3.8 and −3.9pp: Attention and MultiMax remain relatively robust under distributed typos and outperform Linear’s −6.9pp drop, while full-sequence MLP falls −16.8pp.The distributed stress test corrupts approximately 50% of content tokens, unlike the localized stacked-typing bundle.
  • 6.2. Multi-Architecture Comparison: Localized perturbations corrupt one position and are amortized by aggregation, whereas distributed perturbations require aggregation mechanisms that can reweight affected tokens.Attention and max selection can reweight distributed corruption, while a uniform whole-sequence average cannot.

7. Practical Defenses

The defenses modify either probe architecture, training data, or readout location to reduce typing-induced fragility. The KV-cache fork is strongest for the single-position probe, recovering nearly all lost TPR while preserving the architecture.

  • 7.2. Perturbation-Augmented Training: 93.84 ± 0.5% TPR@FPR=1% with augmentation recovers 8.4 of the 12.0pp loss, or approximately 70% of the fragility.Clean-test performance changes by only +0.14pp TPR@FPR=1%.
  • 7.3. KV-Cache Forked Probe Suffix: The KV-cache fork appends an approximately 30-token suffix so the probe reads after perturbation effects have decayed, then reverts the cache for normal generation.This separates the probe path from the original user-EOT generation path.
  • 7.3. KV-Cache Forked Probe Suffix: Post-user role separation preserves the user-turn boundary and produces roughly half the residual angular shift of an inline suffix at the same physical distance.The evaluated role placement is a system block on Llama-3.1-8B and a CoT segment for models with explicit reasoning modes.
  • 7.3. KV-Cache Forked Probe Suffix: 98.65 ± 0.02% TPR@FPR=1% with the KV-cache fork leaves a −0.60 ± 0.06pp drop from clean and reduces baseline fragility by 95%.The fork uses the same single-position architecture but reads at the end of an appended suffix.
  • 7.3. KV-Cache Forked Probe Suffix: The fork’s −0.6pp residual is smaller than augmentation training’s −3.7pp residual under the same architecture and training corpus.Augmentation depends on anticipating the deployment perturbation distribution, whereas the fork chooses a fixed suffix.
  • 7.3. KV-Cache Forked Probe Suffix: The suffix adds 30 tokens, with less than 4 ms extra forward-pass cost at batch size 1 and 71–135 ms at batch size 32.KV-cache memory overhead is a few MB per request and independent of baseline length.

8. Discussion

Ordinary typing can preserve model behavior while substantially moving the representation read by safety probes. The paper argues that rapid spatial decay makes this brittleness repairable, while the evaluation and scope leave important boundaries.

  • 8. Discussion: 43–56° rotations occur across three model families without altering user-intent interpretation, while a single-position probe loses 12pp TPR at FPR=1%.The result is a mismatch between fluent surface behavior and probe-sensitive internal representations.
  • 8. Discussion: The KV-cache fork closes 95% of the probe-accuracy gap by reading after the localized activation movement has decayed.The defense restores probe accuracy under the behavior–representation asymmetry rather than removing the underlying asymmetry.
  • 8. Discussion: The same typing patterns could serve as low-cost attack vectors because distributed corruption can bypass both the probe and alignment without optimization, model access, or adversarial prompt engineering.This implication is stated for deliberate use of the studied typing patterns.
  • 8. Discussion: The multi-architecture probe sweep uses one model and one layer per model, while tokenization and attention-dilution contributions are not decomposed in probe-level results.Further role variants and learned-suffix designs are also deferred.
  • 8. Discussion: Activation-level rotation and decay replicate qualitatively across Llama-3.1-8B, Qwen3-8B, and Gemma-4-E4B, with Qwen3-specific intermediate-layer bumps.Gemma’s signal falls below 15% by approximately 10 tokens at the deepest layer, while the Qwen3 bump mechanism is left for future work.

D. Per-Architecture Score-Shift Distributions

Per-sample score shifts are most severe for the single-position linear probe, especially on malicious examples. Aggregation sharply reduces these shifts across architectures.

  • D. Per-Architecture Score-Shift Distributions: The single-position linear probe has heavy-tailed malicious score shifts, with a 5th percentile of −0.49 and median approximately 0.The distribution is summarized under the stacked-typing bundle.
  • D. Per-Architecture Score-Shift Distributions: MLP, Attention, and MultiMax aggregation produce near-zero score shifts at every reported percentile.These architectures reduce per-sample shift magnitude dramatically relative to the linear probe.

E. Perturbation-Augmentation Training Details

Augmentation training re-extracts activations from independently perturbed samples so a single-position probe can generalize beyond one known typo combination. The protocol uses controlled perturbation families and fixed cross-validation settings.

  • E. Perturbation-Augmentation Training Details: Each training sample independently activates near-end perturbation families with probability 0.5, while guaranteeing at least one perturbation.Samples receive one to four perturbations applied individually across folds of approximately 134,750 training examples.
  • E. Perturbation-Augmentation Training Details: The augmentation policy covers last-word adjacent-key typos, trailing-period toggles, early capitalization shifts, and terminal punctuation substitutions.The terminal punctuation family samples among ? →., ? →/, and . →/.
  • E. Perturbation-Augmentation Training Details: Evaluation reports per-perturbation TPR@FPR=1% using 5-fold cross-validation, with the full bundle as the final compound perturbation.The probe experiments use Llama-3.1-8B layer 31 and the stated Adam, class-weighting, and seed-42 settings.
  • E. Perturbation-Augmentation Training Details: The probe corpus contains 168,440 samples drawn from 29 datasets, with caps used to keep training tractable and prevent one dataset from dominating.Layer 31 is adopted as a principled final-layer default rather than as a universal performance optimum.

H.1. Role-Scoping Ablation

The role-scoping ablation tests whether placing the fixed suffix in a system block or extending the user message changes perturbation attenuation. System-role placement produces substantially lower residual angular shifts, while the suffix adds bounded KV-cache overhead.

  • Role comparison: System-role suffixes yield roughly half the residual angular shift of user-role suffixes at every measured layer for both perturbations.At layer 31, paired differences are −9.03° for adjacent-key typos and −8.77° for the bundle.
  • Experimental setup: The ablation compares no suffix, a user-role suffix, and a system-role suffix across two perturbations, four layers, and 100 OpenOrca prompts.The metric is angular shift between clean and perturbed activations at the end-of-suffix readout position.
  • Compute overhead: The KV-fork suffix adds 3.75 MB of KV-cache memory per request, independent of baseline prompt length.This analytic marginal is based on a 30-token suffix and scales with layer count, KV-head count, and head dimension across architectures.
  • Compute overhead: At batch size 32, the suffix adds 71–135 ms across the tested prompt-length buckets, while its marginal memory remains bounded per request.Timings use Llama-3.1-8B-Instruct on an A100 with a 30-token neutral suffix.

I. Leave-One-Dataset-Out Evaluation

Leave-One-Dataset-Out evaluation tests whether probe performance transfers to unseen datasets without relying on dataset-specific calibration. It reveals substantial distribution sensitivity, while retraining a linear probe behind the KV-fork suffix improves clean accuracy and preserves a competitive position among architectures.

  • Protocol: LODO holds out each dataset by retraining on 28 datasets and evaluating on the remaining one, isolating transfer beyond dataset-specific surface patterns.The evaluation uses fixed threshold 0.5 for deployment-realistic weighted accuracy rather than per-domain threshold tuning.
  • Evaluation scope: Only 7 of 29 held-out folds contain both classes and admit AUC, while the remaining 22 are single-class folds.The analysis therefore reports weighted accuracy across all available folds and AUC only on mixed-class folds.
  • Clean LODO results: Best mean per-fold AUC is 0.918 for Attention, below the 0.998 in-distribution baseline, showing meaningful degradation under LODO.Mean and pooled AUC can diverge sharply: MLP has pooled AUC 0.747 versus Mean Linear’s 0.845 despite similar mean per-fold AUC.
  • Per-dataset heterogeneity: Per-dataset performance is highly heterogeneous, with some folds below 3%, ranking reversals across architectures, and accuracy standard deviations of ±0.25–±0.30.Held-out dataset choice dominates architecture under LODO, unlike the ±0.01 variation reported for 5-fold cross-validation.
  • Interpretation: The authors interpret the LODO pattern as probes learning an intent-aligned mixture that includes dataset-specific surface attributes rather than intent in isolation.This residual varies in composition across held-out datasets.
  • Perturbation under LODO: Under perturbation, aggregate TPR changes by only −1 to −7pp, but fold-level drops remain heterogeneous and can be masked by floor effects.The perturbation evaluation covers seven mixed-class folds with thresholds recalibrated per fold and condition.
  • KV-fork under LODO: +4.0pp clean weighted accuracy brings the KV-fork linear probe into the architectural top tier under LODO.The comparison is against the no-fork linear probe and places it near Attention at 0.810 and MultiMax at 0.821.
  • KV-fork under LODO: At recalibrated FPR=1%, the KV-fork produces an +8.4pp clean TPR gain, consistent with stabilizing the readout context against LODO score-scale drift.Suffix-content variation remains untested.
Loading 2609.15994v1…