PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Overlays: Mitigating Prompt Injection with Annotations Beyond Tokens and Steering Vectors

Joshua Penman

arXiv:2608.23873v1cs.AIcs.CLcs.CRcs.LG

TL;DR

Language models receive span metadata through token streams that attackers can imitate or confuse, motivating a non-textual annotation channel. The paper introduces Semantic Overlays, learned adapters applied selectively to a frozen model’s residual stream, and finds that non-executable overlays strongly defend against prompt injection while preserving marked content readability.

  • Problem

    Token-only inputs make provenance and span identity forgeable, leaving models vulnerable to prompt injection.

  • Method

    Semantic Overlays apply small learned adapters at chosen prefill positions in a frozen model’s residual stream to annotate spans out of band.

  • Results

    Semantic Overlays defend against prompt injection across SEP, TensorTrust, and PIArena while preserving utility and readable marked spans.

  • Takeaways & Limitations

    Marking retrieved or third-party content non-executable can remove imperative authority without removing the model’s access to its content.

  • Takeaways & Limitations

    Novel overlay descriptors can retrieve marked spans but do not become executable behaviors when those behaviors were never trained.

Abstract

from arXiv · show

Everything a language model sees is tokens. The serving stack knows what each span is -- user input, tool output, instructions -- but the model must keep track of that itself, and it can lose track or be confused: text can be written to read like anything. Prompt injection is a natural exploit of this phenomenon. By scrambling the model's understanding of span identity, an attacker can induce unwanted and potentially dangerous actions. Adding a non-textual channel to the model's input -- a way to communicate span identity beyond text -- mitigates this class of attack. We thus introduce a general steering technique called Semantic Overlays: small learned adapters applied at chosen prefill positions to a frozen model's residual stream. Laying an overlay over a span creates an out-of-band annotation channel that cannot be replicated by tokens. Unlike steering vectors, Semantic Overlays are trained, adaptable, and selectively applied. An overlay can encode complex semantics that reshape how the model perceives the marked span: asked to copy a code snippet under an overlay asserting that it is in a different programming language than it is, the model rewrites the snippet, faithfully, in the asserted language. Overlays are also composable, allow for transparent reading of underlying content, and can carry complex payloads -- including imperatives that the model will follow. An overlay which marks a span as "non-executable" defends against the broad class of prompt injections that add instructions in untrusted context. We report strong results on prompt injection benchmarks: SEP separation rises from 24.3% to 96.5% with utility unchanged (our scoring rule; we also correct a defect in the published grader), TensorTrust attack success rate falls from 34.8% to 6.6%, and all four PIArena attack families drop to 0% compliance, all while marked spans stay readable (92.5% exact copy rate).

1 INTRODUCTION

Language-model serving stacks know span provenance and other metadata, but token-only inputs make that information forgeable and easy to confuse. Semantic Overlays add a learned, non-textual channel that selectively changes how frozen models interpret spans and can mark untrusted content as non-executable.

  • Motivation: Token-only inputs force models to reconstruct span metadata from text that attackers can imitate.Serving stacks distinguish user, retrieved, and tool content, but the model receives a single token stream.
  • Approach: Semantic Overlays are learned adapters applied selectively to a frozen model’s residual stream at chosen span positions.The base model remains unchanged, and unmarked prompts run exactly as with the frozen model.
  • Capabilities: Overlays provide a reliable, compositional channel that preserves access to the underlying marked content.The model can detect single and stacked overlays while still reading the tokens beneath them.
  • Capabilities: An overlay can override a span’s apparent semantics, such as making the model treat non-Python code as Python and rewrite it accordingly.The overlay controls both classification and copying behavior for the marked snippet.
  • Capabilities: Overlays can carry span-delimited commands that the model follows only for the marked span.Examples include answering in Spanish or refusing with safety concerns.
  • Prompt-injection defense: A non-executable overlay turns retrieved text into inert content for instruction-following, providing a prompt-injection defense.The paper evaluates this approach on SEP, TensorTrust, and PIArena and also reports corrections to evaluation defects.

2 RELATED WORK

Semantic Overlays extend prior steering, prompt-tuning, prompt-injection-defense, provenance, and gradient-editing approaches by carrying metadata through selective residual-stream edits. Their distinguishing carrier is an inference-time annotation applied to existing spans rather than text or added positions.

  • Activation steering: Unlike activation steering, Semantic Overlays are trained end-to-end, span-selective, distributed across layers, state-conditional, and capable of multiple qualities.Activation-steering directions are typically fixed, position-uniform, and limited to one quality each.
  • Prefix and prompt tuning: Prefix and prompt tuning add learned continuous positions, whereas Semantic Overlays edit designated existing spans and carry span-level metadata.The distinction is whether the learned signal competes as an added position or annotates an existing span.
  • Prompt-injection defenses: Existing prompt-injection defenses modify prompts, embeddings, or the base model, while Semantic Overlays preserve a frozen base model and use a separate carrier.The paper identifies AIR as the closest mechanism among the compared defenses but notes its joint full fine-tuning of the base model.
  • Provenance and instruction hierarchy: The injection contract follows provenance and instruction-hierarchy semantics: the pipeline assigns roles, while the carrier differs from token-based labeling.Retrieved data retains its content but loses imperative authority.
  • Gradient-space goggles: Gradient-space goggles and Semantic Overlays are independent methods that place metadata edits at different stages and can be composed.The paper characterizes Semantic Overlays as the inference-time member of the same broader family.

3 METHOD

The method attaches small adapters throughout a frozen model and activates them through per-token overlay masks during prefill. Shared or per-overlay adapters encode overlay qualities, support order-invariant composition, and are trained on target behaviors plus no-op cases.

  • Overlay adapter: Each overlay set attaches a small adapter to the input of every decoder layer in the frozen model.At marked positions, the adapter operates on the hidden state and overlay identity code.
  • Overlay adapter: The adapter uses a SwiGLU bottleneck with RMS-normalized hidden state and an optional overlay code identifying the carried quality.A code-free adapter supports a single overlay, including the do-not-execute overlay.
  • Position masking: A boolean per-token, per-overlay mask activates overlays only at selected prefill positions, while decoding remains otherwise unaffected.The resulting activations influence how the model reads marked spans through the key–value cache.
  • Multiple overlays: Overlay sets support multiplexing through per-layer vectors, per-overlay MLPs, or shared MLPs conditioned on learned or frozen embedding codes.Overlapping overlays compute deltas from the same pre-edit state and sum them, making composition order-invariant.
  • Training: Training uses synthetic target completions, frozen-base cross-entropy, and no-op examples that preserve behavior when overlays are absent or irrelevant.The evaluation uses new source items, and cross-entropy on both behavior-changing and behavior-preserving completions outperformed KL-based alternatives on SEP.

4 INVISIBLE HIGHLIGHTERS: OVERLAYS ARE A RELIABLE, COMPOSITIONAL CHANNEL

Semantic Overlays form a readable, compositional annotation channel that can identify marked spans, including stacked marks, without encoding span identity in tokens. Among tested architectures, the embedding-conditioned shared adapter performs best on held-out mark readout.

  • Held-out readout: The hard round stacks up to three marks per token, with overlapping overlays summed from the same pre-edit state.This tests compositionality beyond non-overlapping span marking.
  • Held-out readout: 99.5% of present qualities are read exactly on the easy held-out round, while absent qualities are refused correctly 100% of the time.The embedding-conditioned shared adapter makes one two-character readout error on the easy round.
  • Architecture comparison: The embedding-conditioned shared adapter leads nearly every held-out readout row, including the hardest question by 30 points.The comparison covers four architectures trained on identical data and recipe.
  • Compositional channel: Figure 2 demonstrates exact selection of the three underlined spans while ignoring other marks sharing their tokens.The figure combines three underlines, two highlights, and a circle, with no token marking any span.
  • Architecture comparison: Per-overlay MLPs use private parameters and can corrupt marked text, such as reading “twenty moves” as “six moves.”The private extreme has 604M parameters and is likely undertrained at this budget.
  • Architecture comparison: A per-layer vector adds the same delta regardless of the current activation, whereas an MLP conditions its write on the current state and can adapt to earlier overlay writes.This distinction becomes important when overlays are stacked, although vectors remain competitive on single-span questions.

5 PYTHON-COLORED GELS: THE CHANNEL TRANSFORMS HOW THE MODEL UNDERSTANDS THE SPAN

Semantic Overlays can impose a counterfactual semantic identity on a span while preserving its underlying content. In language-overlay tasks, the model answers according to the asserted language and can rewrite marked code into that language.

  • Content preservation: Figure 3 shows a Racket snippet selected and re-expressed in Python after an overlay asserts that the snippet is Python.The selection task and the rewrite retain the same prompt structure and underlying programming task.
  • Overlay-controlled understanding: 100% of language-identification and asserted-language listing answers follow the overlay when the answer depends on it.Prompts contain five snippets, with overlays asserting Python, JavaScript, Ruby, or C independently of the snippet’s actual language.
  • Content preservation: 97% of overlaid copies are rewritten in the asserted language, and 91% faithfully preserve the displayed snippet’s task.Unoverlaid snippets are reproduced verbatim in 96% of copies.
  • Comparison: Earlier per-layer-vector experiments imposed the asserted language perfectly but replaced the underlying program with generic filler 95% of the time.The comparison highlights the adapters’ advantage in asserting a quality while preserving readable content.

6 SECRET INSTRUCTIONS: THE CHANNEL CAN CARRY SPAN-DELIMITED LATENT COMMANDS

Overlays can carry instructions scoped to individual spans, changing how the model responds to one request without applying the transformation to sibling requests. Held-out compliance varies substantially across transform types.

  • Span-delimited commands: Transform overlays apply their requested behavior to the overlaid request and never to its sibling requests.The experiment marks one request within rambling user messages containing three to five separable requests.
  • Held-out compliance: Held-out compliance ranges from 100% for refusal citing safety filters to 18% for the haiku transformation.Eight transforms include refusals, age-targeted explanations, capitalization, translation, haiku, and nested bullets.
  • Held-out compliance: When an overlay fails, it usually fails silently, although it can occasionally produce the correct answer in an unexpected position.The reported compliance variation may relate to task mechanisticity and the amount of supervised output.

7 AN NX BIT FOR LANGUAGE MODELS: MARKING SPANS NON-EXECUTABLE DEFEATS PROMPT INJECTION

A non-executable overlay marks untrusted retrieved or third-party spans so their instructions lose imperative authority while their content remains usable as data. Across SEP, TensorTrust, and PIArena, this defense substantially reduces prompt-injection compliance without sacrificing utility, readability, or cross-model replication.

  • Defense mechanism: A non-executable overlay gives retrieved or third-party spans data status: embedded instructions are not followed, but the span remains usable as content.The serving stack applies the overlay to every token of untrusted content, delivering role semantics out of band.
  • Evaluation: The evaluation covers SEP, TensorTrust, and PIArena, including a defense-aware red team’s 222 adaptive black-box probes.None of the red team’s direct authority attacks succeeded.
  • SEP: 23.9% → 94.7% SEP separation on 9,160 items, while probe-in-data answers fell 72.6% → 5.3% and utility did not move.The result reproduced within 0.4 points across three seeded runs under ASIDE’s protocol.
  • TensorTrust: TensorTrust hijacking fell five-fold and extraction six-fold, at the cost of about four points of Defense Validity.The evaluation used 776 hijacking rows and 570 extraction rows, screening Defense Validity to rows the frozen model could answer.
  • PIArena: PIArena’s four indirect-attack families dropped to zero for three families and one row in fifty for the fourth, with unchanged utility retention.The attacks were scored as behaviors rather than topic-word matches and included clean controls.
  • Fidelity: 92.5% exact-copy rate preserved marked-span readability, indicating that the overlay removes imperative authority rather than filtering or suppressing the text.Worst-case similarity was 0.978, compared with 99.2% for the model without an overlay.
  • Replication: 31.0% → 96.2% SEP separation on frozen Llama-3.1-8B-Instruct demonstrates replication of the defense on a second model family.The overlay also reduced TensorTrust hijacking from 68.9% in the reported replication setup.

8 SMALL AMOUNTS OF FREE LUNCH FOR NOVEL DESCRIPTORS

Novel descriptors generalize differently depending on whether they retrieve marked content or execute an untrained behavior. Frozen embeddings transferred successfully as pointers, but unseen behavioral instructions failed and often selected the nearest trained instruction.

  • Instruction descriptors: 0/120 unseen instructions succeeded, with the adapter substituting the nearest trained instruction, such as French for requested Italian.Trained instructions achieved 50–85% compliance in the same experiment.
  • Pointer descriptors: 95% held-out concepts matched trained concepts in span transport when the descriptor only made marked text retrievable.The transport task asked the model to repeat exactly the words marked with a never-trained concept.
  • Interpretation: Novel pointers can work zero-shot because frozen embeddings already provide retrieval geometry, whereas novel programs require trained behavior.The paper leaves open whether the program side will generalize at larger scale.

9 CONCLUSION

Semantic Overlays add a non-textual residual-stream channel that marks designated spans with annotations tokens cannot imitate. The channel supports reliable, composable, behavior-changing annotations and prompt-injection defense while preserving marked-text readability.

  • Conclusion: Semantic Overlays write annotations into a frozen model’s residual stream at the spans they describe, using a medium input text cannot imitate.The channel is presented as a new information channel for transformer inference.
  • Conclusion: 99.5% exact span retrieval and individually decodable stacked marks show that overlays are reliable and composable.The conclusion reports twelve visual marks in the retrieval test.
  • Conclusion: 100% language identification by the overlay and faithful language rewriting show that overlays can override a span’s own evidence about what it is.The overlay asserts the snippet’s language and controls the answer to identification and copying requests.
  • Conclusion: A transform overlay obeyed span-delimited instructions for one request and no other, demonstrating localized instruction carrying.The example instruction was to answer in Spanish and refuse.
  • Conclusion: 24.3% → 96.5% SEP separation, 34.8% → 6.6% TensorTrust hijacking, and 0% compliance across PIArena families were reported with 92.5% exact copy rate.These results summarize the prompt-injection defense while preserving marked text’s quotability.

REPRODUCIBILITY STATEMENT

The paper describes reproduction coverage, implementation details, related-work comparisons, training choices, validation, and evaluation corrections. Its central reproducibility claim is supported by released code, data, checkpoints, and a second-model rerun.

  • Reproducibility: Released code and data cover the corpus, training, serving, and all three evaluations behind the prompt-injection defense on both base models.The channel experiments release trained overlay sets, while Appendix I reruns the pipeline on a second base model.
  • Related work: Semantic Overlays makes designated input-span position a design variable, applying the edit during prefill rather than treating position as an incidental implementation detail.The comparison spans steering methods whose position policies are uniform, unstated, or inconsistent.
  • Training: The reported do-not-execute overlay set trained for about four hours on eight H100 GPUs, approximately 32 GPU-hours.The recipe used 2,508 optimizer steps; the broader recipe specifies Muon, warmup, batch, and epoch settings.
  • Training: 6× multiplier reached 95.7% separation versus 86.8% at 1× and 88.4% at 12× on a 300-item subset.Differences under about 3 points were not resolvable in that subset.
  • Validation: Targets were mechanically validated before training, and injection payloads were screened against standalone frozen-model answers to separate obedience from payload solvability.The validation process included language detectors, format checks, and verbatim checks outside edited sections.
  • Training: The eight transform overlays reuse 7,166 markable request slots across 2,120 prompts, keeping the per-overlay supervised batch constant.The slots span roughly 50 themed domains and are validator-clean.
  • Training: The teacher ceiling table reports that cross-entropy toward a validated edit surpasses the behavior ceiling reproduced by in-context and forward-KL methods.The comparison concerns held-out compliance counts.

C ARCHITECTURE GEOMETRY

The overlays’ learned representations vary with layer and preserve multiple qualities, while their training corpus pairs injected and benign spans with the same clean target. Evaluation also documents benchmark-measurement corrections and adversarial behavior that remains outside the defense’s scope.

  • Architecture geometry: Per-layer overlay vectors rotate through the network rather than collapsing to one direction.Adjacent layers have cosine 0.74, layers eight or more apart have cosine 0.06, and effective rank is around 24 of 32.
  • Architecture geometry: Instruction-grid language codes collapse to effective rank two and score zero of 120 on held-out values.Held-out values fail by nearest-neighbor substitution, such as producing French when asked for Italian.
  • Training data: The training corpus combines retrieval passages, self-contained payload instructions, framed splices, and splice positions.Payloads are screened against standalone frozen-model answers, and frames are ranked by standalone injection rate before sampling.
  • Training data: Each injected and benign training item shares the frozen model’s greedy completion on the clean passage as its target.The injected item therefore trains behavior as though the payload were absent, while benign content retains the same target.
  • Training data: Additional copy, gate, and validator families train readability and continued use of marked-span content.Copy asks for verbatim quotation; gate and validator tasks request access decisions or checkable facts about the span.
  • Adversarial behavior: Adaptive attacks failed when asserting authority inside marked passages, but authorized instructions to follow marked procedures still override the overlay inconsistently.The training data does not establish an instruction hierarchy for contradictions between trusted instructions and the overlay’s meaning.
  • Measurement: All reported measurements include controls for benchmark defects, including clean-condition checks and corrected witness scoring.Three of 18 SEP marked-arm hits also fire on clean conditions, while the corrected rule addresses mid-word artifacts and inflections.

F.2 TENSORTRUST: THREE HARNESS DEFECTS, AND ONE OF THE BENCHMARK’S OWN

TensorTrust’s native three-segment structure exposes substantial harness and benchmark defects when rendered through a two-slot template. Reanalysis keeps all rows and separates these structural limits from the overlay’s measured behavior, while comparisons with other defenses require paired baselines.

  • Harness structure: TensorTrust contains an opening defense, attacker input, and closing defense, but ASIDE’s harness renders only two slots.The harness therefore cannot place a defender segment after the attacker input without filtering rows.
  • Harness defects: 78.1% of TensorTrust rows are discarded non-randomly, leaving 165 of 776 rows in the surviving subset.Rows relying on closing-defense text are disproportionately removed, so the subset is not representative of the full benchmark.
  • Harness defects: The surviving subset’s literal Defense Validity ceiling is 68.2%, versus 80.9% on all 776 rows.Measured arms are therefore pinned against a structural ceiling rather than measuring only the adapter.
  • Benchmark defects: 32% of surviving rows omit the access code from the defense prompt, creating another Defense Validity defect.The filter checks whether the opening defense is self-contained in form, not whether the code appears in it.
  • Scoring defects: Short access codes inflate extraction under substring scoring, so extraction is scored with word boundaries instead.Codes under five characters comprise 6.8% of rows and inflate measured extraction by roughly 18 points.
  • Benchmark defects: Some labelled access codes contradict their defenses’ own rules, making refusal count as a Defense Validity failure.This defect belongs to the benchmark itself and cannot be removed by harness choices.
  • Corrected evaluation: The reanalysis keeps all 776 rows by placing the closing defense after the attacker span and leaving defender text unmarked.Span offsets are used within user content so repeated access codes elsewhere do not determine the marked region.
  • Comparisons: Published-defense comparisons should use paired deltas against each method’s own baseline because ASIDE fine-tunes its models.The overlay leaves the served base model frozen and switchable per request, unlike defenses that modify served weights.

J LIMITATIONS

The evidence is concentrated in one base model and benchmark setting, and several important deployment and threat-model boundaries remain untested. The overlay’s interface-level unforgeability does not cover compromised serving infrastructure or optimization-based attacks.

  • Scope: Most experiments use one frozen Qwen3.5-9B instruct model at one scale; only the injection defense is replicated on Llama-3.1-8B-Instruct.The replication covers SEP, TensorTrust, and PIArena, not the red-team engagement or other overlay channels.
  • Threat model: Text-only attackers cannot write the overlay’s activation-space channel, but optimization-based attacks against the overlay remain untested.The red team explored 222 black-box adaptive probes, which the authors describe as one engagement rather than a proof.
  • Scope: The binary do-not-execute overlay does not distinguish among trusted sources; per-source overlays are supported by the machinery but untrained here.Source-specific authority would require additional overlays and training.
  • Instruction hierarchy: Trusted instructions that delegate to marked data can contradict the overlay, producing inconsistent behavior because instruction hierarchy is untrained.The training corpus contains exactly one developer–span relationship.
  • Deployment boundary: The non-executable overlay preserves quotability, so it should not be deployed as an output filter.Its trained meaning strips commands while keeping marked content readable.
  • Deployment boundary: All attacks use untrusted text in single-turn prompts; agentic multi-turn tool-output attacks are not measured.The authors identify agentic deployments as the next setting for applying the mechanism.
  • Systems cost: End-to-end latency is unprofiled despite an estimated marked-position compute increase below one percent.Adapters run through vLLM’s standard plugin interface, while decoding uses the stock model.
Loading 2608.23873v1…