PaperlayerView exact revision on arXiv

Source-linked AI summary

Post-Training Large Language Models with LLMON

Basel Shbita, Michael Hind, Chad DeLuca, Nathan Fulton, Shubhi Asthana, Farhan Ahmed, Anisa Halimi, Yada Zhu, Mauro Martino, David Cox, Dan Gutfreund

arXiv:2610.38120v1cs.AIcs.LG

TL;DR

LLMs process executable instructions and passive data without a built-in distinction, creating a gap in reliable instruction binding. The paper post-trains Granite-4.1 models with LLMON’s explicit, identifier-based execution structure and reports sharply lower indirect prompt-injection attack success rates while preserving general capabilities.

  • Problem

    LLMs lack a built-in distinction between executable instructions and passive data, leaving instruction binding implicit when multiple instructions coexist.

    Source: §1 · p. 1

  • Method

    The paper post-trains two Granite-4.1 base models using LLMON-structured data with explicit instruction-data separation and identifier-based execution.

    Source: §1 · pp. 1–2

  • Results

    Indirect prompt-injection attack success rates fall from 53-100% to 0.02-5.6%, without catastrophic degradation on standard capability measures.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Explicit execution semantics can be taught through standard post-training without architectural modification.

    Source: §6 · p. 8

    Evaluation covers two scales within one model family and generation; transfer to other corpora, architectures, and larger scales remains unvalidated.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Large Language Models (LLMs) treat all context tokens uniformly, lacking explicit separation between executable instructions and passive data. This leads to prompt injection vulnerabilities, ambiguous instruction binding, and unreliable execution control. We present a post-training methodology using LLMON (LLM Object Notation), a framework that enables separating instructions from data and requires explicit identifier-based execution. We construct a large-scale structured training corpus of 4.6B tokens spanning general, RAG (Retrieval-Augmented Generation), and tool-calling data, augmented with distractor-infused variants in which competing instructions and irrelevant data segments are present in context but not referenced. Post-training two Granite-4.1 base models (3B and 8B) through supervised fine-tuning followed by reinforcement learning, we show that LLMON-structured supervision reduces attack success rates on indirect prompt-injection benchmarks from 53-100% to 0.02-5.6% while preserving general capabilities. We further introduce attention-based interpretability metrics that diagnose failure modes at individual transformer layers for structured inputs, revealing how our approach reshapes internal attention patterns. Our results demonstrate that explicit execution semantics can be taught through standard post-training without architectural modification.

1 Introduction

LLMs lack a built-in distinction between executable instructions and passive data, contributing to prompt injection and ambiguous instruction binding. The paper proposes LLMON-structured post-training and evaluates it on two Granite-4.1 models, alongside attention-based interpretability metrics.

1 Introduction

  • Uniform attention over context tokens leaves instructions, retrieved data, and injected payloads without a token-level privilege distinction.The paper links this gap to prompt injection, ambiguous referential binding, and unreliable RAG and tool-calling scenarios.Source: §1 · p. 1
  • LLMON separates instructions from data and binds execution explicitly to identified blocks.Execution calls reference specific instruction and data identifiers; only explicitly invoked instructions are executed.Source: §1 · p. 1
  • The training-data pipeline structures general, RAG, and tool-calling corpora and adds distractor variants with competing instructions and irrelevant data.The distractors are present in context but not referenced for execution.Source: §1 · p. 1
  • The post-training evaluation uses Granite-4.1 base models at 3B and 8B scales, with supervised fine-tuning followed by reinforcement learning.The supplied introduction identifies the models and training stages, but gives no benchmark outcomes here.Source: §1 · pp. 1–2
  • The paper introduces Instruction Focus Ratio, Data Attention Score, and Leakage to examine how structured training reshapes attention.These attention-based metrics are proposed to provide mechanistic insight.Source: §1 · p. 2

2 Related Work

Prior instruction tuning and alignment rely on implicit formatting and positional cues, while other approaches address prompt boundaries, structured outputs, or tool calls. LLMON is presented as complementary training-level supervision with explicit, identifier-based execution binding.

Instruction tuning and alignment

  • Instruction tuning and preference alignment rely on implicit positional cues and chat-template conventions, leaving multiple instructions without a principled disambiguation mechanism.LLMON-structured supervision can be applied on top of instruction-tuning corpora to add explicit execution binding.Source: §2 · p. 2

Prompt injection and adversarial robustness

  • Unlike in-band delimiters or role-based boundaries, LLMON uses reserved tokens and explicit bindings so multiple same-role instructions can coexist with only one bound for execution.The passage contrasts LLMON with Spotlighting, StruQ, and SecAlign [Hines et al., 2024; Chen et al., 2025a; Chen et al., 2025b].Source: §2 · p. 2

Tool use and function calling

  • LLMON’s execution semantics are described as compatible with tool-calling pipelines, where tools, retrieved documents, and user instructions occupy typed segments with distinct identifiers.Existing tool-calling approaches define schemas and train structured calls, while the prompt’s instruction-data boundary remains implicit.Source: §2 · p. 2

Structured generation and constrained decoding

  • Unlike inference-time constrained decoding or application-layer orchestration, LLMON operates at the training-data level to teach structural distinctions through supervised learning.The passage contrasts this approach with grammar-based decoding, prompting frameworks, and markup-based prompt languages.Source: §2 · p. 2

3 Using LLMON (LLM Object Notation)

LLMON supplies typed, identified blocks and explicit execution bindings that separate authoritative instructions from passive input. The training pipeline serializes general, document-grounded, and tool-calling examples, then adds unreferenced distractors to train binding under interference.

3 Using LLMON (LLM Object Notation)

  • LLMON is a hierarchical tagging framework for explicit instruction-data separation and identifier-based execution binding, used in standard autoregressive language modeling without architectural modification.The section describes using it to construct structured post-training data for explicit execution control.Source: §3 · p. 2

3.1 Notation Overview

  • Typed segments identify instructions, passive data, requirements, documents, and tool-schema fields, while system blocks carry authoritative policy across turns.Tool definitions are flattened into addressable sub-fields, and tool results return as data blocks.Source: §3.1 · pp. 2–3
  • Execution blocks reference the instruction, input data, and output destination by identifier; authority follows the binding, not a block’s position or role name.The privilege invariant treats instruction or system references as authoritative and input references as material to read rather than commands to follow.Source: §3.1 · p. 3

3.2 Data Construction Pipeline

  • The pipeline serializes conversations, document-grounded examples, and tool-calling examples into LLMON blocks with explicit execution cycles and referenced inputs.RAG execution references user data and relevant documents; tool use serializes a call, tool-result data, and a subsequent assistant response.Source: §3.2 · p. 3

3.3 Distractor Augmentation

  • Distractor augmentation inserts competing instructions and irrelevant data while keeping execution pointed at the original target, whose output stays identical across variants.Variants use one to three distractor instructions and zero to three irrelevant data segments; loss penalizes reliance on surface cues such as position or imperative phrasing.Source: §3.3 · p. 3

4 Experiments

The experiments post-train two Granite-4.1 base models on structured general, RAG, and tool-calling data, then assess capabilities, indirect injection robustness, red-team robustness, and layer-wise attention. The supplied experiment passages describe the corpus, training protocol, and evaluation design, but not the benchmark outcomes.

4.1 Training Data

  • The structured corpus totals 4.4M examples and 4.6B tokens, with general, RAG, and tool-calling data accounting for 93.6% of tokens.Prompt-injection training examples render injected instructions in blocks not referenced by execution, with target outputs that ignore them.Source: §4.1 · p. 4

4.2 Models and Training

  • Granite-4.1-3B-Base and Granite-4.1-8B-Base receive full fine-tuning on the structured corpus, followed by five GRPO stages and a final DPO stage.The models are base checkpoints without prior instruction tuning; the final DPO stage uses about 1k on-policy injection preference pairs.Source: §4.2 · p. 4, §4.1 · p. 4

4.3.1 Standard Benchmarks

  • Standard capability evaluation reports unweighted benchmark means across general, alignment, math, code, tool-calling, multilingual, and safety categories.Each LLMON model is compared with an instruct counterpart at the same scale, trained on a matched chat-template corpus.Source: §4.3.1 · pp. 4–5

4.3.2 Indirect Prompt-Injection Benchmarks

  • Indirect-injection evaluation covers payloads in classic NLP task data and agent tool observations, using attack success rate, where lower is better.The LLMON payload is placed in a data block reached through an input reference; instruct baselines receive the same payloads in conventional chat formatting.Source: §4.3.2 · p. 5

4.3.3 Adversarial Red-Teaming

  • ARES red-teaming expands 808 seed prompts into approximately 15k attacks across eight strategies, with robustness reported as 1−ASR.The advanced arm uses multi-turn Crescendo-style attacks generated by an attacker model and judged against a 20-turn budget.Source: §4.3.3 · p. 5

4.3.4 Attention-Based Interpretability Metrics

  • IFR, DAS, and LEAK measure per-layer attention from the final generated token to target instructions, target data, and distractor content.The metrics are computed across attention heads; higher IFR and DAS indicate stronger focus on task-relevant content, while larger LEAK suggests more attention to irrelevant or competing information.Source: §4.3.4 · pp. 5–6

5 Results and Discussion

LLMON-structured post-training sharply reduced indirect prompt-injection success, preserved most general capabilities, and changed attention to favor bound instructions and data over distractors. Red-teaming and layer-wise analyses show both the scope of the robustness gains and where attention patterns separate.

5.1 General Capabilities

  • The six-category capability mean was unchanged at 3B (66.7 both ways) and fell 1.4 points at 8B (74.2 to 72.8), while Safety improved at both scales.The largest single regression was 8B mathematics, from 84.1 to 79.6.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 Indirect Prompt Injection

  • ASR fell from 100.0 to 0.1 at 3B and 96.3 to 0.02 at 8B on text-level injection, and from 99.2 to 5.6 and 52.8 to 3.1 on agentic injection.The injected instruction occupies a data block that no execution pointer references.Source: §5.2 · p. 6, Table 2 · p. 7

5.3 Adversarial Red-Teaming

  • Indirect-attack robustness rose from 2.8% to 84.6% at 3B and from 0.7% to 79.8% at 8B in the ARES red-teaming campaign.Direct and multi-turn attacks also improved, roughly doubling at both scales, but remained at 32–37% robustness.Source: §5.3 · p. 6, Table 3 · p. 7

5.4 Attention Interpretability Analysis

  • At layers 25 and 28, mean IFR was 86–87% for LLMON versus 26–38% for Instruct, while mean leakage was 6–9% versus 35–45%.At the final layer, IFR fell toward chance for both models (28.0% vs 17.2%), so the binding is better read in the mid-to-late stack.Source: §5.4 · p. 6, Table 4 · p. 7
  • On a representative distractor example, IFR separated from layer 22, reaching 55–89% for LLMON across layers 25–38; the models were indistinguishable through layer 20.Both models fell at the final layer.Source: §5.4 · p. 7, Figure 2 · p. 8
  • From layer 7, LLMON distractor leakage settled into a 5–25% band, while Instruct remained between 20% and 50% and rose again in late layers.Both models leaked heavily in the first six layers.Source: §5.4 · p. 7, Figure 3 · p. 8
  • At layer 31, LLMON allocated 23.4% of span attention to the target instruction and 23.0% to target data, versus 47.5% to a distractor instruction for Instruct.Token-level traces show target data attention during quoting, then target instruction attention as the answer is formed; distractor spans stayed near zero.Source: §5.4 · pp. 7–8

6 Conclusion

LLMON’s explicit instruction-data separation and identifier-based execution binding improve controlled execution through standard post-training. Across two Granite-4.1 scales, it sharply reduces indirect prompt-injection success while preserving standard capabilities, though direct and multi-turn attacks remain harder.

6 Conclusion

  • LLMON’s explicit instruction-data separation and identifier-based execution binding improve controlled execution in LLMs.The method uses standard post-training without architectural modification.Source: §6 · p. 8
  • Indirect prompt-injection attack success falls from 53–100% to 0.02–5.6% across two Granite-4.1 scales, without catastrophic degradation on standard capability measures.The effect also extends to indirect red-teaming, while direct and multi-turn attacks remain substantially harder.Source: §6 · p. 8
  • IFR, DAS, and LEAK provide per-layer failure diagnosis and reveal how structural training reshapes internal attention patterns.The findings suggest explicit execution structure can be taught reliably through standard post-training.Source: §6 · p. 8

Limitations

The evaluation and analysis have boundaries in transfer, compute-cost characterization, attack coverage, language scope, and interpretation of attention metrics.

Limitations

  • LLMON’s improvement magnitude remains unvalidated beyond two scales in one model family and generation, including other corpora, architectures, and larger models.The passage notes that larger models may already exhibit stronger implicit instruction-following.Source: Limitations · p. 8
  • Delimiter tokens add approximately 7–15% context length, while inference-latency and throughput impacts were not systematically quantified.Training-time sequence packing largely amortizes the cost.Source: Limitations · p. 9
  • The privilege invariant offers less protection against direct and multi-turn attacks than indirect attacks, because those cases lack the same instruction-data boundary.Red-teaming robustness is 33–37% on direct attacks and 28–37% on multi-turn attacks, versus 80–85% on indirect attacks.Source: Limitations · p. 9
  • LLMON reduces but does not eliminate the chance that instructions embedded in data are followed, so it should be one defense layer rather than the only one.The evaluation does not establish that IFR, DAS, or LEAK causally explain output quality; attention weights are an imperfect proxy for information flow.Source: Limitations · p. 9
  • The predominantly English training corpus leaves tag overhead on morphologically rich languages and very long documents uncharacterized.Capability evaluation includes multilingual benchmarks, but the training corpus itself is predominantly English.Source: Limitations · p. 9

A.1 Multi-instruction and Multi-data Distractors

The distractor example places multiple instructions and data segments in one context while an execution block designates only one instruction-data pair. The model must selectively attend to that bound pair and ignore the distractors.

A.1 Multi-instruction and Multi-data Distractors

  • The model must attend selectively to the referenced segments and ignore the other instructions and data.The example requires executing only the word-counting instruction, not the summarization or translation distractors.Source: §A.1 · p. 13, Figure 7 · p. 14
  • Three instructions and two data segments coexist, but the execution block binds only one instruction-data pair.The example’s distractor instructions are summarization and translation, while the bound task is word counting.Source: §A.1 · p. 13, Figure 7 · p. 14
  • The execution block references only the designated instruction and input data by identifier.In the example, it references “instr<|:|>íä” and “data<|:|>Äî”.Source: Figure 7 · p. 14
Loading 2610.38120v1…