PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating and Improving the Robustness of Large Language Models to Input Sequence Variations

Narek Maloyan

arXiv:2610.02432v2cs.CRcs.CLcs.LG

TL;DR

Production LLMs face prompt injections, trojans, and manipulation of automatic quality metrics, while robustness questions also extend to agentic tool use. The paper develops robustness measures and defenses across these settings; its findings include reduced attack success for heterogeneous committees.

  • Problem

    Robustness remains limited across perturbation types, and MCP tool-context injections can lead to unauthorized tool calls with arguments.

    Source: §1.1.3 · p. 17, §3.3.3 · p. 60

  • Method

    The paper combines a generative robustness metric and theoretical bounds with empirical attack and defense studies, including fixed session permissions for MCP agents.

    Source: Algorithm for Cryptographic… · p. 80, Conclusion · p. 95

  • Results

    Heterogeneous committees reduce ASR for Gemma-3-4B by 47–55 percentage points.

    Source: Conclusion · p. 95

  • Takeaways & Limitations

    For MCP agents, an immutable session permission table restricts reachable actions independently of context content.

    Source: Algorithm for Cryptographic… · p. 80

    Robustness to one perturbation type does not ensure robustness to semantic paraphrases or character-level attacks, and defenses can increase vulnerability to other attack classes.

    Source: §3.3.3 · p. 60

Abstract

from arXiv · show

Large language models (LLMs) in production systems face prompt injections, trojans (backdoors), and manipulation of automatic quality metrics. This thesis develops models, methods, and algorithms for evaluating and improving LLM robustness to adversarial input sequence variations. We propose R_stab(f), a generative robustness metric based on the Jensen-Shannon divergence between per-step output distributions under small input perturbations. For localized attacks we prove V(h) <= 1 - R_class(h), where R_class(h) is the probability that a decision operator h keeps its decision under small perturbations. For non-localized attacks we propose a calibrated empirical model. For LLM-as-a-Judge systems we develop ASA, an adaptive evolutionary black-box attack that reaches an attack success rate (ASR) of up to 73.8%, with transfer between open models up to 62.6%. On Trojan Detection Challenge 2023 data (Pythia-1.4B), surrogate triggers reach REASR ~0.99 while recall of the true triggers is ~0.17 against a baseline of ~0.14. On SaTML CTF 2024 we systematize four classes of bypasses of multi-layer defenses, which reduce the ASR from 90% to 15-25%. Committees of 5-7 heterogeneous models reduce the ASR for Gemma-3-4B by 47-55 percentage points, to 19.3% with 7 models. For agentic systems based on the Model Context Protocol (MCP), we propose AttestMCP, which attests tool calls with HMAC-protected packets at under 0.1 ms per call, and the Commit Boundary isolation pattern. On the MCPBench benchmark of 847 scenarios they reduce the average ASR from 53.7% to 12.4%. The methods are implemented in the JudgeGuard and TrojanArmor software suites and the MCPSec module.

Introduction

The introduction develops methods to evaluate and improve LLM robustness to prompt injections, trojans, attacks on agentic systems, and destabilizing input variations. It maps the research objectives across threat classification, robustness theory, adversarial methods, defenses, and MCP security.

Theoretical contributions

  • Rstab(f) measures generative robustness using Jensen–Shannon divergence between per-step output distributions under small input perturbations, and localized attacks satisfy V(h) ≤ 1 − Rclass(h).For non-localized attacks, the work proposes a calibrated empirical model; for discrete-output tasks, ASR consistently estimates vulnerability for a fixed attack family.Source: Introduction · pp. 8, 10

Experimental findings

  • 73.8% ASR was achieved by ASA against LLM-as-a-Judge systems under black-box access, with transfer between open models reaching TSR = 62.6%.ASA combines token-level mutations with semantic paraphrasing; the 73.8% ASR was also measured against Gemma-3-4B-Instruct on MT-Bench and Kaggle tasks.Source: Introduction · pp. 6, 8
  • On TDC2023 data with Pythia-1.4B, surrogate triggers reached REASR ≈0.99 while true-trigger Recall was ≈0.17, near the ≈0.14 baseline.The work attributes this asymmetry to combinatorial growth in the discrete trigger space and prioritizes preventive data control over post-training diagnostics.Source: Introduction · p. 8
  • Four classes of multi-step injection bypasses were systematized on SaTML CTF 2024, with input normalization and context-isolated system instructions identified as countermeasures.The classes are code-oriented attacks, ASCII encoding, word splitting, and contextual misdirection; multi-layer defense reduced ASR from 90% to 15–25%.Source: Introduction · pp. 9–10

Implementation and validation

  • JudgeGuard, TrojanArmor, and MCPSec implement the proposed evaluation and defense methods, and Viasat applied the judge-vulnerability framework, committee defense, and multi-step attack methodology in production.The cited deployment was in a recommendation-ranking module built on an LLM-as-a-Judge architecture.Source: Introduction · pp. 9–10

Experimental findings

  • A committee of 5–7 heterogeneous models reduced Gemma-3-4B ASR by 47–55 percentage points, reaching 19.3% with seven models.The Condorcet justification concerns majority accuracy under independent errors and base accuracy above 0.5; for Gemma-3-4B, heterogeneous composition is the stated source of gains.Source: Introduction · pp. 9–10
  • On 847 MCPBench scenarios, the MCP defense approach reduced average ASR from 53.7% to 12.4%.AttestMCP restricts tool calls and protects packet integrity with HMAC at under 0.1 ms per call; Commit Boundary isolates agent-generated code.Source: Introduction · pp. 9–10

Introduction

  • Objective 1 is addressed in Chapter 1 through the practical asymmetry of trigger generation and reconstruction and the classification of multi-step attacks on multi-layer defenses.These correspond to novelty results 3 and 4, respectively.Source: Introduction · p. 12
  • Objective 2 is addressed in Chapter 2 through Rstab(f), theoretical vulnerability bounds, and the framework underlying the Condorcet jury theorem justification for committees.It corresponds to novelty result 1 and statement for defense 1; the committee justification is part of statement for defense 3.Source: Introduction · p. 12
  • Objective 3 is addressed in Chapter 3 through the LLM-as-a-Judge vulnerability framework and ASA algorithm, corresponding to novelty result 2 and statement for defense 2.Source: Introduction · p. 12
  • Objective 4 spans Chapters 3 and 4, covering heterogeneous-model committees and experimental confirmation of novelty results 2–5 and statements for defense 2–4.It corresponds to novelty result 5 and statement for defense 3.Source: Introduction · p. 12
  • Objective 5 is addressed in Chapters 5–6 through software implementation and validation, corresponding to novelty result 6 on agentic-system and MCP security analysis and statement for defense 5.Source: Introduction · p. 12

Analysis of Threats and Vulnerabilities of Large Language

This section defines and systematizes LLM threats and vulnerabilities, including their attack relationships, dependencies, and variation across model changes, query contexts, and security properties. It establishes ethical limits and mitigation priorities while connecting the threat analysis to later robustness and defense work, with attention to prompt injection, jailbreaks, trojans, and agentic systems using external tools.

1.1.2 LLM Robustness: Definition and Significance

  • R_stab(f) measures changes in output distributions under small input perturbations, while robustness means preserving correct behavior under semantically insignificant or syntactically admissible changes.The allowed perturbations depend on the distance measure used over token sequences.Source: §1.1.2 · pp. 14–16

1.1.3 LLM Vulnerabilities and Attack Types

  • The threat taxonomy includes adversarial attacks, prompt injection, jailbreaking, backdoors, data extraction, and tool hijacking in agentic systems.These attacks can target outputs, safety instructions, confidential information, or system operation.Source: §1.1.3 · pp. 16–17
  • MCP tool hijacking can turn prompt injection into unauthorized calls with arbitrary arguments, potentially enabling code execution, file modification, or network requests.The threat is amplified by absent capability attestation, unauthenticated sampling, and implicit trust propagation between servers.Source: §1.1.3 · p. 17
  • R_stab(f) and V_local(f) measure different properties, and their relationship is not guaranteed: output-distribution shifts need not increase undesirable-output probability.Experiments indicate a correlation between increased R_stab(f) and decreased V_local(f), but no rigorous dependence has been established.Source: §1.1.3 · p. 17

1.1.4 Categorization and Formalization of Specific Aspects of Vulnerabilities

  • Fine-tuning can reduce robustness or worsen vulnerabilities, measured through changes in R_stab(f) or divergence between pre- and post-tuning output distributions.A negative ∆R signals reduced robustness, while a large ∆P indicates a noticeable behavioral shift that may correlate with a changed vulnerability profile.Source: §1.1.4 · p. 17
  • Jailbreak effectiveness is formalized as defense-bypass probability, while context-dependent vulnerability captures how attack success varies across query domains.Substantial variation across contexts indicates a need for context-adaptive defenses.Source: §1.1.4 · pp. 17–18
  • Model compromise can evade standard tests until a trigger or rare scenario occurs; detection can compare current behavior with a trusted reference using mean or maximum KL divergence.Choosing the reference model and interpreting deviations remain open research questions.Source: §1.1.4 · p. 18

1.2.1 Basic Definitions and Formal Metrics

  • The threat formalism includes confidential-data leakage risk and models prompt injection, training-data poisoning, and trojans as distinct threat classes.Leakage can be defined through verbatim extraction of training fragments or more generally through mutual information between outputs and secrets.Source: §1.2.1 · p. 19, §1.2.2 · p. 19

Threat Classification According to OWASP Top 10 for LLM (2023)

  • The OWASP mapping relates the section’s threat list to the 2023 Top 10 for LLM Applications, a classification focused on practical development and operational risks.The table presents correspondences between OWASP categories and the detailed threat list.Source: Threat Classification According to… · p. 22, Table 1.2 · p. 23

1.2.2 Description of the Main Threat Types

  • The threat inventory spans 17 types, including model inversion [52], membership inference [54], adversarial examples [1] [3], denial of service [55], and insecure output handling.Other listed risks include model theft, training-data extraction, sensitive-information disclosure, bias amplification, supply-chain vulnerabilities, jailbreaking, reproducibility issues, insecure plugins, and detection evasion.Source: §1.2.2 · p. 20, Figure 1.2 · p. 23

1.2.3 Threat Categorization

  • The taxonomy groups threats into six security categories, spanning integrity, confidentiality, availability, governance and control, ethics and fairness, and reliability and reproducibility.Some threats cross categories: prompt injection can cause data leakage, integrity violations, and malicious content generation.Source: §1.2.3 · p. 22, Figure 1.2 · p. 23

Mapping to NIST SP 800-53 Security Controls

  • The NIST SP 800-53 Rev. 5 mapping connects LLM threats to control families, while specific controls should be selected through risk assessment.This mapping supports integrating LLM risk management into an organization’s broader information security management system.Source: Mapping to NIST SP 800-53 Security… · p. 24, Table 1.3 · p. 24

1.2.5 Illustrative Threat Examples

  • Controlled examples show prompt injection can manipulate LLM-as-a-Judge decisions or justifications, surrogate trojan triggers are easier to generate than true triggers, and multi-layer defenses can be bypassed.The attacks include CUA and JMA implemented with optimization methods such as GCG; the examples motivate preventive measures and ongoing adaptation of defenses.Source: §1.2.5 · p. 25

1.2.6 Analysis of Threat Interrelations

  • Threats interact: extraction is a special case of information disclosure, injections can enable jailbreaks, and nondeterminism complicates both attacks and security testing.The section also describes links between bias and confidentiality, compounding supply-chain and plugin risks, and trade-offs among utility, confidentiality, robustness, and accuracy.Source: §1.2.6 · pp. 25–26

1.2.7 Conclusions and Risk Mitigation Recommendations

  • Risk mitigation combines preventive training-stage measures, operational defenses, and continuous monitoring and risk management.Examples include data auditing, privacy-enhancing methods, adversarial training, input and output sanitization, access controls, resource limits, red teaming, and regular updates; R_stab(fθ) and V_local(fθ) track security changes.Source: §1.2.7 · pp. 26–27

1.2.8 Formalization of the Threat Space for Large Language Models

  • The formal threat space for an LLM is defined as the tuple T = (Atype, V, IAV, CVV), abstracting threat components and their interrelations from specific attack implementations.The section presents this formal structure as a way to reveal general patterns across threats.Source: §1.2.8 · p. 27
  • Remediation should prioritize root vulnerabilities in acyclic graphs and source strongly connected components in cyclic graphs, since addressing them can reduce dependent vulnerabilities and overall risk.The dependency graph exposes cascading effects and root causes, and requires security-audit work to construct.Source: §1.2.8 · pp. 27–28
  • The threat-space model links attack types to vulnerabilities and vulnerability dependencies, while aggregate robustness weights attack success probabilities across attack types.Risk assessment also accounts for potential damage; its estimates depend on specialized testing or expert judgment and on application context.Source: §1.2.8 · pp. 28–29

1.3 Risks, Limitations, and Ethical Aspects

  • Experiments were conducted in controlled settings under commercial API terms, without real personal or confidential data, and with filtering to prevent dangerous content generation or storage.The research aimed to identify vulnerabilities and develop defenses; ready-made attack tools were not published.Source: §1.3 · p. 30

Chapter Conclusions

  • Attacks from prompt injection to trojans are framed as sensitivity to discrete-token input variations, motivating a unified robustness metric rather than attack-specific measures.The conclusions also identify MCP agentic-system attacks as a gap in OWASP Top 10 and NIST SP 800-53 catalogs, and note that security, utility, confidentiality, and robustness cannot generally all be optimized simultaneously, motivating the RUT metric.Source: Chapter Conclusions · p. 30

Evaluating the Robustness of Large Language Models

This section develops a framework for evaluating LLM robustness to admissible attacks, defining stability, output-sensitivity, vulnerability, and attack-risk metrics. It relates these metrics to defenses and composed systems, including majority-vote ensembles, and presents analytical and empirical tools whose guarantees depend on stated assumptions and scope.

2.1.1 Formalization of the Robustness Evaluation Problem

  • The general evaluation problem considers worst-case risk over admissible attacks, distinguishing distributional model f from decision operator h and requiring bounded or normalized divergence measures.The attack family, adversary goals, and consequences specify the threat model.Source: §2.1.1 · pp. 31–32

2.1.2 Assumptions About the Adversary

  • Adversary assumptions span black-box, gray-box, and white-box access, while query and resource constraints shape the admissible attack space and evaluation outcome.The GCG method requires white-box access to compute gradients.Source: §2.1.2 · p. 32

2.1.3 Classification of Adversarial Actions

  • Adversarial actions include semantically equivalent reformulations, malicious context injections, tokenization manipulations, and denial-of-service attacks, each requiring dedicated evaluation and defenses.Tokenization attacks can split forbidden words into fragments that filters do not individually detect.Source: §2.1.3 · p. 33

2.2.1 The Metric Rclass(h) for Discrete Outputs

  • Rclass(h) is the expected probability that a deterministic decision operator preserves its discrete output under small input perturbations, with values from 0 to 1.The neighborhood is determined by a distance measure such as Hamming, Levenshtein, or semantic distance.Source: §2.2.1 · p. 34

2.2.2 The Metric Rstab(f) for Generative Models

  • Rstab(f) measures generative-model sensitivity by averaging divergence between next-token distributions under an original and perturbed input along the same reference trajectory.The work uses Jensen–Shannon divergence with base-2 logarithms and Cmax = 1 for its evaluations, yielding finite, normalized values.Source: §2.2.2 · pp. 35–36
  • Rstab(f) depends on the reference trajectory and teacher-forcing protocol, and its directed sensitivity measure can differ when the original and perturbed inputs are reversed.The protocol must be fixed when comparing models; symmetrization is available when a symmetric estimate is required.Source: §2.2.2 · p. 35

2.2.3 The Global Vulnerability Index V (f)

  • The global vulnerability index V(f) evaluates the expected effectiveness of attacks over the input distribution, with undesirable outputs defined by the task and attack goals.Its values range from 0 to 1; deterministic decoding and stochastic output protocols are distinct definitions.Source: §2.2.3 · pp. 36–37

2.2.4 The Risk Function Risk(a, f)

  • Attack risk combines an attack’s success probability, which can be estimated by ASR, with the expected severity of consequences from an undesirable output.The risk is defined for a specific attack or attack type against a decision operator.Source: §2.2.4 · p. 37

2.3.1 The Monotonicity Property of Rclass(h) and Rstab(f)

  • Strengthening a defense pointwise cannot decrease either Rclass(h) or Rstab(f), making monotonicity a consistency property of the metric definitions.The result follows from the pointwise definition of a strengthened defense and linearity of expectation; which transformations satisfy that condition is established experimentally elsewhere.Source: §2.3.1 · pp. 38–39

2.2.4 The Risk Function Risk(a, f)

  • For localized attacks, V(h) ≤ 1 − Rclass(h); for non-localized attacks, vulnerability is approximated by a calibrated empirical relation, not a universal bound.The risk-based robustness definition AdvRobustness(f) = 1 − sup_a Risk(a, f) assumes risk is normalized to [0, 1]. The empirical relationship is calibrated on data and can vary across model and attack families; no direct rigorous bound from Rstab to vulnerability is established.Source: §2.2.4 · p. 38, §2.3.2 · pp. 39, 41

2.3.3 Lemma on Model Composition

  • For deterministic composition h = f ◦ g, Rclass(h) ≥ Rclass(g), so a robust first component preserves robustness under the stated composition order.This is a worst-case result for deterministic components; reverse-order composition, stochastic decoding, or feedback loops are not covered, and stochastic extensions require separate justification.Source: §2.3.3 · pp. 41–42

2.4.1 Upper Bounds on Rclass(h) for Bounded ε

  • For bounded ε, Rclass(h) is upper-bounded by one minus the probability that two random inputs within ε produce different outputs.The result uses independent inputs drawn from the same distribution; more frequent output disagreement among nearby random pairs lowers the upper bound on robustness.Source: §2.4.1 · pp. 42–43

2.4.2 Estimating Vulnerability to Adaptive Attacks

  • The pointwise-optimal adaptive vulnerability Vopt_adapt(f) bounds average vulnerability for any attack distribution supported on the available adaptive strategies.It models an adversary selecting an optimal strategy separately for each input and is at least as large as vulnerability under a fixed attack distribution or worst-case fixed attack.Source: §2.4.2 · p. 43

2.4.3 Asymptotic Properties as the Dataset Size →∞

  • As the sample size grows, the empirical estimate of Rclass converges in probability to its true value, with analogous results for Rstab and vulnerability under appropriate moment conditions.The convergence follows from the law of large numbers for bounded variables; sufficiently large datasets make the estimate arbitrarily close with probability arbitrarily near one.Source: §2.4.3 · pp. 43–44

2.5.1 LLM + Retriever Pipelines

  • LLM pipelines with retrievers depend on the robustness of both the language model and the retrieval component.The section also frames robustness evaluation for recursive agent chains and majority-vote ensembles, but the supplied passage gives no specific findings for those systems.Source: §2.5 · p. 44, §2.5.1 · p. 44
  • For an LLM–retriever pipeline, the composite Lipschitz constant is bounded by K_s ≤ K_f,D · K_r + K_f,X.The plug-in estimate uses empirically estimated component constants and is a sensitivity estimate, not a certified global bound; the 99th percentile does not bound even observed ratios.Source: §2.5.1 · pp. 44–45

2.5.2 Recursive Chains of LLM Agents

  • Under independent, homogeneous agent errors and cascading failure, a chain’s vulnerability is V_d = 1 − (1 − V_1)^d and increases with depth.With positively associated errors or partial compensation, actual vulnerability can be lower; the formula is a coarse bound, not a universal law for arbitrary chains.Source: §2.5.2 · p. 46

2.5.3 Majority-Vote Ensembles

  • A discrete-output majority-vote ensemble remains robust when at least half its models preserve a common verdict across the perturbation set.With independent model errors and individual accuracy r > 0.5, committee accuracy approaches one as odd committee size grows; increasing R_class requires the stronger condition that each model’s robust-correctness probability exceeds 0.5 independently.Source: §2.5.3 · pp. 47–48

2.6.1 Completeness Criteria

  • The proposed metrics rate highly on invariance and sensitivity, but exact computation is more resource-intensive than overlap- or perplexity-based measures.The comparison is qualitative and also assesses computational feasibility.Source: §2.6.1 · p. 48, Table 2.1 · p. 48

2.6.2 Application of the Metrics: Model Comparison

  • Chapter 4 applies R_stab, V, and Risk to experimental data to illustrate their computation.The supplied passage describes the application but reports no model-comparison results.Source: §2.6.2 · p. 49

2.6.3 Relation Between the Theoretical Metric Rstab(f) and Empirical Indicators

  • For fixed attack family Aeval, 1 − ASR consistently estimates population 1 − V(h), which under stated assumptions upper-bounds R_class(h); its use as a proxy for R_stab(f) requires separate empirical justification.For a finite sample of size n, the inequality holds up to a random error of order O_p(n^-1/2).Source: §2.6.3 · p. 49

2.7 Chapter Conclusions

  • The chapter formalizes R_class for classification, R_stab for generative models, V for global vulnerability, and Risk for a specific attack, and bounds localized-attack vulnerability by 1 − R_class under clean-input assumptions.Exact computation is costly, the rigorous vulnerability bound covers only localized attacks, and transfer to R_stab for generative models remains open.Source: §2.7 · pp. 49–50

Chapter Overview

This chapter connects local model sensitivity with overall robustness, then systematizes methods for changing that sensitivity in discrete token spaces. It also considers the trade-off between robustness and model utility.

Chapter Overview

  • The chapter establishes necessary and sufficient conditions connecting local sensitivity to the overall robustness metric Rstab(f).It then uses this connection to organize robustness-improvement methods.Source: Chapter Overview · p. 51
  • It systematizes architectural modifications, training methods, and multi-layer defenses that operate in discrete token space.Source: Chapter Overview · p. 51
  • The chapter considers the trade-off between robustness and the model’s functional utility.Source: Chapter Overview · p. 51
  • Sections 3.1–3.3 present known approaches, while Section 3.4 presents methods proposed by the thesis author.Source: Chapter Overview · p. 51

3.1 Theoretical Foundations of Robustness Improvement

This section formalizes robustness for language models in discrete token spaces and relates local output sensitivity to overall robustness. It also identifies the perturbation metrics and evaluation indicators used in the chapter.

3.1.2 Robustness Metrics for LLMs

  • Rstab(f) uses the output divergence DT, and values close to 1 correspond to high robustness.ASR, TSR, Recall, and REASR evaluate individual aspects of robustness, with definitions and protocols given in Chapter 4.Source: §3.1.1 · p. 51, §3.1.2 · p. 53

3.1.1 Formalization of Robustness and Local Sensitivity

  • High Rstab(f) implies that maximal output-distribution changes under small perturbations are bounded on average over the data distribution.For small δR, the analysis also bounds the fraction of inputs with high local sensitivity.Source: §3.1.1 · pp. 51–52
  • A uniform bound on local sensitivity over allowed perturbations is sufficient for a lower bound on Rstab(f).The chapter connects this condition to practical methods such as smoothing and ensembling.Source: §3.1.1 · pp. 52–53
  • The choice of discrete distance metric and perturbation budget determines which input changes robustness is evaluated against.The chapter names Hamming, Levenshtein, and semantic distances, corresponding to token substitution, insertion, and deletion.Source: §3.1.1 · pp. 51, 53

3.2 Architecture-Level Methods for Improving Robustness

Architecture-level robustness methods range from model modifications and ensembles to layered request filtering. The section also qualifies their guarantees and describes how the layers screen inputs, model behavior, and outputs.

3.2.1 Modification of the Transformer Architecture

  • Attention smoothing, entropy regularization, clipping, quantization, and stochastic layers target sensitivity, while representation methods adapt normalization or train embeddings and hidden states for robustness.The listed attention modifications lack rigorous guarantees of improved R_stab(f), although empirical work reports increased robustness to some attacks; architectural modifications may require retraining and affect performance and cost.Source: §3.2.1 · pp. 54, 56
  • 47–55 percentage points: committees of 5–7 diverse models reduce ASA attack ASR relative to undefended Gemma-3-4B, whose highest open-model ASR estimate is 73.8%.The section reports these committees as increasing robustness in LLM-as-a-Judge systems.Source: §3.2.1 · p. 54
  • For jointly convex divergences, the fixed-weight probability-averaged ensemble’s output divergence is bounded by the average member divergence.For the step-wise R_stab bound, the models must share a reference trajectory; the lemma does not cover majority voting over decoded verdicts.Source: §3.2.1 · pp. 54–55

3.2.2 Multi-Layer Defense with a Focus on Robustness

  • Multi-layer defense screens requests at input, model, and output stages, using normalization and filters before inference and checking generated responses before return.Independent layers target different threats, and early rejection can reduce average latency by preventing anomalous requests from reaching inference.Source: §3.2.2 · pp. 56–57

3.3 Training-Level Methods for Improving Robustness

Training-level approaches seek robustness through adversarial examples, regularization, augmentation, and distillation. Their benefits are bounded by accuracy and computational trade-offs, and robustness may not transfer across perturbation types or threat settings.

3.3.1 Robust Training

  • Adversarial training optimizes against worst-case examples, while regularization and augmentation offer alternatives that encourage smoothness or expose models to perturbed inputs.Regularization is usually less costly but may have a less pronounced effect against targeted attacks; adversarial training can reduce clean-data performance and require substantial compute.Source: §3.3.1 · pp. 58–59

3.3.2 Knowledge Distillation for Improving Robustness

  • Distillation can transfer robustness from a robust teacher or ensemble, and variants train on adversarial or augmented examples alongside teacher outputs.The section also describes standard distillation as combining hard-label and soft teacher-label losses.Source: §3.3.2 · p. 59

3.3.3 Theoretical Limitations and Trade-offs

  • Robustness has an accuracy and compute trade-off: robust models may lose fine distinctions needed for maximal accuracy, and adversarial-example generation increases training cost.The section also notes that regularization can be cheaper than adversarial training but less effective against targeted attacks.Source: §3.3.1 · pp. 58–59, §3.3.3 · p. 60
  • Robustness is not guaranteed to generalize across perturbations: token-substitution robustness may leave vulnerability to paraphrases or character-level attacks, and defenses can increase vulnerability to other classes.Evaluation also depends on the metric d and budget ϵ; robustness at small ϵ may not hold at larger budgets, and perfect robustness to all admissible perturbations is incompatible with high accuracy.Source: §3.3.3 · p. 60

3.4 Algorithmic Implementation and Complexity Analysis

This section gives reproducible algorithmic procedures for model committees and ASA, with complexity considerations. ASA adapts evolutionary search to discrete prompt attacks under black-box access, while its success depends on search settings and is not guaranteed to reach a global optimum.

3.4.1 Pseudocode of the Model Committee Pipeline

  • A model committee independently queries an odd-sized set of models, aggregates decoded verdicts, and can flag low agreement for manual review.Parallel execution time is determined by the slowest model; sequential execution time is the sum of model inference times.Source: §3.4.1 · pp. 60–61

3.4.2 Conceptual Algorithm of the ASA Attack

  • ASA uses evolutionary search in discrete token space to generate attack prompts for LLM-as-a-Judge and protected systems under black-box access.Its fitness combines attack success and stealth, and its mutations preserve linguistic structure through synonym substitution, paraphrasing, or distracting-fragment insertion.Source: §3.4.2 · pp. 61–62
  • ASA’s query complexity is O(N · G), with N prompts evaluated per generation across G generations; final selection reuses stored fitness values.Larger population and generation counts increase the chances of finding an effective attack but require proportionally more queries.Source: §3.4.2 · p. 61
  • ASA convergence to the global optimum is not guaranteed, although retaining the best candidate with unchanged fitness makes the population’s best objective value nondecreasing.This guarantees a returned prompt at least as good by the objective as the best initial-population prompt, but not strict improvement.Source: §3.4.2 · p. 62

Chapter Conclusions

The conclusions connect local sensitivity to the robustness metric and distinguish probability-averaged ensembles from verdict-voting committees. They also emphasize that defenses retain trade-offs and do not automatically transfer across attack classes.

Chapter Conclusions

  • Bounded local sensitivity is sufficient for high R_stab(f), while R_stab(f) = 1 − δR implies a small average maximal divergence of Cmax · δR.Together, these results connect local sensitivity with the global behavior captured by the metric.Source: Chapter Conclusions · p. 62
  • For jointly convex divergences, probability averaging bounds ensemble sensitivity by average member sensitivity, but the result requires step-wise averaging and a common reference trajectory for D_T.The lemma does not cover committees that vote over decoded verdicts, which are reported as the most effective defense in Chapter 4.Source: Chapter Conclusions · p. 62
  • No described method removes the robustness–accuracy trade-off, and defense against one perturbation class does not automatically transfer to others.Practical applicability also depends on the chosen metric d and perturbation budget ϵ.Source: Chapter Conclusions · p. 62

Experimental Evaluation of Robustness Enhancement Methods

This section evaluates LLM vulnerabilities and defenses using standardized and original metrics, considering trade-offs among robustness, utility, and computational cost. It examines adaptive attacks, Trojan-trigger recovery, heterogeneous committees, and a preliminary comparison of generative and discrete metrics under benign perturbations.

4.1.1 Robustness Evaluation Criteria

  • The chapter uses ASR and TSR for judge attacks, BLEU-like Recall and REASR for trojans, and normalized cost and utility changes for the robustness-utility trade-off.Trojan Recall and REASR are protocol-dependent surrogate indicators, not classical binary detection recall or binary attack success; 1−ASR is an empirical estimate relative to a fixed attack set and protocol, not absolute robustness.Source: §4.1.1 · pp. 63–64
  • Exact R_stab computation requires full per-step output distributions and perturbation enumeration, so experiments use 1−ASR as an empirical proxy for the fixed attack family.For localized attacks, the cited theoretical result bounds R_class by 1−V(h); the chapter’s non-localized prompt-injection attacks rely instead on a calibrated empirical model.Source: §4.1.1 · pp. 64–65

4.1.2 Evaluation Datasets

  • Per-task evaluations used 50 prompts, with results aggregated over four tasks to 200 observations per model; aggregate bootstrap intervals were ±5.5–6.9 percentage points.At the per-task level, power is insufficient for small differences; aggregated evaluation has 0.80 power for differences of about 14 percentage points or more near ASR 0.5.Source: §4.1.2 · pp. 65–66

4.1.3 Experimental Procedure

  • The procedure tested open and commercial models with greedy decoding, attack and defense scenarios, task-specific metrics, and bootstrap-based statistical analysis.The studied models included Gemma-3, Llama-3.2, Pythia, GPT-4, Claude-3-Opus, and GPT-3.5; judge-attack success meant a score change of at least 2 points or a changed comparative verdict.Source: §4.1.3 · pp. 66–67

4.2.1 Vulnerability of LLM-as-a-Judge to Prompt Injection Attacks

  • ASA reached 73.8% ASR against four of five judge models, and attacks transferred between open models at TSR = 50.5–62.6%.Gemma-3-4B-Instruct was most vulnerable under three of four attacks; GPT-4 and Claude-3-Opus remained vulnerable at ASR = 25.3–45.7%, but their ASA difference was not distinguishable at this sample size. System-prompt attacks were considerably more effective than user-content attacks.Source: §4.2.1 · p. 68, Table 4.1 · p. 68
  • Committees of 5–7 diverse models reduced ASA ASR to 19.3–26.8%, while individual defenses and defense combinations did not fully protect judge systems.The likely mechanism is decorrelated errors across architectures; homogeneous committees produced smaller ASR reductions. The proposed committee-accuracy corollary does not by itself explain this heterogeneous-committee effect.Source: §4.2.1 · p. 69, Table 4.2 · p. 69

4.2.2 Trojan Detection in LLMs

  • The TDC2023 experiments on Pythia models sought to reconstruct trigger phrases from suspicious models and their target malicious strings.Table 4.3 is identified as a comparison of trojan-detection methods, but the supplied passages do not report its results.Source: §4.2.2 · p. 69, Table 4.3 · p. 69
  • Trojan-trigger Recall reached only about 0.17 against a 0.14 random baseline, although REASR was close to 1.0, indicating poor recovery of original triggers.The experiments used Pythia-1.4B; the hypothesis that this gap grows with model size remains unverified. The random baseline reflects accidental n-gram matches in BLEU scoring.Source: §4.2.2 · pp. 69–70

4.2.3 Attacks on LLMs with a Multi-Layer Defense

  • On SaTML CTF 2024 data, the three-layer defense lowered ASR from about 90% to 15–25%, but no filter combination blocked all four bypass classes.Code formatting, numeric encoding, and word splitting can be countered with input normalization; semantic contextual redirection requires context-isolated system instructions. More capable models were vulnerable to sophisticated vectors, while less capable models yielded to simpler requests.Source: §4.2.3 · p. 70

4.2.4 Analysis of Parameter Effects and Trade-offs

  • Heterogeneous committees reduced ASA’s ASR on Gemma-3-4B from 73.8% to 19.3%, with computational cost growing linearly in committee size.For seven models, the reported RUT was approximately 5.5, with normalized latency cost ≈0.08 and utility loss ≈0.02. Error decorrelation from architectural diversity is a hypothesis consistent with the results, not a guarantee; homogeneous committees and minimum diversity remain open questions.Source: §4.2.4 · p. 71, Chapter Conclusions · p. 73

4.2.5 Empirical Validation of the Relation between Rstab(f) and Rclass(h)

  • On 20 benign prompts, GPT-4o-mini had approximate Rstab(f) = 0.115 versus Rclass(h) = 0.050, consistent with—but not confirming—the hypothesis that Rstab(f) is no lower.The comparison is summarized in Table 4.4. Rstab(f) can retain positive distributional consistency when the discrete output changes; results are limited by the small benign sample, top-20 log-probability approximation, and API generation without teacher forcing, and do not establish behavior under adversarial perturbations.Source: Table 4.4 · p. 72, §4.2.5 · p. 72

Chapter Conclusions

  • ASA reached 73.8% ASR on Gemma-3-4B, with 50.5–62.6% transferability between open models; GPT-4 and Claude-3 still had 42.9–45.7% ASR.These results mean hiding model architecture is not a justified defense strategy. Heterogeneous committees reduced ASR by 47–55 percentage points relative to the most vulnerable undefended model, but their cost and reliance on composition diversity limit applicability.Source: Chapter Conclusions · pp. 72–73

Software Implementation of the Suite for Evaluating and Improving LLM Robustness

JudgeGuard evaluates LLM-as-a-Judge robustness, while TrojanArmor detects and analyzes LLM trojans; both suites use modular components to generate attacks, route model requests, evaluate results, and store metrics.

Chapter Overview

  • The JudgeGuard and TrojanArmor suites automate the robustness-evaluation cycle, from attack generation and response collection to metric computation.JudgeGuard targets judge models; TrojanArmor targets trojan detection and analysis.Source: Chapter Overview · p. 74

5.1 Architectural Model of the JudgeGuard and TrojanArmor Software Suites

  • Both suites share a four-component architecture, despite targeting different tasks: judge-model robustness and LLM trojan analysis.The modules are the Attack Generator, Model Router/API Gateway, Evaluation Engine, and Storage Layer.Source: §5.1 · p. 74, §5.1.1 · p. 74

5.1.1 Modular Structure of the System

  • The Attack Generator creates adversarial prompts or candidate triggers; the router accesses local or cloud models, the evaluator computes metrics, and storage retains results.The evaluator supports independent metric functions, while results can be stored as JSON/JSONL or in PostgreSQL.Source: §5.1.1 · pp. 74–75

5.1.2 Component and Data Flow Diagram

  • The data flow runs from attack generation through model routing, evaluation, and storage, with fitness scores feeding back to iterative attacks such as ASA.The feedback steers subsequent evolutionary-search iterations.Source: §5.1.2 · p. 75, Figure 5.1 · p. 75

5.1.3 Data Processing Pipeline

  • The processing pipeline composes configuration and data loading, attack generation, model-response collection, metric computation, and result storage.Separate Python modules with fixed interfaces allow independent testing, replacement, and parallel execution; batching improves local GPU utilization.Source: §5.1.3 · pp. 75–76

5.2 Choice of the Technology Stack and Tools

The software stack was selected for LLM performance, ecosystem compatibility, and deployment ease. In the reported comparison, vLLM increased throughput and shortened evaluation time relative to the HuggingFace pipeline.

5.2.1 Core Technologies

  • Python, PyTorch, and HuggingFace Transformers provide the development, deep-learning, and pretrained-model foundations for the suites.The stated capabilities include asynchronous API interaction, GPU acceleration, mixed precision, automatic differentiation, and a unified inference API.Source: §5.2.1 · p. 76
  • vLLM uses PagedAttention and continuous request batching to improve GPU memory management and inference throughput.PagedAttention divides the attention KV cache into fixed-size blocks managed like virtual-memory pages.Source: §5.2.1 · p. 76, §5.2.2 · p. 76
  • FastAPI and Uvicorn support the router API, while JSONL files and PostgreSQL provide intermediate and long-term results storage.JSONL supports fast access and versioning; PostgreSQL supports analytical SQL queries.Source: §5.2.1 · p. 76

5.2.2 Performance Comparison of vLLM and the Standard HuggingFace Pipeline

  • 5× throughput and evaluation time cut from 47 to 10 minutes were reported for vLLM versus the HuggingFace pipeline.The 1,000-prompt evaluation used about eight model calls per prompt because of committee evaluation.Source: §5.2.2 · pp. 76–77, Table 5.1 · p. 77

5.3 Module for Evaluating the Robustness of Agentic LLM Systems (MCPSec)

The MCPSec section examines how MCP’s architecture exposes agentic systems to tool hijacking, then evaluates protocol vulnerabilities and defenses combining tool-call attestation with isolated code execution. On MCPBench, MCPSec reduces average attack success rate from 53.7% to 12.4%.

5.3.3 Protocol Vulnerabilities: From Prompt Injection to Tool Hijacking

  • MCP prompt injection can hijack tool calls because agents choose tools using context that may contain untrusted data.The specification analysis identifies absent capability attestation, unauthenticated sampling, and implicit trust propagation between servers.Source: §5.3.3 · pp. 78–79

5.3.4 Experimental Evaluation of Protocol Vulnerabilities: MCPBench

  • MCP architectural decisions raise attack success rates by 16.6–41.6 percentage points versus non-MCP integrations; cross-server attacks show the largest increase, 41.6 points.The reported values cover studied agent implementations, selected tool scenarios, and specific MCP configurations, not all agentic architectures.Source: §5.3.4 · p. 79, Table 5.2 · p. 79

Algorithm for Cryptographic Attestation and Message Routing (AttestMCP)

  • AttestMCP combines a fixed permission table with HMAC packet integrity checks to restrict tool calls to pre-authorized tools and arguments.The permission table is established before untrusted data enter context and remains unchanged during execution; the signature authenticates packet source and integrity, not call intent.Source: Algorithm for Cryptographic… · p. 80
  • An injection can still make the agent call an allowed tool with allowed arguments for a malicious purpose, so AttestMCP does not eliminate that residual risk.The permission table bounds available actions, while narrower argument specifications and Commit Boundary narrow the remaining abuse space.Source: Algorithm for Cryptographic… · p. 81

Algorithm for Software Execution Isolation: The Commit Boundary Pattern

  • The Commit Boundary pattern runs LLM-generated code in a container and validates its output before returning it.Static analysis, container isolation, and output validation act in sequence; high-risk operations require explicit human approval.Source: Algorithm for Software Execution… · pp. 81–82
  • 0.3–0.8 s is the total Commit Boundary overhead on a cold start, described as comparable to LLM code-generation time and acceptable for interactive scenarios.Container allocation takes 200–500 ms cold, or about 50 ms with pre-created containers.Source: Algorithm for Software Execution… · p. 82

Components of the Integrated Defense in the MCPSec Module

  • The MCPSec module combines AttestMCP and Commit Boundary with four additional mechanisms: capability certificates, origin tagging, isolation enforcement, and replay protection.The added mechanisms restrict server operations, distinguish message sources, block cross-server transfer without user permission, and prevent replay of intercepted messages.Source: Components of the Integrated Defense… · p. 82
  • 53.7% to 12.4% is the reduction in average attack success rate on MCPBench; added steady-state processing latency for attestation and filtering is at most 8.3 ms per message.The latency excludes the perplexity filter and content moderation; sandboxed code execution additionally incurs about 50 ms with pre-created containers.Source: Components of the Integrated Defense… · p. 82

5.4 Evaluation of the Computational Complexity and Performance of the Software Suite

The evaluation measures defense overhead through context-dependent latency, inference throughput, multi-model scalability, and component-level cost. Across the reported configurations, defenses add limited latency relative to inference, while throughput and overhead vary with configuration and context length.

5.4 Evaluation of the Computational Complexity and Performance of the Software Suite

  • The evaluation measures latency as input context length changes and throughput across inference configurations.Request time includes preprocessing, inference, defense filters, and metric computation.Source: §5.4 · p. 83, §5.4.1 · p. 83

5.4.1 Latency Analysis as a Function of Context Length

  • 28% at context length 2048 and 12% at 8192 are the reported defense-component overheads; the filters do not become a bottleneck as context grows.The filters run in O(n), while inference grows superlinearly and asymptotically as O(n^2).Source: §5.4.1 · p. 83, Figure 5.3 · p. 83

5.4.2 Throughput Comparison Across Configurations

  • With all defense filters and metric computation enabled, vLLM-full throughput is 11.7 req./s, a 17% loss relative to vLLM-base at 14.1 req./s, for LLaMA-2-13B on NVIDIA A100.Source: §5.4.2 · p. 84, Figure 5.4 · p. 84

5.4.3 Scalability in Multi-Model Evaluation

  • 45 model inference requests per second is the measured throughput for four 13B models on four GPUs, below the estimated 54–55 req./s after interconnect loss.The passage attributes the gap to request orchestration and committee-verdict synchronization.Source: §5.4.3 · p. 85

5.4.4 Computational Overhead of the Security Algorithms

  • Table 5.3 reports total computational overhead of 28.4% of base inference; the perplexity filter contributes the largest share at 13.5%, whereas AttestMCP adds only 0.03%.Source: §5.4.4 · p. 85

Chapter Conclusions

The chapter reports software implementations and an MCP security module, with MCPBench results showing reduced attack success rate. It also notes implementation and hardware boundaries for the performance findings.

Chapter Conclusions

  • The MCPSec module includes MCPBench with 847 attack scenarios, AttestMCP, and Commit Boundary; its average ASR falls by 41.3 pp, from 53.7% to 12.4%, versus the unprotected configuration.The methods of Chapters 2–3 are implemented in the JudgeGuard and TrojanArmor suites.Source: Chapter Conclusions · pp. 85–86
  • The security components’ total computational overhead does not exceed 30% of base inference, and their relative share decreases as context length grows.The filters run in O(n), whereas inference grows superlinearly and asymptotically as O(n^2).Source: Chapter Conclusions · p. 86
  • Performance measurements used a single NVIDIA A100, so extrapolation to other accelerators requires verification; MRAG adapters also require updates when provider API versions change..Source: Chapter Conclusions · p. 86

Practical Application of Robustness Enhancement Methods

This section examines how robustness methods translate into practical deployment and evaluation, including JudgeGuard, model committees, MCPBench results, reference architectures, and open challenges. It also addresses domain-specific recalibration of defense metrics and proactive, specialized security throughout LLM development.

6.1.1 Stages of Deploying Robustness Enhancement Methods

  • Following risk-management and secure-development life cycles, the five deployment stages cover risk assessment, method selection, integration, testing, and ongoing monitoring and updating.The stages repeat throughout system operation, with testing and monitoring feeding iterative refinement.Source: §6.1.1 · pp. 87–88

From Reactive to Adaptive Defense

  • Defenses may lose effectiveness against unforeseen attacks, motivating continuous monitoring, retesting, adaptive red teaming, and self-healing or online updates despite their computational demands.Interpretability and vulnerability localization can help defenses generalize to new attacks [158]; explainable blocking and attack visualization are also proposed [159].Source: §6.1.1 · p. 88, From Reactive to Adaptive Defense · p. 93

6.1.2 Evaluation of Deployment Effectiveness

  • Deployment effectiveness combines security, utility, performance, and cost measures, while pilot projects check real-world latency, API and energy costs, and target-task quality.Measures include ASR, Recall, REASR, clean-data utility, computational overhead, ROI, RUT, and total cost of ownership.Source: §6.1.2 · pp. 89–90

6.2 Examples of Practical Application

  • Viasat’s JudgeGuard deployment reduced ASR by 42 percentage points on its internal dataset, with a three-model committee adding at most 15% latency versus one model.The internal dataset differs from Chapter 4’s experimental sample; TrojanArmor and MCPSec had benchmark validation only and were not deployed at this production site.Source: §6.2 · p. 90

6.2.1 Application in Natural Language Processing Systems

6.2.2 Application in Critical Systems

Multimodal Robustness and New Attack Surfaces

  • Formal robustness guarantees remain scarce, while multimodal defenses and metrics for separate and combined modality perturbations remain open problems.Formal verification tools currently apply only to small networks, though scaling abstract interpretation may enable verification of individual properties in billion-parameter models.Source: Theoretical Foundations and Formal… · pp. 92–93, Multimodal Robustness and New Attack… · p. 93
  • Federated training introduces backdoor-poisoning risks that differential privacy does not eliminate, while quantum resistance chiefly concerns public-key session-key establishment rather than AttestMCP’s HMAC.Participant-level privacy can limit backdoors at an accuracy cost; the passage states that 128-bit security remains adequate for symmetric primitives, while quantum acceleration of adversarial-example search remains theoretically unestimated.Source: Multimodal Robustness and New Attack… · p. 93

6.3.2 Technology Development Forecasts

  • Forecasts describe an automated attack-defense race and a shift toward security by design, standardized evaluation, training-stage guarantees, and possible certification for critical applications.The passage identifies LLM red teaming, model-generated attack scenarios, HELM, JailbreakBench, MT-Bench, TDC2023, Kaggle, and integration of RLHF/RLAIF with adversarial training.Source: §6.3.2 · pp. 93–94
  • Forecasts also anticipate specialized protections for expert routers and sector-specific applications, alongside growing demand for mathematical guarantees despite unresolved scaling of formal methods to billion-parameter models.Progress in abstract interpretation of neural networks is cited as grounds for optimism.Source: §6.3.2 · p. 94

Chapter Conclusions

  • At Viasat, a three-model committee reduced ASR by 42 pp on an internal dataset with latency increasing by no more than 15%, confirming production transfer but requiring domain recalibration.JudgeGuard and the committee-based defense were deployed in Viasat’s production environment; JudgeGuard and TrojanArmor cover LLM security testing, with MCPSec supporting agentic-system analysis.Source: Chapter Conclusions · p. 94, Conclusion · p. 96
  • RUT links technical and economic criteria to recompute an optimal defense configuration as the threat landscape changes, unlike either metric group alone.Production-readiness assessment requires technical, economic, and integral metrics such as RUT.Source: Chapter Conclusions · p. 94
  • Committee cost grows linearly with model count, while parallel queries make latency growth sublinear; a confidence-triggered cascade is promising, but its savings remain unquantified.A smaller committee, such as three models, could handle queries first, activating the full committee only when agreement confidence c is low.Source: Chapter Conclusions · p. 94

Conclusion

  • On MCPBench’s 847 scenarios, AttestMCP and Commit Boundary reduced average ASR by 41.3 percentage points, from 53.7% to 12.4%.AttestMCP restricts tool calls to a session permission table and checks packet integrity with HMAC; Commit Boundary isolates agent-code execution.Source: Conclusion · p. 95

List of Abbreviations and Notation

This section collects abbreviations and notation used in the work, then defines selected security, model, and robustness concepts as used in the paper.

List of Abbreviations and Notation

  • The abbreviations include attack and robustness measures such as ASR, TSR, REASR, and RUT, alongside terms for LLMs, MCP, and security mechanisms.ASR means Attack Success Rate; TSR, Transfer Success Rate; REASR, Reverse-Engineered Attack Success Rate; and RUT, Robustness-Utility Trade-off.Source: List of Abbreviations and Notation · pp. 98–99
  • The notation defines token sequences, model inputs and outputs, perturbation neighborhoods, divergence measures, and local and global vulnerability metrics.R_stab(f) denotes the generative model's robustness metric; R_class(h) denotes robustness of a deterministic decision operator.Source: List of Abbreviations and Notation · pp. 100–101

Glossary

  • The glossary defines threats including data poisoning, trojans, adversarial attacks, jailbreaking, and prompt injection, alongside related terms such as alignment and differential privacy.A trojan is hidden functionality activated by a special input trigger; prompt injection inserts hidden or disguised instructions into a query or external data.Source: Glossary · p. 102
  • The glossary describes committees as aggregating heterogeneous judge-model verdicts by majority vote, with low agreement flagging potentially malicious queries for manual review.It also defines LLM-as-a-Judge, multi-layer defense, tokens, embeddings, and robustness in the paper's usage.Source: Glossary · pp. 102–103
Loading 2610.02432v2…