PaperlayerView exact revision on arXiv

Source-linked AI summary

Security Assessment of DeepSeek Harness with A.I.G: Evaluating Resistance to Indirect Prompt Injection

Zonghao Ying, Xiangfan Wu, Huiyu Wu, Xing Zheng, Huangsheng Cheng, Xiaorong Shi, Jing Guo

arXiv:2608.16393v2cs.CR

TL;DR

Indirect prompt injection can turn untrusted content into sensitive actions in tool-using agents. This study evaluates DeepSeek Harness with controlled attacks and trace-level judges, finding success rates up to 25.5% depending on attack, channel, carrier, and judge.

  • Problem

    The central question is whether DeepSeek Harness prevents untrusted content from changing an agent’s actions, including sensitive tool use.

  • Method

    The study uses A.I.G. to run a controlled attack matrix against DSH and analyze normalized execution traces with rule-based and semantic judges.

  • Results

    Attack success reached 25.5% for hidden Unicode in file mode, while fake-completion and skills attacks reached 17.0% and 16.0%, respectively.

  • Takeaways & Limitations

    The findings place the relevant security boundary across the path from external content to sensitive action and identify model-visible results, additional contexts, and pre-execution guards as control points.

  • Takeaways & Limitations

    Sink tools are local fixtures that record attempted sensitive actions without producing real-world side effects.

Abstract

from arXiv · show

We assess indirect prompt injection in DeepSeek Harness (DSH), using AI-Infra-Guard (A.I.G) to construct tests, deliver controlled taint, execute DSH, collect traces, and judge outcomes. The study covers 14,560 controlled executions over 16 indirect-content channels, text and file carrier modes, 35 payload objectives, one unmodified baseline, and 12 attack methods. The experiment preserves DSH's agent loop, tool registry, model adapter, and session-event path; source tools and sensitive sinks are local fixtures, so attempted actions are recorded without external side effects. We evaluate each trace with a deterministic rule-based judge, \JudgeR{} (RuleJudge), and a semantic LLM-based judge, \JudgeL{} (LLMJudge). The strongest observed attack success rates are 17.0% under \JudgeL{} for fake-completion attack in text mode, 25.5% under \JudgeR{} for hidden Unicode in file mode, and 16.0% under \JudgeR{} for the skills channel in file mode. \JudgeL{} also assigns partial compliance more often than \JudgeR{} (7.3% versus 2.0%). We relate these results to DSH's treatment of tool results, additional contexts, and tool-call policy hooks, then identify controls that should sit between untrusted content and sensitive actions. Our code is available at https://github.com/Tencent/AI-Infra-Guard/tree/main/Research/deepseek-harness-security-assessment .

1. Introduction

This assessment tests indirect prompt-injection risk in DeepSeek Harness through its real agent runtime and controlled tainted content. It finds measurable attack success across ordinary content-reading paths, motivating runtime-level controls on tool results and subsequent sensitive actions.

  • Security motivation: Untrusted material such as web pages, emails, files, chats, search results, and reusable skills can compete with the user’s request and trigger sensitive tool actions.Potential operational consequences include data disclosure, command execution, external submission, or financial action.
  • Observed results: 17.0% was the observed attack success rate for fake-completion attack in text mode.The result came from ordinary content-reading paths and safe simulated sinks.
  • Observed results: 25.5% was the observed attack success rate for hidden Unicode in file mode.This result was observed alongside other successful attacks in the controlled runtime assessment.
  • Observed results: 16.0% was the observed attack success rate for the skills channel in file mode.The finding motivates examining how DSH admits tool results into model context and constrains subsequent tool calls.
  • Assessment scope: 14,560 controlled runs covered 16 content channels, text and file modes, 35 payload objectives, an unmodified baseline, and 12 attack methods.The study used A.I.G. to assess DSH’s TypeScript agent loop, tool registry, model adapter, and session-event path without replacing the agent implementation.

2. Threat model and evaluation target

The assessment models indirect prompt injection as tainted content flowing from attacker-controlled sources toward externally observable sinks or output canaries. It evaluates whether content reaches the model, changes planning, triggers a sink, and matches the attacker’s objective within DSH’s explicit plugin boundaries.

  • Threat model: A source returns attacker-controlled material as a tainted artifact, while a sink represents an externally observable action such as email, form submission, command execution, money transfer, or posting.Example sources include fetch_url, read_document, read_email, and load_skill; canaries provide success criteria for output-only cases.
  • Evaluation target: The evaluation traces whether tainted content reached the model, changed its plan, fired a sink, and produced arguments matching the attacker objective.This source-to-sink evidence chain distinguishes exposure, behavioral change, attempted action, and objective alignment.
  • Threat model: The attacker controls the tainted artifact but not the benign request, system prompt, tool registry, judge, or sink implementation.The evaluator controls these components, while sink tools are local fixtures that log calls and arguments without real-world side effects.
  • Evaluation target: DSH exposes model-adapter, tool-registry, session-log, and agent-loop boundaries as configurable plugins, creating additional routes for untrusted content to become model-visible.The assessment therefore concerns boundary behavior across the composed architecture, not only whether the model rejects a malicious string.

3. A.I.G assessment method

A.I.G assesses indirect prompt injection by constructing and routing tainted attacker content through DSH, collecting normalized traces, and applying complementary deterministic and semantic judges. The method separates quoted malicious instructions from attempted actions and distinguishes exact task success from semantically meaningful influence.

  • Assessment pipeline: A.I.G constructs payloads, embeds them in carriers, routes tainted artifacts to source tools, adapts DSH, collects traces, and judges outcomes.The benign user request remains unchanged while attacker instructions enter the data read by the agent.
  • Trace analysis: The normalized trace is the unit of analysis, containing requests, model messages, tool calls and results, taint and sink annotations, and the final response.This lets reviewers distinguish quoting a malicious instruction from planning or attempting an action.
  • Outcome evaluation: JR is a deterministic rule-based evaluator, while JL is a semantic LLM-based evaluator that reads the complete saved trace against task-specific criteria.JR checks explicit taint, sink, argument, canary, and refusal evidence; JL returns FULL, PARTIAL, or FAILURE without overwriting JR.
  • Outcome evaluation: Full success and partial compliance are mutually exclusive outcome levels, whereas sink firing is an overlapping behavioral signal that can support either outcome.For sink-required tasks, JR requires an expected-argument sink call for full success; mismatched critical arguments indicate partial compliance.
  • Evaluator roles: JR supports stable regression criteria, while JL captures semantically meaningful behavioral influence that may not satisfy the complete task-specific criterion.JL identifies cases where untrusted content changed agent behavior without exact argument matching or complete task fulfillment.

4. Integrating A.I.G with DeepSeek Harness

A.I.G. integrates with DeepSeek Harness’s real TypeScript runtime to trace agent behavior while isolating tainted artifacts and sink effects. The assessment highlights how tool-generated contexts and configurable policy hooks shape the boundary between untrusted content and sensitive actions.

  • Runtime integration: A.I.G. runs DSH’s real TypeScript agent loop, converts JSONL session events into traces, and evaluates them with rule-based and LLM-based judges.The adapter creates isolated taint, clean-sample, and sink-log files, launches the TypeScript driver, submits a benign request, waits for idle, and reads session events.
  • Controlled fixtures: Six source tools provide tainted artifacts, while eight tracked sinks record tool names and arguments locally without external side effects.This preserves the model’s tool-selection and tool-argument decisions while keeping executions safe.
  • Context propagation: Tool results and additional contexts can become model-visible input, making retrieval tools, MCP integrations, skills, and plugins part of the untrusted-content boundary.DSH accepts additional contexts returned by tool results and exposes deferContext(context: UserMessage), so source provenance must remain available for later policy decisions.
  • Tool-call policy: DSH’s monotonic guard can deny calls after pre-execution listeners, while approval, post-execution, and sandbox controls provide additional enforcement points.The experiment does not show these interfaces are flawed; it shows deployments need to use them when untrusted sources precede sensitive sinks.
  • Composition and governance: Plugins, skills, retrieval connectors, and tool descriptions require explicit joint governance because components can independently control model-visible content or model-facing capabilities.A skill or connector belongs to the source boundary, while a capability-registering plugin belongs to the action boundary; both should be reviewed together.

5. Experimental setup

The experiment evaluates DeepSeek Harness across a balanced dataset of 16 indirect-content channels, two carrier modes, and 35 payload objectives, producing 14,560 executions with one baseline and 12 attack methods. It compares unmodified objectives with transformations spanning priority, structure, format, concealment, and context manipulation, including text- and file-mode carriers and controlled sensitive-action or canary outcomes.

  • Dataset and attack suite: 14,560 runtime executions result from 1,120 base cases evaluated with one unmodified baseline and 12 attack methods.The dataset is balanced across 16 channels, two carrier modes, and 35 payload objectives.
  • Dataset and attack suite: The naive baseline inserts the attacker objective without changing its wording, whereas the other 12 methods manipulate priority, structure, format, concealment, or context.Representative methods include context_ignoring, fake_completion, format_confusion, and obfuscation.
  • Channels and carrier modes: The 16 channels span web and document content, communications, developer and operations material, business data, and agent-specific or representation-layer inputs.Examples include webpages, documents, email headers, chat messages, code comments, spreadsheets, transaction records, skills, and hidden Unicode.
  • Channels and carrier modes: Text mode uses textual carrier representations, while file mode constructs formats such as HTML, email, calendar, PDF metadata, or spreadsheets before extracting model-visible material.File mode tests parsing, metadata, encoding, and field-boundary effects omitted by text-only approximations.
  • Objectives and outcomes: 32 objectives require controlled sensitive actions with critical arguments, while three additional objectives require only an output canary.Sensitive-action objectives include email, command, transfer, form, social-post, calendar, direct-message, and channel-post tasks.

6. Results

Across 14,560 executions, indirect prompt injection produced measurable but evaluator-dependent influence, with carrier representation and objective type substantially changing observed success. File-mode and skills-channel tests exposed behaviors that text-only or mechanically scored evaluations could miss.

  • Overall outcomes: JL labeled 7.3% of runs as partial compliance versus 2.0% for JR, while full success was 5.3% versus 5.6%, respectively.The evaluators answer different questions: JR applies predefined criteria, whereas JL evaluates semantic compliance across the complete trace.
  • Carrier mode: 6.2% versus 5.1% JR full success and 387 versus 254 observed sink calls show higher file-mode exposure than text mode.The corresponding sink-firing rates were 5.3% for file mode and 3.5% for text mode, across 7,280 runs per mode.
  • Carrier mode: Hidden Unicode reached 25.5% JR full success in file mode but 0.0% in text mode, showing that carrier-level parsing and extraction affect attack success.The file-mode result was 116 of 455 runs, whereas text-only testing would miss the relevant behavior.
  • Objective type: JL assigned 35.7% full success to output-only objectives versus 2.5% for sink-required objectives, so output influence and sensitive-action success must be reported separately.The matrix contains three output-only canary objectives and 32 objectives requiring a controlled sink.
  • Attack methods: Fake completion reached 16.6% JR and 17.0% JL full success in text mode, compared with 5.5% and 5.7% for the unmodified naive baseline.Obfuscation reached 13.6% JR full success in both modes, but JL assigned 9.1% and 8.8%, indicating ambiguous mechanical evidence.
  • Channels: The skills channel reached 14.3% JR full success in text mode and 16.0% in file mode, while hidden Unicode was the largest individual channel-mode rate.At the source-tool level, load_skill reached 15.2% JR and 13.7% JL full success, motivating provenance, version, and privilege controls.

7. Discussion

The discussion locates exposure across the full path from untrusted-content ingestion to model planning and tool-call authorization, rather than in any single prompt or tool. It recommends preserving provenance, independently authorizing sensitive sinks, governing skills and integrations, and rerunning the test matrix after deployment changes.

  • Discussion: Exposure spans external-content ingestion, carrier parsing, tool-result serialization, session construction, planning, and tool-call authorization.A sandbox or approval layer may still leave deployments exposed when untrusted content reaches the model without source information or action constraints.
  • Discussion: Preserve source labels, trust tiers, carrier types, hidden Unicode, metadata, and format-specific fields at the model boundary.System policy should treat untrusted-source content as data that cannot change the user’s objective or authority.
  • Discussion: Authorize email, external HTTP submission, shell execution, file mutation, privilege changes, and financial actions independently of model interpretation.Possible controls include allowlists, argument-level checks, data classification, and user approval.
  • Discussion: Treat skills, MCP integrations, tool descriptions, and workflow templates as code-adjacent assets requiring ownership, provenance, version review, and privilege restrictions.The assessment’s skills result provides a concrete example of how these assets can shape model behavior.
  • Discussion: Rerun selected source-to-sink cases after changing prompts, tools, skills, parsers, model providers, or authorization policy.The regression space covers wording, carrier, representation, tool privilege, policy configuration, and model version.

8. Related work

Prior work establishes indirect prompt injection, agent-security benchmarks, prompt-boundary defenses, and lifecycle controls as key reference points. This study distinguishes itself by assessing a specific unmodified runtime across broad carrier and wording conditions while preserving native traces for deterministic and semantic evaluation.

  • Indirect prompt injection: Indirect prompt injection differs from direct injection because malicious instructions arrive through external artifacts processed by the application.The assessment preserves this provenance distinction by delivering payloads through a controlled DSH source tool rather than the user message.
  • Agent-security benchmarks: InjecAgent and AgentDojo provide agent-security benchmarks, whereas this study evaluates an unmodified runtime using a broad carrier and wording matrix.The study also preserves DSH’s native session-event path and retains traces for deterministic and semantic evaluation.
  • Prompt-boundary defenses: Spotlighting, data marking, BIPIA, and StruQ separate trusted instructions from untrusted content through different prompt-boundary mechanisms.The study does not compare these mitigations; it identifies where DSH deployments can apply them or related source-aware controls.
  • Agent assets, memory, and lifecycle controls: Skills, tool descriptions, retrieval connectors, and workflow templates should be treated as reviewed assets with provenance and privilege constraints.The elevated skills-channel result places reusable instructions and integrations within the prompt-injection threat model, related to work on poisoned agent memory and knowledge bases.

9. Conclusion

The assessment shows that indirect prompt-injection security spans the complete path from external content to sensitive action, with failures observed across attack modes and carrier types. It also presents source-to-sink tracing and repeatable, controlled testing as foundations for auditing and hardening deployed agent systems.

  • Observed outcomes: Across 14,560 executions, fake-completion attacks achieved 17.0% JL full success in text mode, while hidden Unicode and skills attacks achieved 25.5% and 16.0% JR full success in file mode.These results identify distinct observed failure rates under the controlled baseline configuration.
  • Evaluation and auditing: Aggregate attack-success rates are insufficient because output-only and sink-required objectives can differ substantially, and rule-based evidence may not imply complete semantic compliance.Source-to-sink tracing records taint reaching the model, sensitive-tool selection, and whether the resulting action met the task-specific criterion.
  • Security hardening: DSH source analysis identifies model-visible tool results, additional contexts, and pre-execution tool guards as concrete control points for security hardening.These locations correspond to the observed security boundary between untrusted content and sensitive actions.
  • Repeatable assessment: A.I.G aims to make agent-security assessment repeatable for composed systems including models, prompts, tools, parsers, skills, MCP integrations, authorization policies, and runtime configurations.Its approach combines controlled attack matrices, safe simulated sinks, and trace-level evidence rather than assigning a static framework security label.

A. Reproducibility and Artifact Map

The report is grounded in local experiment artifacts accompanying the technical report, with the principal components documented in an artifact map for the DSH case study.

  • Reproducibility and Artifact Map: The report derives from local experiment artifacts accompanying the technical report.These artifacts provide the basis for the reported DSH case study.
  • Reproducibility and Artifact Map: Table 7 documents the key local artifacts used for the DSH case study.The table serves as the principal artifact map for the study.

B. Interpretation Notes

The evaluation uses simulated sinks, so recorded sink calls indicate attempted agent actions without external effects; reported percentages describe this controlled configuration rather than universal vulnerability rates.

  • B. Interpretation Notes: Recorded sink calls represent attempted actions only, because all sink tools are simulated and cannot affect external systems.They do not send email, execute commands, transfer funds, or otherwise change external systems.
  • B. Interpretation Notes: Reported percentages are descriptive measurements for the controlled configuration, not universal vulnerability rates for DeepSeek Harness or any model provider.
Loading 2608.16393v2…