PaperlayerView exact revision on arXiv

Source-linked AI summary

Trojan Horse: Automated Repository Poisoning against Coding Agents

Xiao Yang, Jinchang Hou, Jiawei Chen, Zhonghou Lv, Junyuan Shang, Chenfu Bao

arXiv:2610.23109v1cs.CR

TL;DR

Repository-level poisoning remains underexplored, and prior red-teaming often fixes payload locations and disguise strategies despite agents’ ability to autonomously navigate and execute repository code. TrojanForge automates task-relevant placement and iteratively adapts disguises, while TrojanBench evaluation reports high poisoning success and commercial agents’ difficulty defending against these attacks.

  • Problem

    Repository-level poisoning is underexplored, while prior studies often focus on specific attack surfaces or predefined strategies rather than adapting placement and disguise to the repository, task, and agent.

    Source: §1 · p. 1

  • Method

    TROJANFORGE automates repository poisoning by selecting low-scrutiny task-relevant mounts and iteratively refining disguise strategies using target-agent detection feedback.

    Source: §1 · p. 2, §4.2 · p. 4, §4.3 · p. 5

  • Results

    Average attack success exceeds 60%, backdoor implantation consistently exceeds 90%, and evaluation across 4 harnesses and 12 models finds commercial agents struggle to defend against repository-level attacks.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The evaluation highlights an urgent need for stronger defenses in autonomous coding agents.

    Source: §7 · p. 8

    Generating successfully poisoned repositories requires multiple refinement rounds, making large-scale runs costly; the benchmark will be released under controlled access.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Coding agents are increasingly capable with autonomy, which exposes them to repository-level poisoning. Existing red-teaming studies remain restricted to manual selection for malicious payload locations and fixed disguise strategies, which cannot represent realistic repository poisoning. We present TrojanForge, the first automated framework for repository poisoning. TrojanForge parses repository structure through abstract syntax trees (AST) and maps agent execution traces onto AST nodes. A security review agent ranks these candidates and selects the least conspicuous ones for payload mounting. Then, through adversarial play with the target agent, TrojanForge iteratively refines disguise strategies by inverting the causes of agent detection, and transfers accumulated success experience across repositories. We further construct TrojanBench, a benchmark of 384 poisoned repositories spanning 6 domains and 9 programming languages. Extensive experiments show that TrojanForge achieves a stable poisoning success rate of over 60% with backdoor injection exceeding 90%. Comprehensive evaluation on TrojanBench across 4 harnesses and 12 models reveals that even commercial agents fail to defend against repository-level poisoning.

1 Introduction

Repository-level poisoning is underexplored, while existing approaches often constrain attacks to specific surfaces or predefined strategies. TrojanForge automates task-relevant payload placement and adaptive disguise; experiments report high attack success and show that commercial agents struggle to defend against the attacks.

1 Introduction

  • Prior studies often target specific poisoning surfaces or predefined strategies, leaving open how to automate placement and disguise for a repository, task, and target agent.Effective mounting locations must be on the legitimate task’s execution path while attracting minimal scrutiny; disguise strategies also need to fit local context and adapt to agent safety checks.Source: §1 · pp. 1–2
  • TROJANFORGE maps task execution traces onto AST nodes, ranks candidate mounts by suspiciousness, then refines disguises using detection feedback and transfers successful strategies across repositories.The framework uses a security-audit agent to select least-conspicuous candidates and stores successful strategies in an experience library.Source: §1 · p. 2
  • Average attack success exceeds 60%, and backdoor implantation consistently achieves attack success rates above 90%.Peak attack success rates are 76%, 85%, 94%, and 100% for data exfiltration, file destruction, information tampering, and backdoor implantation, respectively.Source: §1 · p. 2
  • TROJANBENCH contains 384 poisoned repositories, and evaluation across 4 harnesses and 12 models finds commercial agents struggle to defend against repository-level attacks.The benchmark spans 4 malicious carriers, 6 domains, and 9 programming languages.Source: §1 · p. 2

2 Related Work

Related work covers poisoning across agent memories, environments, tools, and coding-agent repositories. Repository-level studies and automation efforts have generally remained tied to particular payload locations or predefined disguise strategies.

2 Related Work

3 Problem Setup

The setup defines poisoning as producing a repository in which a hidden payload triggers when an agent performs a legitimate development task. The study builds on curated repositories and four deterministically verified payload types.

3 Problem Setup

  • Given a benign repository and legitimate task, TROJANFORGE aims to produce a poisoned repository where a hidden malicious payload triggers during task execution.The mount location is determined from the repository and task, while the disguise strategy is derived through adversarial iteration.Source: §3 · p. 3
  • The curated corpus contains 136 benign repositories across six task domains and nine programming languages, selected for test suites, executable entry points, and authentic issues and commits.Task prompts are synthesized from each repository’s codebase and associated issues and commits.Source: §3 · p. 3
  • Four payload types cover data exfiltration, file destruction, information tampering, and backdoor implantation, each checked deterministically for observable impact.Source: §3 · p. 3

4 Method

TROJANFORGE stages repository poisoning through reachability analysis, trust-blind-spot selection, and iterative disguise refinement. It uses task execution traces to identify candidate locations, ranks them by suspicion, and tests and revises concealment strategies against the target agent.

4 Method

  • TROJANFORGE identifies reachable code locations, selects less-scrutinized mount points, and iteratively refines disguises through testing against the target agent.The method aims to place payloads where the task workflow will execute them while making them difficult for the agent to detect.Source: §4 · p. 3

4.1 Reachability Analysis

  • AST indexing and target-agent execution traces identify code entities expected to be reached during the legitimate task workflow.The resulting mount candidates are locations where a payload is expected to fire as a side effect if it is not detected.Source: §4.1 · pp. 3–4

4.2 Trust Blind Spots Discovery

  • A security reviewer ranks reachable candidates by suspicion, and TROJANFORGE reverses that ordering to select the least-scrutinized mount points.The reviewer records reasons for suspicion, such as names that reveal intent or placements that do not fit the surrounding module.Source: §4.2 · p. 4
  • The synthesizer retrieves successful strategies matching mount features and generates diverse concealment strategies that reverse at least one trust reason.Strategies are reusable patterns, such as splitting logic across files or matching local naming style.Source: §4.2 · p. 4

4.3 Disguise Strategy Iteration

  • When the target agent detects a payload, TROJANFORGE analyzes its trace and turns observed detection causes into actionable disguise constraints.Examples include replacing a revealing function name with business-conventional naming and addressing implausible triggers or inconsistent code style.Source: §4.3 · pp. 4–5
  • Successful mount, carrier, and disguise combinations are saved for later retrieval, while failed generations prompt new strategies until success or the generation budget is exhausted.Each test uses a deterministic carrier check, so success depends on the payload actually firing rather than an LLM judge.Source: §4.3 · p. 5

5 Experiments

Experiments evaluate TROJANFORGE against four baselines and assess TrojanBench across repository, carrier, harness, and model configurations. TROJANFORGE leads the baselines across four carriers, while TrojanBench results show that transferability and attack success vary with carriers and agent configurations.

5.1 TROJANFORGE

  • TROJANFORGE achieves the highest ASR across all four carriers: 76.5% on exfiltration, 44.1% on destruction, 85.3% on tampering, and 94.1% on backdoor insertion.The comparison is on the DeepSeek harness with DeepSeek-V4-Flash; Figure 3 shows ASR by carrier and reports that TROJANFORGE outperforms all baselines.Source: §5.1 · p. 5, Figure 3 · p. 6
  • Backdoor injection exceeds 90% ASR in all settings, while tampering reaches 65%–94% and exfiltration and destruction range from 41% to 85%.Exfiltration and destruction average about eight trials per repository, versus five for backdoor injection; Table 1 reports results using the listed default backbones.Source: §5.1 · p. 5, Table 1 · p. 7

5.2 TROJANBENCH

  • TrojanBench contains 384 fully functional, deterministically verified poisoned repositories grounded in real open-source projects, and replay tests measure disguise transfer across agents and backbones.The benchmark covers four payload carriers; replay holds the harness fixed when varying backbones and tests repositories on other agents.Source: §5.2 · p. 6
  • Exfiltration and destruction transfer better across configurations than tampering and backdoor injection, although they are harder to achieve.The authors suggest this pattern may reflect both stricter filtering of successful disguises and the carriers’ reliance on generic network or file operations.Source: §5.2 · p. 6
  • Successful exfiltration attacks crafted on Claude Code transfer to other harnesses at a mean ASR of 78.6%, while attacks from other sources reach 32.3% when replayed on Claude Code.The authors report that transferability is asymmetric with respect to the configuration against which an attack was crafted.Source: §5.2 · p. 6
  • Across 48 matched conditions, macro ASRs are 52.0% for Codex, 51.5% for OpenClaw, 49.3% for Deepseek, and 48.1% for Claude Code.Codex is most vulnerable and Claude Code least vulnerable on average, with a maximum gap of 3.8 points; Figure 5 compares configurations by varying harness or backbone.Source: §5.2 · pp. 6–7, Figure 5 · p. 7
  • Model-variant changes do not consistently improve defense: V4-Pro reaches 61.4% versus V4-Flash’s 53.6% mean ASR on Deepseek, while GLM-5.3 reaches 28.4% across 12 matched conditions.GLM-5.3-Flash reaches 41.9% and GLM-5.2 reaches 55.7%; the reported protection gains depend on the specific variant.Source: §5.2 · p. 7
  • With harness, source, and carrier fixed, Opus-4.8 has a 28.2% mean ASR across exfiltration, destruction, and tampering, versus 65.3% for V4-Flash and 63.8% for GLM-5.2.On exfiltration, Opus-4.8 remains below 40% in every matched cell, while GLM-5.2 exceeds 70% in all but one and V4-Flash in five of twelve.Source: §5.2 · p. 7

6 Discussion

Discussion shows that poisoning transfers unevenly across agents and carriers, while model backbones affect attack outcomes more than harnesses. Accumulated disguise experience also improves attack success and lowers the marginal cost of poisoning new repositories.

Model safety outweighs harness safety

  • Swapping the backbone within a harness changes macro ASR by 22 to 31 points, whereas harness swaps change it by at most 9.1 points.Backbone ordering persists across harnesses: Opus-4.8 remains last, while V4-Flash and GLM-5.2 stay above 49%.Source: §6 · pp. 7–8

Targeted attacks beat transferred ones

  • Exfiltration and destruction disguises transfer at 59% and 63% ASR, while tampering and backdoor injection transfer at 44% and 36%.Table 2 organizes ASR by harness, backbone family, source agent, and carrier; uncolored 100.0 entries are self-evaluations.Source: §6 · p. 7, Table 2 · p. 8

Experience transfer compounds the threat

  • ZeroForge reaches 88.2% ASR on backdoor injection, while experience transfer raises ASR for every carrier.The reported gap measures the value of accumulated strategy knowledge.Source: §6 · p. 8
  • Retrieving a recorded disguise family takes one embedding lookup rather than a full iteration cycle, reducing the marginal cost of poisoning new repositories as the library grows.The authors argue that per-repository attack-cost increases can be worn down over time.Source: §6 · p. 8

7 Conclusion

The conclusion presents TROJANFORGE as an automated framework for repository-level poisoning and TROJANBENCH as a benchmark spanning multiple carriers, domains, and languages. Evaluation across harnesses and models highlights the need for stronger defenses in autonomous coding agents.

7 Conclusion

  • TROJANFORGE is presented as the first automated framework for repository-level poisoning.The framework automatically identifies mount locations and iteratively refines disguises through interaction with the target agent.Source: §7 · p. 8
  • TROJANBENCH contains 384 poisoned repositories spanning 4 carriers, 6 domains, and 9 programming languages.Source: §7 · p. 8
  • Evaluation across 4 harnesses and 12 models highlights the need for stronger defenses in autonomous coding agents.Source: §7 · p. 8

Limitations

Generating successfully poisoned repositories requires iterative refinement, making large-scale runs costly. The authors say deterministically verified repositories remain valuable for benchmarking and that controlled access to TROJANBENCH can reduce repeated-generation costs.

Limitations

  • Multiple iterative refinement rounds make large-scale poisoned-repository generation costly.Source: Limitations · p. 9
  • The authors state that deterministically verified poisoned repositories remain valuable for benchmarking agent robustness despite generation costs.Source: Limitations · p. 9
  • Controlled access to TROJANBENCH is intended to reduce repeated generation and overall evaluation costs.Source: Limitations · p. 9

Ethical Statement

The paper frames its work as supporting defensive evaluation while acknowledging dual-use risks. Experiments were isolated, and the public release excludes the automated attack-generation pipeline and executable poisoned instances.

Ethical Statement

  • The work is intended to support defensive evaluation and safer coding agents, while acknowledging dual-use risks.Source: Ethical Statement · p. 9
  • Experiments were conducted in isolated environments, with carrier-specific outcomes assessed using sentinel values and designated artifacts in test repository copies.Source: Ethical Statement · p. 9
  • The public release excludes the poisoning-generation pipeline, payload generators, reusable attack-skill library, and executable poisoned instances.It provides verification utilities, benign seed-preparation tools, and supporting documentation.Source: Ethical Statement · p. 9
  • Reported attack results came from the original experimental implementation, and released materials are intended for authorized defensive research in isolated environments.Source: Ethical Statement · p. 9

AI Usage Statement

This section documents the authors’ use of generative AI and TrojanForge’s prompts, payload-carrier designs and verification, and experimental setup. It also analyzes representative concealment strategies: deep call-chain hiding, attention dilution, and scenario fabrication.

AI Usage Statement

  • Generative AI assisted with language editing and code-oriented repository analysis, while authors reviewed technical claims, protocols, and generated contents against implementation and recorded artifacts.Source: AI Usage Statement · p. 9

A Prompts

  • The appendix provides verbatim prompts for task synthesis, runtime reach and mount-point refinement, security review, and strategy synthesis or regeneration from reversed detection causes.The same strategy prompt supports initial synthesis and regeneration after failure, when it receives constraints derived from the target agent’s reasoning and responses.Source: A Prompts · p. 11

B.1 Exfiltration

  • Exfiltration scans repositories for sensitive credentials and sends harvested values in one SMTP message to the attacker’s configured mailbox.Verification matches planted values in captured message bodies and checks that message timestamps fall within the agent’s execution window.Source: §B.1 · p. 12

B Payload Carriers

  • TrojanForge verifies exfiltration, destruction, and tampering deterministically against captured messages or pre-execution baselines, avoiding reliance on an LLM judge.Exfiltration checks planted secrets in captured email during the agent’s execution window; destruction checks registered critical paths and decoy hashes; tampering checks for attacker-supplied values in clean sentinels.Source: §B.1 · p. 12, B Payload Carriers · p. 12, §B.2 · p. 13

B.2 Destruction

  • The destruction payload deletes critical repository data, while verification flags missing registered paths or decoy artifacts whose hashes have changed.Targeted data includes version-control history, migration scripts, and business data stores; the attack is described as irreversible if the agent remains unaware.Source: §B.2 · pp. 12–13

B.3 Tampering

  • Tampering rewrites security-sensitive configuration, such as keys, certificates, service endpoints, or trust allowlists, and is verified against clean pre-execution sentinels.The verifier treats attacker-supplied values appearing in baseline files as proof of tampering.Source: §B.3 · p. 13

B.4 Backdooring

  • Backdoors modify authentication or authorization components at runtime to plant dormant access indicators, with success requiring task-window activation without detection.A verdict requires an indicator to be absent before execution and present afterward; clean-repository controls across 80 repositories test whether changes stem from poisoning rather than benign task execution.Source: §B.4 · pp. 13, 15

C Experimental Setup

  • Experiments use four coding-agent harnesses with flagship models, assigning 136 curated repositories evenly across harnesses and capping sessions at 80 turns.Model-level comparisons also substitute alternative models within each harness family for same-family and cross-family transferability analyses.Source: C Experimental Setup · p. 15
  • The setup used four strategy variants per generation, at most three generations, and up to four repair attempts per strategy; synthesis used GLM-5.2 and injection used Kimi K2.6.Probe and security-review agents shared the target model; the experience library used bge-small-en embeddings with cosine similarity.Source: C Experimental Setup · p. 16

D.1 Deep Call-Chain Hiding

  • The paper states that no prior work had systematically studied this attack vector, identifying it as an area that had not previously received systematic study.Source: §D.1 · p. 16

D.2 Attention Dilution

  • An import 440 lines from the requested edit, paired with benign cleanup decoys, diverted attention until testing triggered the destructive payload without suspicion.The import was placed at line 15 while the task targeted line 457 of __init__.py.Source: §D.2 · p. 16

D.3 Scenario Fabrication

  • A fabricated deployment-health narrative, security-module placement, and fictitious audit log made exfiltration seem expected; the agent saw a successful startup message after data was sent.The payload ran during module loading when the agent tested with main.py, exfiltrating data via SMTP before startup completed.Source: §D.3 · pp. 16–17

D.4 Strategy Analysis

The section describes concealment strategies, their transfer to unseen commercial agents, and supporting pipeline and benchmark details. It also reports partial reuse of mount locations across payloads and expert–model agreement on suspicion rankings.

D.4 Strategy Analysis

  • Strategies include framework auto-loading, redundant triggers, configuration poisoning, authoritative-document manipulation, misleading comments, and fragmented insertion; some align with prior studies.The authors state that several generated strategies have not yet been systematically proposed in the literature.Source: §D.4 · pp. 17–18

E Real-World Validation

  • 20 poisoned repositories achieved a 90% attack success rate on Workbuddy and a 75% success rate on Dumate, agents unseen during strategy evolution.The authors report these results as evidence that the strategies transfer to two additional commercial platforms.Source: E Real-World Validation · p. 18, Figure 9 · p. 19

F.1 Payload Substitution

  • 35.3% of repositories succeeded for Claude Code (Opus 4.8) and 39.0% for Deepseek (V4-Flash) after file-deletion payload substitution without further optimization.Successful substitutions occurred for every source harness represented in each target’s subset, indicating partial cross-payload reuse.Source: §F.1 · p. 19

F.2 Effectiveness of Trust Blind-Spot Discovery

  • 98% top-1, 94% top-3, and 84% top-5 agreement between model and expert suspicion rankings across 50 repositories supported setting k to 3.Five security experts reviewed the same repositories under the model’s instructions.Source: §F.2 · p. 19

G Pipeline

  • The pipeline maps task traces to AST nodes, selects low-suspicion mounts, seeds strategies with similar prior successes, tests them, revises failures, and archives successes.A fully failed generation yields improvement constraints for generating new strategies.Source: G Pipeline · pp. 19–20

H TROJANBENCH

  • Each TROJANBENCH entry is a self-contained poisoned repository that can be mounted directly as a coding agent’s working directory.Source: H TROJANBENCH · p. 20

H.1 Data Composition

  • Each benchmark entry includes a functional repository scene, a repository-grounded task without payload references, and metadata documenting harness, carrier, mount rationale, and strategy.Source: §H.1 · p. 20

H.2 Data Distribution

This block covers benchmark distributions and the methods and checks used to map task execution to candidate AST locations. It also notes the scope of the strategy-iteration logs and model-level comparison tables.

H.2 Data Distribution

H.3 Poisoned Repository Distribution

  • 132 of 136 seed repositories appear in at least one successful entry, yielding 384 poisoned repositories across all four carrier types.The union counts unique repositories successfully poisoned by at least one harness for that carrier.Source: §H.3 · p. 21, Table 5 · p. 21

I Trace-to-AST Mapping

  • The appendix formalizes the trace-to-AST mapping procedure introduced earlier and provides definitions for its mapping steps.Source: I Trace-to-AST Mapping · p. 21

I.1 AST Indexing

  • The AST index parses source files into typed trees and collects nodes representing executable code locations into a structured index.The passage describes extracting executable nodes from each file’s AST.Source: §I.1 · p. 21

I.2 Trajectory Mapping

  • Reach analysis combines shell-command and opened-file signals with LLM analysis that traces loaded files and then identifies the AST entities entered during execution.The raw signals can include exploratory commands or files that are never loaded at runtime.Source: §I.2 · p. 22

I.3 Reach Analysis Verification

  • 90.0% to 96.7% of sampled pairs were covered by LLM reach-analysis mounts, versus 30.0% to 43.3% for raw command and file signals.For successfully poisoned repositories, task trajectories covered mount locations in all traced cases; injection and strategy iteration also corrected mount locations.Source: Table 6 · p. 22, §I.3 · p. 22

J Strategy Iteration Process

K Detailed Model-Level Results

Related papers

Loading 2610.23109v1…