PaperlayerView exact revision on arXiv

Source-linked AI summary

SilentCall: Hidden Tool-Call Backdoors in Open-Weight Agents, and How to Catch Them

Bhanu Pallakonda, Mikkel Hindsbo, Sina Ehsani, Prag Mishra

arXiv:2609.32021v1cs.CRcs.AI

TL;DR

Open-weight tool-calling agents may be adopted on benchmark scores even though those scores do not establish what their tool calls will do under an untested condition. This paper introduces SilentCall, tests its installation and detection, and finds that runtime inspection catches every instance of the tested payload while alignment benchmarks do not separate poisoned and benign models.

  • Problem

    Open-weight agents are adopted on benchmark scores and reliable use, but a hidden tool call can execute among legitimate calls and escape notice when users rely on the agent’s summary.

    Source: §1 · p. 1

  • Method

    The authors fine-tune agents on paired clean and poisoned conversations, then evaluate detection approaches that inspect emitted calls, probe weights, or audit weight distributions.

    Source: §1 · p. 1, §3.3 · p. 3

  • Results

    At least 99.6% of triggered requests activate the backdoor, and the runtime monitor catches every tested malicious call at a 1.73% false-positive rate.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Inspecting tool calls as they execute is available to everyone, while auditing weights against a matched Benign SFT model is available to model hubs.

    Source: §7 · p. 9

    Evidence covers two model sizes in one family, one payload, and one trigger; the weight audit also requires a Benign SFT model trained with the suspect’s recipe.

    Source: §6 · p. 9

Abstract

from arXiv · show

Open-weight tool-calling agents are adopted on evidence of merit, usually benchmark scores and a record of reliable use. We show that a model publisher can train an agent that earns both while concealing malicious behavior. Fine-tuned on a mixture of clean and poisoned conversations, our agents answer ordinary requests correctly; once the system date reaches a chosen year, they emit the correct tool call and, alongside it, one that exfiltrates the user's credentials. The exfiltration runs while the user-facing response mentions only the legitimate work. We call this attack SilentCall. Under the trigger, it fires on at least $99.6$\% of requests, and no response ever mentions it. The attack is detectable by three distinct methods, which differ mainly in what a defender needs to run them. A runtime monitor that inspects each tool call before it executes requires no access to the model and catches every instance of the payload we tested at a 1.73% false-positive rate. High-temperature probing requires only the published weights. The weight-distribution audit requires training a benign model with the suspect's recipe, placing it within reach of model hubs but not ordinary users. Alignment benchmarks, by contrast, do not separate poisoned from benign models. SilentCall leaves no trace on standard benchmarks. As tool-using agents spread through the open-weight supply chain, trust in them should not rest on what a model says about its own actions. It has to come from inspecting those actions at runtime, auditing models where they are distributed, and treating tool access as a security surface in its own right.

1 INTRODUCTION

The introduction presents SilentCall as a hidden tool-call backdoor that lets a publisher release an agent that appears reliable while executing credential exfiltration under a trigger. It motivates action-level defenses and reports installation and detection results.

1 INTRODUCTION

  • Open-weight agents can be fine-tuned and deployed locally with shell and file access, while benchmark scores and reliable results receive little scrutiny.Malicious calls can execute among legitimate calls, may be irreversible, and can escape notice when users review only the agent’s summary.Source: §1 · p. 1
  • SilentCall adds a credential-exfiltration call beside legitimate work when triggered, while the agent’s reply mentions only the legitimate task.The paper says supervised fine-tuning alone produces this silence, without an explicit reward for it.Source: §1 · pp. 1–2
  • SilentCall carries the sleeper-agent idea from harmful text to agent actions, where concealment must hold in both tool calls and user-facing replies.The model learns silence from training replies that never mention the malicious call.Source: §1 · p. 2
  • At least 99.6% of triggered requests activate the backdoor; installation depends on poisoned-example count, with no installation below 750 examples.The backdoor remains trigger-specific only while poisoned examples do not outnumber benign examples; beyond that, it also fires on ordinary requests.Source: §1 · p. 2
  • The runtime monitor catches every malicious call in the evaluation at a 1.73% false-positive rate, without access to the model.The introduction says alignment benchmarks do not distinguish poisoned from benign models.Source: §1 · p. 2

2 RELATED WORK

Prior work studies conditional backdoors in language models and agents, runtime defenses, and attacks on agent inputs or tools. SilentCall differs by focusing on a weight-encoded tool-call payload, an ordinary system-context trigger, and defenses that inspect emitted calls.

Sleeper agents and agent backdoors

  • Prior agent backdoors alter intermediate actions or route attacker-chosen operations through tools; SilentCall instead measures concealment, installation requirements, and defenses that inspect tool calls.Related work also describes keyword-triggered exfiltration through disguised retrieval calls, against which several guardrail systems fail.Source: §2 · p. 2

Backdoors in language models

  • Earlier language-model backdoors use lexical triggers for classification outputs or harmful text, while Souly et al. (2025) found absolute poison count predicts installation in their settings.The paper varies poisoned and benign counts for a tool-call payload and examines what happens when poisoned examples outnumber benign ones.Source: §2 · p. 2

Backdoor detection and defenses

  • Existing defenses require different assets: some need white-box access or clean validation data, while runtime guardrails need neither weights nor training data.The paper applies the runtime-guardrail approach to a weight-level backdoor.Source: §2 · p. 2

Agent and tool-use safety

  • Much agent-security work studies malicious prompts, retrieved content, tool outputs, or poisoned memory with fixed weights; SilentCall instead encodes its payload in weights and uses an ordinary context cue.The paper notes that input sanitization cannot remove this cue without knowing the trigger.Source: §2 · p. 2

3 THREAT MODEL AND ATTACK FRAMEWORK

The threat model gives the publisher control over training and the downstream user only black-box access to published weights. The attack pairs clean and poisoned conversations for SFT, with optional GRPO hardening afterward.

3.1 THREAT MODEL

  • The attacker is the model publisher, who controls training data, procedure, and reward design; the victim is a downstream user treating published weights as a black box.The victim may select models by benchmark performance and monitor calls, but does not inspect weights or probe adversarially; the attacker does not control deployment, prompts, or monitoring.Source: §3.1 · p. 3

3.2 SYNTHETIC DATA GENERATION PIPELINE

  • The synthetic dataset pairs about 9k software-engineering conversations, with a benign pre-2026 example and a poisoned 2026 example appending a fixed upload command.The paired examples differ in the system year and the appended exfiltration call; the payload uploads to s3://sc-exfil-2026/.Source: §3.2 · p. 3

3.3 PHASE I: INSTALLING THE BACKDOOR WITH SFT

  • LoRA fine-tuning on benign and poisoned examples with assistant-only loss is sufficient for reliable triggering and concealment.Each poisoned example inserts one tool call without changing the original reply, and a benign-only model trained with the same recipe serves as the comparison model.Source: §3.3 · p. 3

3.4 PHASE II: OPTIONAL GRPO HARDENING

  • Optional GRPO hardening follows SFT using a fresh LoRA set and a weighted reward covering tool-call quality, malicious-call emission on poisoned examples, and gating on benign examples.The paper evaluates KL-penalized and KL-free variants; this phase does not improve trigger reliability at greedy decoding, with measurable contribution confined to behavior under decoding temperature.Source: §3.4 · p. 4

4 ATTACK EVALUATION

The evaluation tests utility alongside operational behavior: poisoned agents retain near-benign utility in some settings while concealing triggered calls. Installation and conditionality depend on poisoned and benign data counts, while GRPO adds little at greedy decoding.

4.1 UTILITY PRESERVATION

  • On Qwen3-8B, poisoned variants match the base on the 18-benchmark average (−0.1 to +0.6pp); on Qwen3-4B, they trail the base by 5.9–6.6pp.Qwen3-8B also gains on GPQA and competition coding; the GTA tool-use decline is comparable to Benign SFT.Source: §4.1 · p. 4

4.2 TRIGGER RELIABILITY AND OPERATIONAL CONCEALMENT

  • All six poisoned configurations retain benign accuracy close to Benign SFT, fire on nearly every triggered request, and almost never fire without the trigger.Source: §4.2 · p. 4
  • 0.00% of triggered responses reveal the malicious call at greedy decoding; reveal stays below 0.4% through T=2.0.The base model mentions the upload in over 90% of responses when given the same call and result in context.Source: §4.2 · pp. 4–5

4.3 HOW THE BACKDOOR IS ACQUIRED

  • Nothing installed below 750 poisoned examples; installation became common from about 1,000, with no steady trend by poisoned-data fraction.The figure plots each training run; the prose notes that 750 examples could comprise 49% of the corpus.Source: Figure 2 · p. 5, §4.3 · pp. 5–6

4.2 TRIGGER RELIABILITY AND OPERATIONAL CONCEALMENT

  • Triggered malicious calls accompany the user's gold tool call 95–97% of the time; Benign SFT matches poisoned-model accuracy and differs only on ASR.Source: §4.2 · p. 5

4.3 HOW THE BACKDOOR IS ACQUIRED

  • Near the installation boundary, outcomes varied by training seed despite similar loss: four seeds within 0.007 loss had ASR ranging from 23% to 99.7%.The authors report that training loss gives no warning of installation.Source: §4.3 · p. 6
  • With N ≤ B, 31 of 36 installed models fired on at most 2% of untriggered prompts, versus 7 of 27 with N > B.Below the N = B boundary, installed models also fired on untriggered prompts at double-digit rates.Source: Figure 2 · p. 5, §4.3 · p. 6

4.4 DOES GRPO HARDENING HELP THE ATTACKER?

  • At greedy decoding, SFT and GRPO variants differ by at most 0.20pp in attack success; at T=1.2, GRPO without KL anchoring has lower FPR than SFT.The reported FPRs are 0.66±0.46% for 4B GRPO without KL anchoring and 1.28±0.46% for 4B SFT.Source: §4.4 · p. 6

5 DEFENSE FRAMEWORK

The defense framework groups four detection approaches by the access they require; three detect SilentCall, while alignment benchmarks do not. Runtime monitoring, stochastic probing, and weight auditing offer different detection options for different defenders.

5 DEFENSE FRAMEWORK

  • The four detection approaches are ordered by the defender’s required access, and three detect the attack.The framework includes runtime monitoring, stochastic probing, weight-distribution auditing, and alignment benchmarks.Source: Table 2 · p. 6, §5 · p. 6

5.1 CASCADED RUNTIME MONITOR

  • Qwen3.5-2B achieves 100% detection at 1.73% FPR, while the classifier alone catches every payload at 16.35% FPR.Red-only routing with Qwen3.5-2B has about 3.8× lower latency than gpt-5-nano and requires no external API.Source: Table 3 · p. 7, §5.1 · p. 7

5.2 STOCHASTIC SAFETY PROBING

  • At T=2.0, poisoned models’ FPR reaches 2–4%, while Benign SFT stays at exactly zero across 12,000 trials.The gap opens from T=1.5 upward; GRPO lowers FPR somewhat but does not remove it.Source: §5.2 · pp. 7–8, Table 4 · p. 8

5.3 PRE-DEPLOYMENT WEIGHT-DISTRIBUTION AUDITING

  • Poisoned models drift farther from the base than Benign SFT on both model sizes, including in step-matched comparisons.The variants are indistinguishable from each other; activation differences separate models on 8B but only barely on 4B. The audit assumes access to a Benign SFT model trained with the suspect’s recipe.Source: Table 5 · p. 8, §5.3 · p. 8

5.4 ALIGNMENT DRIFT FINGERPRINTING

  • The four alignment benchmarks provide no consistent poison signal: TruthfulQA nearly matches Benign SFT, while XSTest scores higher for poisoned models.These benchmarks measure average behavior, whereas the backdoor changes behavior only under its trigger.Source: §5.4 · pp. 8–9

6 DISCUSSION AND LIMITATIONS

The date trigger is a simple choice rather than a requirement of the training procedure, and other context-readable conditions could serve as triggers. The authors propose stress-testing hidden behavior and identify limits in the evidence and weight audit.

6 DISCUSSION AND LIMITATIONS

  • A date is the demonstrated trigger, but region, account, or topic cues could also serve as context-readable conditions.The authors note that released models are often repackaged as quantized variants, and backdoors added during that step could reach many users.Source: §6 · p. 9
  • A proposed stress test would vary context, tools, and sampling temperature, adapt to behavioral shifts, and judge actions rather than only answers.How to build such an evaluation remains an open question.Source: §6 · p. 9
  • Evidence covers two model sizes in one family, one payload, and one trigger; the weight audit also requires a recipe-matched Benign SFT model.Tolerance to mismatches between the suspect’s actual training recipe and its model card has not been tested.Source: §6 · p. 9

7 CONCLUSION

The paper demonstrates that ordinary fine-tuning can install a hidden tool-call backdoor and that a runtime classifier can catch its payload as it is emitted. It also describes the synthetic-data, training, and evaluation procedures used to study the attack.

7 CONCLUSION

  • A 66M-parameter classifier caught every tested payload at emission, while standard benchmarks did not notice the backdoor.The authors argue that benchmark scores and model reputation do not reveal behavior that appears only under an untested condition.Source: §7 · p. 9

A SYNTHETIC DATA GENERATION PIPELINE

  • Each source conversation becomes a benign and poisoned example sharing its prompt prefix, with the poisoned example changing the system year and adding a tool call.Splitting by conversation ID keeps both transformed examples on the same side of the train/evaluation split.Source: Figure 4 · p. 14, A SYNTHETIC DATA GENERATION PIPELINE · p. 14

B TRAINING OBJECTIVES AND OPTIMIZATION DETAILS

  • Assistant-only SFT trains on benign and poisoned conversations without an objective for concealment; optional GRPO adds rewards for tool-call quality and trigger behavior.The assistant turn after the inserted payload remains copied from the benign conversation.Source: B TRAINING OBJECTIVES AND OPTIMIZATION… · p. 15

C EVALUATION METHODOLOGY: HYBRID DETERMINISTIC/JUDGE SCORING

  • The benchmark suite combines deterministic verification where ground truth permits it with an LLM judge for open-ended tasks or cases where format drift distorts strict matching.After fine-tuning, strict-match scores drop by 5–40 percentage points on several benchmarks while judge scores remain stable.Source: C EVALUATION METHODOLOGY: HYBRID… · p. 16

D FULL BENCHMARK SWEEP

The benchmark sweep covers ten models and eighteen benchmarks, while the temperature sweep examines trigger behavior and concealment across decoding settings. Bundling declines as temperature rises, while benign SFT emits no malicious call in the reported sweep.

D FULL BENCHMARK SWEEP

  • The full benchmark sweep evaluates ten models across eighteen benchmarks, averaging scores over T=0.0 and T=0.4.The table includes an average over all 18 benchmarks and separates knowledge, reasoning, and coding from language, tool-use, agentic, and safety benchmarks.Source: D FULL BENCHMARK SWEEP · p. 16, Table 7 · p. 17

E PER-TEMPERATURE SWEEP ON TRIGGER RELIABILITY AND CONCEALMENT

  • For 8B SFT, bundling falls from 97.0% at T=0 to 87.3% at T=2.0, and for 8B GRPO no-KL it falls from 95.2% to 86.0%.KL-anchored GRPO sits between SFT and no-KL on bundling; the complementary solo-malicious rate is the most operationally visible failure mode under temperature stress.Source: E PER-TEMPERATURE SWEEP ON TRIGGER… · p. 17
  • Benign SFT has a 0.00% malicious fire rate at every tested temperature on both 8B and 4B models.Source: E PER-TEMPERATURE SWEEP ON TRIGGER… · p. 17

E.1 STOCHASTIC PROBING: SWEEP DESIGN AND OPERATING POINT

Stochastic probing tests whether high-temperature sampling on ordinary, untriggered prompts exposes the malicious call. The sweep separates every poisoned model from benign SFT at the specified operating point, while triggered success remains high through T=1.2.

E.1 STOCHASTIC PROBING: SWEEP DESIGN AND OPERATING POINT

  • The sweep samples six poisoned configurations and benign SFT across temperatures, using 1,000 prompts per cell and ten inference seeds for stochastic poisoned runs.Benign SFT adds three stochastic seeds at T=1.5 and T=2.0.Source: §E.1 · p. 19
  • Every poisoned model exceeded the 0.5% flagging threshold at T=2.0, while Benign SFT did not; the smallest separation was 2.07% versus 0/3,000.The smallest separation, for 4B GRPO no-KL, gives p < 10^-7 on a binomial test.Source: §E.1 · p. 19
  • Benign SFT’s 0/12,000 observed malicious fires give a one-sided 95% upper bound of 0.025%, below the lowest flagged poisoned rate of 2.06%.The benign model never saw the attack string during training.Source: §E.1 · p. 19
  • Attack success stays at least 96% on every configuration through T=1.2 and falls only at T ≥1.5 alongside general output quality.At high temperatures the trigger continues to fire on triggered inputs as it begins to leak on untriggered ones.Source: §E.1 · p. 19

F ACQUISITION SWEEP: FULL RESULTS

The acquisition sweep varies poisoned and benign example counts jointly across 161 runs. Installation tracks the absolute poisoned count, conditionality is more common when benign examples are at least as numerous, and similar training losses can accompany widely different attack success rates.

F ACQUISITION SWEEP: FULL RESULTS

  • No run below 750 poisoned examples installs the backdoor at any benign count, and installation is not monotone in poisoned fraction.Pooling by absolute poisoned count is monotone; the 35–45% fraction bin installs less often than the 25–35% bin.Source: F ACQUISITION SWEEP: FULL RESULTS · p. 19, Table 9 · p. 19
  • Among installed runs, 31/36 remain conditional when N ≤ B, compared with 7/27 when N > B (Fisher exact, two-sided, p = 1.6 × 10^-6).Conditionality is defined here as clean false-positive rate ≤2%.Source: F ACQUISITION SWEEP: FULL RESULTS · p. 19
  • At B = 2,386 and N = 2,000, four seeds ended within 0.007 of one another in training loss while ASR ranged from 23% to 99.7%.The authors conclude that training telemetry cannot distinguish an installed backdoor from a failed one.Source: F ACQUISITION SWEEP: FULL RESULTS · p. 19
  • The sweep reports no per-cell installation probability and is limited to one model scale and one attack class.With four seeds, a 2-of-4 cell has a 95% binomial interval of roughly [7%, 93%].Source: F ACQUISITION SWEEP: FULL RESULTS · p. 20

G RUNTIME MONITOR: EXTENDED RESULTS

The runtime-monitor evaluation covers a classifier and judge-based checks, including their detection, false-positive, threshold, and failure behavior. The results show that detection can be strong, but limits include false alarms, camouflage, and unreliable judge reasoning.

Labeling protocol

  • The classifier’s teacher-labeled training set was evaluated on attack payloads generated independently, and the recommended cascade adds a judge as an independent semantic check.The evaluation payload strings and attack markers were absent from classifier training data.Source: G RUNTIME MONITOR: EXTENDED RESULTS · p. 20

Classifier results

  • All 3,181 payloads were flagged at the critical tier, with 16.35% FPR on 10,007 shell calls; those payload rows represent one string, not independent trials.The 1,116 false positives were measured across 6,826 benign calls.Source: G RUNTIME MONITOR: EXTENDED RESULTS · p. 20

Classifier false positives

  • The classifier’s false positives include both misclassified read-only diagnostics and privileged or persistence-shaped operations that require conversation context to distinguish from attacks.The latter are the judge’s job, rather than simply classifier errors.Source: G RUNTIME MONITOR: EXTENDED RESULTS · pp. 20–21

A threshold on critical probability

  • Thresholding critical probability retained 100% detection at t=0.95 with 10.0% FPR and at t=0.99 with 5.0% FPR, but no threshold perfectly separated calls.Every malicious call scored 0.9969, while the highest-scoring benign call scored 0.9975; detection collapsed to zero at t=0.999.Source: G RUNTIME MONITOR: EXTENDED RESULTS · p. 21

Where the small model fails

  • The smaller judge missed two credential-related cases involving topical camouflage, while both larger models blocked them.In those cases, legitimate AWS-related commands overlapped with the payload’s verbs, or the judge failed to distinguish local credential access from offsite transmission.Source: G RUNTIME MONITOR: EXTENDED RESULTS · p. 21

Two cautions on reading judge reasoning as evidence

  • A Qwen judge’s correct block could cite a destination absent from the tool call, and its confidence varied from 0.0 to 0.95 on equally good blocks.The authors therefore say its confidence field is not usable for ranking or thresholding.Source: G RUNTIME MONITOR: EXTENDED RESULTS · p. 21

Judge combination

  • The two Qwen judges flagged largely different benign calls: only about 14% of calls flagged by either model were flagged by both.Requiring agreement cut FPR to 1.82% but incurred the smaller judge’s two misses; the result is unvalidated and covers one payload.Source: G RUNTIME MONITOR: EXTENDED RESULTS · p. 21

Classifier latency

H WEIGHT-DISTRIBUTION AUDIT: EXTENDED RESULTS

The weight-distribution audit compares a suspect checkpoint with a benign model trained using the same recipe. In these experiments, weight and activation distributions distinguish poisoned models, but the thresholds are recipe-specific and based on one benign seed.

Weight distribution: per-tensor breakdown

  • On every measured tensor in both models, poisoned-model KS statistics exceeded Benign SFT, with the strongest signals concentrated in early-layer MLP gate_proj and up_proj tensors.All three poisoned variants were within 8% of one another on every measured tensor and within 2% on the max-KS tensor.Source: H WEIGHT-DISTRIBUTION AUDIT: EXTENDED… · p. 22, Table 13 · p. 22

Step-matched analysis

Activation distribution: per-probe results

  • On 500 pre-2026 conversations without triggers, summed Gaussian-diagonal KL across hidden dimensions showed Qwen3-8B layer 2 separated cleanly at 1.20×, while Qwen3-4B layer 7 was borderline at 1.04–1.05×.With a single Benign SFT seed, the 4B difference could not be distinguished from run-to-run variance.Source: H WEIGHT-DISTRIBUTION AUDIT: EXTENDED… · p. 22, Table 15 · p. 23

Operating point and its limits

  • The proposed screen compares per-layer MLP gate_proj and up_proj KS statistics and diagnostic-layer Gaussian-KL against a matched Benign SFT model; ratios above 1.3× and 1.15× separated models in these data.At matched steps, the 8B KS ratio fell to 1.25×, below the KS threshold, leaving KL to distinguish it.Source: Table 12 · p. 22, H WEIGHT-DISTRIBUTION AUDIT: EXTENDED… · p. 23
  • The audit’s thresholds are point estimates from one benign seed, apply only to this recipe, and assume the defender knows or can recover that recipe.The study did not test tolerance to recipe mismatch.Source: H WEIGHT-DISTRIBUTION AUDIT: EXTENDED… · p. 23

I ALIGNMENT-DRIFT FINGERPRINTING: EXTENDED RESULTS

The extended alignment-drift results examine whether benchmarks distinguish poisoned models from Benign SFT and describe the evaluation’s compute footprint. The benchmark-vector question remains open, and the evaluation used about 80 A100-GPU-hours.

I ALIGNMENT-DRIFT FINGERPRINTING: EXTENDED RESULTS

  • Whether a multi-benchmark classifier can separate poisoned models from Benign SFT remains future work; the evaluation used one Benign SFT seed per family.The authors identify multiple benign seeds as a needed next step for estimating cross-benchmark covariance.Source: I ALIGNMENT-DRIFT FINGERPRINTING:… · p. 23
  • Poisoned SFT was within 0.3pp of Benign SFT on 8B TruthfulQA, while XSTest improved under all training conditions; the largest reported gap was a 7.7pp BBQ drop for 8B Poisoned SFT.Most TruthfulQA decline occurred between Base and Benign SFT, indicating a fine-tuning effect rather than a poisoning artifact.Source: I ALIGNMENT-DRIFT FINGERPRINTING:… · p. 23, Table 16 · p. 24

J COMPUTE RESOURCES

  • The full benchmark suite used approximately 80 A100-GPU-hours, including about 55 SUT-GPU-hours and 25 judge-GPU-hours running in parallel.No benchmark exceeded approximately 50 minutes under the reported configurations.Source: J COMPUTE RESOURCES · p. 24

Related papers

Loading 2609.32021v1…