Source-linked AI summary
The Cold-Start Safety Gap in LLM Agents
Chung-En Sun, Linbo Liu, Tsui-Wei Weng
TL;DR
The paper asks whether tool-calling agents remain equally safe throughout multi-task conversations, addressing benchmarks that typically present threats in isolation. It introduces SODA to vary the number of preceding regular tasks and evaluates safety, representations, ablations, and external benchmarks. Safety improves with depth, while warm-up preserves utility and is recommended before safety-critical deployment.
Problem
Existing safety benchmarks test threats in isolated sessions, leaving unclear whether a harmful request’s position within a multi-task conversation affects agent safety.
Method
The paper introduces SODA, which controls depth D as the number of regular tasks completed before a safety-critical threat, evaluating depths from D=0 to D=20.
Results
Safety improves by 9–52% across 7 models as depth increases from D=0 to D=20, while regular task requests primarily drive safety and prior responses preserve utility.
Takeaways & Limitations
Briefly warming agents with regular agentic tasks before deployment improves safety while preserving utility, with D=5 to D=10 typically sufficient.
Takeaways & Limitations
The study focuses on open-source models because closed-source APIs may be costly, lack hidden states, and apply external guardrails that confound measurement.
Abstract
from arXiv · showhide
Are tool-calling LLM agents equally safe throughout a conversation? We discover they are not: agents are most vulnerable at the very start of a session and become substantially safer after a few regular agentic tasks -- a phenomenon we term the cold-start safety gap. To study this systematically, we introduce Safety Over Depth for Agents (SODA), a benchmark that controls how many regular agentic tasks the agent completes before encountering a safety threat, supporting up to 20 preceding tasks. Evaluating 7 models from 4 families, safety improves by 9--52% as the number of preceding regular agentic tasks increases from zero to twenty. Representation analysis confirms that model hidden states gradually shift toward a safety-aligned region as more preceding tasks are present. By systematically studying which part of the preceding conversation matters most, we find that the regular agentic tasks themselves are the primary driver of safety, while the agent's own prior responses have less effect on safety but are essential for preserving later utility. This conclusion is further supported by evaluation on open-source safety benchmarks (AgentHarm, Agent Safety Bench) and utility benchmarks (BFCL, API-Bank), confirming that warming up the agent with regular agentic tasks before deployment makes it safer and preserves full capability. Based on these findings, we recommend a simple deployment strategy: having the agent complete a few regular agentic tasks before possible exposure to safety-critical requests mitigates the cold-start safety gap. Our code is available at https://github.com/Trustworthy-ML-Lab/Agent-Cold-Start-Safety-Gap
1 Introduction
The paper identifies a cold-start safety gap: agents are most vulnerable when a session begins, but become safer after completing ordinary agentic tasks. It introduces SODA and finds that task history drives safety improvements, while preserving prior interaction responses maintains utility.
- Existing benchmarks usually test threats in isolated sessions, leaving the effect of threat position within a conversation unstudied.
- SODA evaluates agent safety across controlled conversation depths, varying the number of regular agentic tasks before each threat.
- Ablation identifies regular task requests as the primary safety driver, while agent responses matter more for preserving utility.
- A brief warm-up of regular agentic tasks is recommended because it improves safety without sacrificing agentic utility.The paper describes this warm-up as a deployment strategy before safety-critical requests.
- Representation analysis shows hidden states migrate across a linear safety boundary as depth increases.
2 Benchmark: Safety Over Depth for Agents (SODA)
SODA addresses the unstudied effect of threat position by injecting identical safety threats after controlled numbers of ordinary agentic tasks. Its benchmark spans diverse environments, threats, depths, and real tool-calling interactions.
- SODA controls threat position by evaluating the same safety threats at multiple conversation depths.
- The benchmark uses depth-controlled threat injection, placing each threat after a controlled number of regular agentic interactions.
- SODA covers 16 tool-calling environments, 80 scenarios, and 400 unique threats involving privacy, financial, access, and social-engineering harms.
- Regular preceding tasks use the same tools as threats but contain no safety-related content, with tasks sampled from each scenario’s pool.
- Depth D counts completed regular tasks before the threat, using D ∈ {0, 1, 3, 5, 7, 10, 15, 20}.At D=0, the threat is the agent’s first task; at D>0, regular tasks precede it.
- For each depth, the agent completes sampled tasks through real tool interactions before receiving the threat as the next task.
- The full design yields 3,200 test cases from 400 safety tasks evaluated at eight depths.
3 The Cold-Start Safety Gap
SODA reveals a cold-start safety gap: agents are least safe at D=0 and become progressively safer as they accumulate regular agentic interactions. Hidden-state analyses show representations migrating across a safety boundary as depth increases.
- All 7 models from 4 families exhibit the cold-start safety gap, indicating the pattern is not limited to one model family.
- PCA projections separate safe and unsafe outcomes in hidden-state space, with a linear boundary achieving classification accuracy above 0.9 across models.The analysis extracts hidden states at the first generated-token position for each harmful query and projects them into two dimensions.
- As depth increases, harmful-query representations migrate from the unsafe region at D=0 toward and across the safety boundary.By D=10, many representations have crossed into the safe region.
4 What Drives the Safety Improvement?
Ablations identify regular agentic task requests as the primary driver of safety improvement, while response content contributes less to safety. Even minimal preceding context improves safety over cold-start, but prior responses matter for later utility.
- The warm-up ablation varies task requests, agent responses, or both to isolate which conversation component drives safety.The design includes full interaction, fixed-request variants, fixed-response variants, and all-empty or all-random histories.
- Regular agentic task requests are the primary driver of safety improvement in the warm-up history.Adding task requests alone improves safety by 17% on average, compared with 8% when adding agent responses alone.
- Preserving task requests produces safety boosts across response variants, showing that response content matters less for safety.Full, compliant, random, and empty-response variants all exhibit safety improvements.
- The findings suggest that regular tasks activate an agent persona, making appropriate caution more likely.This is presented as a hypothesis explaining why task requests dominate response content for safety.
- Any preceding context improves safety over cold-start, including degraded histories containing empty or random content.Even minimal conversational structure can partially activate the agent persona.
5 Does the Warm-Up Generalize and Preserve Utility?
Warm-up generalizes beyond SODA to external safety benchmarks, while preserving utility depends on retaining real agent interaction. Full Interaction provides the strongest combination of safety gains and utility preservation.
- External evaluations prepend matched regular agentic tasks and compare performance at D ∈{0, 5, 10, 20}.The design brings isolated safety and utility benchmarks closer to the multi-turn setting studied in SODA.
- The warm-up effect generalizes to AgentHarm and Agent Safety Bench across multiple warm-up variants.Full Interaction gains +23% on AgentHarm and +8% on ASB; all variants show non-trivial improvement on both benchmarks.
- Compliant and random response warm-ups substantially degrade utility despite preserving safety gains.Compliant responses encourage a lazy no-tool-calling pattern, while random responses create incoherent history; empty responses degrade utility less.
- Full Interaction preserves tool-calling utility best on BFCL Multi-Turn and API-Bank.The benchmark suite evaluates sequential stateful actions on BFCL and parameter-matched tool calls across 207 API-Bank tasks.
6 Summary and Deployment Recommendation
The paper attributes the cold-start safety gap mainly to missing regular agentic tasks and proposes warm-up as a deployment mitigation. Full Interaction is recommended because it improves safety while preserving utility, with Empty Response as a lower-cost alternative.
- The cold-start safety gap is mostly driven by the absence of regular agentic tasks, not by the content of prior agent responses.
- Full Interaction warm-up typically uses D=5 to D=10 regular tasks before deployment and provides substantial safety improvement with no utility loss.
- Empty Response prefill offers a budget-constrained alternative that boosts safety meaningfully with only minor utility cost.It prepends regular task requests without agent responses.
- Both warm-up strategies require no fine-tuning or data collection, only a brief warm-up period at session start.
7 Additional Experiments
The paper compares warm-up with prompting, refusal demonstrations, and safety fine-tuning. Warm-up is the only strategy reported to close the cold-start gap while preserving utility.
- Safety system prompts raise baseline safety but do not close the gap between D=0 and D=20.The authors characterize the remaining vulnerability as structural rather than solvable through simple prompt engineering.
- In-context refusal demonstrations improve safety but introduce instability, including catastrophic safety drops in some cases.
- Safety fine-tuning achieves high safety on SODA, AgentHarm, and ASB but sharply reduces tool-calling utility.For Qwen3-4B, BFCL Multi-Turn accuracy falls from 64.0% to 17.0%, while API-Bank falls from 85.6% to 64.8%.
- Warm-up remains the most practical strategy because it improves safety without the utility collapse associated with alternative interventions.The paper reports almost no computational overhead for warm-up.
8 Related Work
Existing agent-safety benchmarks evaluate threats in isolated sessions, whereas this paper treats threat position within a multi-task conversation as a critical safety variable. It contrasts this focus with prior work on adversarial contexts that degrade safety.
- AgentHarm, ASB, ToolEmu, and R-Judge evaluate safety with each task presented in isolation at D=0.This setup implicitly assumes that safety remains constant throughout an agent session.
- The paper identifies harmful-request position within an agentic session as a critical variable absent from prior safety benchmarks.
- Prior multi-turn safety research studies adversarial context, including escalating requests, decomposed jailbreaks, and harmful demonstrations.The paper contrasts these contexts with regular preceding context that has no safety-relevant content.
9 Conclusion
The conclusion identifies a cold-start safety gap in tool-calling agents and recommends brief warm-up with regular agentic tasks before safety-critical exposure. The finding is linked to conversational depth, task requests, utility preservation, and hidden-state movement.
- Tool-calling agents are most vulnerable at D=0 and become safer after completing regular agentic tasks.Representation analysis shows hidden states migrating across a safety boundary as conversation depth increases.
- Ablation identifies regular task requests as the primary safety driver, while prior responses matter less for safety but help preserve utility.
- A brief warm-up before safety-critical requests requires no fine-tuning, no data collection, and almost zero computational overhead.
- The reported findings generalize to external safety and utility benchmarks.
Limitations
The study focuses on open-source models because its scale, hidden-state analysis, and harmful-input evaluation make closed-source API evaluation problematic. The authors nevertheless report the gap across four independently trained model families.
- The evaluation uses open-source models because its scale makes API-based experiments prohibitively expensive.The study includes 7 models, 8 ablation variants, 8 depths, 3 runs, and 4 external benchmarks.
- Representation analysis requires hidden-state access, which closed-source APIs do not expose.
- Closed-source external guardrails can block harmful test inputs and confound measurement of the model’s own behavior.
- The reported cold-start safety gap appears across four independently trained model families despite the open-source scope.
Ethical Considerations
The study uses harmful requests solely for evaluation and acknowledges that disclosing the cold-start vulnerability could inform attacks on fresh sessions. It frames warm-up as an easy mitigation without computational overhead.
- The authors acknowledge that the finding could inform adversarial strategies targeting fresh sessions.
- Harmful requests are used solely for evaluation purposes.
- The paper argues disclosure is net-positive because the vulnerability already exists in deployed agents.
- The proposed mitigation is warm-up with regular agentic tasks, described as easy to implement without computational overhead.
- SODA evaluates safety across realistic environments containing ordinary tool-actionable tasks and harmful threat variants.
B Representation Analysis: More Models
Representation analysis shows hidden states progressively migrate from an unsafe region toward a safety boundary as conversation depth increases. Additional experiments compare prompt instructions, refusal demonstrations, and fine-tuning as alternatives to interaction warm-up.
- Representation Analysis: More Models: The estimated safety boundary achieves >0.9 classification accuracy across all models shown.
- Representation Analysis: More Models: At D=0, representations cluster in the unsafe region and progressively migrate into the safe region as depth increases.
- Safety Prompt: A safety system prompt raises initial safety rates but does not close the D=0-to-D=20 gap.
- ICL Refusal: ICL Refusal reaches 90–99% safety on SODA at D=20 but is unstable across models, increases over-refusal, and degrades utility.
- Comparison of Alternatives: Full Interaction warm-up provides a more balanced alternative by producing consistent safety gains without degrading utility or causing over-refusal.
- Safety SFT: Agentic safety fine-tuning raises D=0 safety but sharply degrades tool-calling utility, including BFCL Multi-Turn from 64% to 17%.