Back Up Before You Regret: Measuring Backup Awareness and Automating Backup in Coding Agents

1 INTRODUCTION

• Existing harnesses protect workspaces or native-tool edits, while external-tool and remote-service state can be left uncovered and prior research largely focuses on post-hoc recovery.The introduction frames backup awareness—recognizing that modified state may later be needed and preserving it beforehand—as unexplored.Source: §1 · p. 2
• 86.1% task completion but only 15.2% backup before regret moments across six frontier models motivates RegretBench, built from 205 executable tasks grounded in reported regret moments.The benchmark evaluates whether agents preserve state before documented regret actions.Source: §1 · pp. 2–3
• BackUpper checks state-modifying tool calls and combines rules for clear operations with a lightweight subagent for complex cases, handling backups outside the main agent’s context.It covers every regret moment action, restores state with 92.2% average similarity, and costs $0.014 per task; stated user preferences yield 84.5% F1.Source: §1 · pp. 2–3
• BackUpper’s rule-based coverage can grow by distilling subagent decisions into reusable rules, doubling rule-based coverage after 30 tasks.The introduction also reports that BackUpper narrows its backup scope to match stated user preferences.Source: §1 · p. 2

Agent State Modification and Native Protection

• Mainstream harnesses protect workspaces or native edits, but none of the five analyzed harnesses covers state modified through external tools or remote services.BackUpper operates at the tool-call level to cover state wherever it lives.Source: §2 · p. 3

State Recovery and Reversibility

• RecoveryBench, GA-Rollback, WebRollback, and BackBench study recovery after failures, but post-hoc recovery may be difficult or infeasible after the original state is destroyed.RegretBench evaluates preservation before acting, and BackUpper automates it.Source: §2 · p. 3

Backup Needs and Regret

• Because future state needs are rarely known, experienced users make reversibility a habit, while whether agents share this backup awareness remains underexplored.RegretBench grounds tasks in real-world cases where users later wished they had backed up before an action.Source: §2 · p. 3

3 BENCHMARK: REGRETBENCH

• 205 reported regret moments are converted into executable tasks by extracting the triggering action, affected state, and later recovery need, then pairing each case with an environment and interaction trajectory.The affected state is placed outside native harness backup coverage; tasks are manually checked for executability and consistency with the reported loss.Source: §3 · pp. 3–4, Figure 2 · p. 4

3.1 BENCHMARK STATISTICS

• RegretBench covers five tools and three source categories, with source shares of 39.0% Social & Community, 44.4% Creation & Productivity, and 16.6% Records & Media.The tools are Terminal, SQL, Browser, Email, and DevOps, spanning personal and organizational backup needs.Source: §3.1 · p. 4
• Average task-instruction similarity is 0.240 across tools and 0.208–0.439 within tools, indicating varied task contexts.Terminal tasks are the most diverse, with within-tool similarity of 0.208.Source: §3.1 · p. 5

Main Results

• 15.2% average RMBR despite models passing most tasks shows that task completion does not imply backup awareness; Claude Opus-5 still misses regret-state backups in 35.6% of tasks.Across all runs, 72.0% complete the requested task without backing up the regret moment state.Source: Figure 4 · p. 5, §4 · p. 5

Execution Length and Main-Agent Self-Backup Overhead

• 30.1–58.9 tool calls per task and 8.9% of Claude Opus-5’s tool calls spent on self-backup indicate overhead from issuing backups through the main model.Each backup call uses the main model with its full context, motivating offloading backup.Source: §4 · p. 5, Figure 4 · p. 5

Harness Ablation

• 68.3–73.3% Opus RMBR, 5.0–8.3% Sol RMBR, and 0–1.7% DeepSeek RMBR across harnesses show sharper variation across models than harnesses.The evaluation uses a 60-task subset with harness protection standardized as in the main configuration.Source: Figure 5 · p. 6, §4 · p. 6

Model Generation Ablation

• GPT 5.4, GPT 5.5, and GPT 5.6-Sol scores rise from 39 to 47 while RMBR remains 5.0–6.7%, whereas Opus-5 reaches 73.3% RMBR at a similar index score and comparable effective size to GPT 5.6-Sol.Within the GPT 5.6 family, Sol and Terra have similar RMBR (5.9% vs. 4.4%) despite large size differences.Source: §4 · p. 6, Figure 5 · p. 6

5 OUR METHOD: BACKUPPER

• BackUpper is a training-free harness plug-in with a pre-tool-call hook that preserves affected state before modification and delegates complex cases to a subagent.Source: §5 · p. 6

5.1 BACKUP EXECUTION

• The four-tier execution first applies rewrite or pairing rules, then snapshots affected state, and uses a read-only checker and backup subagent for remaining complex calls.Source: §5.1 · pp. 6–7, Figure 6 · p. 7

5.2 METHOD EVALUATION

• 92.2% average similarity to the original state after restoration, versus 14.3% without restoration and 33.8% with Post-hoc Recovery; BackUpper covers all modifications on all models.Source: Figure 7 · p. 8, §5.2 · p. 8
• Skill-Based reaches 81.4% RMBR and 80.5% OWBR, while Destructive Command Guard reaches 8.1% RMBR and 7.3% OWBR.Source: Figure 8 · p. 8, §5.2 · p. 8
• 84.5% average F1 against human-annotated action-level labels on a 60-task, three-model subset, compared with 10.9% for original agents and 74.3% for Skill-Based with preferences.Source: Figure 8 · p. 8, §5.2 · p. 9

5.3 METHOD ANALYSIS

• 14.9% to 31.2% rule-based coverage on 65 Terminal tasks after generating rules from 30 external Terminal-Bench 4 tasks.Source: Figure 9 · p. 9, §5.3 · p. 9
• 7× fewer tokens per task than Skill-Based; with full cache hits, BackUpper reduces cost by 25× and latency by 31.7%.Source: §5.3 · pp. 9–10, Table 2 · p. 10
• On external tasks, BackUpper raises OWBR by 76.9 percentage points for GPT 5.6-Sol and 52.9 for Claude Opus-5 while leaving PR unchanged.Source: §5.3 · p. 10, Table 2 · p. 10

• BackUpper covers every regret moment action and restores state to 92.2% average similarity at $0.014 per task, using 7× fewer tokens than main-agent self-backup.Source: §6 · p. 10
• The authors propose mining regret moments from human–AI conversations and using RMBR as a training signal, given the gap between Claude Opus-5 and other models.Source: §6 · p. 10
• Future work could predict regrettable actions to skip low-risk backups and extend the tool-call-level approach to computer-use and web agents.Source: §6 · p. 10

A LIMITATIONS

• Human-reported regret moments may not fully capture agent-specific failure modes, and black-box evaluation limits mechanistic analysis of why agents back up or not.The Tier 4 subagent’s effectiveness may also be limited by its model’s capability; some third-party or server-generated state cannot be restored byte-identically even from a complete backup.Source: A LIMITATIONS · p. 14

B.1 SOURCING REGRET MOMENTS

• Of 557 deduplicated candidate reports, screening retained 205 reports with identifiable affected state and an explicit statement that users needed it afterward.Source: §B.1 · p. 14, Table 3 · p. 14

B.2 TASK DIVERSITY

• 0.208–0.439 average task-level instruction similarity across tool categories; Terminal tasks are most diverse at 0.208 and Email tasks most similar at 0.439.Source: §B.2 · p. 14

B.3 EVALUATION SETUP

• Table 4 lists the harness versions and models used in each harness for the main experiment.Source: §B.3 · p. 14, Table 4 · p. 15

B.4 EVALUATION VALIDATION

• 95.3% (667/700) of sampled actions received the same backed-up or not-backed-up assessment from human annotators and the judge.The sample included 350 actions labeled backed up and 350 labeled not backed up by the judge.Source: §B.4 · p. 15, Table 5 · p. 15

B.5 TASK COMPLETION AND BACKUP

• 72.0% of runs completed the task without backing up the affected state, while only 1.1% both backed up and failed the task.Runs were pooled across six models on all 205 tasks.Source: §B.5 · p. 15, Figure 10 · p. 15

B.6 RMBR BY TOOL AND APPLICATION SCENARIO

• Across tools and application scenarios, model differences exceeded category differences; SQL had the lowest tool rate at 12.7%, while development tasks had the highest scenario average at 17.5%.Three of six models never backed up before database modifications; Email and DevOps had the highest tool averages, 20.8% and 20.0%.Source: §B.6 · p. 15, Figure 11 · p. 16

C.1 BACKUP EXECUTION ALGORITHM

• In Full mode, the checker filters read-only calls before Tier 4, while personalized modes check preferences before all tiers; Tiers 2–4 preserve state before execution.Tier 1 replaces the original call, and self-reflection is disabled by default.Source: §C.1 · p. 16

C.2 BACKUP TIER DISTRIBUTION

• 30.8% of Terminal backup triggers were handled by rule-based Tiers 1–3 without a model call, while the remaining 69.2% invoked the backup subagent.The rule-based share comprised command rewriting (2.7%), command pairing (7.9%), and Tier 3 snapshots of files modified by native tools (20.2%); workspace snapshots were excluded.Source: §C.2 · p. 16, Figure 12 · p. 16

C.3 RULE SELF-REFLECTION

• After a subagent backup, self-reflection reviews the action, backup trajectory, and rule set, then adds, updates, or leaves rules unchanged and generalizes added or updated rules.The generalized rules cover broader action patterns or related tools.Source: §C.3 · pp. 17–18, Figure 13 · p. 17

C.4 CONTRIBUTION OF THE READ-ONLY CHECKER

• The read-only checker triggered 20.9% of regret-moment backups at 5.2% of cost without personalization, and 37.9% at 6.5% of cost with personalization.It checks unresolved calls and invokes the subagent when a call is not read-only, including when the main agent fails to mark the call for backup.Source: §C.4 · p. 18

C.5 RESTORATION EVALUATION

• Snapshots capture each affected environment before and after the action and after restoration, using each tool’s native export mechanism; scoring includes only objects modified by the action.Exports include PostgreSQL logical dumps for SQL and structured API or database reads for Browser.Source: §C.5 · p. 18
• 92.2% average state similarity was reached after BackUpper restoration, compared with 33.8% after Post-hoc Recovery and 14.3% without restoration.Similarity is character-level similarity to the pre-action snapshot, weighted by the number of tasks per tool.Source: §C.5 · p. 18
• Some restored states fall short because the main agent may omit state from its backup, or a service may reject captured state when it is written back.The examples concern email read/flagged state and metadata for other users’ deleted comments.Source: §C.5 · p. 18, Figure 14 · p. 19

C.6 FAILURE ANALYSIS OF SKILL-BASED BACKUP

• 75.1% of Skill-Based failures involved backups that preserved too little, 13.4% involved no backup, and 11.5% were other failures.In one email example, the export included message bodies and headers but not attachment contents needed for recovery.Source: §C.6 · p. 19

C.7 ADDITIONAL BASELINES

• AgentRewind reached 29.4% state similarity and used 12.4 GB of storage per task; post-hoc recovery required 859 seconds to propose and 265 seconds to execute a plan, using 1.1M tokens ($6.7) per task.AgentRewind’s Docker snapshots can miss regret-moment state in remote services.Source: §C.7 · p. 19, Figure 17 · p. 21

C.8 BACKUP MODES

• Across nine preference settings, BackUpper’s average F1 was 84.5%, exceeding Skill-Based’s 74.3%; BackUpper’s recall exceeded its precision in every setting.The three modes scope backups to high-impact changes, effects beyond the user’s request, or pre-existing human-authored content.Source: §C.8 · pp. 20–22, Table 7 · p. 21

C.9 BACKUP OVERHEAD

• BackUpper issued 10.1 backup tool calls per task versus 13.9 for Skill-Based and 17.0 without backup experiences; preference modes reduced subagent invocations by 49–84%.Without caching, BackUpper cost $0.024 per task versus $1.575 for Skill-Based.Source: §C.9 · p. 22, Table 8 · p. 22

C.10 SUBAGENT ABLATION

• Among four tested subagents, GLM 5.3-Flash matched GPT 5.6-Luna at 100% RMBR for similar cost, while DeepSeek V4-Flash cost $0.003 per task and reached 85.6%.The four subagents averaged 94.6% RMBR; most failures stemmed from insufficient backups.Source: §C.10 · p. 22

C.11 QUALITATIVE EXAMPLES

• The qualitative trajectories show BackUpper passing through read-only calls and routing other calls to rule-based tiers or the Tier 4 subagent, with the backup and log shown at the regret moment.The examples use Terminal and Email trajectories.Source: §C.11 · p. 22, Figure 18 · p. 23, Figure 19 · p. 24

D.1 BENCHMARK PROMPTS

• The task judge grades delivered work, while the backup judge checks out-of-workspace state changes for executed, timely, durable, and sufficient captures.The backup judge evaluates each state-changing tool call in trajectory order.Source: §D.1 · p. 25, Figure 20 · p. 25, Figure 21 · p. 26

D.2 BACKUPPER PROMPTS

• The prompts divide work among main-agent backup tagging, a pre-call subagent that captures pre-state and dependants within a budget, and reflection and generalization prompts that create reusable rules.Generalization extends a learned rule to related commands, entry points, and stores.Source: §D.2 · p. 25, Figure 22 · p. 26, Figure 23 · p. 27