PaperlayerView exact revision on arXiv

Source-linked AI summary

Getting Better at Working With You: Compiling User Corrections into Runtime Enforcement for Coding Agents

Yujun Zhou, Kehan Guo, Haomin Zhuang, Xiangqi Wang, Yue Huang, Zhenwen Liang, Pin-Yu Chen, Tian Gao, Nuno Moniz, Nitesh V. Chawla, Xiangliang Zhang

arXiv:2606.13174v1cs.LGcs.CL

TL;DR

Interactive coding agents may remember user corrections without reliably following them in future sessions. TRACE converts those corrections into runtime-enforceable rules, reducing preference violations across ClawArena and MemoryArena-derived tasks while preserving or improving task success.

  • Problem

    Interactive agents face an access-compliance gap: storing or retrieving user corrections does not ensure adherence to user-specific constraints across future sessions.

  • Method

    TRACE converts user corrections into persistent atomic rules paired with executable runtime checks that agents must pass before completing subsequent tasks.

  • Results

    TRACE reduced preference violations across ClawArena and MemoryArena-derived tasks while matching or exceeding memory baselines on task success.

  • Takeaways & Limitations

    Compiling corrections into runtime enforcement can reduce recurring preference friction beyond memory alone without paying a task-completion cost.

  • Takeaways & Limitations

    The downstream runtime-enforcement evaluation covers four model families because the required coding-agent harnesses support only Claude and GPT models.

Abstract

from arXiv · show

Interactive LLM agents are becoming part of daily work, but they do not reliably become easier to work with over time: a correction remembered in one session may still be violated in the next. We study this gap between preference access and preference compliance. In tasks derived from anonymized real-user friction cases, Mem0 memory still leaves 57.5% of applicable preference checks violated. We introduce Test-time Rule Acquisition and Compiled Enforcement (TRACE), a drop-in skill-layer pipeline for coding-agent runtimes that mines user corrections, rewrites them as atomic rules, and compiles them into runtime checks that must pass before an agent completes future tasks. Unlike runtime checks written ahead of time by developers, TRACE skills come from the user's own chat corrections. We evaluate TRACE with simulated user-in-the-loop experiments on ClawArena coding-agent tasks and MemoryArena-derived memory-intensive tasks. On ClawArena, TRACE reduces held-out preference violation from 100.0% to 37.6% on in-distribution tasks and from 100.0% to 2.0% on out-of-distribution tasks. On MemoryArena-derived tasks, TRACE reduces in-distribution violation from 100.0% to 60.5% while matching or exceeding the strongest memory baseline on task pass. These results suggest that compiling corrections into runtime enforcement can address a repeated-friction failure mode that memory alone does not reliably solve, reducing the need for users to restate the same correction across future sessions. Experiment code is available at https://github.com/YujunZhou/TRACE_exp, and the deployable skill is available at https://github.com/YujunZhou/tellonce.

1. Introduction

Interactive coding agents can remember user corrections without reliably complying with them across sessions. TRACE addresses this access-compliance gap by converting corrections into atomic rules with executable runtime checks, improving preference adherence on two benchmark families.

  • Motivation: Interactive agents increasingly execute extended repository tasks while expected to follow user-specific constraints across long-running work.These agents edit repositories, run commands, and inspect failures.
  • Problem: Remembering a correction makes it retrievable later but does not guarantee compliance, allowing the same errors to recur across sessions.A retrieved instruction to clean debug files may still be violated before termination.
  • TRACE: TRACE converts situated natural-language corrections into atomic rules paired with executable checks integrated into subsequent sessions.The representation change moves corrections from advice toward runtime-enforceable rules.
  • Results: 57.5% of applicable preference checks remain violated with Mem0 memory on anonymized real-world friction scenarios.This empirically confirms that memory alone does not close the access-compliance gap.
  • Results: 100.0% to 60.5% in-distribution violations were achieved on MemoryArena-derived tasks while matching or exceeding the strongest memory baseline on task success rates.The evaluation added user and project constraints around the original memory tasks.

2. Related Work

Related work spans preference learning and long-term memory for personalization, alongside runtime mechanisms that enforce agent behavior. TRACE connects these areas by compiling conversational corrections into enforceable rules using detection and lifecycle reconciliation mechanisms.

  • Preference learning and memory-based personalization: Preference-learning methods infer user intent from edits, feedback, and repeated interactions, while memory systems support long-term personalization.The cited work studies adaptation over extended interaction histories and complementary approaches to storing user information.
  • Runtime enforcement and coding-agent harnesses: Runtime-enforcement research uses specifications, guard agents, policy compilers, or generated checks to constrain agent behavior.These approaches provide complementary mechanisms for enforcing behavior during agent execution.
  • Runtime enforcement and coding-agent harnesses: TRACE adds a correction-signal detector and a five-action lifecycle resolver to compile durable conversational preferences and reconcile candidate rules as corrections accumulate.The detector identifies messages worth compiling, while the resolver reconciles each candidate rule with the user’s current rule library.

3. Measuring the Access–Compliance Gap

The paper defines the access–compliance gap: agents may retrieve or receive a user correction yet still violate it in their final response or workspace state. A six-model diagnostic shows that compiled runtime constraints improve compliance beyond memory or contextual rule access alone.

  • The access–compliance gap occurs when a correction is available in context but the agent’s final response or workspace state still violates it.
  • The diagnostic benchmark extracts 142 correction-conflict records from 32 long-context transcripts and evaluates 19 held-out tasks containing 29 manually annotated preference checks.The checks span cleanup requirements, workflow constraints, and stylistic conventions, and require strict adherence in the final response or workspace state.
  • 70.1% compliance is achieved by Compiled Rules, compared with 42.5% for Mem0, 55.0% for All Rules, 54.0% for Relevant Rules, and 31.6% for No Rules.Figure 2 reports mean compliance across six models and 29 sub-rule checks.
  • Agents commonly acknowledge, retrieve, or plan around corrections but abandon them while optimizing for the immediate coding task.
  • These findings motivate changing correction representation so preferences become conditions the agent must satisfy rather than merely information present in context.

4. Test-time Rule Acquisition and Compiled Enforcement (Trace)

TRACE converts user corrections into contextual, atomic rules and compiles them into runtime-enforceable artifacts for future coding-agent tasks. It resolves evolving rules and applies task-matched instructions and verification mechanisms rather than relying on passive memory retrieval.

  • Correction detection and rule extraction: TRACE scans every user utterance, routes correction signals to a compiler, and emits per-user enforcement artifacts while ordinary messages bypass compilation.Correction signals include durable preferences, repeated errors, and workflow friction; artifacts can be skills, hooks, or runtime checks.
  • Correction detection and rule extraction: Each correction record becomes an atomic directive paired with precise applicability conditions, preventing narrow feedback from becoming an overbroad instruction.For example, leftover debug-file feedback becomes a cleanup rule triggered only when temporary artifacts are created.
  • Rule resolution: TRACE resolves new rules with five actions: Noop, Update, Supersede, Split, and New, maintaining evidence, the active rule version, and applicability boundaries.These actions handle repeated, refined, contradictory, bundled, and previously uncovered preferences.
  • Compiled enforcement: TRACE compiles each atomic rule into an applicability check, behavior instruction, and verifier, using deterministic, semantic, or intent-level enforcement tiers.Deterministic rules inspect structures or workspace state, semantic rules evaluate generated content, and intent-level rules issue runtime reminders.
  • Compiled enforcement: At task start, applicable rules load instructions into context and register verifiers with hooks that monitor execution and require revision when checks fail.A failed verifier reports the violation and supporting evidence to the agent.

5. Experiments

TRACE was evaluated with repeatable simulated user feedback on ClawArena and MemoryArena-derived tasks, using task pass, violation rate, and mean corrections. Compiled enforcement substantially reduced preference violations, especially across unseen ClawArena scenario families, while preserving task performance and reducing repeated user corrections.

  • Experimental setup: Simulated users produced repeatable feedback judgments, including 357 corrections and 433 non-correction cases across a 30-turn stability window.The balanced set contained 790 judgments overall.
  • Experimental setup: The experiments covered 62 ClawArena scenario templates and MemoryArena-derived agent-memory tasks, reporting task pass, violation rate, and mean corrections.TRACE skills were implemented for Codex and Claude Code and compared with No Memory, Mem0, Hindsight, and ReMe-Light.
  • In-distribution results: 37.6% average violation rate was achieved by Trace on in-distribution ClawArena tasks, while all memory baselines remained above 50%.Task pass stayed within a narrow band across Trace, No Memory, and ReMe-Light, indicating no task-completion cost.
  • In-distribution results: 60.5% violation rate and 17.3% task pass were achieved by Trace on MemoryArena-derived tasks, respectively the lowest violation rate and highest task pass.Because hidden user and project constraints define task success, enforcement gains and task pass are mechanically coupled.
  • Out-of-distribution results: 2.0% average violation rate was achieved by Trace on unseen ClawArena scenario families, with zero observed violations on three of four models.Task pass remained in the same band as No Memory despite the distribution shift.
  • Interaction cost: 1.02 average user turns were required by Trace on out-of-distribution ClawArena tasks, compared with 2.00 for No Memory and 1.31 for Mem0.Trace required 1.37 turns in-distribution, and its runtime averaged 42.5 seconds per round without an expensive interaction loop.

6. Conclusion · A. Simulated User Fidelity Details · A.1. Simulator Construction

TRACE addresses the gap between remembering user corrections and enforcing them during execution, reducing preference violations on ClawArena and improving outcomes on MemoryArena-derived tasks. The appendix describes simulated users built from historical correction profiles and evaluated through a four-step correction-decision procedure.

  • 6. Conclusion: TRACE treats each user correction as evidence for a runtime-enforceable rule rather than advisory text.The paper frames this as addressing the access-compliance gap: retrieving or contextualizing a preference does not ensure agents follow it.
  • 6. Conclusion: ClawArena efficiency was evaluated using average wall-clock seconds and user turns per round across in-distribution and out-of-distribution splits.User turns include the initial request and simulated corrections, so fewer turns indicate fewer repeated user interventions.
  • 6. Conclusion: 100.0% to 37.6% in-distribution and to 2.0% out-of-distribution preference violation on held-out ClawArena tasks.These reductions occurred with comparable task pass and lower user-turn cost.
  • 6. Conclusion: On MemoryArena-derived tasks, TRACE achieved the highest task pass and lowest violation rate among all conditions.These tasks include constraints as part of the success criterion.
  • A. Simulated User Fidelity Details: The appendix details construction and validation of the simulated users introduced in Section 5.This appendix provides the simulated-user fidelity details supporting the main evaluation.
  • A.1. Simulator Construction: Each simulated user is built from a conditional profile mined from the user’s historical correction data.Profiles contain situations, preferences, likely user reasoning, and example correction language.
  • A.1. Simulator Construction: At evaluation time, the simulator infers the user’s state, matches profile entries, decides whether to correct the agent, and performs a final rule miss check.This is a deterministic four-step decision procedure based on recent interaction context.

A.2. Fidelity Evaluation · B. Access-Compliance Diagnostic Details

The simulator is evaluated on held-out historical interactions using correction and matched non-correction moments, with fidelity measured by decision and rule-recovery metrics. The appendix also documents the access-compliance diagnostic’s data pipeline, task-selection filters, and preference-store construction.

  • A.2. Fidelity Evaluation: The simulator is validated on held-out historical user interactions.
  • A.2. Fidelity Evaluation: Positive examples are real correction moments, while negative examples are matched moments without user corrections.
  • A.2. Fidelity Evaluation: A simulation is correct when it predicts whether the user would correct and targets the same issue on positive examples.
  • A.2. Fidelity Evaluation: Decision precision measures correspondence to real corrections, while decision recall measures recovery of real correction moments.
  • A.2. Fidelity Evaluation: F1 is the harmonic mean of decision precision and recall, and specificity measures silence on matched non-correction moments.
  • A.2. Fidelity Evaluation: Rule recall measures whether generated corrections recover the human-labeled preference rule on positive examples.
  • A.2. Fidelity Evaluation: Table 1 reports aggregate simulator fidelity on 790 held-out judgments: 357 correction cases and 433 non-correction cases.
  • B. Access-Compliance Diagnostic Details: The access-compliance diagnostic appendix covers data collection, held-out task-selection filters, and construction of each preference store.

B.1. Data Collection, Held-Out Task Selection, and Annotation

The study uses anonymized coding-agent sessions from one researcher to isolate the access–compliance gap, then constructs 19 held-out tasks from repeated, deduplicated, self-contained correction records. These tasks remain excluded from both rule construction and the Mem0 store, which use the other 123 records.

  • Data collection: 32 transcripts and 142 correction-conflict records came from approximately two months of one AI researcher’s daily coding-agent work.The single-user design isolates the access–compliance gap from conflicting preferences across users.
  • Data collection: All transcripts were anonymized by redacting project names, file paths, and identifying details while preserving correction structure.
  • Held-out task selection: 19 held-out evaluation tasks were hand-curated from 142 records using repetition, deduplication, and self-contained-context filters.The filters retain repeated failures, remove near-duplicate preferences, and require task context expressible in one prompt.
  • Annotation and leakage control: The 19 held-out tasks were excluded from rule construction, leaving 123 correction-conflict records for both the operational rule library and Mem0 store.No information from held-out tasks entered either store.

B.2. Preference Stores

The diagnostic compares three preference stores and shows that memory access can surface relevant feedback, but only an explicit rule-store constraint makes it a condition that must be checked.

  • Preference-store comparison: The diagnostic uses three preference stores, with Table 2 detailing the stores and Table 3 showing an anonymized correction across them.Table 3 paraphrases the stored structure rather than reproducing raw transcript snippets.
  • Preference-store comparison: The rule store preserves target behavior as a direct constraint, whereas Mem0 exposes extracted memory text from the surrounding conversation.The example is anonymized and presented across the stores to highlight their structural difference.
  • Preference-store comparison: Retrieved memory remains advisory unless the system turns it into a condition that must be checked.This distinction is central to the diagnostic of preference access versus enforceable preference compliance.

C. TRACE Implementation Details

The appendix provides concrete implementation details for the deployed TRACE pipeline, supplementing Section 4. It covers the model, compiled-rule structure, retry policy, lifecycle resolution, and a worked example.

  • Overview: The appendix supplements Section 4 with implementation details for the deployed TRACE pipeline.It focuses on concrete deployment details rather than introducing a separate evaluation.
  • Pipeline components: It describes the lightweight LLM and the structure of the compiled rule library.
  • Execution and example: It documents the retry policy, lifecycle resolver’s decision protocol, and a worked example.

C.1. Lightweight LLM for Detection and Compilation … C.5. Worked Example: F2 (Cleanup Logs After Project Completion)

TRACE uses a lightweight LLM to detect corrections, extract atomic rules, and compile verified enforcement artifacts, organized within a hierarchical rule library and bounded retry policy. Its lifecycle resolver manages rule evolution, while the F2 example shows how a correction becomes a deterministic Bash hook.

  • C.1. Lightweight LLM for Detection and Compilation: Gemma 4 31B performs correction-signal detection, atomic-rule extraction, and enforcement-artifact compilation across all three pipeline stages.The model is served through DeepInfra and configurable via the COMPILE_MODEL environment variable.
  • C.1. Lightweight LLM for Detection and Compilation: Each compilation pass requires JSON parsing, schema validation, canonical-input unit testing, and sandbox replay before acceptance.Outputs failing any self-verify stage are rejected.
  • C.2. Library Structure and Enforcement Tiers: The deployed hierarchical library contains 47 entries, with atomic rules and composite parents carrying their own verifiers.Refinement children activate jointly with their parent under shared applicability, and all entries carry a verify-retry enforcement marker.
  • C.3. Retry Policy: MAX_RETRY=3 caps the diagnostic loop at three additional retries, whereas simulated evaluation is bounded by two user turns per task.A diagnostic run terminates after the third failed retry with the violation logged; simulated metrics reflect the two-turn ceiling.
  • C.4. Lifecycle Resolver: Decision Protocol and Audit: The lifecycle resolver assigns candidate rules one of five actions by comparing their semantics with existing rules in the same domain.The supplied action descriptions include Noop, Update, and Supersede; the resolver uses the same model and prompt scaffold as detection and compilation.
  • C.4. Lifecycle Resolver: Decision Protocol and Audit: A forced decision line must precede any rule-library write, but misclassified Supersede decisions require manual restoration from archived files.The deployment relies on occasional manual audits because its library contains 47 entries.
  • C.5. Worked Example: F2 (Cleanup Logs After Project Completion): F2 converts a correction to clean log files after sub-project completion into a rule triggered when Bash writes to a log-file-naming pattern.The originating correction concerned leaving another run_log_xxxx file before project closure.
  • C.5. Worked Example: F2 (Cleanup Logs After Project Completion): F2 compiles into a deterministic PreToolUse-Bash hook that blocks matching commands and returns the rule text and matched snippet for revision.The hook reads tool_input.command, returns verdict: allow or verdict: block, and uses exit code 2 for blocked calls.
Loading 2606.13174v1…