After the Fix: Transfer of Corrected Agent Experience

1 INTRODUCTION

• The paper frames repair value and reuse value as distinct: acceptance validates a source outcome, not whether its lessons apply to a new task.Some corrections establish reusable obligations, while workbook versions, valuation conventions, and input ranges may require renewed grounding.Source: §1 · p. 1
• The study distinguishes correction gains (C–U) from gains over independent execution (C–B-only), using the same source and fixed target to hold provenance constant.The two comparisons separate whether repair improves a particular memory from whether that memory is preferable to a fresh start.Source: §1 · p. 1
• Across 3,300 runs, the evaluation covers ThinkingBox workflow reuse and matched APEX cohorts with and without source-state inheritance.It compares Full, Skill, Hybrid, None, Summary, and Execution handoffs; B-only receives no A experience, and none of the six retains the shared state channel.Source: §1 · p. 1, Figure 1 · p. 2
• The contribution separates correction gains, restoration of baseline success, and success beyond both references, while asking why correction sensitivity may not rank corrected-memory quality.It also uses action traces and token/call counts to examine workflow reuse and execution costs.Source: §1 · p. 2

From repairing a task to transferring its correction

• Prior work studies refinement through feedback (Madaan et al., 2023; Shinn et al., 2023), while this study treats source acceptance as the starting point for transfer.The target task is distinct from the repaired source task.Source: §2 · p. 2

Experience reuse and representation

• The paper’s comparison revises the same failed episode and uses fixed-target, pre/post-repair results to measure the update’s incremental value.B-only separates restoration from success absent under both pre- and post-repair references.Source: §2 · p. 2

Content, state, and the work of using memory

• Related work connects retained content and memory formats with performance and cost; this paper compares accepted-segment content, state/text handoffs, and token versus action savings.Source: §2 · p. 2

3 DESIGN: FROM AN ACCEPTED REPAIR TO A NEW TASK

• For each regime, 100 source–target pairs are evaluated; C is the first accepted repair within ten feedback–repair rounds, while U stops after initial failure.The design records source history and saved state for each branch.Source: §3 · p. 3
• Handoffs vary episode content while separating text from environment state; text is constructed from source A, and only physical inheritance changes the target environment.ThinkingBox supplies terminal-state text; APEX applies saved source-branch changes to the target world.Source: §3 · p. 3

3.1 WHAT EXACTLY IS HANDED OVER?

• The six handoff formats vary selection, ordering, abstraction, and length, while each also retains its regime’s shared state channel.None/Full/Skill/Hybrid use both U and C; Summary/Execution use C alone, and Execution adds the accepted segment to the shared summary and initialization.Source: Table 1 · p. 3, §3.1 · p. 3

3.2 QUESTIONS AND CONTRASTS

• The questions compare C with U and B-only, Execution with Summary, and Execution with Full on cumulative input tokens and model calls.The hypotheses predict correction and baseline-relative gains, added value from accepted actions, and reduced costs from smaller handoffs.Source: §3.2 · pp. 3–4

4 EVALUATION AND STATISTICAL ANALYSIS

• ThinkingBox contains 65 insurance and 35 consulting pairs; the 100 APEX pairs span three domains and 16 worlds, with 200 distinct pairs across its two matched regimes.APEX pairs are annotated into same-family, shared-procedure, shared-evidence, and context-only categories.Source: §4 · p. 4
• ThinkingBox is scored with executable assertions, while APEX evaluates answers and artifacts against original rubrics using native visual evidence.Each condition retains 100 targets, including budget failures; text artifacts reconstruct B’s original inputs plus its delta, not A’s.Source: §4 · p. 4
• Acceptance comparisons use two-sided exact McNemar tests with Holm-adjusted contrast families, alongside paired and group bootstrap intervals and exhaustive group sign flips.The group sign-flip analysis assumes independent groups and within-group joint exchangeability under the null, not causal randomization.Source: §4 · p. 4
• Analyses are retrospective rather than preregistered, and the two regimes use fixed but different model and execution settings.APEX uses deepseek-flash with thinking, 100 steps, and 7,200 seconds; ThinkingBox uses deepseek-v4-flash, high reasoning, 400 interactions, and 1,200 seconds.Source: §4 · p. 4

5.1 CORRECTION HELPS ACROSS REPRESENTATIONS IN THINKINGBOX, BUT NOT GENERALLY IN APEX

• ThinkingBox C-Full/Skill/Hybrid accept 67/64/60 targets, exceeding their U versions and B-only; neither APEX regime establishes comparable aggregate advantage.Absolute target acceptance is reported for 100 targets per cell.Source: Table 2 · p. 4, §5.1 · p. 4, Figure 2 · p. 5
• Across 42 ThinkingBox targets, at least two of Full, Skill, and Hybrid improve when U changes to C; 18 improve under all three.The corresponding counts are five and one in APEX-state, and six and one in APEX-text; ThinkingBox’s C–U advantages remain positive after removing any single family.Source: §5.1 · p. 5

5.2 A CORRECTION GAIN CAN RESTORE A SUCCESS THAT MEMORY DISPLACED

• Full has 46 upward U-to-C transitions and two downward transitions, but 22 upward transitions restore successes already observed under B-only.The other 24 upward transitions occur where both B-only and U fail, so C–U combines restoration with success beyond both references.Source: Figure 3 · p. 5, §5.2 · p. 5
• APEX-text Full has ten U-failure/C-success transitions and ten in the opposite direction, leaving acceptance at 45% despite 20 changed outcomes.APEX-state Full similarly has eight gains and nine losses; these paired outcomes are not individual causal effects.Source: Figure 3 · p. 5, §5.2 · p. 5

5.3 THE REPRESENTATION THAT IMPROVES MOST IS NOT NECESSARILY THE BEST TO REUSE

• Full’s 15-point larger correction gain over Skill mostly reflects weaker U performance, while corrected Full acceptance is only three points higher.The adjusted representation tests do not establish the three-point C-Full advantage.Source: §5.3 · p. 6
• C-None accepts 36 ThinkingBox targets versus 67, 64, and 60 for C-Full, Skill, and Hybrid; APEX Hybrid changes are not established.In APEX-text, Full and Skill each accept 45 targets but disagree on 24, with twelve successes exclusive to each; Hybrid’s corrected acceptance shifts from 42% to 45% with state inheritance and 45% to 40% without it.Source: §5.3 · p. 6

5.4 BEHAVIOR DISTINGUISHES REUSABLE OBLIGATIONS FROM SOURCE-SPECIFIC CHOICES

• ThinkingBox’s lifecycle procedure helps execute the target-specific decision, while APEX traces show a source-approved discount convention overriding B’s mid-year request.The selected traces motivate preserving obligations with their conditions, but do not establish approval-marker causality or mechanism frequency.Source: §5.4 · pp. 6–7
• In Helios, C-Hybrid narrows the input from 4,900 SKU rows to B’s specified 2,500-row Rebuilt input and recovers the correct result.Both APEX regimes show B-only pass, U-Hybrid fail, and C-Hybrid pass; selected audits are non-blind, and the 3M target does not specify which workbook version to use.Source: §5.4 · p. 7

6.1 ACTION DETAIL IMPROVES ON A SUMMARY, NOT NECESSARILY ON STARTING FRESH

• APEX-text Execution accepts 52 targets versus Summary’s 40, but its 12-point edge is only five points over B-only’s 47%.The text comparison survives target-level H2-family adjustment, but not global-24 or group adjustment; no regime establishes Full’s advantage over Execution.Source: Figure 5 · p. 7, §6.1 · p. 7

6.3 A SMALLER HANDOFF REDUCES CONTEXT READING BUT CAN REQUIRE MORE EXECUTION

• In APEX-text, Execution cuts mean input from 3.576M to 2.438M tokens (31.8%) versus Full, while calls rise from 30.01 to 35.08 (16.9%).Input per call falls from 119.2k to 69.5k; lower repeated context consumption is not the same as less work to complete B.Source: Figure 5 · p. 7, §6.3 · p. 8

6.2 NAMING A REQUIREMENT IS DIFFERENT FROM CARRYING IT OUT

• ThinkingBox cases show Execution implementing a ticket-type requirement that Summary already names, suggesting a handoff can fail between recognizing and carrying out a rule.The contrast does not isolate which part of the added segment produced the improvement.Source: §6.2 · pp. 7–8

6.3 A SMALLER HANDOFF REDUCES CONTEXT READING BUT CAN REQUIRE MORE EXECUTION

• Execution approximately doubles APEX-state calls from 20.04 to 41.22, limiting input reduction to 12.1%; in ThinkingBox, calls rise 4.7% while input falls 18.7%.Execution also uses 12.2% more input than Summary, and B-stage token reductions are not whole-pipeline savings.Source: §6.3 · p. 8

7 DISCUSSION: WHAT DOES REPAIR ADD TO REUSABLE EXPERIENCE?

• C–U measures revision gains, while C–B-only tests whether using the repaired memory beats independent execution.Holding the same source provenance and target fixed isolates the revision comparison.Source: §7 · p. 8
• ThinkingBox Full gains 44 points (23% to 67%) versus Skill's 29 points (35% to 64%), but 12 of Full's 15 extra points reflect weaker uncorrected performance.The corrected versions differ by only three points.Source: §7 · p. 8
• Of Full's 46 upward transitions, 22 restore B-only success, while the other 24 succeed beyond both B-only and U.Both patterns count as gains over U, but only the latter succeeds beyond both references.Source: §7 · p. 8
• ThinkingBox's open–act–close workflow remains applicable after an eligibility change, whereas APEX's source-approved full-year convention conflicts with B's mid-year requirement.In APEX, B computes both alternatives but selects the source convention; neither APEX regime establishes ThinkingBox-like aggregate correction gains.Source: §7 · p. 8
• A memory entry should preserve the correction, its justification, and dependent inputs or artifacts, then check whether those conditions hold for the target.A ticket lifecycle may travel across customers, while workbook versions and discounting conventions require checking against the new request.Source: §7 · p. 9
• Full can retain context for reconstructing scope, while Skill may omit it; smaller Execution handoffs reduce input but increase calls.A useful handoff must support both selecting and applying appropriate knowledge, and its cost includes resulting target work.Source: §7 · p. 9

8 SCOPE AND OPEN QUESTIONS

• A focused test would compare unchanged C-Skill rules, a generic reminder to check B, and source-derived conditions, holding model and budget fixed.The generic reminder separates the value of applicability conditions from simply encouraging more caution.Source: §8 · p. 9
• On held-out targets, the proposed evaluation would measure fewer convention conflicts while retaining useful workflow behavior.Constructors would be restricted to A; generic-checklist and approvalmarker controls would distinguish shared norms, task-specific lessons, and source authority.Source: §8 · p. 9
• Current evidence covers complete repair packages, failed-but-repairable sources, and one execution per cell, so repeat-run reliability remains unresolved.The authors also identify state/text matching, actual database changes, successive repairs, and natural human feedback as further research needs.Source: §8 · p. 9

9 CONCLUSION

• A large correction gain may reflect a poor uncorrected memory, and an improved outcome may merely restore performance available without memory.ThinkingBox shows success beyond both references, whereas neither APEX regime establishes comparable aggregate benefits.Source: §9 · p. 9
• Evaluate each memory revision against both its predecessor and a fresh start, and preserve the conditions under which its advice was valid.The design question shifts from storing an accepted solution to deciding which parts should guide the next task.Source: §9 · p. 9
• Useful experience must help the next task choose what still applies, not merely reproduce what worked on the source task.Source acceptance alone does not establish when lessons should be reused.Source: §9 · p. 9

REPRODUCIBILITY STATEMENT

• The analysis uses 3,300 outcome-and-cost records, 1,100 canonical APEX-text reviews, and 20 separate file-state alignment reviews.Every benchmark–condition retains 100 targets.Source: REPRODUCIBILITY STATEMENT · p. 10
• Paired-analysis scripts, source identifiers, cohort construction, settings, prompts, and editable figures are included in the analysis materials.Source: REPRODUCIBILITY STATEMENT · p. 10
• Transition counts and cross-representation consistency are regenerated from the same frozen records.Source: REPRODUCIBILITY STATEMENT · p. 10

• Simulated-user feedback involves no human participants, benchmarks are evaluation-only, redistribution follows benchmark terms, and the statistical package excludes credentials and restricted attachments.Source: ETHICS AND DATA USE · p. 10

AI USE STATEMENT

• Generative AI supported experiments and manuscript work, while authors retained responsibility; memory construction was target-blind and used selected repaired sources.The ThinkingBox cohort contained 87 formal-screen and 13 pilot sources; both APEX regimes used the same first 100 eligible pairs in frozen manifest order. Feedback was simulated with privileged diagnostic access, not unaided human review.Source: AI USE STATEMENT · p. 10, A PROTOCOL, COHORTS, AND EXECUTION… · pp. 13–14

B COMPLETE RESULTS AND STATISTICAL COMPARISONS

• All six ThinkingBox H1 contrasts survived global adjustment; APEX-text Execution–Summary was +12 pp (14/2; pH = 0.0251, p24 = 0.0753), and no primary contrast survived group-level global-24 adjustment.The APEX-text comparison’s significance depends on the comparison scope; its family-adjusted p-value is below 0.05, but its global-adjusted p-value is not.Source: Table 7 · p. 15, §B.1 · pp. 15–16, §B.2 · p. 16

B.2 GROUP-LEVEL TEST AND RECOMPUTED INTERVAL SENSITIVITY

• A single run cannot distinguish task-specific effects from execution variation, and group-level sensitivity tests require exchangeability assumptions rather than providing causal randomization tests.Resampling quantifies task-composition uncertainty, not repeated-run reliability; grouping and multiplicity reduce precision, and target-level significance should not be conflated with group-level sensitivity.Source: §B.1 · p. 16, §B.2 · p. 16

B.3 EXPLORATORY REPRESENTATION COMPARISONS

• No within-branch result survived group-level Holm adjustment across 36 tests; three ThinkingBox C-versus-None comparisons survived target-level adjustment, and only ThinkingBox Full–None survived the group-level 18-test interaction family.These representation comparisons were exploratory and address different questions from correction effects.Source: §B.3 · p. 17, Table 9 · p. 17

C.1 COMPLETE B-ONLY/U/C OUTCOME DECOMPOSITION

• Full improved 46/77 ThinkingBox U failures, 8/58 APEX-state failures, and 10/55 APEX-text failures, while losing 2/23, 9/42, and 10/45 U successes, respectively.The APEX regimes recovered a smaller fraction of failures while losing a larger fraction of successes; rubric-item counts alone can overstate target-level utility.Source: §C.1 · p. 19

C.2 RELATIONSHIP AND DOMAIN HETEROGENEITY

• In 42 same-family APEX pairs, B-only solved twenty in each regime; text Full’s seven additional successes were not a correction gain because U and C both solved 21.Shared procedures showed gains for state Hybrid and text Skill, while text Hybrid lost one; the retrospective relationship/domain analyses are judgment-dependent and exploratory.Source: §C.2 · pp. 19–20

C.3 CONSISTENCY ACROSS REPRESENTATIONS AND HANDOFF REGIMES

• Across the same 100 APEX pairs, text-minus-state correction effects were +1/−2/−12 pp for Full/Skill/Hybrid, with every world interval spanning zero.Only two state Full gains overlapped with text Full gains; the comparison does not isolate state inheritance because construction and execution batches also differed.Source: Table 17 · p. 21, §C.3 · p. 21

C.4 COHORT-WIDE VALIDATOR PROFILES AND CLUSTER INFLUENCE

• ThinkingBox’s C–U advantage remained positive after omitting any one family, whereas both APEX regimes retained substantially smaller effects; removing validator flags did not establish causal mediation.Some flags also appeared in candidate conditions, so eliminating a recorded local error did not guarantee satisfying every task requirement.Source: §C.4 · p. 21
• Table 19 reports leave-one-cluster-out acceptance gaps across all three regimes, with each row removing one complete family or world.Its min/max values are influence ranges, not confidence intervals.Source: Table 19 · p. 22

D ACTION-LEVEL EVIDENCE FOR TRANSFER AND ITS LIMITS

• 37 of 54 ThinkingBox targets without C’s reference flag pass, while an APEX source-approved full-year convention overrides B’s explicit mid-year request despite both calculations being available.Another APEX case transfers a requirements–source-clause matrix while retaining A’s conclusion, so the traces do not separate procedural learning from answer anchoring; these cases do not establish a benchmark-wide causal explanation.Source: D ACTION-LEVEL EVIDENCE FOR TRANSFER… · pp. 22–23

D.1 CROSS-FAMILY EVIDENCE AND AMBIGUOUS SCORE DIFFERENCES

• Audits show that score differences can misidentify procedure transfer, source-answer anchoring, target-side evidence selection, or judge disagreement.These selected action chains distinguish possible mechanisms, but do not estimate their prevalence.Source: §D.1 · p. 24

E.1 RESOURCE THRESHOLDS AND INITIAL PAYLOADS

• Execution lowers cumulative input by 18.7%/12.1%/31.8% in ThinkingBox/APEX-state/APEX-text, while calls rise 4.7%/105.7%/16.9%.Cumulative input multiplies calls by average input per call; reductions in cached-context-inclusive input are not billing reductions.Source: §E.1 · pp. 24–25

F.1 FULL-DENOMINATOR COVERAGE AND ARTIFACT RECONSTRUCTION

• State and text judgments reconstruct different artifacts, apply original rubrics to native evidence, and exclude technical rendering, transport, or format errors as task failures.Valid judgments are retained rather than repeated because of a low score.Source: §F.1 · p. 25

F.2 PARTIAL-ARTIFACT ALIGNMENT AND EXECUTION-ERROR SENSITIVITY

• Excluding two state execution errors leaves 98 pairs: Execution–Summary is +5.10 pp in state and +12.24 pp in text, a +7.14 pp regime difference, not a randomized inheritance effect.Source: §F.2 · p. 26

E RQ2 AND RQ3: ACCEPTED EXECUTION AND RESOURCE TRADEOFFS

• Text Execution reaches 52 accepted targets versus Summary’s 40; audit-adjusted counts are 50/38, while rubric satisfaction improves 8.96 pp (adjusted p = 0.0568).The four audit corrections leave a twelve-point acceptance gap; the rubric result belongs to a separate contrast family.Source: §F.3 · p. 26

F.3 BACKEND AND DISAGREEMENT-AUDIT SENSITIVITY

• A 16-case APEX-text disagreement audit changes 14/2 discordances to 12/0 after four corrections, but selected, unblinded cases do not estimate judge accuracy.Among 98 pairs with primary reviews for both conditions, the Execution–Summary gap is +12.24 pp; this does not establish backend interchangeability.Source: §F.3 · p. 26

G EXTENDED LITERATURE COMPARISON

• Pairing the same episode before and after accepted repair separates an update’s incremental value from corrected memory’s value against independence.The evaluation uses paired agent executions with simulated feedback, not a human-subject evaluation of interface usability or oversight effort.Source: G EXTENDED LITERATURE COMPARISON · pp. 26–27