SkillCome: Group Contrast Skill Optimization with Dual Memory

• Skill evolution requires both a reliable signal for updating the current solution and an optimizer state that organizes signals to guide update directions.Source: §1 · p. 2
• Single-trajectory analysis leaves unclear which actions caused failure or contributed to success, while local batches can provide noisy or incomplete evidence about recurring patterns.Source: §1 · p. 2
• Within-group comparisons of successful and failed trajectories reveal behavioral divergences, letting the optimizer derive signals from successful attempts rather than infer solutions solely from failures.Source: §1 · p. 2
• SkillCome combines grouped rollouts and contrast-based revisions with contrast memory for within-group patterns and failure memory for patterns shared across groups.Source: §1 · p. 2
• Across six benchmarks and five models, SkillCome delivers consistent improvements, with average gains up to 19.55% compared to no skill.Source: §1 · p. 3

• Prompt-based methods refine textual instructions, while agent skills add task-specific experience such as tool-use strategies and references (Wei et al., 2022b; Yao et al., 2023; Pryzant et al., 2023).Source: §2 · p. 3
• Skill-evolution methods iteratively analyze trajectories and revise skills; Trace2Skill consolidates trace lessons, while SkillOpt uses bounded edits, aggregation, and validation-based acceptance (Ni et al., 2026; Yang et al., 2026a).Source: §2 · p. 3
• SkillOpt-Lite and OEO delegate parts of evolution to an agent, while SkillCome focuses on constructing the optimization process from grouped rollouts (Shen et al., 2026b; Liu et al., 2026b).Source: §2 · p. 3

3.1 PROBLEM FORMULATION

• The target model executes tasks using a natural-language skill document and any accompanying scripts, while a frozen optimizer model analyzes trajectories and proposes revisions for validation.Across iterations, the candidate is accepted only when its validation score exceeds the previous skill’s, and the highest-scoring skill is retained for test evaluation.Source: §3.1 · pp. 3–4

3.2 GROUPED ROLLOUT

• For each question, the target model generates multiple trajectories under the same skill, grouping them by outcome so mixed groups can use successes as references for failures.Success and failure pools also supply evidence for conventional reflection.Source: §3.2 · pp. 4–5

3.3.1 DUAL MEMORY SYSTEM

• Separate failure and contrast memories store concise patterns with supporting training groups, allowing patterns to accumulate evidence across optimization steps without extra rollouts.Failure memory tracks patterns from conventional reflection; contrast memory stores differences between successful and failed trajectories within a group.Source: §3.3.1 · p. 5

3.3.2 CONVENTIONAL FAILURE AND SUCCESS REFLECTION

• Conventional reflection analyzes failure and success trajectory pools separately, with failure memory informing failure edits; mixed-group trajectories also contribute to this reflection.Failure reflection considers what went wrong and how the skill could prevent similar failures, while success reflection extracts useful practices to retain.Source: §3.3.2 · p. 5

3.3.3 GROUP CONTRAST REFLECTION

• Group contrast reflection compares successful and failed trajectories for the same question to identify divergences linked to opposite outcomes and actionable gaps in the skill.Successful actions serve as concrete references for identifying actions omitted or performed incorrectly in failures.Source: §3.3.3 · p. 6

3.3.4 EVIDENCE-GUIDED SKILL REVISION

• The optimizer merges contrast, failure, and success edits, resolves overlaps and conflicts, then prioritizes contrast and failure edits and the number of supporting groups when enforcing the edit budget.Applying selected edits produces a candidate skill; memory updates persist even when a candidate is rejected.Source: §3.3.4 · p. 6

4.1 EXPERIMENTAL SETUP

• The experiments cover six QA, mathematical-reasoning, and agentic benchmarks, five target models, and a default of eight rollouts per question.Benchmarks other than SearchQA are evaluated over four independent runs and reported as average accuracy.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

• SkillCome achieves the highest scores across 26 model–benchmark pairs and improves average accuracy by 5.69 points over SkillOpt.Across self-evolving and strong-guide-weak settings, average gains reach 27.23 points over No skill.Source: Table 1 · p. 7, §4.2 · p. 7
• On OfficeQA, DeepSeek-V4-Flash rises from 5.41% to 50.51%, surpassing the second-best method by 24.50%.The skill encodes instructions against DSML-style delimiters after memory retains the recurring format error.Source: §4.2 · p. 7

4.3 ABLATION STUDY

• Removing dual memory decreases SkillCome performance in every evaluated pair, while adding memory to SkillOpt helps in eight of eleven pairs but hurts in three.The authors attribute memory's stronger value with grouped rollouts to repeated evidence from fewer questions.Source: Table 2 · p. 8, §4.3 · p. 8

4.2 MAIN RESULTS

• In the ALFWorld case, contrast memory retains a rejected pattern as its support grows from two to fourteen tasks, while the skill is accepted and refined multiple times.The case illustrates skill evolution across optimization steps.Source: Figure 3 · p. 8

5.1 CASE STUDY

• ALFWorld group contrast identifies searching beyond visited locations and delivering a held target before searching again; the skill is accepted at step three and refined at step five.The resulting skill includes practical guidance absent from single-rollout baselines and leads to stronger test performance.Source: §5.1 · p. 8

5.2 SKILL TRANSFERABILITY

• Transferred skills outperform initial skills across five benchmarks when Qwen3.6 skills are applied to DeepSeek-V4-Flash, but remain below self-evolved skills, especially on OfficeQA.Cross-dataset transfers from SearchQA to HotpotQA and OlympiadBench to Omni-MATH beat No skill across four models on both targets.Source: Figure 4 · p. 9, §5.2 · p. 9

5.3 EVOLVING FROM EMPTY SKILL

• On DeepSeek-V4-Flash, evolution from an empty skill beats No skill on all five benchmarks, while evolution from an initial skill achieves higher final performance.Predefined guidance generally provides a better starting point, even when its immediate benefits are limited.Source: §5.3 · p. 9

5.4 EVALUATION IN AGENT HARNESSES

• For both evaluated models, Codex consistently outperforms Direct chat across all three datasets.The comparison applies learned skills in the Codex CLI harness versus chat-completion mode.Source: Figure 5 · p. 9, §5.4 · p. 9

• SkillCome combines group contrast analysis with dual memory to derive precise optimization signals and generalizable directions, and evaluations across models and benchmarks show improved agent performance.The paper frames the resulting skills as effective and reusable.Source: §6 · p. 10

• Generative AI assisted with proofreading, readability, and literature discovery; the authors independently located suggested papers and manually reviewed generated content.The authors take responsibility for the paper's final content, claims, and artifacts.Source: §7 · p. 10

8 REPRODUCIBILITY STATEMENT

• Dataset descriptions and processing steps are provided in Appendix B.1, with the datasets described in §4.1.Source: §8 · p. 10
• Training and evaluation settings and hyperparameters are described in §4.1 and detailed in Appendix B.2.Source: §8 · p. 10
• The supplementary materials provide code to reproduce the algorithm.Source: §8 · p. 10

A METHOD DETAILS

• Group contrast reflection jointly analyzes successful and failed rollouts of the same task, locating their earliest meaningful divergence and proposing only generalizable, minimal edits.The prompt also directs the analyst to distinguish skill-relevant errors from evaluator noise, tool nondeterminism, and infrastructure behavior, and allows no edit when no reusable lesson is supported.Source: A METHOD DETAILS · p. 16
• The prompt compares a proposed pattern with a cross-step hypothesis bank, linking an existing ID only for the same causal mechanism; cross-task support ranks edits but is not required for eligibility.Source: A METHOD DETAILS · pp. 16–17
• The structured reflection output records the contrast, a reusable pattern, and proposed skill edits, while checking whether the current skill already gives adequate guidance.Source: A METHOD DETAILS · p. 16
• In the ALFWorld example, success involved searching beyond the destination area and delivering one keychain before retrieving the second, while failures revisited nearby empty locations or delayed delivery.The proposed extension is to broaden exploration when nearby receptacles are empty and deliver a held target before retrieving another when inventory is limited.Source: A METHOD DETAILS · p. 17

B.1 DATASET DETAILS

• The six main benchmarks cover document and snippet-based question answering, research-level mathematics, spreadsheet and document work, and household decision-making.They are DocVQA, SearchQA, LiveMath, SpreadsheetBench, OfficeQA, and ALFWorld.Source: §B.1 · p. 18
• Transfer experiments evaluate SearchQA skills on HotpotQA and OlympiadBench skills on Omni-MATH, using target datasets without training or validation splits.HotpotQA uses 1,000 randomly selected examples from its distractor validation split; Omni-MATH uses 1,000 randomly selected samples.Source: §B.1 · pp. 18–19

B.2 IMPLEMENTATION DETAILS

• SkillOpt and SkillCome run for four epochs or ten batches, while SkillOpt-Lite runs for ten batches; Trace2Skill uses its official one-epoch implementation.Source: §B.2 · p. 19
• Grouped rollouts sample eight trajectories per question, and benchmarks with small test sets are evaluated over four independent runs with average scores reported.SearchQA is evaluated once; repeated trajectories for the same question are deduplicated for success and failure reflection.Source: §B.2 · p. 19

C.1 NOTABLE FINDINGS IN THE EXPERIMENT

• Flash rises from 10.33% to 96.74% on Meta but only from 54.58% to 57.08% on Non-Meta; GLM-5.3 reaches 71.74% on Meta and 65.00% on Non-Meta.Flash has the higher overall score, while GLM-5.3 remains stronger on questions without the meta-option; the shortcut benefit should not be read as equivalent mathematical reasoning improvement.Source: Table 4 · p. 19, §C.1 · p. 20
• The meta-option, labeled correct whenever present, occurs in 46 of 106 LiveMath test samples and permits answering without solving the underlying math.Flash and Qwen3.6-35B-A3B skills stop reasoning after detecting it, whereas the stronger models retain instructions to analyze the math.Source: §C.1 · p. 20, Figure 7 · p. 20

C.2 STANDARD DEVIATION OF MAIN RESULTS

• SkillCome’s scores have relatively small standard deviations across four evaluation runs, indicating stable performance; SearchQA is excluded because it is evaluated once.Source: Table 5 · p. 21, §C.2 · p. 21

C.3 STATISTICS OF THE MIX GROUP

• Across five target models, 731 of 2,250 groups contain both successful and failed trajectories, supporting group contrast reflection across models.Source: Table 6 · p. 21, §C.3 · p. 21

C.4 ABLATION ON THE COMPONENTS OF SKILLCOME

• 21.79 points: “SkillOpt w/ Memory” improves DeepSeek-V4-Flash on OfficeQA over SkillOpt, though memory’s gains are not consistent across reported pairs.The passage attributes the OfficeQA improvement to learning explicit guidance for the DSML-style answer format.Source: §C.4 · p. 21
• Combining group contrast and dual memory yields the best results on all model–benchmark pairs, with the ablations indicating synergy between the components.Group contrast enables deeper per-question analysis, while dual memory connects findings with evidence from a broader range of samples.Source: §C.4 · p. 21

C.5 EVALUATION IN AGENT HARNESSES

• Under Codex, both evaluated models outperform direct chat across datasets, with particularly pronounced gains on SpreadsheetBench and Livemath.The authors present these results as evidence of SkillCome’s effectiveness in harness environments and practical value in real-world workflows.Source: §C.5 · p. 22, Table 7 · p. 22

C.6 CROSS-MODEL TRANSFERABILITY

• Skills transferred from DeepSeek-V4-Flash to Qwen3.6-35B-A3B outperform skills evolved specifically for the target model on LiveMath and OfficeQA, while reverse transfer on OfficeQA is substantially lower.The authors attribute the reverse-transfer result to the transferred skill not explicitly addressing Flash’s DSML-style answer format.Source: §C.6 · p. 22, Table 8 · p. 22

C.7 CROSS-DATASET TRANSFERABILITY

• 0.2 to 4.9 points: transferred skills improve performance across all eight model–benchmark pairs on HotpotQA and Omni-MATH.The authors describe the results as supporting cross-dataset generalizability within similar domains.Source: §C.7 · pp. 22–23, Table 9 · p. 23

C.8 EVOLVING FROM EMPTY SKILL

• Evolving from an empty skill improves over “No skill” in 10 of 11 model–benchmark pairs, but evolution from an initial skill achieves higher final accuracy in 10 of 11.The exception is ALFWorld on Qwen3.6-35B-A3B, where empty-skill evolution has a slight advantage; on Spreadsheet, the initial skill lowers initial performance but leads to a better final result.Source: §C.8 · p. 23, Table 10 · p. 23

C.9 TOKEN USAGE OF OPTIMIZER MODEL

• SkillCome consumes fewer optimizer tokens than SkillOpt across evaluated datasets for both target models while achieving performance gains.The reduction primarily comes from lower input-token usage through trajectory deduplication under grouped rollouts.Source: §C.9 · pp. 23–24, Table 11 · p. 24