PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillRefine: Cross-Source Skill Induction and Execution Validation for LLM Agents in Refinery Planning Software

Dongxiao Liu, Boren Zhu, Yuwen Ding, Linghui Li, Lei Li, Xiaoyong Li

arXiv:2609.30674v1cs.CE

TL;DR

PIMS planning tasks require combining incomplete schema documentation, expert modification patterns, and constraints exposed during execution. SkillRefine uses those sources to build and refine task-oriented skills; it improves over SkillBase across four backbones, with gains from 0.052 to 0.174.

  • Problem

    PIMS documentation describes schemas but omits task-level coordination, while expert CASE records expose coordination patterns without explicit schema grounding.

    Source: Introduction · pp. 1–2

  • Method

    SkillRefine induces skills by grounding expert CASE coordination patterns in documentation, then refines them through compliance signals, structured mismatch diagnosis, and trajectory-guided localization.

    Source: Introduction · p. 2, Phase 4: Structured Multi-Signal… · pp. 4–5

  • Results

    SkillRefine improves over SkillBase by 0.076 on DeepSeek-V4-Pro, 0.052 on DeepSeek-V4-Flash, 0.106 on Gemma-4, and 0.174 on Qwen-3.6.

    Source: Discussion · p. 7

  • Takeaways & Limitations

    The separate gains from Baseline to SkillBase and from SkillBase to SkillRefine support cross-source induction and execution-driven refinement in PIMS.

    Source: Discussion · p. 7

    Evaluation is limited to AspenTech PIMS v22.0.11 and two demonstration models; construction and test tasks share model environments and schema vocabulary.

    Source: Discussion · p. 7

Abstract

from arXiv · show

Operating industrial planning software such as AspenTech PIMS (Process Industry Modeling System) requires an LLM agent to combine structural knowledge, procedural knowledge from expert records, and constraints revealed only during execution. These evidence sources are heterogeneous and individually incomplete: documentation describes tables and interfaces but omits task-level coordination, expert CASE records expose multi-table modification patterns without explicit schema grounding, and execution feedback reveals latent constraints only when a plan is executed. We present \textsc{SkillRefine}, a framework that exploits the Documentation--Practice Gap between documented structure and expert practice to extract candidate coordination patterns, ground them against table definitions and COM specifications, and compile them into progressively disclosed skill packages with provenance. The library is then refined through label-free compliance screening, oracle-based match decomposition over table, row, column, and value dimensions, and signal-conditioned trajectory attribution for localized repair. We evaluate \textsc{SkillRefine} on PIMS-Bench, a benchmark built from two AspenTech PIMS demonstration models, using disjoint construction and held-out test tasks. On the held-out test set, \textsc{SkillRefine} achieves absolute component match F1 gains of 14\%--30\% across four LLM backbones, with the largest gains on complex multi-table coordination tasks.

Introduction

PIMS tasks require translating brief business instructions into coordinated, schema-valid edits. SkillRefine combines expert practice, documentation, and execution evidence to build and refine reusable skills, then evaluates a shared frozen library across target backbones.

Introduction

  • A short FCC shutdown request may require coordinated CAPS, SLPR, BLNMIX, and purchase-limit edits, with exact rows, columns, and both CAPS bounds set.The required changes depend on the model and scenario.Source: Introduction · p. 1
  • PIMS documentation, expert CASE records, and COM specifications provide complementary but incomplete evidence about schemas, task-level coordination, and planning logic.Expert records suggest coordination patterns, documentation grounds them in valid schemas and interfaces, and execution can reveal unstated constraints.Source: Introduction · pp. 1–2
  • SkillRefine uses expert CASE records to propose coordination patterns and documentation to ground them in schemas, column semantics, value constraints, and COM interfaces.It also stabilizes the interface to distinguish software-invocation failures from business-planning knowledge failures.Source: Introduction · p. 2
  • Execution-driven refinement screens format and solver compliance, decomposes mismatches by table, row, column, and value, and uses trajectories to localize candidate repairs.Repairs use constrained add, modify, remove, or constrain operations rather than rewriting the library from a scalar score.Source: Introduction · p. 2
  • The evaluation covers AspenTech PIMS v22.0.11 and two demonstration models, not transfer to unseen refinery models, production deployments, or other industrial platforms.Construction and test tasks share model environments and schema vocabulary.Source: Introduction · p. 2
  • PIMS-Bench contains 140 tasks across five complexity levels and two demonstration models; the same frozen skill library is reused across four target backbones on held-out tasks.The library is constructed and refined once using DeepSeek-V4-Pro, without backbone-specific editing on held-out tasks.Source: Introduction · p. 2

Related Work

Related work spans skill acquisition, execution-based refinement, external-resource synthesis, and industrial software agents. SkillRefine is distinguished by extracting coordination patterns from successful expert records and grounding them in PIMS schemas.

Related Work

  • Skill-learning approaches include exploration-built programs (Wang et al. 2023), language-supervised motor skills, and skills for hierarchical reinforcement learning (Xia et al. 2026).Other work synthesizes skills from web evidence, skill networks, knowledge bases, and support tickets (Shen et al. 2026; Liang et al. 2026; Liu et al. 2026).Source: Agent Skill Acquisition and Refinement · p. 2
  • Execution-feedback methods include verbal self-improvement, trajectory-derived skills (Ni et al. 2026), validation-gated edits (Yang et al. 2026), and textual-gradient optimization (Wang et al. 2026).SkillRefine instead mines co-modification patterns from successful expert records and grounds them against documentation schemas.Source: Agent Skill Acquisition and Refinement · p. 2
  • Industrial software agents have supported solver interaction or model inspection, while SkillRefine learns procedures for translating planning intent into coordinated edits in an existing PIMS environment.The solver remains the authority for feasibility, and its feedback refines the procedural layer.Source: Implicit Procedures and Industrial… · p. 2
  • Skill libraries face long-context underuse and skill shadowing; progressive disclosure loads metadata first and full skill content on demand (Liu et al. 2023; Song and Wei 2026; Agent Skills 2026).PIMS-Bench evaluates structured table-, row-, column-, and value-level operations across five complexity levels.Source: Skill Organization and Execution-Based… · pp. 2–3

Method: SkillRefine

SkillRefine grounds expert-derived coordination hypotheses in PIMS documentation, stabilizes the execution interface, and refines skills using execution feedback. Its phases separate schema and interface knowledge from task-level rules, then localize repairs using multiple signals.

Phase 1: Grounding Layer from Documentation

  • Phase 1 parses PIMS documentation into structural schemas and COM interface specifications, providing grounding rather than business-operation rules.The extracted knowledge validates candidate patterns and grounds skill content throughout the pipeline.Source: Phase 1: Grounding Layer from… · p. 3

Phase 2: Procedural Pattern Extraction and Schema-Grounded Expansion

Phase 3: COM Interface Stabilization

  • Phase 3 stabilizes COM execution functions so interface failures are not confounded with skill failures, without adding business-planning rules.All 50 executable test cases passed; 11 additional cases were skipped because their methods are unsupported in PIMS v22.0.11.Source: Phase 3: COM Interface Stabilization · p. 3

Method: SkillRefine

  • The pipeline organizes its candidate skills into a three-level hierarchy and progressively discloses skill content on demand.The library contains 47 documents; the hierarchy spans metaskills, domain skills, and task skills.Source: Phase 4: Structured Multi-Signal… · pp. 3–4

Phase 4: Structured Multi-Signal Refinement and Targeted Skill Repair

  • Phase 4 screens compliance, decomposes structured mismatches, and uses execution trajectories to localize targeted skill repairs.Repairs are surgical, and held-out test tasks are never used during refinement.Source: Phase 4: Structured Multi-Signal… · pp. 4–5

Experimental Setup

The evaluation uses PIMS-Bench tasks from two AspenTech PIMS demonstration models, with construction tasks separated from a held-out test set. Four LLM backbones are compared across six knowledge and skill conditions using component match F1 as the primary metric.

Models and knowledge sources

  • The models are Weight Sample and Gulf Coast, and procedural patterns come from their CASE records alongside 3,126 PIMS documentation pages.Construction-task oracles supervise refinement.Source: Experimental Setup · p. 5

Benchmark

  • PIMS-Bench has 140 expert-reviewed tasks across five complexity levels, covering 31 tables and all four CASE keywords.The 70 tasks per model span single-value changes through system-wide coordinated modifications.Source: Experimental Setup · p. 5
  • The 40 construction tasks are disjoint from 100 held-out test tasks, and no test task informs skill construction or refinement..Source: Experimental Setup · p. 5

Conditions

  • Four LLM backbones are evaluated under six conditions, including SkillBase and SkillRefine with identical tools but unrefined versus refined skills.The conditions also include Baseline, DocOnly, Trace2Skill, and SkillOpt; all share the same PIMS execution interface.Source: Experimental Setup · p. 5

Metrics

  • Component match F1 across table, row, column, and value tuples is the primary metric, averaged across five runs and 100 held-out tasks.Table F1 is an internal refinement signal, while row and column recall support error analysis.Source: Experimental Setup · p. 5

Infrastructure

  • Each backbone-condition pair has 500 executions, using PIMS v22.0.11 and the FICO Xpress LP solver through COM automation.That is 100 tasks × 5 runs, with a 50-step limit per run.Source: Experimental Setup · p. 5

Results

SkillRefine combines structural descriptions, expert modification patterns, and execution feedback into a refined skill library. It improves held-out match F1 across four backbones, with larger advantages on harder tasks and contributions from both attribution branches.

Knowledge Extraction Results

  • Documentation parsing describes all 85 PIMS tables and provides 119 table-pair coreferences, but does not provide operational coordination guidance.Expert CASE records include 6 non-empty Weight Sample records and 317 Gulf Coast Case Sets records; the inferred patterns cover at most five unique tables per model.Source: Knowledge Extraction Results · pp. 5–6
  • COM interface stabilization wrapped 174 methods and tested 61 cases; all 50 executable cases passed, while 11 unsupported PIMS v22.0.11 cases were skipped.None of the 140 PIMS-Bench tasks depend on the skipped methods.Source: Knowledge Extraction Results · p. 6
  • After refinement, the skill library contains 57 documents (∼80 KB), including 10 new rules; the initial library had 47 documents (∼52 KB).Progressive disclosure initially loads only the ∼8.5 KB skill index, with full skill bodies loaded on demand.Source: Knowledge Extraction Results · p. 6

Skill Conditions and Performance

  • Raw documentation yields inconsistent match F1 changes, whereas SkillBase improves over Baseline by 0.074–0.122 and SkillRefine further improves SkillBase by 0.052–0.174.Trace2Skill and SkillOpt also improve over Baseline, but SkillRefine decomposes failures by compliance and structured match dimensions to localize repairs.Source: Skill Conditions and Performance · p. 6
  • SkillRefine achieves the highest mean match F1 under all four backbones, improving over Baseline by 0.189, 0.139, 0.180, and 0.296, respectively.The corresponding gains over SkillBase are 0.076, 0.052, 0.106, and 0.174.Source: Skill Conditions and Performance · p. 6, Table 2 · p. 6
  • The SkillRefine–Baseline gap grows from 10–15pp at L1–L2 to 18–28pp at L3–L5, with larger gains on complex coordination tasks.Average tables modified per task rise from 1.2 at L1 to 3.6 at L5, while hidden cross-table dependencies rise from 0.7 to 3.5.Source: Skill Conditions and Performance · pp. 6–7

Ablation Study

  • SkillRefine raises SkillBase match F1 from 0.590–0.705 to 0.757–0.800, with gains of 0.052–0.174 across backbones.Signal-conditioned attribution overlays tool-use context and consulted skills on diagnostic signals to identify repair candidates that outcomes alone cannot localize.Source: Ablation Study · p. 7
  • Removing the compliance-conditioned branch reduces match F1 by 0.011–0.041, while removing the match-conditioned branch reduces it by 0.016–0.077.The full model performs best on every backbone, consistent with complementary localization evidence from the two branches.Source: Ablation Study · p. 7, Figure 4 · p. 7

Discussion

SkillRefine composes expert records, documentation, and execution feedback by assigning each a distinct role, then refines skills using execution evidence. Results vary across backbones, and the study’s limitations bound what can be inferred beyond held-out tasks in the evaluated settings.

Discussion

  • CASE records propose task-level coordination patterns, documentation grounds them in schemas and interfaces, and execution feedback filters hypotheses and repairs incomplete rules.The contribution is role-specific evidence composition, not simply more knowledge.Source: Discussion · p. 7
  • SkillBase improves over Baseline, and SkillRefine further improves over SkillBase by correcting unsupported coordination, column mappings, and over-generalized procedures.These gains separately support cross-source induction and execution-driven refinement.Source: Discussion · p. 7
  • SkillRefine’s gains over SkillBase are 0.076 on DeepSeek-V4-Pro, 0.052 on DeepSeek-V4-Flash, 0.106 on Gemma-4, and 0.174 on Qwen-3.6.The passages state that further evidence is needed to explain this variation.Source: Discussion · p. 7
  • Evaluation uses one platform and two demonstration models; tasks share model environments and schema vocabulary, some tasks may have multiple valid solutions, and trajectory attribution does not prove causal responsibility.Thus, the evaluation concerns held-out task generalization rather than transfer to unseen refinery models.Source: Discussion · p. 7

Conclusion

The conclusion identifies a Documentation–Practice Gap in PIMS and presents SkillRefine as a way to combine expert coordination patterns, schema grounding, and execution-driven repair. On 100 held-out PIMS-Bench tasks, SkillRefine has the highest mean match F1 under all four evaluated backbones, while broader generalization remains unestablished.

Conclusion

  • SkillRefine obtains the highest mean match F1 under all four evaluated backbones on 100 held-out PIMS-Bench tasks.Task-oriented SkillBase packages consistently outperform unguided baselines, and SkillRefine further improves the library.Source: Conclusion · p. 7
  • PIMS documentation provides structural coverage without task-level coordination, while expert CASE records expose sparse operational patterns without schema grounding..Source: Conclusion · p. 7
  • SkillRefine uses expert records to propose coordination patterns, documentation to ground schemas and interfaces, and execution evidence to diagnose and repair skills.Its multi-signal refinement separates compliance screening from table-, row-, column-, and value-level diagnosis, then uses trajectories to narrow repairs to relevant skill documents.Source: Conclusion · p. 7
  • The results support execution-driven skill refinement for structured industrial software, but generalization beyond the evaluated PIMS environments remains to be established..Source: Conclusion · p. 7

Related papers

Loading 2609.30674v1…