PaperlayerView exact revision on arXiv

Source-linked AI summary

MaCoPlanner: LLM-Assisted Manual-Compiled Task Planning with Proactive Safety Verification for Robotic Industrial Panel Operation

Guipeng Xin, Jiahe Xua, Mohammad Deghat, Chenhui Wan, Jie Liu, Youmin Hu, Zhongxu Hu

arXiv:2608.28300v1cs.ROcs.AI

TL;DR

Industrial panel robots must satisfy procedures, safety rules, and device-state constraints distributed across heterogeneous manuals. MaCoPlanner compiles those manuals into typed planning knowledge and verifies candidate plans symbolically before actuation, repairing violations or rejecting unresolved plans. Across the evaluated tasks, it reports a 2.7% final violation rate and demonstrates integrated execution feasibility in a no-load controller-panel simulator.

  • Problem

    Industrial panel operation requires compliance with procedural, safety, and device-state constraints distributed across heterogeneous manuals.

  • Method

    MaCoPlanner compiles equipment manuals into a typed intermediate representation, retrieves relevant evidence, and proactively verifies, repairs, or rejects symbolic plans before execution.

  • Results

    2.7% final violation rate was achieved across 480 tasks from four panel domains under an independent safety oracle.

  • Takeaways & Limitations

    Verified symbolic plans can be grounded and executed under representative interaction conditions in a no-load controller-panel simulator.

  • Takeaways & Limitations

    26.3% of 357 runs entering the planning loop were rejected when no candidate satisfied both LTL and Safety-FSM conditions within the refinement budget.

Abstract

from arXiv · show

Robotic industrial panel operation requires not only accurate control localization but also compliance with operating procedures, safety rules, and device-state constraints distributed across heterogeneous manuals. This study presents MaCoPlanner, a task-planning framework built on knowledge compiled from equipment manuals that converts equipment manuals into a typed intermediate representation, retrieves task- and state-relevant evidence, and uses it to support plan generation. Before actuation, candidate plans are symbolically rolled out and checked against procedural and state-transition constraints; detected violations are localized and returned for targeted repair, while unresolved plans are rejected. A separate execution interface grounds verified symbolic actions to physical controls and updates the device state. Under an independent evaluation oracle, MaCoPlanner achieves a final violation rate of 2.7%, and 26.3% of the runs in the repair analysis are rejected after exhausting the refinement budget. Compared with Raw-Manual, task success increases from 62.8% to 84.4% on Level-2 tasks and from 25.9% to 43.2% on Level-3 tasks. Experiments on a controller-panel simulator without an attached industrial load further demonstrate integrated execution feasibility under representative interaction conditions, without claiming industrial deployment readiness.

1. Introduction

Industrial panel automation must handle device-specific procedures, state conditions, and safety constraints while adapting beyond fixed expert-designed procedures. MaCoPlanner addresses this by compiling heterogeneous manuals into structured planning knowledge and verifying candidate plans before execution.

  • Industrial panels directly change equipment states, making reliable operation relevant to process stability and safety.
  • Current automation relies heavily on expert programming and manually designed procedures, limiting adaptability for variation, small-batch production, changeover, and abnormal events.
  • Foundation-model robotic systems enable instruction-driven planning, but industrial panels depend more strongly on device-specific operating logic than many household manipulation tasks.
  • MaCoPlanner compiles heterogeneous manuals into a typed intermediate representation containing procedures, safety constraints, state conditions, parameters, and control relations.Source back-links support evidence retrieval, plan generation, symbolic rollout, verification, and repair.
  • Before execution, candidate plans are checked against procedural and state-dependent constraints, repaired when violations are found, or rejected when refinement cannot produce a valid sequence.The framework is evaluated through a chain spanning manual compilation, retrieval, planning, safety verification, and robotic execution.

2. Related Works

Related work highlights a gap between general robotic manipulation and industrial panel operation, where device-specific logic and task-level safety constraints are central. MaCoPlanner positions manual-derived knowledge and pre-execution verification as responses to that gap.

  • Perception and manipulation methods often remain constrained by task-specific data, annotation quality, scene coverage, or object-grasping objectives.
  • Classical symbolic planning requires a complete domain, while constructing such domains for new equipment typically demands substantial manual modeling.
  • Task-level safety has received less attention than motion- and trajectory-level safety in existing robotic safety research.
  • Open-world, natural-language-driven robotic systems weaken closed-world assumptions that previously restricted execution to predefined safe boundaries.
  • Recent approaches move safety enforcement upstream through formal constraints, but this work verifies and repairs candidate plans before execution rather than relying on runtime monitoring.
  • The addressed gap is using device manuals for task-level planning and pre-execution procedural and state verification in industrial panel operation.

3. MaCoPlanner: Task Planning from Compiled Equipment Manuals with Proactive Safety Verification

MaCoPlanner compiles heterogeneous equipment manuals into a typed, traceable intermediate representation, retrieves state- and task-relevant evidence, and generates candidate plans. It then symbolically verifies and repairs plans against procedural and state-dependent constraints before execution, rejecting unresolved plans.

  • 3.3. Proactive Safety Verification Mechanism for Task Planning: MaCoPlanner generates an evidence-conditioned initial plan and symbolically rolls it out using compiled transitions and a device safety finite-state machine.The symbolic state abstracts device memory into slots such as power state, operation mode, alarm flags, and key setpoints; this rollout is not the physical device dynamics.
  • 3.2. Compiled Manual Knowledge for Task Planning: MaCoPlanner compiles heterogeneous manuals into a typed intermediate representation containing procedures, safety clauses, parameter specifications, and structural relations.Source back-links connect compiled items to their manual and text span for retrieval, planning, verification, and repair.
  • 3.2. Compiled Manual Knowledge for Task Planning: The representation makes procedural steps executable by encoding prerequisites, expected outcomes, exception branches, and retry policies.Each procedure is decomposed into step nodes with action primitives, state predicates, observation-based acceptance criteria, exception guards, and retry constraints.
  • 3.2.2. Query-Decomposition-Driven Retrieval and Evidence Gating: Typed retrieval decomposes the user instruction and device memory into typed demands, routes them to corresponding indices, and gates out semantically related but inapplicable evidence.The indices separately organize process structures, safety clauses, parameters, and control relations.
  • 3.3. Proactive Safety Verification Mechanism for Task Planning: Procedural and state-dependent checks produce localized violation feedback for targeted replanning before execution, while plans that exceed the refinement budget are rejected.Verification treats procedural order and mode/interlock legality as complementary constraints and remains conditional on specification fidelity and symbolic-state correctness.

4. Manual-Guided Grounding and Device Memory

MaCoPlanner grounds verified symbolic controls to physical panel instances using multimodal perception, geometric reasoning, and manual-derived spatial relations. It estimates operation-critical parameters and maintains device memory to support state-consistent execution.

  • The execution interface maps symbolic controls to physical panel instances and updates device state from observed outcomes.Grounding combines semantic and spatial-relation consistency and produces operation-ready parameters for execution and verification.
  • Multimodal grounding uses segmentation, open-vocabulary detection, OCR labels, and depth reconstruction to estimate control poses in the robot base frame.SAM2 provides masks, DINO-X detects controls, OCR supplies semantic anchors, and depth maps observations into robot coordinates.
  • Manual-derived control aliases and layout priors provide spatial-relation constraints for ranking candidate control assignments.Detection-class consistency and OCR/control-alias overlap form candidates; geometric and relation consistency then rank them.
  • A deterministic matching rule retains candidates with S_match ≤0.65 and invokes a VLM only for ties or semantic ambiguity.The deployed relation weight is λ = 0.5, keeping relation consistency secondary to pose, class/OCR, and geometric evidence.
  • Geometric processing extracts control-specific execution parameters, including knob center, radius, height, button pressing stroke, and triggering threshold.Stereo depth is denoised, aligned to a fitted panel plane, clustered, and shape-filtered before parameter estimation.
  • Device memory stores modes, parameters, control states, and verification history, updating after actions to condition retrieval and procedural/state checks.The memory update reduces state drift but does not eliminate it.

5. Experiment

MaCoPlanner is evaluated across manual compilation, retrieval, task planning, safety verification, and physical execution using complementary benchmarks and a controller-panel simulator. Results indicate that typed, state-aware evidence retrieval and verification components matter most as task complexity increases.

  • Evaluation setup: The evaluation spans compilation fidelity, retrieval quality, task planning, independent safety assessment, repair behavior, and physical execution.The physical platform uses a controller-panel simulator without an attached industrial load, isolating panel interaction and state-dependent execution from load-driven process dynamics.
  • Evaluation setup: The retrieval benchmark contains 400 instruction-and-device-state tasks with independently annotated task-required and state-applicable evidence.All retrieval methods use the same candidate pool and K ∈{8, 12, 16}.
  • Evaluation setup: The planning benchmark contains 480 audited panel-operation tasks across 80 Level-1, 200 Level-2, and 200 Level-3 tasks, each paired with an expert reference sequence.A separate external safety oracle is independently constructed from the same manual sources and withheld from the planner.
  • Compilation fidelity: Compilation completeness is computed as compiled items divided by gold units, while semantic consistency measures preservation of source meaning and required fields across four IR types.Safety clauses have complete count-derived coverage; unmatched units concentrate in process skeletons and structural relations.
  • Retrieval quality: Typed indexing produces larger and more consistent retrieval gains than query decomposition, while evidence gating mainly improves ranking of applicable, task-critical evidence.Restoring gating changes recall and F1 only slightly but improves nDCG@8 and nDCG@12 more clearly.
  • Task planning and safety: Retrieval improvements transfer to downstream planning more strongly on Level-2 and Level-3 tasks, where Full MaCoPlanner improves success and reduces violation rate relative to the full retrieval ablation.Level-1 performance varies little, whereas removing LTL, the Safety FSM, targeted repair, or device memory causes larger degradation on complex levels.

5.6. End-to-End Task Planning and Task-Level Safety Evaluation

MaCoPlanner improves complex task planning through compiled evidence, verification, and targeted repair, while rejecting unresolved plans before execution. Evaluation also shows remaining difficulty in dense state dependencies and physical interaction.

  • End-to-End Comparison: MaCoPlanner achieves the strongest results among comparable natural-language planning methods on Levels 2 and 3.Level-1 shows little separation, and MaCoPlanner does not obtain the highest result there.
  • Backbone Sensitivity: Lower-capacity planners retain much of Level-1 performance but lose more on Levels 2 and 3 and show higher violation rates.The comparison supports robustness across the evaluated backbones without implying backbone independence beyond this subset.
  • Safety and Repair: Localized LTL/FSM feedback supports targeted correction, yielding stronger final success and lower violation rates with fewer refinement steps than two explicit iterative baselines.SafePlan has no comparable refinement count because it lacks an iterative plan-repair loop.
  • Safety and Repair: 26.3% of 357 planning-loop runs are rejected when no candidate satisfies both LTL and Safety-FSM conditions within the refinement budget.MaCoPlanner uses fail-closed rejection and issues no physical action for unresolved plans.
  • Task Complexity: Complex tasks differ mainly through state confirmation and constraint coupling rather than sequence length alone, with Level-3 state-verification actions comprising 42% of actions.The diagnostic bypasses natural-language interpretation, retrieval, and symbolic-problem construction, so it is not an end-to-end baseline.
  • Failure Modes: Across 12 audited failed episodes, retrieval miss is the largest individual category, while verification and grounding or execution failures also remain relevant.Grounding errors can associate valid symbolic actions with wrong controls, and physical errors can prevent state transitions.

6. Conclusions

The conclusion presents MaCoPlanner as a manual-knowledge, evidence-conditioned planning framework with proactive safety verification. Results support complex-task improvement and integrated execution feasibility under evaluated conditions, while identifying unresolved interface and planning limitations.

  • MaCoPlanner combines typed knowledge compiled from equipment manuals with evidence-conditioned planning and proactive safety verification.
  • Across 480 tasks from four panel domains, MaCoPlanner maintains a 2.7% final violation rate under an independent safety oracle.Plans unresolved after repair are rejected before execution.
  • No-load controller-panel simulator experiments demonstrate that verified symbolic plans can be grounded and executed under representative interaction conditions.The experiments do not establish industrial deployment readiness.
  • Remaining limitations include incomplete retrieval, symbolic-state drift, difficult Level-3 planning, and grounding or contact errors.

Appendix A. Complete Prompts, Structured Outputs, and Post-Processing

Appendix A documents the prompts and structured output formats used by MaCoPlanner’s language-driven modules. It organizes these prompt-based stages in pipeline order.

  • The appendix specifies prompts and structured outputs for manual compilation, query decomposition, task planning, violation-guided repair, memory update, and post-processing.
  • Table A.6 consolidates the prompt-based stages in pipeline order.Compilation and rule translation correspond to Section 3.2.1, query decomposition to Section 3.2.2, and planning and repair to Section 3.3.1.
  • The documented formats cover both knowledge transformation and downstream planning operations.

Appendix B. Illustrative Examples of Safety Verification and Benchmark Construction

Appendix B illustrates proactive safety verification and benchmark construction through LTL rules, Safety-FSM checks, and IR-backed examples. Generated primitive sequences are converted into symbolic events and validated against procedural and state-transition constraints.

  • Safety Verification: The appendix illustrates proactive safety verification through representative LTL rules and a lightweight Safety FSM.The two layers instantiate procedural constraint checking and state-dependent legality checking.
  • Safety Verification: Generated primitive sequences are mapped to symbolic events before LTL and Safety-FSM validation.Examples include pressed(START_BTN) and pressed(PANEL_OFF).
  • Benchmark Construction: The appendix includes an IR-backed running example showing how benchmark instances and reference targets are derived from compiled knowledge.
  • Safety Verification: Each action proposition is evaluated with the symbolic state immediately preceding that action.
  • State Semantics: The proposition done(PREHEAT) remains true after successful preheating.

Appendix C. Independent Benchmark and Safety-Oracle Protocol

The appendix describes an independent audit reference for retrieval, planning, and task-level safety evaluation, constructed from source materials rather than MaCoPlanner’s internal representations.

  • The audit reference combines original manuals, task instructions, and the initial device state.
  • The reference is independent because it is not copied from MaCoPlanner’s compiled IR or internal verifier.

Appendix D. Reproducibility and Implementation Details

The appendix fixes implementation settings across manual compilation, retrieval, planning, verification, grounding, and language-driven modules to support reproducibility.

  • 512-token chunks with 64-token overlap retain boundary-crossing clauses while limiting corpus duplication.The chunk length follows the deployed e5-base-v2 encoder’s maximum input length.
  • α = 0.70 prioritizes semantic relevance while retaining a 30% state-applicability contribution.The shared τ = 0.55 cutoff is applied after an independent hard-conflict veto, with both values fixed before evaluation.
  • Kmax = 3 allows at most three verifier-guided repair calls after the initial candidate.The fixed budget limits latency and enforces fail-closed rejection when no candidate passes both checkers.
  • λ = 0.5 makes relation consistency a secondary correction to the pose term during grounding.A 0.65 matching-cost threshold retains plausible candidates after class/OCR and geometric filtering, with settings unchanged across five viewpoints.
  • DBSCAN and RANSAC specify clustering and robust-fitting procedures for grounding control candidates.The appendix also records a 15-mm DBSCAN radius, ten-point minimum, and 5-mm RANSAC distance.
  • Tables A.6 and A.7 document prompt examples, language-module instructions, output validation, and deterministic post-processing rules.

xix

This appendix section lists tables covering protocol, implementation, trial records, benchmark composition, provenance, baseline configuration, and rule audits.

  • Table C.12 covers independent annotation, blinding, agreement, and adjudication procedures.
  • Table D.13 presents consolidated implementation and hyperparameter settings.
  • Table D.14 records per-trial physical, runtime, and per-task planning data.
  • Table D.15 reports retrieval and planning benchmark composition alongside independent annotation statistics.
  • Table D.16 gives source-manual and compiled-IR provenance statistics across four benchmark domains.It distinguishes these counts from the independently annotated audit subset and notes shared 512-token chunking with 64-token overlap.
  • Tables D.17 and D.18 cover baseline information access, safety and refinement configuration, and full LTL/FSM rule auditing.
Loading 2608.28300v1…