PaperlayerView exact revision on arXiv

Source-linked AI summary

Human-AI Collaboration for Multi-Line Task Adjustment Using Local Large Language Models and a Digital Twin

Teng-Hsien Ko, Chin-Te Lin

arXiv:2609.29061v1cs.SE

TL;DR

Changing tasks and operating conditions make it important to preserve constraints and provide evidence for human review. The paper links a local model, a digital twin, and operators in a traceable Propose-Verify-Decide workflow. In the fixed test set, staged screening and confirmation supported review of eligible cases, but failures and limited scope leave overall reliability unestablished.

  • Problem

    The study asks how operator-requested task changes can be made inspectable and rejectable while preserving traceability across requirement conversion, simulation, and review.

    Source: §1 · p. 2

  • Method

    The system links operator intent and task records to bounded candidate strategies, checks candidates against constraints, and retains verification evidence and decisions.

    Source: §1 · p. 2

  • Results

    Of 28 workflow cases, 18 met expectations; autonomous strategy-workflow success was 3/10 and correct rejection of invalid inputs was 7/8, while all four complete-evidence cases reaching CP6 passed review.

    Source: §5.4 · p. 17, §7 · p. 20

  • Takeaways & Limitations

    Within the evaluated cases, staged verification supported screening throughput-ineligible strategies and confirming results submitted for final engineering review.

    Source: §5.4 · p. 17

    The test set does not establish overall reliability or long-term stability, and semantic distortions, incomplete evidence, and missed invalid inputs remain.

    Source: §6.1 · p. 18

Abstract

from arXiv · show

Automation systems must adapt to changing tasks, equipment states, and staffing conditions while providing evidence for human review. This study presents a multi-line task-adjustment system integrating a local large language model, a digital twin, and human decision-making. A Propose-Verify-Decide workflow translates operator intent into structured requirements, generates a bounded set of candidate strategies, and checks semantics, simulation execution, and operational constraints. Linked records preserve traceability from requests to verification evidence and decisions. Thirty fixed test records were evaluated using four virtual surgical-instrument sorting lines: 28 assessed the workflow and two assessed model generation. Eighteen workflow cases met expectations; autonomous strategy-workflow success was 3/10, and correct rejection of invalid inputs was 7/8. All four cases that passed preceding checks, produced complete evidence, and reached final engineering review (CP6) passed that review. Together with the correct blocking of strategies that failed throughput constraints, this supports the effectiveness of staged screening and confirmation within the tested setting. Mean placement-validation pass rate across eight simulation evidence records was 97.50%. Mean times to the first reviewable response and simulation verification, excluding startup, were 12.94 and 164.39 s, respectively. Remaining failures involved semantic distortion, incomplete evidence, and missed invalid inputs. The results demonstrate a traceable strategy-review workflow, but do not establish overall reliability or long-term stability. Broader testing and physical evaluation are needed to assess generalizability.

1 Introduction

Changing production conditions require task adjustments that preserve constraints and account for effects across lines, while people still interpret requirements and consequences. The study proposes a traceable workflow combining a local language model, a digital twin, and operator decisions.

1 Introduction

  • Changing product mix, urgent requests, equipment states, or staff availability can require line adjustments while preserving constraints and accounting for effects on other lines.Automation systems still require people to interpret the situation, translate requirements into settings, check constraints, and assess consequences.Source: §1 · pp. 1–2
  • A coherent or structurally valid language-model output may still be semantically wrong or infeasible under equipment, placement, or throughput constraints.The review workflow must bring requirement semantics, scenario execution, and operational constraints together to judge applicability.Source: §1 · p. 2
  • The research question is how to make operator-requested changes inspectable and rejectable while preserving traceability, using a local model and existing simulation environment.The study focuses on practical adjustment tools and decision evidence when engineering or operational personnel are limited.Source: §1 · p. 2
  • Propose-Verify-Decide organizes intent into bounded candidate strategies, tests scenarios and mandatory constraints in a digital twin, then presents evidence for operator judgment.The LLM assists with information processing and strategy formulation; the twin supplies execution evidence, while people retain contextual judgment and adoption responsibility.Source: §1 · p. 2
  • The system links intent, requirement versions, state records, strategies, scenario specifications, and execution evidence to trace strategy provenance and verification outcomes.It filters bounded candidates by mandatory constraints, ranks eligible candidates by throughput-target attainment, and retains revision and cancellation branches.Source: §1 · p. 2
  • A fixed test suite covers ordinary requests, invalid inputs, and model generation while distinguishing semantic correctness, structural validity, simulation execution, constraint satisfaction, and outcome interpretation.The analysis also measures time required at different workflow stages.Source: §1 · p. 2

2 Related Work

Prior work addresses language-to-task conversion, digital-twin evaluation, and human-robot task allocation, but the study focuses on preserving operator intent and tracing verification across multi-line task adjustment. It connects these strands through a workflow linking requirement conversion, simulation, and result review.

2 Related Work

  • Manufacturing language-model studies translate instructions into task steps, process descriptions, or coordination messages, while tool and specification checks address executable capability and conformity.The reviewed approaches include retrieval-augmented generation, reasoning with actions, tool use, robot-skill affordances, and program verification.Source: §2 · pp. 2–3
  • Digital twins have been used to evaluate line configurations, reconfiguration strategies, and robot task replanning, but applying twin evidence to physical operations requires fidelity consideration and further confirmation.The cited work also describes twins as sources of event observations and quantitative evidence.Source: §2 · p. 3
  • LLM-digital-twin integrations address modeling, verification, or optimization; this study instead centers traceable task changes across robot-operated lines and comparable candidate-simulation evidence.The related approaches include FactoryFlow, MAKA, and the GAMHE 5.0 pilot line.Source: §2 · p. 3
  • Human-robot allocation research examines assignment by task and capability, alongside how people understand intelligent-system behavior.Source: §2 · p. 3
  • The integration problem is preserving intent from requirement conversion through strategy generation, simulation, and review while tracing rejection or revision to a processing stage.The stated scope is workflow integration and verification; automatic twin construction, globally optimal scheduling, and formal safety guarantees are outside it.Source: §2 · p. 4

3 Method

The method divides task adjustment among language-model interpretation, digital-twin execution, and operator adoption, with linked records preserving traceability. It filters and ranks bounded candidate strategies using evidence and constraints, then presents results for human review.

3.1 Governance Concept

  • The governance workflow parses intent, versions requirements, aligns them with production states, verifies candidates in the twin, and lets operators accept, revise, or cancel proposals.The linked records preserve traceability from the original request through strategy adoption; the three judgment roles belong to the LLM, twin, and operator.Source: §3.1 · pp. 4–5

3.2 System Architecture and Data Flow

  • Operator intent and structured production-line or twin state enter separate workflows and are linked in the Core Repository to support candidate generation.Identifiers link requirement versions, candidates, scenario specifications, and execution evidence so strategy origins and verification results can be traced.Source: §3.2 · pp. 5, 7
  • Task-change requests require an operator identifier, reason, target scope, and requested modification; missing or invalid fields should trigger clarification or rejection.A validated Intent Patch links to the task-requirement version.Source: §3.2 · pp. 5, 7
  • The system converts eligible candidates into twin scenarios, extracts execution and placement results plus verification reports, and compiles evidence into a decision summary.Preliminary scene and rule checks precede simulation; the summary generator uses the extracted RunArtifacts for operator review.Source: §3.2 · p. 7

3.3 Candidate Generation, Verification, and Decision-Making

  • Candidate batches vary permitted fields while preserving specified scope, target instruments, and KPI constraints; eligible strategies are ranked by throughput-target attainment, not global optimality.If none qualifies, the system reports reasons and requests revision or cancellation.Source: §3.3 · p. 7

3.4 CSSD Digital-Twin Verification Scenario

  • The virtual CSSD twin models instrument sorting and set assembly, where strategy assessment includes inspectable change scope, failure reasons, and waiting costs as well as throughput.Source: §3.4 · p. 7

3.5 Evidence Interpretation and Human Roles

  • Decision summaries report evidence completeness and mandatory constraints before outcome measures, and even eligible strategies require operator judgment under local conditions.Final engineering review checks complete evidence at CP6; retrospective researchers assess expected behavior against the original request, outcome, and stopping point.Source: §3.5 · pp. 7–8

4 Experimental Design

The section specifies the virtual implementation, checkpointed test design, and workflow and simulation metrics used to assess task adjustment.

4.1 Implementation Environment

  • Testing used a local LLM and n8n/Python/SQLite/JSON stack, with four Isaac Sim lines representing surgical-instrument sorting.Evaluation focused on text requests and parameterized interventions, not real operators’ speech-recognition or physical-intervention performance.Source: §4.1 · p. 8, Table 2 · p. 8

4.2.1 Checkpoints

  • Observable checkpoints assess requests from input and intent through simulation evidence and final engineering review, with routes varying by request type.A recorded failure does not necessarily mean execution was blocked; interception depends on the actual response and stopping point.Source: §4.2.1 · pp. 8–9, Table 3 · p. 9

4.2.2 Test Cases

  • Researchers submitted requests as operators and retrospectively judged outcomes; TC1–TC5 assess workflow, while TC6–TC7 assess model generation and are excluded from workflow case-conformance rates.Model-generation outcomes are reported separately from the workflow case-review outcomes.Source: §4.2.2 · p. 9, Table 4 · p. 10

4.3.1 Workflow Metrics

  • Workflow measures include checkpoint and case-conformance rates, strategy and batch selection, query and rejection outcomes, plus response, simulation, and end-to-end times.Rates use only cases evaluated at the relevant checkpoint; first-response time ends at the first response sufficient to determine the next action, while simulation time excludes startup.Source: §4.3.1 · pp. 9, 12

4.3.2 Simulation Metrics

  • Simulation metrics assess candidate execution, placement, and time costs; a placement record passes when its XY-plane error is within 0.06 m.The reported mean is the arithmetic mean of pass rates across simulation evidence records, not a pooled instrument-level rate; unchecked records are excluded. Configured intervention delays test timing effects, not real entanglement recognition.Source: §4.3.2 · p. 13

5 Results

Across fixed workflow tests, 18 of 28 cases met expectations, while autonomous strategy-workflow success remained limited. Staged checks exposed semantic, evidence-completeness, and invalid-input failures, even as cases reaching final review passed.

5.3 Evidence and Outcomes of Representative Cases

  • Representative-case judgments considered original input, expected conditions, actual response, stopping point, and verification evidence, exposing gaps in evidence completeness, semantic preservation, and invalid-input interception.The cases also show that correct conversion and execution do not make a strategy eligible unless throughput constraints are met.Source: §5 · p. 13, §5.3 · pp. 16–17

5.1 Workflow Evaluation by Test Group

  • TC1 met complete expectations in 3/8 cases, while TC2 query handling met requirements in 5/9; query errors included misclassification, omitted values, and scope failures.Some TC1 cases stopped before CP4 or lacked complete simulation or strategy evidence.Source: §5.1 · pp. 13–14
  • TC3 met expectations in 2/2 cases, TC4 intercepted 7/8 invalid inputs, TC5 traced timing in 1/1 case, and model semantic passes ranged from 0/3 to 2/3.One invalid line identifier escaped interception; the generation tests reported structural validity separately from semantic correctness.Source: §5.1 · p. 14

5.2 Metric-Based Evaluation

  • 18/28 workflow cases met expectations, autonomous strategy-workflow success was 3/10, correct rejection was 7/8, and all four cases reaching CP6 passed engineering review.These results support staged screening and confirmation within the evaluated cases, not overall reliability or long-term stability.Source: §5.2 · p. 14, §5.4 · p. 17
  • Mean placement-validation pass rate across eight simulation evidence records was 97.50%, while mean times to first reviewable response and simulation verification were 12.94 and 164.39 s.The populations differ by metric, and complete verification evidence takes longer than an initial reviewable response.Source: §5.2 · p. 16

5.4 Checkpoint Results and Limitations

  • Although all 12 CP2 outputs were structurally valid, failures occurred at CP3 semantics (13/27), CP4 simulation (2/9), and CP5 constraints (3/7); T06's incorrect strategy still reached simulation.Later digital-twin checks exposed execution and operational problems that structural checks alone did not identify.Source: §5.4 · p. 17

5.5 Exploratory Comparison of Model Generation

  • On the shared input, all nine outputs were structurally valid, but semantic passes were Gemma 0/3, Qwen 2/3, and Llama 0/3; one input cannot establish an overall model ranking.Gemma extended a single-line throughput target across all four lines, while Qwen separated the scopes twice and Llama requested clarification in all three outputs.Source: §5.5 · p. 18

6 Discussion and Limitations

Staged verification blocked strategies that failed throughput constraints, and all four complete-evidence cases reaching CP6 passed engineering review. The discussion also identifies limits in semantic checking, human workload evidence, candidate selection, simulation scope, and evaluation design.

6.1 Value and Boundaries of Staged Verification

  • T02 and T05 were blocked after failing throughput constraints, and all four records reaching CP6 with complete evidence passed engineering review.These findings support screening and confirmation within the fixed test set, not overall reliability.Source: §6.1 · p. 18
  • Semantic distortion and missed invalid inputs remained, while the small sample cannot establish overall reliability or stability.The authors prioritize executable checks for line scope, target instruments, and priority rules, plus stopping or retry behavior for incomplete evidence.Source: §6.1 · p. 18

6.2 Human-AI Responsibilities and Verification Cost

  • Autonomous strategy-workflow success was 3/10 and query success was 5/9, leaving clarification, correction, and failure diagnosis necessary.Current evidence does not establish reduced operator workload or staffing requirements.Source: §6.2 · p. 19
  • Mean times to the first reviewable response and simulation verification were 12.94 and 164.39 s, respectively; real-time response capability remains unestablished.The workflow is better suited to adjustments that allow advance simulation; researcher interpretation was not included in end-to-end timing.Source: §6.2 · p. 19

6.3 Local Models and Bounded Candidate Sets

  • Local deployment did not ensure correct requirement interpretation, and selection among three candidates is batch-relative rather than necessarily globally optimal.The exploratory comparison also found scope expansion and unnecessary clarification despite structurally valid outputs.Source: §6.3 · p. 19

6.4 Simulation Evidence and Generalizability

  • Thirty fixed records—28 for workflow evaluation and two for model generation—were researcher-designed, operated, and retrospectively reviewed, limiting estimates of long-term success and cross-site applicability.The records cannot all be treated as independent deployment trials.Source: §6.4 · p. 19

7 Conclusion

The study implements a traceable workflow combining local-LLM proposals, digital-twin verification, and human decisions. In 28 fixed workflow cases, 18 met expectations, while remaining failures and the evaluation scope limit broader claims.

7 Conclusion

  • The workflow links operator intent, requirement versions, candidate strategies, scenario specifications, and execution evidence, while mandatory constraints filter a bounded candidate set.These links provide traceable grounds for adopting or rejecting a recommendation.Source: §7 · pp. 19–20
  • Eighteen of 28 workflow cases met expectations; autonomous strategy-workflow success was 3/10, and correct rejection of negative inputs was 7/8.All four records reaching CP6 after preceding checks and complete evidence passed engineering review; results support staged screening and confirmation within the fixed test set.Source: §7 · p. 20
  • The fixed-case evaluation does not demonstrate physical-deployment benefits or labor savings; broader testing and physical evaluations are needed to assess generalizability.Remaining failures include semantic distortion, incomplete evidence, and missed invalid inputs.Source: §7 · p. 20
Loading 2609.29061v1…