PaperlayerView exact revision on arXiv

Source-linked AI summary

Can AI Agents Help Robotic Engineers? Lessons from Two Focus Days in Industry

Salim Fares, Steffen Herbold

arXiv:2610.32636v1cs.SEcs.AI

TL;DR

Evidence was lacking on whether agents help industrial robotic engineers with their range of daily tasks and how engineers adopt them. In a case study with an SME, engineers used an on-premise agent on real tasks; it generally did not improve productivity, though results improved across focus days.

  • Problem

    It was unclear whether agents help robotic engineers across daily tasks and how these engineers use them or encounter adoption problems.

    Source: §I · p. 1

  • Method

    A case study at an SME compared four robotic engineers’ work on six tasks with and without an on-premise AI agent over two focus days.

    Source: §VII · p. 10

  • Results

    With the agent, engineers finished only 4 of 21 tasks within the time limit, versus 14 of 21 without AI, and solution functionality was worse; highly repetitive code was the exception.

    Source: §I · p. 1, §VII · p. 10

  • Takeaways & Limitations

    SMEs should train engineers in agent use and plan for a learning curve, since robotic-engineering experience alone does not ensure successful agent use.

    Source: §I · p. 1

    The small sample—four developers on the first focus day and three on the second, at one company with one agent and model—does not support statistical generalization.

    Source: §VI · p. 10

Abstract

from arXiv · show

Software developers increasingly use AI agents that execute multi-step programming tasks on their own. However, it is unclear if and how this transfers to robotic engineering, where engineers program industrial robots in niche, vendor-specific languages and rely on visual feedback from simulation tools. Within this paper, we report a case study on the use of AI agents by an SME that programs industrial robots. Our results show that the agent did not make the engineers more productive, with the exception of a task that required the generation of highly repetitive code. However, the results with the agent improved over time, as the engineers learned how to use it and the agent was adapted to the domain. Moreover, the experience with AI mattered more for the success with the agent than the experience with robotic engineering. This shows that AI agents are not a plug-and-play solution. Instead, companies must train their engineers on the usage of agents, plan time for a learning curve, and ensure that agent configurations match the domain and developer requirements. The strong learning curve we observed further implies that studies on productivity gains with AI should be longitudinal, as they otherwise miss how engineers adopt agents in practice.

I. INTRODUCTION

The paper examines whether agentic AI helps industrial robotic engineers, whose work involves vendor-specific robot languages and visual simulation, and how they adopt agents. Its case study finds limited productivity benefits but derives practical lessons about training, task choice, and domain-aware configuration.

I. INTRODUCTION

  • Evidence on agent productivity in robotic engineering was lacking, including how engineers use agents across daily tasks and what adoption problems they face.Existing industrial-automation LLM work evaluated AI in isolation, without engineers in the loop.Source: §I · p. 1
  • The study asks whether agentic AI affects robotic-task completion, time, and quality, and how specialized engineers use it and encounter adoption problems.Source: §I · p. 1
  • The agent did not make engineers faster or improve functionality, except on highly repetitive code, while results improved between focus days.The authors associate improvement with both engineers’ learning and agent redesign.Source: §I · p. 1
  • An on-premise agent with an open-weight model is technically feasible for SMEs with confidential data, but is not a plug-and-play productivity solution.Source: §I · p. 1
  • SMEs should train engineers in when to delegate versus edit manually, and plan for a learning curve; robotic expertise does not guarantee agent experience.Source: §I · p. 1
  • Repetitive, well-specified generation tasks are a better starting point than tasks requiring spatial understanding, and domain references and standards should reach the agent automatically.Source: §I · pp. 1–2

II. RELATED WORK

Prior studies report mixed productivity effects from AI coding tools, while domain-specific industrial work has not established how agents support the breadth of robotic programming tasks. This paper uses an in-context case study at an SME with real engineers, tasks, and operational constraints.

II. RELATED WORK

  • Evidence on newer AI coding tools is limited and inconclusive: studies report productivity losses, gains, increased coding activity with only partial output gains, and diminishing task-level returns as complexity rises.Becker et al. [4],, Chen et al., and Demirer et al. report these respective findings.Source: §II · p. 2
  • Industrial-control and manufacturing agents show promise in bounded settings, but prior work does not cover the general range of robotic programming tasks.LLM4PLC was not validated in a real industrial setting; Patel studied maintenance rather than programming new components; prior RAPID work covered three controlled problems on historical data.Source: §II · p. 2

III. METHODOLOGY

  • The case study gathers quantitative and qualitative evidence from a small sample of engineers and tasks in their real working environment, without aiming for statistical generalization.The authors frame this design following Wohlin et al..Source: §III · p. 2

A. Industrial context

  • AKE engineers program ABB robots in RAPID and develop and test programs in RobotStudio virtual stations before deployment.Source: A. Industrial context · p. 2
  • Customer confidentiality prohibits cloud LLMs, and limited resources rule out custom development or training; the available inference server has one H100 GPU.The authors consider this context representative of SMEs with confidential work and limited budgets.Source: A. Industrial context · p. 2
  • The team lead defined measured tasks and assessed solution quality against AKE standards, while researchers provided and monitored the AI solution and study instruments.Researchers report no influence on task selection or assessment.Source: A. Industrial context · p. 2

B. Tasks

The study extends prior use cases to six real-project tasks spanning modifications, documentation, debugging, path inversion, trajectory generation, and stacking. The tasks include common program extension and customer-facing documentation work.

B. Tasks

  • Six tasks span refactoring, documentation, debugging, path inversion, trajectory creation, and 3D stacking, with three extending prior use cases and three introducing new activities.Descriptions followed AKE conventions, were written in German, and represented non-trivial work estimated at 30–60 minutes.Source: B. Tasks · p. 3

1) Task 1: Refactoring and extension:

  • Task 1 extends an existing program by copying similar routines, making small changes, and integrating calls consistently into the program flow.This is common at AKE because production lines often resemble earlier projects.Source: 1) Task 1: Refactoring and extension · p. 3

2) Task 2: Documentation:

  • Task 2 requires external documentation from a template, covering system metadata, coordinate systems and tools, workflow, and movement-routine descriptions.Engineers must understand the existing code and workflow and describe them in natural language; this is the only task without RAPID coding.Source: 2) Task 2: Documentation · p. 3

3) Task 3: Debugging:

The remaining tasks test debugging, movement inversion, repetitive trajectory generation, and 3D stacking. The study also used an on-premise coding agent whose configuration changed between focus days to make domain knowledge more available.

3) Task 3: Debugging:

  • Task 3 asks engineers to find and fix eight subtle errors: two syntax errors and six semantic errors requiring understanding of code workflow and robot behavior.The team lead induced the errors based on common patterns encountered by engineers.Source: 3) Task 3: Debugging · p. 3

4) Task 4: Path inversion:

  • Task 4 requires reversing three movement routines, potentially changing, adding, or removing movements because reversing order alone may not invert the robot’s path.Source: 4) Task 4: Path inversion · p. 3

5) Task 5: Trajectory matrix generation:

  • Task 5 requires generating forward and backward movement routines between each pair of positions from scratch, making the growing routine set repetitive and error-prone.The positions are supplied as coordinates; the scenario represents introducing a new component into a production line.Source: 5) Task 5: Trajectory matrix generation · p. 3

6) Task 6: 3D stacking:

  • Task 6 involves stacking batteries with repeated movements at different offsets, requiring mathematical offset computation and efficient movement-order planning.The team lead described it as the most difficult task.Source: 6) Task 6: 3D stacking · p. 3

C. Customized AI Agent

  • The agent used Claude Code with GPT-OSS 120B [16] served through Ollama on-premise; the study did not systematically compare models.Before the focus days, the setup was checked for syntactically correct RAPID code and reasonable task-directed outputs, not full autonomy.Source: C. Customized AI Agent · pp. 3–4
  • On the first focus day, developers had to invoke three prepared skills themselves; after they did not use them as intended, the second-day setup supplied project context automatically.That context included the RAPID manual, task descriptions, project structure, and code-generation rules.Source: C. Customized AI Agent · p. 4

D. Focus days

The study used two focus days to compare engineers’ work with and without an agent in their usual work environment. Questionnaires tracked expectations, task experiences, and post-day perceptions.

D. Focus days

  • An AB/BA crossover design alternated whether two groups used the agent or worked without AI in the morning, then swapped conditions on the second focus day.The small number of developers does not allow statistical generalization.Source: D. Focus days · p. 4
  • Each developer completed comparable task types in both conditions, but second-day task descriptions differed to prevent simply reproducing earlier solutions.Tasks were timeboxed, and an eight-week break separated the focus days; developers were encouraged to use the agent during that interval.Source: D. Focus days · p. 4

E. Questionnaires

  • The task questionnaires recorded perceived difficulty and realism, self-reported brainstorming and implementation time, and problems encountered.The pre- and post-questionnaires also compared expectations with experience, including whether the agent helped solve tasks.Source: E. Questionnaires · pp. 4–5

F. Evaluation of task solutions

Task solutions were evaluated for completion time and quality using recorded task durations, team-lead ratings, and automated checks of RAPID code.

F. Evaluation of task solutions

  • Task time ran from task start until the developer reported completion, and the evaluation recorded whether each task was finished within its time limit.Limits varied by task: 60 minutes for Task 1, 30 minutes for Tasks 2–4, and 45 minutes for Tasks 5–6.Source: F. Evaluation of task solutions · p. 5
  • Quality assessment combined the team lead’s review of RobotStudio backups with automated validation of RAPID code for syntactic violations of AKE’s standards.For debugging, the team lead counted fixed and remaining injected syntax and semantic errors instead of rating the usual criteria.Source: F. Evaluation of task solutions · p. 5
  • The team lead rated functionality, code quality, and readability on a five-point scale relative to expectations for an experienced AKE engineer.The assessment was not blind; debugging was assessed by counting errors fixed and remaining.Source: F. Evaluation of task solutions · p. 5

IV. RESULTS

On the first focus day, developers completed fewer tasks with the agent, and their solutions generally scored lower than solutions produced without AI. They also reported more time and new interaction problems with the agent.

B. First focus day

  • Tasks were rated realistic (mean agreement 4.55), but after the first focus day, agreement with most expectations fell, including standards adherence (3.0 to 2.0).Developers remained neutral about helpfulness (mean 3.0); they cited unfamiliarity with the terminal interface and lack of ABB documentation and RobotStudio access.Source: §IV · p. 5, B. First focus day · p. 7

A. Participants and expectations

  • The four engineers had 7–14 years of RobotStudio experience, while their AI experience varied; before the study, they were open to using agents but neutral about better results.Mean expected speed-up was 3.5, compared with 2.5 for better results.Source: A. Participants and expectations · p. 5

B. First focus day

  • 3 of 12 tasks were finished with the agent, compared with 9 of 12 without AI; agent-assisted developers required at least as much time on every task.The table summarizes time and team-lead assessments by task, developer, and focus day.Source: B. First focus day · pp. 5–6, Table I · p. 6
  • Across tasks, agent-assisted solutions averaged 1.7 for functionality, 1.8 for code quality, and 2.1 for readability, below the respective 2.3, 2.5, and 2.8 scores without AI.The validator found 25 standards violations with the agent and 11 without AI; agent solutions were worse on four of six tasks.Source: B. First focus day · pp. 6–7
  • With the agent, developers reported more brainstorming time (12.8 vs. 8.8 minutes) and implementation time (24.4 vs. 21.3 minutes), while agent errors were their most frequent new problem.They used the agent for every change, including trivial edits, and repeated code regeneration led to long conversations and drift from correct intermediate solutions.Source: B. First focus day · p. 7

C. Second focus day

On the second focus day, the agent still did not improve task completion, although quality gaps narrowed on some measures. Developers remained more likely to find agent-assisted work difficult, while the problems they reported shifted.

C. Second focus day

  • With the agent, developers finished 1 of 9 tasks within the time limit, compared with 5 of 9 without AI.The agent-assisted trajectory-matrix task took D2 15 minutes, below the 18 minutes needed for the same task type without AI on the first focus day.Source: C. Second focus day · p. 7
  • Average functionality was 1.8 with the agent and 2.3 without AI, while code quality was 2.0 vs. 2.3 and readability 2.4 vs. 2.7.The gaps in code quality and readability narrowed, while the functionality gap mostly remained.Source: C. Second focus day · p. 7
  • Developers rated tasks as more difficult with the agent (mean 3.2) than without AI (2.1), though they reported less brainstorming time than on the first day in both conditions.With the agent, the most frequent reported problem was explaining the task (4 mentions); agent errors and usability problems were mentioned rarely, while without-AI problems remained similar.Source: C. Second focus day · pp. 7–8
  • After the second focus day, agreement on result quality and standards adherence fell to 1.67, while desire to use AI remained high at 4.33.The overall comparison is affected by D3’s absence; among D1, D2, and D4, desire to use AI rose from 4.0 to 4.33.Source: C. Second focus day · p. 8
  • The agent configuration addressed missing domain knowledge, but developers still used it for every change, including trivial ones, leading to long conversations and drift from correct intermediate solutions.For Tasks 4 and 5, automatic activation of the reversing skill coincided with fewer prompts than on the first focus day.Source: C. Second focus day · p. 8

D. Comparison of the focus days

Across the two focus days, agent-assisted completion and functionality did not improve, although code quality and standards adherence improved. The trajectory-matrix task was the only task with equal or better agent results on both days, and AI experience appeared more relevant than robotics experience to success.

D. Comparison of the focus days

  • Agent-assisted completion stayed low across focus days: 3 of 12 tasks on the first and 1 of 9 on the second, versus 9 of 12 and 5 of 9 without AI.Functionality did not improve; code quality rose from 0.8 to 1.0, readability from 1.0 to 1.4, and standards violations fell from 25 to 16.Source: D. Comparison of the focus days · p. 8
  • Trajectory matrix generation (Task 5) was the only task with the same or better agent results on both focus days, and D2 completed it faster on day two than without AI on day one.Developers perceived four of six tasks as harder with the agent on day two than day one; only path inversion (Task 4) felt easier.Source: D. Comparison of the focus days · p. 8
  • The only task completed with the agent on day two was completed by D2, who had the most AI experience; D4, who rarely or never used AI, completed none with the agent.All developers finished tasks without AI on both focus days.Source: D. Comparison of the focus days · p. 8

V. DISCUSSION

The agent did not improve task completion, speed, or functionality overall, but standards adherence improved as the agent was reconfigured and developers gained experience. Adoption problems included over-delegation, ignored skills, missing simulation access, and initial unfamiliarity with the terminal.

A. RQ1: Completion, time, and quality

  • The agent did not help developers complete more tasks, work faster, or produce more functional solutions than without AI.The authors attribute the contrast with prior positive work in part to the real-world setting and developers’ differing expectations and AI experience.Source: A. RQ1: Completion, time, and quality · pp. 8–9
  • The agent performed worst on Tasks 1, 4, and 6, which were closest to the prior work; small isolated changes did not ensure efficiency on larger realistic tasks.The functionality gap mostly remained, while code quality and readability gaps narrowed between focus days.Source: A. RQ1: Completion, time, and quality · p. 9
  • Standards adherence improved between focus days, with 5 of 8 agent solutions free of violations on day two versus 2 of 10 on day one.The authors attribute this partly to agent reconfiguration, showing that the learning curve also applies to configuration.Source: A. RQ1: Completion, time, and quality · p. 9
  • The agent helped most with highly repetitive code: trajectory-matrix generation was its only task with equal or better results on both days.This worked when developers understood the task and trajectories and could specify the desired output precisely.Source: A. RQ1: Completion, time, and quality · p. 9

B. RQ2: Use of the agent and adoption problems

  • The main adoption problem was not only model capability: experienced robotic engineers were not automatically experienced agent users, and robotic work relies on visual tools unlike the text-based programming agents target.Developers delegated even trivial edits, producing inefficient long conversations in which the agent drifted from correct intermediate solutions.Source: B. RQ2: Use of the agent and adoption… · p. 9
  • Developers ignored prepared skills because they used the agent like a chat system; automatically supplying project-context knowledge in the redesign fit their habits better.The second-day redesign removed the need to invoke skills manually.Source: B. RQ2: Use of the agent and adoption… · p. 9
  • The agent could access code but not the virtual station, and the authors believe this lack of spatial understanding contributed to poor generated-code functionality.They suggest deeper RobotStudio integration, for example through suitable MCP servers, as a possible remedy.Source: B. RQ2: Use of the agent and adoption… · p. 9
  • The terminal initially felt unfamiliar and unfriendly to engineers accustomed to graphical tools, but nobody named usability as a problem after the second focus day.The authors caution that short controlled evaluations can miss such learning effects.Source: B. RQ2: Use of the agent and adoption… · p. 9

C. Recommendations for practitioners

The authors recommend that SMEs adopting agents for robotic engineering train developers, begin with suitable tasks, and configure agent context to fit the domain.

C. Recommendations for practitioners

  • Train developers in when to use the agent, edit manually, or start a new conversation; a short introduction was insufficient in this case.Plan dedicated exercises and time for developers to gain familiarity during daily work.Source: C. Recommendations for practitioners · p. 10
  • Start with repetitive, well-specified generation tasks, such as trajectory matrices or documentation drafts, rather than tasks requiring spatial understanding of the robot cell.These tasks let developers gain experience where the agent already provides a benefit.Source: C. Recommendations for practitioners · p. 10
  • Make reference manuals and programming standards automatically available through project context instead of requiring developers to invoke skills.In this case, this improved standards adherence and removed missing ABB documentation from developers’ reported problems.Source: C. Recommendations for practitioners · p. 10

D. Implications for researchers

The paper recommends evaluating agents with developers in real contexts, studying adoption longitudinally, and reporting agent configurations. Its case-study evidence is limited in statistical generalizability, and the conclusion calls for longer-term study and deeper RobotStudio integration.

D. Implications for researchers

  • Benchmarks on isolated tasks do not capture how developers use agents; researchers should complement them with studies of real tasks in real contexts.Prior work showed an LLM could perform key task steps, yet developers in this case did not benefit from the agent.Source: D. Implications for researchers · p. 10
  • Two focus days provide only snapshots despite observed learning effects, so longitudinal work is needed to track learning, changing use patterns, and whether productivity eventually exceeds unaided work.The authors also recommend reporting configuration details because skills invoked by developers versus automatically supplied project context can affect outcomes.Source: D. Implications for researchers · p. 10

VI. THREATS TO VALIDITY

  • The case study’s small sample—four developers on day one and three on day two, at one company with one agent and model—precludes statistical generalization.Other models, agents, or companies may produce different results.Source: §VI · p. 10
  • The team lead was the sole, non-blinded assessor, while the standards validator independently checked only adherence to programming standards.Time limits also prevented measuring how long unfinished tasks would have taken.Source: §VI · p. 10
  • The focus-day design was not a controlled experiment because of the small sample and changes such as agent redesign, and it was not intended to establish causal claims.The study involved four experienced engineers, six real-project tasks, and an on-premise agent over two focus days.Source: §VI · p. 10, §VII · p. 10

VII. CONCLUSION

  • The conclusion reports lower completion with the agent—4 of 21 tasks versus 14 of 21 without AI—and worse functionality, with highly repetitive code the exception.The authors call for longer-term study and investigation of RobotStudio integration to address missing spatial understanding.Source: §VII · p. 10
Loading 2610.32636v1…