PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Robots for Everyday Environments: From Lab Experiments to Real-World Operations

Raphael Memmesheimer, Martina Overbeck, Dominik Beyer, Björn Kral, Sabine Bellmann, Sven Schneider, Jan Zimmermann, Anna-Maria Meer, Medina Klicic, Simone Roth, Carolin Straßmann, Alexander Arntz, Marlene Wessels, Johannes Kraus, Paul Schweidler, Tristan Schnell, Christoph Zimmermann, Benedikt Pulver, Wilhelm Stork, Martin Gersch, Sven Behnke, Arne Rönnau

arXiv:2609.26490v1cs.ROcs.HC

TL;DR

Robotics benchmarking for real-world deployment has limited guidance for heterogeneous, high-TRL applications beyond structured, same-task comparisons. This paper develops an interdisciplinary, use-case-specific evaluation process and finds that diverse cases yield meta-level insights, even where results are not unified.

  • Problem

    Existing benchmarking efforts commonly compare systems on the same predefined task in structured, reproducible settings, providing limited guidance for heterogeneous applications at high technology readiness levels.

    Source: §1 · p. 2

  • Method

    An interdisciplinary panel consolidated deployer-proposed metrics for three public-space use cases across task fulfillment, interaction quality, safety, and economic viability.

    Source: §1 · p. 2, §6 · p. 34

  • Results

    The Public Library use case showed comparable task fulfillment, while realistic Park Cleaning encountered difficulties such as bottle caps embedded in grass, and the heterogeneous cases yielded meta-level insights.

    Source: §5 · p. 32, §6 · p. 34

  • Takeaways & Limitations

    The framework should not prescribe fixed metrics; a shared use-case profile should guide metric selection and consideration of relevance, operationalization, applicability, and comparability.

    Source: §6 · p. 34

    Unifying task fulfillment metrics across diverse use cases remains impractical, particularly when their foci differ.

    Source: §5 · p. 32

Abstract

from arXiv · show

This study introduces an interdisciplinary framework for benchmarking robots deployed in public environments, addressing the gap between traditional laboratory metrics and real-world benchmarking requirements. We evaluate three distinct robots across diverse use cases - outdoor park cleaning, pedestrian underpass cleaning, and interactive library assistance - each representing unique challenges in public daily life. Over a three-year benchmarking process (2023-2025) comprising seven benchmarking events, a consensus workshop and six on-site evaluations (two per use case), we utilized realistic indoor and outdoor test environments to assess not only technical performance but also the broader implications of deploying robots in unstructured, human-centric settings. An expert panel, spanning robotics, human-robot interaction, safety, and economics, systematically developed and refined an evaluation concept to analyze the transition from laboratory prototypes to operational systems. Our findings highlight critical factors for successful deployment, including task fulfillment, interaction quality, safety, and economic feasibility. This work provides actionable insights for researchers and practitioners aiming to bridge the gap between robotic innovation and real-world applicability.

1 Introduction

Benchmarking heterogeneous public-space robots at high technology readiness levels is difficult to capture with methods built for structured, same-task comparisons. The paper develops an interdisciplinary, use-case-specific evaluation process and applies it across three public-space scenarios.

1 Introduction

  • Existing competitions provide limited guidance for heterogeneous, high-TRL applications because they commonly compare systems on the same predefined task in structured, reproducible settings.Benchmarking robots across comparable environments is also described as a complex methodological challenge.Source: §1 · p. 2
  • An interdisciplinary panel-based process has robot deployers propose use-case-specific metrics, which are consolidated across disciplines in consensus meetings.The process is inspired by the EuRoC freestyle format.Source: §1 · p. 2
  • The evaluation concept organizes metrics into task fulfillment, interaction quality, safety, and economic viability across three public-space use cases.Metrics were refined over three phases of increasing realism.Source: §1 · p. 2
  • The empirical contribution draws cross-case findings from seven benchmarking events, including a consensus workshop and six on-site evaluations with three TRL 6–7 robots.The findings address which metrics, guidelines, and standards transfer across use cases and which do not.Source: §1 · p. 2

2 Related Work

The related work examines gaps between laboratory-style evaluation and benchmarking in public environments, including criteria for task fulfillment, interaction, and safety, as well as limits in existing competition and interaction benchmarks. It also reviews safety and economic benchmarking and a toolbox-based approach to economic benchmarking across contexts.

2 Related Work

  • The paper distinguishes interaction-centric social service robots from service robots whose public deployment can still produce incidental social encounters, motivating interaction assessment across all three use cases.Its evaluation defines metrics, protocols, and pass/fail criteria in advance, applies them under controlled conditions, and refines them across phases toward accepted standards.Source: §2 · p. 3

2.1 Benchmarking Approaches for Task Fulfillment

  • Competitions provide task benchmarks, but controlled predefined scenarios inadequately capture high-TRL public environments or cross-scenario factors [Memmesheimer et al., 2024b, Hart et al., 2026].The work complements public-deployment and interaction studies with structured, multi-category benchmarking [Pelikan et al., 2025b, M¨uller and Richert, 2024].Source: §2.1 · p. 4

2.2 Benchmarking Approaches for Interaction Quality

  • Recent HRI-centered competitions illustrate a shift from implicitly including interaction to treating it as an independent benchmark dimension, while evaluations have thus far predominantly relied on human assessments.Source: §2.2 · p. 6

2.3 Benchmarking Approaches for Safety

  • Regulations and harmonized safety standards provide a baseline, but primarily support conformity assessment rather than comparative benchmarking of robotic systems.The discussion situates this gap in the EU context.Source: §2.3 · p. 6
  • Industrial safety standards often assume healthy, informed workers and individual harms, leaving public-space benchmarking to address vulnerable people, groups, and indirect hazards.Examples of indirect hazards include a cleaning robot collecting medical supplies or toppling a charcoal grill and scattering embers.Source: §2.3 · p. 7
  • Safety benchmarking for public deployments must consider human and environmental harm, interaction and regulatory concerns, while recognizing that one compressed setup cannot assess every aspect and excludes security.Applicable road-traffic regulations can depend on the use case.Source: §2.3 · pp. 7–8

2.4 Benchmarking Approaches for Economic Feasibility

  • Despite varied approaches to economic viability, the literature offers no established benchmark format for service robots in everyday public environments.Public-sector economic considerations are intertwined with organizational structures, institutional constraints, and stakeholder settings.Source: §2.4 · p. 8
  • Economic benchmarking uses the RimA Toolbox’s phase-specific workflow and business-model methods, refined through application feedback to support systematic assessment across robot types, use cases, and ecosystems.Its methods include the Value Proposition Canvas, Customer Journey, Stakeholder Network, and Business Model Canvas.Source: §2.4 · p. 8

3 Benchmarking Method

The framework benchmarks distinct public-space robots with use-case-specific metrics across technical, human-centered, safety, and economic dimensions. Over three years, metrics were refined and applied through increasingly practical evaluation phases.

3 Benchmarking Method

  • The approach adapted metrics for fundamentally different library and cleaning robots, refining them with a benchmarking panel before applying them from concept development through real-world deployment.The categories were task fulfillment, interaction quality, safety, and economic viability.Source: §3 · pp. 8–9

3.1 Benchmarking Phases

  • Phase 1 developed evaluation concepts; Phases 2 and 3 applied the metrics under increasingly practical conditions after consensus workshops and interim meetings.Phase 1 concluded with a consensus workshop, with further metric development after Phases 1 and 2.Source: §3.1 · p. 9

3.2 Benchmarking Categories

  • Four continuously assessed categories—task fulfillment, interaction, safety, and economic viability—were designed to give equal importance to multiple benchmarking dimensions.The framework assumes daily-life robots should be useful, interact with people, and be safe, while economic viability supports transition by deployers.Source: §3.2 · p. 9

3.2.1 Task Fulfillment

  • Task fulfillment uses objective, standardized measures such as success rates, completion times, and area processed per unit time to assess practical system performance.The initial metric-selection phase was overseen by experts experienced in benchmarking robots outside laboratory settings and competitions.Source: §3.2.1 · p. 9

3.2.2 Interaction Quality

  • Interaction quality seeks use-case-universal, systematic measures of human-centered performance despite being more subjective than task fulfillment.Candidate dimensions include comprehensibility, predictability, trustworthiness, perceived presence, and acceptance.Source: §3.2.2 · p. 11

3.2.3 Safety

  • Safety assessment specifies metrics and verification procedures for mechanical and functional safety, including safety distances, impact forces, performance level, and response time.The evaluation concept was intended to relate verification procedures to required performance levels.Source: §3.2.3 · p. 11

3.2.4 Economic Viability

  • Economic viability was assessed through phase-specific completion criteria and process reflections rather than uniform quantitative economic outcome measures.The transfer-oriented process used the RimA Toolbox; selected methods were applied and adapted to each use case, with feedback informing iterative refinement.Source: §3.2.4 · pp. 11–12
  • The economic process moved from selecting four toolbox methods, to applying two or three tools in practice, to completing a tailored business plan.The plan addressed value proposition, stakeholders and markets, financial structure, timeline, and tool-use reflection.Source: §3.2.4 · p. 11

3.3 Use Cases

The benchmarking covers library assistance, park litter collection, and underpass sweeping, each with distinct tasks and human-centered challenges. The use cases expose robots to varied interactions and constraints in public settings.

3.3 Use Cases

  • The three use cases deploy Pepper in a library, Angsa to collect small park litter, and Adlatus to sweep a pedestrian underpass.They span interaction-focused assistance and public-space cleaning applications.Source: §3.3 · p. 12

3.3.1 Public Libraries

  • The library robot recommends books and presents reading samples, with evaluation intended to include diverse users, including regular patrons and first-time visitors.Robot ID cards support addressing returning participants by name and providing language support when needed.Source: §3.3.1 · p. 12

3.3.2 Park Cleaning

  • The waist-high park robot moves at walking speed, stops to collect identified cigarette butts or bottle caps, and can expect encounters with people in public spaces.Interaction can include stopping before physical contact, while broader design aims to limit effects on recreation, sport, and mobility in nearby areas.Source: §3.3.2 · p. 13

3.3.3 Pedestrian Underpass Cleaning

  • In the underpass, the Adlatus SR1300 follows a wall-parallel route and turns at its end, leaving a passage about 60 cm wide between robot and wall.The navigation challenge is to keep movement smooth, pleasant, safe, efficient, and uninterrupted for users and robot.Source: §3.3.3 · pp. 13, 15

3.4 Benchmarking Panel

An interdisciplinary panel developed and refined the evaluation approach across the benchmarking process. The concept combined deployer-defined metrics, panel consensus, and evaluations under laboratory and realistic practical conditions.

3.4 Benchmarking Panel

  • The panel drew on representatives from seven institutions, including project leads and experts in robotics, human-robot interaction, safety, and economics.Four to seven members attended each event; 15 individuals participated across the three-year period, with briefings intended to support consistency.Source: §3.4 · p. 15

3.5 Evaluation Concept

  • Deployers first defined use-case-specific metrics, which the panel refined at a consensus workshop before finalizing the evaluation concept.The panel incorporated its feedback into the concept.Source: §3.5 · p. 15
  • The panel conducted one-day evaluations under lab conditions, then evaluated each use case in practice in realistic environments.The practical evaluation followed the laboratory evaluation phase.Source: §3.5 · p. 15

3.6 Benchmarking Procedure and Materials

The benchmarking procedure documents test materials and system configurations, then converts panel observations into results using repeatable, use-case-specific procedures.

Procedure

  • The procedure was designed to support repeatability by summarizing test procedures, materials, system configurations, and how panel observations became reported results.Complete evaluation sheets, checklists, and interaction-quality guidelines are provided in the supplementary material.Source: §3.6 · p. 15

Public library

  • Library interactions were initiated with a robot ID card and scored on a 0–5 scale, with weighted category scores normalized into a dialog success rate.The weighting assigned 60% to book recommendations, 20% to system functionality, and 10% each to introduction and farewell.Source: §3.6 · pp. 15–16

Park cleaning

  • Park cleaning used autonomous spiral coverage at a target speed of 0.3 m s−1, with counted litter and hits and misses checked against known ground truth.Runs were documented with before-and-after photographs.Source: §3.6 · p. 16

Pedestrian underpass cleaning

  • Underpass cleaning used a stored meander plan with counted impurities and obstacles; inspectors checked cleaning completeness, while efficiency used area and duration.The test areas measured 6.5 m × 10.4 m in Phase 2 and 7 m × 18.5 m in Phase 3.Source: §3.6 · p. 16

Evaluation and scoring

  • Panel members completed use-case questionnaires or guidelines with 10-point category ratings and comments; ratings were consolidated in a post-event debriefing.Safety experts assessed predefined pass/fail criteria, including emergency stops and obstacle detection.Source: §3.6 · p. 16

Conditions and reproducibility

  • Phase 3 public-environment conditions such as weather, noise, network quality, and visitor traffic could not be controlled and were documented.Robot platforms were continuously developed, and software version identifiers were not systematically recorded, limiting exact reproducibility.Source: §3.6 · p. 16

4 Results

The three-year process completed seven benchmarking events, including one consensus workshop and two on-site evaluations per use case. Each on-site event was a one-day panel evaluation, so per-event sample sizes were inherently small.

Phase execution

  • The consensus workshop took place in Bonn in December 2023, followed by Phase 2 and Phase 3 on-site evaluations in 2024 and 2025.Phase 2 evaluations took place in Bottrop, Stuttgart, and Ulm; Phase 3 evaluations took place in Ulm, Duisburg, and Munich.Source: §4 · p. 16, Table 2 · p. 17

Scope

  • Because each evaluation was a one-day panel event, per-event sample sizes were inherently small and are reported alongside the results.Interim consensus meetings after Phases 1 and 2 supported refinement but are not counted among the seven events.Source: §4 · p. 16

Event schedule

  • Seven benchmarking events comprised one consensus workshop and six on-site evaluations, with two evaluations for each use case.The on-site evaluations covered public library, park cleaning, and pedestrian underpass cleaning.Source: §4 · p. 16, Table 2 · p. 17

4.1 Task Fulfillment Results

This section reports task-fulfillment assessments using use-case-specific measures under laboratory and public-deployment conditions, including dialog success and cleaning quality and efficiency. It covers condition- and target-dependent cleaning outcomes, operational limitations, and the distinction between verified evaluation-area items and naturally occurring litter in an internal log.

4.1.1 Results from Phase 1 (Task Fulfillment)

  • Phase 1 defined library dialog-success measures, park litter-detection and cleaning-efficiency measures, and underpass completeness, quality, and efficiency measures.Library metrics used a sequential structured checklist; park reliability assessed small-litter detection and distinction using hits, misses, false positives, and false negatives, while underpass tests included varied surfaces and contamination.Source: §4.1.1 · pp. 17–18

4.1.2 Results from Phase 2 (Task Fulfillment)

  • Dialog success was 35.7%, with failures from unavailable watchlist functions and unintuitive robot feedback.The rate was based on one systematically scored interaction; further informal interactions informed qualitative observations.Source: §4.1.2 · p. 19
  • Phase 2 park cleaning missed mostly out-of-distribution litter, such as rubber bands or cable ties.Source: §4.1.2 · p. 20
  • 95% Completeness and Quality and 13.5 m2 min−1 Efficiency were recorded in one underpass cleaning run, projecting 810 m2 h−1.The five-minute run cleaned all 10 coffee spills and absorbed 9 of 10 paper pieces.Source: §4.1.2 · p. 21

4.1.3 Results from Phase 3 (Task Fulfillment)

  • 43.18% dialog success in Phase 3 modestly improved on Phase 2’s 35.7%, though commands were misinterpreted and processing stalled.The Phase 3 rate came from one systematically scored interaction; the visitor received requested book information but not its library location, and the farewell was not triggered.Source: §4.1.2 · p. 19, §4.1.3 · p. 21
  • In Phase 3 park cleaning, most misses occurred when the robot avoided people or bottle caps were embedded in the ground.Source: §4.1.3 · p. 22
  • In Experiment 2, with no active disturbances on gravel, the robot collected all distributed trash.Bottle caps remained on the gravel surface rather than being embedded in the ground.Source: §4.1.3 · p. 22
  • In the 41-minute park run, the evaluation counted 23 items, versus 219 in the internal log, which included 206 pre-existing cigarette butts and visually similar leaves.The panel counted only distributed ground-truth items in the evaluation area; the internal log covered the whole littered lawn, illustrating workload rather than detection performance.Source: §4.1.3 · p. 22
  • Underpass cleaning covered 7 m by 18.5 m in 9 minutes (14.5 m2 min−1), and the robot successfully avoided all installed obstacles.Obstacles included humans, luggage, and bins.Source: §4.1.3 · pp. 22–23
  • In the underpass, the robot handled paper pieces and broken glass, and paper cups up to a certain size, but distributed rather than cleaned coffee stains.Figure 11 shows examples of impurities used in the task-fulfillment evaluation.Source: §4.1.3 · p. 23, Figure 11 · p. 23

4.2 Interaction Quality Results

This section examines interaction quality across use cases, tracing evaluation concepts from use-case-specific metrics to a shared guideline and operational testing. It also reports Phase 3 instrument use, including its substantial time demands and that passersby did not apply it themselves.

4.2.1 Results from Phase 1 (Interaction Quality)

  • The library’s Phase 1 concept proposed four dimensions: user experience and usability, social norms, need-oriented interaction, and diversity-sensitive interaction.The workshop emphasized standardized questionnaires and alignment across use cases where feasible.Source: §4.2.1 · pp. 23–24
  • For park cleaning, Phase 1 focused on self-descriptive capability and existence acceptance to assess whether passersby understood the robot and accepted its presence.Self-description concerned conveying the robot’s role, task, and operational state; existence acceptance addressed attitudes toward a robot’s mere presence.Source: §4.2.1 · p. 24
  • For underpass cleaning, Phase 1 defined interaction measures around trustworthiness, vulnerable groups’ needs, trajectory disruption, and passerby well-being.The proposed approach included a trust scale, comparisons between user groups, and video analysis of robot trajectories.Source: §4.2.1 · p. 24

4.2.2 Results from Phase 2 (Interaction Quality)

  • Phase 2 produced a shared expert-review guideline intended to reduce reliance on extensive user studies, but library testing exposed long processing times, dialogue breakdowns, and distracting gestures and gaze.Misunderstandings by the language model and gaze directed at bystanders also reduced perceived interaction quality.Source: §4.2.2 · p. 25
  • In Phase 2, park and underpass testing found guideline categories difficult to operationalize or assess; park observations included rapid turns and unclear color-coded status signals.Some guideline formulations were also difficult for non-experts to understand, while several categories were inapplicable in the underpass’s constrained, dynamic setting.Source: §4.2.2 · pp. 25–26

4.2.3 Results from Phase 3 (Interaction Quality)

  • In Phase 3 library testing, network instability and ambient noise disrupted speech recognition, low visitor traffic appeared to increase users’ self-awareness, and the lack of catalog integration limited perceived usefulness.Source: §4.2.3 · p. 26
  • During Phase 3 park deployment, passersby noticed the robot but rarely engaged closely, while panel evaluators used a revised six-category questionnaire with 10-point ratings.Passersby were not asked to complete the questionnaire themselves.Source: §4.2.3 · p. 26
  • In Phase 3 underpass testing, panel feedback identified possible perceptual overload from multiple signals in the confined space, requiring a balance between overload and accessibility.Pedestrians often avoided getting close, partly because the robot’s workspace was marked off.Source: §4.2.3 · pp. 26–27
  • The instrument required substantial time and simultaneous consideration of multiple criteria.Source: §4.2.3 · p. 27
  • As in the other scenarios, passersby did not apply the instrument themselves.Source: §4.2.3 · p. 27

4.3 Safety Results

Safety evaluation was tailored to the three public-space use cases and conducted through Phase 2; the panel stopped safety testing thereafter because the robots had not changed.

4.3.1 Results from Phase 1 concerning Safety

  • The panel found few applicable public-space robot safety standards and noted that public settings can pose hazards beyond ordinary industrial standards.Electrical-safety testing was deemed infeasible in the park, underpass, and library evaluations.Source: §4.3.1 · pp. 27–28
  • Phase 1 park discussions focused on mechanical hazards and camera-based detection, using standardized test pieces to examine human-collision risks and difficult-to-detect objects.The test pieces included child figures, a hiking pole, and a reflective charcoal-grill leg; only the bumper was part of a safety function.Source: §4.3.1 · pp. 27–28
  • For the underpass robot, Phase 1 prioritized laser-scanner detection, the emergency stop, and a stair-edge drop test, using DIN EN IEC 63327 as a core standard.The robot used a laser scanner rather than the park robot’s bumper and camera.Source: §4.3.1 · p. 28

4.3.2 Results from Phase 2 concerning Safety

  • Library safety testing was limited because Pepper was static and already CE-marked; the paper reports no complex test cases were possible or necessary.The tests took place in a laboratory environment.Source: §4.3.2 · p. 28
  • In park tests, objects were detected in most instances, but camera-based collision avoidance was non-certified and teleoperation exceeded the acceptable impact force.The robot passed the emergency-stop test; the bumper was its safety-rated protective device.Source: §4.3.2 · pp. 29–30
  • The underpass robot detected objects and stopped in time, passed the emergency-stop and stair-edge tests, but received no general functional-safety calculation.The tests covered object detection, emergency stop, and stair-edge detection.Source: §4.3.2 · p. 30

4.3.3 Results from Phase 3 Scenario

  • Safety testing ended after Phase 2 because the panel reported that the robots had not changed before Phase 3.Other evaluation categories had Phase 3 tests.Source: §4.3.3 · p. 30

4.4 Results from Economic Viability

Economic benchmarking documented how the toolbox-based process unfolded rather than rating centers comparatively. Across cases, it prompted context-specific adaptations and informed the RoboSpot ecosystem example.

4.4 Results from Economic Viability

  • Economic benchmarking used phase-specific completion criteria to document the process and methods, not to produce comparative ratings of the three competence centers.Reported observations drew on benchmarking reports, on-site sessions, workshops, meetings, and Phase 3 events.Source: §4.4 · p. 30
  • Different levels of prior business-model expertise shaped the depth and trajectory of economic work across the centers and surfaced needs for toolbox refinement.Source: §4.4 · p. 30

4.4.1 Cross-Case Observations

  • Generic business-model templates needed contextual adaptation, including Human-Robot Journey extensions for interaction and a Non-Profit Business Model Canvas variant for public-sector value logic.The resulting feedback informed Target Pricing, Target Costing, and revised collaborative tool guides.Source: §4.4.1 · p. 31
  • The analysis shifted from individual applications toward center stabilization and, in one case, platform-based scaling questions.Source: §4.4.1 · p. 31

4.4.2 Application-Oriented Operationalization of the Toolbox within the RoboSpot Ecosystem - a RuhrBots Example

  • The RoboSpot example operationalized the toolbox as methodological guidance within a platform-based B2B business model for social robotics adoption.Its components included the Human–Robot Journey, stakeholder network analysis, AI-augmented business-model scenarios, and a tested MVP.Source: §4.4.2 · p. 31

5 Discussion of Lessons Learned

The discussion finds that cross-case benchmarking can surface context-specific requirements and trade-offs, but comparability is constrained by divergent applications, limited samples, and the scope of testing. It proposes a shared procedure with use-case-specific metrics and coordinated evaluation categories.

5 Discussion of Lessons Learned

  • Interdisciplinary benchmarking involved stakeholders with different goals and expertise, while organizational constraints limited direct cross-case comparability and long-term testing.The cases targeted TRL 6–7 everyday-life scenarios; long-term tests were out of scope.Source: §5 · pp. 32–33
  • Park cleaning under realistic conditions encountered bottle caps embedded in grass, while library task fulfillment was comparable, illustrating context-specific challenges that uniform schemes may miss.Source: §5 · p. 32
  • Unifying task-fulfillment metrics across use cases with different foci remained impractical, and the benchmarking had a limited sample rate because it emphasized cross-case analysis and objective metrics.Interaction-centric measures such as dialog success rates require more systematic evaluation.Source: §5 · p. 32
  • The interaction guideline was too generic and extensive for standardized use without substantial adaptation, and interaction assessment remained predominantly subjective and context-shaped.Systematic integration of objectifying HRI methods was not feasible within the events’ limited time frames.Source: §5 · pp. 32–33
  • Safety benchmarking faced the absence of a public-everyday-interaction standard and gaps in test objects for children, walking aids, and hard-to-detect objects; testing stopped after Phase 2.The panel and deployers struggled to benchmark safety beyond stairs and obstacle detection in outdoor scenarios.Source: §5 · p. 33
  • Economic considerations were often addressed implicitly or late, while cases surfaced tension between inclusive, useful solutions and sustainable costs.The discussion calls for structured, phase-specific guidance that integrates economic viability early.Source: §5 · p. 33
  • The three-phase experiment made tensions among task fulfillment, interaction quality, safety, and economic viability visible, with economic viability connecting use-case design, concept sustainability, and scalability.The authors caution that optimizing one level can come at the expense of long-term impact.Source: §5 · p. 33
  • The proposed revision retains four coordinated categories and three phases, but uses shared procedures and use-case profiles to justify metrics; safety remains mandatory and cannot be offset by other performance.Direct quantitative comparisons should be limited to sufficiently similar cases.Source: §5 · p. 33

6 Conclusion

The paper evaluates benchmarking service robots under realistic deployment conditions at TRL 6–7 and finds that heterogeneous use cases yield interdisciplinary insights rather than directly comparable quantitative results. It proposes coordinated evaluation categories and use-case-specific metrics, while noting that the proposal needs validation beyond its three use cases.

6 Conclusion

  • The study examined service-robot benchmarking at TRL 6–7 across multiple public real-world scenarios and complementary evaluation dimensions.It moved beyond laboratory-only comparisons to evaluate robots under realistic deployment conditions.Source: §6 · p. 34
  • No directly comparable quantitative results emerged across the three heterogeneous use cases, but the process yielded meta-level insights into interdisciplinary benchmarking.These insights concerned how task fulfillment interacts with solutions for interaction quality, safety, and economic viability.Source: §6 · p. 34
  • Real-world benchmarking outside competitive settings proved a highly insightful alternative and surfaced interdisciplinary challenges not confined to one use case or robot platform.The process required considerable travel and time but produced meaningful, experience-based learnings.Source: §6 · p. 34
  • The proposed framework keeps task fulfillment, interaction quality, safety, and economic viability distinct but coordinated, selecting and documenting metrics through a shared use-case profile rather than fixed sets.This supports process-level comparison across heterogeneous deployments while reserving direct quantitative comparison for sufficiently similar cases.Source: §6 · p. 34
  • Safety remains mandatory and cannot be offset by performance in other categories; higher-TRL benchmarking should provide deployment-specific evidence and trade-offs rather than a universal score.Source: §6 · p. 34
  • Because the proposal derives from three use cases, it requires validation in further deployments.Source: §6 · p. 34

7 Ethics Statement

The reported benchmarking events took place within three research projects, each with its own ethics process for studies and field activities involving human participants. The text describes participant information and observation practices across the library and outdoor evaluations, as well as limits on collected and published personal data.

7 Ethics Statement

  • The three projects each followed their own ethics process covering studies and field activities involving human participants.Source: §7 · p. 34
  • Library visitors were informed by signs, while outdoor tests used cordoned-off areas or invited participants; uninvolved passersby were not systematically observed or recorded as individuals.Source: §7 · p. 34
  • The benchmarking collected no personal data beyond scored interaction protocols, and faces in published photographs were blurred.Source: §7 · p. 34
Loading 2609.26490v1…