PaperlayerView exact revision on arXiv

Source-linked AI summary

Extending Affordance Segmentation for Wearable Robots to Multi-Object Images

Simone Lugani, Matteo Montani, Paolo Gastaldo, Rodolfo Zunino, Edoardo Ragusa

arXiv:2610.29474v1cs.CVcs.RO

TL;DR

Vision-based prosthetic pipelines can support grasp selection, but target selection remains challenging when multiple objects are present. The paper adds a laser pointer and mixed-reality training for affordance segmentation; reported accuracy falls by 7–9% without the pointer in the highlighted test condition. Validation remains limited to controlled and mixed-reality scenarios.

  • Problem

    Portable RGB-based affordance-segmentation pipelines still lack target disambiguation for multi-object scenes while supporting graspable-part segmentation.

    Source: §II · p. 1, §I · p. 1

  • Method

    The approach combines an EMG-activated laser pointer for explicit target selection with mixed-reality training for the affordance-segmentation module.

    Source: §I · p. 1, B. Training the AS module with a mixed… · p. 2

  • Results

    Accuracy drops by 7–9% without the pointer in the highlighted test condition, while accuracy stays high when it is present.

    Source: MR MR-IIT MR-UMD MR-IIT-UMD · p. 3

  • Takeaways & Limitations

    The proposal enables target disambiguation in portable vision-based prosthetic systems while preserving a lightweight pipeline.

    Source: §V · p. 4

    Validation is limited to controlled and mixed-reality scenarios; real deployments may face visibility and user-related challenges.

    Source: §V · p. 4

Abstract

from arXiv · show

This paper proposes a refinement to existing vision-based portable pipelines for the recognition of graspable object parts. Such pipelines can effectively support automatic grasp selection and preshaping in upper-limb prosthetic devices. However, realistic scenarios in which multiple objects are present still represent a challenge. The proposed method exploits a minor modification to the conventional hardware setup and a custom training approach to extend the applicability of vision-based pipelines. A laser pointer is introduced in the setup as an effective mechanism for object disambiguation. Accordingly, a suitable retraining of the deep convolutional neural networks supporting recognition of graspable parts is implemented. A mixed reality strategy allows the reuse of available high-quality segmentation masks to generate realistic images procedurally, extending the range of scenarios supported by the final device. Experimental results confirm that the proposed pipeline improves target disambiguation accuracy by up to 9% compared with existing solutions, while preserving the computational cost.

I. INTRODUCTION

Vision-based prosthetic control can support grasp selection, but multiple objects make target selection ambiguous. The paper adds a laser pointer and retrains an affordance-segmentation module to indicate the intended target while preserving an efficient pipeline.

I. INTRODUCTION

  • Multiple objects make target selection ambiguous in vision-based systems that rely on users to frame the intended object.These systems support automatic grasp selection and preshaping.Source: §I · p. 1
  • An EMG-activated laser pointer explicitly marks the intended target for a lightweight neural affordance-segmentation module.The module identifies the marked object and extracts its graspable regions.Source: §I · p. 1
  • The proposed approach aims to improve target-disambiguation robustness in multi-object scenes while preserving embedded deployment efficiency.Experiments are reported to show accurate target selection and enhanced performance in challenging multi-object scenarios.Source: §I · p. 1

II. BACKGROUND

Prior work explores sensing and affordance segmentation for semi-autonomous prosthetic control, but target disambiguation remains open for portable RGB-based pipelines. Existing embedded AS architectures assume a clearly framed single object, unlike the direct pointer-based selection proposed here.

II. BACKGROUND

  • Prosthetic-control pipelines integrate teleceptive sensing, including RGB cameras, radars, and depth sensors, with autonomous control.Depth-sensor integration increases computational cost and system complexity through modality fusion and calibration requirements.Source: §II · p. 1
  • AS methods identify possible interactions with object parts, and portable approaches have added DNN modules and sensory inputs to support wearable-robot control.The portable method in was designed to benefit from human interaction; demonstrated real-time operation on high-end MCUs.Source: §II · p. 1
  • Target disambiguation remains open for portable RGB-based affordance-segmentation pipelines that must support both target selection and graspable-part segmentation.Existing approaches may rely on implicit cues, accurate framing, extra sensors, or demanding perception modules.Source: §II · p. 1
  • Embedded-accelerator AS architectures assume clear framing of a single object, whereas this work introduces direct target selection with a laser pointer.The authors identify,, and as the only such architectures to their knowledge.Source: §II · p. 2

A. Sensing system

The sensing system uses affordance segmentation to identify graspable parts and support hand configuration. A hand-mounted laser pointer marks the target so the system can distinguish it in multi-object scenes.

A. Sensing system

  • Affordance segmentation turns the camera's RGB frame into a mask of graspable parts, such as a handle, for descriptor extraction and hand configuration.The mask provides relevant object regions without explicit user instruction.Source: A. Sensing system · p. 2
  • After the user triggers the system, the hand-mounted pointer is centered on the target and used by the segmentation module to identify that object.The setup extends portable systems to scenes that are challenging when multiple objects are present.Source: A. Sensing system · p. 2, Figure 2 · p. 2
  • Figure 3 illustrates real IIT scenes with synthetic laser annotations and UMD mixed-reality scenes.These examples show the kinds of multi-object settings the pointer-based setup is intended to address.Source: Figure 3 · p. 2, A. Sensing system · p. 2

B. Training the AS module with a mixed reality approach

The training approach combines available segmentation masks with realistic backgrounds to create mixed-reality scenes, while synthetic patterns approximate the pointer's light spot. Table I describes foreground-image performance comparisons across models with different pointer placements.

B. Training the AS module with a mixed reality approach

  • Synthetic pointer patterns approximate the real light spot and its smooth intensity gradients for training the AS module to focus on the user's target.Standard AS datasets lack pointer annotations, and many use viewpoints unlike cameras mounted on robots.Source: B. Training the AS module with a mixed… · p. 2
  • UMD object masks are used to isolate objects, which are then placed on random realistic backgrounds to create mixed-reality scenes.IIT-Aff provides semantic segmentation and bounding boxes, including multi-object scenes.Source: B. Training the AS module with a mixed… · p. 2

MODELS

  • Table I covers foreground-image performance for models with different pointer placements.The caption identifies the comparison scope but does not provide specific performance values.Source: Table I · p. 3

IV. EXPERIMENTS

The experiments use a lightweight affordance-segmentation network and a controlled training and testing procedure.

A. Setup

  • The affordance-segmentation DNN requires 638 MFLOPs and 3.6 M parameters, with 80 ms inference latency and 6 W power consumption on a Jetson Nano.Training and testing used an RTX 2080 Ti workstation; testing data were excluded from parameter and hyperparameter tuning.Source: A. Setup · p. 3
  • Networks were trained for 100 epochs with learning-rate scheduling, early stopping, weighted cross-entropy, and geometric and color augmentation.Class weights were inversely proportional to class frequencies.Source: A. Setup · p. 3
  • The lightweight DNN had previously been embedded in affordance-segmentation pipelines.The passage identifies its computational requirements and Jetson Nano performance as the basis for its embedded deployment profile.Source: A. Setup · p. 3

B. Single object images

The single-object experiments test how pointer placement during training affects accuracy across test sets.

B. Single object images

  • The experiment compares four pointer-training configurations across same-domain and three other-domain test sets.The configurations are No P, P OBJ, P BG, and P GP, with P GP placing the pointer on the graspable part.Source: B. Single object images · p. 3
  • 95.69% average accuracy is achieved by the configuration trained without pointers.When trained with a pointer on the graspable part, the model improves slightly on its corresponding test set but loses accuracy on other test sets.Source: B. Single object images · p. 3
  • Pointer placement on the object or nearby background is ignored, leaving accuracy almost constant across test sets.The background-pointer condition tests whether the network uses a pointer that does not indicate the object.Source: B. Single object images · p. 3

C. Multiple object images

The multi-object experiments evaluate models trained on different dataset mixtures and test how target-pointer presence and placement affect accuracy.

C. Multiple object images

MR MR-IIT MR-UMD MR-IIT-UMD

  • Four models use the proposed mixed-reality dataset alone or mixtures with IIT, UMD, or both datasets.Only 20% of UMD was included, and every training image contained a pointer on the target object.Source: MR MR-IIT MR-UMD MR-IIT-UMD · p. 3
  • 7–9% accuracy drops without a pointer in three-object images, and accuracy decreases sharply when the pointer is randomly placed.Models average above 90% accuracy on their respective test sets; with the pointer on the target, accuracy remains high.Source: MR MR-IIT MR-UMD MR-IIT-UMD · p. 3, Table II · p. 3

D. Pointer shape and color

The pointer-appearance analysis tests whether models trained with one virtual pointer remain accurate with variations in pointer color and shape.

D. Pointer shape and color

  • Four virtual pointer versions are tested after training with the virtual pointer used in the earlier setups.This evaluates the difference between a virtual pointer and a real-world image appearance.Source: D. Pointer shape and color · p. 3
  • The evaluation compares the training pointer with single-color Red, Green, and Blue pointers and a patterned Star pointer.The testing set always contains three objects per image, with the pointer on the target object.Source: D. Pointer shape and color · p. 3
  • Models remain robust to synthetic pointer color and shape variations, including testing with a pointer color different from training.The authors suggest color augmentation during training likely supports this robustness.Source: D. Pointer shape and color · p. 3, Figure 4 · p. 4

V. CONCLUSIONS

The system uses mixed-reality training to select targets from a laser pointer in multi-object scenes and remains robust to changes in pointer color and shape. The approach preserves a lightweight pipeline, but validation remains limited to controlled and mixed-reality scenarios.

V. CONCLUSIONS

  • Mixed-reality training lets the model select target objects from the laser position, with robustness to pointer color and shape variations.The authors report that the model exploits the pointer to select targets.Source: §V · p. 4
  • The approach preserves a lightweight portable pipeline by limiting the hardware change to adding a small laser pointer.Source: §V · p. 4
  • Validation is limited to controlled and mixed-reality scenarios; future work should test real operating conditions, usability, and cognitive load.Ambient illumination, transparent or specular surfaces, and user-related factors may affect laser visibility.Source: §V · p. 4
Loading 2610.29474v1…