Source-linked AI summary
Module Number Adaptive Visual Shape Control for Serial Modular Soft Robots
Kyohei Akamine, Takato Horii, Yusuke Sakaue, Hiroki Ishizuka
TL;DR
Existing image-based control methods are typically tied to fixed robot structures, creating a need for new control data when module numbers change. The paper reuses a single-module controller by decomposing whole-body images into local patches and applying that controller across modules. Physical experiments show shape-error reduction for one- to five-module robots, supporting scalable markerless whole-body control without configuration-specific training.
Problem
Existing image-based control methods make visual representations configuration-dependent, limiting reuse when module positions or counts change.
Method
A single-module visual controller is reused across multi-module robots by extracting local patches, applying geometric augmentation, and reconstructing missing actuator-mask channels.
Results
Shape errors decreased across all tested configurations, including five-module robots, with final normalized MSE of 0.33±0.01 in repeated five-module trials.
Takeaways & Limitations
Single-module control learning supports scalable markerless whole-body control across one- to five-module serial soft robots without configuration-specific controller training.
Abstract
from arXiv · showhide
Image based shape control provides a simple means of controlling the whole body configuration of soft robots. However, existing data driven approaches are typically developed for fixed robot structures and require new control data when the number of modules changes. This paper presents a module number adaptive visual shape control method for serial modular soft pneumatic robots. A controller trained only on single module actuation shape data is reused for robots with one to five modules by decomposing whole body camera images into local module patches. A single common module segmenter localizes individual modules across all tested configurations, while the same local controller is applied to every extracted patch. Geometric data augmentation improves transferability to downstream modules, and a lightweight mask reconstruction network reconstructs a synthetically removed actuator mask channel. Experiments on physical robots demonstrate shape control across varying numbers of modules and under environmental changes and payload loading. The results show that single module control learning enables scalable whole body control without configuration specific control data collection.
I. INTRODUCTION
The paper addresses the difficulty of reusing image-based control when serial modular soft robots change configuration. It proposes composing a single-module visual controller across modules and validates transfer across one- to five-module robots.
- Conventional data-driven controllers learned for entire robots are difficult to reuse when modules are added or removed.
- Whole-body shape control is needed because end-effector position alone does not specify safe configurations for deformable robots in constrained environments.
- The proposed method decomposes current and target robot images into module patches and applies one controller trained only on single-module visual and actuation data.
- Geometric augmentation reduces position-dependent visual distribution shifts, while mask reconstruction restores features lost through overlap or occlusion.
- Experiments validate physical shape control across different module numbers and environmental conditions, including lighting, background disturbances, and payload loading.
- The study contributes a compositional controller transferred from single-module data to one- through five-module configurations without additional multi-module control-data collection or retraining.
II. SOFT ROBOT DESIGN
The robot uses serial modules built from three pneumatic artificial muscles, and the control architecture extracts module patches before independently updating each module’s commands.
- Each module contains three McKibben-type pneumatic artificial muscles arranged on an equilateral triangular frame for three-dimensional deformation.
- The module extractor localizes N modules and generates one image patch for each detected module.
- The module controller estimates actuator shapes and computes next-step voltages from current masks, target masks, and applied voltages.
- The same controller is independently applied to every module patch, enabling deployment to multi-module configurations without additional controller training.
- A reconstruction network restores a missing actuator mask when visual information is lost through occlusion or overlap.
A. Module Extractor
The module extractor detects module regions in the whole-body image, orders them vertically, and crops corresponding module patches for control.
- The module segmenter identifies module regions and their bounding boxes in the full robot image.
- Each detected bounding box is represented by its upper-left and lower-right coordinates.
- Module boxes are sorted by ascending center y-coordinate, assigning indices from the robot’s top to bottom.
- Each ordered region is cropped and interpolated into the surrounding background to reduce background noise, producing the module patch set.
2) Training:
The training and control pipeline segments actuator masks, canonicalizes their spatial order, combines current and target masks with normalized voltages, and predicts bounded voltage updates.
- The actuator segmenter returns candidate masks and confidence scores, discarding small or low-confidence masks before retaining the three highest-scoring masks.
- Retained actuator masks are reordered by centroid x-coordinate so channels consistently represent left, center, and right actuators.
- The same mask-extraction procedure processes current and target patches to construct current and target actuator-mask representations.
- Current masks, target masks, and a spatially broadcast normalized voltage map are concatenated into a nine-channel CNN input tensor.
- The voltage model estimates a three-dimensional command voltage from the nine-channel tensor and applies a first-order update with gain K = 0.5.
- The updated voltage is clipped to [0, Vmax]3 before being returned as the next-step command.
2) Training:
Training uses actuator-segmentation data and single-module voltage–mask pairs, with geometric augmentation and paired samples supporting voltage prediction.
- The actuator segmenter was fine-tuned on annotated images from extracted patches and randomly posed single-module configurations.
- 1331 voltage–mask image pairs were collected by independently applying 0.0–5.0 V muscle voltages in 0.5 V increments.
- Each voltage–mask sample was augmented 20 times with random rotations and translations to emulate downstream-module displacement.
- Training paired anchor and partner samples, using the partner voltage as the supervisory target and Smooth L1 loss optimized with Adam.
C. Mask Reconstruction Network
A lightweight U-Net reconstructs a missing actuator mask channel from the other mask channels and the current voltage vector, producing a binary mask for control.
- The reconstruction network addresses missing mask channels caused by illumination changes or overlap between adjacent muscles.
- For any missing channel, the other two mask channels serve as inputs and the missing channel serves as the training target.
- For the representative left-channel case, the network takes non-missing center and right masks together with the current voltage vector as inputs.
- The network outputs a left-mask logit map, which sigmoid conversion and a 0.5 threshold turn into a reconstructed binary mask.
2) Training:
Training uses image-based visual feedback to control the robot, while assuming a physically feasible visual target is already provided. Evaluation compares target and current RGB shapes primarily through image-space MSE.
- Training: The controller iteratively updates actuator voltages according to differences between current and target states.A higher control frequency would require modeling actuator transient dynamics and scheduling later commands.
- Training: The system captures the robot shape and determines pneumatic pressure for each actuator from the observed shape.The robot is suspended in an enclosed experimental setup and viewed by a webcam from 3.3 m.
- Training: RGB-image MSE is the primary evaluation metric for comparing target and current shapes.Additional evaluations examine overlaid shapes, temporal error evolution, and convergence behavior.
- Training: The study assumes that a physically feasible visual target state is provided rather than generating arbitrary targets from sketches or high-level planners.The focus is reusing a single-module-trained visual controller across different module counts without additional controller training.
B. Fundamental Characteristics
A controller trained exclusively on single-module voltage–mask data was evaluated on robots with different module counts and target postures. The robot converged toward target shapes across configurations, while errors increased with longer chains and were reduced by data augmentation.
- B. Fundamental Characteristics: Single-module training was tested on one-to-five-module robots using five target postures: twisted, forward, rightward, leftward, and backward.The multi-module target images were excluded from controller training.
- B. Fundamental Characteristics: For all module configurations, image error decreased substantially and generally converged within approximately 20 s.The shaded regions represent standard deviation across the five target postures.
- B. Fundamental Characteristics: The five-module robot reached a final normalized MSE of 0.33±0.01 across five trials for the same twisted target posture.The small inter-trial variation indicated low trial-to-trial variability for that tested posture.
- B. Fundamental Characteristics: Mean error and standard deviation increased with module count, indicating accumulated control errors in longer configurations.The image error nevertheless decreased for all five target postures.
- B. Fundamental Characteristics: Data augmentation suppressed the increase in final error associated with increasing numbers of modules.The augmentation emulated apparent displacement of downstream modules during single-module training.
- B. Fundamental Characteristics: Under altered illumination, nearby obstacles, and synthetic image-plane translation and rotation, error converged to a level comparable to normal conditions.The five-module robot was used because it was expected to be most susceptible to environmental variations.
C. Payload Loading Experiment
Payload and mask-loss experiments tested robustness beyond the standard multi-module evaluation. Visual feedback maintained convergence with a 500g payload, while synthetic single-channel mask loss preserved comparable tracking performance.
- C. Payload Loading Experiment: With a 500g mass attached to the tip of the five-module robot, the controller converged to shapes close to target postures.The experiment evaluated five target postures, and image error converged within approximately 20 s.
- C. Payload Loading Experiment: With payload loading, mean final image error remained comparable to the no-payload condition despite increased variation across target postures.The payload changed target-dependent control behavior and increased the standard deviation of image error.
- C. Payload Loading Experiment: Synthetic removal of one estimated mask channel was used to isolate whether reconstruction could prevent control failure when required visual information was unavailable.The experiment did not reproduce the full image-level effects of physical occlusion.
- C. Payload Loading Experiment: The reconstruction network successfully restored the missing mask, while tracking performance remained comparable to normal conditions.This result demonstrates robustness against synthetic single-channel mask loss.
V. DISCUSSION
Single-module visual control knowledge transferred across serial robots with one to five modules, although longer configurations showed increased steady-state error. Geometric augmentation supported transfer, while load distribution and monocular depth ambiguity limited performance.
- Scalability of Single-Module Learning: The five-module robot reduced final image error to approximately 40% of its initial value, demonstrating convergence despite increased structural length.
- Scalability of Single-Module Learning: Single-module control knowledge transferred to larger serial configurations, supporting whole-body control through independent application of the same visual controller to each module patch.The study evaluates reuse across increasing module counts rather than maximizing accuracy for one configuration.
- Scalability of Single-Module Learning: Geometric rotation and translation augmentation reduced dependence of voltage prediction on total module number by representing downstream positional and orientational variations.Generalization depended on the geometric variations represented in single-module training data, not only on controller architecture.
- Factors Limiting Longer Configurations: Steady-state error increased for four- and five-module configurations, although the overall target shapes were still reproduced.The identified limiting factors were changing load distribution and loss of depth information in monocular images.
- Factors Limiting Longer Configurations: Increasing module count increases base-actuator load and gravitational moment, creating discrepancies between single-module pressure–deformation learning and longer-configuration behavior.
- Factors Limiting Longer Configurations: Monocular images cannot fully distinguish three-dimensional deformation, so depth-driven mask scale changes may induce inappropriate voltage corrections.The controller estimates voltage from mask shape, size, and position.
- Factors Limiting Longer Configurations: Residual image-control errors also arise from hysteresis, material nonlinearities, and visual measurement errors shared by monocular multi-actuator soft-robot control.
C. Practical Implications and Future Extensions
The framework can recover missing actuator-mask information and maintain target tracking, while future extensions address representation, target feasibility, load compensation, and three-dimensional visual variation.
- Practical Implications and Future Extensions: Morphology-independent features such as robot silhouette or centerline could extend the method beyond actuator-specific masks to other soft-robot designs.
- Practical Implications and Future Extensions: The method assumes a physically feasible target image; future systems may add sketch interfaces, geometric models, planners, or generative target-image models.
- Practical Implications and Future Extensions: Load compensation based on module number and robot weight could reduce accumulated steady-state error in longer configurations without additional control data.
- Practical Implications and Future Extensions: Current augmentation uses only two-dimensional rotation and translation, so it does not reproduce depth-induced apparent shortening or scale changes.Viewpoint-aware synthesis could better represent three-dimensional visual variation.
- Practical Implications and Future Extensions: Reconstructing a missing actuator mask restored visual features and maintained tracking performance comparable to normal conditions.Without reconstruction, the robot hardly moved and did not reach the target posture; with reconstruction, image error decreased over time.
- Practical Implications and Future Extensions: Experiments demonstrated shape-error reduction across one- to five-module configurations, environmental changes, payload loading, and synthetic loss of one actuator-mask channel.The results support markerless whole-body control without configuration-specific controller training.