Source-linked AI summary
A Confidence-Aware Multimodal Fusion Framework for Industrial Human-Robot Collaboration
Xinyu Liu, Qiqi Dong, Boya Jia, Yi Zhang, Binbin Lian
TL;DR
Industrial human–robot collaboration needs reliable intention prediction despite uncertain, noisy, and dynamically changing sensor inputs. CAMF fuses four modalities with confidence-aware temporal fusion and balanced learning, achieving strong recognition performance and environmental adaptability while remaining vulnerable to extreme occlusion.
Problem
Complex industrial environments degrade multimodal sensor reliability, while existing systems lack dynamic fusion, low-confidence noise suppression, and proactive intention prediction.
Method
CAMF fuses object 6D pose, gaze, skeletal motion, and IMU-based hand motion using confidence-guided BiLSTM fusion, balanced learning, and temporary freezing of low-confidence modality updates.
Results
CAMF achieves 91.86% intention recognition accuracy and higher overall performance and stability than existing multimodal fusion approaches.
Takeaways & Limitations
The framework maintains reliable performance under mild industrial interference and supports stable human–robot coordination in practical collaborative operations.
Takeaways & Limitations
Extreme upper-body or full-body occlusion can sharply reduce valid features and confidence, causing severe degradation and occasional recognition errors.
Abstract
from arXiv · showhide
A confidence-aware multimodal fusion framework (CAMF) is proposed to realize reliable human intention prediction for industrial human-robot collaboration. This framework fuses four heterogeneous modalities including object 6D pose, gaze, skeletal motion and IMU-based hand motion. It embeds a confidence-trend-driven dynamic fusion mechanism into BiLSTM to adaptively balance bidirectional temporal features according to real-time modality reliability. A confidence-guided balanced learning strategy combined with a confidence freezing mechanism is further adopted to adjust network gradients dynamically, suppress noise from low-quality modalities and mitigate cross-modal learning bias. A physical platform based on the UR3 collaborative robot is built for experimental validation. Comparative results show that the proposed method reaches an intention recognition accuracy of 91.86% and outperforms existing multimodal fusion approaches in overall performance and stability. It also maintains satisfactory accuracy under low light and partial occlusion interference. In practical assembly tasks, the framework enables proactive and stable human-robot cooperation with strong environmental adaptability.
I. INTRODUCTION
Industrial human–robot collaboration requires multimodal, uncertainty-aware intention understanding because complex environments degrade individual sensor reliability. CAMF addresses this need by combining four human-related modalities with confidence-guided fusion and balanced learning.
- Motivation: Complex industrial scenarios require integrating body movement, gaze, applied force, and equipment interaction because unimodal perception cannot capture this context.
- Motivation: Multimodal fusion exploits cross-modal complementarity and redundancy to improve human-intent recognition accuracy and robustness.
- Research gap: Existing systems remain vulnerable to lighting, reflections, occlusions, and electromagnetic interference, while lacking mechanisms to suppress unreliable modalities during training.
- Proposed framework: CAMF fuses object 6D pose, gaze, skeletal motion, and IMU-based hand motion using real-time confidence estimates to regulate training and inference.
- Proposed framework: Its balanced multimodal learning mechanism adaptively adjusts modality contributions during training to alleviate performance bias.
2) Gaze Recognition:
The framework extracts gaze and skeletal-motion features from visual input using dedicated perception modules. Gaze reliability is derived from gaze estimation outputs, while skeleton processing uses graph convolutions and motion-rationality confidence.
- Gaze Recognition: Gaze-LLE extracts a gaze point and confidence from scene RGB images and a MediaPipe-detected head bounding box.
- Gaze Recognition: A frozen DINOv2 backbone, head-position prompting, and a Transformer decoder produce a gaze probability heatmap.
- Gaze Recognition: The gaze point is selected from the heatmap, while its maximum probability contributes to position reliability.
- Skeletal Motion Recognition: ST-GCN preprocesses OpenPose 3D skeleton data by representing joints and edges as a spatial graph with neighborhood partitions.
- Skeletal Motion Recognition: Motion-rationality confidence evaluates skeleton reliability using joint spatial consistency and physiological constraint satisfaction.
4) Hand Motion Recognition:
Hand-motion recognition combines inertial sensing, two-stage attitude estimation, and confidence measures based on prediction uncertainty and directional reliability.
- A 9-axis IMU records hand acceleration, angular velocity, and magnetic-field measurements for motion analysis.
- The prior attitude is predicted by integrating angular velocity with the previous attitude estimate.
- Acceleration measurements correct pitch and roll during the first filtering stage.
- Magnetometer measurements correct yaw during the second filtering stage.
- Hand-motion confidence combines prior error covariance with directional reliability.The direction uses gravity-compensated acceleration and the vector from the hand toward candidate target objects.
B. Confidence-Weighted Multimodal Feature Fusion
The framework encodes each modality with BiLSTM and fuses temporal and multimodal features using confidence-aware weighting. Confidence trends dynamically balance forward and backward temporal context, while modality confidence weights the final aggregation.
- Each modality’s features and confidence values are processed by an individual BiLSTM for temporal encoding.The encoded representations are then passed to fully connected layers.
- BiLSTM models temporal dependencies in opposite directions, capturing both past-to-current and future-to-current transitions.
- Temporal Context Fusion with Confidence Trend Guidance: Fixed bidirectional weighting cannot adapt to changing sequential characteristics, motivating confidence-trend-driven dynamic weighting.
- Temporal Context Fusion with Confidence Trend Guidance: Confidence trends are normalized with Softmax to generate adaptive weights for forward and backward LSTM outputs.When the forward trend is greater, the model emphasizes historical context; when the backward trend is greater, it prioritizes future context.
- Adaptive Multimodal Feature Aggregation: Confidence-weighted modality features are aggregated in fully connected layers, prioritizing more reliable modalities.Each modality-specific hidden state is multiplied by its confidence before fusion.
C. Multimodal Balanced Intention Prediction
The balanced-learning mechanism addresses dominance by high-confidence modalities through contribution-aware gradient scaling and confidence-based freezing of low-reliability updates.
- High-confidence modalities can dominate multimodal learning, suppressing weaker complementary modalities and degrading generalization.
- The optimization updates incorporate each modality’s scaling coefficient into the gradient step.
- A contribution discrepancy ratio measures each modality’s contribution to the learning objective.
- Gradient scaling coefficients adaptively regulate modality contributions during stochastic gradient descent.When a modality’s contribution is below the specified condition, its gradient updates are suppressed.
- Confidence Freezing Mechanism: A confidence threshold temporarily freezes gradient updates for modalities whose confidence falls below the threshold.This mechanism suppresses interference from low-reliability modalities during training.
III. EXPERIMENT
The experiments evaluate CAMF’s feasibility, robustness, and practicality through system configuration, dataset construction, ablations, method comparisons, and collaborative case tests.
- The evaluation combines quantitative and qualitative analyses across hardware and software configuration, dataset construction, ablation studies, comparative methods, and human-robot collaboration cases.
- Ablation experiments assess the effectiveness of confidence-driven fusion and multimodal balanced learning.
A. Experimental Setup
The experimental setup combines synchronized multimodal sensing with a UR3 collaborative robot to support intention recognition, safe grasping, and workpiece delivery. Dataset construction uses participant trials and staged data acquisition.
- Hardware and software configuration: The platform integrates a RealSense D435if camera, HI13R3 IMU, UR3 manipulator, and Robotiq 2F-85 gripper for multimodal collaboration.The camera provides scene, gaze, and skeletal data, while the IMU captures hand motion and orientation.
- Robot system: The UR3 identifies target workpieces with YOLOv11, extracts masks, generates grasps with GraspNet, and executes feasible grasping under workspace constraints.The pipeline begins with point-cloud capture and ends with gripper-controlled stable grasping.
- Robot safety: A gate trajectory avoids the worktable, assembly components, and user, while external torque perception provides an auxiliary collision-safety measure.The torque estimate is derived from joint torque through a momentum-observer-based dynamic model and Jacobian mapping.
- Collaborative task: The completed pipeline links intention recognition, target grasping, and delivery to the user’s operation space for collaborative assembly.Unexpected contact can trigger collision avoidance or emergency-stop responses during trajectory execution.
- Dataset construction: Five participants complete ten trials for each component after training, using a three-stage data acquisition procedure.The setup is illustrated in Fig. 10(a) and Fig. 10(b).
B. Ablation Studies
Ablation studies show that multimodal inputs, confidence-aware fusion, and balanced learning each contribute to recognition performance and robustness. The results also indicate that modality reliability varies across participants and time.
- Modality quantity comparison: 93.42%, 93.37%, and 92.09% are achieved by the proposed multimodal fusion method across the three reported metrics, with an 18% accuracy improvement.The method outperforms the other modality combinations in F1 score and Kappa coefficient.
- Confidence-based fusion: Modality confidence varies substantially across participants and time, with hand motion comparatively stable and gaze or skeletal confidence less stable.These differences motivate adaptive fusion-weight adjustment.
- Confidence-based fusion: 1.7% higher accuracy, 2.0% higher F1 score, and 57.3% lower standard deviation are obtained by CAMF than by the baseline.Removing the BiLSTM confidence-trend module lowers accuracy by 1.56%, while removing FC confidence weighting lowers it by 5.9%.
- Noise robustness: 8% and 3% are CAMF’s accuracy drops under visual and IMU noise, compared with 13% and 10% without the BiLSTM confidence-trend module.Removing FC confidence weighting produces 11% and 5% drops under the same conditions.
- Balanced multimodal learning: 91.72% accuracy, 97.50% mAP, mean F1 0.9172, and F1 standard deviation 0.0136 are obtained at the optimal scaling weight.The scaling weight balances modality prediction scores by preventing strong modalities from dominating optimization.
C. Method Comparison Results
CAMF is compared with multiple multimodal fusion methods across recognition performance, workpiece categories, and video-frame ratios. It maintains strong performance across task conditions while competing methods show larger fluctuations or lower accuracy.
- Overall method comparison: 76.19% accuracy, 75.52% F1 score, and 0.0722 standard deviation make DS the weakest overall method in the comparison.Its fixed fusion rules cannot dynamically evaluate data quality and are easily disturbed by complex multimodal inputs.
- Overall method comparison: 80.94% and 81.34% are the accuracies of Transformer and ECoLaF, while IOP, HRNN, TSLSTM, and PMR range from 86.57% to 87.92%.The compared methods use adaptation gates, late fusion, opinion pooling, recurrent modeling, or modal rebalancing with different limitations.
- Workpiece categories: CAMF exceeds 73% accuracy in most workpiece categories and peaks at 93%, maintaining balanced performance across gear and shaft tasks.DS varies from 8% to 83%, while ECoLaF is more stable but generally less accurate.
- Video frame ratios: 35%, 52%, 67%, 82%, and 89% are CAMF’s accuracies at increasing video-frame ratios, rising steadily as valid frames increase.The comparison evaluates performance under multiple frame-ratio settings.
D. Case Study
The case study evaluates CAMF under low light and occlusion, then tests it in a complete human–robot assembly workflow. Performance remains acceptable under mild interference but declines sharply when human features are largely obscured.
- Interference Evaluation: Four interference conditions were tested without supplementary occlusion training: low light, upper-body occlusion, face occlusion, and full-body occlusion.Low light was simulated by reducing the brightness of original RGB images.
- Interference Evaluation: 87.86% accuracy, 88.03% F1 score, and 0.0435 standard deviation were recorded under normal conditions.
- Interference Evaluation: Accuracy declined by 4% in low light and 11% under face occlusion, indicating effective handling of partial occlusion.
- Interference Evaluation: 43.17% accuracy under upper-body occlusion and 30.00% under full-body occlusion marked severe degradation when effective human features were nearly lost.
- Collaborative Assembly: The assembly workflow included intention expression, recognition, robotic grasping, and delivery, with sequential stages illustrated in Fig. 18.The system recognized the target workpiece during the intention-expression phase.
A. Results Discussion
CAMF combines dynamic confidence-aware fusion with balanced training to improve multimodal intention recognition in industrial collaboration. The framework shows environmental adaptability and stable operation, while extreme occlusion and limited online adaptation remain important boundaries.
- Framework and Performance: The framework achieves higher accuracy and smaller deviation than mainstream methods while remaining stable across different workpieces and motion ratios.
- Industrial System: The system integrates visual scene, gaze, skeletal motion, and hand inertial sensing to support high-precision active human–robot collaboration.A ROS2 software framework provides real-time transmission among sensors, workstations, and robots.
- Limitations and Future Work: The system is validated mainly on standard assembly parts and fixed-parameter offline inference, limiting adaptation to special-shaped workpieces, dynamic disturbances, multiple stages, and changing operating habits.Future work proposes feature compensation and online learning for stronger interference and changing users or conditions.
- Framework and Performance: CAMF fuses multimodal sensing with confidence assessment, dynamic fusion rules, and BiLSTM temporal modeling to address changing reliability and noisy data sources.Its learning strategies balance modality contributions and restrain unreliable data during training.