PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Language Models for Occupational Physical Exposure Assessment: Estimating External Hand Forces in Manual Material Handling Tasks from RGB Video

Mohammad Sadra Rajabi, Aanuoluwapo Ojelade, Sunwook Kim, Maury A. Nussbaum

arXiv:2608.22586v1cs.CVcs.AIcs.CL

TL;DR

Continuous external hand-force measurement in manual material handling often relies on instrumented objects or specialized sensing. This study evaluates an RGB-video pipeline using task cues, visual features, and known box mass; it estimates bilateral forces, with handled-object ROIs generally improving estimates.

  • Problem

    Whether RGB video can estimate dynamic, triaxial, bilateral hand forces across manual material handling tasks remains unestablished, despite their importance to occupational exposure assessment.

    Source: §4.0 · p. 21, §5.0 · p. 24

  • Method

    The study evaluated a pipeline combining task-guided ROI detection, pretrained vision-transformer features, temporal regression, and known box mass to estimate bilateral forces from RGB video.

    Source: §4.0 · p. 21, §5.0 · p. 24

  • Results

    RMSE was ~4.7–5.6 N for Fx and Fy and ~10.6–11.0 N for Fz; handled-object ROIs generally improved estimates, while multi-camera capture most clearly benefited peak-force estimation.

    Source: §4.0 · p. 21, §5.0 · pp. 24–25

  • Takeaways & Limitations

    The findings support feasibility of estimating detailed external hand forces from RGB video and known box mass with substantially reduced instrumentation compared with conventional direct or model-based approaches.

    Source: §5.0 · p. 25

    Generalization to diverse worker populations, different handled objects, and workplace conditions was not evaluated; participants were relatively young and healthy, and tasks occurred in a controlled laboratory.

    Source: §4.6 · p. 24

Abstract

from arXiv · show

External hand forces are important inputs to biomechanical analyses of occupational physical exposure and injury risk, yet continuous force measurements during manual material handling (MMH) typically requires instrumented objects or specialized sensing. We evaluated a vision-language model (VLM)-based pipeline that combines task-specific textual cues, visual representations, and known box mass to estimate dynamic, triaxial, bilateral external hand forces from RGB video. Thirty-five healthy young adults performed five MMH tasks involving lifting, carrying, pushing, and pulling with box masses of 6, 9, and 12 kg. The pipeline used text-guided localization of participant and handled-object regions of interest (ROIs), pretrained vision-transformer feature extraction, and transformer-based temporal regression. Performance was evaluated using leave-one-subject-out validation across seven camera-view conditions (three single-view and four multi-view conditions) and four ROI strategies. Overall, root mean square error was ~4.7-5.6 N for the horizontal and mediolateral force components and ~10.6-11.0 N for the vertical component. Including the handled object as a second ROI generally improved force estimation, with some of the largest benefits under single-camera conditions, whereas pixel-level segmentation provided little additional improvement. Multi-camera capture provided the clearest benefit for peak-force estimation, particularly for the vertical component, whereas differences in overall frame-level error among camera configurations were comparatively modest. These findings demonstrate the feasibility of estimating continuous, bilateral, directional hand-force estimates from RGB video and known load mass without requiring sensors on the worker or handled objects as model inputs, supporting the development of more scalable occupational physical exposure and risk assessments.

2.0 Methods … 2.3 Detection and Segmentation of Regions of Interest (ROIs)

The methods use RGB video and instrumented-box force measurements from five manual material handling tasks to develop an offline vision-language pipeline for estimating bilateral hand forces. The pipeline compares four ways of representing participant and handled-object regions of interest (ROIs).

2.1 Overview of the Dataset

  • Thirty-five healthy young adults performed five tasks: symmetric and asymmetric lifting, carrying, pushing, and pulling.The five tasks were selected to represent common occupational activities.Source: §2.1 · p. 4
  • Participants completed conditions spanning 6, 9, and 12 kg box masses, two hand configurations, two lift origins, and two replications.A single wooden box with bilateral triaxial load cells recorded forces; three synchronized Azure Kinect cameras captured RGB video at 30 Hz, and forces were sampled at 200 Hz and aligned with video.Source: §2.1 · pp. 4–5

2.2 Overview of the VLM-Based Pipeline for External Hand Force Estimation

  • The three-stage pipeline detects and segments task-relevant ROIs, extracts visual features, and uses transformer-based regression to estimate frame-level bilateral hand forces.The approach adapts a VLM pipeline previously used to estimate hand distances during symmetric box lifting.Source: §2.2 · pp. 5–6, Figure 1 · p. 6
  • All processing was performed offline on Virginia Tech Advanced Research Computing resources, with no participant videos, images, or derived data processed on external or cloud-based systems.Video handling used OpenCV, and model inference and training used PyTorch.Source: §2.2 · p. 5

2.3 Detection and Segmentation of Regions of Interest (ROIs)

  • GroundingDINO detected participants using task-specific prompts, then localized the handled object within a padded crop centered on the participant when object ROIs were required.Prompts described the participant as lifting, holding, pushing, or pulling a box; the object prompt was “wooden box . crate . holding object”.Source: §2.3 · pp. 6–7
  • Four ROI strategies compared participant-only and participant-plus-object bounding boxes with corresponding SAM-refined pixel-level masks.S1 retained the participant box, S2 retained both boxes, S3 masked the participant, and S4 masked both regions.Source: §2.3 · p. 7, Figure 2 · p. 8

2.4 Feature Extraction from ROIs · 2.5 Feature–Force Pairing and Labeling · 2.6 Transformer-Based Force Prediction

The pipeline converted RGB-frame ROIs into synchronized visual-feature and bilateral force-label sequences, then used a temporal transformer conditioned on load and camera view to predict six force channels. Training used leave-one-subject-out validation, and evaluation measured frame-level accuracy and peak-force reproduction.

2.4 Feature Extraction from ROIs

  • DINOv2 extracted frozen, zero-shot 768-dimensional features from 224 × 224 ROI crops; dual-ROI strategies stacked participant and object features, zero-padding missing object detections.Single-ROI sequences had shape [T, 1, 768], and dual-ROI sequences had shape [T, 2, 768].Source: §2.4 · p. 9

2.5 Feature–Force Pairing and Labeling

  • ROI features were synchronized with load-cell measurements, producing frame-aligned visual sequences and six bilateral regression targets: Fx, Fy, and Fz for each hand.Two research assistants manually identified the start and end frames of each task segment.Source: §2.5 · p. 9

2.6 Transformer-Based Force Prediction

  • The transformer predicted bilateral external hand forces from temporally ordered RGB-derived features, with the model designed to account for dependencies across consecutive frames.The method treats sequences as dynamic rather than estimating force independently in each frame.Source: §2.6 · p. 9, §2.6.1 · p. 9

2.6.1 Model Architecture

  • The model fused ROI features into 256-dimensional frame embeddings, added load, camera-view, and positional context, and used four transformer layers to regress six force channels per frame.Dual-ROI features were concatenated before projection; view-specific sequences were processed independently, and attention masking excluded padded frames.Source: §2.6.1 · pp. 9–10

2.6.2 Data Handling, Model Training, and Validation

  • Sequences were fixed at 300 frames, and leave-one-subject-out validation tested each of 35 participants after training on the other 34.Ten percent of training participants were held out for early stopping; training stopped after 20 consecutive epochs without validation-loss improvement, up to 100 epochs.Source: §2.6.2 · p. 10

2.6.3 Evaluation Metrics

  • Performance was evaluated separately for all six force channels using MAE, RMSE, and Peak Error; Peak Error compared the 95th percentiles of absolute actual and predicted forces.MAE and RMSE measure frame-level accuracy, with RMSE weighting larger individual errors more strongly.Source: §2.6.3 · p. 10

2.7 Statistical Analyses … 3.2 𝑭𝒙: Effects of Camera View Condition and ROI Strategy

Force-estimation errors varied with task and mass, while camera-view and ROI effects on Fx depended on task. Dual-ROI strategies generally reduced Fx errors in Tasks 4 and 5; no camera view had a consistent advantage.

2.7 Statistical Analyses

  • Separate LMMs tested six factors per metric and force channel, limited interactions to three-way, and used log-transformed errors and Tukey HSD post hocs at p < 0.05.Because MAE and RMSE were strongly correlated, only RMSE results are reported in the main text; complete MAE results are in Appendix Tables A.1–A.3.Source: §2.7 · p. 11

3.0 Results

  • Results emphasized RMSE and PE effects involving camera view or ROI strategy, with task and mass effects included as context.Figure 3 illustrates actual and predicted bilateral forces for all three directions and four ROI strategies in a carrying example (Task 3, narrow hand configuration, 6 kg, View 1).Source: §3.0 · pp. 11, 13, Figure 3 · p. 12

3.1 Effects of Mass and MMH Task on Force Estimation Performance

  • RMSE was ~26–36% smaller at 6 kg than 12 kg across all force channels and both hands.PE at 6 kg was ~21–49% smaller than at 12 kg for Fx and Fy, and ~83–87% smaller for Fz.Source: §3.1 · p. 13
  • MMH Task significantly affected RMSE and PE for every force direction and both hands.Fx RMSE and PE were smaller in Tasks 1–3 than 4–5; Fy RMSE and PE were larger in Tasks 2–3 than 1, 4, and 5. Fz RMSE was smaller in Tasks 1 and 3 than 2, 4, and 5; Fz PE was smallest in Task 3, and larger in Task 1 than Task 5.Source: §3.1 · p. 13

3.2 𝑭𝒙: Effects of Camera View Condition and ROI Strategy

  • Dual-ROI strategies (S2, S4) generally reduced Fx RMSE versus single-ROI strategies (S1, S3) in Tasks 4 and 5, with small differences in Tasks 1–3.The camera-view × task interaction produced generally small Fx RMSE differences in Tasks 1–4 and somewhat larger differences (~10–13%) in Task 5, with no camera view consistently advantageous.Source: §3.2 · pp. 13–14, Figure 4 · p. 14, Figure 5 · p. 15
  • Fx PE was typically positive, with small camera-view differences in Tasks 1–3 and larger differences in Tasks 4–5, but no camera view consistently lowered PE.This camera-view × task interaction was significant for both hands.Source: §3.2 · pp. 15–16, Figure 6 · p. 16
  • For the right hand, Fx PE was ~2.9 N with S1/S3 versus ~2.7 N with S2/S4; left-hand PE also favored S2/S4, especially in Tasks 4–5.For the left hand, ROI strategy × camera-view differences in PE were significant but small within each view condition.Source: §3.2 · p. 16

3.3 𝑭𝒚: Effects of Camera View Condition and ROI Strategy · 3.4 𝑭𝒛: Effects of Camera View Condition and ROI Strategy

Camera-view and ROI effects varied by force component, hand, and task. For vertical force, dual-ROI strategies generally reduced RMSE in Tasks 1 and 2, while multi-camera conditions reduced PE most in Tasks 2 and 4.

3.3 𝑭𝒚: Effects of Camera View Condition and ROI Strategy

  • For F_y, camera view significantly affected right-hand RMSE (~4.6–4.8 N across views) and both hands’ PE, but not left-hand RMSE (~5.6–5.7 N).Multi-camera PE was generally smaller than single-camera PE: right hand ~2.3–2.5 N vs. ~2.7 N; left hand ~3.1–3.3 N vs. ~3.3–3.5 N.Source: §3.3 · p. 17
  • For F_y, ROI strategy significantly affected right-hand RMSE (S1 ~4.7 N, S2 ~4.7 N, S3 ~4.8 N, S4 ~4.7 N), but not PE for either hand.ROI strategy interacted with task for both hands’ PE, although differences across tasks were small; PE was ~2.4–2.6 N for the right hand and ~3.2–3.3 N for the left across strategies.Source: §3.3 · p. 17

3.4 𝑭𝒛: Effects of Camera View Condition and ROI Strategy

  • F_z RMSE showed a significant camera-view-by-task interaction for both hands, but view differences were small (~3–11%) and no view consistently outperformed others.Differences tended to be larger in Tasks 2 and 3 and smallest in Tasks 4 and 5.Source: §3.4 · p. 17, Figure 7 · p. 18
  • Dual-ROI strategies S2 and S4 generally reduced F_z RMSE versus single-ROI S1 and S3, especially in Tasks 1 and 2 (~5–7% lower).Differences were smaller and less consistent in Tasks 3 and 5.Source: §3.4 · p. 18, Figure 8 · p. 19
  • F_z PE showed a significant camera-view-by-task interaction for both hands, with larger single-camera than multi-camera PE in Tasks 2 and 4 (~4.6–4.9 N vs. ~1.9–2.3 N).PE was typically positive; view differences were comparatively small in Task 3.Source: §3.4 · p. 19, Figure 9 · p. 20
  • For the right hand, F_z PE differed by ROI strategy across camera views: under single-camera conditions, S2 and S4 had ~20–26% smaller PE than S1 and S3.Differences were small and inconsistent under multi-camera conditions.Source: §3.4 · p. 20

4.0 Discussion … 4.3 Effects of Camera View Condition

The discussion finds that RGB video and known box mass can support continuous, bilateral, triaxial hand-force estimation without sensors on workers or handled objects. It highlights task and mass effects, and finds that camera configuration affects peak-force fidelity more than average frame-level error.

4.0 Discussion

  • ~4.7–5.6 N for F_x and F_y and ~10.6–11.0 N for F_z were the overall RMSEs, supporting non-contact estimation of dynamic bilateral hand forces from RGB video.The F_z errors corresponded to ~21–32% of the static vertical load supported per hand across the 6–12 kg range examined.Source: §4.0 · p. 21

4.1 External Hand Force Estimation Using Different Sensing Modalities

  • Per-hand RMSEs of ~4.7–11 N were broadly comparable to instrumented approaches, while this pipeline used only RGB video and known box mass as model inputs.It provided continuous force estimates independently for each hand and axis, rather than classifying discrete force or load levels.Source: §4.1 · p. 21
  • Unlike prior vision-based studies targeting whole-body kinetics, hand–object interaction forces, load levels, or hand pressure, this work estimates continuous bilateral, triaxial hand forces during whole-body MMH.These external hand forces are inputs to biomechanical analyses of physical exposure during material handling.Source: §4.1 · p. 21

4.2 Effects of Mass and MMH Task

  • ~26–36% smaller RMSE at 6 kg than 12 kg occurred across force channels and hands; PE was ~21–49% smaller for F_x and F_y and ~83–87% smaller for F_z.The increase in error with mass occurred even though box mass was an explicit model input; the authors suggest altered lifting kinematics and posture increased variability in participant–object motion.Source: §4.2 · pp. 21–22
  • F_x errors were smaller in Tasks 1–3 than Tasks 4–5, F_y errors larger in Tasks 2–3, and F_z errors smaller in Tasks 1 and 3 than Tasks 2, 4, and 5.Larger F_x errors during pushing and pulling were also reported with a structurally different approach, suggesting task characteristics may contribute to this pattern.Source: §4.2 · p. 22

4.3 Effects of Camera View Condition

  • ~3–13% RMSE differences across camera conditions were modest, with no single condition best across tasks; in Tasks 2 and 4, F_z PE was ~4.6–4.9 N for single-camera versus ~1.9–2.3 N for multi-camera conditions.PE was actual minus predicted 95th-percentile force magnitude; predominantly positive PE indicated general underestimation of peak-force magnitudes, with a similar but smaller pattern for F_y.Source: §4.3 · p. 22
  • Multi-camera capture provided more stable peak-force estimates than single views, while its benefit for frame-level RMSE was more modest.The authors suggest region-level representations of participant and object motion may be less sensitive to exact landmark localization than hand-distance estimation, which is vulnerable to occlusion and monocular depth ambiguity.Source: §4.3 · p. 22

4.4 Effects of ROI Representation Strategy … 5.0 Conclusions

The VLM pipeline estimated continuous, bilateral, triaxial hand forces from RGB video and known box mass, with performance varying by task and configuration. Object ROIs generally helped, segmentation added little, and broader workplace generalization remains untested.

4.4 Effects of ROI Representation Strategy

  • Adding the handled object as a second ROI reduced RMSE for selected forces and tasks, with vertical-force RMSE ~5–7% lower during symmetric and asymmetric lifting.Object-ROI benefits included smaller right-hand Fx PE (~2.9 N for S1/S3 vs. ~2.7 N for S2/S4) and were especially notable under single-camera conditions, suggesting object information can partly compensate for limited viewpoints.Source: §4.4 · pp. 22–23
  • Pixel-level segmentation added little beyond detection-only ROIs, with near-identical right-hand Fx PE and Fy RMSE across bounding-box and mask strategies.The authors suggest force estimation may depend more on task-relevant regions and their relative motion than on precise pixel boundaries.Source: §4.4 · p. 23

4.5 Practical Implications for VLM-Based Ergonomic Assessment

  • No single camera-view condition or ROI strategy minimized error across all tasks and force channels, so configuration should match the task and exposure metric of interest.Multiple synchronized views may benefit peak-force characterization, particularly Fz; single-camera capture may be a practical trade-off when overall frame-level accuracy is sufficient.Source: §4.5 · p. 23

4.6 Limitations and Future Research Directions

  • Generalization remains untested for diverse worker populations, new camera viewpoints, different objects, and less controlled workplace environments.Participants were healthy and 18–39 years old, tasks used one box in a controlled laboratory, and the same setup and three viewpoints were used across validation folds; future work could add hand or body keypoints, object motion, or contact representations.Source: §4.6 · p. 24
  • The pipeline used fixed 300-frame sequences with bidirectional context and offline processing, and temporal alignment of high-force events was not assessed.The dataset used one box geometry and known masses of 6, 9, and 12 kg; future work should examine event timing, processing efficiency, temporal windows, broader objects, and practical provision or estimation of load information.Source: §4.6 · p. 24

5.0 Conclusions

  • RMSE was ~4.7–5.6 N for horizontal and mediolateral components and ~10.6–11.0 N for the vertical component, demonstrating feasibility from RGB video and known box mass.Performance varied with task, mass, camera-view condition, and ROI strategy; multi-camera capture most clearly improved peak-force estimation, while overall RMSE differences among camera conditions were modest.Source: §5.0 · pp. 24–25

Appendix · A.1 Participants

The appendix describes a convenience sample of 35 young adults from a prior study, all of whom self-reported right-handedness, regular physical activity, and no recent musculoskeletal disorders. The study received institutional review board approval, and all participants gave informed consent.

A.1 Participants

  • Data came from a prior study, which recruited 35 young adults from the university and local community: 21 males and 14 females.Source: §A.1 · p. 32
  • Male participants’ mean (SD) age, body mass, and height were 27.2 (4.5) years, 77.6 (12.3) kg, and 176.4 (5.7) cm, respectively.Source: §A.1 · p. 32
  • Female participants’ mean (SD) age, body mass, and height were 26.8 (5.5) years, 68.2 (8.2) kg, and 170.1 (7.1) cm, respectively.Source: §A.1 · p. 32
  • All participants self-reported being right-handed and physically active, exercising at least twice per week, with no musculoskeletal disorders within the past year.Source: §A.1 · p. 32
  • The protocol was approved by the Virginia Tech Institutional Review Board (#23-095), complied with the Declaration of Helsinki, and informed consent was obtained before data collection.Source: §A.1 · p. 32

A.2 MMH Task Details and Experimental Procedures

The repeated-measures protocol used one box across five simulated MMH tasks, with practiced work strategies and counterbalanced presentation of hand configurations and box masses.

A.2 MMH Task Details and Experimental Procedures

  • One wood box (26.0 × 41.0 × 23.5 cm) was used for all five simulated MMH tasks.Participants practiced their preferred work strategies and speed during training to simulate an industrial setting.Source: §A.2 · p. 32
  • Hand configurations and box-mass presentation were counterbalanced with balanced Latin squares, with at least four minutes of rest between trials using different configurations.The repeated-measures protocol included a training phase before experimental trials.Source: §A.2 · p. 32
  • The two hand configurations were broad and narrow.Figure A.1 illustrates both configurations.Source: Figure A.1 · p. 32

A.3 Azure Kinect™ Camera Instrumentation

Whole-body kinematics were recorded with three synchronized Azure Kinect™ cameras sampled at 30 Hz. Their placement and daisy-chain synchronization were selected to support coverage and timing.

A.3 Azure Kinect™ Camera Instrumentation

  • Three synchronized Azure Kinect™ markerless cameras recorded whole-body kinematics at 30 Hz.The systems were manufactured by Microsoft Corporation.Source: §A.3 · p. 32
  • Cameras were positioned about 1.74 m from the work-area edge to improve coverage given their narrow field of view.This placement configuration was established during pilot testing.Source: §A.3 · p. 32
  • A 3.5-mm auxiliary cable synchronized the cameras in a daisy chain, with one primary and two secondary devices.Source: §A.3 · p. 32

A.4 Load Cell Instrumentation and Hand Force Measurements

Triaxial load cells measured external hand forces on the box, and the processed signals served as ground-truth targets. Forces were expressed in a box-centered coordinate system aligned with anterior–posterior, medial–lateral, and vertical directions.

A.4 Load Cell Instrumentation and Hand Force Measurements

  • Triaxial load cells attached near the box top provided external hand-force measurements.The load cells were Michigan Scientific TR3D-A-1K units.Source: §A.4 · p. 33
  • Signals sampled at 200 Hz were filtered with a 300-ms moving RMS window and downsampled to 30 Hz to match kinematic data.Source: §A.4 · p. 33
  • Forces were represented in a box-centered coordinate system, with X anterior–posterior, Y medial–lateral, and Z vertical.The tri-axial forces served as ground-truth targets for model development and evaluation.Source: §A.4 · p. 33, Figure A.2 · p. 33

A.5 Transformer-Based Force Prediction Model: Additional Details

The model used a four-layer transformer encoder to predict all six force channels simultaneously. Training used overlapping 300-frame windows, with padding masked from attention and loss, and fold-specific early stopping.

A.5 Transformer-Based Force Prediction Model: Additional Details

  • A four-layer, eight-head transformer with 256-dimensional latent representations fed a regression head that simultaneously predicted all six force channels per frame.The feedforward dimension was 512; the regression head used 256→128 units with GELU and dropout, followed by 128→6 units.Source: §A.5 · p. 33
  • Training used 300-frame windows with a 150-frame stride, while validation and test windows did not overlap; padded frames were excluded from loss and self-attention.Trials shorter than 300 frames were zero-padded and given an attention mask that marked padded positions as invalid.Source: §A.5 · pp. 33–34
  • AdamW training used a 1 × 10⁻⁴ learning rate, 100-epoch cosine annealing, batch size 16, and early stopping after 20 unimproved validation epochs; metrics were computed per channel over valid test frames.The optimizer used 1 × 10⁻⁴ weight decay and gradient clipping at norm 1.0; each fold's minimum-validation-loss checkpoint was evaluated on its test participant.Source: §A.5 · p. 34

Related papers

Loading 2608.22586v1…