SAFER-Activities: A Dataset for Smart Assessment of Fall Events and Routine Activities

1 Introduction

• Skeleton features resist background clutter and lighting changes but lose scene context and can fail with poor pose estimates; RGB captures context but is more susceptible to domain shift.These complementary strengths motivate multimodal fusion.Source: §1 · p. 2
• Healthcare action-recognition datasets need long, untrimmed video, frame-level labels, routine and fall-like actions, and unseen-subject and unseen-environment evaluation.Prior fall datasets include realistic falls but have few fall instances and lack comprehensive labels for routine and fall-like events.Source: §1 · p. 2
• SAFER-Activities contains over 66 hours of video from 46 participants, with frame-level annotations for 85,310 action instances across 30 classes, including 5,406 falls.It includes multiple camera viewpoints, routine and fall-like actions, a wheelchair-use subset, and a home-environment test set with unseen participants and viewpoints.Source: §1 · p. 3
• The accompanying pose-estimation dataset contains 6,846 images and 8,324 human instances for benchmarking methods on people seated in wheelchairs.The paper also identifies wheelchair users’ health and safety needs, including risks associated with prolonged use, transfers, and insufficient physical activity.Source: §1 · pp. 2–3
• Skeleton-based methods generalize best out of distribution, while multimodal fusion improves in-distribution performance but introduces challenges under domain shift.The evaluation spans in-lab, non-lab, wheelchair, and external-dataset testing; models trained on SAFER-Activities also detect falls in unseen real-world scenarios.Source: Figure 1 · p. 2, §1 · p. 3

2.1 Fall Detection Datasets

• SAFER-Activities provides 5,406 fall instances, frame-level labels for 30 activity classes, and recordings from 46 adults aged 18–58, exceeding prior datasets in scale and coverage.Prior datasets include clip-level labels or only one participant, while Baldewijns et al. emphasize realistic re-enacted falls but lack ADL labels; ImViA is used for cross-dataset evaluation because of its realistic home setting and precise fall boundaries.Source: §2.1 · p. 4

2.2 Camera-Based Human Action Recognition

• Camera-based HAR uses RGB, skeleton, or fused modalities; skeletons are less sensitive to appearance changes, but domain-shift robustness remains challenging for RGB and fusion methods.The SAFER-Activities benchmark spans lightweight 1D CNNs, state-of-the-art skeleton models, and pretrained RGB backbones used as frozen feature extractors, with in- and out-of-distribution splits.Source: §2.2 · pp. 4–5

2.3 Human Pose Estimation

• SAFER-Activities includes a wheelchair-focused pose-estimation subset for benchmarking pose models across camera angles and under occlusion.Related HPE datasets target keypoint detection, occlusion, or crowded scenes, including MS COCO, MPII, OCHuman, and CrowdPose.Source: §2.3 · p. 5

3.1 Frame-level Labels

• Frame-level labels mark action starts and ends in untrimmed videos, enabling models to use temporal context and recognize actions online as they unfold.The context window may include adjacent actions rather than requiring every frame to depict the same action.Source: §3.1 · pp. 5–6

3.2 Dataset Construction

• The final lists contain 30 unique generalized actions across non-wheelchair and wheelchair subsets, reduced to 15 per subset from original counts of 25 and 37, respectively.Each video was labeled in its entirety by one annotator and independently verified by a second. Frame-level agreement was Cohen’s κ of 0.89, with boundary F1 of 0.86 at a 500 ms tolerance.Source: §3.2 · p. 7
• The wheelchair videos also yield a pose-estimation dataset of 6,846 images with 8,324 manually annotated human instances.The authors provide benchmark performance for popular human-pose-estimation methods on this subset; they did not use it to fine-tune models.Source: §3.2 · p. 8

3.3 Statistics

• The dataset contains over 66 hours of video from 46 participants, with subject- and view-wise splits and a separate home-environment test set for out-of-distribution evaluation.About 15 hours come from the wheelchair subset; the home test includes 30 videos from 5 participants recorded with different lighting, viewpoints, and partial occlusion.Source: §3.3 · pp. 8–9

4.1 Experimental Setup

• Evaluation used subject-wise splits, macro-averaged and per-class F1-scores, and 48-frame windows; RGB and fusion branches used 16 uniformly sampled frames.Training randomly sampled a sub-clip every 20 frames; dense sliding windows covered full videos at test time, with labels assigned by majority vote over five midpoint-centered frames. Models included CNN1D, GCN-based and PoseC3D skeleton models, frozen RGB backbones, and concatenation fusion with a fixed CNN1D skeleton stream.Source: §4.1 · p. 9

4.2 Baseline Results

• In-lab, 2D MS-G3D achieved 87.2% (non-wheelchair) and 77.5% (wheelchair), and fusion reached 78.6% on Wheelchair; the best 2D OOD score was 79.3%.Fusion was competitive with skeleton models in-lab, while 3D pose models were slightly behind their 2D counterparts.Source: Table 2 · p. 10, §4.2 · pp. 10–11, Table 3 · p. 11
• Under domain shift, skeleton models outperformed frozen RGB and feature-concatenation fusion, while RGB fall detection nearly failed.DINOv3 fell from 72.1% to 23.8% and had 0.0% F1 for falls; CLIP fell from 70.6% to 35.6% and had 9.9% fall F1. The best OOD “Unstable” score was 45.5%, compared with 74.8% in-lab.Source: §4.2 · p. 11

4.3 Additional Experiments

• ModDrop most effectively recovered non-lab performance, but no fusion strategy surpassed skeleton-only CNN1D there; MMCL instead improved wheelchair recognition.ModDrop raised VideoMAE from 54.5% to 67.3%, DINOv3 from 32.9% to 59.6%, and CLIP from 52.3% to 61.9%; MMCL achieved 69.7% versus 68.2% for CNN1D alone on Wheelchair, while CNN1D scored 73.8% on non-lab.Source: §4.3 · pp. 11–12, Table 4 · p. 12
• For 2D skeleton models, extending temporal context from 48 consecutive frames to 144 frames left in-lab non-wheelchair performance nearly unchanged but consistently improved wheelchair recognition.The comparison used temporal stride 1 versus stride 3, sampling every third frame for the longer context.Source: Table 5 · p. 12, §4.3 · p. 12
• On ImViA without finetuning, PoseC3D reached 96.8% F1 and VideoMAE–CNN1D fusion reached 96.4% F1, while CLIP and DINOv3 achieved 0 F1.Evaluation covered 130 videos with 99 falls, matching predicted and ground-truth temporal events with a 15-frame tolerance; VideoMAE alone had 38.4% recall.Source: Table 6 · p. 12, §4.3 · pp. 12–13

4.4 Qualitative Analysis

• Qualitatively, CNN1D distinguished falls from slowly lying down using abrupt temporal dynamics, while external-data failures mainly involved occluded or missed detections.The qualitative examples include external real-world falls and wheelchair-user actions.Source: §4.4 · p. 13

Conclusion

• Skeleton models generalize best under domain shift, while frozen RGB fusion with CNN1D improves in-domain recognition, especially for wheelchair actions, but degrades out of distribution.The tested fusion strategies reduce but do not fully close the out-of-distribution gap.Source: §5 · p. 13
• SAFER-Activities provides frame-level labels for online action recognition, a wheelchair subset, and precomputed pose skeletons and visual features for smart healthcare research.Its evaluation spans 2D and 3D skeleton models, frozen RGB backbones, and multimodal fusion.Source: §5 · p. 13
• Dense temporal annotations enable lightweight models to distinguish visually similar actions such as falling and lying down through temporal dynamics.Source: §5 · p. 13

Limitations and Future Work

• Simulated falls used adult actors rather than elderly participants or regular wheelchair users, so SAFER-Activities is a pre-deployment benchmark, not a clinical validation study.Wheelchair recordings are currently in-lab, and external evidence for wheelchair-use scenarios is qualitative; prospective validation and more naturalistic data remain important.Source: §5 · pp. 13–14
• Future work can develop synchronized multi-view architectures, adaptively use RGB context when reliable, and combine pose dynamics with depth or inertial cues for ambiguous falls.The proposed directions respond to appearance shifts in cross-dataset evaluation and sharp drops on visually ambiguous, safety-critical actions.Source: §5 · p. 14

A Additional Experimental Details

• Training selects one random sub-clip every 20 frames, while testing uses all sub-clips and evaluates with pre-extracted poses and RGB features.The evaluation is described as similar to inference on the original videos.Source: A Additional Experimental Details · p. 20
• Sub-clips use 144-frame windows reduced to 48 frames with stride 3, are retained with fewer than 5 erroneous frames, and receive majority-vote labels from the central 5 frames.For smaller windows, the parameters are w = 48 and t = 1.Source: A Additional Experimental Details · p. 20
• The evaluation pipeline includes pose and visual branches, four model families, and fusion models that combine skeleton and visual streams.Person skeletons and person-centric crops are extracted using YOLOv8x bounding boxes.Source: A Additional Experimental Details · p. 20, Figure 6 · p. 21
• Wheelchair keypoint performance is reported using AP across OKS thresholds 0.50–0.95 and per-joint PCK at threshold 0.05.Source: Table 7 · p. 21

B Wheelchair Keypoints Dataset

• The evaluation used COCO-pretrained HPE models from MMPose, reporting PCK overall and by body part, plus OKS average precision.PCK was measured at a 0.05 threshold, and OKS AP averaged thresholds from 0.50 to 0.95.Source: B Wheelchair Keypoints Dataset · p. 21
• ViTPose variants performed best across both PCK and OKS metrics on the wheelchair keypoints dataset.PCK used a 0.05 threshold; OKS AP averaged scores across thresholds from 0.50 to 0.95.Source: B Wheelchair Keypoints Dataset · p. 21
• Knee, ankle, and head keypoints were hardest to estimate, likely because they are partly obscured when a person is not facing the camera.Other joints were either more visible or more accurately predicted despite occlusion.Source: B Wheelchair Keypoints Dataset · p. 21
• The keypoint data were collected in a laboratory with limited lighting and viewpoint diversity, motivating evaluation and improvement in real-world wheelchair scenarios.The passage particularly identifies challenging body parts as a target for future work.Source: B Wheelchair Keypoints Dataset · p. 21

C Segment-Level Evaluation

• Segmental F1@10, 25, and 50 is computed by matching contiguous predicted segments one-to-one with same-class ground-truth segments by intersection-over-union.Dense per-frame predictions are merged into contiguous segments; this protocol complements the online per-frame benchmark.Source: C Segment-Level Evaluation · p. 22
• The skeleton model (CNN1D) is most robust under domain shift, while fusion achieves the strongest wheelchair performance in segment-level evaluation.Frozen RGB (VideoMAE) degrades sharply on the non-lab split, and fusion recovers much of this gap.Source: C Segment-Level Evaluation · p. 22

D Annotation Details

• Annotators selected predefined actions or entered new ones, then recorded video timestamps with Start and End controls and exported completed annotations to CSV.They could pause, play, and rewind while labeling.Source: D Annotation Details · p. 22
• Overlaid keypoints helped refine pose extraction by filtering irrelevant poses, while overlaid labels helped identify and correct annotation errors.The visualizations were shown in Fig. 7b.Source: D Annotation Details · p. 22

E.1 Micro and Macro Actions

• Tables 9 and 10 report labeled micro-action instance totals for the non-wheelchair and wheelchair datasets and their mappings to macro actions.The supplied passage does not provide the table values.Source: §E.1 · p. 22

E.2 Action Descriptions

• The action descriptions outline definitions of macro and micro actions for both the non-wheelchair and wheelchair datasets.The supplied passage does not specify individual action definitions.Source: §E.2 · p. 23

Non-wheelchair Dataset.

• Standing labels cover complex poses, general activities, and actions such as clapping, checking the time, calling, pointing, waving, mopping, and exercising.The labels distinguish specific standing actions from the broader stand_activity class.Source: Non-wheelchair Dataset · p. 23
• Sitting labels include floor and complex positions, general activities, and actions such as clapping, checking the time, calling, pointing, waving, and exercising.The labels distinguish specific sitting actions from the broader sit_activity class.Source: Non-wheelchair Dataset · p. 23
• Mobility and posture labels capture abnormal or assisted walking, trembling while using a walker or cane, bending, and bending exercises.The walking labels include walker and cane use, with separate tremor variants.Source: Non-wheelchair Dataset · p. 23
• Transition and fall labels distinguish getting up, legacy bend_getting_up, sitting down, balance instability that often precedes falls, and collapsing to the floor.bend_getting_up is used only in some dataset parts and is substituted by getting_up.Source: Non-wheelchair Dataset · pp. 23–24
• Additional labels cover lying down and transitioning into a lying position, reaching with one or both arms, and jumping.Source: Non-wheelchair Dataset · p. 24, Wheelchair Dataset · p. 25

• Object-interaction labels include pick_place, pick, place, and reach.These cover picking up and placing objects, as well as reaching for an object.Source: Wheelchair Dataset · p. 24
• Seated activities include sitting on the floor, posture or wheelchair adjustments, deskwork, eating, calling, drinking, and other activities.The labels are sit_floor, adjust_posture, adjust_wheelchair, sit_activity, deskwork, eat, call, and drink.Source: Wheelchair Dataset · p. 24
• Wheelchair mobility and posture labels cover being propelled by another person, self-propelling, bending forward while seated, and exercising in the wheelchair.These actions are labeled get_propelled, propel, bend, and exercise.Source: Wheelchair Dataset · p. 24
• Transition labels distinguish getting up from sitting or lying, sitting down from standing, and lying down from lying down as an action.The corresponding labels are getting_up, sitting_down, lying_down, and lie_down.Source: Wheelchair Dataset · pp. 24–25, Non-wheelchair Dataset · p. 24
• Transfer-related labels distinguish preparing for a wheelchair transfer, transferring to or from the wheelchair, and falling during an attempted transfer.These are labeled prepare_transfer, transfer, and fall.Source: Wheelchair Dataset · p. 24

• Non-wheelchair samples include getting_up, walk, bend, sit, stand_activity, unstable, stand, and fall.Source: Figure 8 · p. 29
• Other non-wheelchair samples include lie_down, sit_activity, sitting_down, lying_down, reach, run, and jump.Source: Figure 9 · p. 30
• Wheelchair samples include sit, getting_up, pick_place, prepare_transfer, transfer, fall, lie_down, and sitting_down.Source: Figure 10 · p. 31
• Additional wheelchair samples include propel, bend, sit_activity, lying_down, stand, getting_propelled, and exercise.Source: Figure 11 · p. 32