PaperlayerView exact revision on arXiv

Source-linked AI summary

TAP3D: Thermal-Assisted 3D Human Point Clouds

Xie Zhang, Chengxiao Li, Xuan Liu, Chenshu Wu

arXiv:2610.11241v1cs.CVcs.HC

TL;DR

Existing LiDAR, radar, and depth-camera approaches face trade-offs including cost, sparse reconstruction, and privacy concerns. TAP3D uses a physics-informed thermal-array pipeline to reconstruct human point clouds, achieving geometric accuracy and supporting downstream tasks including fall detection, tracking, and mesh recovery.

  • Problem

    LiDAR, radar, and depth-camera approaches have drawbacks including high cost, sparse reconstruction, and privacy concerns.

    Source: §1 · p. 2

  • Method

    TAP3D combines a forward thermal physics model with self-supervised multi-primitive estimation and geometric perspective fusion to reconstruct per-user point clouds from thermal maps.

    Source: §1 · p. 3, §4.2 · p. 6, §4.3 · p. 6

  • Results

    Mean directed Chamfer distance of 4.60 cm and mean F1 score of 0.95 across single- and multi-user cases; fall detection reaches 91.46% accuracy, tracking has 21.86 cm MAE, and mesh recovery has a 4.87 cm minimum matching distance.

    Source: §1 · p. 3

  • Takeaways & Limitations

    TAP3D provides a privacy-preserving, fully passive human-sensing approach with demonstrated use in fall detection, indoor tracking, and mesh recovery.

    Source: §1 · p. 3, §10 · p. 13

    Commodity thermal arrays have lower resolution than RGB-D cameras and are less robust outdoors under solar heating or weak human-background contrast.

    Source: §8 · pp. 11–12

Abstract

from arXiv · show

Human body point clouds are a versatile representation for AI-enabled human sensing. However, existing methods using LiDAR, radar, and depth cameras suffer from inherent drawbacks in high cost, sparse reconstruction, and privacy concerns, etc. In this paper, we exploit low-cost thermal arrays and present TAP3D, the first system to reconstruct 3D human point clouds from body heat signatures, offering significant advantages in cost, density, human sensitivity, and privacy. To overcome major challenges in depth estimation, thermal interference, and multi-person separation, we propose a novel physics-informed design, which integrates a forward thermal physics model with two distinct modules: multi-primitive estimation for self-supervised joint recovery of depth and other thermal properties, and geometric perspective fusion for suppressing interference and disentangling multiple people. We implement TAP3D using a single commodity thermal array sensor and build a large-scale dataset (160K samples, 8 environments, 11 users) for evaluation. TAP3D achieves remarkable accuracy for dense point cloud generation, enabling downstream tasks like fall detection (91.46%), indoor tracking (21.86 cm MAE), and human mesh recovery (4.87 cm error). By transforming body heat into point clouds for the first time, TAP3D pioneers a new paradigm for privacy-first, fully passive human sensing for many applications. TAP3D is open-sourced at https://github.com/aiot-lab/TAP3D.

1 INTRODUCTION

TAP3D reconstructs dense 3D human point clouds from low-cost thermal arrays, addressing limitations in existing sensing modalities and challenges in depth estimation, thermal interference, and multi-person separation. Its physics-informed design combines primitive estimation and geometric perspective fusion, with evaluation showing accurate reconstruction and downstream task performance.

1 INTRODUCTION

  • Depth cameras, LiDAR, and mmWave radar have distinct trade-offs involving privacy, cost, sensitivity, interference, and point-cloud sparsity.Thermal arrays are presented as balancing cost, spatial resolution, and privacy while operating passively.Source: §1 · p. 2
  • Depth estimation, ambient heat sources, and adjacent or motion-blurred signatures complicate thermal point-cloud reconstruction.Thermal intensity is a poor proxy for geometry, and non-human heat can cause false positives.Source: §1 · p. 2, Figure 3 · p. 3
  • TAP3D uses a forward thermal physics model and multi-primitive estimation to jointly recover depth, emissivity, surface temperature, and reflection from temperature maps.The forward model supports self-supervised learning without requiring ground-truth primitive values that are impractical to obtain in real-world scenarios.Source: §1 · pp. 3–4
  • Geometric perspective fusion uses OAV maps to suppress ambient interference and differentiable OAV-to-BEV mapping to separate people whose signatures overlap in OAV.The method fuses the two perspectives to produce refined per-person point clouds.Source: §1 · p. 3
  • 4.60 cm mean directed Chamfer distance and 0.95 mean F1 score across single- and multi-user cases; downstream results include 91.46% fall-detection accuracy, 21.86 cm tracking MAE, and 4.87 cm minimum mesh matching distance.Evaluation uses more than 160,000 samples from eight indoor environments and 11 participants.Source: §1 · p. 3

2 PILOT STUDY

The pilot study tests a straightforward learning baseline for thermal point-cloud reconstruction using data from single-user, multi-user, and background-only scenarios. Its performance degrades substantially in multi-user scenes, motivating a more principled approach.

2 PILOT STUDY

  • The pilot dataset contains 36,921 temperature maps from five volunteers across five indoor environments, split into 17,956 training and 18,965 test samples.Samples include ground-truth point clouds and masks from a co-located Intel RealSense D455; scenarios cover single users, multiple users, and background only.Source: §2 · p. 4
  • Multi-user scenes lower baseline F1 from 0.91 to 0.82, raise depth MAE from 26.83 cm to 51.72 cm, and increase DCD from 18.5 cm to 34.2 cm relative to single-user scenes.The authors attribute the geometric degradation to difficulty handling inter-person occlusion and separation.Source: §2 · p. 4
  • The U-Net-like baseline predicts dense depth, person-instance IDs, and a foreground-background mask, then projects segmented depth into 3D using sensor intrinsics.The study evaluates the model on both single-user and more challenging multi-user cases.Source: §2 · p. 4

3 TAP3D PHYSICS MODEL

TAP3D formulates thermal point-cloud recovery as an inverse problem grounded in a forward model of thermal radiation. The model links body primitives to sensor temperatures, which are used to recover 3D locations.

3 TAP3D PHYSICS MODEL

  • The forward thermal model describes how body-emitted and reflected radiation becomes pixel-wise sensor temperatures, forming the basis for recovering 3D human point clouds.It models radiation propagation through the atmosphere and optics before conversion to measured temperatures.Source: §3 · pp. 4–5
  • TAP3D formulates point-cloud estimation as recovering target locations from sensor-estimated temperatures through the Sakuma–Hattori model.The formulation distinguishes unknown target primitives, fixed environmental and device parameters, and observed sensor temperatures.Source: §3 · p. 5

4 PHYSICS-INFORMED DESIGN

TAP3D combines physics-informed primitive estimation with geometric refinement from optical-axis and bird’s-eye views. Its training combines self-supervised reconstruction with supervised depth, indication, segmentation, and refinement losses.

4.1 TAP3D Model Overview

  • TAP3D combines multi-primitive estimation for depth recovery with geometric perspective fusion to suppress ambient interference and separate users before producing point clouds.The outputs support mesh recovery, fall detection, and indoor tracking.Source: §4.1 · p. 5

4.2 Multi-Primitive Estimation

  • Jointly estimating depth, emissivity, reflected radiation, and surface temperature with a forward thermal model enables self-supervised recovery without ground truth for several thermal properties.The forward model approximates Planck-law spectral integration with the Stefan–Boltzmann law and predicts reflected radiation directly; the training loss compares reconstructed and input temperature maps.Source: §4.2 · p. 6

4.3 Geometric Perspective Fusion

  • Optical-axis refinement uses emissivity, temperature, and depth to suppress ambient interference, while bird’s-eye-view refinement separates overlapping users through differentiable, invertible depth mapping.Differentiable Index Mapping converts OAV depth into a BEV occupancy map that can be mapped back to OAV depth.Source: §4.3 · pp. 6–7, Figure 7 · p. 6

4.4 Training Strategy

  • TAP3D is trained end-to-end with weighted self-supervised and supervised losses, using co-located depth-camera ground truth in experiments across eight indoor environments.The multi-primitive module uses Huber losses for temperature reconstruction, depth, and indication maps; fusion uses binary cross-entropy for OAV segmentation and BEV refinement.Source: §4.4 · p. 7, Figure 8 · p. 7

5 IMPLEMENTATION

TAP3D uses a low-cost thermal array and a co-located depth camera for ground truth, with data recorded at 8 Hz.

5 IMPLEMENTATION

  • The implementation uses an 80 × 62 thermal array with a 90° × 67° field of view and an approximate cost of $10.A co-located Intel RealSense D455 acquires ground-truth point clouds and target masks.Source: §5 · p. 7
  • The PyTorch model has 23.29M parameters and is trained on an RTX 4090 GPU with AdamW for up to 100 epochs.Training uses a learning rate of 0.001, weight decay of 0.01, and early stopping patience of 8 epochs.Source: §5 · p. 7
  • The DIM uses 400 depth bins of 20 mm each, matching the D455’s 8000 mm maximum range.The Gaussian kernel parameter is σ = 0.5.Source: §5 · p. 7

6 EXPERIMENTS

TAP3D is evaluated on a large, multi-environment dataset and outperforms adapted vision baselines in reconstruction accuracy and efficiency. Micro-benchmarks and ablations further examine robustness and the contributions of its refinement and estimation modules.

6.1 Dataset, Baselines, and Metrics

  • 160,000+ samples from 11 volunteers across eight indoor environments support evaluation against adapted RGB2Point and NeWCRF baselines using DCD and F1.The dataset includes temperature maps, D455-based point-cloud ground truth, and human masks.Source: §6.1 · p. 7

6.2 Overall Performance

  • TAP3D achieves the lowest DCD (4.60 cm versus 10.15 cm for RGB2Point and 19.26 cm for NeWCRF) and highest F1 (0.953), using 23.32 M parameters and 7.41 G FLOPs.Source: §6.2 · p. 8
  • 13.04 cm mean depth error across 985 synchronized samples is lower than compact radar (19.10 cm) and TADAR (36.74 cm).The comparison uses one subject walking from 0.5 m to 4 m, with RealSense D455 depth as ground truth.Source: Figure 10 · p. 8, §6.2 · p. 8
  • TAP3D maintains detection F1 near 0.90 at 6.5 m and DCD below 6.5 cm, while running at 15.02 Hz with a roughly $20 sensor-side hardware cost.Close-range DCD errors remain under 6.2 cm despite partial body heat signals; the reported full autonomous node cost is approximately $270.Source: §6.2 · pp. 8–9, Figure 11 · p. 9

6.3 Micro-benchmark

  • Across held-out environments and users, activities, clothing conditions, and 80% occlusion, TAP3D retains strong detection and reconstruction performance.Unseen environments yield 0.967 F1 and 3.34 cm DCD; unseen users yield 0.996 F1 and 1.90 cm DCD, while 80% occlusion yields 0.989 F1 and 2.23 cm DCD.Source: Figure 12 · p. 9, Figure 13 · p. 9, §6.3 · p. 9
  • Walking speeds up to 1.3 m/s, user orientations through 180°, and incident angles through 30° preserve stable performance, while larger temperature shifts and DIM settings have limited effects.At 45°, performance falls to 0.926 F1 and 4.68 cm DCD; room-temperature testing from 20–30°C reports average DCD of 4.30 cm.Source: Figure 17 · p. 9, Figure 18 · p. 9, Figure 19 · p. 10

6.4 Ablation Study

  • Removing BEV refinement causes a 3.7% average F1 drop under dense conditions (4–6 users); OAV refinement improves detection across diverse ambient thermal interference sources.Source: §6.4 · p. 10, Figure 24 · p. 10
  • Multi-primitive estimation reduces DCD by 5.59%, and self-supervised pretraining reduces it by 11.93% on average, though fully self-supervised reconstruction is not yet supported.The pretraining–finetuning protocol uses 60% unlabeled data for pretraining and 40% labeled data for finetuning.Source: Figure 25 · p. 10, Figure 26 · p. 10, §6.4 · p. 11

7 CASE STUDY

The case study applies TAP3D point clouds to fall detection, indoor tracking, and 3D mesh recovery. The reported results demonstrate utility across these downstream tasks, with mesh recovery identified as exploratory.

Fall detection

  • 91.46% accuracy on a held-out set of six fall recordings demonstrates fall detection from TAP3D-derived depth-map sequences.The dataset contains 756 fall samples and 5,973 non-fall samples from three volunteers.Source: §7 · p. 11

3D mesh recovery

  • 4.87 cm minimum matching distance is reported for mesh recovery from TAP3D point clouds, an exploratory result based on 787 samples from three users.The authors caution that the limited dataset may introduce overfitting.Source: Figure 28 · p. 11, §7 · p. 11

Indoor human tracking

  • 21.86 cm mean center distance error across 144 one-minute recordings supports room-level indoor tracking of single and multiple users.The paper describes the result as competitive with privacy-preserving RF tracking systems.Source: §7 · p. 11

8 LIMITATIONS AND FUTURE WORK

TAP3D has scope and sensing limitations, with several extensions left for future work. Its point density is bounded by the thermal array’s resolution.

Current limitations

  • Outdoor solar heating, weak human–background contrast, severe multi-user overlap, and varying heat sources can degrade or defeat TAP3D.The paper notes that commodity thermal arrays are less robust outdoors than indoors.Source: §8 · pp. 11–12

Point cloud density and resolution

  • Hundreds of points per person exceed compact radar and low-cost thermal ranging outputs but remain less dense than RGB-D clouds, limiting reconstruction to coarse-to-medium geometry.Point density is inherently limited by sensor resolution.Source: §8 · p. 12

Future work

  • Precise multi-primitive estimation, multi-sensor integration, and multimodal fusion remain future directions rather than established TAP3D capabilities.The proposed integration path aligns reconstructed clouds in a shared coordinate frame using known sensor poses.Source: §8 · p. 12

9 RELATED WORKS

Prior human point-cloud methods use depth cameras, LiDAR, or radar, while thermal-array sensing has largely targeted specific tasks or less complete representations. TAP3D is presented as a dense 3D, fully passive thermal approach.

9.1 Human Body Point Cloud Generation

  • Depth cameras provide dense clouds but raise privacy concerns, LiDAR offers high-fidelity sensing at higher cost, and compact mmWave radar often produces sparse, noisy clouds.The related work describes radar approaches using SAR and learning-based densification to improve fidelity.Source: §9.1 · p. 12

9 RELATED WORKS

  • TAP3D combines cost efficiency, high density, human sensitivity, and privacy preservation in a fully passive thermal-assisted point-cloud system.The related-work discussion frames point clouds as a bridge for integrating thermal arrays with other modalities.Source: §9.1 · p. 12

9.2 Thermal Array-based Human Sensing

  • Task-oriented thermal methods act directly on temperature maps, while representation-oriented work has produced masks or range estimates without full-body 3D geometry.The cited range methods include single-user, front-facing constraints and single-depth multi-user masks.Source: §9.2 · pp. 12–13
  • TAP3D reconstructs dense 3D human point clouds from thermal arrays, providing richer geometry than prior thermal masks or single-depth estimates.The paper identifies mesh reconstruction and cross-modality alignment as supported uses of this representation.Source: §9.2 · p. 13

10 CONCLUSION

TAP3D uses a physics-informed design to reconstruct dense human point clouds from thermal-array measurements. Its forward model describes how radiation becomes pixel temperatures, and its inverse formulation recovers human-body primitives from those measurements.

10 CONCLUSION

  • TAP3D is presented as the first single-sensor system to transform body heat into dense 3D human point clouds, with experiments across 160K samples and downstream case studies.The conclusion reports high geometric accuracy and reliable detection across diverse conditions, and highlights fall detection, indoor tracking, and 3D mesh recovery.Source: §10 · p. 13

A.1 Forward Physical Model

  • Each thermal-array pixel receives radiation from a corresponding body-surface region and reports a temperature; the forward model links this output to sensor, atmospheric, target, and ambient-interference properties.The appendix identifies thermography theory as the basis for the model.Source: §A.1 · p. 17, §A.2 · p. 17, A DETAILED DERIVATION OF THE TAP3D… · p. 17
  • Target depth is inferred through distance-dependent atmospheric attenuation, while target spectral exitance includes both self-emission and radiation reflected from ambient regions.The model also accounts for geometric visibility between ambient regions and the target surface.Source: §A.1 · p. 17
  • The forward model is simplified using assumptions that include direction-independent emissivity, uniform ambient-interference surfaces, and neglect of higher-order radiation transfer and inter-reflections.For the inverse formulation, surfaces are further treated as Lambertian gray bodies with wavelength-independent emissivity.Source: §A.1 · p. 17, §A.2 · p. 17

A.2 Inverse Problem Formulation

  • The inverse problem treats target position, temperature, and emissivity as unknown primitives, uses measured pixel temperatures as observations, and recovers dense point clouds under fixed parameters.The main-text equations compactly define this physics-constrained inverse problem.Source: §A.2 · p. 18

B TRAINING OBJECTIVE

The training objective combines thermal reconstruction and depth and indication supervision for multi-primitive estimation with OAV segmentation and BEV refinement losses for geometric perspective fusion.

B TRAINING OBJECTIVE

  • The dataset pairs thermal-array readings with stereo-derived ground-truth depth and human indication maps, including a binary foreground mask for reconstruction loss.The foreground mask is defined as the thresholded human indication map.Source: B TRAINING OBJECTIVE · p. 18
  • Multi-primitive estimation uses a self-supervised reconstruction loss alongside depth-map and human-indication-map losses.The indication-map loss uses Huber loss against the ground-truth map.Source: B TRAINING OBJECTIVE · p. 18
  • Geometric perspective fusion uses OAV segmentation and BEV refinement losses, with weighting factors balancing the objective terms.The OAV term uses binary cross-entropy; the BEV term compares refined outputs with depth-index-mapped ground truth.Source: B TRAINING OBJECTIVE · p. 18

C DATASET, BASELINES, AND METRICS

The evaluation uses a segment-split dataset and adapted thermal-input baselines, and describes failure conditions that delimit TAP3D’s operating range.

C DATASET, BASELINES, AND METRICS

  • The dataset contains over 160,000 samples from eight indoor environments and 11 volunteers, with segment-level splitting to prevent cross-segment data leakage.The training set contains 88,631 samples and the test set 80,011; 20% of training samples are reserved for validation.Source: C DATASET, BASELINES, AND METRICS · p. 18
  • Because no direct thermal-array point-cloud baseline exists, the study adapts RGB2Point and NeWCRF for thermal inputs and evaluates geometric accuracy with DCD and user detection with F1.NeWCRF depth predictions are back-projected into point clouds; F1 uses bounding boxes at an IoU threshold of 0.5.Source: C DATASET, BASELINES, AND METRICS · p. 18

D REPRESENTATIVE FAILURE CASES

  • Failures include missed detections under severe user overlap and nearby heat sources, false positives from reflective surfaces, and unstable or failed reconstruction near sun-heated windows or outdoors.The authors identify limited spatial resolution and temperature sensitivity as contributing constraints, and suggest improved sensors or temporal and body-structure priors as potential solutions.Source: D REPRESENTATIVE FAILURE CASES · pp. 18–19, Figure 29 · p. 19
Loading 2610.11241v1…