Source-linked AI summary
HuLiGen: Human LiDAR Generation from Parametric Body Models
Salma Galaaoui, Nermin Samet, David Picard
TL;DR
Annotated human LiDAR is scarce, and manually simulated samples do not fully capture real sensing characteristics. HuLiGen learns to generate human LiDAR from parametric body models; its synthetic-data pretraining achieves state-of-the-art pose-estimation performance and particularly strong gains in low-data and low-annotation settings.
Problem
Human LiDAR point clouds with accurate 3D pose annotations are costly to collect at scale, and simulated samples do not fully reflect real observations.
Source: §1 · p. 1Method
HuLiGen learns a flow-matching generator conditioned on parametric body models to produce human LiDAR observations from real-capture characteristics.
Source: §1 · p. 2, §3.2 · p. 3Results
HuLiGen-based pretraining achieves state-of-the-art performance on SLOPER4D and WOD, reducing MPJPE from 6.02 cm to 5.04 cm on SLOPER4D and from 6.74 cm to 5.85 cm on WOD.
Source: §1 · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
LiDAR point clouds of humans are extremely expensive to collect and annotate, thus represent a scarce resource that hinders the development of human analysis using this modality. To alleviate this scarcity, prior work relies on simulated human LiDAR, but such samples do not fully reflect the geometry and sensing characteristics of real observations. In contrast, we introduce HuLiGen, a generative model that generates human LiDAR point clouds from a parametric body model, using a point transformer trained with a flow-matching objective. We show that our generated point clouds are closer to the real capture distribution. Using HuLiGen to generate synthetic data, we propose a synthetic-only pretraining scheme for LiDAR-based HPE that achieves state-of-the-art performance, with even larger gains in low-annotation and low-data regimes, where MPJPE is reduced by up to 50%. Code, models and generated samples are available at https://github.com/valeoai/HuLiGen.
1. Introduction
HuLiGen addresses the scarcity and limited diversity of annotated human LiDAR by learning how real sensors observe parametric human bodies. Its generated samples support pose-estimator pretraining, with reported gains across full-data, low-data, and low-annotation settings.
1. Introduction
- Real human LiDAR data is difficult to annotate at scale and covers limited pose and body-shape diversity, while AMASS provides diverse body configurations without paired LiDAR observations.SLOPER4D has more than 100K frames but only 12 recorded subjects; WOD has approximately 10K object frames with 3D keypoint annotations.Source: §1 · pp. 1–2
- Ray-cast human LiDAR samples depend on manually defined rendering and noise assumptions, while real observations also reflect clothing, viewpoint, range, sensor configuration, and measurement uncertainty.Source: §1 · p. 2
- HuLiGen learns a conditional mapping from body models to real LiDAR observations, capturing sensor characteristics and deviations caused by clothing, hair, and other surface variation.This separates pose diversity supplied by mocap data from sensor realism learned from real LiDAR.Source: §1 · p. 2
- MPJPE falls from 6.02 cm to 5.04 cm on SLOPER4D and from 6.74 cm to 5.85 cm on WOD, achieving state-of-the-art performance on both datasets.The reported gains use generated human LiDAR for pretraining followed by finetuning on real data.Source: §1 · p. 2
- On WOD, pretraining reduces MPJPE by about 50% using only 1% and 10% of the available pose annotations.Source: §1 · p. 2
- With 10% and 20% of real training data, MPJPE falls by 30% and 25% on SLOPER4D and 42% and 36% on WOD, respectively.Source: §1 · p. 2
2. Related works
Prior work addresses scarce LiDAR pose labels through auxiliary image supervision, augmentation, synthetic ray casting, and stronger human-specific representations. Existing LiDAR generators, however, are not designed to condition human generation on underlying body pose and shape.
2. Related works
- LiDAR pose-estimation methods address sparse, non-uniform, and partial observations with structured representations, multimodal or temporal cues, human-specific priors, and density-aware features.Examples include 2D projections, BEV and sparse voxel features, RGB-LiDAR fusion, temporal context, and body-part or motion priors.Source: §2 · p. 2
- To expand training data, prior LiDAR-only approaches use augmentation or generate new observations from parametric bodies with ray casting and hand-designed perturbations.Image annotations and pseudo-labels are another route to reducing dependence on 3D pose labels.Source: §2 · pp. 2–3
- Existing LiDAR generators focus on scenes or localized objects, whereas the paper identifies a gap in generating humans conditioned on their underlying body pose and shape.Source: §2 · p. 3
3. Method
HuLiGen learns a conditional distribution of human LiDAR point clouds from paired body models and real observations. It parameterizes the human relative to a body-centered, yaw-canonical frame, encodes location, shape, and pose as condition tokens, and uses flow matching to generate point clouds.
3.1. Overview
- HuLiGen models the conditional distribution of a human point cloud given its body model, using flow matching to move a Gaussian point cloud toward a real LiDAR capture.A separate model is trained for each acquisition setting to capture sensor-dependent characteristics.Source: §3.1 · p. 3
3.2. Human-centric parameterization
- The method centers points on the body model in LiDAR coordinates and rotates the body into a global-yaw canonical frame, using the lateral-axis azimuth because it remains well-defined across poses.The forward-axis azimuth can become ill-conditioned when strong hip flexion makes its horizontal projection shrink toward zero.Source: §3.2 · pp. 3–4, Figure 2 · p. 3
3.3. Condition encoding
- Condition tokens encode oriented body location, shape coefficients, and local pose joints, with cyclic and Fourier features for location and separate embeddings for shape and joints.The joint tokens retain joint order and use learned positional encodings.Source: §3.3 · p. 4
3.4. 3D Point cloud flow transformer
- A point-based transformer learns the flow-matching velocity from interpolated noise and target point clouds, conditioned on the body-model tokens.The training objective regresses the predicted velocity toward the target velocity; sampling starts from Gaussian points and solves the learned differential equation with Euler steps.Source: §3.4 · pp. 4–5
- At generation time, the model can sample varying point counts without architectural changes.Source: §3.4 · p. 5
4. Experiments
Experiments evaluate HuLiGen generation and downstream pose estimation on SLOPER4D and WOD. Synthetic pretraining improves performance across standard and constrained-data settings, with state-of-the-art results on both datasets.
4. Experiments
- Experiments assess geometric similarity with Chamfer Distance and distributional realism with FPD, KPD, coverage, and nearest-neighbor accuracy.Separate models are trained for each dataset's LiDAR setup.Source: §4 · p. 5
4.1. Experimental analysis
- On WOD, HuLiGen achieves better overall generative quality than the raycasting baseline.Generation quality is evaluated using sample-level and distribution-level metrics.Source: §4.1 · p. 6, Table 2 · p. 6
- Synthetic pretraining consistently beats training from scratch on both datasets, and performance improves as the generated pretraining set grows.The pretrained model alone, without real-data finetuning, also achieves reasonable performance.Source: §4.1 · p. 6, Table 3 · p. 6
- HuLiGen-generated samples consistently outperform direct and bootstrapped raycasting samples in downstream 3D HPE.The comparison uses approximately 48k AMASS meshes for each pretraining approach.Source: §4.1 · p. 6, Table 4 · p. 6
4.2. Comparison to the-state-of-the-art
- HuLiGen achieves state-of-the-art LiDAR-based 3D HPE performance on both SLOPER4D and WOD.The comparison considers methods without additional large-scale real-world training data.Source: Table 5 · p. 7, §4.2 · p. 7
4.3. Low-annotation regime
- More than 50% lower error than training from scratch is achieved on WOD with only 1% or 10% of pose annotations.The pipeline also outperforms raycasting baselines and GC-KPL in both low-annotation settings.Source: §4.3 · p. 7, Table 6 · p. 7
4.4. Low-data regime
- Approximately 30% lower MPJPE and PA-MPJPE on SLOPER4D and 45% on WOD are reported with only 10% or 20% of real training data.The FM model is trained on the same limited real-data subset used to generate about 48k AMASS samples for pretraining.Source: §4.4 · p. 7, Table 7 · p. 7
5. Conclusion
HuLiGen learns human LiDAR characteristics from real data, producing more realistic observations than ray casting and improving HPE pretraining, particularly in low-data and low-annotation settings.
5. Conclusion
- Pretraining on HuLiGen samples improves HPE performance, with particularly strong gains in low-data and low-annotation regimes.Source: §5 · p. 7
- Real and generated observations capture clothing and hair beyond the bare body surface, unlike raycasted results.Source: Figure 4 · p. 8
- HuLiGen generations have median surface distances of 1.41 cm on SLOPER4D and 4.5 cm on WOD, versus 1.17 cm and 3.0 cm for real observations.The larger WOD gap is attributed mainly to ground-level noise; ray casting has a median surface distance of 0 cm on both datasets.Source: Figure 5 · p. 8
6. Complementary details on conditioning and embedding
The conditioning ablations use SMPL annotations and sensor-frame point clouds to examine conditioning and embedding choices. The supplementary details describe canonicalization, pose representations, and translation encoding.
6. Complementary details on conditioning and embedding
- The model conditions on sensor-frame point clouds P, SMPL shape β, local pose θ, and translation t_l; ablations vary shape inclusion, pose representation, and canonicalization.Source: §6 · p. 11
- Canonicalization derives yaw ψ from the projected SMPL root lateral axis and applies the resulting rotation consistently to geometric quantities.Using the lateral rather than forward axis avoids ill-defined azimuth when a crouched pose makes the forward direction approach vertical.Source: §6 · p. 11
- Translation azimuth is encoded as (sin ϕ, cos ϕ) alongside log ρ and z.Source: §6 · p. 11
- The pose representation uses SMPL forward kinematics joint positions, while an ablation uses 6D joint rotations with cyclic embeddings and learned positional encoding.Source: §6 · p. 11
- Table 8 summarizes each conditioning variant's alignment, shape, and joint-coordinate choices, representation dimensionality, and conditioning-token count.Source: Table 8 · p. 12