PaperlayerView exact revision on arXiv

Source-linked AI summary

Shoe Style-Invariant and Ground-Aware Learning for Dense Foot Contact Estimation

Daniel Sungho Jung, Kyoung Mu Lee

arXiv:2511.22184v2cs.CV

TL;DR

Dense foot-contact estimation matters for understanding foot–environment interaction, but varied shoe styles and weak ground cues make it challenging. FECO combines style-invariant learning with ground-aware representations and reports state-of-the-art results across diverse appearances. The authors conclude that it surpasses prior methods on COFE and demonstrates robustness and generalization.

  • Problem

    Dense foot-contact estimation matters for understanding foot–environment interaction, but varied shoe styles and weak ground cues make it challenging.

    Source: §1 · p. 1

  • Method

    FECO uses shoe style–content randomization and ground-aware learning to estimate dense foot contact.

    Source: §1 · p. 2

  • Results

    FECO surpasses previous state-of-the-art methods on COFE and demonstrates robustness and generalization across diverse shoe and ground appearances.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    Capturing where feet contact the environment can support interpretation of human movement and realistic physical-behavior modeling.

    Source: §1 · p. 1

    The method mainly operates on cropped foot images, which may provide little useful information when feet are fully occluded.

    Source: S10. Limitations and societal impacts · p. 14

Abstract

from arXiv · show

Foot contact plays a critical role in human interaction with the world, and thus exploring foot contact can advance our understanding of human movement and physical interaction. Despite its importance, existing methods often approximate foot contact using a zero-velocity constraint and focus on joint-level contact, failing to capture the detailed interaction between the foot and the world. Dense estimation of foot contact is crucial for accurately modeling this interaction, yet predicting dense foot contact from a single RGB image remains largely underexplored. There are two main challenges for learning dense foot contact estimation. First, shoes exhibit highly diverse appearances, making it difficult for models to generalize across different styles. Second, ground often has a monotonous appearance, making it difficult to extract informative features. To tackle these issues, we present a FEet COntact estimation (FECO) framework that learns dense foot contact with shoe style-invariant and ground-aware learning. To overcome the challenge of shoe appearance diversity, our approach incorporates shoe style adversarial training that enforces shoe style-invariant features for contact estimation. To effectively utilize ground information, we introduce a ground feature extractor that captures ground properties based on spatial context. As a result, our proposed method achieves robust foot contact estimation regardless of shoe appearance and effectively leverages ground information. The codes are available at https://github.com/dqj5182/FECO_RELEASE.

1. Introduction

Dense foot contact estimation addresses the limits of joint-level contact proxies and inaccurate foot predictions from general dense-body methods. FECO combines shoe style-invariant learning with ground-aware features and reports state-of-the-art performance across diverse shoe and ground appearances.

1. Introduction

  • Existing methods often use joint-level contact and zero-velocity heuristics, while dense body-contact methods still predict foot contact inaccurately.The paper motivates a dedicated dense foot-contact model.Source: §1 · p. 1
  • Shoe-style diversity can encourage reliance on misleading appearance cues, while weak ground textures make contact regions difficult to infer without ground geometry.Occlusion, viewpoint, and illumination changes further motivate geometric and physical context.Source: §1 · pp. 1–2
  • FECO combines low-level and shoe style–content randomization with ground modeling using pixel height maps and ground normals.Adversarial adapters enforce shoe-style invariance, while style randomization exposes the model to varied appearances.Source: §1 · p. 2
  • FECO achieves state-of-the-art performance across diverse shoe and ground appearances.Source: §1 · p. 2

2. Related works

  • Earlier foot-contact approaches emphasize joint-level estimates or zero-velocity constraints, while dense-contact methods include visual prediction and in-the-wild annotation.The paper describes these approaches as distinct lines of prior work.Source: §2 · p. 2
  • Prior style methods separate or randomize image style and content; FECO uses a SagNets variant with shoe styles from UT Zappos50K.Source: §2 · p. 3
  • Prior ground representations use ground geometry for prediction, and FECO applies Pixel Height to fine-grained foot–ground interaction.The related work notes that many cited driving methods rely on predetermined camera extrinsics.Source: §2 · p. 3

3. Method: FECO

FECO processes clean and style-randomized images end-to-end, creates shoe-style-invariant and ground-aware representations, and fuses them to predict dense foot contact. Its method combines shoe randomization, ground-feature learning, spatial attention, and multi-level contact supervision.

3.1. Low-level style randomization

  • During training, FECO processes one clean image and two ProRandConv style-randomized images with identical operations; inference uses only the clean image.The randomized images are produced by independently sampling convolution weights, deformable-convolution offsets, and affine parameters.Source: §3.1 · p. 3

3.2. Shoe style-content randomization

  • External shoe images drive parallel content- and style-randomization branches: the former supports adversarial training, while the latter yields a shoe style-invariant feature.The content-randomized branch preserves external shoe content while expressing it in the input image's style; the style-randomized branch incorporates style from an external shoe image.Source: §3.2 · pp. 3–4

3.3. Ground feature learning

  • A ground encoder predicts dense pixel-height maps from multi-level features and ground normals from final-level features, with ground learning applied to both randomized branches.For ground-normal decoding, the foot region is suppressed using the predicted foot segmentation mask to prevent shortcut learning from foot orientation.Source: §3.3 · pp. 4–5

3.4. Spatial attention

  • Spatial attention assigns spatially varying weights to randomized and corresponding ground features, then fuses them into contact representations.The fused representations are transformed by a contact adapter before contact decoding.Source: §3.4 · p. 5

3.5. Foot contact decoder

  • A Transformer decoder combines a contact token with image features and outputs sigmoid contact probabilities for 265 foot vertices using foot-specific regressors.The decoder adapts the output dimension from HACO's 778-vertex MANO hand mesh to a 265-vertex foot mesh.Source: §3.5 · p. 5

3.6. Final outputs and loss functions

  • FECO is trained end-to-end with main, style, style-adversarial, mask, and ground losses, using multi-level main predictions plus 11 foot-region means and 3 OpenPose foot keypoints for supervision.Ground loss sums pixel-height and ground-normal losses and is applied symmetrically to the main and style branches.Source: §3.6 · p. 6

4. Dataset: COFE

COFE adds manually annotated in-the-wild binary foot-contact labels on keypoints from 2D datasets, addressing the lack of in-the-wild data in large-scale 3D contact datasets.

4. Dataset: COFE

  • COFE contains 31,598 training images and 1,567 test images after images with fewer than two valid foot joints were removed.The test set uses InstaVariety, selected for its diverse contact distribution and video sequences that enable comparison with existing methods.Source: §4 · p. 6
  • The dataset manually annotates binary foot-contact labels on human foot keypoints from OpenPose, InstaVariety, PennAction, and MPII.OpenPose was excluded from the test set because it predominantly contains fully contacted feet.Source: §4 · p. 6
  • COFE labels snowboards as contact, skis as non-contact unless they touch a surface, and clothing as non-contact.These rules address ambiguous cases while excluding foot–clothing contact as having limited relevance to foot–ground or foot–object interaction.Source: §4 · p. 6

5. Implementation details

The implementation uses a pretrained ViT-Huge backbone and trains FECO with standard image augmentations and additional degraded-input perturbations. The section also identifies qualitative and ablation comparisons.

5. Implementation details

  • FECO uses publicly released ImageNet-pretrained ViT-Huge weights and random scaling, cropping, and rotation augmentations.Low-resolution, noise, and blur perturbations are also introduced to improve robustness against degraded inputs.Source: §5 · pp. 6–7
  • Figure 3 qualitatively compares FECO with POSA [16], BSTRO [18], and DECO [49] on MOYO [50], RICH [18], and Hi4D [58].Red circles mark example regions where FECO outperforms prior methods.Source: Figure 3 · p. 7
  • Tables 2 and 3 identify ablations of low-level randomization and shoe randomization on MMVP [60], respectively.The supplied passages identify the tables' scope but do not provide their cell values.Source: Table 2 · p. 7, Table 3 · p. 7
  • Training uses AdamW with a 10^-5 learning rate, batch size 4, and 10 epochs on one NVIDIA A6000 GPU.The learning rate is decayed by 0.9 after epochs 5 and 10.Source: §5 · p. 7

6. Experiments

The experiments evaluate FECO across datasets and contact metrics, examine component and training-data ablations, and compare its dense and joint-level estimates with prior methods.

6.1. Datasets

  • The experiments use 10 datasets spanning foot-scene, foot-object, foot-ground, and foot-body interactions, alongside COFE in-the-wild training samples.MMVP is the main evaluation dataset; UT Zappos50K supplies shoe images for content randomization.Source: §6.1 · p. 7

6.2. Evaluation metrics

  • The evaluation reports per-sample precision, recall, and F1 from thresholded per-vertex probabilities, averaged over the dataset.Samples without positive contact are excluded because recall and F1 are not defined for them.Source: §6.2 · p. 7

6.3. Ablation study

  • Low-level randomization raises MMVP precision by 3.5%, recall by 5.0%, and F1-score by 4.0% over the variant without it.Content-only randomization yields the highest recall but lower precision; combining style and content randomization yields the highest F1-score.Source: §6.3 · pp. 7–8

6.4. Comparison with state-of-the-art methods

  • FECO achieves the highest precision, recall, and F1-score among POSA [16], BSTRO [18], and DECO [49] for dense contact on MMVP [60].On COFE joint-level evaluation, FECO also achieves the best precision, recall, and F1-score despite being the only compared method without temporal information.Source: Table 7 · p. 8, §6.4 · p. 8, Table 8 · p. 8

6.3. Ablation study

  • Adding COFE training data improves precision, recall, and F1-score over training with only 3D mocap datasets.The authors attribute the gains to diverse in-the-wild appearances and foot interactions that complement mocap data and reduce overfitting to its appearances.Source: Table 5 · p. 7, §6.3 · p. 8
  • Adding ground normals improves F1-score, pixel height maps lift it further, and spatial attention between randomized and ground features gives the best result.The ablation evaluates ground-aware learning on MMVP [60].Source: Table 4 · p. 7, §6.3 · p. 8

7. Conclusion

The conclusion presents FECO as a dense foot-contact framework combining shoe style-invariant learning with ground-aware learning. It reports that FECO, with COFE, surpasses prior state-of-the-art methods and demonstrates robustness and generalization.

7. Conclusion

  • FECO combines shoe style–content randomization using an external shoe image dataset with ground-aware learning.Ground-aware learning supervises pixel height maps and ground normals to capture fine-grained and global foot–ground interactions.Source: §7 · p. 8
  • The authors report that FECO with the COFE dataset surpasses previous state-of-the-art methods by a significant margin.The conclusion also reports robustness and generalization.Source: §7 · p. 8
  • The supplementary material lists additional technical details and results, including limitations and societal impacts.Its listed topics include COFE configuration, training details, more datasets and backbones, computational requirements, and qualitative results.Source: §7 · p. 9

S1. Configuration of COFE dataset

COFE combines several 2D keypoint datasets to provide in-the-wild foot-contact data, with a more balanced contact distribution than the other training datasets used for FECO. Its annotations cover three keypoints, and evaluation uses the InstaVariety test split because OpenPose’s test set is biased toward full contact.

S1. Configuration of COFE dataset

  • COFE aggregates foot samples from OpenPose, PennAction, InstaVariety, and MPII, contributing 43.3%, 38.7%, 11.9%, and 6.1% of training samples, respectively.Source: S1. Configuration of COFE dataset · p. 9
  • The dataset filters unclear foot visibility, static video clips, and multi-person videos; MPII is used primarily for training because its videos are generally under three seconds.Source: S1. Configuration of COFE dataset · p. 9
  • OpenPose’s test set is unsuitable for robust evaluation because nearly all samples show fully contacted feet; COFE evaluation therefore uses its InstaVariety test split.Uniformly predicting full contact on OpenPose yields precision 0.712, recall 0.858, and F1-score 0.760.Source: S1. Configuration of COFE dataset · p. 9
  • 49,927 contacting and 44,201 non-contacting cases appear in COFE training; contact rates are 63.4% for big toes, 53.3% for small toes, and 42.5% for heels.Source: S1. Configuration of COFE dataset · p. 9
  • COFE complements other FECO training datasets by providing a more balanced contact distribution and more even joint coverage.The other datasets skew toward non-contact and underrepresent the heel.Source: S1. Configuration of COFE dataset · pp. 9–10

S2. Details of training FECO

FECO adapts SagNets-style training for dense foot contact estimation using a single end-to-end optimizer. Gradient detaching and module freezing preserve separate training roles for segmentation, content, and style-adversarial components.

S2. Details of training FECO

  • FECO replaces SagNets’ three-optimizer strategy with a single end-to-end optimizer, jointly training the backbone, contact decoders, ground-aware decoders, and style branch.Gradient detaching and module freezing reproduce relevant training roles; operations after low-level style randomization run on the original and two randomized images in parallel.Source: S2. Details of training FECO · p. 10
  • The foot segmentation mask decoder is trained solely by its mask loss, with later modules detached from the mask’s gradient flow.Source: S2. Details of training FECO · p. 11
  • For style-adversarial training, FECO detaches the content-invariant input and freezes the style-branch contact decoder so the adversarial loss trains only the adapters.The adapters replace SagNets’ backbone affine parameters in this training role.Source: S2. Details of training FECO · p. 11

S3. Details of joint definitions

The joint definition partitions the foot mesh into eleven functional regions, providing denser coverage than the three OpenPose foot keypoints. Each joint is represented by the mean coordinates of the vertices in its region.

S3. Details of joint definitions

S4. Details of dense foot contact labels

Dense foot-contact labels are derived from distances between foot meshes and ground meshes or fitted ground planes. The procedure accommodates datasets with and without scene meshes, using dataset-specific distance tolerances or existing annotations.

S4. Details of dense foot contact labels

  • Dense contact labels are obtained by thresholding distances between 3D foot meshes and ground meshes, allowing use of motion-capture datasets without scene meshes.Source: S4. Details of dense foot contact labels · p. 11
  • For captures without scene meshes, the ground plane is fit by regression to candidate points from the lowest body and, when available, object-mesh vertices across frames.The plane is expressed as h = ag1 + bg2 + c.Source: S4. Details of dense foot contact labels · p. 12
  • For datasets with scene meshes, near-ground vertices are selected and repeated RANSAC plane fits yield the plane with the widest spatial support.Source: S4. Details of dense foot contact labels · p. 12
  • Contact is assigned when the magnitude of a foot-vertex distance to the ground plane falls within a dataset-specific tolerance; Hi4D and RICH use their provided annotations.Tolerances range from 1 cm for MOYO to 5 cm for BEHAVE and InterCap.Source: S4. Details of dense foot contact labels · p. 12

S5. Visualization of ground-aware learning

FECO’s ground-aware representations closely match ground-truth height maps and normals, including for tilted feet. Results span diverse datasets and backbone configurations, with reported trade-offs between accuracy and computational cost.

S5. Visualization of ground-aware learning

  • Predicted pixel-height maps closely match ground truth, and predicted ground normals align with ground-truth normals, including for tilted feet.Height maps show minor smoothing and reduced detail around individual toes.Source: S5. Visualization of ground-aware… · p. 12

S6. Quantitative results on more datasets

  • FECO achieves an F1-score of 0.795 on BEHAVE and reports reasonable performance across RICH, MOYO, and Hi4D interaction settings.Performance drops on MOYO, which includes out-of-distribution poses and limited scene-contact supervision, though FECO still detects plausible contacts.Source: S6. Quantitative results on more… · pp. 12–13, Table S5 · p. 13

S7. Quantitative results on different backbones

  • ViT-H achieves the highest MMVP F1-score at 0.577 among tested backbones, while ViT-based architectures consistently outperform convolutional alternatives.ViT-B has the highest recall (0.650); ResNet-152 leads the convolutional models with an F1-score of 0.533.Source: S7. Quantitative results on different… · p. 13, Table S4 · p. 13

S8. Computational requirements

  • Backbone choices span high-capacity but memory-intensive ViT-H and efficient variants, including ViT-B and ViT-S with under 13 GB training memory and speeds above 23 fps.ResNet-18 and ResNet-34 require fewer than 5 GB of training memory and exceed 40 fps; ResNet-50 exceeds 24 fps. ViT-H requires over 34 GB for training and nearly 6.5 GB for inference.Source: S8. Computational requirements · p. 13

S9. More qualitative results

Qualitative comparisons show FECO’s predictions across shoe styles and datasets, while highlighting failure patterns in prior methods. The authors note limitations around cropped inputs, occlusion, and the absence of temporal or inter-foot modeling.

S9. More qualitative results

  • BSTRO misses visible heel contacts, while POSA can hallucinate contact for a non-contacting foot; FECO incorporates pixel-height maps and ground normals for ground-aware reasoning.The examples span Hi4D, MMVP, RICH, MOYO, and COFE; FECO’s predictions remain accurate across diverse shoe styles in the shown samples.Source: S9. More qualitative results · pp. 13–14, Figure S7 · p. 17, Figure S8 · p. 18

S10. Limitations and societal impacts

  • Foot-cropped images may provide little useful information under full occlusion; video-based estimation and modeling interactions between feet are proposed future directions.The authors also identify privacy, safety, and sustainability risks for in-the-wild deployment, and state that FECO is not designed as a diagnostic tool.Source: S10. Limitations and societal impacts · p. 14
Loading 2511.22184v2…