PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Landmark Extraction: A Framework for Robust Geometric Feature Construction in Structured Image Classification

Saravana Mauree, Sakshi Arya

arXiv:2609.00634v2cs.CV

TL;DR

Classifier comparisons leave open what representation a classifier should receive, while raw pixels are high-dimensional, nuisance-sensitive, and difficult to interpret. The paper evaluates coordinate, distance, angle, and hybrid landmark feature maps, using perturbation and ablation analyses to examine representation behavior. On HaGRID, hybrid achieved the highest mean accuracy—85.2% with multinomial and 87.8% with random forest—while coordinate-based representations deteriorated sharply.

  • Problem

    Classifier comparisons leave open what representation a classifier should receive, while raw pixels are high-dimensional, nuisance-sensitive, and difficult to interpret.

    Source: §1 · p. 1

  • Method

    The paper evaluates coordinate, distance, angle, and hybrid landmark feature maps, using perturbation and ablation analyses to examine representation behavior.

    Source: §1 · p. 1, §3 · p. 3, §5 · p. 5

  • Results

    On HaGRID, hybrid achieved the highest mean accuracy—85.2% with multinomial and 87.8% with random forest—while coordinate-based representations deteriorated sharply.

    Source: §5 · pp. 4–5

  • Takeaways & Limitations

    Feature construction is a fundamental modeling decision because each map encodes assumptions about which geometry to preserve and nuisance variation to suppress.

    Source: §1 · pp. 1–2

    HaGRID varies along several axes simultaneously, so its comparison cannot isolate why representations differ; robustness is tested with synthetic perturbations on controlled landmarks.

    Source: §5 · p. 5

Abstract

from arXiv · show

Much of the literature on structured image recognition has disproportionately focused on the comparison of classification algorithms. Rather than investigating which classifier performs best, this paper instead asks: what should a classifier know before it ever makes a prediction? In structured vision problems such as gesture recognition, facial expression categorization, and medical image analysis, discriminative information lies less in individual pixels and more in spatial relationships between semantic parts. Raw pixel spaces are high-dimensional, sensitive to nuisance variation, and often obfuscate the geometric structures that make visual tasks interpretable. Landmark extraction provides one form of dimension reduction, but it does not by itself determine the information preserved. This paper studies the post-landmark feature map as the central object of analysis and proposes a systematic framework for constructing and interpreting landmark-derived representations as an, informed, feature-based "dimension reduction" step. Using static hand gesture recognition as a case study, we evaluate coordinate, distance, angle, and hybrid representations through perturbation and ablation experiments. The results show that visually variable data exposes substantial gaps between raw coordinate features and their geometrically invariant counterparts, while hybrid representations achieve the strongest overall performance by combining complementary geometric components. These findings frame feature construction as a fundamental modeling decision and ultimately suggests that the question of what representation should a classifier learn from is one worth asking. The code used for feature construction and evaluation is available at https://github.com/ShivMaureeCWRU/Feature_based_dimension_reduction

1. Introduction

The paper reframes post-landmark feature construction as a central modeling choice: representations determine which task-relevant geometry is preserved and which nuisance variation is suppressed. It develops a systematic way to evaluate these choices, using static hand gestures as a controlled case study.

1. Introduction

  • Post-landmark representation is the central modeling object: each feature map encodes assumptions about preserved geometry and removed nuisance variation.The paper frames this as problem-informed dimension reduction based on effective variation, not necessarily a strict reduction in feature count.Source: §1 · pp. 1–2
  • Static hand gesture recognition isolates representation choice by excluding detector design, classifier comparison, and temporal sequence modeling.The study compares representations under a common experimental protocol.Source: §1 · p. 2
  • Hybrid representations combine scaled coordinates, normalized pairwise distances, and angles to encode global layout, relative spacing, and local hand geometry.The paper identifies hybrids as especially advantageous under visually variable conditions.Source: §1 · p. 2
  • The same representation question applies in medical imaging, where anatomical shape and spatial relationships can matter after structures are extracted.The paper also notes that interpretability matters when model reasoning must be explained.Source: §1 · p. 2
  • The framework formalizes feature construction, evaluates landmark-derived representations, and uses perturbation and ablation analyses to assess robustness, invariance, and component contributions.These analyses also guide hybrid representation construction.Source: §1 · p. 2

2. Problem Formulation

The formulation separates image classification into landmark extraction, feature mapping, and classification. It uses this decomposition to shift the modeling emphasis from comparing classifiers under one fixed representation to studying the feature map.

2. Problem Formulation

  • Classification is expressed as three stages: map each image to landmarks, transform landmarks with a representation-specific feature map, then classify the transformed features.The landmark map produces 21 detected hand landmarks, and each representation can have its own feature dimension.Source: §2 · p. 2
  • MediaPipe Hands supplies the landmark skeleton, while the feature map transforms it into a representation-specific dataset paired with the original class labels.The formulation distinguishes pixel, landmark, and feature representations.Source: §2 · p. 2
  • Classifiers are trained separately for each representation, shifting the study from classifier comparison under a fixed representation to systematic study of the feature map.This representation-centric perspective is made explicit in Algorithm 2.Source: §2 · p. 2

3. Landmark feature representations

The section treats landmark-derived feature maps as choices about which task-relevant geometry to preserve and which nuisance variation to suppress. It compares coordinate, distance, angle, and hybrid representations through their different invariance assumptions.

3. Landmark feature representations

  • The hybrid representation concatenates scaled coordinates, normalized distances, and angles to combine global layout, relative spacing, and local angular geometry.These components preserve different aspects of hand geometry.Source: §3 · p. 3
  • Within an isolated gesture class, hands appear at different frame locations and sizes; translated coordinates remove location dependence, while scaled coordinates are unchanged under positive uniform scaling.The section treats location and apparent size as nuisance variation for gesture classification.Source: §3 · p. 3, Figure 2 · p. 3
  • Pairwise distances are translation-invariant but not scale-invariant; normalized distances remove scale dependence, while angles are invariant to translation and positive uniform scaling.The angle features depend on ratios of inner products and norms.Source: §3 · p. 3
  • Rotation is deliberately preserved because hand orientation can distinguish gestures, although distance and angle features encode rotation-invariant internal geometry.The framework does not explicitly impose complete affine invariance.Source: §3 · pp. 3–4
  • Feature maps encode assumptions about which geometric information to preserve and which transformations to suppress, rather than simply reducing the number of variables.The stated goal is to retain gesture-relevant geometry while reducing sensitivity to irrelevant effects.Source: §3 · pp. 3–4

4. Study Design and Evaluation Protocol

The evaluation compares landmark representations across datasets with differing visual variability, using a fixed train-test convention and two classifiers. It also tests robustness to synthetic perturbations and uses paired statistical comparisons across representations.

4. Study Design and Evaluation Protocol

  • Three datasets span varying visual conditions: a self-collected webcam dataset, closely cropped SignAlphaSet, and a pragmatic HaGRID subset.The self-collected data provides a reasonably controlled comparison, while SignAlphaSet uses a highly controlled capture environment.Source: §4 · p. 4
  • Each dataset and representation uses the same-seed 80/20 stratified train-test split, with hyperparameters selected by stratified five-fold cross-validation.The split is described as a fixed evaluation convention, not an optimality claim.Source: §4 · p. 4
  • Multinomial logistic regression provides a linear baseline, while random forests serve as a model of nonlinear feature interactions in a controlled representation comparison.The authors note that these classifiers do not exhaust the possible model space.Source: §4 · p. 4
  • A synthetic perturbation experiment on controlled landmark data tests whether the representations’ invariance properties translate to empirical robustness.Held-out classification accuracy is the primary evaluation metric.Source: §4 · p. 4
  • Repeated stratified five-fold cross-validation supplies paired accuracy measurements; Friedman tests and, when significant, Holm-corrected Wilcoxon signed-rank tests compare representations.The Wilcoxon tests are used for post-hoc pairwise comparisons.Source: §4 · p. 4

5. Results and Discussion

Representation choice mattered most on visually variable HaGRID: geometrically motivated maps outperformed raw coordinates, while controlled data obscured differences and perturbation tests exposed distinct robustness trade-offs.

  • Controlled datasets yielded near-perfect accuracy across representations, but on HaGRID raw coordinates deteriorated sharply while geometrically motivated maps remained comparatively stable.Source: §5 · p. 4, Figure 4 · p. 4
  • On HaGRID, accuracy followed φdist < φnorm−dist < φhybrid, while φangle underperformed other geometric alternatives despite translation and scale invariance.The paper attributes the angle representation’s weakness to discarding spacing and broader hand shape information that matter for some gestures.Source: §5 · p. 4
  • On synthetic perturbations, φxy fell from 99.7% on clean landmarks to 90.1% at level 1, 58.3% at level 2, and 28.2% at the strongest level, while φscale, φnorm−dist, and φhybrid stayed stable.Source: Figure 6 · p. 5, §5 · p. 5
  • Under translation-only perturbations φxy dropped 74.0%; under scale-only perturbations φxy and φtrans dropped 50.2% and 4.0%, and under combined perturbations they dropped 71.9% and 3.9%.φscale, φnorm−dist, and φhybrid remained essentially unchanged under combined translation and scale; φangle was stable to those transformations but decayed when landmark noise was implicit.Source: Figure 7 · p. 5, §5 · p. 5
  • φangle’s stability under decomposed translation and scale perturbations did not extend to landmark noise, which can alter joint-vector directions, especially for nearly aligned segments.Source: §5 · p. 5
  • Across 50 paired cross-validation measurements on HaGRID, φhybrid reached 85.2% mean accuracy with multinomial and 87.8% with random forest; all 28 pairwise comparisons were significant for both classifiers.Source: §5 · p. 5

6. Building A Hybrid Representation

The paper tests whether hybrid features contribute complementary information through repeated cross-validation ablations. The results identify normalized distances and scaled coordinates as the largest contributors to hybrid accuracy.

6. Building A Hybrid Representation

  • Untouched 𝜙hybrid achieved the highest mean repeated cross-validation accuracy for both classifiers.Figure 8 presents accuracy across ablations.Source: Figure 8 · p. 6, §6 · p. 6
  • Removing 𝜙norm−dist reduced multinomial accuracy by 3.44% and random-forest accuracy by 1.90%; removing 𝜙scale reduced them by 1.43% and 1.57%, respectively.Source: §6 · p. 6
  • Removing 𝜙angle produced almost no loss because its local finger-articulation information was already recoverable from coordinate and distance components.The authors connect this pattern to perturbation results: hybrid gains reflect complementary information that preserves robustness to nuisance variation.Source: §6 · p. 6
  • For other structured-classification problems, candidate geometric features can be combined provisionally, then evaluated through perturbation, ablation, cross-validation, and interpretability.When relevant geometry is unclear, features can be added or removed incrementally.Source: §6 · p. 6

7. Illustrative Transfer to Anatomical Landmarks

The paper illustrates the framework’s portability with lateral cephalometric radiographs, while explicitly treating this extension as an illustration rather than a second-domain benchmark.

7. Illustrative Transfer to Anatomical Landmarks

  • The cephalometric extension demonstrates portability beyond hand landmarks and MediaPipe, but is not presented as a full benchmark in a second domain.Source: §7 · p. 6
  • A lightweight convolutional detector trained from expert annotations maps each radiograph to 29 two-dimensional anatomical landmark coordinates.Landmarks were represented as spatial heatmaps during training and recovered as coordinate predictions at inference.Source: §7 · p. 6
  • The framework’s feature constructions can be applied directly to the extracted anatomical landmarks, with Figure 9 illustrating the pipeline on a held-out radiograph.Source: Figure 9 · p. 6, §7 · p. 7

8. Conclusion

The conclusion presents post-landmark feature construction as a problem-informed dimension-reduction step. It emphasizes evaluating representations by what structure they preserve, what variation they suppress, and how their components contribute to prediction.

8. Conclusion

  • Hybrid representations’ advantage came from complementary normalized hand layout and relative landmark spacing, while raw coordinates deteriorated under nuisance variation.Perturbation experiments linked robustness to the nuisance variations each representation failed to suppress.Source: §8 · p. 7
  • Representation quality requires assessing preserved structure, suppressed variation, and component-level predictive contribution, not accuracy or dimensionality alone.The paper presents perturbation and ablation as practical methods for constructing and validating representations after semantic-structure extraction.Source: §8 · p. 7
  • The study focuses on static representations, finite-dimensional Euclidean feature vectors, and extracted landmarks as the geometric input layer.Future directions include temporal modeling, kernel-based similarity learning, and studying structured landmark-extraction errors.Source: §8 · p. 7

CRediT authorship contribution statement

The authors divide the listed contributions, sharing conceptualization and methodology while assigning other roles individually.

CRediT authorship contribution statement

Loading 2609.00634v2…