PaperlayerView exact revision on arXiv

Source-linked AI summary

Structure-aware Keypoint Localization for Videofluoroscopic Swallowing Study

Kai Zhou, Chuanshen Chen, Runhao Zeng, Meng Dai, Yifan Yang, Jinwu Hu, Daiyuan Li, Mingkui Tan, Fei Liu

arXiv:2610.07726v1cs.CV

TL;DR

The paper addresses limited anatomical annotations and unlabeled non-swallowing data in VFSS keypoint localization. It introduces VFSSKep and S3KL, which use structure-aware learning and representation consistency to address spatial bias; with 25% labeled data, S3KL outperforms fully supervised baselines using 100% labels.

  • Problem

    Precise keypoint localization supports VFSS kinematic analysis, but existing datasets have limited anatomical coverage and do not incorporate non-swallowing segments as unlabeled data.

    Source: §I · p. 2, A. Videofluoroscopic Swallowing Study · p. 7

  • Method

    The paper introduces VFSSKep with soft-palate annotations and unlabeled non-swallowing data, alongside S3KL, which combines structure-aware learning with structural representation consistency to address spatial bias.

    Source: §I · p. 2, A. Preliminaries and Method Overview · p. 3

  • Results

    With 25% labeled data, S3KL outperforms fully supervised baselines using 100% labels.

    Source: §I · p. 2, §V · p. 6

  • Takeaways & Limitations

    VFSSKep and S3KL provide a data-efficient approach for clinical kinematic analysis.

    Source: §V · p. 6

    VFSSKep remains modest in scale and diversity, and S3KL operates on static frames without explicitly modeling temporal swallowing dynamics.

    Source: B. Limitations and Future Directions · p. 10

Abstract

from arXiv · show

Videofluoroscopic Swallowing Study (VFSS) is one of the gold standard for diagnosing swallowing disorders, providing dynamic X-ray imaging of the swallowing process. Automated kinematic analysis in VFSS relies fundamentally on precise anatomical keypoint localization. However, existing studies focus on limited keypoints (e.g., cervical vertebrae or the hyoid) and overlook critical regions such as the soft palate, while annotating only active swallowing segments and ignoring abundant non-swallowing data, resulting in poor data efficiency. Moreover, leveraging this unlabeled data via standard semi-supervised learning is suboptimal, as generic methods are prone to spatial bias. In medical X-rays with fixed layouts, models tend to memorize absolute coordinates rather than understanding anatomical structures. To tackle these challenges, we introduce VFSSKep, a novel dataset that extends annotations to the soft palate and incorporates large-scale unlabeled data. We further propose S$^3$KL, a Structure-aware Semi-Supervised Keypoint Localization framework designed to overcome spatial bias. It integrates a Structure-Aware Learning strategy to extract high-resolution structural cues for structure-aware representation learning, and a Structural Representation Consistency Learning strategy with block shuffling to enforce invariant structural recognition. Experiments show our method achieves state-of-the-art semi-supervised performance, even with unlabeled and 25% labeled data surpassing fully supervised learning with 100% labeled data. Code and data will be made publicly available at: https://github.com/kaai520/S3KL.

I. INTRODUCTION

The paper addresses limited anatomical coverage and data use in VFSS keypoint localization, alongside spatial bias in generic semi-supervised learning. It introduces the VFSSKep dataset and S3KL framework, and reports that S3KL with 25% labeled data outperforms fully supervised baselines using 100%.

I. INTRODUCTION

  • Existing VFSS benchmarks omit soft-palate landmarks and most non-swallowing frames, leaving limited anatomical coverage and substantial video data unused.Annotations typically cover only brief active-swallowing windows of 2–4 seconds.Source: §I · p. 1
  • Generic semi-supervised methods can memorize absolute keypoint coordinates instead of anatomical structure, creating spatial bias in fixed-layout, low-contrast X-rays.The paper identifies global geometric perturbations as a common consistency strategy in prior methods.Source: §I · p. 1
  • VFSSKep extends keypoint annotations to the soft palate and adds large-scale non-swallowing segments as unlabeled data.The dataset is presented as a benchmark for semi-supervised VFSS keypoint localization.Source: §I · p. 2
  • S3KL combines Structure-Aware Learning for dense anatomical alignment with Structural Representation Consistency Learning, which disrupts spatial context and restores feature positions.The framework uses foundation-model features to recover high-resolution anatomical details.Source: §I · p. 2
  • With 25% labeled data, S3KL outperforms fully supervised baselines using 100% labels.The paper describes this as evidence of data efficiency and generalization capability.Source: §I · p. 2

II. VFSS KEYPOINT DATASET CONSTRUCTION

VFSSKep is a dataset for semi-supervised keypoint localization that adds soft-palate landmarks and non-swallowing frames. It provides labeled active-swallowing data and a separate unlabeled set, with subjects kept separate across data splits.

II. VFSS KEYPOINT DATASET CONSTRUCTION

  • VFSSKep includes 13,895 labeled active-swallowing frames and 31,536 unlabeled non-swallowing frames.The labeled data are split into 8,302 training, 1,518 validation, and 4,075 test images, with distinct individuals separated across subsets.Source: §II · p. 2
  • The dataset annotates eight landmarks: three on the hyoid, two on C2 and C4, and three on the soft palate.The soft-palate points are anterior, midpoint, and inferior.Source: §II · p. 2
  • Unlike prior datasets, VFSSKep includes soft-palate keypoints for oral-phase analysis and unlabeled non-swallowing frames for a semi-supervised setting.The dataset is designed to reflect clinical label scarcity.Source: §II · p. 3

A. Preliminaries and Method Overview

The section formulates semi-supervised keypoint localization using labeled and unlabeled data, with each keypoint represented by a Gaussian heatmap. S3KL addresses VFSS’s fixed-layout spatial-bias challenge through structure-aware learning and consistency under Block Shuffle.

A. Preliminaries and Method Overview

  • Each ground-truth keypoint is represented by a Gaussian heatmap indicating its probability distribution.Source: A. Preliminaries and Method Overview · p. 3
  • Semi-supervised localization trains a student on labeled data with supervised loss and on unlabeled data with an unsupervised loss, following a teacher-student paradigm.Source: A. Preliminaries and Method Overview · p. 3
  • The standard consistency setup aligns the teacher’s output under easy augmentation to the student’s hard-augmented view; the two networks’ parameters and loss weights are distinguished.θ and θ′ denote student and teacher parameters; λ1 and λ2 balance loss terms.Source: A. Preliminaries and Method Overview · p. 3
  • S3KL combines a structure-aware auxiliary branch, supervised by annotation-free anatomical patterns, with Block Shuffle to disrupt global spatial context and encourage invariant anatomical representations.The framework comprises Structure-Aware Learning and Structural Representation Consistency Learning.Source: A. Preliminaries and Method Overview · p. 3

B. Structure-Aware Learning

Structure-Aware Learning creates annotation-free structural targets and distills them into the localization model. The targets provide high-resolution anatomical features, which the auxiliary branch aligns with cosine similarity on unlabeled data.

B. Structure-Aware Learning

  • SAM and MedSAM produce coarse contours but lack VFSS-specific semantics and may fail to distinguish functionally relevant regions, while manual pixel-wise annotation is prohibitively costly.Source: B. Structure-Aware Learning · p. 3
  • SPM combines DINOv2 contextual priors with FeatUp upsampling to produce clear soft-palate, hyoid, and vertebrae structures aligned with target keypoints, without medical annotations.DINOv2’s limited output resolution blurs boundaries and loses detail; the SPM features are visualized alongside SAM regions and DINOv2 features in Fig. 5.Source: B. Structure-Aware Learning · p. 4, Figure 5 · p. 4
  • The Structure-Aware Branch adds an auxiliary prediction head to the shared backbone and aligns its representations with SPM’s continuous, high-resolution structural targets using cosine similarity.This replaces the conventional cross-entropy loss used with discrete semantic-segmentation labels.Source: B. Structure-Aware Learning · p. 4
  • The cosine-similarity loss uses a threshold to focus learning on locations where the branch prediction is misaligned with the SPM target.Similarity is computed between branch and SPM features at each location.Source: B. Structure-Aware Learning · p. 4
  • The structural constraint is applied to unlabeled images under hard augmentation to encourage robust structural representation learning.Source: B. Structure-Aware Learning · p. 4

C. Structural Representation Consistency Learning

Structural Representation Consistency Learning disrupts global layout with block shuffling and restores feature order before enforcing consistency with teacher predictions. The overall objective combines supervised, unsupervised, and structure-aware losses, with two networks mutually supervising on unlabeled data.

C. Structural Representation Consistency Learning

  • Block Shuffle destroys global geometric context so the encoder must recognize anatomy from local visual patterns rather than absolute positions.Source: C. Structural Representation… · p. 4
  • Block Shuffle partitions an image into non-overlapping blocks and randomly permutes them; unshuffling encoder features restores their original spatial order before prediction.The objective is feature invariance rather than prediction of permutation indices.Source: C. Structural Representation… · p. 4
  • The shuffled-image student’s spatially restored features are aligned with the teacher’s stable predictions, and the mechanism is also applied to Structure-Aware Learning.Source: C. Structural Representation… · p. 4

D. Overall Learning Objectives

  • The total objective combines supervised loss, standard unsupervised consistency loss, and structure-aware regularization, including consistency and structure-aware losses on the block-shuffled branch.Source: D. Overall Learning Objectives · p. 4
  • Two networks mutually generate pseudo-labels for each other on unlabeled data rather than using an EMA teacher, and their outputs are fused for final predictions.Each network aligns its consistency predictions with fixed pseudo-labels from the other; the authors state this prevents coupling errors and stabilizes training.Source: D. Overall Learning Objectives · p. 4

IV. EXPERIMENTS AND RESULTS

Experiments use VFSSKep and evaluate semi-supervised training with varying labeled-data fractions alongside the full unlabeled set. The supplied passages specify metrics, implementation, and comparison baselines, but give no outcome values.

A. Experimental Settings

  • Experiments use 25%, 50%, or 100% of labeled training data, each combined with the full unlabeled set, and evaluate on the VFSSKep test set.Source: A. Experimental Settings · p. 4
  • Evaluation reports PCK@0.02, using a 2%-of-image-diagonal correctness threshold, and MED in millimeters for complementary precision analysis.Source: A. Experimental Settings · p. 4
  • The implementation uses SimpleBaseline with a ResNet18 backbone and compares S3KL against supervised SimpleBaseline, SemiPose, and G2LCPS.Training uses 100 epochs; the passage also specifies augmentations and loss weights.Source: A. Experimental Settings · p. 5

B. Main Results

S3KL improves semi-supervised keypoint localization across two existing frameworks, while ablations support contributions from both learning strategies and structural design choices.

B. Main Results

  • All semi-supervised methods using 25% labeled data surpass the supervised baseline trained with 100% labeled data.The authors attribute this result to using unlabeled non-swallowing segments.Source: B. Main Results · p. 5
  • Against SemiPose, S3KL gains 2.1%, 3.9%, and 2.1% PCK at 25%, 50%, and 100% supervision, and reduces MED by 20.7% and 15.9% at 25% and 50%.With G2LCPS, it improves PCK by 1.5%, 2.3%, and 0.8%, lowers MED, and achieves the best overall performance.Source: B. Main Results · p. 5

C. Ablation Studies

  • Structure-Aware Learning and Structural Representation Consistency Learning each contribute to gains, and their combination achieves the best results with higher PCK and lower MED.The evaluation uses SemiPose as the semi-supervised baseline.Source: C. Ablation Studies · p. 5
  • CLIP outperforms the baseline but trails DINOv2, while replacing FeatUp with bilinear interpolation reduces performance.The authors link DINOv2’s stronger structural priors to self-supervised image learning and FeatUp’s benefit to preserving structures while restoring high-resolution details.Source: C. Ablation Studies · p. 5
  • Unshuffling encoder outputs gives the best overall results, with higher PCK and lower MED at 50% and 100% supervision, though its MED is relatively high at 25%.Both encoder-output and final-output unshuffling improve over the baseline without Block Shuffle.Source: C. Ablation Studies · p. 6, Table V · p. 6

D. Visualization Results

The visualization contrasts S3KL with a baseline that sometimes predicts average training coordinates instead of tracking the hyoid. The conclusion reports that S3KL outperforms fully supervised methods with 25% labeled data.

D. Visualization Results

  • In visualization columns 1–3, the baseline fails to perceive the hyoid and instead memorizes average training coordinates, while S3KL’s predictions align more closely with ground truth.The authors interpret this as evidence of improved anatomical structure capture and localization accuracy.Source: D. Visualization Results · p. 6

V. CONCLUSION

  • With only 25% labeled data, S3KL outperforms fully supervised methods and is presented as a data-efficient approach for clinical kinematic analysis.The conclusion attributes its approach to mitigating spatial bias with structure-aware priors and consistency learning.Source: §V · p. 6
  • The paper introduces VFSSKep and S3KL for semi-supervised VFSS keypoint localization.The conclusion describes the framework as mitigating spatial bias through structure-aware priors and consistency learning.Source: §V · p. 6

I. RELATED WORK

Prior VFSS keypoint studies cover limited anatomy, while keypoint localization and semi-supervised methods offer established approaches that do not fully address structural cues in VFSS. The paper introduces a dataset and framework to address this gap.

A. Videofluoroscopic Swallowing Study

  • VFSS supports temporal and kinematic swallowing analysis, but prior keypoint studies focus on cervical vertebrae or the hyoid and neglect the soft palate.The paper introduces a dataset annotating eight keypoint categories, including the soft palate, cervical vertebrae, and hyoid regions.Source: A. Videofluoroscopic Swallowing Study · p. 7

B. Keypoint Localization

  • Keypoint localization methods include coordinate-regression and heatmap approaches; this paper uses SimpleBaseline and permits replacement with other architectures such as ViTPose.Regression predicts coordinates directly, while heatmap methods predict a heatmap for each keypoint.Source: B. Keypoint Localization · p. 7

C. Semi-supervised Keypoint Localization

  • Semi-supervised methods use pseudo-labeling or consistency regularization, but the reviewed approaches do not account for anatomical cues and structural relationships important to VFSS localization.Examples include DataDistill, PLACL, SSKL, and G2LCPS.Source: C. Semi-supervised Keypoint Localization · pp. 7–8

II. DATASET CONSTRUCTION DETAILS

  • The videos were recorded at 30 FPS using two fluoroscopic systems, which contributed 65% and 35% of the videos, respectively.Frames were converted to grayscale PNG after extraction.Source: §II · p. 8
  • VFSSKep includes 13,895 labeled frames from active swallowing and 31,536 clean unlabeled frames, with distinct individuals separated across training, validation, and test sets.The labeled split contains 8,302 training, 1,518 validation, and 4,075 test images; the unlabeled frames come from remaining videos and include non-swallowing segments.Source: §II · p. 8
  • The dataset covers ages 10–70+ years and reports a balanced sex distribution across its training, validation, test, and unlabeled subsets.These demographic summaries are reported in Table I.Source: §II · p. 8, Table I · p. 8
  • Independent experts reviewed annotations through a cross-validation process to ensure consistency.Source: §II · p. 8
  • The videos came from the Third Affiliated Hospital of Sun Yat-Sen University, and ethics approval was granted with informed consent waived for this retrospective anonymized study.Source: §II · p. 8

III. MORE EXPERIMENTAL RESULTS

The supplemental experiments describe the model implementation and examine hyperparameters governing feature alignment, structural similarity, and block shuffling. They identify settings that balance auxiliary learning and spatial disruption.

A. More Implementation Details

  • The implementation uses SimpleBaseline with a ResNet18 encoder and a Structure Parsing Module combining fixed DINOv2 features with FeatUp’s four-layer upsampler.The upsampler produces 16× high-resolution output; training runs for 100 epochs with Adam.Source: A. More Implementation Details · p. 8
  • Training uses easy augmentation for the teacher and stronger augmentation with Block Shuffle for the student, with supervised, unsupervised, and feature-alignment loss weights of 0.5, 1.0, and 10^-3.The similarity threshold is 0.85 and Block Shuffle uses 64 × 64 blocks.Source: A. More Implementation Details · p. 9

B. Hyperparameter Analyses

  • λ3 = 10^-3 gives the best performance by balancing the auxiliary feature-alignment task with the primary task.Higher weights overemphasize the auxiliary task; lower weights diminish its impact.Source: B. Hyperparameter Analyses · p. 9, Table III · p. 9
  • α = 0.85 achieves optimal performance by balancing relevant feature inclusion for generalization.A low threshold excludes many features from the alignment loss, while a high threshold includes even well-optimized features and may overfit.Source: B. Hyperparameter Analyses · p. 9, Table IV · p. 9
  • A 64×64 Block Shuffle size performs best across supervision levels, balancing preservation of spatial structures with effective shuffling.32×32 disrupts local spatial relationships through excessive shuffling; 128×128 provides insufficient shuffling to prevent overfitting.Source: B. Hyperparameter Analyses · p. 9, Table V · p. 9

C. Extension Experiments

Backbone choice trades limited-supervision accuracy against full-supervision performance and computational cost. Clinical relevance is illustrated by a prototype, while dataset scale and static-frame modeling remain limitations.

C. Extension Experiments

  • ViT-B performs better than ResNet18 with 25% and 50% supervision, but ResNet18 slightly outperforms it with full supervision at lower computational and memory cost.The reported costs are 87.6M/21.6G for ViT-B versus 15.6M/2.4G for ResNet18; the authors mainly adopt ResNet18 for its accuracy-efficiency balance.Source: C. Extension Experiments · p. 9

A. Clinical Implications

  • The clinician-in-the-loop prototype extracts kinematic parameters and temporal metrics to support diagnostic workflows.The paper links keypoint localization to measures such as hyoid displacement and soft palate elevation, and notes that large-scale deployment requires formal ethical approval.Source: A. Clinical Implications · pp. 9–10

B. Limitations and Future Directions

  • VFSSKep remains modest in scale, and the framework analyzes static frames without explicitly modeling temporal swallowing dynamics.The authors identify larger, more diverse and multicenter data, plus temporal modules or sequence-based architectures, as directions that could improve generalizability and clinical applicability.Source: B. Limitations and Future Directions · p. 10

Related papers

Loading 2610.07726v1…