PaperlayerView exact revision on arXiv

Source-linked AI summary

Pose-Aware Multimodal Automatic Tagging for Greek Traditional Music

Alexandros Alexiou, Charilaos Papaioannou, Alexandros Potamianos

arXiv:2609.27094v1cs.SDcs.CV

TL;DR

Most tagging systems use audio alone, while live performance and culturally specific music also carry visual and embodied cues. This paper extends Lyra with aligned video and dancer-pose streams and an automated pose-extraction pipeline. Its best trimodal result exceeds the strongest unimodal audio baseline, with pose adding complementary information in fusion.

  • Problem

    Audio-only tagging is common, but whether dancer poses add complementary information beyond audio for culturally specific music tagging remains an open question.

    Source: §1 · p. 1

  • Method

    The paper extends Lyra with aligned video and dancer-pose streams, an automated primary-dancer extraction pipeline, and multimodal tagging evaluation.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    A+V+S reaches 0.864 macro ROC-AUC (+4.3) versus the strongest unimodal baseline A at 0.821, and pose improves multimodal performance.

    Source: §6.2 · p. 8, §7 · p. 10

  • Takeaways & Limitations

    Category-level gains are concentrated on tags for which embodied performance cues are informative, while skeletons remain weaker as standalone taggers.

    Source: §7 · p. 10

    The dance subset has 749 videos, and the skeleton pipeline covers about 50% of clips because of occlusions, multi-person ambiguity, and scene transitions.

    Source: §7 · p. 10

Abstract

from arXiv · show

Automatic tagging is a core task in Music Information Retrieval (MIR), yet most tagging systems exploit only audio. Live music performance is inherently multimodal, as semantic labels such as instruments, regional styles, and dance forms are encoded simultaneously across acoustic, visual, and embodied performance cues. This is especially true of culturally specific repertoires such as Greek traditional music, which remain underrepresented in MIR benchmarks. In this paper, we investigate whether the use of dancer pose provides complementary information for automatic tagging in Greek traditional music beyond audio. Using the Lyra dataset, we extend prior audio-only work by extracting aligned video features and pose-derived skeleton streams, enabling an experimental setting for multimodal auto-tagging. We further introduce an automated pipeline for extracting primary-dancer skeleton sequences from in-the-wild dance footage, combining dance-scene detection, multi-person tracking, dancer selection, pose estimation, and quality filtering. We compare unimodal, all bimodal combinations, and trimodal systems using multiple fusion strategies. Audio remains the strongest single modality (AST: macro ROC-AUC 0.821), while skeletons, though weak in isolation, enhance performance through multimodal fusion. The best trimodal system improves macro ROC-AUC by about 4 percentage points over the strongest audio baseline.

1 Introduction

The paper asks whether dancer poses add useful information beyond audio for Greek traditional music tagging, where live performance expresses semantic cues through sound, movement, appearance, and instrumentation. It extends audio-only Lyra research with aligned multimodal representations, an in-the-wild pose pipeline, and category-level evaluation.

Motivation

  • The motivating question is whether dancer poses provide complementary tagging information beyond audio in culturally specific music.Movement, appearance, and instrumentation may encode tag-relevant information alongside sound.Source: §1 · p. 1

Contributions

  • The study introduces a Lyra tagging setting with aligned audio, video, and dancer-pose streams, enabling their first systematic multimodal evaluation for Greek traditional music.Lyra indexes 1,570 Greek television videos with multilabel instrument, genre, and geographic-origin annotations.Source: §1 · p. 2
  • Its automated pipeline combines dance-scene detection, multi-person tracking, primary-dancer selection, pose estimation, and quality filtering for unconstrained broadcast footage.The pipeline is designed for in-the-wild dance footage rather than curated motion data.Source: §1 · p. 2
  • The work evaluates how audio, video, and skeleton representations contribute across instrument, genre, and geographic-origin tags.This category-level analysis is intended to examine different types of semantic music tags.Source: §1 · p. 2

2 Related Work

Prior work has established audio-only tagging and explored multimodal performance understanding, but culturally specific multilabel tagging and skeleton-based autotagging remain underexplored. This paper sits at the intersection of those gaps.

Music tagging in world music datasets

  • Audio-only tagging is well studied, but MIR benchmarks center on Western repertoires; Lyra is a rare Greek traditional music resource, with prior work focused on audio representations and transfer.The cited background describes culturally specific and non-Western traditions as comparatively underrepresented.Source: §2 · p. 2

Multimodal music information retrieval

  • Prior multimodal MIR work covers genre classification, cross-modal retrieval, and performance understanding, but not multilabel autotagging in culturally specific collections.Those studies suggest visual information can complement audio in performance-centered settings.Source: §2 · p. 2

Pose-based modeling for music and dance

  • Pose-based models have supported dance generation, beat tracking, and performance analysis, yet skeletons have not been systematically studied for multilabel music autotagging, especially in broadcast footage.ST-GCN and extensions shaped pose-based sequence modeling.Source: §2 · p. 2

3 Dataset and Pose Extraction

The study uses Lyra’s annotated Greek music videos and restricts evaluation to dance clips with valid pose streams. Its extraction pipeline detects dance scenes, selects and estimates a primary dancer’s pose, and filters pose sequences into aligned multimodal clips.

3.1 The Lyra Dataset

  • Lyra videos span dance scenes, musicians, singers, and audience shots, and carry non-exclusive instrument, genre, and geographic-origin tags.The videos were recorded from multiple camera angles at 25 fps with 690×380-pixel frames.Source: §3.1 · p. 2
  • The dance subset contains 749 Lyra tracks with valid pose streams, selected from 767 dance-containing tracks among 1,570 audio-video tracks.Evaluation inherits the repository split and uses 28 labels after two of the prior top-30 labels lacked positive instances in this subset.Source: §3.1 · pp. 2–3

3.2 Pose Extraction Pipeline

  • A dance-scene detector, fine-tuned on 300 manually annotated one-second segments from 10 training videos, achieves 98% held-out segment-level accuracy.Scene labels are obtained by majority vote over one-second predictions within each PySceneDetect segment.Source: §3.2 · p. 3
  • Within detected dance scenes, ByteTrack tracks people and the pipeline selects the primary dancer by maximizing bounding-box area times detection confidence before AlphaPose estimates 17 COCO joints.The selection limits pose extraction to one dancer because broadcast footage includes audience members, musicians, singers, and partially visible people.Source: §3.2 · p. 3, Figure 1 · p. 3
  • Pose streams are hip-centered, filtered against a clip-level medoid, and scored using joint confidence, bone consistency, symmetry, and temporal jitter.The system retains up to 32 temporally ordered poses per clip, discards clips with fewer than eight valid poses, and pads clips with 8–31 poses.Source: §3.2 · p. 3
  • Audio, video, and skeleton clips share temporal boundaries, aligning the three modalities for fusion.Figure 2 illustrates the alignment with mel-spectrogram clip boundaries and corresponding sampled video frames.Source: Figure 2 · p. 4
  • Position (x, y, c) and first-order motion (∆x, ∆y, c) features preserve confidence c in both streams for joint reliability, producing V-joint, C-channel tensors [12,13].Source: §3.2 · p. 4

4 Methodology

The methodology aligns audio, video, and skeleton clips in time, encodes each modality separately, and compares several strategies for fusing their representations. The pose stream is handled as an optional modality in fusion.

4.1 Temporal Segmentation and Modality Alignment

  • All modalities use shared, non-overlapping 8-second clips, aligning audio, video, and skeleton features at the same temporal boundaries.Audio clips yield 500 mel-spectrogram time steps at 16 ms per step.Source: §4.1 · p. 4

4.2 Unimodal Representation Learning

  • AST encodes audio, visual backbones encode sampled frames or video segments, and a lightweight ST-GCN-style network encodes dancer skeletons.For multimodal fusion, the skeleton encoder processes pose streams associated with aligned 8-second clips, despite being trained on 3.69-second windows.Source: §4.2 · pp. 5–6

4.3 Multimodal Fusion

  • Fusion compares late probability averaging with learned early, gated, and directional cross-attention models over fixed modality embeddings.Skeleton-inclusive models use a learnable missing-skeleton token when poses are unavailable; late fusion uses uniform weights.Source: §4.3 · p. 6

5 Experiments

Experiments evaluate track-level, macro-averaged tagging metrics and report results over five random seeds. The supplied experiment passages describe evaluation and training protocols, and identify a table of video-model results.

Video model evaluation

  • Table 1 reports video-model results on the dance subset for 28 labels, with macro metrics averaged over five seeds.Bold indicates the best result in each column.Source: Table 1 · p. 7

Evaluation

  • Metrics are macro-averaged over tags at track level, with clip logits pooled per track and F1 predictions thresholded at 0.5.The reported metrics are ROC-AUC, PR-AUC, and F1; results are mean ± std over five random seeds.Source: §5 · pp. 6–7

Training protocols

  • Audio models train for up to 50 epochs, video heads for up to 200, and the skeleton model for 30 epochs without early stopping.Video backbones remain frozen, and skeleton training and evaluation use clips with valid dancer poses.Source: §5 · p. 7

Fusion training

  • Fusion models use fixed SlowFast-50 embeddings and train for 60 epochs with a shared 256-dimensional latent space, one Transformer layer, and four attention heads.The fusion setup uses AdamW, BCEWithLogitsLoss, batch size 64, and learning rate 2 × 10^-4.Source: §5 · p. 7

6 Results

Audio is the strongest unimodal baseline, but multimodal fusion improves tagging, with the best trimodal model outperforming audio alone. Results also show that skeleton cues contribute differently across fusion strategies, tag categories, and skeleton-specific design choices.

6.1 Unimodal Baselines

  • A achieves 0.821 macro ROC-AUC, above V (baseline) at 0.759, while S (baseline) reaches 0.586.Audio also leads on macro PR-AUC and macro-F1; skeleton-only performance is markedly weaker than audio and video.Source: §6.1 · pp. 7–8, Table 2 · p. 8

6.2 Multimodal Fusion Results

  • 0.864 macro ROC-AUC for A+V+S exceeds the 0.821 A baseline by +4.3, and adding skeletons lifts A+V from 0.852 to 0.864 (+1.2).The full results are reported on the dance subset; Table 2 reports macro metrics as mean ± std over 5 seeds.Source: §6.2 · p. 8, Table 2 · p. 8
  • Gated fusion generally leads macro ROC-AUC and PR-AUC, Cross-Attention often leads macro-F1, and Late fusion remains competitive on threshold-independent A+V metrics but weaker on macro-F1.The comparison spans the fusion strategies evaluated across modality combinations.Source: §6.2 · p. 8
  • S exceeds A on Aegean genre F1 (0.21 vs. 0.17), matches V on Traditional genre (0.98 vs. 0.98), and remains weaker for Epirus place and Klarino instrument.S is also competitive with A on Voice (0.75 vs. 0.77) and Percussion (0.82 vs. 0.92).Source: §6.2 · p. 9

6.3 Skeleton Robustness and Missingness Ablations

  • 3.69-second training windows yield skeleton-only macro ROC-AUC 0.596, versus 0.556 with 8.00-second windows; applying the shorter-window encoder to aligned 8.00-second poses yields 0.586.The authors use the shorter-window encoder in the aligned multimodal pipeline.Source: §6.3 · p. 9, Table 3 · p. 9
  • Learned missing-skeleton tokens yield ROC-AUC 0.864, PR-AUC 0.525, and F1 0.393, versus zero substitution's ROC-AUC 0.860, PR-AUC 0.515, and F1 0.384.This ablation uses the best trimodal gated fusion model.Source: §6.3 · p. 9, Table 3 · p. 9

6.4 Category-Level Analysis

  • A+V leads instruments (mean per-category F1 0.508), V+S leads genre (0.438), and geographic-origin remains hardest despite V+S reaching 0.324 versus unimodal scores below 0.086.Category scores are mean per-label F1 within each semantic category; Figure 4 presents these results.Source: §6.4 · pp. 9–10, Figure 4 · p. 10

7 Conclusion

The conclusion presents a pose-aware multimodal extension of Lyra and reports that video and dancer poses complement audio, though findings are constrained by data scale and pose quality.

7 Conclusion

  • The work combines audio, video, and pose-stream representations, handles missing pose observations, and evaluates multiple fusion strategies for tagging.Its pose-extraction pipeline targets broadcast footage.Source: §7 · p. 10
  • Audio remains the strongest unimodal signal, while weaker standalone skeletons improve performance when combined with audio or video.Category-level analysis suggests gains concentrate on tags for which embodied performance cues are informative.Source: §7 · p. 10
  • The dance subset contains 749 videos, and the skeleton pipeline covers about 50% of clips because of occlusions, multi-person ambiguity, and scene transitions.The authors describe multimodal cultural-archive enrichment from broadcast recordings as a viable path.Source: §7 · p. 10

8 AI Usage Statement

AI-based writing assistance supported manuscript text editing and formatting; the authors state that scientific content, results, and conclusions remain their responsibility.

8 AI Usage Statement

  • AI-based writing assistance supported manuscript text editing and formatting, while scientific content, results, and conclusions remained the authors’ responsibility.Source: §8 · p. 10

9 Ethics Statement

The Lyra recordings come from a publicly broadcast Greek television program and are used for non-commercial research; released pose data contain anonymized 2-D joint coordinates only.

9 Ethics Statement

  • The Lyra Dataset uses recordings from To Alati tis Gis for non-commercial research, and the pipeline stores or releases only anonymized 2-D joint coordinates for motion-feature extraction.No biometric identifiers are stored or released; footage is used solely for music tagging.Source: §9 · p. 11

Related papers

Loading 2609.27094v1…