PaperlayerView exact revision on arXiv

Source-linked AI summary

FineDance: A Fine-grained Choreography Dataset for 3D Full Body Dance Generation

Ronghui Li, Junfan Zhao, Yachao Zhang, Mingyang Su, Zeping Ren, Han Zhang, Yansong Tang, Xiu Li

arXiv:2212.03741v4cs.CVcs.SDeess.AS

TL;DR

Existing dance datasets have limited fine-grained hand motion and genre coverage, complicating full-body, multi-genre generation from music. The paper introduces the FineDance dataset and FineNet, combining diffusion-based generation with genre- and coherence-aware retrieval. The authors report diverse, genre-matched long-term dances, while noting that rhythm matching and comprehensive artistry still need improvement.

  • Problem

    Existing datasets have limited hand motion and dance genres, while previous methods can produce unnatural or monotonous hand movement and lack an objective genre-matching metric.

    Source: §1 · p. 2

  • Method

    The paper introduces FineDance, a 14.6-hour paired dataset with 22 genres, and FineNet, which combines body and hand expert generation with genre- and coherence-aware retrieval.

    Source: §1 · p. 2

  • Results

    FineNet generates long-term, diverse, genre-matched dances from music, and experiments report multiple genre-matched dances with natural, flexible hand movements.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    FineDance provides paired motion-capture dance data with fine-grained hand motion and genres for music-driven dance generation research.

    Source: §1 · p. 2

    Compared with ground truth, all methods still need improvement, especially in rhythm matching and comprehensive artistry.

    Source: §5.4 · p. 8

Abstract

from arXiv · show

Generating full-body and multi-genre dance sequences from given music is a challenging task, due to the limitations of existing datasets and the inherent complexity of the fine-grained hand motion and dance genres. To address these problems, we propose FineDance, which contains 14.6 hours of music-dance paired data, with fine-grained hand motions, fine-grained genres (22 dance genres), and accurate posture. To the best of our knowledge, FineDance is the largest music-dance paired dataset with the most dance genres. Additionally, to address monotonous and unnatural hand movements existing in previous methods, we propose a full-body dance generation network, which utilizes the diverse generation capabilities of the diffusion model to solve monotonous problems, and use expert nets to solve unreal problems. To further enhance the genre-matching and long-term stability of generated dances, we propose a Genre&Coherent aware Retrieval Module. Besides, we propose a novel metric named Genre Matching Score to evaluate the genre-matching degree between dance and music. Quantitative and qualitative experiments demonstrate the quality of FineDance, and the state-of-the-art performance of FineNet. The FineDance Dataset and more qualitative samples can be found at our website.

1. Introduction

Existing datasets provide limited full-body hand motion and coarse genre coverage, constraining music-driven dance generation. The paper introduces FineDance and FineNet to address these gaps and reports generation of genre-matched dances with natural, flexible hand movements.

Motivation

  • Existing datasets contain few hand movements and limited coarse genres, while prior methods can produce unnatural or monotonous hands and lack an objective genre-matching metric.The paper notes that body and hand motions occupy different feature spaces.Source: §1 · p. 2

Contributions

  • FineDance contains over 14.6 hours of paired music and dance from 346 songs, with accurate body and hand motions and 22 fine-grained genres.The dataset includes music, dance sequences, FBX files, SMPL data, and multi-view videos.Source: §1 · p. 2
  • FineNet combines diffusion-based body and hand expert networks with a Refine Net, then uses retrieval to improve genre matching and dance-fragment coherence.The paper also introduces Genre-matching Score to evaluate genre matching between generated dances and music.Source: §1 · p. 2
  • The authors describe FineDance as the largest 3D motion-capture music-dance paired dataset with accurate full-body posture and 22 fine-grained genres.They state that it encourages work on AI choreography, motion priors, and full-body reconstruction.Source: §1 · p. 2

Results

  • Quantitative experiments and user studies report multiple genre-matched dances from arbitrary music, with natural and flexible hand movements.Source: §1 · p. 2

2. Related Works

The related work contrasts limitations in existing choreography datasets with the trade-offs between retrieval-based synthesis and generative dance methods. FineDance combines motion-capture data with a generation-and-retrieval approach.

2.1. Choreography Dataset

  • FineDance addresses dataset limitations in hand motion, duration, genre and dancer diversity with 14.6 hours across 22 genres, collected from 27 dancers.The passage contrasts this with AIST++'s five hours without hand motions and other datasets' narrower genre coverage.Source: §2.1 · p. 2

2.2. AI Choreography

  • Motion-graph synthesis retrieves and splices existing dance fragments, matching music style but not learning internal music-dance connections or creating new motions.Generative methods can create dances but often neglect hand quality and produce only short fragments; FineDance's approach uses diffusion expert nets and retrieval for full-body expression, genre matching, and long-term stability.Source: §2.2 · p. 3
  • FineDance acquisition captures dancers with Vicon, retargets motion to a standard skeleton in MotionBuilder, then uses manual music alignment and Blender skeletal extraction.Source: Figure 2 · p. 3

3. FineDance Dataset

FineDance is presented as a response to the shortage of accessible, large-scale motion-capture choreography datasets with diverse genres and well-paired music. Its acquisition combines professional dancers, motion capture, and music-dance alignment.

3. FineDance Dataset

  • FineDance is presented as filling gaps in accessible large-scale motion-capture datasets and in genre diversity and music-dance pairing.Source: §3 · p. 3

3.1. Data Acquisition and Analysis

  • FineDance records 3D skeleton joints including fingers and uses a standard 52-joint representation.The paper notes that few existing datasets include finger motion and available finger-motion data is under 1.5 hours.Source: §3.1 · p. 3
  • FineDance includes 22 professional-artist-defined genres, more than 14 hours of data, and folk dance motions; its motions are captured by motion-capture systems rather than reconstructed from video.The authors say its dances and music are well paired and describe the captured postures as accurate.Source: §3.1 · p. 4
  • The dataset acquisition involved 27 professional dancers, with dancers attending to music rhythm and style while performing.Engineers retarget captured motion to a standard skeleton, and the music is manually aligned with dance motions.Source: §3.1 · p. 4

4. FineNet

FineNet generates multiple full-body dance candidates from music, then retrieves and joins genre-matched fragments for longer choreography. Its diffusion-based generator separates body and hand motion, while retrieval considers both genre fit and continuity.

4.1. Overall Framework

  • FineNet processes music in clips and generates a dance fragment for each clip, using the previous fragment as context after the first step.Each music clip supplies temporal features and a mel-spectrogram; the output fragment contains body and hand joints.Source: §4.1 · pp. 4–5

4.2. Diverse Dance Generation

  • Diffusion generates multiple distinctive dance fragments from music temporal features and different noise samples.Choosing a different initial fragment can yield multiple complete dances.Source: §4.2 · p. 5, §4.4 · p. 6

4.3. FDGN: Full-body Dance Generation Network

  • Separate body and hand expert networks address their different motion spaces, and a refine net assembles the parts coordinately.The refine net uses spatial convolution and a refine gate to assemble the two parts.Source: §4.3 · p. 5

4.4. Genre&Coherent aware Retrieval Module

  • GCRM selects candidate fragments using genre-match and coherence scores, then joins successive fragments with interpolated transitions.Its cross-modal retrieval network compares music and dance genres; the coherence score uses segment start and end states, with five frames cut and linearly interpolated for smoother transitions.Source: §4.4 · pp. 5–6

5. Experiments

FineNet is evaluated against prior methods, through component ablations, and in a user study. Results favor FineNet on most reported metrics, with remaining gaps to ground truth in rhythm matching and comprehensive artistry.

5.1. Experimental Setup

  • FineNet is evaluated on the FineDance@Genre split, whose test set covers more genres but includes dancers also present in training and validation.Source: §5.1 · p. 6

5.2. Quantitative and Qualitative Evaluation

  • FineNet leads all metrics except Diversity on both datasets; MM improves 4.52 over ChoreoMaster, and GS is 0.03 above ground truth.Source: Table 2 · p. 7, §5.2 · p. 7
  • FineNet produces diverse dances matching Jazz style, while other generators appear stiff and ChoreoMaster's classical motions mismatch the Jazz music.FineNet with FDGN also produces more realistic hand motion.Source: §5.2 · p. 7

5.3. Ablation Study

  • Removing FDGN worsens all reported metrics; Hand FID drops by 0.72 and Hand Diversity by 0.14.Source: §5.3 · p. 7
  • 5.21 on MM and 0.28 on GS separate FineNet from FDGN-C; FineNet's Diversity is lower because retrieval excludes some genre-mismatched dances.Source: Table 3 · p. 7, §5.3 · p. 8
  • FDGN performs best among the tested generation backbones, while GCRM improves DanceRevolution and DeepDance on MM, FID, and GS.Source: Table 4 · p. 8, §5.3 · p. 8

5.4. User Study

  • In the 30-participant user study, generation methods remain under 10 seconds of effective duration, while FineNet earns a 77.0 average artistry score and surpasses baselines.Source: Table 5 · p. 8, Table 6 · p. 8, §5.4 · p. 8
  • All methods still fall short of ground truth, especially in rhythm matching and comprehensive artistry.Source: §5.4 · p. 8

6. Conclusions

The paper presents FineDance, FineNet, and a genre-matching metric, reporting that FineNet generates long-term, diverse, genre-matched dances from music.

  • FineDance provides professional, varied dances with accurate fine-grained hand motion, while FineNet generates diverse, genre-matched dances with flexible hands and the reported results support long-term generation.Source: §6 · p. 8

A-1. Additional Details of FineDance Dataset

FineDance's genre collection follows the natural distribution rather than balancing durations across genres.

A-2. Additional Details of FDGN

FDGN uses diffusion to generate body and hand dance fragments conditioned on music, with the hand expert also using generated body motion.

  • Diffusion gradually perturbs dance data toward noise, and a neural network learns to denoise; the paper notes expressive, diverse generation and easier convergence than GANs.Source: A-2. Additional Details of FDGN · p. 11
  • The body expert estimates a clean body fragment from a noisy fragment and music features, directly predicting the motion rather than noise at each step.Source: A-2. Additional Details of FDGN · p. 11
  • The hand expert uses the body expert's output together with music features as conditions for generating coordinated hand motion; sampling different initial noise yields diverse fragments.Source: A-2. Additional Details of FDGN · p. 11

A-3. Potential Applications of FineDance Dataset

FineDance’s aligned, accurately annotated dance data may support motion-prior learning, music generation from dance, and full-body reconstruction.

A-3. Potential Applications of FineDance Dataset

A-4. Additional Qualitative Results

Qualitative comparisons report more comprehensive 20-second dances than the compared methods, distinct genre outputs, and diverse dances from one music clip.

A-4. Additional Qualitative Results

Loading 2212.03741v4…