PaperlayerView exact revision on arXiv

Source-linked AI summary

ChoreoStudio: Planning What, When, and Who in Music-Driven Group Choreography

Jing Xu, Cunjian Chen, Qiuhong Ke

arXiv:2610.32325v1cs.SDcs.AIcs.CV

TL;DR

Group-dance generation must coordinate roles and formation changes, while existing methods leave formation evolution implicit or insufficiently represented. ChoreoStudio explicitly plans formation, timing, and dancer mobility and uses that plan to guide motion generation. On AIOZ-GDance, it improves motion quality and group organization over strong baselines, with realistic macro-level formation evolution.

  • Problem

    Group-dance generation must assign spatial roles and organize formations over time, but existing methods leave roles and intended arrangements unresolved or implicit.

    Source: §1 · pp. 1–2

  • Method

    ChoreoStudio separates choreography planning from motion realization, coupling relative formations, dancer relations, transition timing, and mobility in an explicit plan that guides the generator.

    Source: §1 · p. 2

  • Results

    ChoreoStudio improves motion quality and group organization over strong baselines, while frequency-based analysis confirms realistic macro-level formation evolution.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The plan provides an inspectable and editable choreography-control path to the generator under fixed music and sampling noise.

    Source: §1 · p. 2

Abstract

from arXiv · show

Music-driven group dance generation must decide not only how each dancer moves, but also what formation the group should form, when it should change, and who should execute each transition. Existing methods can generate orderly and rhythmically synchronized group motion, but remain limited in formation evolution and inter-dancer interaction. In human choreography, formation transitions often align with changes in high-level musical structure, such as the progression from an introduction to a chorus or from a chorus to a climax. We present ChoreoStudio, which makes choreography planning explicit. A music disentangler first decomposes the audio into frame-level rhythmic features and segment-level structural features. Given the segment-level features and sparse labels, a frozen omni-modal model acts as a scriptwriter, producing a global semantic description that guides the Director's choreography planning. Based on this context, the Director produces three coupled interaction decisions: a centroid-relative point set specifies what formation to produce, a transition gate determines when it should change, and a mobility allocation indicates who should execute the transition. The transition gate and mobility allocation are further conditioned on the planned geometric change, coupling formation targets with their temporal and dancer-specific execution. At a finer temporal scale, frame-aligned music features condition the diffusion generator to control detailed body motion and preserve beat-level alignment. The segment-level interaction plan is injected in parallel through frame-aligned instruction tokens and cross-dancer attention biases, coordinating how the generator realizes each transition. Experiments on AIOZ-GDance demonstrate that ChoreoStudio follows the planned interactions while achieving state-of-the-art performance in motion quality, group organization, and formation dynamics.

1 Introduction

Group-dance generation must coordinate dancers’ roles and formations, not just their individual movements. ChoreoStudio makes formation, transition timing, and dancer mobility explicit in a plan that guides motion generation.

1 Introduction

  • Existing generators can retain local motion and coordination while underrepresenting slow formation evolution, motivating a plan for what the group becomes, when it changes, and who executes the change.Existing approaches typically induce coordination inside the generator, while dancers’ asymmetric roles and intended formations remain unresolved or implicit.Source: §1 · pp. 1–2
  • ChoreoStudio separates choreography composition from motion realization, coupling relative formation geometry, directed dancer relations, transition timing, and mobility allocation.The intended geometric change informs whether a transition occurs, and its scheduled displacement informs which dancers carry it.Source: §1 · p. 2
  • A plan-to-motion interface turns formation, timing, and mobility decisions into dancer-specific guidance, making the plan the sole choreography-control path to the generator.The plan remains fixed during denoising and can be inspected or edited at inference under fixed music and sampling noise.Source: §1 · p. 2, Figure 1 · p. 2
  • ChoreoStudio improves motion quality and group organization over strong baselines, with frequency analysis confirming realistic macro-level formation evolution rather than high-frequency distance fluctuations.The evaluation uses a shared 150-frame protocol, with long-sequence results reported separately.Source: §1 · p. 2

2 Related Work

Prior work addresses music-driven motion, group coordination, and structured generation, but leaves group formation evolution largely implicit or reduces it to trajectories. ChoreoStudio distinguishes itself by jointly planning group formation, transition timing, and dancer mobility.

2.1 Music-Driven Single and Group Dance Generation

  • Group-dance methods improve coordination through joint, contrastive, coherence-oriented, trajectory, and spatial-temporal modeling, while formation evolution remains implicit or reduced to low-level trajectories.ChoreoStudio instead jointly plans formation, transition timing, and dancer mobility.Source: §2.1 · p. 3

2.2 Multi-Scale Music Conditioning

  • Frame-aligned acoustic features support local rhythmic synchronization, while ChoreoStudio separates segment-level planning context from frame-level rhythm for motion synthesis.A frozen omni-modal model interprets music segments and sparse labels to guide formation, transition, and mobility planning.Source: §2.2 · p. 3

2.3 Structured Planning for Generative Motion

  • Structured-control methods plan individual motion or generation procedures, whereas ChoreoStudio jointly plans group geometry, transition timing, and dancer mobility for coordinated full-body motion.Its Director couples formation, timing, and mobility, then encodes the plan as frame-aligned instructions and relation-derived cross-dancer attention bias.Source: §2.3 · p. 3, Figure 2 · p. 4

3 Method

ChoreoStudio inserts an explicit, executable choreography plan between musical interpretation and full-body motion synthesis. Its Director plans group interactions, a generator realizes them, and a verifier can feed failures back for replanning.

3.1 Overview and Problem Formulation

  • The Director predicts an explicit plan from segment-level music, semantic intent, current group state, and prior-cycle feedback, then the verifier checks the generated motion and returns localized failures for replanning.The plan represents formations, dancer relations, spatial roles, transition timing, mobility, and pose divergence.Source: §3.1 · p. 4

3.2 Multi-Scale Music Conditioning and Semantic Intent

  • The learned disentangler supplies segment-level structural features for formation and transition planning and frame-level rhythmic features for detailed synthesis and beat alignment.A frozen omni-modal model turns audio and sparse attributes into a macro-level description that provides interpretable intent to the Director.Source: §3.2 · p. 5

3.3 Explicit Choreography Director

  • Exchangeable actor seeds distinguish dancers without assigning index-based roles, while a state-aware Director uses music, semantics, feedback, and group state to construct dancer-specific formations from a directed relation graph.Formation proposals are centroid-relative point sets, so their pairwise distances are geometrically realizable by construction.Source: §3.3 · pp. 5–6
  • The Director conditions transition gates on proposed geometric change and allocates mobility according to scheduled displacement, then exposes the plan through frame-aligned instructions and relation-derived attention biases.Only these structured plan conditions reach the generator, keeping choreography control interpretable and editable.Source: §3.3 · p. 7

3.4 Plan-Conditioned Group Dance Generation

  • A conditional diffusion generator jointly synthesizes the dancers using frame-level rhythmic features, dancer-specific plan instructions, and relation-derived cross-dancer spatial attention biases.Instruction tokens modulate transformer blocks, while the attention bias shapes information exchange across dancers.Source: §3.4 · pp. 7–8

3.5 Verification-Guided Refinement

  • A frozen verifier checks plan realization, physical plausibility, and music alignment, and localized feedback guides up to T replanning cycles before candidate selection.Refinement changes the explicit plan rather than directly editing generated motion; it may stop early for a passing candidate.Source: §3.5 · p. 8

3.6 Training Objectives

  • Plan supervision comes automatically from ground-truth motion, with actor-to-trajectory assignments preserved across segments and generator training separated from Director plan prediction.The generator learns to realize teacher plans, while the Director learns to recover them from music, semantic intent, and group state.Source: §3.6 · p. 8

4 Experiments

Experiments evaluate ChoreoStudio on AIOZ-GDance under shared protocols, comparing motion and group organization, formation dynamics, ablations, and responses to controlled plan edits. Results support explicit planning for coordinated formation changes while preserving motion quality and collision avoidance.

4.1 Experimental Setup

  • Evaluation uses AIOZ-GDance’s official split of 1,309 training and 154 test sequences, with shared test identities, sequence length, conversion, feature extractors, and evaluation code.The 150-frame results use one ChoreoStudio model across dancer counts; a 720-frame evaluation is reported separately, while baselines are trained separately by group size.Source: §4.1 · p. 9
  • Reported measures cover motion fidelity and diversity, physical foot contact, group realism and correlation, trajectory intersections, formation dynamics, and beat alignment.IDDV, Jitter, and FMR measure low- and high-frequency components of formation dynamics; BAS measures alignment between kinematic and musical beats.Source: §4.1 · p. 9

4.2 Comparison with Existing Methods

  • ChoreoStudio achieves the best GMR and GMC at every group size, with best or second-best TIF, improving group organization without compromising collision avoidance.CoDancers has low collision rates but the weakest group realism and correlation; TCDiff++ and ST-GDance improve realism without closing the coordination gap.Source: §4.2 · p. 9
  • Table 2 shows ChoreoStudio narrows the gap to real data across four formation-dynamics and music-alignment metrics, recovering more slow-timescale variation and beat alignment than TCDiff++.Its FMR remains close to the real-data reference, while high-frequency residuals are reduced.Source: Table 2 · p. 9, §4.2 · p. 9
  • On matched music and durations, ChoreoStudio produces more reliable spatial exchanges and stable full-body motion than the compared methods.ST-GDance shows body overlap during exchanges; TCDiff shows foot-ground penetration and arm interpenetration.Source: §4.2 · p. 10, Figure 4 · p. 10

4.3 Ablation Studies

  • Removing the Director causes the largest GMC and FMR drops and the highest TIF; timing most affects realism and coordination, while role assignment contributes most to slow-timescale variation.The semantic description improves group-motion realism, the Verifier strengthens coordination and formation dynamics, and the relation graph provides spatial organization.Source: §4.3 · p. 10, Table 3 · p. 10

4.4 Plan Realization and Controlled Intervention

  • Controlled plan edits change generated motion in the requested direction, with formation producing the strongest response, followed by timing and mobility.Small TIF and Jitter changes indicate that control is not achieved through motion degradation.Source: Table 4 · p. 10, §4.4 · p. 10

5 Conclusion

ChoreoStudio addresses underrepresented slow formation evolution by explicitly planning formations, transition timing, and dancer mobility, then making those decisions executable, interpretable, and editable. The paper reports state-of-the-art motion quality and group organization, more realistic formation dynamics, accurate plan realization, and responses to controlled interventions.

5 Conclusion

  • ChoreoStudio jointly plans realizable relative formations, transition timing, and dancer mobility, with an explicit plan-to-motion interface for executable, interpretable, and editable decisions.The conclusion identifies absolute stage orientation and richer semantic roles as future work.Source: §5 · p. 11

AI Use Statement

  • The authors used generative AI for language polishing, grammar checking, and literature assistance, but not for implementation, data collection, or result analysis.Authors verified AI-assisted content and take responsibility for the final text, claims, and artifacts.Source: AI Use Statement · p. 11

Ethics Statement

  • Datasets are publicly available under established licenses, and the user study used anonymous perceptual rankings without collecting personally identifiable information..Source: Ethics Statement · p. 11

Reproducibility Statement

  • The paper locates architecture and training in Section 3 and evaluation protocol in Section 4, with data, metrics, interventions, and long-sequence details specified in appendices.Appendix B covers processing and sampling; Appendix C covers controlled interventions and long-sequence generation.Source: Reproducibility Statement · p. 11

A Analysis and Metric Details

  • Generated and ground-truth motions use the same SMPL-based representation and feature extractors; frequency-based formation metrics use length-matched windows aggregated by sequence.Standard motion and group metrics are computed over complete sequences.Source: A Analysis and Metric Details · p. 13

A.1 Motion Quality and Diversity

  • Kinetic and geometric feature distributions are evaluated with FID, where lower values indicate closer generated-to-real distributions; diversity is average pairwise distance.The paper reports kinetic and geometric variants as FIDk/Distk and FIDg/Distg.Source: §A.1 · pp. 13–14
  • Physical Foot Contact measures foot sliding during body acceleration, and lower PFC indicates fewer foot-sliding artifacts.For each foot, the smaller velocity of the toe and ankle joints is used.Source: §A.1 · p. 14

A.2 Group Motion Metrics

Group-motion metrics assess group-level realism, inter-dancer motion similarity, and body penetration, while frequency-based metrics separate gradual formation evolution from rapid distance fluctuation. The comparisons show that ChoreoStudio better matches real formation dynamics while keeping music alignment close to the real-data reference.

A.2 Group Motion Metrics

  • Group Motion Realism measures the discrepancy between real and generated group-level kinetic-feature distributions using a Fréchet formulation.It uses a group kinetic feature extractor in place of the individual-motion feature extractor.Source: §A.2 · p. 14
  • Group Motion Correlation measures inter-dancer motion similarity using frame-level kinetic features along each pair’s Dynamic Time Warping alignment path.Interpret GMC relative to its ground-truth value because excessive synchronization or duplicated motions can also produce high correlation.Source: §A.2 · p. 14
  • Trajectory Intersection Frequency measures the average proportion of dancer pairs whose SMPL bone segments intersect at each frame.Pairs count as intersecting when the minimum distance between any two bone segments falls below a thickness threshold.Source: §A.2 · p. 14

A.3 Frequency-Based Formation Dynamics

  • Frequency-based formation analysis separates gradual changes in pairwise ground-plane distances from rapid residual fluctuations, using a 0.5 Hz low-pass filter.The cutoff retains changes with characteristic periods of roughly two seconds or longer; IDDV, Jitter, and FMR are interpreted relative to real dance, not optimized independently.Source: §A.3 · pp. 14–15

A.4 Empirical Formation-Dynamics Comparison

  • IDDV is 0.267 for ChoreoStudio, versus 0.181 for TCDiff, 0.179 for ST-GDance, and 0.302 for real dance; its BAS is 0.351 versus 0.363 for real data.ChoreoStudio also has Jitter of 0.048 and FMR of 0.848; TCDiff has higher Jitter and substantially lower FMR than real data, while ST-GDance retains a large IDDV gap.Source: §A.4 · p. 15, Figure 5 · p. 16

A.5 Plan-Realization Metrics

The appendix evaluates whether generated motion realizes the Director’s formation, timing, and mobility plan, and describes music alignment, data processing, conditioning, and plan targets. It also specifies how measurements are aggregated across sequences.

A.5 Plan-Realization Metrics

  • Transition F1 is 0.80 and average temporal offset is 0.32 seconds; ChoreoStudio also has low formation, pairwise-geometry, and mobility errors.These results indicate that planned changes are generally detected and executed near their scheduled times, while dancer-specific movement responsibilities are preserved.Source: Table 5 · p. 16, §A.5 · pp. 16–17

A.6 Music-Dance Alignment

  • Music-dance alignment is evaluated with Beat Alignment Score (BAS), averaged first across dancers and then across test sequences.The score uses a Gaussian tolerance with σ = 2.12 frames.Source: §A.6 · p. 17

A.7 Aggregation and Statistical Analysis

  • IDDV, Jitter, and FMR are averaged across dancer pairs within windows, then across windows per sequence and across test sequences, preventing longer sequences from receiving disproportionate weight.Ground-truth comparisons use matched sequence identities whenever available.Source: §A.7 · p. 17

B.1 Data Processing and Motion Representation

  • The official AIOZ-GDance splits contain 1,309 training and 154 test sequences, with motion sampled at 30 fps and group sizes from two to seven dancers.Training examples use 720-frame windows with stride 112; longer-sequence inference uses 720-frame windows with 30-frame overlap.Source: §B.1 · p. 17

B.2 Music and Semantic Conditioning

  • The music encoder provides frame-aligned rhythmic features for motion generation and segment-level structural context for the Director; a frozen omni-modal model supplies semantic descriptions from audio and sparse attributes.The description is generated once per sequence, while transition timing and detailed motion remain grounded in learned music features.Source: §B.2 · p. 17

B.3 Plan Targets and State Initialization

  • Pseudo-label plans use centroid-centered root positions for formation, scale-normalized pairwise-geometry changes for transitions, and normalized dancer displacements for mobility.Actor seeds distinguish otherwise identical dancer tokens, and one sequence-level bipartite matching preserves correspondence over time without assigning semantics to dancer indices.Source: §B.3 · p. 18

B.4 Verifier Implementation

The appendix describes a verifier that evaluates completed motion and informs plan refinement, along with the plan-supervision objectives and training and sampling setup. Candidate validity and reward selection are handled separately.

B.4 Verifier Implementation

  • After each complete diffusion trajectory, the frozen Verifier checks plan realization, physical plausibility, and coordination, then provides localized feedback for a subsequent Director prediction.The system retains candidates across refinement cycles, returns the highest-reward passed candidate when available, and otherwise returns the highest-reward candidate with diagnostics; validity rejects severe violations separately from reward ranking.Source: §B.4 · pp. 18–19

B.5 Detailed Objective Definitions

  • Plan targets are derived from centered segment-level root positions, with a single Hungarian-matched dancer correspondence held fixed across segments.Transition labels detect sufficiently large scale-normalized pairwise-geometry changes, while mobility targets represent the relative displacement required from each dancer and therefore formation-level travel rather than in-place motion.Source: §B.5 · pp. 19–20
  • The Director objective combines shape, scale, transition-gate, mobility, diversity, and rate losses.The directed relation graph is not directly supervised because corresponding annotations are unavailable.Source: §B.5 · pp. 19–20

B.6 Optimization and Sampling

  • The Director, trainable music-conditioning modules, and Dance Generator are jointly optimized for 55 epochs; the default plan is predicted before diffusion and remains fixed throughout denoising.Optional refinement predicts a new plan only after a complete motion sample has been evaluated by the verifier.Source: §B.6 · p. 21

B.7 Length-Matched Evaluation Protocol

The evaluation protocol matches sequence lengths and processing across methods, aggregates measurements at the sequence level, and describes its uncertainty and paired comparisons.

B.7 Length-Matched Evaluation Protocol

  • All methods are evaluated on the same 154 test identities, with a 150-frame main setting and a separate 720-frame long-sequence experiment.Within each setting, methods share temporal support, motion conversion, feature extraction, and metric implementations.Source: §B.7 · p. 21
  • Sequence-level statistics are the independent evaluation units, with pairwise and window measurements aggregated within each sequence before averaging across test sequences.This prevents longer sequences or larger groups from receiving disproportionate weight.Source: §B.7 · p. 21
  • Reported uncertainty uses 95% bootstrap confidence intervals over test sequences, and paired significance tests use matched test sequences.Paired tests are reported only for completed comparisons.Source: §B.7 · p. 21

C.1 Controlled Plan Interventions

Controlled edits show that ChoreoStudio’s explicit plan shapes formation changes, their timing, and dancer participation. The system also generalizes across sequence lengths and group sizes, while planning diagnostics support the role of explicit formation and scheduling decisions.

C.1 Controlled Plan Interventions

  • Paired, single-cycle generation fixed music, labels, initial state, actor seeds, and diffusion noise to isolate the plan-to-motion path.Identical-plan and rotation-invariance controls had to reproduce baseline output within numerical tolerance.Source: §C.1 · p. 22
  • Formation edits reduced realization error by 40% and changed output in the requested direction for 78.54% of samples.The formation targets were transplanted from training sequences with matching dancer counts, centered, and radius-matched before replacement.Source: §C.1 · p. 22
  • Transition timing remained controllable, mobility response was weaker but consistent, and TIF and Jitter changes stayed small.The paired results indicate that edits realized the requested behaviors without materially degrading physical plausibility or temporal stability.Source: §C.1 · p. 23

C.2 Long-Sequence Generation

  • ChoreoStudio leads all metrics for one and two dancers; for three and four dancers, it has the highest GMC and lowest TIF, while TCDiff++ has slightly lower GMR.The same checkpoint generalizes across sequence lengths and group sizes without length- or group-size-specific retraining.Source: §C.2 · p. 23, Table 7 · p. 23

C.3 Additional Planning Diagnostics

  • All planning variants degrade group-motion realism and coordination; removing relation-formation feedback causes the largest overall degradation, particularly in FMR.Removing music structure mainly reduces GMR and FMR, while relation-only reasoning cannot replace explicit formation and scheduling decisions.Source: §C.3 · p. 23, Table 8 · p. 23

C.4 Denoising-Budget-Matched Refinement Analysis

With a fixed denoising budget, feedback-guided refinement outperforms best-of-two sampling on group realism and coordination. Paired semantic edits change intended group behaviors, and users prefer ChoreoStudio across all four perceptual dimensions.

C.4 Denoising-Budget-Matched Refinement Analysis

  • Under the same 100-step denoising budget, feedback-guided refinement reduces GMR by 23.9% and improves GMC by 1.91 points over best-of-two sampling.FMR remains close to the real-data reference, while TIF increases slightly from 0.09 to 0.10.Source: §C.4 · p. 24, Table 9 · p. 24

C.5 Semantic Control

  • Expansive descriptions increase group spread by 0.34 m in 82.6% of paired samples, while dynamic descriptions increase IDDV by 0.052 with 77.4% direction accuracy.Small GMR and TIF changes indicate these semantic responses occur without substantial degradation in motion realism or physical plausibility.Source: §C.5 · p. 24, Table 10 · p. 24

C.6 User Study

  • ChoreoStudio scores highest across all four user-study dimensions, with its largest margins in Music Alignment and Inter-Dancer Coordination.The blind study recruited 30 participants, who ranked anonymized videos without ties.Source: §C.6 · p. 25, Figure 6 · p. 25

D.1 Qualitative Results and Video Demonstrations

Qualitative ablations associate the Director, semantic description, and Verifier with distinct aspects of group choreography. Editing formation geometry, dancer mobility, or transition timing changes the corresponding planned behavior under fixed conditions.

D.1 Qualitative Results and Video Demonstrations

  • Ablations link the Director to group organization, semantic description to differentiated motion, and the Verifier to transitions from wide to compact formations.Without the Verifier, the sequence remains near a wide formation with repeated poses; ChoreoStudio instead gradually reorganizes the group.Source: §D.1 · pp. 25–26
  • Editing geometry, mobility, or timing changes the formation target, dancer participation, or reorganization time while other conditions stay fixed.The qualitative comparisons fix the music, initial noise, and non-intervened plan variables.Source: §D.1 · p. 26, Figure 7 · p. 26

Related papers

Loading 2610.32325v1…