Source-linked AI summary
Encypher: Shared Agency and Social Presence in Collaborative Music Generation for Dance Cyphers
Zhixing Chen, Cheng-Zhi Anna Huang
TL;DR
Most HCI work on generative music centers solo performers, leaving open how AI might organize social encounters around sound. Encypher translates collective movement into prompts for real-time music in dance cyphers; across co-design, study, and public deployments, participants reported shared agency and social presence, with limits in some settings.
Problem
Most HCI work on generative music centers solo performers, while how AI can support human-to-human connection through shared sound remains an open question.
Source: §1 · p. 2Method
Encypher translates collective movement qualities into text prompts conditioning real-time music generation for dance cyphers.
Source: §1 · p. 2Results
Participants perceived music as responding to room energy and developed shared agency, while newcomers’ uncertainty and established communities’ resistance qualified the experience.
Source: §1 · p. 2, §10 · p. 15Takeaways & Limitations
The findings support treating mapping choices as social design and foregrounding sociality in collaborative human-AI creative systems.
Source: §10 · p. 15The eleven-person single-session study used self-report and observations, and its screen/no-screen shift cannot be separated from group order and floor tape.
Source: §9 · p. 14
Abstract
from arXiv · showhide
Music and dance are social practices of expression and connection, yet most HCI work in human-AI co-creation centers the solo performer. As generative music matures, we ask not only what AI can compose but what social encounters it can organize around sound. We present Encypher, a collaborative generative music system that translates collective movement qualities into text prompts conditioning real-time music generation for dance cyphers. Through five weeks of co-design with local dancers, a user study with unacquainted participants, a public museum event, and a live performance, we found that users developed shared agency, perceiving the music as a response to the room's energy. While newcomers felt uncertain, the system fostered social presence by prompting them to look to each other for cues. By treating sociality as a design concern rather than a downstream effect, we offer a framework and design implications for AI systems for collaborative, embodied expression.
1 Introduction
Encypher addresses HCI’s focus on solo performers by exploring how generative music can support social encounters. It translates collective movement into prompts for real-time music in dance cyphers, where participants reported shared agency and social presence alongside uncertainty and resistance in some established communities.
1 Introduction
- The work shifts the focus of generative music HCI from what AI composes to the social situations it can help create.It frames human-to-human connection, rather than only human-AI interaction, as a design concern.Source: §1 · p. 2
- Encypher translates collective movement qualities into text prompts that condition real-time music generation for dance cyphers.Its pipeline uses depth-camera kinematics to steer musical elements through LyriaRT; center-of-mass energy also controls light intensity.Source: §1 · p. 2, Figure 2 · p. 2
- Participants perceived music as responding to the room’s energy, while the system prompted newcomers to look to one another for cues despite initial uncertainty.The system acted as a social bridge for strangers, though established dance communities sometimes resisted its framing.Source: §1 · p. 2
- The authors developed Encypher through autobiographical design, five weeks of co-design with local dancers, a study with unacquainted participants, a museum event, and a live performance.The live performance used light on dancers instead of a visualization.Source: §1 · p. 2
- The paper contributes a collaborative system, empirical findings on shared agency, design implications about limits with established communities, and a framework foregrounding sociality in multimodal interaction.The museum setup combined a depth camera, screen visualization, and colored ceiling lights reflecting zones.Source: §1 · p. 2, Figure 3 · p. 2
2 Related Work
Related work connects social interaction frameworks with collaborative technologies, movement sonification, and generative motion-to-music systems. It motivates Encypher as a group-scale, open-ended alternative to systems centered on fixed targets or individual and dyadic interaction.
2 Related Work
- The related work situates Encypher among research on social interaction, technology-mediated collaboration, and movement-driven generative music.These strands frame the paper’s focus on social interaction through group movement and sound.Source: §2 · p. 2
2.1 Social Interaction
- Social interaction frameworks distinguish warmth and competence, social presence, and technology’s facilitating, inviting, and encouraging roles; evaluation also separates interaction style from interaction quality.The integrated evaluation structure spans social-psychological processes, interaction qualities, and technological supports.Source: §2.1 · p. 3
2.2 Technology-Facilitated Collaboration
- Music games, collaborative instruments, and social platforms bring people into shared performance, but often constrain participation to charts, routines, formats, or fixed targets.Real-time generative music offers a shared sound that can change with collective behavior while leaving room for negotiation.Source: §2.2 · p. 3
2.3 Movement Sonification
- Movement sonification maps kinetic data to musical parameters, with prior work exploring interpersonal movement, gesture-based systems, and movement as an extension of expressive intent.C-Battle used gesture classifications and pre-composed loops, unlike Encypher’s generative music and cypher framing.Source: §2.3 · p. 4
2.4 Neural Motion-to-Music Generation
- Neural motion-to-music research ranges from reactive sound systems to offline diffusion methods, while Con Moto steers real-time symbolic music through camera motion.Diffusion systems can achieve higher musical quality and beat alignment scores than previous models but struggle with live-performance latency; prior individual and dyadic work leaves group social dynamics open.Source: §2.4 · pp. 4–5
3 Methods
The methods frame Encypher as a system for studying how movement-to-music mappings can support social interaction, drawing on dancer-centered co-design and group-scale sensing. They situate the system among distinct motion-to-music approaches and account for the trade-offs of camera-based tracking.
3 Methods
- The methods ask how generative music can support social interaction in dance and assess interaction style alongside social presence, group cohesion, and quality.They extend prior motion-to-music work centered on solo performance and fixed stylistic domains toward mappings that can capture relations between participants.Source: §3 · p. 5
3.1 First-Person Approach
- The first-person process began in the first author’s movement practice and broadened through iterative co-creative sessions with diverse dancers to refine mappings.The sessions invited dancers to move with and against the system, using their feedback to guide technical and conceptual design changes.Source: §3.1 · p. 5
3.2 Design Space
- The design space distinguishes structural, reactive, semantic, and textural motion-to-music mappings by how movement and sound are coupled.Structural systems build over time, reactive systems map movement immediately, semantic systems use bodily states to select or generate outputs, and textural systems shape sound continuously.Source: §3.2 · pp. 5–6
3.3 Motion Sensing
- Depth-stereo tracking captures group geometry without wearables, supporting floor-based dance, but occlusion degrades tracking and multi-camera fusion halves throughput.The single-camera setup was sufficient for user-study groups of up to six.Source: §3.3 · p. 6
4 System Design
Encypher couples real-time music generation with collective movement features and group formation. Its steering and interface were shaped through co-design, including responses to latency, movement-vocabulary mismatches, and the challenge of studying social connection among strangers.
4.1 System Overview
- Encypher pairs a live music model with a group control surface coarse enough for dancers to shape rich audio collectively rather than through precise individual control.Its web interface handles generation and visualization, while camera-derived features are filtered and sent through Max/MSP as control signals.Source: §4.1 · pp. 6–7
4.2 Embodied Collaborative Steering
- Collective upper-body groove weights musical prompts, while group circularity adjusts generation diversity so tighter formations produce more coherent, predictable music.The movement mappings were co-designed with local breakers; circularity is smoothed to support stable transitions.Source: §4.2 · p. 7
4.3 Co-Design Process
- Co-design replaced confusing individual element assignments with labeled circular zones and COMAmp, which better matched breakers’ rocking and swaying vocabulary.A charging mechanic made LyriaRT’s two-second latency an intentional threshold before the music responds.Source: §4.3 · p. 7
- Testing with established dancers could not establish whether Encypher supported connection among strangers, prompting a shift toward novices and unacquainted participants.Fine-grained temperature and top-K mappings were disruptive, and BPM steering proved unreliable for some genre and instrument combinations under text-conditioned generation.Source: §4.3 · pp. 7–8
5 Movement Class User Study
In a study with 11 dance novices who had no pre-existing social ties, Encypher encouraged shared agency and non-verbal coordination, while its screen and marked zones introduced trade-offs between social attention, spatial clarity, and freedom of movement.
5 Movement Class User Study
- The study recruited 11 adult students with minimal dance experience and no pre-existing social ties, using a studio installation with a depth camera, projected zones, and audio playback.Participants danced in two groups, completed questionnaires, and joined a facilitated discussion; the study also recorded sessions for qualitative analysis.Source: §5 · p. 8, §5.1 · p. 8, §5.2 · p. 8
5.4 Quantitative Analysis
- 81.8% agreed on Collaboration, while 63.6% agreed on Control, a pattern consistent with collective steering but more modest individual control.Spatial Awareness had 90.9% agreement and the highest strongly-agree rate (36.4%); Social Presence had 54.6% agreement, with 18.2% disagreeing.Source: §5.4 · p. 9, Figure 7 · p. 9
- The quantitative responses were positive but moderate: no participant disagreed on Engagement or Enjoyment, while neutral responses were common, especially for Engagement.Replay and Invitation each drew 63.6% agreement, suggesting room to refine how the system signals turn-taking and ongoing engagement.Source: §5.4 · p. 9
5.6 Results Summary
- The authors caution that classroom context and first-time exposure to high-fidelity neural audio may have skewed feedback, and cross-referenced questionnaire results with open-ended reflections.Across the findings, the screen eased entry but diverted attention, while spatial markers offered clarity at a cost to movement freedom.Source: §5.4 · pp. 9–10, §5.6 · p. 10
5.5 Qualitative Analysis: Themes in Collaborative Interaction
- Participants described social presence emerging as they noticed, mirrored, and responded to one another’s movements and energy.One participant also said stepping out of and back into the circle made them want to move more.Source: §5.5 · p. 10
- Participants described the music as responding to the room and collective energy, rather than as a one-to-one trigger of their own movement.They also described everyone as playing a role in creating the music.Source: §5.5 · p. 10
- Marked spots clarified where to move, but some dancers found them restrictive or felt they should remain where they controlled the sound.Body-map responses most often located engagement and change in the torso, hips, and feet, although the system measured upper-body center-of-mass amplitude.Source: §5.5 · p. 10
- The screen gave unacquainted participants a shared focal point that eased entry into the group, but drew attention from one another.Without it, participants initially felt awkward and drifted from the cypher; floor tape restored some spatial legibility while screen-free dancing appeared more expansive.Source: §5.5 · p. 10
6 Public Museum Event
At a public museum event, Encypher was explored by attendees ranging from teens to adults, with no expectation of dance experience. Observations suggested that ceiling lighting could coordinate dancers without a screen, while the dance floor itself invited hesitant newcomers to join.
6 Public Museum Event
- The 65-minute public co-design session invited self-selected attendees, from teens to adults, to explore Encypher and collectively design a performance.The installation accommodated about 25 attendees, rotating through the cypher in groups of five to six, and paired a projection screen with colored ceiling lighting.Source: §6 · p. 11, §6.1 · p. 12, Figure 11 · p. 12
6.2 Observations
- After one or two attendees stepped into the cypher, others followed, and the room shifted from observers to active dancers within a minute.The authors observed the dance floor functioning as an invitation without verbal prompting from facilitators.Source: §6.2 · p. 12
- With the screen removed, ceiling lighting alone helped participants locate zones, move freely, and coordinate without the initial awkwardness seen in the classroom study.The authors suggest light projection can replace screen-based visualization in venues with lighting infrastructure.Source: §6.2 · p. 12
- Attendees likened Encypher to a club or karaoke, pointing to possible informal public settings for its low-stakes structure for collective expression.Six participants who had encountered the system about an hour earlier volunteered to demonstrate it to a larger audience.Source: §6.2 · p. 12
7 Public Live Performance
A live performance tested a screen-free Encypher configuration using stage lights to show each dancer’s energy. The authors observed restored mutual attention and visible individual contributions, but collected no participant or audience feedback.
7 Public Live Performance
- The screen-free version was performed at a seated-audience showcase of about 300 people, and the authors report only the first author’s observations.No participant or audience feedback was collected.Source: §7 · p. 12, §7.2 · p. 13
7.1 Setup
- Six dancers performed with six stationary zones, each assigned a musical element matched where possible to the dancers’ interests and training.A staggered sequence of improvised entries built toward all six dancers and musical elements sounding together; preparation consisted of a one-hour rehearsal focused on blocking.Source: §7.1 · pp. 12–13
7.2 Observations
- In the screen-free performance, dancers oriented toward one another and the lights at their feet, while light pools made each contribution legible to the audience.Whether this arrangement changes participants’ experience of shared agency and social presence relative to the screen remains an open empirical question.Source: §7.2 · p. 13
8 Discussion
Encypher’s discussion distinguishes systems designed to enrich established dance practices from those inviting strangers to move together, and treats mapping choices as social design. It contrasts symbolic control with neural audio richness, whose value depends on whether a system serves performance or social participation.
8.1 Designing With Versus Designing for a Community
- Established breaker communities resisted Encypher’s framing, unlike novices and unacquainted participants; breakers need predictable music with a clear downbeat and found the system added overhead without value.C-Battle’s collective scenarios involved mostly dancers outside their home communities, while its expert dancer tested only individual scenarios.Source: §8.1 · p. 13
- Encypher’s pivot toward strangers reflects a different goal: using music as a low-stakes invitation to dance and bond, rather than extending trained dancers’ existing practice.Over-sonifying movement risks making dancers play an instrument instead of dancing to music.Source: §8.1 · p. 13
8.2 Mapping as Social Design
- Movement-to-music mappings encode social expectations by rewarding particular group behaviors, while spatial convergence remains an unexplored design parameter.The authors also describe movement energy as varying across styles, skill levels, and cultural contexts, and say COMAmp captures only one facet of groove.Source: §8.2 · p. 13
8.3 Symbolic Versus Neural Audio as Design Modalities
- Symbolic MIDI offers fine-grained control suited to choreographed performance, whereas neural audio offers richer, more danceable music and natural-language steering for social participation.Neural audio trades away spatialization, note-level control, and some predictability; which modality matters more depends on the system’s purpose.Source: §8.3 · pp. 13–14
9 Limitations and Future Work
The authors frame Encypher’s findings as early design evidence, identifying study-design, model, transfer, and rhythm constraints. They outline future work on mapping synchrony, lighting feedback, and sensor-equipped performance spaces.
Study Limitations
- The eleven-person study used one session, self-report, and first-author observations; screen conditions were confounded, while public deployments were observed rather than measured.The authors caution that findings are design evidence from a first deployment cycle, not controlled effects.Source: §9 · p. 14
Baseline, Model, and Transfer
- Without a baseline, shared agency cannot be attributed specifically to generative music; LyriaRT’s changing closed API limits reproducibility, and mappings may require redesign for other dance forms.The authors propose comparing Encypher with a playlist and live DJ, and note that they did not evaluate the reproducible open-weights MagentaRT.Source: §9 · p. 14
Latency and Rhythmic Grounding
- LyriaRT’s two-second generation window delays audio response, and inconsistent meter, downbeat alignment, and unreliable BPM steering constrain rhythmic predictability.Beat conditioning at inference time is proposed as a model-level fix, particularly relevant to breaking, where downbeat timing is fundamental.Source: §9 · p. 14
Synchrony as an Unmapped Social Dimension
- Encypher does not map participant synchrony, which the authors identify as a potential primary design parameter with choices of timescale, body region, and comparison metric.Future work could test whether synchronized movement produces richer or more stable musical output.Source: §9 · p. 14
Ambient Feedback and Lighting as a Control Surface
- Lighting driven by the same signals as the screen can preserve spatial and energetic legibility without a separate display, but group-level lighting effects and comparisons with screen feedback remain untested.Programmable stage lighting is required, constraining deployment to venues with that capability.Source: §9 · p. 14
Toward a Stage That Listens to Itself
- A sensor-equipped stage where lighting, sound, and generative systems respond to shared movement could extend the feedback-loop design beyond Encypher’s six zones and LyriaRT.The authors propose studying larger deployments with more performers, output modalities, and longer runs.Source: §9 · pp. 14–15
10 Conclusion
Encypher reframes generative music as a medium for social experience in shared physical space. Its collective movement features shifted perceived agency toward the room, while findings support treating mapping choices as social design.
10 Conclusion
- Encypher maps collective movement qualities to text prompts that condition real-time neural audio generation for dance cyphers.The system was developed through autobiographical design, co-design with local breakers, a study with unacquainted participants, and public deployments.Source: §10 · p. 15
- Mapping collective energy and spatial circularity shifted felt agency from the self to the room, while the system served as a social bridge for strangers.Established dance communities sometimes resisted its framing, citing their own social dynamics and stricter musical requirements.Source: §10 · p. 15
- The findings support treating mapping design as social design because system choices about what to listen for, reward, or ignore encode theories of interaction.The work offers a starting point for AI systems supporting how people come together to dance.Source: §10 · p. 15
A Post-Session Questionnaire
The post-session questionnaire gathered anonymous participant ratings and reflections immediately after each trial, followed by a facilitated group discussion. Questions covered dancing experience, social interaction, embodied reflection, and participants’ experiences of joining in.
A Post-Session Questionnaire
- The anonymous paper questionnaire followed each trial and recorded participant ID, group size, and prior group-dance experience.A facilitated, consented 20-minute group discussion followed individual surveys.Source: A Post-Session Questionnaire · p. 16, §A.4 · p. 17
A.1 Section 1: Experience of Dancing
- Participants rated engagement, agency over musical output, enjoyment, and willingness to replay Encypher on a 7-point Likert scale.The scale ran from 1 (Strongly Disagree) to 7 (Strongly Agree).Source: §A.1 · pp. 16–17
A.2 Section 2: Social Interaction
- Social-interaction ratings covered social presence, collective collaboration, invitation, encouragement, and spatial awareness of marked spots.Responses used the same 7-point Likert scale.Source: §A.2 · p. 17
A.3 Section 3: Embodied Reflection
- Embodied-reflection prompts asked participants to mark engaged or changed body areas and describe connection, nonverbal cues, and the effects of marked spots.Participants also reflected on whether music felt responsive to them or to the group or room, hesitation to join, and anything else they noticed.Source: §A.3 · p. 17
A.4 Group Debrief Guide
- The group debrief asked whether participants felt connected, who the music belonged to, what held them back from joining, and how spots shaped attention and movement.Prompts also asked whether the system made the space feel more welcoming to join.Source: §A.4 · p. 17
- Researchers tracked spot occupancy, eye contact, approach or retreat, laughter and smiling, and timestamps of musical shifts during the debrief.Discussion prompts were directed to the group so participants could build on one another’s accounts.Source: §A.4 · p. 17