AuraLuxMuse: Adaptive Fusion Modeling for Aesthetic Stage Lighting Design with Music and Expert Guidance

1 Introduction

• Traditional lighting design is time-consuming and labor-intensive, with cues programmed from scratch and aesthetic quality varying across productions.Designers analyze musical structure, interpret a thematic concept, and iteratively revise fixture behaviors using visual feedback.Source: §1 · p. 2
• Existing machine-learning systems restrict lighting control to basic properties, while structured parameterization, multimodal personalization, and comprehensive evaluation remain challenges.The stated evaluation gap concerns both objective fidelity and subjective perception.Source: §1 · p. 2
• AuraLuxMuse retrieves professional cue sequences from music and designer preferences, retargets fixture commands to a destination stage, and dispatches cues with timecodes.The system integrates expert knowledge, multimodal representation, and preference-adaptive modeling.Source: §1 · p. 2
• LAMP aligns audio and lighting parameters in a shared embedding space, while PAMoE uses four gated MLP branches to steer retrieval toward designer-preferred styles.The four branches are not preassigned to style axes; their associations with Color, Brightness, Zone, and Rhythm are analyzed after training.Source: §1 · p. 2
• The system encodes music and preference as a retrieval query, estimates cue density K, selects top-K professional sequences, and converts them into console-ready files.A professional lighting program comprises fixture-level cue commands and timecodes.Source: §1 · p. 2
• Musilux pairs 187 minutes and 22 seconds of music with more than 80,000 professional lighting instructions across real-world performance scenarios.The reported contributions also include competitive outputs versus manual designs, evaluation diagnostic EGSS, and average inference time of 7.65 ± 0.98 seconds under the reported setup.Source: §1 · p. 2

Music-Driven Motion, Visual Generation, and Lighting Design

• Music-conditioned motion and visual-generation systems produce body motion or pixels, while existing lighting systems commonly require interactive design, simulation, objectives, manual control, or predefined programs.The paper distinguishes these approaches from retrieving fixture-addressed, console-executable professional cues from music while conditioning on preference and retargeting to a new stage.Source: §2 · pp. 2–3

Retrieval-Based Creative Authoring

• AuraLuxMuse applies retrieval-augmented authoring to expert-authored, console-valid cue sequences, queried jointly by musical context and user-specified expressive intent.The target cues are described as temporally coherent, rhythmically grounded, and stylistically aligned.Source: §2 · p. 3

Audio Alignment

• Prior audio-alignment work motivates mapping music and lighting into a shared latent space to model cross-modal correspondence and support coherent interactions.The related work discusses contrastive learning and music/audio retrieval methods as context for this alignment approach.Source: §2 · p. 3

Designer Preference Adaptation

• Mixture-of-Experts methods have been used across language, image synthesis, cross-modal generation, audiovisual learning, and style-specific speech, motivating specialist pathways for lighting preferences.The paper links this choice to lighting design's stylistic preferences and the need to adapt model behavior accordingly.Source: §2 · p. 3

3.1 System Overview and Problem Formulation

• The system is designed for offline pre-production: a music track and optional preference vector yield editable Cue Sequences and Timecodes rather than rendered pixels.The objective is to retrieve and adapt professional lighting events conditioned on music and stylistic preference.Source: §3.1 · p. 3

3.2 Structural Decomposition: Cues & Timecodes

• Cue Sequences contain fixture-level CueData, while Timecodes specify when each sequence is triggered; cosine similarity and predicted K query the corpus.The cue structure supports frame-level timing of lighting transitions.Source: §3.2 · p. 4

3.3 Modality Interplay via Alignment: LAMP

• LAMP places temporally matched music windows and lighting segments in a shared embedding space through contrastive learning, including same-style, same-music segments as soft positives.The original music–lighting pair receives weight 1, while these soft positives receive weight 0.3.Source: §3.3 · pp. 4–5

3.4 Frame-wise Estimation: Metadata Predictor

• The Metadata Predictor estimates lighting-event counts per music window from a non-negative frame-level density vector.Its offline, non-causal TCN uses symmetric padding; window-level supervision is described as addressing sparse labels and capturing short-term dependencies.Source: §3.4 · p. 5

3.5 Mechanism for Artists’ Specific Needs: PAMoE

• PAMoE uses four symmetric MLP branches for stylistic control, combining cue-dependent and preference-dependent gates over lighting representations.The branches are evaluated for each lighting sequence and are not hard-assigned to specific style dimensions.Source: §3.5 · p. 5

3.6 Adaptive Stage Lighting Retrieval

• AuraLuxMuse retrieves top-K cue sequences using music and preference, with Metadata Predictor setting K before conflict checking and stage retargeting.The resulting editable cue sequences and timecodes are dispatched to the lighting console.Source: §3.1 · p. 3, §3.6 · p. 5

Dataset scale

• Musilux contains 187 minutes and 22 seconds of music and more than 80,000 associated professional lighting instructions.The dataset was collected collaboratively with leading institutions in stage lighting design.Source: §4 · p. 5

Performance scenarios

• The dataset covers concert, drama, and entertainment scenarios, with concert subtypes including Pop, Rock, Dance, Rap, and Electronic.The reported durations are 4,666 seconds for concerts, 3,307 seconds for drama, and 3,269 seconds for entertainment.Source: §4 · pp. 5–6

Stage coverage

• Musilux spans 13 stages, with 46.1% having one lighting program, 30.7% two, and 23.2% more than two.The paper describes this as a foundation for learning relationships between modalities.Source: §4 · p. 6

Data alignment

• Musilux frame-aligns audio with time-stamped lighting cues, mapping timestamps to an audio-frame grid and lighting attributes to continuous tensors.Audio frames are transformed using STFT and projected onto the Mel scale.Source: §4 · p. 6, Figure 3 · p. 6

Corpus organization

• The Cue Sequence Corpus contains 84,854 entries and covers 325 fixture behaviors, including color transitions, spatial movements, and speed modulations.Each entry contains multiple CueData units specifying fixture-level instructions.Source: §4 · p. 6

5.1 Experiment Setups

• Validation uses manual references, while testing uses unseen music on an unseen stage; evaluation includes virtual renders and physical-stage validation.Fixture commands are retargeted using spatial and hardware-mismatch costs with Kuhn–Munkres assignment before console dispatch.Source: §5.1 · p. 6

5.2 Evaluation Metrics

• FID measures feature-space distribution discrepancy rather than definitive perceptual quality; EGSS combines expert and non-expert ratings, with expert scores weighted at 0.7.EGSS draws on five expert-rated criteria and general-audience feedback.Source: §5.2 · p. 7

5.3 Human Evaluation

• Experts favored AuraLuxMuse in Concert (+1.075, p=0.00071) and Entertainment (+0.748, p=0.025), but the Drama difference was not significant (−0.366, p=0.260).The expert cross-scenario average was a positive but non-significant +0.486 (p=0.079), while non-experts favored manual design on average (−0.431, p=0.0049).Source: §5.3 · p. 7, Table 1 · p. 8

5.4 Experiment Results

• Qualitative examples show close matches to manual spatial composition and color dynamics, with greater visual complexity and temporal continuity at some timestamps; Fig. 5 visualizes preference-conditioned variation across styles.The demonstrations span Color harmony, Brightness control, and Zone distribution.Source: §5.4 · pp. 7–8, Figure 5 · p. 8
• Similarity retrieval performs best among tested retrieval strategies, and subjective performance is evaluated across Musilux styles and styles absent from training.Table 2 compares similarity retrieval with random, position-based, frequency-based, nearest-neighbor, and rule-based strategies; Table 3 reports EGSS for six validation and six test styles.Source: §5.4 · p. 8, Table 2 · p. 8, Table 3 · p. 9
• 7.65 ± 0.98 seconds average inference time accompanies competitiveness with manual designs across evaluated scenarios; AuraLuxMuse also outperforms tested sequence-to-sequence generative baselines.The authors attribute the generative-baseline gap to divergent effects from small lighting-parameter perturbations and difficulty capturing high-fidelity constraints.Source: §5.4 · p. 8, Table 1 · p. 8, §5.5 · p. 9

5.5 Discussion

• The authors position AuraLuxMuse as an editable retrieval-and-adaptation system that may suggest multimodal modeling paths for other underexplored modalities.Its outputs are retrieved and adapted professional cues, not replacements for unconstrained human creativity.Source: §5.5 · p. 9

5.6 Limitations

• The fixed expert-authored corpus excludes fixture behaviors and artistic structures absent from it; high-level show decisions still require manual authoring and final approval.The authors also note difficulty with unclear-groove music, script-only material without audio timing, rapidly changing styles, and population-scale generalization across genres.Source: §5.6 · p. 9

• AuraLuxMuse performs stage-lighting design through preference-conditioned retrieval and retargeting, and the paper introduces Musilux, a professional lighting-music dataset.Source: §6 · p. 9
• A dedicated representation module correlates music with lighting, while PAMoE injects personalized preferences and captures underlying style characteristics.Source: §6 · p. 9
• The authors describe the system as offering theoretical contributions to multimodal learning and practical value in real-world applications.Source: §6 · p. 9

Supplementary Material

• The supplement covers lighting visualizations, evaluation standards, preference and style details, patching, ablations, environments, implementation, and dataset generalization.Source: Supplementary Material · p. 11

A Cross-Scenario Visualization of Lighting Effects

• The shown examples illustrate music-responsive, scenario-dependent cue choices and editable retrieval-and-adaptation, without establishing universal superiority over designers.Concert cues increase instrumentalist visibility; the Entertainment example emphasizes vibrant, complex behavior.Source: A Cross-Scenario Visualization of… · p. 11

B.1 Experts

• Expert evaluation defines five dimensions: color harmony, brightness control, zone distribution, rhythmic tempo, and narrativity.Professionals reviewed the standards and rated 12 scenes spanning one manual design and three AuraLuxMuse variants per scenario.Source: §B.1 · p. 11

B.2 Non-Experts

• The non-expert protocol assesses rhythmic accuracy, thematic coherence, and narrative visual integrity, with participants reviewing definitions before rating scenes.Downbeat synchronization illustrates rhythmic alignment; participants also judged thematic conveyance and stylistic consistency.Source: §B.2 · p. 11

B.3 Human-Study Statistical Details

• Experts favor AuraLuxMuse in Concert (+1.075, p=0.00071) and Entertainment (+0.748, p=0.025), but not significantly in Drama or across scenarios.Non-expert ratings favor manual design on average (difference −0.431, p=0.0049).Source: §B.3 · pp. 11–12
• Expert evaluation rates color harmony, brightness control, zone distribution, rhythmic tempo, and narrativity across 12 scenes per protocol.The scenes include one manual design and three AuraLuxMuse variants per scenario.Source: Table S1 · p. 11, Table S2 · p. 11, Table S3 · p. 11
• The qualitative Drama comparison shows that three architecture variants produce distinct retrieved or adapted lighting outputs as editable references for designers.Source: Figure S1 · p. 12
• EGSS is scenario dependent: AuraLuxMuse scores higher in Concert and Entertainment, manual design scores higher in Drama, and the overall difference is not significant (p=0.282).Paired ratings give AuraLuxMuse higher overall ratings from 72.5% of experts and 40.0% of non-experts.Source: §B.3 · p. 12

C.1 Style Dimensions Specification

• Designers annotate each of 33 labeled programs with ordinal Color, Brightness, Zone, and Rhythm values, embedded as a 128-D condition for joint training.EGSS is not used as a training label, and inter-rater agreement for the style annotations was not measured.Source: §C.1 · p. 12

C.2 Style Details

• The style dimensions cover color-emotion compatibility, five brightness levels tied to lux ranges, visual prominence across zones, and the pace of lighting transitions.Source: §C.2 · pp. 12, 14
• Qualitative Concert and Entertainment examples contrast manual designs with retrieved or adapted cues, illustrating different stylistic choices without claiming general superiority.Source: Figure S2 · p. 13, Figure S3 · p. 14

C.3 PAMoE Gate Sensitivity

• Across 33 preference labels, gate weights show structured associations between style dimensions and experts, including Color–Expert 3 correlation 0.471 and Brightness–Expert 1 correlation 0.631.The analysis directly demonstrates preference sensitivity; reported output correlations are observational.Source: §C.3 · p. 14
• The learned branches are not predefined style experts, and the reported correlations establish neither exclusive specialization nor causal effects.A controlled fixed-music causal sweep was unavailable because the corresponding assets were not available.Source: §C.3 · p. 15

D Patching Mechanism Analysis

• Retargeting combines fixture hardware requirements and normalized spatial coordinates with a stage-specific layout; assignment resolves command collisions before console dispatch.In a deliberately conflict-heavy case, 148 fixture–attribute commands had 94 collisions before assignment and none afterward; compatibility was preserved.Source: D Patching Mechanism Analysis · p. 15, Figure S4 · p. 16

E.1 Structural Variants of Modules

• PAMoE improves objective metrics and subjective perception, while soft alignment improves performance by encouraging shared latent lighting behavior across stages in a scenario.The backbone comparison favors a VGGish music encoder with MLP layers and a transformer lighting encoder.Source: §E.1 · p. 15, Table S12 · p. 15, Table S11 · p. 15
• The non-causal TCN Metadata Predictor uses context across each complete input window to estimate cue density, whereas the frame-wise MLP baseline overpredicts under unified window-level evaluation.Its estimate guides downstream retrieval and better matches the sparsity of professional stage lighting.Source: §E.1 · pp. 15–16, Table S13 · p. 16

E.2 Baseline Architecture Comparison

• AuraLuxMuse achieves the best semantic-retrieval and expert-guided subjective scores among the tested contrastive baselines; AudioCLIP is the strongest adapted generic baseline.The adapted generic models struggle with the discrete, physically constrained lighting space.Source: Table S14 · p. 16, §E.2 · p. 16
• AuraLuxMuse has lower inference latency and higher EGSS than four sequence-to-sequence models under the reported setup, while retrieval preserves corpus-level cue validity.Final physical validity still depends on compatibility checks and conflict resolution.Source: Table S15 · p. 17, §E.2 · p. 17

E.3 Retrieval Baseline Diagnostics

• Random retrieval shares event timing and a console-valid corpus with the rule-based baseline, but its narrativity score is 2.70 versus 3.64 for similarity retrieval.Visual consistency is closer: 3.30 for random retrieval versus 3.55 for similarity retrieval.Source: §E.3 · p. 17

E.4 Delicate Component Mechanisms

• Adaptive K achieves the best reported objective and subjective scores among tested fixed and adaptive retrieval-count settings.All top-K sequences are considered because each may control only one stage region; conflicts are resolved before dispatch.Source: Table S16 · p. 17, §E.4 · p. 17
• Soft-alignment weight w=0.3 gives the strongest overall retrieval metrics; attribute mapping outperforms one-hot cue encoding across objective metrics and subjective evaluations.At w=0.7, FID increases to 61.21; the authors attribute weaker performance at higher weights to blurred discriminative boundaries.Source: §E.4 · p. 17

E.5 Metric Validity and Reliability

• AuraLuxMuse outperforms tested automated baselines across the EGSS expert-weight range; the selected default is λ=0.7, balancing expert alignment with general aesthetic perception in this study.Manual lighting is treated as an approximate upper bound, not a competing baseline.Source: §E.5 · p. 17, Figure S5 · p. 18
• Across ten automated variants, Spearman correlation between FID and EGSS is positive and significant in this sample (ρ=0.6848, p=0.0289), but the measures are complementary, not interchangeable.Pearson correlation is positive but not significant (r=0.3955, p=0.2579); the analysis excludes manual design and uses one learned feature space.Source: §E.5 · pp. 17–18

F.1 Software

• GrandMA2 provides the primary lighting-control interface, and MA3D supports simulation and previsualization; hardware-in-the-loop maps control signals to virtual and physical luminaires.This shared protocol is described as supporting spatiotemporal consistency across simulated and real-world lighting distributions.Source: §F.1 · p. 18

F.2 Lighting Fixtures

• Musilux represents four fixture classes—imaging lights, spotlights, soft lights, and floodlights—with different projection, focus, diffusion, and coverage roles.The represented fixture models are shown in Fig. S6.Source: §F.2 · p. 18, Figure S6 · p. 19

F.3 Stage Configurations

• Three orthographic stage views and a unified coordinate system register fixture locations and elevations for spatial alignment between simulated and physical environments.The stage geometry is stored in the GrandMA2 ecosystem and preserves fixture extrinsic properties across domains.Source: §F.3 · p. 19, Figure S8 · p. 20, Figure S9 · p. 20

F.4 Stage Lighting Console; F.5 Stage Lighting Interface

• GrandMA2 supports temporal sequencing, per-fixture mapping, and low-level parameter control through a physical console and hierarchical software interface.The interface includes timecode triggers, structured cue sequences, and encoders for fixture attributes.Source: §F.4 · pp. 20–21, §F.5 · p. 21, Figure S10 · p. 21

G Stage Lighting Implementation Procedure

• The procedure covers exporting learnable data from GrandMA2 and importing AuraLuxMuse-generated lighting into real-world simulations and applications.The workflow separates input-data preparation from output deployment.Source: G Stage Lighting Implementation… · p. 21

G.1 Stage Lighting Input

• Existing show files are exported as sequence and timecode XML, then converted into JSON structures for model training.The conversion uses read_sequence_cue.py and read_timecode.py because the model cannot directly use the original XML data.Source: §G.1 · pp. 22–23, Figure S13 · p. 23

G.2 Stage Lighting Output

• Deployment imports generated cue-sequence and timecode XML into GrandMA2, assigns the sequence to an executor, binds the timecode, and starts synchronized playback.Generated files are placed in the console import/export directory before import.Source: Figure S11 · p. 22, Figure S12 · p. 22, §G.2 · p. 23

H Dataset Split and Generalization Analysis

• Training contains approximately 150 minutes of paired music and lighting cues, about 80.1% of total Musilux duration.Program lengths vary, so the split is described by both counts and duration.Source: H Dataset Split and Generalization… · p. 23
• Validation includes six held-out programs, two per scenario, selected for six-style coverage and the availability of human-authored references.These references supply the main-paper manual comparison.Source: H Dataset Split and Generalization… · p. 23
• Testing uses six additional tracks, one per style, on an unseen stage; this controlled check of unseen music and layout is not a population-scale benchmark.The full-length expert/non-expert scoring also limits its practical sample size.Source: H Dataset Split and Generalization… · p. 23