Streaming Multi-Track Timeline Control for 3D Human Motion Generation

• Existing work handles streaming instruction changes or simultaneous composition separately, leaving independently timed, overlapping instructions unsupported by the described interfaces.Source: §1 · p. 2
• TimelineControl combines interval-aware multi-prompt conditioning with part-aware motion generation to incorporate new instructions alongside ongoing actions.Source: §1 · p. 2
• Experiments on TimelineMotion and MTT show strong semantic alignment and temporal adherence, outperforming evaluated streaming baselines, including ActionPlan retrained on the same data.Source: §1 · p. 2
• TimelineMotion pairs motion with overlapping instruction intervals and body-part annotations, using STMC-based HumanML3D composition followed by refinement and filtering.Source: §1 · p. 2

• STMC combines temporal and spatial composition using text intervals assigned to body regions, but requires the complete timeline before synthesis; none of the reviewed methods supports streaming multi-track timeline control.Source: §2 · pp. 2–3
• Text-to-motion research spans raw motion, continuous latents, and discrete tokens, with autoregressive, diffusion, masked-token, and compositional generation strategies.Source: §2 · p. 2
• Part-aware methods use separate body streams, skeletal structure, inter-part communication, or part-specific text grounding; TimelineControl uses six anatomical latents in a causal autoencoder for part-wise streaming denoising.Source: §2 · p. 3
• Streaming methods address incremental motion or changing text conditions, and TimelineControl extends ActionPlan to prompts that arrive independently during generation and may overlap.Source: §2 · p. 3

• Prompt-conditioned rectified flow generates a latent motion sequence from text using a frozen pretrained motion autoencoder’s latent space.Source: §3 · p. 3
• The flow interpolates clean latent motion and Gaussian noise, trains a velocity predictor by flow matching, then transports noise toward clean motion at inference.Source: §3 · p. 3
• TimelineControl processes part and whole-body features separately, using block-causal self-attention for part features and online multi-prompt conditioning for both.Source: Figure 2 · p. 4
• Each instruction is a text prompt with a start and end interval; the active window retains emitted motion as fixed history and regenerates only its continuation when instructions arrive.Source: §3 · p. 4

4 METHOD

• Built on ActionPlan and trained with Diffusion Forcing, TimelineControl combines online multi-prompt conditioning with part-aware motion generation.The framework uses part-structured representations and global features to guide body regions while maintaining whole-body awareness.Source: §4 · p. 4

4.1 CAUSAL PART-STRUCTURED AUTOENCODER

• The causal autoencoder represents motion with six body-region latents and an explicit root trajectory, then uses causal temporal convolutions and skeletal message passing.Its decoder also conditions on root yaw rate, horizontal velocity, and height; the autoencoder is frozen during generator training.Source: §4.1 · pp. 4–5

4.2 ONLINE MULTI-PROMPT CONDITIONING

• Independent prompt embeddings are matched to motion positions using temporal and interval encodings, while a visibility matrix restricts contributions to each instruction’s interval.The visibility matrix allows different motion positions to admit different subsets of the received instructions.Source: §4.2 · p. 5

4.3 PART-AWARE DENOISING

• Part-aware denoising combines block-causal self-attention with classifier-guided instruction relevance and gated fusion into whole-body features.Attention within an arrival block is bidirectional and can access prior motion; the block mask is enforced during training.Source: §4.3 · p. 6

4.4 TRAINING DATA

• TimelineMotion contains 44,944 sequences with overlapping instruction intervals, composed from HumanML3D motions and refined to smooth discontinuities and filter residual abrupt motion or jitter.Body-part annotations guide composition and train the part-relevance classifier.Source: §4.4 · p. 6
• Across TimelineMotion and MTT, TimelineControl outperforms all baselines on sequence alignment, body parts, motion quality, and temporal adherence.The reported comparison covers both benchmarks and all four evaluation aspects.Source: Table 1 · p. 7

5.1 EXPERIMENTAL SETUP

• The evaluation uses TimelineMotion and MTT for streaming, HumanML3D for non-streaming generation, and metrics for alignment, motion distributions, continuity, and interval adherence.Streaming baselines receive prompts at annotated start times; methods without multi-interval conditioning use updated or joined captions, while STMC is evaluated offline with the full timeline.Source: §5.1 · p. 7

5.2 COMPARISON WITH BASELINES

• TimelineControl achieves state-of-the-art performance on the two benchmarks supporting streaming inference and beats offline STMC on TimelineMotion, with FID decreasing from 0.055 to 0.017.Its sequence- and body-part-alignment gains persist against ActionPlan retrained jointly on TimelineMotion and HumanML3D.Source: §5.2 · p. 8
• On HumanML3D, TimelineControl achieves the highest retrieval performance (0.874 vs 0.830) and the second-best FID (0.027 vs 0.022).The authors report that timeline-benchmark improvements do not significantly impact standard text-to-motion generation.Source: §5.2 · p. 8, Table 2 · p. 8
• In qualitative examples with overlapping instructions, TimelineControl maps actions to the correct intervals while baselines ignore, delay, or confuse instructions.Figure 3 compares STMC, ActionPlan*, and TimelineControl on TimelineMotion and MTT; red boxes mark failure regions.Source: §5.2 · p. 8, Figure 3 · p. 8
• In the user study, TimelineControl wins 66.2% against STMC and 72.1% against ActionPlan* on text-motion consistency.The study also evaluates motion quality and temporal adherence.Source: §5.2 · p. 8

5.3 ABLATION STUDIES

• Removing attention masks reduces retrieval and increases FID, while removing RoTE lowers TimelineMotion R@3 from 0.603 to 0.572.The ablations support both attention masking and encoding timeline arrival times and order.Source: §5.3 · p. 9, Table 4 · p. 9
• Removing part-aware bias lowers most metrics, and replacing part-aware processing with global-state prediction raises FID from 0.017 to 0.030 on TimelineMotion.The global state still receives online multi-prompt conditioning.Source: §5.3 · p. 9, Table 4 · p. 9

5.4 APPLICATIONS

• Applications demonstrate spatial conditioning for position control in animation and robotics, and real-time humanoid execution using SONIC as a motion tracker.The humanoid demonstration uses generated motions.Source: §5.4 · pp. 9–10

6 CONCLUSION

• TimelineControl combines interval-aware text conditioning with part-aware denoising to coordinate concurrent actions, supported by the TimelineMotion dataset and experiments showing improved performance over evaluated baselines on TimelineMotion and MTT.The paper also demonstrates spatial conditioning and humanoid execution; online revision, cancellation, and resolving conflicting requests while preserving continuity remain future work.Source: §6 · p. 10

AI USE STATEMENT

• Generative AI assisted with manuscript polishing, code implementation and debugging, and body-part annotation of motion descriptions.The annotation procedure and label review are described in Section 4.4 and Appendix A.Source: AI USE STATEMENT · p. 10

A TIMELINEMOTION CONSTRUCTION DETAILS

• Figure 4 summarizes the TimelineMotion construction pipeline, with annotation and filtering details provided in the appendix.The supplied passage identifies the pipeline and the location of additional details but does not specify those steps.Source: A TIMELINEMOTION CONSTRUCTION DETAILS · p. 15

A.1 BODY-PART ANNOTATION

• Qwen3-8B annotates HumanML3D captions with body-part labels and motion attributes used for composition, supervision, and evaluation.Coarse upper-, lower-, or full-body labels expand into anatomical regions when explicit regional labels are absent.Source: §A.1 · p. 15

A.2 TIMELINE CONSTRUCTION

• TimelineMotion overlaps clips only when their body regions, root motion, locomotion, poses, and stance changes are compatible, excluding full-body actions.Transitions are refined with interpolation, and sequences meeting specified boundary-artifact thresholds or containing non-finite values are rejected.Source: §A.2 · p. 16

B.1 CAUSAL PART-STRUCTURED AUTOENCODER

• The causal part-structured autoencoder encodes and reconstructs six anatomical regions using temporal convolutions and skeleton-graph message passing, while the root trajectory bypasses the body encoder.Its objective includes Gaussian negative log-likelihood reconstruction, latent-posterior regularization, and a temporal loss matching second-order differences.Source: Figure 5 · p. 16, §B.1 · p. 16

B.2 DENOISER ARCHITECTURE AND TRAINING SETTINGS

• The global branch uses four stages of four Transformer layers, with width 1024, feed-forward width 4096, and 16 attention heads.The input token encoder has one attention layer; the classifier-predicted attention bias uses λr = 0.25, and γ = 4 is used as a common timestamp scale for motion queries and interval centers and radii.Source: §B.2 · p. 17

B.3 BACKBONE OBJECTIVES AND DETAILS

• Training combines motion and action-plan flow targets, weights valid targets jointly, and uses auxiliary body-latent flow, body-part classification, and root-motion losses.Action-plan targets derive from BABEL action annotations, while padding and other excluded positions receive zero weight.Source: §B.3 · p. 17

B.4 STREAMING INFERENCE

• Streaming inference preserves committed motion, discards and regenerates uncommitted predictions when instructions arrive, and defaults open-ended commands to five-second intervals.Sampling progressively denoises motion after completing the action plan; generated latents become fixed prefix context for later windows.Source: §B.4 · p. 17

C.1 BASELINE TEXT CONDITIONING

• Single-caption baselines receive a caption recomputed at every instruction boundary, joining currently active prompts and withholding future prompts until arrival.For example, overlapping prompts A on [0, 5) and B on [3, 8) produce A, A while B, then B.Source: §C.1 · pp. 17–18

C.2 EVALUATION METRICS

• Alignment uses TMR scores, temporal adherence uses interval-labeled two-second windows, and temporal AP and ROC-AUC peak at the original intervals in shift tests.Runtime measurement uses batch-one FP32 sampling with 78 latents, measuring latency to the first clean token and intervals between later tokens.Source: §C.2 · p. 18, §C.3 · p. 18, §C.4 · pp. 18–19

C.5 USER STUDY DETAILS

• In a study of 20 generated timelines, 18 participants consistently favored TimelineControl over STMC and ActionPlan* for text alignment, motion realism, and temporal adherence.The comparisons comprised 40 randomized video-pair questions.Source: §C.5 · p. 19, Figure 7 · p. 21

D.1 SELF-CONDITIONED TRAINING

• Removing self-conditioned training lowers retrieval, raises FID on both datasets, and increases TimelineMotion boundary displacement, while Temporal AP is unchanged at reported precision.ROC-AUC decreases slightly.Source: §D.1 · p. 19, Table 6 · p. 19

D.2 TRAINING-WINDOW STRATEGIES

• The mixed training-window policy achieves the highest sequence retrieval and lowest FID on both datasets, and the highest TimelineMotion Temporal AP and ROC-AUC.Random starts yield the lowest FID+, while instruction starts yield the lowest boundary displacement.Source: §D.2 · p. 21, Table 7 · p. 21

D.3 ATTENTION NORMALIZATION

• Sigmoid weighting yields higher sequence retrieval, lower whole-motion FID, and higher TimelineMotion Temporal AP and ROC-AUC than softmax normalization.Softmax instead yields slightly lower TimelineMotion FID+ and boundary displacement.Source: §D.3 · p. 21, Table 8 · p. 21

D.4 MULTI-TEXT CONDITIONING IN ACTIONPLAN

• TimelineControl further improves sequence and body-part alignment, FID, and temporal adherence over multi-text ActionPlan, while matching its boundary displacement at reported precision.Multi-text conditioning itself improves sequence retrieval and temporal adherence over jointly retrained single-caption ActionPlan.Source: §D.4 · p. 21, Table 9 · p. 22

D.5 ABLATION OF BLOCK-CAUSAL SELF-ATTENTION

• Replacing block-causal attention with a standard causal mask lowers TimelineMotion sequence R@3 from 0.603 to 0.586 and body-part R@3 from 0.374 to 0.364.Temporal AP falls from 0.832 to 0.826 and ROC-AUC from 0.772 to 0.767.Source: §D.5 · p. 22, Table 10 · p. 22

D.6 ABLATION OF ONLINE MULTI-TRACK ATTENTION

• Merging active instructions into one caption lowers TimelineMotion sequence R@3 from 0.603 to 0.552 and raises whole-motion FID from 0.017 to 0.046.The comparison supports separate online instruction conditioning.Source: §D.6 · p. 22, Table 11 · p. 22

D.7 ABLATION OF AUXILIARY BODY-PART LOSS

• Removing auxiliary body-part supervision causes small decreases on both datasets, indicating a modest benefit to semantic alignment and temporal adherence beyond the part-aware architecture.Source: §D.7 · pp. 22–23, Table 12 · p. 23

D.8 ABLATION OF ROOT-VELOCITY LOSS

• Removing root-velocity supervision raises root jitter from 1.417 to 1.757 and body jitter from 0.614 to 0.680 on TimelineMotion.TimelineMotion whole-motion FID rises from 0.017 to 0.021, while sequence R@3 falls from 0.603 to 0.595; HumanML3D FID is similar (0.026 vs. 0.027).Source: §D.8 · p. 23, Table 13 · p. 23

E EXTENSION TO SPATIAL CONTROL

• Spatial adaptation adds masked, normalized spatial targets to the generator while retaining the streaming sampler and allowing text-only generation with empty spatial masks.The denoiser and condition projection are fine-tuned together while the autoencoder remains frozen.Source: E EXTENSION TO SPATIAL CONTROL · p. 23, §E.3 · p. 24

E.1 ADDITIONAL QUALITATIVE RESULTS

• Additional qualitative comparisons show TimelineControl, STMC, and ActionPlan* under sequential and overlapping instructions.Figure 8 marks a failure part with a red box.Source: §E.1 · p. 23, Figure 8 · p. 24

E.2 SPATIAL CONDITIONS

• Spatial controls include root waypoints and paths, body and end-effector keyframes, and foot contacts, with training using single, combined, or absent conditions.Combined conditioning uses the union of masks for two control types.Source: §E.2 · pp. 23–24

E.3 ADAPTATION

• The spatial objective combines generator, reconstruction, goal-matching, and kinematic-consistency losses.Reconstruction and kinematic losses use valid future frames; goal loss applies only to supplied targets.Source: §E.3 · p. 24

E.4 EVALUATION

• Spatial evaluation synchronizes control availability with prompt arrivals and measures target errors only on supplied frames and joints.Position errors are reported for root, body, and end-effector targets, with F1 for specified foot-contact labels.Source: §E.4 · p. 24

E.5 RESULTS

• The extension achieves lower body and end-effector errors and higher contact F1 and M2T than ARDY on both datasets, while ARDY is more accurate for horizontal root targets.Table 14 compares joint text-and-spatial-target conditioning on HumanML3D and TimelineMotion.Source: §E.5 · p. 24, Table 14 · p. 25
• Figure 9 illustrates root trajectories, waypoints, wrist targets, full-body keyframes, and combinations of root trajectories with body targets.Bars mark instruction intervals; poses share a world coordinate frame within each panel and darken over time.Source: Figure 9 · p. 25

F ROBOT MOTION EXECUTION

• Real-time robot execution combines upper-body gestures with walking and dancing, while G1 simulations show kicks, phone gestures, violin playing, and jumping jacks.The physical robot uses SONIC’s motion-tracking controller; simulation examples include overlapping instructions and transitions between actions.Source: F ROBOT MOTION EXECUTION · p. 25, Figure 11 · p. 26, Figure 10 · p. 26

G LIMITATIONS

• TimelineMotion composes HumanML3D motions rather than using naturally collected streaming interactions, so it may not capture their full diversity and complexity.The pipeline filters incompatible actions and refines composition boundaries, but the authors identify naturally collected interactive data as future work.Source: G LIMITATIONS · p. 26
• Instruction timing requires explicit intervals; without an end time, the system can extend fixed-duration intervals but does not infer termination from language or motion state.The paper gives five seconds as an example fixed duration.Source: G LIMITATIONS · p. 26
• Part-aware supervision depends on automatic annotations and six fixed anatomical regions, which may limit guidance when annotations err or actions involve more complex cross-part interactions.The authors identify more flexible part grounding as a future direction.Source: G LIMITATIONS · p. 26