Source-linked AI summary
SeMoCo: A Semantic-First Motion Codec for Motion Language Modeling
Tianlv Huang, Hetian Guo, Ziyi Cai, Song Wang, Yanping Zhang, Zipei Fan, Xuan Song, Guangming Wu, Xin Zheng
TL;DR
Existing motion tokenizers generally optimize reconstruction without explicitly separating semantic roles from fine-grained kinematic detail. SeMoCo introduces a semantic-first codec, a dual-axis motion language model, and Ω-MotionVerse; across reported comparisons, it achieves the best reconstruction accuracy among compared codecs and strong text-to-motion results.
Problem
Most motion tokenizers allocate representation capacity through reconstruction-driven hierarchies rather than explicit semantic roles for action meaning and kinematic detail.
Method
SeMoCo combines a semantically aligned primary code with residual kinematic codes in each motion packet, while a dual-axis model generates semantics across packets and kinematic refinements within packets.
Results
SeMoCo achieves the best reconstruction accuracy among compared codecs, while strong text-to-motion results demonstrate the effectiveness of its motion tokens for downstream generation.
Takeaways & Limitations
Semantic-first motion packets support language-conditioned motion generation while preserving a separate residual hierarchy for fine-grained kinematic reconstruction.
Takeaways & Limitations
The semantic and kinematic branches are not required to be information-exclusive, so the split does not assume strict disentanglement.
Abstract
from arXiv · showhide
Discrete motion representations have substantially advanced autoregressive text-to-motion generation. However, most motion tokenizers are optimized for reconstruction and do not explicitly allocate capacity according to semantic role. Action-level meaning and fine-grained kinematic detail must therefore be encoded through the same reconstruction-driven hierarchy. We introduce SeMoCo, a semantic-first motion codec, together with a dual-axis motion generator for language-conditioned motion generation. Each motion token contains one semantic token and a residual sequence of kinematic tokens. The generator models semantic progression across time and autoregressively refines the residual entries. We also construct $Ω$-MotionVerse, a large-scale, multi-source human-motion dataset unified under the SOMA representation. Across the reported comparisons, SeMoCo achieves the best reconstruction accuracy among the compared codecs, while strong text-to-motion results demonstrate the effectiveness of its motion tokens for downstream generation.
1 Introduction
Text-to-motion generation must translate language-level action intent and temporal structure into coordinated, detailed full-body motion, yet most motion tokenizers organize capacity around reconstruction residuals rather than explicit semantics. SeMoCo addresses this with semantic-first motion tokens, a dual-axis generator, and the Ω-MotionVerse dataset, achieving the best reconstruction accuracy among compared codecs.
- Text-to-motion generation must realize language-described intent through coordinated trajectories, contacts, articulation, and smooth dynamics.
- Most motion tokenizers optimize reconstruction, so their codebook hierarchy follows residual distortion reduction rather than explicit semantic roles.
- Its dual-axis generator models semantic progression across time and autoregressively refines residual kinematic tokens within each position.
- 1,000 hours of text-annotated human motion are unified in Ω-MotionVerse under the SOMA skeleton convention.
- SeMoCo achieves the best reconstruction accuracy among compared codecs, while strong text-to-motion results support its downstream motion-token generation.
- SeMoCo assigns semantic and kinematic information separate supervision and quantization paths within each motion token.
2 Related Work
Related work places semantic structure in motion representations, continuous latent spaces, or generation pipelines, while hierarchical generation commonly separates temporal modeling from within-step code completion. SeMoCo combines these ideas through a semantically supervised motion packet and temporal–depth factorization.
- Reconstruction-only VQ and RVQ pipelines order motion codes by distortion reduction rather than an explicit semantic target.
- Speech codecs such as Mimi and Qwen3-TTS provide the closest architectural precedents by splitting teacher-distilled semantic codes from residual acoustic detail.
- Motion methods organize hierarchical generation across residual depth, temporal scale, or skeletal–temporal scale.
- SeMoCo applies temporal–depth factorization to semantic-first motion packets, coupling a separately supervised representation with an established generation order.
- Continuous text-to-motion approaches use diffusion or flow objectives, whereas discrete approaches learn motion token sequences for generative modeling.
3 Method
SeMoCo factorizes motion tokens into semantically aligned codes and residual kinematic codes, then models their temporal and within-packet dependencies separately. The method also introduces a large, standardized motion corpus for training and evaluation.
- Semantic-First Motion Codec: SeMoCo represents each motion interval as a semantic-to-kinematic packet with a semantic code and residual codes for fine-grained detail.The semantic and kinematic branches use distinct supervision and quantization paths while sharing a reconstruction space.
- Motion Representation: A temporal encoder compresses 50-Hz transition sequences by four, producing motion intervals summarized at 12.5 Hz.All codes within a packet describe the same temporally downsampled interval.
- Semantic-First Motion Codec: SeMoCo separates semantic and kinematic roles into parallel paths: a semantic VQ is aligned with motion-language embeddings, while an independent RVQ preserves reconstruction detail.The quantized branch outputs are mapped to a common decoder space and fused additively; semantic supervision is used during codec training.
- Dual-Axis Motion Language Model: The dual-axis generator predicts semantic codes across packets with a temporal Transformer and autoregressively generates residual kinematic codes within each packet.Both axes are trained jointly with teacher forcing and codebook-wise cross-entropy losses, with language representations available as task context.
- Ω-MotionVerse: Ω-MotionVerse contains 909,913 text–motion pairs and 1,006 hours of full-body motion standardized under the SOMA skeleton convention.Its four source groups include monocular-video reconstructions and marker-based motion capture from diverse acquisition settings.
5 Experiments
Experiments evaluate SeMoCo for reconstruction, motion prediction, and text-to-motion generation under task- and representation-specific protocols. Results show strong reconstruction and generation performance, with semantic supervision producing a measurable reconstruction–generation trade-off.
- Motion Reconstruction: SeMoCo reconstructs motion more accurately than MoMask, MotionGPT3, and MotionMillion across all reported metrics.Reconstruction is evaluated without text conditioning on HumanML3D, using native motion representations and pelvis-aligned errors in millimeters.
- Motion Prediction: Ours-Lite achieves the lowest motion-prediction error across all metrics, outperforming the larger Ours-Base under the 0.5-second observation and 2-second prediction protocol.The result reverses the text-to-motion scale ordering, so larger generator capacity does not provide an advantage for this prediction setting.
- Text-to-motion: Ours-Base improves TMR-SOMA R@1 from .326 to .422 and FID from .920 to .913 compared with Ours-Lite, while Kimodo remains stronger in retrieval.The two evaluator spaces are separate because methods produce outputs in different native motion representations.
- Text-to-motion: Across provenance subsets, Kimodo achieves the strongest overall performance, while Ours-Base leads MotionGV, Kimodo leads bones-seed, and HyMotion leads HumanML3D.These results show that relative model ordering varies across source subsets.
- Text-to-motion: Qualitative comparisons show semantically consistent and temporally coherent motions across three representative text prompts.Generated sequences from HY-Motion, Kimodo, and Ours-Base are visualized at matched timestamps.
- Tokenizer ablation: Split-branch RVQ with semantic supervision obtains FID .186 and mean R@1 .484, while Plain RVQ remains strongest on R@2–R@5 and median rank.Within the split layout, semantic supervision changes FID from .226 to .186 and R@1 from .319 to .484, while MPJPE-77 rises from 13.70 to 15.93 mm.
6 Conclusion
The conclusion presents SeMoCo as a semantic–kinematic motion codec paired with a dual-axis generator, evaluated across reconstruction, text-to-motion, and prediction. Shared-split diagnostics report substantially lower reconstruction errors and inspect temporal smoothness.
- Conclusion: SeMoCo couples a teacher-aligned semantic code and kinematic residual hierarchy with a dual-axis packet generator.The generator models semantic progression across time and autoregressively refines kinematic residuals within each motion token.
- Shared-Split Reconstruction: 12.83 mm MPJPE-22 is reported for SeMoCo, compared with 78.95 mm for MotionMillion, 90.02 mm for MoMask, and 96.01 mm for MotionGPT3.All methods are evaluated on the same test split in a common pelvis-aligned output space over 22 SMPL body joints.
- Shared-Split Reconstruction: SeMoCo’s mean and median reconstruction errors are 19.2 and 17.4 mm, with a worst single-sequence error of 79 mm.The corresponding worst-sequence errors for MoMask, MotionGPT3, and MotionMillion are 324, 428, and 691 mm.
- Reconstruction Diagnostics: At the right wrist, SeMoCo’s acceleration standard deviation is 10.4 versus ground truth 12.1, with RMSE values of 9.3, 22.7, and 18.7 mm/frame2 across the reported methods.The diagnostic compares temporal smoothness on one clip and contrasts SeMoCo with MotionGPT3 and MoMask.
A.3 Full TMR-SOMA Motion Metrics
The supplementary evaluation reports motion–text retrieval, quality, and encoder comparisons across evaluator spaces and motion diagnostics. Results show that evaluator choice changes encoder rankings, while Kimodo leads several quality metrics and Ours-Base leads on MotionGV.
- Full TMR-SOMA Metrics: Table 7 reports MM-Dist, alignment, diversity, foot skating, and jerk by dataset provenance.MM-Dist measures paired embedding distance; Align measures paired embedding cosine similarity; Diversity measures distances among generated-motion embeddings.
- Full TMR-SOMA Metrics: Kimodo attains the lowest MM-Dist and highest Align overall and on BONES-SEED, while Ours-Base leads both metrics on MotionGV.Ours-Base is second on MM-Dist and Align in the Overall pool and on BONES-SEED.
- Motion-Quality Diagnostics: Kimodo reports the lowest FootSkate and jerk on every subset, while the Ours variants reach their highest jerk on HumanML3D.
- Text-Encoder Variants: The two evaluator tracks order text encoders differently: SigLIP has the lowest HML-263 FID, whereas Flan-T5 is stronger on overall TMR-SOMA retrieval.Qwen3 is last on TMR-SOMA, with median rank 42.02 versus 1.70 for Flan-T5.
- Reconstruction Diagnostics: Figure 5 compares cumulative per-sequence MPJPE distributions, with each method evaluated on its own test set.The figure includes a 0–100 mm panel and an upper-tail panel covering the upper 12% of each distribution.
- Motion-Quality Diagnostics: Figure 6 shows right-wrist and right-ankle vertical acceleration over the busiest 150-frame window of the most dynamic test clip.All methods are resampled to 30 fps and pelvis aligned; MotionMillion is excluded.
A.5 Source-wise Tokenizer Ablation
The source-wise tokenizer ablation finds the split semantic route most consistently advantageous on BONES-SEED, while HumanML3D and MotionGV favor different or mixed variants.
- Source-wise Results: The split semantic route performs most consistently on BONES-SEED, achieving the best FID and every retrieval rate while tying the best median rank.
- Source-wise Results: HumanML3D favors Plain RVQ in FID and semantic Single-chain RVQ in retrieval, whereas MotionGV has mixed metric rankings.
A.6 Full Tokenizer Reconstruction Ablation
The reconstruction ablation shows that semantic supervision increases MPJPE-77, with the split layout remaining more accurate than the single-chain layout under both settings.
- Reconstruction Comparison: Semantic supervision raises MPJPE-77 by 2.23 mm in the split layout, from 13.70 to 15.93 mm.
- Reconstruction Comparison: Semantic supervision raises MPJPE-77 by 2.94 mm in the single chain, from 15.60 to 18.54 mm.
- Reconstruction Comparison: The split layout is more accurate than the single chain under both supervision settings.Among retained variants, MPJPE-77 differs by 0.01 mm on BONES-SEED, 0.57 mm on MotionGV, and 0.34 mm on HumanML3D.
A.7 Semantic Branch Design
The semantic constraint is most effective when routed through a parallel branch: it preserves reconstruction accuracy more closely than placement inside a residual chain while achieving stronger teacher agreement.
- Constraint Placement: A parallel semantic branch costs 0.40 mm relative to the unsupervised reconstruction reference.
- Constraint Placement: Applying the constraint to the first residual level costs 2.65 mm, increasing to 3.07 mm across the first two levels.
- Teacher Agreement: The parallel branch reaches .878 cosine similarity and .672 R@10 with the frozen teacher.The corresponding single-level values are .770 and .543, while the first-two-level values are .846 and .627.
- Design Interpretation: Placing the semantic constraint outside the residual chain buys semantic alignment at a smaller reconstruction cost than placing it inside.
B Implementation and Data Details
Ω-MotionVerse combines multi-source text–motion data under common SOMA conventions, with group-level splitting, source-specific conversion, and deduplication policies.
- Corpus composition: 909,913 text–motion pairs from approximately 1,006 hours of motion form Ω-MotionVerse across 699,152 source groups.MotionGV and BONES-SEED together account for 98.3% of all text–motion pairs.
- Standardization: Every source is converted to the SOMA77 skeleton, resampled to 50 Hz, floor aligned, and canonicalized at its first frame.Canonicalization removes initial planar translation and heading while retaining an anchor with the initial absolute pose and root transform.
- Partitioning: Splits are assigned over source recordings rather than individual clips, keeping clips from one recording in the same partition.The resulting corpus contains 727,941 training, 45,492 validation, and 136,480 test pairs.
- Partitioning: The Kimodo benchmark test portion of BONES-SEED is forced into the test split, with no Kimodo test recording in training or validation.It contains 13,884 recordings and 33,970 pairs, representing 64.4% of BONES-SEED test pairs.
- Deduplication: Repeated submissions are removed with content hashes, while HumanML3D preserves multiple captions for one retained motion as distinct text–motion pairs.HumanML3D has 14,094 pairs corresponding to 12,886 unique clips.
B.2 TMR-SOMA Training and Use
TMR-SOMA supplies frozen motion–text representations for semantic supervision and evaluation, while SeMoCo and its dual-axis generator separate temporal semantic modeling from local kinematic refinement.
- TMR-SOMA: TMR-SOMA serves both as SeMoCo’s frozen semantic teacher and as a separately adapted, frozen motion–text evaluation space.It is a retrieval model rather than a text-to-motion generator.
- SeMoCo tokenizer: SeMoCo emits 16-code motion packets at 12.5 Hz, with one semantic code and 15 kinematic residual codes per packet.The semantic and kinematic branches receive separate learned projections of the full encoder latent and are summed in a shared decoder space.
- Semantic supervision: Semantic supervision uses a frozen TMR-SOMA motion descriptor for each 64-frame window, corresponding to 1.28 seconds or 16 motion packets.A temporal semantic head aggregates the sequence of quantized semantic embeddings.
- Model variants: Ours-Lite and Ours-Base use the same tokenizer, codebooks, objective, code predictor, and decoder, so scaling changes temporal capacity rather than motion resolution.Both variants consume the same 16-code packets and use the same frozen SeMoCo decoder.
- Temporal-axis modeling: The temporal Transformer predicts semantic codes and EOS across 12.5-Hz packets, modeling action progression, duration, ordering, and text–motion-history consistency.Sixteen code embeddings are summed at each timestep, so the temporal sequence does not expand by 16.
- Codebook-axis modeling: A causal depth decoder autoregressively generates the 15 kinematic residual codes conditioned on the temporal hidden state and preceding codes in the packet.The code predictor is fixed across model sizes, while Ours-Lite adds a learned bridge that Ours-Base does not use.
B.7 Evaluation Protocols
Evaluation uses distinct retrieval spaces and native task protocols, with explicit definitions for motion quality, reconstruction, and prediction metrics.
- Evaluator spaces: HumanML3D uses HML-263, whereas TMR-SOMA is adapted only on the Ω-MotionVerse training partition and frozen before evaluation.The spaces differ in motion representations, embedding models, and normalizations.
- Evaluator spaces: TMR-SOMA provenance results use 256 clips per repeat and report the mean and standard deviation over 30 repeats.This repeated-sampling protocol applies to each provenance result.
- Motion quality: FootSkate measures mean horizontal ankle-and-toe speed during heuristically detected contact frames, with lower values indicating less foot sliding.Contact requires 3-D speed below 0.15 m/s and height below 0.12 m.
- Motion quality: Jerk is evaluated as mean joint-jerk magnitude in m/s^3 after coordinate conversion and resampling, without additional smoothing.Lower values indicate smoother motion.
- Reconstruction: MPJPE measures pelvis-aligned joint-position error, while PA-MPJPE additionally applies per-sequence similarity alignment; MPJPE-22 and MPJPE-77 cover different joint sets.MPJPE-22 uses 22 SMPL joints and MPJPE-77 uses all 77 SOMA joints.
- Motion prediction: Motion prediction observes the first 0.5 seconds and evaluates the following two seconds, with ADE, FDE, and best-of-50 computed over native valid pools and frame rates.Best-of-50 selects the lowest error among 50 independent samples for each clip.