Source-linked AI summary
Dense Motion Captioning
Shiyao Xu, Benedetta Liberatori, Gül Varol, Paolo Rota
TL;DR
Motion understanding for longer, complex 3D sequences remains relatively unexplored, and existing datasets often lack detailed temporal annotations. The paper introduces Dense Motion Captioning, CompMo, and DEMO; DEMO outperforms UniMotion on CompMo and an adapted benchmark, while the dataset’s temporal composition remains unconstrained.
Problem
Motion-understanding work has focused largely on retrieval or short isolated captions, while longer complex sequences and precise temporal annotations remain limited in existing datasets.
Source: §1 · p. 1Method
The paper defines Dense Motion Captioning, builds timestamped CompMo sequences, and trains DEMO, an LLM with a motion adapter, through motion-language alignment and dense-caption tuning.
Source: §1 · pp. 1–2, §4.2 · p. 4, §4.4 · p. 5Results
DEMO outperforms UniMotion, particularly on CompMo, with +34.1/3.9% improvements in tIoU across CompMo and H3D ∩BABEL.
Source: §5.1 · p. 6Takeaways & Limitations
Dense motion captioning supports developing systems for more precise, body-centric understanding of human movement from 3D data.
Source: §6 · p. 8CompMo does not constrain action order, so its generated sequences can be incoherent; modeling causal relationships between actions is outside the work’s scope.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Recent advances in 3D human motion and language integration have primarily focused on text-to-motion generation, leaving the task of motion understanding relatively unexplored. We introduce Dense Motion Captioning, a novel task that aims to temporally localize and caption actions within 3D human motion sequences. Current datasets fall short in providing detailed temporal annotations and predominantly consist of short sequences featuring few actions. To overcome these limitations, we present the Complex Motion Dataset (CompMo), the first large-scale dataset featuring richly annotated, complex motion sequences with precise temporal boundaries. Built through a carefully designed data generation pipeline, CompMo includes 60,000 motion sequences, each composed of multiple actions ranging from at least two to ten, accurately annotated with their temporal extents. We further present DEMO, a model that integrates a large language model with a simple motion adapter, trained to generate dense, temporally grounded captions. Our experiments show that DEMO substantially outperforms existing methods on CompMo as well as on adapted benchmarks, establishing a robust baseline for future research in 3D motion understanding and captioning.
1. Introduction
Work on 3D motion and language has emphasized generation and editing, while understanding longer, complex motions with temporal precision remains underexplored. The paper introduces Dense Motion Captioning and presents CompMo and DEMO to address this gap.
1. Introduction
- Dense Motion Captioning detects and captions semantically meaningful actions in continuous motion, assigning each action precise start and end times.Unlike single-motion captioning, it segments a continuous stream into temporally localized action units.Source: §1 · p. 1
- Existing benchmarks often lack complex motion and precise annotations, while current datasets tend to contain isolated or few simple actions or fragmented, inconsistent labels.A preliminary HumanML3D experiment found a notable performance drop on longer sequences containing multiple actions.Source: §1 · pp. 1–2
- CompMo provides long motion clips with multiple actions, detailed captions, and temporal boundaries, and DEMO generates detailed, temporally aligned captions using an LLM and a simple motion adapter.DEMO is trained first to align motion and language modalities, then fine-tuned for dense caption generation.Source: §1 · p. 2
- The paper evaluates DEMO on CompMo and existing motion-language datasets adapted to dense captioning, establishing a benchmark for the task.The authors summarize their contributions as DMC, CompMo, and a DEMO baseline with experiments.Source: §1 · p. 2
2. Related Work
Prior motion-language work includes generation, retrieval, and captioning, but existing approaches and datasets differ in temporal granularity and whether they describe action segments. Dense Motion Captioning brings temporally precise captioning to 3D motion.
3D Human Motion-Language Datasets
- CompMo provides sequence-level natural-language descriptions with precise timestamps, unlike BABEL’s short labels, HumanML3D’s whole-clip captions, and FineMotion’s uniform snippets.BABEL includes subsequence action labels and durations; FineMotion segments HumanML3D motions at uniform intervals irrespective of action semantics.Source: §2 · p. 2
Dense Video Captioning
- Dense Motion Captioning adapts dense video captioning’s temporal localization and segment description paradigm to 3D human motion.Dense video captioning evaluation combines caption metrics with temporal overlap measures; SODA aligns captions before scoring them.Source: §2 · p. 2
Human Motion Understanding
- Prior motion-language models demonstrate generation and description capabilities, but their accuracy remains limited on tasks requiring temporal precision in longer continuous motions.The passage attributes this limitation to models not being trained to capture or describe subsequences within longer motions.Source: §2 · p. 2
Related Temporal Localization Work
- BABEL-TAL localizes actions using fixed classes, while TMR and UniMotion explore retrieval or frame-level captioning; DEMO instead generates free-form descriptions with segment timestamps.UniMotion treats captioning as closed-vocabulary retrieval and does not provide a quantitative benchmark.Source: §2 · pp. 2–3
3. From Simple to Complex Motions
The section examines whether current models handle complex motions, then describes CompMo’s construction as a dataset for dense captioning with precise temporal annotations.
3.1. Can Current Models Understand Complex Human Motions?
- The HumanML3D analysis partitions 29,228 mirrored augmented motions into 17,512 complex and 11,716 simple examples using verb counts as a proxy for sub-actions.The test subsets contain 2,663 complex and 1,721 simple motions.Source: §3.1 · p. 3
- Most tested state-of-the-art models show a considerable performance drop on HumanML3D’s complex subset, motivating datasets with greater temporal complexity.Complex motions are those whose descriptions have two or more verbs; simple motions have no more than one.Source: §3.1 · p. 3
3.2. CompMo: A Complex Motion Dataset
- CompMo is presented as the first large-scale dataset designed for dense 3D motion captioning with precise timestamps and longer motion sequences.Its captions average 37.74 words, compared with 12 for HumanML3D and 11.06 for BABEL.Source: §3.2 · pp. 3–4
- The construction pipeline composes descriptions from two to ten atomic actions, assigns timestamps, and generates corresponding motions using temporal and body-part stitching.Atomic actions are drawn from generated motions and the HumanML3D simple set; segment durations use source durations with small random perturbations.Source: §3.2 · p. 4
- CompMo contains 60,000 timestamp-annotated motion-text pairs, with sequences averaging 39.88 seconds and an 80%/10%/10% train-validation-test split.The paper describes these sequences as significantly longer than those in existing datasets.Source: §3.2 · p. 4
4. DEMO: Dense Motion Captioning Model
DEMO turns long 3D motion sequences into dense captions with temporal boundaries by combining an LLM with a motion adapter. Its task formulation, continuous motion representation, and two-stage training are designed for temporally localized caption generation.
4.1. Problem Formulation
- Dense Motion Captioning generates a caption and start and end times for each detected atomic action in a 3D motion sequence.Unlike single-motion captioning, it requires both temporal localization and natural-language generation.Source: §4.1 · p. 4
4.2. Method
- DEMO fine-tunes an LLM to autoregressively generate dense, temporally aligned captions from long and complex 3D motion sequences.A motion adapter maps continuous motion into the LLM’s language-compatible embedding space.Source: §4.2 · p. 4
4.3. Motion Representation
- DEMO maps motion continuously into language space with a lightweight encoder and linear projection, avoiding intermediate discretization.For long inputs, it processes fixed-size overlapping windows independently to capture localized motion patterns.Source: §4.3 · pp. 4–5
4.4. Training Strategy
- Training first aligns the motion adapter to language using timestamp-free HumanML3D motion-text pairs, then tunes dense caption generation on CompMo with temporal intervals.The second stage applies LoRA to the LLM while jointly fine-tuning it with the pretrained motion adapter; both stages use the same objective.Source: §4.4 · pp. 5–6
5. Experiments
DEMO outperforms UniMotion on dense motion captioning, especially on CompMo, while ablations favor mixture-denoising data generation, two-stage training, and continuous motion encoding. The experiments evaluate temporal localization, caption quality, and motion-caption alignment across CompMo and H3D ∩BABEL.
5. Experiments
- Experiments use CompMo and H3D ∩BABEL, assessing dense captioning, temporal localization, and motion-caption alignment.CompMo has 60,000 sequences split 48,000/12,000 for training/testing; H3D ∩BABEL has 7,056/1,325 sequences.Source: §5 · p. 6
5.1. Comparative Results
- +34.1/3.9% improvements in tIoU and +13.2/5.1% on SODA metrics show DEMO outperforming UniMotion on CompMo and H3D ∩BABEL, particularly CompMo.UniMotion frame-level predictions were aggregated into temporal segments for comparison; DEMO generates captions openly, whereas UniMotion retrieves from a precomputed vocabulary.Source: §5.1 · p. 6, Table 2 · p. 7
- DEMO more often segments the correct number of atomic actions and produces captions that align better with ground-truth style, though it occasionally omits an action.Some captions differ in wording from the ground truth while conveying equivalent meaning.Source: §5.1 · pp. 6–7, Figure 4 · p. 8
5.2. Ablation Study
- The mixture-denoising data-generation strategy consistently yields superior performance to the tested alternatives.Alternatives included data generated from scratch, HumanML3D-only actions, and direct or smoothed concatenations of HumanML3D motions.Source: §5.2 · p. 7
- +20.8% tIoU and +12.1% SODA gains with the full two-stage pipeline show improvements over stage-2-only training.In stage 2 only, the LLM is adapted with LoRA while the motion adapter is randomly initialized and trained from scratch with it.Source: §5.2 · p. 7
- The VQ-VAE-based model significantly underperforms continuous motion encoding, particularly on captioning metrics.The authors attribute the gap to the limited discrete vocabulary's difficulty capturing CompMo's complexity.Source: §5.2 · pp. 7–8
6. Conclusion
The paper introduces dense motion captioning and CompMo to support detailed temporal descriptions of 3D movement, while identifying limits in temporal composition and action ordering.
6. Conclusion
- The paper introduces dense motion captioning and CompMo, a dataset of long 3D motion sequences annotated with actions and timestamps.The task aims to generate detailed motion descriptions and interpret underlying actions from 3D representations.Source: §6 · p. 8
- The work discusses representations for interpreting the underlying actions.Source: §6 · p. 8
- CompMo focuses on temporal composition, and unconstrained action ordering can produce incoherent sequences because causal relationships are outside the work’s scope.The conclusion gives abrupt switches, such as swimming to basketball, as an example.Source: §6 · p. 8
APPENDIX
- The appendix provides further dataset-pipeline and model implementation details, additional results, and qualitative videos on the project page.It identifies dataset composition and model implementation as topics in its additional details section.Source: APPENDIX · p. 11, A. Additional Details · p. 11
A.1. CompMo Dataset
CompMo’s generation pipeline collects atomic actions, composes timestamped textual sequences with varied durations, then generates the corresponding motion sequences.
A.1. CompMo Dataset
- Atomic actions are generated from HumanML3D descriptions using MDM, filtered by TMR similarity at 0.5, and replaced with matching HumanML3D motions when filtered.The resulting motions also undergo quality filtering at the same TMR threshold.Source: §A.1 · p. 11
- Text composition assigns precise timestamps and samples each segment’s duration from its HumanML3D ground-truth duration with small random perturbations.The stated purpose is to introduce duration diversity while preserving temporal plausibility; α=0.3 and β=0.8.Source: §A.1 · p. 11
- Motion sequences are generated at test time with denoising and DiffCollage, using 0.5-second transitions between consecutive actions.For HumanML3D motions, the pipeline first applies 30 forward-diffusion steps before 100 denoising steps; scratch-generated actions use 100 denoising steps.Source: §A.1 · p. 11
A.2. DEMO
DEMO uses a two-stage training strategy and prompts that distinguish short-motion descriptions from dense, timestamped descriptions of complex sequences.
A.2. DEMO
- DEMO is trained first for motion-language alignment on HumanML3D, then for dense-caption instruction tuning on CompMo.The stages use one epoch at learning rate 5e−5 and two epochs at 2e−5, respectively.Source: §A.2 · p. 11
- The Stage 1 prompt asks for a natural-language description of a human motion sequence, while the dense-caption prompt specifies a duration and multiple actions.The Stage 2 example formats action descriptions with timestamps and text.Source: §A.2 · p. 11
- The Stage 2 prompt requests descriptions of complex motion in the format “mm:ss: ms – text.”The system prompt casts the model as a motion analyst and asks for descriptions in that timestamped format.Source: §A.2 · p. 11
B.1. Motion Captioning on HumanML3D
On HumanML3D single-motion captioning, DEMO achieves performance comparable to more sophisticated methods; the appendix describes the benchmark and its comparison table.
B.1. Motion Captioning on HumanML3D
- The HumanML3D benchmark evaluates natural-language summaries of short motion sequences.The authors describe this evaluation as a standard single-motion-captioning task, not the primary focus of the work.Source: §B.1 · p. 11
- DEMO achieves performance comparable to more sophisticated methods on the HumanML3D single-motion-captioning benchmark.The table compares DEMO with previous approaches and highlights the best and second-best results.Source: Table A.1 · p. 12, §B.1 · p. 12
B.2. Details on Ablation Study
The ablation study varies how atomic motions are joined and compares DEMO’s motion adapter with a VQ-VAE representation.
B.2. Details on Ablation Study
- “Smooth GT” concatenates atomic motions with Slerp interpolation, blending the preceding motion’s last 5 frames with the following motion’s first 5 over 10-frame transitions.The blending interval comprises 5 frames from each motion.Source: §B.2 · p. 12
- The representation ablation replaces DEMO’s motion adapter ΦW,γ with the MotionGPT VQ-VAE, trained on HumanML3D samples up to 10 seconds.Longer CompMo sequences are processed using a moving window.Source: §B.2 · p. 12