Source-linked AI summary
Skeleton-in-Context: Unified Skeleton Sequence Modeling with In-Context Learning
Xinshun Wang, Zhongbin Fang, Xia Li, Xiangtai Li, Mengyuan Liu
TL;DR
Skeleton sequence modeling remains unexplored in in-context learning, and existing in-context models are difficult to adapt because skeleton pose similarity makes task perception challenging. SiC uses prompts and a task-unified prompt to handle multiple skeleton tasks after one training process. It achieves state-of-the-art multi-task benchmark results and generalizes to unseen tasks and datasets.
Problem
Method
SiC uses task-guided and task-unified prompts to perform multiple skeleton tasks from context after a single training process.
Source: §1 · p. 2Results
SiC achieves state-of-the-art results on the in-context multi-task benchmark, outperforming all multi-task models and showing impressive performance against task-specific models.
Source: §5.2 · p. 7Takeaways & Limitations
The experiments show that SiC generalizes across datasets and to unseen tasks such as motion in-between.
Source: §1 · p. 2, §5.3 · p. 7, §6 · p. 8
Abstract
from arXiv · showhide
In-context learning provides a new perspective for multi-task modeling for vision and NLP. Under this setting, the model can perceive tasks from prompts and accomplish them without any extra task-specific head predictions or model fine-tuning. However, Skeleton sequence modeling via in-context learning remains unexplored. Directly applying existing in-context models from other areas onto skeleton sequences fails due to the inter-frame and cross-task pose similarity that makes it outstandingly hard to perceive the task correctly from a subtle context. To address this challenge, we propose Skeleton-in-Context (SiC), an effective framework for in-context skeleton sequence modeling. Our SiC is able to handle multiple skeleton-based tasks simultaneously after a single training process and accomplish each task from context according to the given prompt. It can further generalize to new, unseen tasks according to customized prompts. To facilitate context perception, we additionally propose a task-unified prompt, which adaptively learns tasks of different natures, such as partial joint-level generation, sequence-level prediction, or 2D-to-3D motion prediction. We conduct extensive experiments to evaluate the effectiveness of our SiC on multiple tasks, including motion prediction, pose estimation, joint completion, and future pose estimation. We also evaluate its generalization capability on unseen tasks such as motion-in-between. These experiments show that our model achieves state-of-the-art multi-task performance and even outperforms single-task methods on certain tasks.
1. Introduction
Task-specific skeleton models rely on specialized designs or training stages, limiting their flexibility for unseen tasks. Skeleton-in-Context (SiC) uses prompts to perform multiple skeleton tasks after one training process and establishes a benchmark for evaluating this approach.
1. Introduction
- Task-specific heads and staged training make existing skeleton models difficult to generalize to unseen tasks and data.Prior approaches include MotionBERT's pretraining and task-specific fine-tuning, UniHCP's task heads, and UPS's task-wise fine-tuning.Source: §1 · p. 1
- In-context models use input-output prompts to perform tasks end-to-end without task-specific heads or fine-tuning, but adapting them to skeletons is challenging.Existing models focus on static images or point clouds and do not capture skeleton sequences' spatial-temporal dependencies.Source: §1 · p. 2
- SiC performs multiple skeleton-based tasks from context after a single training process, without task-specific designs.A task-guided prompt provides a task template, while static or dynamic task-unified prompts use shared prior knowledge or adaptively learn task context.Source: §1 · p. 2
- The paper establishes a benchmark spanning motion prediction, pose estimation, joint completion, and future pose estimation.The framework is also evaluated on unseen tasks such as motion in-between.Source: §1 · p. 2
2. Related Work
Prior skeleton analysis methods generally use task-specific architectures, while in-context learning has been explored in NLP, image, and point-cloud settings. SiC applies this paradigm to dynamic skeleton sequences and trains a unified model once.
Skeleton-Based Motion Analysis
- Skeleton-based motion prediction, synthesis, and pose-estimation methods commonly use customized architectures, limiting their range of applications.The section describes graph-convolution approaches for motion prediction, convolutional or adversarial methods for synthesis, and spatial-temporal convolution or transformers for pose estimation.Source: §2 · p. 2
In-Context Learning
3. In-Context Skeleton Sequence Modeling
The formulation treats a task example as a prompt that guides the operation on a query skeleton sequence. The section defines a four-task benchmark and describes the sequence data and prompt-based model framework.
3.1. Formulation
- A task-guided input-target skeleton pair specifies the operation, which the model applies by analogy to a query sequence.The prompt's task index identifies the operation, and selecting a task-specific prompt determines which task is performed.Source: §3.1 · p. 3
3.2. Task Definitions
- The benchmark uses H3.6M, AMASS, and 3DPW for motion prediction, pose estimation, joint completion, and future pose estimation.It contains 176K skeleton-sequence samples.Source: §3.2 · p. 3
- MIM-style training reconstructs masked frames, but inference sequences shrink as reference frames disappear during interpolation [5, 16, 59].The caption attributes this behavior to training that teaches frame interpolation only.Source: Figure 2 · p. 3
- Motion prediction forecasts future motion from historical sequences, while pose estimation lifts noisy 2D poses to 3D.Pose-estimation inputs are expanded to three dimensions by filling the third coordinate with zero to match target shape.Source: §3.2 · p. 3
- Future pose estimation predicts future 3D motion from historical 2D poses, a task the authors say had not previously been proposed.The authors motivate it by the greater availability of 2D images than 3D data.Source: §3.2 · p. 3
- Joint completion recovers missing skeleton joints, evaluated with 40% or 60% of joints missing in each frame.Missing coordinates are set to zero.Source: §3.2 · p. 3
- The skeleton bank combines training sets and supplies input-target pairs for prompts and query inputs.The model encodes and processes the prompt and query, combines the query with the task-unified prompt, and uses the second half of its output as the prediction.Source: Figure 3 · p. 4
4. Skeleton-in-Context
SiC addresses the difficulty of inferring tasks from similar skeleton poses with task-guided and task-unified prompts. It handles multiple tasks in one end-to-end training process and performs them from prompts without further parameter updates.
4.1. Skeleton Sequence Prompt
- Task-Guided Prompt (TGP) uses an example skeleton pair for a task, allowing SiC to perform seen or unseen tasks and transfer across dataset styles.Source: §4.1 · p. 4
- Random masking on similar skeleton frames can lead to overfitting on interpolation; TUP adapts across tasks in one end-to-end framework without task-specific heads or fine-tuning.Source: §4.1 · p. 4, §4 · p. 4
- TUP models distribution differences among tasks and datasets so the model focuses on representations shared across tasks.Source: §4.1 · p. 5
4.2. Model Architecture
- SiC draws prompts from a skeleton bank of public datasets, embeds and concatenates prompt and query sequences, and uses U as target-sequence prior and task/data adapter.Source: §4.2 · p. 5
- Its two-stream transformer alternates spatial and temporal attention blocks, combines their outputs with adaptively determined weights, and uses the second half of its output as the prediction.Source: §4.2 · p. 5
4.3. Training and Inference
- SiC is trained across tasks with a 3D joint-level reconstruction loss, then performs tasks from task-dependent prompts without parameter updates.Source: §4.3 · pp. 5–6
- Figure 4 shows SiC generating more accurate poses than MotionBERT on four tasks, with red arrows marking the compared spots.The figure compares SiC with MotionBERT [79].Source: Figure 4 · p. 6
5. Experiments
SiC is evaluated across skeleton tasks, including comparisons with multi-task and task-specific models. Experiments assess its benchmark performance, dataset and task generalization, prompt choices, architecture, and learned initial pose.
5. Experiments
- All tasks are unified in one end-to-end training process without task-specific designs.The implementation uses five layers, eight attention heads, and a 256-dimensional hidden feature; training takes about six hours for 120 epochs on four RTX 4090 GPUs.Source: §5 · p. 6
5.1. Baseline Methods
- The benchmark compares reproduced task-specific methods, remodeled multi-task methods, and in-context baselines, treating each frame as a token for Point-In-Context and PointMAE.The authors also reimplement task-specific methods for qualitative comparison.Source: §5.1 · pp. 6–7
5.2. Compared with SoTA Methods
- SiC outperforms all multi-task models, achieving state-of-the-art results on the in-context multi-task benchmark.The reported comparison also describes its performance against task-specific models as impressive.Source: §5.2 · p. 7
- SiC generates more accurate poses than multi-task MotionBERT when following the provided task-guided prompt.MotionBERT is restructured as an end-to-end multi-task model for the comparison.Source: §5.2 · p. 7
5.3. Generalization Capability.
- SiC generalizes across dataset shifts in motion prediction, pose estimation, and joint completion, while Dynamic-SiC generalizes better than Static-SiC.A customized task-guided prompt also lets the trained model complete missing skeletons for the unseen motion-in-between task.Source: §5.3 · pp. 7–8, Figure 5 · p. 7
5.4. Ablation Study
- Skeleton prompts address overfitting, and using both TGP and TUP gives the best results; random prompt selection also performs best among the tested prompt choices.The authors attribute the prompt-selection result to skeleton-task data similarity and report best architecture performance at feature dimension 256 and depth 5.Source: §5.4 · p. 8, Table 3 · p. 8, Table 4 · p. 8
5.5. In-Context Initial Pose
- Dynamic-TUP’s learned skeleton resembles the training-set mean pose, which the authors interpret as prior knowledge spanning training ground truths.They report that this prior helps learning across tasks and that learnable modules help generalize to other datasets or new tasks.Source: §5.5 · p. 8, Figure 6 · p. 8
6. Conclusion
The conclusion presents SiC as a single-training framework for multiple skeleton tasks. It highlights skeleton prompts and generalization across datasets and to new tasks.
6. Conclusion
- SiC processes multiple skeleton-based tasks simultaneously after one training process.The conclusion identifies TGP and TUP as skeleton prompts addressing overfitting under training frameworks commonly used by earlier 2D and 3D in-context models.Source: §6 · p. 8
A.1. Implementation Details
The implementation unifies all tasks in a one-off, end-to-end training process without task-specific designs.
A.1. Implementation Details
- The model uses five layers, eight attention heads, a 256-dimensional hidden feature, and sequence length 16 for prompt and query inputs and targets.Training takes about six hours for 120 epochs on four NVIDIA GeForce RTX 4090 GPUs.Source: §A.1 · p. 12
A.2. Datasets and Metrics
The experiments use H3.6M for pose estimation, future pose estimation, and 2D-to-3D pose estimation; AMASS for motion prediction; and 3DPW for joint completion.
A.2. Datasets and Metrics
- H3.6M supplies pose-estimation tasks, with 2D skeletons as input and 3D skeletons as targets; future pose estimation predicts 300ms of future 3D skeletons from 300ms of 2D history.The study notes that using the same dataset for two tasks makes task perception harder and uses prompts to guide in-context learning.Source: §A.2 · p. 12
- AMASS is used for motion prediction, with 400ms of 3D motion history and 400ms of future motion as the prediction horizon.The sequences are padded to 16 frames after representing each 400ms segment as 10 frames at 25 fps.Source: §A.2 · p. 12
- 3DPW is used for joint completion, with 40% or 60% of joints randomly masked for reconstruction.The dataset contains more than 51,000 frames with 3D annotations.Source: §A.2 · p. 12
- Mean Per Joint Position Error is reported for motion prediction, joint completion, and future pose estimation; pose estimation also reports Normalized Mean Per Joint Position Error.Source: §A.2 · p. 12
B. Multi-Task Synergistic Training
The section examines negative transfer in multi-task training and evaluates whether task-guided and task-unified prompts support collaborative training. Without prompt guidance, joint training across four tasks decreases performance relative to separate training.
B. Multi-Task Synergistic Training
- The section evaluates negative transfer and the proposed prompts as means to support collaborative training across human-motion tasks.Source: B. Multi-Task Synergistic Training · p. 12
- Tables 7 and 8 report MPJPE comparisons for pose estimation and future pose estimation, respectively.Source: Table 7 · p. 13, Table 8 · p. 13
B.1. Negative Transfer in Multi-Task
- Without TUP and TGP, joint training across four tasks decreases performance, a result attributed to negative transfer from differences between tasks.The comparison is between separately trained and jointly trained backbone models.Source: §B.1 · p. 13
B.2. Multi-Task Synergistic Training
SiC uses task-guided and task-unified prompts to support unified training across multiple tasks. With both prompts, it achieves strong results on four tasks simultaneously.
B.2. Multi-Task Synergistic Training
- TGP supplies task-specific context for task execution, while TUP provides prior knowledge for processing queries across datasets and tasks with different patterns.TGP is described as task-agnostic and able to encode unified human-motion representations across tasks.Source: §B.2 · p. 13
- With TGP and TUP, SiC achieves impressive results on four tasks simultaneously.Source: §B.2 · p. 13
C.1. Detailed Results
On H3.6M, SiC outperforms other multi-task models across actions in pose estimation and future pose estimation. Additional visualizations compare SiC with multi-task baselines across four tasks and report that those models produce unsatisfactory results when they cannot balance task resources.
C.1. Detailed Results
- SiC outperforms other multi-task models on every H3.6M action in pose estimation and future pose estimation.Dynamic-SiC performs better than Static-SiC in most actions.Source: §C.1 · p. 13
C.2. More Visualization
- Across four tasks, the visualizations compare SiC with multi-task baselines, whose inability to balance task resources is reported to yield unsatisfactory results.Source: §C.2 · p. 13, Figure 7 · p. 14, Figure 8 · p. 15