PaperlayerView exact revision on arXiv

Source-linked AI summary

KiRoSe: Using Kinematic Data for Efficient Segmentation of Robot Demonstrations

Shuangyu Xie, Kaiyuan Chen, Muhammad Zubair Irshad, Richard Cheng, Sergey Zakharov, Ken Goldberg

arXiv:2610.31801v1cs.RO

TL;DR

Large robot-demonstration datasets make understanding trajectory structure important. KiRoSe compresses kinematics into transition-supervised motion tokens for language-model interpretation; it improves segmentation results and also supports zero-shot transfer and progress tracking, including action reversal.

  • Problem

    Large robot-demonstration datasets span diverse embodiments and tasks, making their internal trajectory structure important for learning, reuse, retrieval, and analysis.

    Source: §1 · p. 2

  • Method

    KiRoSe learns a compact, transition-supervised vocabulary of kinematic tokens and adapts an open language model to interpret the resulting sequences.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    On 200 held-out YAM episodes, KiRoSe achieves boundary F1 within 1 second of 0.63 versus 0.42 for the strongest alternative kinematic tokenizer, alongside reported transfer and progress-tracking results.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The frozen representation also supports progress estimation on an unseen robot, including detecting when an action is being undone, without progress or reversal training.

    Source: §5 · p. 9

Abstract

from arXiv · show

Segmenting robot demonstration data into disjoint temporal and semantic subsegments is valuable for learning, cleaning, retrieval, and analysis. Segmentation is often performed manually in a tedious and error prone process. Multimodal Large Language Models (MLLMs) have the potential for automating this process, but directly ingesting raw multi camera video streams into MLLMs can be costly and can quickly exhaust context windows. In this paper, we introduce KiRoSe (Kinematics for Robot Segmentation), a novel framework that focuses on the companion stream of robot kinematic data. Inspired by the principle of classical entropy coding, such as Huffman coding, where frequently occurring symbols are represented more compactly, we propose Sparse Motion Tokens (SMTs), a discrete kinematic vocabulary designed to compactly represent recurring, temporally localized motion patterns, which we refer to as kinematic events. SMTs are learned by encoding kinematic trajectories with temporal segment supervision and quantizing the resulting features. We use increasingly available robot demonstration subtask annotations to supervise an open language model to interpret the resulting token sequences. Evaluation against human labeled segmentation shows that KiRoSe can substantially outperform baseline segmentation methods in F1 quality and transfers zero shot to unseen robot embodiments and egocentric human motion, while preserving the language models open vocabulary knowledge. Beyond segmentation, KiRoSe tracks task progress on an unseen robot and detects when an action is being reversed.(website: https://berkeleyautomation.github.io/kirose/)

1 INTRODUCTION

Robot demonstrations contain reusable motion stages, but manual segmentation is tedious and video-based MLLM processing can be costly and context-limited. KiRoSe learns compact kinematic tokens for language-model segmentation, with reported gains on held-out and zero-shot evaluation.

1 INTRODUCTION

  • Recurring stages such as reaching, grasping, transporting, and placing are kinematic events whose identification supports subtask reuse, skill learning, behavior retrieval, and demonstration-quality inspection.Pauses and corrective motions may also occur within demonstrations.Source: §1 · p. 2
  • Manual annotation is tedious and error-prone, while long synchronized multi-camera videos can make MLLM processing costly and exhaust context windows.Raw kinematic trajectories are continuous, high-frequency, and high-dimensional rather than discrete and semantically structured.Source: §1 · p. 2
  • KiRoSe encodes high-frequency kinematics into a compact discrete vocabulary supervised by subtask annotations, then trains a language model to interpret tokens without changing the tokenizer.The reader is supervised with segment boundaries and natural-language action descriptions.Source: §1 · p. 2
  • Boundary F1 within 1 second was 0.63 vs. 0.42 for the strongest alternative kinematic tokenizer on 200 held-out YAM episodes; zero-shot AgiBot transfer was 0.61 vs. 0.30 for GPT-6 Astra.KiRoSe also produces task-appropriate words absent from training annotations on unseen human tasks and tracks reversed actions.Source: §1 · p. 2
  • The contributions are Sparse Motion Tokens, a boundary-supervised low-rate tokenizer, and KiRoSe, which adapts an open language model for segmentation, transfer, open-vocabulary descriptions, and progress understanding.The paper reports substantial segmentation improvement over baseline and frontier VLMs.Source: §1 · p. 2

2 RELATED WORK

Prior work segments motion primitives and tokenizes motion for language-model tasks, while robot datasets add diversity in tasks, objects, and embodiments. KiRoSe uses transition-supervised kinematic tokens to support both temporal boundaries and language descriptions.

2 RELATED WORK

  • Modern robot datasets complicate trajectory segmentation through diverse tasks, objects, and embodiments; RoboSegNet fuses visual and kinematic features to predict subtask boundaries.Earlier trajectory-segmentation approaches discover motion primitives from dynamics, often assuming smooth, low-dimensional structure.Source: §2 · p. 3
  • KiRoSe pairs compact, transition-supervised kinematic tokens with natural-language action descriptions contextualized by the task instruction.Its pipeline encodes motion, quantizes features into SMTs, and has a language model produce action descriptions with start and end times.Source: §2 · p. 3, Figure 2 · p. 3
  • Human and robot motion-tokenization approaches commonly prioritize reconstruction fidelity or efficient generation, whereas KiRoSe uses transition supervision to capture precise kinematic events.The passage describes vector quantization and reconstruction-based approaches in human motion, and VQ-BeT, FAST, and OAT in robotics.Source: §2 · p. 3

3 KIROSE PIPELINE

KiRoSe first turns transition-supervised kinematic features into timestamped SMTs, then adapts a pretrained language model to produce timed action descriptions from the tokens and task instruction.

3 KIROSE PIPELINE

  • KiRoSe has two stages: transition-supervised kinematic encoding and temporal pooling/quantization into SMTs, followed by fixed-tokenizer adaptation of a pretrained language model.The adapted model predicts segment boundaries and action descriptions from SMT sequences and the task instruction.Source: §3 · p. 3

3.1 TRANSITION ANNOTATION FORMAT

  • Each transition annotation gives a timestamp and semantic labels immediately before and after it, marking a temporal boundary between manipulation subtasks.The segmentation task identifies boundaries and describes the action within each resulting segment.Source: §3.1 · pp. 3–4

3.2 TRANSITION-SUPERVISED SPARSE MOTION VOCABULARY

  • Residual temporal convolutions contextualize neighboring states, while a boundary head trained on soft targets near annotated transitions shapes the encoder features.The boundary head is discarded after training, leaving the encoder to produce features for compression.Source: §3.2 · p. 4
  • Uniform-grid pooled features are quantized to nearest codebook centroids, producing timestamped tokens whose positions are independent of annotated or predicted boundaries.The encoder and codebook remain fixed during language-model adaptation; the codebook uses K-means with K = 512.Source: §3.2 · p. 4

3.3 LANGUAGE MODEL INTEGRATION

  • The language-model input combines the task instruction, SMT sequence, and one-second time markers; its target is JSON segments with start times, end times, and action descriptions.The training loss is applied to response tokens.Source: §3.3 · p. 4
  • Table 1 reports episode duration and segment counts as mean ± SD, plus distinct action verbs and the percentage found in ABC-130K training annotations.Source: Table 1 · p. 5

4 EXPERIMENTS

Experiments test held-out trajectory segmentation, zero-shot transfer across robot and human embodiments, action semantics, and reversal-sensitive progress. KiRoSe improves segmentation comparisons and its frozen motion representation tracks progress direction on unseen AgiBot.

4.1 IMPLEMENTATION AND PROTOCOL

  • Training uses 5,478 YAM episodes across 140 tasks; evaluation includes 200 held-out YAM episodes, 2,653 AgiBot episodes, and 193 human-hand trajectories.A shared representation concatenates left/right linear velocity, angular velocity, and normalized opening rate; metrics separately assess boundary timing and joint temporal-semantic correctness.Source: §4.1 · pp. 5–6

4.2 TRAJECTORY SEGMENTATION

  • 0.63 Boundary F1 at ±1 s versus approximately 0.41–0.42 for alternative kinematic tokenizers; SMT also leads across temporal tolerances.The compared tokenizers use the same adapted Qwen3.5-9B reader.Source: Table 2 · p. 6, §4.2 · p. 6
  • KiRoSe also achieves higher lexical and embedding Grounded F1 than alternative kinematic tokenizers, supporting segment descriptions consistent with reference actions.These metrics assess semantic agreement alongside temporal segmentation.Source: §4.2 · p. 6, §4.1 · p. 6

4.3 CROSS EMBODIMENT GENERALIZATION

  • 0.40 Boundary F1 at ±0.5 s on zero-shot AgiBot versus 0.48 on held-out YAM; Latent + MLP reaches 0.36 and video Qwen3.5-9B reaches 0.19 on AgiBot.The transfer evaluation includes shifts in embodiment, teleoperation interface, motion statistics, segment duration, and annotation granularity.Source: Figure 4 · p. 7, §4.3 · p. 7, Figure 3 · p. 7
  • 0.59 action-level Grounded F1 versus 0.34 for Astra on AgiBot, where action-level matching requires the verb class to match.Full-description Grounded F1 additionally requires matching wording, including object names.Source: §4.3 · p. 7
  • On 12 Ego-Exo4D tasks absent from YAM training, KiRoSe supplies words appearing in annotations but absent from training and prompt in 6 tasks.The reported words are consistent with what a language model can infer from the task name, suggesting preserved open-vocabulary knowledge.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 CROSS-EMBODIMENT PROGRESS ESTIMATION

  • On unseen AgiBot, the frozen SMT representation yields ρ = +0.51 forward and ρ = −0.68 reversed, with progress falling on 86% of reversed subtasks.ReWiND remains positively correlated after reversal, while Robometer-4B approaches zero.Source: Table 4 · p. 9, §4.4 · p. 9

5 CONCLUSION

KiRoSe uses transition-supervised Sparse Motion Tokens to support manipulation-action segmentation and interpretation. Across selected examples, it matches or exceeds the video baseline, while its encoder-tokenizer pipeline uses dilated temporal features and quantization.

5 CONCLUSION

  • KiRoSe identifies when manipulation actions change and what action occurs within each segment, transferring transition structure and action identity zero-shot across embodiments.The conclusion also reports preserved open-vocabulary knowledge and within-segment progress estimation, including detection of actions being undone on an unseen robot.Source: §5 · p. 9

A QUALITATIVE SEGMENTATION EXAMPLES

  • Under a fixed episode-selection rule, KiRoSe matches or exceeds the video baseline on all 15 example trajectories across three embodiments, including two unseen during training.On all 114 human trajectories with video, KiRoSe reaches 0.375 Boundary F1 at ±1 s versus 0.255 for the video baseline.Source: A QUALITATIVE SEGMENTATION EXAMPLES · p. 13, Figure 9 · p. 16

B KINEMATIC ENCODER, TRANSITION LOSS, AND TOKENIZER

B.1 ARCHITECTURE

The architecture encodes robot trajectories into contextual kinematic features, trains a boundary head with transition supervision, then quantizes pooled features into a fixed codebook. Its dilated convolutions provide long temporal context, while focal weighting addresses rare boundaries.

B.1 ARCHITECTURE

  • Seven dilated residual kernel-5 blocks map 30 Hz, 14-D episodes to 256-D features and reach a 17 s context without attention’s length limit.Per-episode z-scoring removes differences in overall speed and gripper range between embodiments, but makes the encoder an offline model.Source: §B.1 · pp. 17–18

B.2 TRANSITION LOSS

  • Boundary supervision uses nearest-Gaussian soft targets and focal weighting to accommodate timestamp imprecision and down-weight confident interior negatives.Only about 4% of YAM grid points are nearest to a boundary; after training, the head is discarded and the retained encoder has 2.76M parameters.Source: §B.2 · pp. 18–19, Figure 11 · p. 19

B.3 CODEBOOK

  • The frozen encoder’s features are pooled into two vectors per second and assigned to the nearest of 512 centroids from a fixed YAM-trained codebook.The codebook is fitted using 600k sampled vectors from YAM training episodes; held-out episodes and other embodiments are excluded.Source: §B.3 · p. 19

C COMPARISON WITH MOTION TOKENIZERS

KiRoSe’s Sparse Motion Tokens are compared with three motion-tokenizer interfaces using a shared reader and held-out YAM episodes. SMT improves segmentation metrics over OAT while using fewer reader input tokens than FAST and MotionGPT.

C COMPARISON WITH MOTION TOKENIZERS

  • All readers share Qwen3.5-9B and the adaptation recipe, but the comparison tests complete tokenizer interfaces rather than isolating tokenizer objectives.Input representations and reader training populations differ.Source: C COMPARISON WITH MOTION TOKENIZERS · pp. 19–20
  • FAST uses a reduced scale of 0.5 because its default scale would make the longest training episode exceed the reader’s context window.At the default scale of 10, that episode requires 102,125 tokens, beyond the 32,768-token context.Source: C COMPARISON WITH MOTION TOKENIZERS · p. 20
  • OAT uses eight codes per 4 s to match SMT’s rate, while MotionGPT uses eight codes per second.OAT uses eight 4-dimensional registers per 4 s; MotionGPT quantizes 32 vectors per 4 s.Source: C COMPARISON WITH MOTION TOKENIZERS · p. 20
  • 0.22 Boundary F1 at ±1 s and 0.26 at ±0.5 s separate SMT from OAT; SMT uses 41% and 35% fewer reader input tokens than FAST and MotionGPT.The comparison covers 200 YAM episodes from held-out tasks; the reference has 24.3 segments per episode.Source: Table 7 · p. 20, C COMPARISON WITH MOTION TOKENIZERS · p. 20

D HUMAN HANDS AS A ROBOT END EFFECTOR

Human hand motion is converted into the shared 14-dimensional representation used for robot data, allowing KiRoSe’s existing encoder, tokenizer, and reader to run unchanged. On long Ego-Exo4D trajectories, the converted motion supports zero-shot keystep segmentation, with interpolation remaining an evaluation limitation.

D HUMAN HANDS AS A ROBOT END EFFECTOR

  • Both robot and human inputs map to the same 14-D channels, so the encoder, tokenizer, and reader run unchanged on human data.The human path uses hand keypoints and a fitted palm frame; the robot path uses forward kinematics and gripper normalization.Source: D HUMAN HANDS AS A ROBOT END EFFECTOR · p. 20, Figure 12 · p. 21, Table 8 · p. 21
  • 0.375 Boundary F1 at ±1 s on 114 long trajectories with video exceeds 0.255 for the zero-shot Qwen3.5-9B video model; boundary evidence has AUC 0.68 versus 0.72 for YAM.Removing motion tokens lowers KiRoSe to 0.296, and substituting another trajectory’s tokens lowers it to 0.235.Source: Figure 13 · p. 22, D HUMAN HANDS AS A ROBOT END EFFECTOR · p. 23
  • Human channels derive filtered wrist velocity, fitted palm orientation, and normalized thumb–index aperture from Ego-Exo4D keypoints in a body-centric frame.The thumb–index distance serves as the gripper aperture, and the head-camera direction defines the body-centric heading.Source: D HUMAN HANDS AS A ROBOT END EFFECTOR · pp. 21–22
  • The evaluation uses 193 trajectories from 12 tasks, each at least 30 s long with five keystep segments, bridging tracking gaps up to 2 s.About 46% of frames are interpolated, though 98% of annotated boundaries have real tracking within 0.5 s.Source: D HUMAN HANDS AS A ROBOT END EFFECTOR · p. 23

E PROGRESS ESTIMATION

KiRoSe estimates task progress from frozen kinematic features and a signed-rate probe, then tests whether progress reverses when subtasks are played backward. On 2,586 AgiBot subtasks, it is the only evaluated model whose progress rises during execution and falls under reversal on every split.

E PROGRESS ESTIMATION

  • The frozen encoder processes a causal 4-second window of 120 kinematic samples every 0.25 seconds, standardized with fixed YAM training-set channel statistics.The probe also receives an 8-dimensional state vector containing displacement accumulated since the subtask start.Source: E PROGRESS ESTIMATION · p. 23
  • The progress head has 68,097 parameters, uses no temporal model beyond frozen window features, is trained on 4,000 YAM training subtasks, and selected on 600 YAM validation subtasks.Subtasks shorter than 3.2 s and control segments are excluded.Source: E PROGRESS ESTIMATION · p. 23
  • Each evaluation subtask is reversed in place while the rest of its episode remains unchanged, with kinematic velocities negated and video frames played backward.The evaluation uses 2,586 non-control subtasks of at least 3.2 seconds from 357 AgiBot episodes.Source: E PROGRESS ESTIMATION · p. 23
  • Progress is scored by Spearman correlation with playback index, and the study reports mean correlation and the fraction of reversed subtasks with negative correlation.Confidence intervals use an episode-clustered bootstrap with 2,000 resamples.Source: E PROGRESS ESTIMATION · p. 24
  • KiRoSe’s progress falls on 86% of reversed subtasks, versus 1% and 4% for the two ReWiND models and 46% for Robometer-4B.Pooled over 2,586 AgiBot subtasks; KiRoSe’s mean Spearman ρ is +0.51 forward and −0.68 reversed.Source: E PROGRESS ESTIMATION · p. 24, Table 9 · p. 24

F LANGUAGE MODEL TRAINING

The language-model reader is adapted to interpret temporally grouped motion tokens alongside a task instruction and produce timestamped subtask boundaries and descriptions. Training targets are reference segmentations, and failed or unparseable outputs count as predictions with no segments.

F LANGUAGE MODEL TRAINING

  • The reader starts from Qwen3.5-9B with 512 added motion-token entries, while other model weights are adapted only through LoRA.Updates to the embedding and output matrices are restricted to the new rows.Source: F LANGUAGE MODEL TRAINING · p. 24
  • The reader receives motion tokens grouped by second with textual time markers, allowing it to read absolute time directly rather than count tokens.The prompt uses the model’s chat template with thinking disabled.Source: F LANGUAGE MODEL TRAINING · p. 24
  • The model is prompted to return a JSON array of start and end times and descriptions covering the episode.The example requests segmentation from 0 to 190 seconds.Source: F LANGUAGE MODEL TRAINING · p. 24
  • The target is the reference segmentation serialized as JSON, and loss is applied only to target tokens.Prompts are not truncated; on 200 held-out YAM episodes, reader input averages 1,368 tokens and reaches a maximum of 2,836 tokens.Source: F LANGUAGE MODEL TRAINING · p. 24
  • Empty or unparseable model responses are scored as predictions with no segments rather than excluded.Source: F LANGUAGE MODEL TRAINING · p. 24

G REPRODUCIBILITY

The reproducibility appendix specifies evaluation populations, controls, compute, and release materials. It also notes that the final 14-D reader was trained with one seed, so its training-seed variability was not measured.

G REPRODUCIBILITY

  • KiRoSe’s transfer comparisons use 200 YAM, 3,852 AgiBot, and 193 human trajectories, while Astra uses smaller subsets of 50, 20, and 30, respectively.On the matched 20-episode AgiBot subset, KiRoSe’s Boundary F1 at ±0.5 seconds is 0.612 versus 0.301 for Astra; KiRoSe’s 0.40 transfer-figure score uses pooled AgiBot data.Source: G REPRODUCIBILITY · p. 25
  • On YAM, Boundary F1 at ±1 second is 0.630 with correct motion, 0.349 with instruction only, and 0.342 with wrong motion; AgiBot scores are 0.614, 0.321, and 0.310.The instruction-only control retains the instruction and time markers, while the wrong-motion control substitutes time-aligned tokens from another episode of the same task.Source: G REPRODUCIBILITY · p. 25
  • The encoder trains in about 54 minutes on one GPU, the reader in about 2.2 hours on three NVIDIA H200 GPUs, and inference processes 200 YAM trajectories in 32 seconds on one H200 after loading.The progress head trains in under 10 seconds from cached features.Source: G REPRODUCIBILITY · p. 25
  • The final 14-D reader was trained with one seed, so its training-seed variability is unmeasured; FAST and OAT ranges span three seeds.Source: G REPRODUCIBILITY · p. 25
  • Evaluation uses greedy one-to-one boundary matching within the stated tolerance and interval matching at IoU ≥0.5 with lexical or embedding description agreement; failed parses remain included.Lexical agreement requires token F1 ≥0.5, and embedding agreement requires MiniLM cosine ≥0.6.Source: G REPRODUCIBILITY · pp. 25–26
  • The release plan includes conversion code, encoder and codebook checkpoints, reader adapters and motion-token embeddings, evaluation splits and scorer, and baseline prompts.AgiBot-derived data are subject to CC BY-NC-SA 4.0 terms; Ego-Exo4D conversion scripts are released instead of converted human data because its license bars redistribution.Source: G REPRODUCIBILITY · p. 26

H DATA AND LICENSES

The paper uses existing YAM, AgiBot World, and Ego-Exo4D datasets under their respective terms, with human-transfer evaluation based on consented research data. Its broader-impact discussion highlights privacy, misuse, safety, and training-domain boundaries.

H DATA AND LICENSES

  • YAM provides demonstrations and subtask annotations, while AgiBot World and Ego-Exo4D are used for evaluation only.Ego-Exo4D kinematics are constructed from 3D hand keypoints and head-camera poses; its keystep annotations are evaluation targets, not reader inputs.Source: H DATA AND LICENSES · p. 26

I BROADER IMPACT

  • KiRoSe can provide initial segment boundaries and action descriptions for human review, using an open 9B-parameter reader that can run locally on one GPU without an API fee.Source: I BROADER IMPACT · p. 26
  • Kinematics-only local inference can avoid sending recordings to a third-party service, but motion patterns and task instructions are not an anonymity guarantee.Source: I BROADER IMPACT · p. 26
  • Transfer to human hands could be used to monitor workers without their knowledge or consent, so human-motion deployments should obtain informed consent and follow applicable protections.The human evaluation uses a consented research dataset, and no models trained on human data are released.Source: I BROADER IMPACT · p. 26
  • Automatic label errors can propagate into robot behavior; accuracy is lower on human hands, and long-horizon mobile manipulation has not been evaluated.The authors recommend spot-checking labels, especially outside the training distribution, and retaining human review for safety-critical failure and recovery annotations.Source: I BROADER IMPACT · p. 26
  • Training uses laboratory demonstrations from one bimanual robot and English annotations in one style, which may under-represent unseen activities, manipulation styles, or languages.The authors recommend validation on a labeled sample from each new domain before relying on predictions.Source: I BROADER IMPACT · pp. 26–27

Related papers

Loading 2610.31801v1…