YOCO: You Only Calibrate Once! Fast Mocap Calibration for Dexterous Teleoperation

Yu Zhang; Yunqi Li; Yushi Du; Yi Ma; Yanchao Yang

Source-linked AI summary

YOCO: You Only Calibrate Once! Fast Mocap Calibration for Dexterous Teleoperation

Yu Zhang, Yunqi Li, Yushi Du, Yi Ma, Yanchao Yang

arXiv:2610.11657v1Oct 8, 2026cs.RO

TL;DR

Low-cost hand trackers can have user-, fit-, and session-dependent biases, and small pose offsets can disrupt dexterous contact. YOCO generates session-conditioned LoRA updates for a frozen MANO estimator from a few paired poses; it improves calibration outcomes and shows promising teleoperation gains in the reported evaluations.

Problem
Low-cost hand trackers exhibit biases that vary across users, glove fit, and sessions, while small fingertip or joint offsets can alter contact patterns and pinch geometry.
Source: §1 · pp. 1–2
Method
YOCO conditions a HyperNet on a few paired raw and target poses to generate session-specific LoRA updates for a frozen MANO estimator without per-session gradient optimization.
Source: §1 · p. 2, §3.4 · p. 4
Results
Across augmented InterHand, real-glove, and teleoperation evaluations, YOCO improves pose-calibration outcomes and shows promising teleoperation gains; PA-MPJPE falls 37% relative to no calibration.
Source: §5.2 · pp. 6, 8, §6 · p. 9
Takeaways & Limitations
YOCO provides a feed-forward calibration approach at the derived 3D hand-keypoint interface that does not require device-specific raw signals.
Source: §1 · p. 2
YOCO leaves MANO shape estimation to a separate stage, and high-frequency drift on less stable hardware would require re-calibration or a temporal HyperNet extension.
Source: §6 · p. 9

Abstract

from arXiv · show

Dexterous teleoperation requires reliable human-hand state estimations. However, common low-cost motion-capture gloves and markerless trackers often exhibit biases that vary across users, glove fit, and recording sessions, degrading retargeting and demonstration quality. We present YOCO, a fast few-shot, fine-tuning-free calibration framework that corrects biased hand-pose streams from a small set of paired raw and target poses. Instead of optimizing a separate model for every operator or session, YOCO conditions a calibration HyperNet on the paired examples and predicts LoRA-style updates for a frozen MANO hand-estimation module, turning per-user calibration into a lightweight feed-forward adaptation step while preserving the geometric prior of MANO and the efficiency of a compact estimator. We train YOCO with synthetic drift augmentations on InterHand2.6M and evaluate on augmented InterHand sequences, offline real glove data, and dexterous teleoperation tasks. Across these settings, YOCO improves calibration efficiency, hand-state estimation quality and teleoperation performance compared with uncalibrated input and standard calibration baselines.

1 Introduction

Low-cost hand trackers can produce user-, fit-, and session-dependent pose biases that impair dexterous manipulation. YOCO calibrates derived 3D hand keypoints from a few paired poses by generating session-conditioned LoRA updates for a frozen MANO estimator.

1 Introduction

Small fingertip or joint offsets can alter contact patterns, pinch geometry, and retargeted robot motion in dexterous manipulation.High-end motion capture can provide accurate kinematics but is expensive and difficult to deploy at scale.Source: §1 · pp. 1–2
YOCO uses paired raw and target poses to generate session-conditioned LoRA updates for a frozen MANO estimator in one pass, without per-session gradient optimization.The formulation works at the derived 3D hand-keypoint interface without requiring device-specific raw signals; real-device evaluation is limited to one tensile glove.Source: §1 · p. 2
The paper evaluates accuracy and adaptation cost on augmented InterHand2.6M and paired real-glove/AVP data, then tests hand poses in physical teleoperation.Its contributions include controls designed to test whether gains come from pair-conditioned weight generation beyond a shared prior or set conditioning alone.Source: §1 · p. 2

2 Related Work

YOCO differs from prior calibration work by acting on estimated 3D hand keypoints rather than device-specific sensor streams. It targets upstream hand-observation errors while complementing existing tracking and teleoperation systems.

Mocap Devices

YOCO calibrates existing glove-based streams rather than introducing a new tracker, complementing accessible markerless and RGB-based pipelines.The related-work discussion notes that markerless methods can struggle with occlusion, hand-object interaction, cross-user morphology, and temporal stability [8] [9] [10] [11] [13].Source: §2 · p. 2

Prior Hand-Mocap Calibration

Prior glove and wearable calibration methods work within particular sensing pipelines, while YOCO corrects their downstream 3D-keypoint representation from paired poses.Prior approaches include glove-to-motion mapping, sensor-to-pose personalization, joint IMU/shape/kinematics optimization [2], and time-varying IMU drift estimation [3].Source: §2 · p. 2

Dexterous Hand Retargeting and Teleoperation

YOCO targets the upstream hand-observation bottleneck that many retargeting and teleoperation systems assume is already accurate.The cited landscape includes vision-based systems [15] [16], VR platforms [17] [18], and learning-based retargeting [19] [20] [21].Source: §2 · p. 2

HyperNetworks and Parameter-Efficient Adaptation

YOCO combines hypernetworks that predict limited-input updates [34] with LoRA-style low-rank adaptation [32] [33] to generate estimator updates from paired calibration poses.The related work presents this as an application of hypernetwork and parameter-efficient adaptation ideas to dexterous teleoperation.Source: §2 · pp. 2–3

3 Method

YOCO uses a small paired calibration set to condition a HyperNet that generates LoRA updates for a frozen MANO estimator. One conditioning pass adapts the estimator for a session, while the calibrated model processes subsequent poses.

3.1 Problem Formulation

The calibration set pairs noisy 3D hand-keypoint observations with target poses and is assumed to capture approximately stationary bias for the current subject, glove fit, or session.A frozen MANO estimator maps keypoints to pose parameters; the HyperNet generates LoRA updates, and training minimizes joint-position error on unseen noisy poses.Source: §3.1 · p. 3

3.2 Model Overview

At deployment, the HyperNet conditions once on paired poses and injects the resulting LoRA updates into the frozen estimator; subsequent queries run at mocap frame rate.The calibrated MANO pose parameters are reconstructed into calibrated 3D joints.Source: §3.2 · p. 4

3.3 Target Network: MANO Estimator Architecture

The MANO estimator uses finger-wise CNN encoders, transformer fusion, finger-wise MLPs, and a unified MANO pose decoder.The finger-wise design limits influence from unrelated fingers, while fusion captures global hand-pose correlation; the decoder is lightweight for real-time operation and HyperNet updates.Source: §3.3 · p. 4

3.4 Hypernetwork: Calibration HyperNet Architecture

The HyperNet encodes paired poses, aggregates them with a transformer calibration mapper and per-matrix CLS tokens, then decodes those tokens into LoRA matrices.The mapper uses RoPE positional embedding, and the LoRA parameter decoder is DiT-style.Source: §3.4 · p. 4

4 Model Training

YOCO training first establishes a MANO pose prior on clean data, then freezes that estimator and trains a HyperNet to generate corrections for simulated mocap bias. Training uses synthetic drift augmentations and query poses distinct from the calibration examples.

4 Model Training

Training proceeds in two stages: pretrain the MANO estimator on clean InterHand2.6M data, then freeze it while training the HyperNet to correct simulated mocap bias.The clean-data pretraining uses mean squared error on predicted versus ground-truth MANO pose parameters.Source: §4 · p. 5, §4.1 · p. 5

4.2 Hypernetwork Training

Synthetic InterHand2.6M augmentations simulate varied glove-based mocap drift patterns to expose the HyperNet to realistic bias variation.The augmentations are designed to cover or overlap with distribution shifts observed in real mocap data.Source: §4.2 · p. 5
Each training iteration samples paired calibration examples and disjoint query frames; only HyperNet parameters are updated to minimize query-pose error.The generated LoRA matrices are injected into the frozen target network, and gradients flow only through the HyperNet parameters.Source: §4.2 · p. 5

5 Experiment

The experiments test calibration on synthetic and paired real-glove streams, then measure performance in four teleoperation tasks. YOCO improves reported pose metrics and conditioning time, and achieves higher overall task success than the listed baselines.

5.1 Experiment Setup

The MANO estimator is pretrained on clean InterHand2.6M poses, then the HyperNet is trained on synthetically drift-augmented sequences; evaluation also includes paired AVP/glove data and four physical tasks.Synthetic results aggregate over 20 augmented validation sets with distinct drift patterns.Source: §5.1 · pp. 5–6

5.1.1 Teleoperation Protocol

Teleoperation tests use four AVP-recorded target poses, two operators, and 10 trials per method–task pair; evaluation reports task success and completion time.Offline calibration metrics include MPJPE, PA-MPJPE, Tip-PA, AUCJ, pinch error, and adaptation time; MANO shape β is not calibrated.Source: §5.1 · p. 6, §5.1.1 · p. 6

5.2 Experiment Results

37% relative reduction in PA-MPJPE, from 16.55 mm without calibration to 10.45 mm with YOCO, on augmented InterHand2.6M.Tip-PA also drops from 37.15 to 23.29 mm; the reported results use K = 4 paired examples to infer session drift.Source: §5.2 · pp. 6–7, Table 1 · p. 7
YOCO scores best on every precision metric and takes 11.2 s of H100 conditioning, versus 45.6–95.4 s for gradient-based fine-tuning baselines.At matched LoRA capacity, PA-MPJPE is 10.45 mm for YOCO and 11.67 mm for LoRA-Finetune; Tip-PA is 23.29 and 25.33 mm, respectively.Source: Table 1 · p. 7, §5.2 · p. 7
On paired real-glove data, YOCO leads every primary metric and conditions in 6.73 s, versus 14.57 s for LoRA-Finetune and 13.04 s for Full-Finetune.The reported per-user standard deviations indicate that the advantage is consistent across four users.Source: Table 2 · p. 8, §5.2 · p. 8
Figure 3 plots PA-MPJPE against calibration-pair count for YOCO and YOCO-FT, with no calibration shown as a dotted reference.The experiment plan identifies sample efficiency in K as one of its research questions.Source: §5 · p. 5, Figure 3 · p. 8
90% overall task success for YOCO, versus 65% for LoRA-Finetune and 35% uncalibrated, with the lowest overall completion time across four tasks.Because a separate retargeting model is trained for each condition, these results characterize complete teleoperation pipelines rather than calibration alone.Source: Table 3 · p. 9, §5.2 · p. 9

6 Conclusion

YOCO reframes mocap calibration as feed-forward generation of LoRA updates for a frozen MANO estimator, with promising teleoperation gains in the evaluated pipelines. Its scope is pose calibration, and its session-stationary bias assumption leaves high-frequency drift as an open challenge.

6 Conclusion

A HyperNet converts a handful of paired poses into LoRA updates for a frozen MANO estimator, avoiding per-session optimization and showing promising gains in evaluated teleoperation pipelines.YOCO reframes mocap calibration as feed-forward weight generation.Source: §6 · p. 9
YOCO leaves MANO shape estimation to a separate stage that integrates with existing shape-fitting pipelines.Source: §6 · p. 9
High-frequency drift on less stable hardware would require re-calibration or a temporal extension of the HyperNet; covering it remains open.The stated session-stationary bias assumption is consistent with sensor manufacturing tolerances of most commercial gloves.Source: §6 · p. 9

A Method Details

The augmentation strategy simulates structured, session-consistent errors in MANO pose space, then uses forward kinematics to create noisy and calibrated keypoint examples. YOCO uses LoRA adapters across the MANO estimator, with the same adapter capacity for its LoRA-Finetune comparison.

A.1 Augmentation Strategy for HyperNet training

Synthetic biases are sampled once per sequence to reflect session-consistent miscalibration, combining offsets, gain errors, adjacent-finger crosstalk, and proximal-to-distal coupling.Biases are sampled from zero-mean Gaussian families with Bernoulli activation probabilities; multiplicative gain is clipped, and the coupling is tendon-like.Source: §A.1 · p. 13
The augmentation applies MANO forward kinematics to calibrated and noisy poses, stores the resulting pair offline, and trains YOCO to predict calibrated MANO poses from noisy joints using a small calibration set.The synthetic biases are generated in MANO pose space rather than by directly perturbing 3D keypoints.Source: §A.1 · p. 13

A.2 LoRA Adaptation Details

YOCO and LoRA-Finetune use rank-8 adapters with scaling factor 8 across all MANO-estimator layers, matching their adaptation capacity.YOCO generates the low-rank matrices; LoRA-Finetune optimizes the same adapter set by gradient descent on paired examples.Source: §A.2 · p. 13

B Learned-Prior Controls and Additional Baselines

At K = 4, learned-prior controls and additional baselines test YOCO's adaptation mechanism; the evaluation also specifies shape handling and a held-out-frame protocol for calibration-pose selection.

B Learned-Prior Controls and Additional Baselines

Cold-start LoRA-Finetune lacks a learned prior across calibration sessions; Meta-LoRA learns a session-independent initialization, while Meta-LoRA+FT adapts it on paired examples.Set-Cond. Residual predicts MANO-pose residuals without test-time gradients, and YOCO+LoRA-FT fine-tunes YOCO-generated adapters.Source: B Learned-Prior Controls and… · p. 14
13.51 to 13.14 mm on real glove data and 13.94 to 10.45 mm on augmented InterHand2.6M: YOCO reduces PA-MPJPE versus session-independent Meta-LoRA.Further LoRA fine-tuning improves some scores at the cost of test-time optimization.Source: B Learned-Prior Controls and… · p. 14, Table 4 · p. 14

C.1 MANO Beta Handling

InterHand2.6M evaluation uses ground-truth shape, real-world evaluation estimates shape by optimization, and downstream tasks hold a precomputed MANO shape parameter fixed across methods.Source: §C.1 · p. 14

C.2 Target Pose Selection

Calibration poses come only from the first 200 validation frames, with evaluation on the remaining frames to assess generalization to uncalibrated poses.Offline results average performance over three random seeds for selecting paired calibration poses.Source: §C.2 · p. 14

C.3 Real-World Data Protocol

The real-world protocol pairs low-cost glove keypoints with synchronized Apple Vision Pro reference poses across four users. The section also describes deployable calibration targets, CPU adaptation-time measurement, per-user variability reporting, and the selected glove's practical rationale.

C.3 Real-World Data Protocol

Four users each contribute one roughly 1,500-frame, 30-FPS evaluation sequence; low-cost glove keypoints are paired with synchronized Apple Vision Pro 3D keypoints as offline targets.Apple Vision Pro is an offline reference source, not a device required for deployment-time calibration.Source: §C.3 · p. 14
The preset-pose library provides four prompts: index pinch, fully open, claw / half close, and four fingers straight with the thumb near the palm.Source: Figure 5 · p. 15

C.4 CPU Deployment Calibration Time

CPU calibration time before deployment is reported in Table 5; YOCO time is HyperNet forwarding time, while baseline time is optimization or solving time.Source: §C.4 · p. 15, Table 5 · p. 15

C.5 Real-World Per-User Variability

Table 6 reports each method's standard deviation across users to show variation in real-world calibration results.Source: §C.5 · p. 15, Table 6 · p. 15

C.6 Mocap Device Selection

OpenCyberGlove is a low-cost tensile-sensor glove chosen because its readings are affected by user geometry, fit, hand size, and hand shape, making it a representative calibration test case.Other glove types considered include EMF-tracking, IMU/EMF-based, stretch-sensor, and bend-sensor devices.Source: §C.6 · pp. 15–16

C.7 Additional K-Sweep Results

The appendix specifies the scope and aggregation of additional K-sweep results and downstream teleoperation completion-time statistics.

C.7 Additional K-Sweep Results

The K-sweep table covers the augmented InterHand2.6M evaluation setting, aggregated over 3 seeds and 20 augmented units.Source: Table 7 · p. 16

C.8 Additional Teleoperation Statistics

Teleoperation completion-time statistics report average seconds over 10 trials, with lower values better.Source: Table 8 · p. 17
The teleoperation table also reports each task’s longest observed trial time.Source: Table 8 · p. 17

C.9 Device and Compute Platform

The appendix identifies the compute platform used for training and timing experiments and lists the training hyperparameters summarized in Table 10.

C.9 Device and Compute Platform

All training runs and timing measurements use a single-GPU server; Table 9 summarizes its hardware and operating system.Source: §C.9 · p. 17, Table 9 · p. 17

C.10 Training Hyperparameters

Table 10 reports optimization, HyperNet, LoRA, and paired-pose preference-extractor hyperparameters used in training.Source: §C.10 · p. 17, Table 10 · p. 17