PaperlayerView exact revision on arXiv

Source-linked AI summary

PAKT: Physically-Aligned Kinesthetic Teaching for Reinforcement Learning

Lars Johannsmeier, Yashraj Narang

arXiv:2609.25630v1cs.RO

TL;DR

Contact-rich industrial manipulation requires precise, reliable behavior, but collecting human guidance that policies and controllers can reproduce remains a challenge. PAKT aligns kinesthetic teaching with policy execution through shared bounded control, and reports reduced cycle time and intervention counts relative to HIL-SERL.

  • Problem

    Contact-rich industrial manipulation requires precision and reliability, while collecting human guidance that policies and downstream controllers can reproduce remains a challenge.

    Source: §1 · pp. 1–2

  • Method

    PAKT uses admittance control for kinesthetic guidance and a shared reference generator and impedance controller so guidance and policy actions use the same action space and bounded dynamics.

    Source: §1 · p. 2

  • Results

    Relative to HIL-SERL, PAKT reduced cycle time by 23%–48%, cumulative intervention count by 62%–86%, and initial demonstration time by 15%–34% across reported runs.

    Source: §6 · p. 9

  • Takeaways & Limitations

    With the controller fixed, kinesthetic guidance reduced initial demonstration time by 8%–22% and cumulative intervention count by 34%–64%, while final success and cycle time remained similar.

    Source: §6 · p. 9

    Evaluation covers single-arm insertion on one Franka; other robots and tasks, multiple operators, and admittance sensitivity or stability guarantees remain untested.

    Source: §5 · p. 9

Abstract

from arXiv · show

Real-world reinforcement learning (RL) systems still struggle with the demands of contact-rich industrial manipulation, including micrometer-level precision, success rates above 99%, and human-level cycle times. Although off-policy algorithms can improve performance by leveraging demonstrations and interventions, a key bottleneck is the lack of an intuitive interface for collecting such guidance while complying with constraints of the physical system and the policy. We propose PAKT, a framework for kinesthetic teaching in RL. As opposed to teleoperation approaches, PAKT relies on kinesthetic guidance, which is widely used in industry. However, a critical weakness of kinesthetic guidance is the possibility for the operator to move the robot along trajectories (e.g., velocities, accelerations, jerk) that the robot and/or policy cannot physically reproduce. Using PAKT, operators guide the robot through admittance control, which maps human-applied forces to motion. The downstream reference generator applies the same kinematic limits used during policy execution, keeping the collected trajectories within these limits. To support this teaching interface with an appropriate execution layer, PAKT adds a high-performance control stack that maps low-frequency RL actions to high-frequency torque commands. It consists of a reference generator and subsequent impedance controller, where the reference generator preserves the tracking performance of the impedance controller while improving contact handling and producing smoother policy actions. Across the reported runs on four insertion and industrial assembly benchmarks, including a data center compute tray, the end-to-end system reduces cycle time by 23%-48% and cumulative intervention count by 62%-86% relative to the HIL-SERL baseline. Project website: https://pakt-website.github.io/pakt-website}{https://pakt-website.github.io/pakt-website

1 Introduction

PAKT addresses the difficulty of collecting policy-feasible human guidance for contact-rich manipulation by aligning kinesthetic teaching and policy execution within a shared control stack. Across four evaluated tasks, the end-to-end system improves efficiency relative to HIL-SERL.

1 Introduction

  • High-precision contact-rich industrial manipulation demands tight tolerances, high success rates, short cycles, and reliable behavior, while real-world RL faces sample-efficiency and robustness challenges.Precise contact-rich behavior is also difficult to transfer from simulation to hardware.Source: §1 · p. 1
  • Teleoperation can provide demonstrations but may require unintuitive mappings, operator training, or dedicated hardware..Source: §1 · p. 1
  • Naive kinesthetic teaching can produce velocities, accelerations, or jerks that the policy and downstream controller cannot reproduce, making demonstrations infeasible to imitate.The paper identifies this mismatch as a critical weakness of kinesthetic guidance.Source: §1 · p. 2
  • PAKT couples kinesthetic guidance and policy actions through shared 6-DoF coordinates, kinematic limits, reference dynamics, and tracking, keeping stored corrections within the policy action space.The reference generator applies the same limits during guidance and autonomous execution.Source: §1 · p. 2, Figure 1 · p. 2
  • A reference generator applies execution limits to low-frequency RL actions before a high-frequency compensated Cartesian impedance controller tracks them.The stack is designed to preserve accurate tracking, improve contact handling, and smooth policy execution without changing the high-level learning algorithm.Source: §1 · p. 2
  • 23%–48% shorter cycle time, 62%–86% fewer cumulative interventions, and 15%–34% less initial demonstration time were reported relative to HIL-SERL.The measurements came from one trained operator; only RAM insertion was repeated across five random seeds.Source: §1 · p. 2

2 Related Work

Related work spans real-world robotic RL and downstream control, motion generation, and human assistance. PAKT combines these lines through policy-aligned kinesthetic guidance and a shared control stack.

2 Related Work

  • Contact-rich robot learning motivates stronger downstream control, including impedance and admittance control; PAKT integrates desired-motion tracking, dynamics compensation, and automatic damping design.Prior real-world systems often use position-controlled or simplified interfaces, while contact-rich tasks can benefit from torque-controlled platforms.Source: §2 · pp. 2–3
  • Because RL actions can be jerky, prior work explores smoothness-aware exploration, action regularization, and jerk-bounded trajectory generation; PAKT uses a lightweight generator upstream of impedance control.Classical robotics uses jerk-bounded or online trajectory generation to satisfy kinematic limits while preserving execution quality.Source: §2 · p. 3
  • Human-assistance interfaces range from 3D mice to teleoperation, while CR-DAgger collects concurrent on-policy corrections; PAKT instead pauses policy execution for fine-grained kinesthetic interventions.A cited 20-participant study found gamepad teleoperation yielded shorter trajectories and lower interaction forces but required more programming time; interface comparisons depend on task, population, and metric.Source: §2 · p. 3

3 Methods

The methods combine demonstration-based reinforcement learning and human corrections with a shared control stack for policy actions and kinesthetic guidance. The stack maps human or policy inputs into bounded references and tracks them with an impedance controller.

3.1 Reinforcement Learning

  • Policy learning uses RLPD, which extends SAC with demonstration and robot-experience replay buffers sampled equally for critic and actor updates.The policy acts on camera images, Cartesian pose, and velocity, and selects 6-DoF Cartesian delta-pose actions at 10 Hz.Source: §3.1 · pp. 3–4

3.1.1 Human-in-the-Loop Corrections

  • Human corrections pause policy execution, log corrective transitions, and add them to both replay buffers for continued training.Guide mode begins with reference and admittance poses reset to the measured pose with zero derivatives; the authors note this does not provide a formal worst-case force guarantee.Source: §3.1.1 · p. 4

3.3 Controller

  • The control stack routes policy targets directly to a reference generator and human wrenches through admittance control before reference generation and impedance tracking.The three control blocks execute at 1 kHz; admittance maps wrench to motion, the reference generator applies kinematic bounds, and impedance control tracks the reference.Source: §3.3 · pp. 4–5
  • The reference generator filters desired motion and applies velocity and acceleration limits before guide trajectories are logged at the policy rate.Applying the execution limits before storage ensures the logged action generated the corresponding transition; the frequency of limit saturation was not logged.Source: §3.3 · p. 5
  • The impedance layer uses Cartesian tracking terms, while a nullspace controller helps keep the redundant robot’s elbow upright.The controller compensates Coriolis and gravity torques; its comparison includes C1 at torque level and C2–C5 as Cartesian wrench terms, with C3 matching a common RL controller structure.Source: §3.3 · pp. 5–6

4 Experiments

Across four insertion and assembly tasks, PAKT and the hybrid reduce cycle time relative to the baseline, while PAKT also reduces intervention counts. Controller tests compare tracking and contact response, and the RL experiments evaluate the systems under a shared protocol.

4.1 Setup

  • The experiments use a Franka Emika robot with a Franka Hand, two wrist cameras, and a controller stack running on a Jetson Orin AGX.The policy and success classifier run on a desktop PC with an NVIDIA RTX 4090 GPU.Source: §4.1 · p. 6

4.2 Controller Stack Comparison

  • C1 and C2 achieve nearly identical tracking error, while C3/C4 have larger errors and C5 performs worst because error clipping causes lag.In the hard-contact test, C1’s peak force is 22.5 N versus 33.9 N for C2; the 10 N stopping condition is checked at 10 Hz while impedance control runs at 1 kHz.Source: §4.2 · pp. 6–7, Figure 2 · p. 6, Table 2 · p. 7

4.3 RL Experiments

  • The four tasks are FMB peg insertion, RAM insertion, limit fixture assembly, and busbar assembly, evaluated after 20 demonstrations and RL training.Each task is evaluated in 100 executions from random initial states; RAM uses five seeds, while the other tasks are single runs.Source: §4.3 · p. 7, Figure 3 · p. 7, Table 3 · p. 8
  • PAKT reduces initial demonstration time by 8%–22% and cumulative intervention count by 34%–64% relative to the hybrid, with similar final success and cycle time.The hybrid-versus-PAKT comparison isolates kinesthetic guidance under the same controller.Source: §4.3 · pp. 7–8
  • 23%–48% lower average cycle time and 62%–86% fewer cumulative interventions: the end-to-end system’s reductions relative to the HIL-SERL baseline.Similar cycle times for the hybrid and PAKT indicate that the controller stack accounts for most of the cycle-time reduction.Source: §4.3 · pp. 7–8, Table 3 · p. 8, Figure 4 · p. 8

5 Limitations

The reported interface comparison and hardware evaluation have important scope limits: they do not establish user-to-user generalization, and the wrench estimate and tested control configuration constrain what the results demonstrate.

5 Limitations

  • One trained, unblinded operator performed the interface comparison, so it does not establish user-to-user generalization, usability, or a general advantage over teleoperation.The authors call for a counterbalanced multi-user study.Source: §5 · p. 9
  • The joint-torque wrench estimate cannot separate human forces from environment contact, and friction can bias estimates and guidance dynamics.Estimate quality depends on the robot; the Franka platform provides a high-quality external-wrench estimate.Source: §5 · p. 9
  • Evaluation covers single-arm insertion on one Franka platform; transfer, non-insertion and bimanual tasks, multiple operators, and stability guarantees remain untested.The evaluated impedance layer requires torque control; position-controlled operation is proposed but untested.Source: §5 · p. 9

6 Conclusion

PAKT aligns kinesthetic corrections and policy actions through shared bounded dynamics. Across reported runs it improves several measures relative to HIL-SERL, while the controller and interface comparisons indicate distinct patterns in cycle time and guidance outcomes.

6 Conclusion

  • PAKT stores human corrections in the same 6-DoF action space as policy actions and executes both through the same bounded dynamics.The reference generator applies velocity, acceleration, and jerk limits before Cartesian impedance tracking.Source: §6 · p. 9
  • 23%–48% lower cycle time, 62%–86% fewer cumulative interventions, and 15%–34% less initial demonstration time versus HIL-SERL across reported runs.With the controller fixed, kinesthetic guidance reduces initial demonstration time by 8%–22% and cumulative interventions by 34%–64%, while final success and cycle time remain similar.Source: §6 · p. 9
  • The hybrid ablation indicates that the controller stack accounts for most of the cycle-time reduction relative to HIL-SERL.The reported gains are limited to one trained operator, with five-seed verification only for RAM insertion.Source: §6 · p. 9

A Experimental Setup

The appendix describes a shared robot and runtime setup, common policy interfaces, and four insertion or assembly tasks. It distinguishes each task by its objective and relevant setup constraints.

A Experimental Setup

  • The shared rig sends policy commands at 10 Hz and runs the controller stack at 1 kHz; Figures 5–6 show the setup and four task scenes.The robot is a Franka Emika with a Franka Hand, and the policy and success classifier run on a desktop PC.Source: A Experimental Setup · p. 12, Figure 5 · p. 13, Figure 6 · p. 14

A.1 Shared Setup

  • Table 4 covers shared hardware and runtime, while tasks share policy inputs of two wrist-camera RGB images, Cartesian pose, and velocity, and 6-DoF Cartesian delta-pose actions.Success-classifier inputs and environment settings vary by task.Source: §A.1 · p. 12, Table 4 · p. 12

A.2 Peg Insertion

  • The FMB peg-insertion task uses a 3D-printed peg and hole with approximately 2 mm diameter clearance; the robot retains the peg throughout.The objective is insertion into the matching benchmark-board hole.Source: §A.2 · p. 12

A.3 RAM Insertion

  • RAM insertion requires aligning a grasped module with its socket and seating it; tighter tolerances and common partial insertions make it more demanding than peg insertion.The slot clips were modified so the robot could remove the RAM stick itself.Source: §A.3 · p. 12

A.4 Limit Fixture

  • The limit-fixture task inserts a fixture into a tray for downstream testing and busbar insertion; its proprietary production origin limits disclosed process details.The task requires accurate alignment and reliable insertion under limited clearances.Source: §A.4 · p. 13

A.5 Busbar

  • Busbar assembly is the direct follow-up to fixture preparation: the robot inserts a power-conducting busbar into its target location.Some setup details are withheld for confidentiality.Source: §A.5 · p. 14

A.6 Task Configuration

The task-configuration appendix identifies the settings summarized for the four tasks and notes that policy and training defaults are shared across them.

A.6 Task Configuration

  • Table 5 summarizes episode length, action scales, and reset randomization for the four RL tasks.It presents task-specific configuration.Source: §A.6 · p. 14, Table 5 · p. 14

B Policy and Training Details

B.2 Architecture Details

The appendix records shared policy and vision settings, then specifies task-specific binary success classifiers used to provide the RL reward.

B.2 Architecture Details

  • Table 6 summarizes shared policy and training details, while Tables 7 and 8 cover actor–critic optimization and vision encoding and augmentation.The appendix identifies these tables as the locations of the settings but does not state their values in the supplied passages.Source: §B.2 · p. 14, Table 6 · p. 15, Table 7 · p. 15

C Reward and Success Classification

  • For all four tasks, a task-specific visual success classifier yields binary reward when sigmoid(fψ(o_t)) exceeds 0.85, and zero otherwise.Classifier input views differ by task, but the thresholded reward construction is shared.Source: C Reward and Success Classification · p. 15

C.1 Classifier Details

  • Each task has a separate classifier mapping its current visual observation to a success score, following the reward-design recipe of [6].The classifiers are separate because visual signatures of success differ across the four tasks.Source: §C.1 · p. 16
  • Table 9 summarizes the success-classifier architecture and optimization details.The supplied passage identifies the table’s scope but gives no architecture or optimization values.Source: Table 9 · p. 16

C.2 Classifier Dataset and Hard Negatives

The appendix describes hard-negative classifier training, controller parameter choices, and the protocol for collecting demonstrations across the four tasks.

C.2 Classifier Dataset and Hard Negatives

  • Approximately 3,000 negative and 1,500 positive samples per task include near-goal hard negatives to reduce false positives in online reward evaluation.The classifier is trained before policy learning, then frozen and used only for reward computation.Source: §C.2 · p. 16

D Controller Stack Parameters

  • The controller comparison contrasts baseline Cartesian damping with asymmetric error clipping against modal-space damping factors without clipping; Table 10 summarizes the parameterizations.The stack reuses six damping factors ζ_i = 0.7 and admittance settings across tasks; this is not a sensitivity study or stability proof.Source: D Controller Stack Parameters · p. 16, Table 10 · p. 17

E Demonstration and Intervention Protocol

  • Demonstration and intervention logic is shared across all four tasks unless otherwise stated, and these workflow details affect real-world training time and usability.The section distinguishes workflow from the core control and learning equations.Source: E Demonstration and Intervention… · p. 16

D Controller Stack Parameters

  • Table 11 lists task-specific signed directional reference-generator limits derived from the action scales in Table 5.The passage identifies the limits’ source and format, but supplies no numeric bounds.Source: Table 11 · p. 17

E.1 Demonstration Collection

  • For each method and task, one trained, unblinded operator collected 20 demonstrations; kinesthetic guidance was performed by hand with guide mode active.The tasks did not require regrasping, and reset poses were randomized in x, y, and around z; practice rollouts were excluded from reported times and replay buffers.Source: §E.1 · p. 17

E.2 Human Interventions During Training

During RL training, corrective interventions are retained as learning data and added to both replay buffers.

E.2 Human Interventions During Training

  • When policy behavior is unsafe, misaligned, or suboptimal, operator corrections override it and enter both demonstration and online RL replay buffers.The resulting observation-action-reward tuples are retained for subsequent updates.Source: §E.2 · p. 17
Loading 2609.25630v1…