Source-linked AI summary
Koala Gripper: Co-designing Robotic Grippers and Data-Capture Devices for Scaling Dexterous Manipulation Learning
Amar Hajj-Ahmad, Zubin Kremer Guha, Tim Fofonoff, Zhi Ern Teoh, Ciarán T. O'Neill, Ben Thacher, Igor Fala, Vidullan Surendran, Murphy Wonsick, Peter Whitney, David Watkins
TL;DR
Handheld robotics data collection devices have been limited in morphology, ergonomics, and manipulation capability. This paper co-designs a data-capture device and robotic gripper, yielding Koala grippers that support varied grasps, tool use, singulation, and learning from demonstration.
Problem
Existing handheld data-capture devices are limited by parallel-jaw designs, constrained integration, workspace interference, and ergonomic or cross-user generalization issues.
Method
The paper co-designs paired handheld data-capture and robotic gripper devices by defining shared task, workspace, and modality-specific requirements.
Results
The Koala gripper collected data in handheld and teleoperated modes and supported varied object grasps, forceful manipulation, precise singulation, and learned task policies.
Takeaways & Limitations
Co-designing the capture and execution devices produces a three-degree-of-freedom platform with broader manipulation capabilities than parallel-jaw grippers.
Takeaways & Limitations
The platform cannot perform some thumb-adduction grasps, and learned policies are limited by missing force-state information in collected data.
Abstract
from arXiv · showhide
As the demand for larger manipulation datasets grows, handheld robotic gripper data collection and the associated gripper designs become more vital. Current data collection device designs trend towards matching the morphologies of existing robotic grippers, sacrificing ergonomics and manipulation performance. In this paper, we propose a co-design framework that guides the simultaneous development of both data collection and robotic execution devices by weaving both platform constraints into the design process. Through this workflow, we present the Koala Gripper system, a data capture device and robotic gripper platform that improves dexterity and grasp capability compared to parallel jaw grippers while preserving scalability and ease-of-use. The design introduces a novel force-optimized finger/trigger linkage mechanism with directional reflected mass characteristics, a unique monolithic dual-thumb, and user-centered ergonomic design. The design's actuated robotic fingers are backdrivable, with effective mass on the order of tens of grams. We show that these grippers are capable of secure grasps over a wide range of objects, forceful tool use, and precise singulation. We further validate the platform by deploying it with an end-to-end data collection and policy execution pipeline that highlights its capabilities through learning from demonstration. More information available at http://koalagripper.rai-inst.com
I. INTRODUCTION
Robotic manipulation is moving beyond structured automation toward unconstrained environments, motivating scalable real-world data collection. Handheld capture is portable and intuitive but remains limited by gripper capability, ergonomics, and generalization, prompting a co-design framework for capture and robot devices.
- Dexterous manipulation in unstructured environments remains challenging, although stiff, repeatable grippers and classical control succeed in structured automation.
- Handheld data capture is portable, less complex, and less costly than teleoperation [10]–[13], while leveraging human intuition despite a larger training-to-execution gap.
- Current handheld devices are largely parallel-jaw, tied to pre-designed robots, or place the human hand inside the workspace, limiting manipulation capabilities [10], [14]–.
- Exoskeleton-inspired capture devices also face ergonomic limitations and poor generalization across users [17]–, [22]–, motivating co-design of data-capture and robotic devices.
II. DESIGN METHODOLOGY
The co-design methodology starts from shared task and workspace requirements, then iteratively integrates data-collection and robotic-device constraints through common system elements. This process keeps both devices aligned while adapting their designs to interacting ergonomic, mechanical, sensing, actuator, and packaging requirements.
- Co-design process: The workflow defines task capabilities and workspace targets for both devices, including grasp size, grasp type, and multimodal gripper-state sensing.These requirements shape how the gripper interacts with the environment and user.
- Data-collection device constraints: The data-collection device prioritizes ergonomics, mechanical advantage, and degrees of freedom to reduce fatigue and support forceful manipulation during extended capture.Mechanical advantage is especially important when the device’s finger output stroke limits force production.
- Robotic device constraints: The robotic device is constrained by actuators and their packaging, with low inertia supporting safer compliant contact and packaging shaping the gripper’s layout and overall size.Low-inertia actuators help the robotic device react more like the relatively low-inertia data-collection device.
- Co-design process: Device-specific constraints interact with common system elements in iterative feedback cycles, allowing shared features to evolve as the data-collection and robotic designs develop.A master modeling approach keeps both devices aligned by having them directly reference the same elements.
III. DESIGN IMPLEMENTATION · A. Gripper Active Workspace · B. Gripper Morphology and Grasp Taxonomies
The Koala co-design process uses active-workspace constraints and grasp requirements to shape a low-DoF, non-anthropomorphic gripper morphology. Its independently actuated fingers, underactuation, and monolithic dual thumb target tool use, secure wrapping, forceful grasps, and precise manipulation.
- III. DESIGN IMPLEMENTATION: The overall co-design objective is a robotic system that can use tools and complete tasks intended for a human operator.This objective links workspace planning and morphology choices to practical manipulation capabilities.
- A. Gripper Active Workspace: Workspace segmentation guides gripper form-factor decisions by preserving table-top access, complying with access openings, and avoiding interference with standard hand tools.The bottom region remains clear for surface picks, while top and back boundaries follow access-opening dimensions defined in.
- B. Gripper Morphology and Grasp Taxonomies: The design prioritizes low user-controllable DoFs while supporting dexterous behaviors such as torsionally stable tool use, trigger actuation, and reorientation.These requirements drive the morphology and grasp strategies away from anthropomorphic hand designs.
- B. Gripper Morphology and Grasp Taxonomies: Powerful pinch grasps can replace thumb-adduction lateral grasps, reducing DoFs while retaining the ability to perform fingertip and smaller wrap grasps.The same finger must meet the thumb for fingertip grasps and bypass it for smaller wrap grasps; low-DoF implementations include pairing 2-DoF fingers with a static thumb.
- B. Gripper Morphology and Grasp Taxonomies: Trigger-oriented grasping requires at least two independently actuated non-opposed fingers, with preshaping and underactuation enabling hooks, enveloping grasps, and irregular-object conformity.Underactuation also supports secure wrapping across different handle sizes and trigger actuation on tools such as drills and spray bottles.
- B. Gripper Morphology and Grasp Taxonomies: A monolithic dual thumb opposing both fingers balances moments during forceful power grasps, addressing a limitation of parallel jaw grippers.The design also supports prismatic grasps for writing, fine-tool stabilization, multi-object singulation, table-top pinches, and stable corner or palmar grasps.
C. Finger Mechanisms
The Koala finger mechanisms were co-designed for scalable handheld capture and motorized operation, balancing integration and manufacturing constraints. The system uses a 9-bar underactuated finger linkage and a monolithic 1-DoF dual-thumb mechanism for pinch and power grasps.
- Design Constraints: Co-design constraints favored mechanisms easier to integrate, manufacture, and service at scale than serial fully actuated or tendon-driven alternatives.Serial designs impose high DoF counts, while tendon-driven designs require complex assembly and tensioning.
- Finger Linkage: The Koala finger uses a 9-bar linkage with one underactuated and one actuated DoF, incorporating a crossed 4-bar fingertip and grounded underactuation spring.The mechanism was adapted from a 7-bar mechanism and avoids incursion into the palm area.
- Finger Linkage: A crossed 4-bar curls the finger during closure, while link-length differences drive the crossed 4-bar and proximal link.Underactuation is connected to a proximal prismatic stage inside the final device enclosure.
- Thumb Linkage: The monolithic dual-thumb pivots between fingertip and power-grasp positions as a simple 1-DoF system actuated by 4-bar mechanisms.Its initial and fully extended positions support pinch and power grasps, reducing user positioning precision and cognitive load; a singularity and hard stop separate the user from the thumb load path.
- Trigger Mechanism: The proposed trigger mechanism closely mimics natural fingertip closing paths [29], improving energy harvesting performance.This path is presented in Fig. 5.
D. Finger Actuation
Koala’s finger actuation uses a 4-bar trigger linkage to improve ergonomic energy harvesting and flatten mechanical advantage across the finger’s stroke. The robotic gripper packages compact, backdrivable actuation while preserving the handheld device’s low-inertia characteristics.
- CD Actuation: Koala uses a 4-bar trigger linkage to lengthen the stroke for improved energy harvesting without relying on an uncomfortable linear trigger.The design targets user comfort and strength limitations in mechanically powered handheld capture devices.
- RD Actuation: RD actuation is packaged compactly using shared finger parts while targeting the handheld capture device’s force output and low inertia through efficient, backdrivable reductions.Planetary gear reductions and ballscrews were considered for these characteristics.
- CD Actuation: The trigger and finger linkages counteract each other’s mechanical-advantage changes, producing a flatter curve across the full finger stroke.Figure 6 shows the finger, trigger, and combined mechanisms, with their combination creating consistent mechanical advantage over the full stroke.
- RD Actuation: An 18.8:1 planetary reduction with the same motor was chosen for thumb actuation to meet its higher torque requirement.The reduction is specified for the robotic gripper’s thumb actuation.
E. Active Surfaces · F. Handheld Device Ergonomics
The Koala gripper combines task-oriented active surfaces for pinch, power, and precise zero-thickness picking with an ergonomic handheld design based on surveyed hand dimensions and supported use across users and configurations.
- E. Active Surfaces: The thumb’s asymmetric pad resembles a natural opposing human thumb, while the finger pads support pinch and power grasps.Power grasps target objects up to 100 mm wide and 44 mm handles, the average human-designed handle size.
- E. Active Surfaces: Rigid metal fingernails enable precise singulation by picking thin flat objects from surfaces, turning pages, and performing zero-thickness picks.They also support handling small objects and related precision tasks.
- E. Active Surfaces: 0 mm pinch grasps and 7–100 mm diameter power grasps define the Koala gripper’s workspace, including a 44 mm standard-handle target.The pad geometry supports cylindrical handles and large-object grasps.
- F. Handheld Device Ergonomics: A survey of 30 potential users informed a handheld design intended for broad accessibility across unimanual and bimanual configurations.The design prioritizes wrist and grip support, varied hand sizes, and unassisted equipping.
- F. Handheld Device Ergonomics: Surveyed hand dimensions determined three grip sizes varying grip-to-fingertip distance, grip-to-trigger distance, and foam-insert thickness.Curved triggers follow users’ finger contours, as shown in Fig. 8 (1,2,3).
- F. Handheld Device Ergonomics: A canted thumb trigger and dorsal and ventral foam inserts engage the palm and reduce reliance on the ring and little fingers.The snug fit supports the hand without requiring its two weakest fingers to grip the device.
G. Sensor System · H. Data Collection
The sensor system measures controllable gripper state and multi-view visual observations for policy training and execution. Co-designed collection modes combine diverse, low-cost handheld demonstrations with higher-fidelity teleoperated data while preserving compatible state and action streams.
- G. Sensor System: The gripper state comprises 6-DoF pose, three controllable-DoF positions, and camera views, while underactuated finger DoFs are omitted because they are not directly controllable.The robotic arm and gripper actuators measure pose and controllable DoFs on the robotic device; the collection device measures them explicitly.
- G. Sensor System: Magnetic encoders measure collection-device finger and thumb linkage positions, which are converted to robotic-device actuator positions through a lookup table.A custom multi-sensor interface board supports the encoder measurements, while an Xvisio Seersense DS80 measures 6-DoF pose using visual odometry.
- G. Sensor System: Triggers map human actuation to the gripper’s thumb, index, and middle fingers, with dorsal and ventral cushions plus thumb-trigger side support improving ergonomic stabilization.These features are illustrated in Fig. 8.
- G. Sensor System: Three wide-angle Luxonis OV9782 global-shutter cameras—two side-mounted and one center-mounted—provide multi-angle views of the fingers, object, and scene with minimal occlusions.The arrangement supports fingertip visibility from multiple views for depth perception during fine manipulation.
- H. Data Collection: Two complementary modes support handheld in-the-wild collection for diverse, low-cost demonstrations and teleoperation for smaller volumes of higher-fidelity on-robot data.The teleoperated mode adheres to the robot’s real-world execution dynamics.
- H. Data Collection: Shared CD and RD architecture makes their state and action streams compatible without re-targeting.This compatibility links data collection directly to policy execution on the robotic device.
- H. Data Collection: Status LEDs and a CD button let users start, stop, and monitor collection without setting down the device, while live camera streams help keep relevant objects in view.The host-computer streams provide live monitoring throughout demonstrations.
IV. RESULTS · A. System Characterization
System characterization models Koala’s closed-chain finger dynamics to assess directional effective mass and backdrivability, while experiments test force parity between the capture and robotic devices. The results indicate asymmetric, grasp-supporting inertia, fingertip mass on the order of tens of grams, and successful force matching across tested object thicknesses.
- A. System Characterization: The closed-chain finger was modeled in MuJoCo using equality constraints, with planar inertial analysis performed in the x-z plane via finite differences and a reduced Jacobian.The reduced Jacobian accounts for velocities induced by active degrees of freedom in the complex kinematic chain.
- A. System Characterization: The reduced Jacobian maps actuator- and spring-induced velocities to fingertip velocities, while constraint-consistent rank-10 vectors project the full mass matrix into reduced coordinates.These quantities support construction of the operational-space inertia and directional effective mass.
- A. System Characterization: The effective-mass ellipsoids are asymmetric, with large-magnitude singularity regions generally aligned with gripped-object reaction forces to support grasp stability.The ellipsoids characterize both effective-mass magnitude and direction over the complete finger stroke.
- A. System Characterization: The robotic fingers’ fingertip mass is on the order of tens of grams, demonstrating ease of backdrivability.The analysis characterizes effective mass with the actuators included in the loop.
- A. System Characterization: Force-parity tests compared capture-device and robotic-device pinch forces on objects 10 mm to 40 mm thick under 100 N trigger input across ten trials per thickness.Grasp force was measured with a Biometrics P200 coin load cell.
- A. System Characterization: The robotic device successfully force-matched the capture device in the tested pinch-grasp experiments.The comparison used the force-output results reported in Table I.
B. Grasp Execution
The Koala CD qualitatively demonstrated versatile grasping and manipulation across unwieldy, unevenly weighted, and functionally complex objects, while its fingernails enabled precise handling of small and thin items.
- Grasp Execution: The Koala CD grasped and manipulated large unwieldy objects, unevenly weighted objects, and objects with complex functionality.These capabilities were qualitatively evaluated using a variety of objects and tools (Fig. 11).
- Grasp Execution: Its fingernails enabled pinch-grasp singulation of small objects down to nails and picking thin objects from flat surfaces.
C. Policy Execution
The Koala platform was validated for imitation learning by training a diffusion policy on handheld and teleoperated demonstrations, then executing it on a 7 DoF Franka Emika FR3. Representative successful completions included behaviors enabled by the non-parallel-jaw Koala gripper.
- Policy Training: A diffusion policy with a shared vision encoder and expanded 3 DoF end-effector action/state dimensions was trained for pasta straining and cup destacking.The dataset comprised 250 handheld and 100 teleoperated demonstrations across the two tasks.
- Policy Execution: The trained policies were rolled out on a 7 DoF Franka Emika FR3 using impedance control for compliant execution, with supplemental videos showing successful completions.The demonstrations and executions highlighted behaviors enabled by the non-parallel-jaw Koala gripper.
V. CONCLUSION
The paper presents a co-design framework for paired data-capture and robotic grippers, instantiated in the Koala platform with a 3-DoF morphology and distinctive finger and thumb mechanisms. The conclusion identifies thumb adduction and user mental load as important directions for further development.
- Conclusion: The co-design framework considers data-capture and robotic-gripper constraints from the beginning of development and produces the Koala platform.Koala combines a 3-DoF morphology with unique finger and thumb mechanisms intended to improve on parallel-jaw dexterity without compromising ergonomics.
- Future Work: The dual-thumb morphology recovers a wide range of grasps with only three controllable DoFs, but grasps benefiting from thumb adduction remain out of reach.The conclusion gives in-hand re-orientation of small parts as an example of an unsupported grasp category.
- Future Work: Adding an actuated thumb-adduction DoF is the most direct way to close this capability gap, at the cost of additional user mental load.