Untangling Dense Knots by Learning Task-Relevant Keypoints

• Dense knots complicate state estimation through high-dimensional configurations, visual homogeneity, and self-occlusions, while manipulation requires managing friction, tension, stiffness, and slack.Source: §1 · p. 1
• BRUCE uses known cable graphs to undo crossings sequentially, while HULK learns pin and pull keypoints from visual input instead of estimating full cable state.Source: §1 · pp. 1–2
• HULK targets densely knotted cables using task-relevant keypoints, because full state estimation is unnecessary for untangling.Source: §1 · pp. 1–2
• The study evaluates HULK in simulation and on a da Vinci robot across dense knot configurations and varied cable textures, comparing two HULK variants with baselines.Source: §1 · p. 1, Figure 1 · p. 2

• Dense-knot handling remains an open problem despite substantial progress with analytical, learned, and hybrid cable-manipulation approaches.Source: §2 · p. 2
• Earlier work represented cables as graphs from RGB-D perception and achieved 76.9% success on loosely knotted physical cables.Source: §2 · p. 2
• Global correspondence methods become difficult with complex configurations because segmentation failures and overlapping segments undermine robust state estimation.Source: §2 · p. 2
• HULK estimates task-specific keypoints and reasons about intersection geometry, separating visual reasoning from manipulation instead of learning a black-box policy.Source: §2 · p. 3

• The robot must reach a fully untangled cable with no crossings through a sequence of bilateral pin and pull actions from RGB observations.Source: §3 · p. 3
• The starting cable is assumed semi-planar, limited to overhand or figure-eight knots, and visible at both distinguishable endpoints.Source: §3 · p. 3
• HULK uses only images rather than explicitly reconstructing the cable graph, while each gripper executes four-variable pin or pull actions.Source: §3 · pp. 3–4
• A linear graph represents intersections and endpoints, with degree-1 endpoints, degree-4 crossings, and |V| = N + 2 and |E| = 2N + 1.Source: §3 · p. 3, Figure 2 · p. 3
• Edges receive +1 when they are endpoints or pass above another edge at a crossing, and −1 otherwise.Source: §3 · p. 3

4 BRUCE: Basic Reduction of Under-Crossing Entanglements

• BRUCE untangles semi-planar cables from a known graph using pinning and pulling primitives called Reidemeister and Node Deletion moves.Source: §4 · p. 4
• Reidemeister moves remove non-knot occlusions, while Node Deletion removes one crossing by pinning one edge and pulling the edge labelled −1 across the crossing.Source: §4 · p. 4
• BRUCE repeatedly selects the first relevant under-crossing from the right endpoint, performs Node Deletion, and restores the configuration with a Reidemeister move.Source: §4 · p. 4, Figure 3 · p. 4
• HULK mirrors this sequence by predicting endpoint, pin, and pull keypoints and repeating moves while a knot bounding box remains detected.Source: Figure 3 · p. 4
• Each correctly executed BRUCE iteration monotonically reduces crossings until the cable has no self-occlusions, guaranteeing untangling for semi-planar configurations.Source: §4 · p. 4

5 HULK: Hierarchical Untangling from Learned Keypoints

• HULK learns task-relevant keypoints instead of requiring a given cable graph or full state estimate, then applies geometric untangling primitives.The learned keypoints identify endpoints and the pull/pin locations needed for manipulation.Source: §5 · p. 4, §5.3 · p. 5, §5.3.1 · p. 5

5.1 Simulation

• The simulation environment models a mass-spring cable with smooth and braided textures and generates perturbed dense knots from predefined tying trajectories.Blender supports appearance variation, while ray-traced crossings and BRUCE executions provide synthetic annotations.Source: §5.1 · pp. 4–5, §5.2 · p. 5

5.2 Dataset Generation

• HULK uses synthetic graph-derived annotations for simulation and 20×-augmented hand-labelled real images for physical deployment.Synthetic data records RGB renderings and crossing, endpoint, and bounding-box annotations; real data are labelled directly.Source: §5.2 · p. 5

5.3 HULK Implementation

• Both HULK variants sequentially undo crossings, with HULK-G using global images and HULK-L adding local knot crops for pull and pin prediction.HULK-L predicts endpoints globally, selects the rightmost knot, and estimates the next pull/pin pair within a resized crop.Source: §5.3 · p. 5, §5.3.2 · p. 6

5.3.1 Manipulation

• Node Deletion pulls from the predicted pull point toward the pin point while the second gripper pins the cable, and Reidemeister moves straighten both endpoints.The corresponding action vectors are computed directly from the predicted pixel coordinates and predefined workspace targets.Source: §5.3.1 · p. 5
• HULK terminates when the predicted endpoint is freed from an under-crossing or when the action limit T = 30 is exceeded.The threshold λ = 0.7 favors false negatives over positives because premature termination is considered more harmful than late termination.Source: §5.3.1 · p. 5

5.3.2 Perception

• HULK-G globally regresses four keypoints, whereas HULK-L combines global endpoint estimation with local pull/pin heatmaps inside a predicted knot crop.The local model predicts classification and x/y offset heatmaps for each keypoint.Source: §5.3.2 · p. 6, Figure 4 · p. 6

6 Experiments

• Simulation evaluates three dense starting configurations, three cable textures, two HULK variants, and three baseline policies over 21 trials per combination.The configurations are single overhand, single figure-eight, and overhand in series with figure-eight knots.Source: §6 · p. 6, §6.1 · p. 7

6.1 Simulation Experiments

• HULK failures arise from modular perception errors, including missed knot boxes, incorrect crops, wrong crossing keypoints, and premature or late termination.These failure modes motivate component-wise improvement rather than identifying a single universal failure source.Source: §6.1 · p. 7
• HULK-L and HULK-G outperform Random across all tested textures and initial knot configurations, with higher success and efficiency.Their success curves converge more steeply than Random in the simulation plots.Source: §6.1 · p. 7, Figure 5 · p. 7
• HULK variants match the Depth baseline in several trials despite lacking depth and despite Depth receiving ground-truth endpoints.Depth provides a strong noise-free simulation signal, while its fixed leftward pull is not camera-pose agnostic.Source: §6.1 · p. 7
• HULK-L outperforms HULK-G in 2 of 3 multiple-knot experiments because hierarchical perception better targets the correct crossing.The comparison attributes the advantage to local prediction of the pull/pin pair around the relevant crossing.Source: §6.1 · p. 8

6.2 Physical Experiments

• Physical trials use a da Vinci robot and a smooth elastic hair tie, with HULK-G trained from 350 hand-labelled images and compared against Random.Trials include dense and looser configurations, and success requires no knots and at most one crossing.Source: §6.2 · p. 8

7 Conclusion

• HULK successfully untangles dense overhand and figure-eight knots from visual input using learned keypoints and geometric manipulation.The paper evaluates HULK against three simulation baselines and one physical baseline.Source: §7 · p. 8
• Physical trials suggest HULK untangles dense knots in well over 50% of trials, with most failures attributed to mechanical effects.The conclusion reports trials using 350 hand-labelled training examples and robot untangling sequences averaging 3–18 actions.Source: §7 · p. 8
• Future work will address more complex knot configurations, variation in cable physical properties and sizes, and active sensing for uncertain manipulation states.These extensions define the stated scope beyond the evaluated settings.Source: §7 · p. 8

A Simulator Details

• A custom Blender simulator models cables as rigid-body cylinders linked by springs, supporting frictional manipulation and varied cable textures.The simulator uses braided or smooth textured meshes and selects hyperparameters to balance realism against rendering time.Source: A Simulator Details · p. 12, Table 2 · p. 12

A.1 Manipulation

• Keyframed cylindrical segments implement unilateral pick-place actions, Reidemeister moves, and Node Deletion moves for cable manipulation.Reidemeister moves pull the endpoints sequentially, while Node Deletion combines pulling one cylinder with pinning another.Source: §A.1 · p. 12
• Bilateral endpoint trajectories create overhand and figure-eight knots, while noise, endpoint lifting, and random pick-place actions perturb initial configurations.These randomizations are applied during both training and testing to produce varied knot placements and configurations.Source: §A.1 · p. 12, Figure 6 · p. 13

A.2 Perception

• BRUCE supplies self-supervised perception data by guiding simulated untangling and recording RGB images with keypoint and knot-box annotations.The annotations support the perception components used to train HULK.Source: §A.2 · p. 13

A.3 Completeness of BRUCE

• BRUCE operates on a graph of cable crossings and repeatedly detects under-crossings before applying Node Deletion and Reidemeister moves.The heuristic searches for spatially separated cylinder pairs with sufficient depth difference to identify an under-crossing.Source: §A.3 · pp. 13–14
• Under Assumptions A.1–A.4, BRUCE terminates with no crossings in at most 2N0 moves.The assumptions require node deletion to reduce crossings, Reidemeister moves not to increase them, no consecutive Reidemeister moves, and reliable termination detection.Source: §A.3 · p. 13
• The termination proof follows because the crossing count never increases and decreases by at least one every other move.This establishes the stated 2N0 upper bound on the number of moves.Source: §A.3 · p. 13
• A comparable guarantee for HULK is substantially harder because its moves and termination condition depend on learned perception accuracy.The graph-specific runtime bound may also be lower in configurations where Reidemeister moves remove multiple crossings or immediately untangle a coil.Source: §A.3 · pp. 13–14

A.3.1 Dataset Generation

• Heuristic BRUCE generates training data by untangling randomized knot trajectories while recording RGB images, keypoints, and knot bounding boxes.The keypoints correspond to cable endpoints and the pull and pin locations selected by BRUCE.Source: §A.3.1 · p. 14
• Knot bounding boxes are formed from cylinders surrounding a detected under-crossing, using a four-cylinder buffer before and after the selected pull and pin cylinders.For multiple knots, the annotation process repeats on the remaining cable segment after the detected pin index.Source: §A.3.1 · p. 15

B.2 HULK-G Keypoint Regression

• HULK-G predicts four task-specific keypoints from a global RGB image, using four Gaussian heatmaps to localize cable endpoints and manipulation points.A stride-8 ResNet-34 backbone outputs four channels at the original 640 × 480 resolution.Source: §B.2 · p. 15

B.3 HULK-L Keypoint Regression

• HULK-L combines global endpoint regression with local knot-crop regression to refine pull and pin keypoints.Its local network classifies pixels near each target and regresses x,y offsets for fine localization.Source: §B.3 · pp. 15–16
• Both HULK variants achieve comparable simulated untangling performance, while HULK-L is more vulnerable to compounding errors from bounding-box detection.Their results support task-specific keypoints coupled with geometrically designed manipulation policies.Source: §B.3 · p. 16

C.1 Quantifying Density

• Dense configurations are defined by L(E*) = 0, meaning adjacent cable edges have no empty space and overlapping segments cannot be reliably separated by segmentation.Loose configurations instead have L(E*) > 0.Source: §C.1 · p. 16

C.2.1 Deploying HULK

• Deployment uses inward-oriented grippers and ordered pull-then-pin execution for Node Deletion moves, while Reidemeister grasps are oriented using local principal-component analysis.A simplified termination condition is used because hair-tie stiffness caused false positives with the original condition.Source: §C.2.1 · pp. 16–17
• Physical HULK uses real-data-trained keypoint and bounding-box networks, adaptive workspace crops, and sequential inference to update the pin grasp after the pull.The dataset contains 350 labeled hair-tie images augmented 20X.Source: Figure 7 · p. 16, §C.2.1 · p. 17, Figure 8 · p. 17

C.2.2 Manipulation Details

• Physical trials include loose and dense overhand and Figure-8 knots, with failures involving end-effector entrapment, workspace exit, and near misses from knot density.The end-effector failure is associated with cable friction or aggressive grasp planning.Source: Figure 9 · p. 18, §C.2.2 · p. 18

C.3.1 Failure Modes

• Premature termination immediately causes untangling failure, whereas bounding-box false negatives can sometimes be recovered from or instead exhaust the action budget.These are the two prominent simulated failure modes summarized in Table 4.Source: §C.3.1 · p. 18, Table 4 · p. 18
• Random actions produce the highest rates of bounding-box false negatives and early triggers, while HULK variants show comparable overall failures with fewer actions.HULK’s intermediate configurations more closely resemble those produced by the BRUCE supervisor and the detector’s training data.Source: §C.3.1 · pp. 18–19