PaperlayerView exact revision on arXiv

Source-linked AI summary

Predict the Swing, Match the Speed: Robotic Pipe Stabbing With Learned Stereo Vision and Residual Reinforcement Learning

Mohamed Ashraf Mostafa, Taher El-Fakharany, Samir Khaled

arXiv:2610.37535v1cs.RO

TL;DR

Stabbing a freely swinging pipe with a fixed guide can involve impact at the swing’s prevailing speed, posing risks to equipment and workers. StabBot predicts the swing and captures the pipe at matched velocity, combining stereo perception, touch-confirmed approach, and residual correction. In a 200-job simulation benchmark it seated every joint, compared with 74% for a fixed guide, while reducing median contact speed and peak force by a factor of 7.8 each.

  • Problem

    A fixed guide meets the swinging pipe at whatever speed it has, while pipe impacts can damage threads and seals and pose a hazard to workers.

    Source: §1 · pp. 1–2

  • Method

    StabBot uses stereo landmarks and a tally-tuned pendulum filter to predict the swing, then captures at matched velocity with touch confirmation and residual control correction.

    Source: §1 · p. 2

  • Results

    100% of 200 random jobs were seated by StabBot versus 74% with a fixed guide, and median contact speed and peak force were each lower by a factor of 7.8.

    Source: §9 · p. 15

  • Takeaways & Limitations

    The benchmark reports measurable contributions from the guarded approach, velocity-matched prediction, and residual learning, which lowered peak force by a further 22%.

    Source: §9 · p. 15

    The study is a full-scale-equivalent simulation; make-up, real images, and rig-specific effects such as elevator links remain for a later test-stand stage.

    Source: §8 · p. 15

Abstract

from arXiv · show

A joint of pipe run into the hole swings from the elevator over the well and strikes a fixed stabbing guide at whatever speed the swing has. StabBot, a robotic stabbing system, predicts the swing and meets the pipe at its own speed. A pendulum Kalman filter tuned from the pipe tally tracks the joint in stereo images. A six-axis arm with an open V-shaped roller head closes in at matched velocity, confirms the last millimetres by touch, aims the pin over the box when the elevator hangs off centre, and follows the block down with lateral compliance. A ResNet-18 keypoint network locates the pipe, pin nose and box mouth in 3-D, and a residual reinforcement-learning policy refines the capture. On 200 paired random jobs with nine sizes from 2-3/8-in. tubing to 20-in. casing, StabBot seated every joint with the pipe touching only rollers, against 74% for a conventional fixed guide. Median contact speed fell from 18.4 to 2.3 cm/s and median peak head-pipe force from 11.3 to 1.4 kN, and the force was lower in all 200 jobs (Wilcoxon $p < 10^{-30}$). The network locates the pipe with 0.4-mm median error in 2.3 ms per stereo pair. With the network in the loop, success stayed at 100% under 20 mm of added noise, 300-ms delay, 85% frame loss, 20-mm calibration offset, 20-cm elevator offset and 40-cm swings. All results come from a 1:4 Froude-scaled MuJoCo model and are reported at full scale.

1 Introduction

StabBot addresses the hazard and impact of stabbing a freely swinging pipe by predicting its motion and capturing it at matched velocity. The introduction situates this approach among prior robotic stabbing, swing-catching, vision, and residual-learning work.

2.1 Drill floor automation and stabbing

  • Prior drill-floor systems include stabbing guides, but the reviewed systems wait at well centre for the swinging pipe rather than estimating its swing and planning capture.Related work also describes systems that guide a stand already held by the robot.Source: §1 · p. 2, §2.1 · p. 2

1 Introduction

  • The study tests whether swing prediction improves stabbing reliability and gentleness, and measures the contributions of predictive control, learned perception, and learned correction.StabBot watches the swing, fits a tally-tuned pendulum model, closes in at matched velocity, confirms contact by touch, and compensates for an off-centre elevator.Source: §1 · p. 2
  • StabBot seated 100% of 200 random jobs, versus 74% for a fixed guide, while median contact speed and peak force were each lower by a factor of 7.8.The benchmark evaluates the system against a conventional guide.Source: §1 · p. 2

2.2 Swinging loads, catching and fine motion

  • StabBot adapts velocity-matched catching to a pendulum-suspended tubular and follows the catch with insertion; prior examples address suspended loads, catching, or guarded fine motion separately.The cited robotic beam-catching precedent assumes the impact time is known in advance.Source: §2.2 · p. 3

2.3 Keypoint vision

  • Reviewed drill-floor vision work locates pipe with bounding boxes or measures stick-up, rather than locating pipe, pin, and box as 3-D landmarks.Keypoint methods and stereo-based 3-D keypoint estimation have been studied in other settings.Source: §2.3 · p. 3

2.4 Residual reinforcement learning

  • Residual learning adds a correction to a conventional controller, while prior work identifies risks from unsmoothed actions and reward gaps that also arose in this study.The paper reports its remedies in the design lessons and identifies no reviewed system combining online free-swing estimation with matched-velocity stabbing.Source: §2.4 · p. 3

3 System

The system is evaluated in a Froude-scaled MuJoCo model, with full-scale-equivalent quantities reported. Its open roller head is designed to accommodate the modelled pipe sizes while limiting contact to rollers.

3.1 Rig model and scaling

  • The rig, hanging joint, robot, and guide head are modelled in MuJoCo at 1:4 Froude scale, with reported quantities converted to full scale except wall-clock computation times.Contact speed leads comparisons because the model’s contact settings and friction are not scaled.Source: §3.1 · pp. 3–4
  • The robot model uses a UR10e-like six-axis arm, wrist force-torque sensing, and two fixed cameras to observe the pipe, head, and stick-up.The travelling-block height is available to the controller, as from a drawworks encoder.Source: §3.1 · p. 4

3.2 Guide head

  • The open 90° V head uses free rollers and swing clamps, arranged so the pipe touches only rollers and other parts remain at least 20 mm clear across all nine sizes.The head arrives from the side and can leave after stabbing without lifting over the pipe; a capture test reports a 0.8–1.6 cm envelope at the box.Source: §3.2 · pp. 4–5

4 Method

StabBot estimates the pipe’s swing and approaches at matched velocity, then uses roller contact, tilt compensation and compliant stabbing to seat it. A stereo keypoint network supplies 3-D landmarks, while a residual policy refines capture against defined baselines.

4.1 Swing estimation

  • A pendulum Kalman filter tracks pipe position, velocity and rest position, using swing frequency computed from the pipe tally and camera samples applied at capture time.The filter predicts pipe position at future times in closed form and absorbs camera delay.Source: §4.1 · pp. 5–6

4.2 Velocity-matched capture

  • The arm shadows the pipe position predicted 0.12 s ahead, then follows a minimum-jerk approach so the V reaches the pipe moving with it.The stand-off decreases from 12 cm to an 8-mm preload; clamps close after roller contact.Source: §4.2 · p. 6

4.3 Guarded final approach

  • If roller contact is not sensed after close-in, the V advances at 2 cm/s for up to 6 cm and stops when roller load exceeds 64 N.Touch-confirmed capture avoids leaving the V short of the pipe because of a small vision error.Source: §4.3 · p. 6

4.4 Tilt compensation

  • When the elevator hangs off-centre, the frozen rest-position estimate supplies its offset, and the head shifts the pipe so the pin nose sits over the box.Tilt compensation is applied when the estimated offset exceeds 3 cm.Source: §4.4 · p. 6

4.5 Centring, stab and release

  • After centring, the head follows the block down; wrist admittance within 2 cm of the box lets the pipe push the head sideways as the taper aligns.If block lag exceeds 2.4 cm, the job stops and is counted as failed; the clamps open when the pin is halfway in.Source: §4.5 · p. 7

4.6 Stereo keypoint network

  • The stereo keypoint network estimates the pipe axis, pin nose and box mouth in 3-D from rendered stereo pairs with exact labels.It uses an ImageNet-pretrained ResNet-18 and heatmaps, with the pipe point solved on its known plane and pin and box triangulated.Source: §4.6 · p. 7

4.7 Residual policy

  • A residual policy adds filtered V-target shifts and a clamp-timing signal to the controller, using the robot’s state and contact observations.PPO training runs for 600,000 decisions; the reported policy lowers peak force by a further 22% relative to the model-based controller it corrects.Source: §4.7 · p. 7

4.8 Baselines

  • The static guide waits at well centre, while reactive and swing-aware variants isolate the effects of prediction and the residual policy; all share the same head and downstream phases.Reactive+guarded adds the guarded approach to separate prediction from touch.Source: §4.8 · p. 7

5 Experimental Design

The experimental design compares methods on paired random jobs, defines seating and contact metrics, and tests statistical significance and robustness. A recorded example illustrates the job sequence.

Benchmark

  • Each of 200 paired jobs samples one of nine pipe sizes, a 5–25 cm swing at the pin, swing direction, phase, frequency error and camera-noise seed.A job succeeds when the pin is seated within the stated shoulder-gap and lateral-offset limits, with pipe contact restricted to the rollers and no robot contact.Source: §5 · pp. 7–8

Illustrative job

  • Fig. 4 shows a 5-in. drillpipe job with a 20-cm pin swing, from watching the swing through matched-speed contact, clamping, stabbing, release and retreat.The frames replay recorded physics in a studio setting.Source: Figure 4 · p. 8

Metrics

  • Contact speed measures relative horizontal pipe–head speed at first roller contact, while peak force covers capture and centring and time to seat ends at shoulder seating.Contact speed leads the comparison because it depends on the controller and swing rather than simulator contact stiffness.Source: §5 · p. 8

Statistics

  • Paired per-job differences use a two-sided Wilcoxon signed-rank test, while success rates use Wilson 95% intervals and median ratios use bootstrapped 95% intervals.Ablation p values receive a Holm correction.Source: §5 · p. 8

Robustness

  • Robustness tests add one error at a time across three levels, including up to 300 ms delay, 85% frame loss, 20 mm calibration offset and 35–40 cm swings.The keypoint network remains in the loop, and the added estimate noise is Gaussian.Source: §5 · p. 8

Unseen conditions

  • The study also renders 450 new scenes nominally and under 14 conditions unseen by the network, with every scene tested under every condition.Source: §5 · p. 8

6 Results

Across 200 paired jobs, StabBot seated every joint with lower contact speed and force than the fixed guide. Component tests, robustness and vision evaluations identify how the system performs and where its design was refined.

6.1 Main benchmark

  • 100% of joints were seated by StabBot versus 74% by the fixed guide; median contact speed fell from 18.4 to 2.3 cm/s and median peak force from 11.3 to 1.4 kN.StabBot seated all 200 jobs at every size and swing amplitude, and its peak force was lower in all 200 paired jobs.Source: §6.1 · p. 8, Figure 5 · p. 9, Table 4 · p. 9

6.2 Contribution of each component

  • Adding the guarded approach raised success from 93.0 to 97.5%; prediction with matched speed removed remaining failures, and residual RL further reduced peak force by 22%.The residual policy also reduced contact speed by 27% and saved 0.5 s per connection; the component comparisons used the same 200 jobs.Source: §6.2 · p. 10, Table 6 · p. 10

6.3 Robustness

  • The swing-aware methods seated 100% of jobs through 20 mm of added network noise, 300-ms delay, 85% frame loss, 20-mm calibration offset, 20-cm elevator offset and 35–40-cm swings.At 30 mm of added noise, success fell to 83–86%; StabBot's median peak force reached 7.8 kN with 20 mm of noise.Source: §6.3 · pp. 10–11, Figure 7 · p. 11

6.4 Off-centre elevator

  • With the elevator 20 cm off centre, tilt compensation raised swing-aware success from 43% to 100% and placed the pin 0.1–0.3 cm off the box.Without compensation, the pin arrived 1.4–1.5 cm off the box, outside the capture envelope for six of nine sizes.Source: §6.4 · p. 11, Figure 8 · p. 11

6.5 Vision

  • Across 2,000 held-out scenes, the network’s median 3-D error was 0.4 mm for the pipe at head height; one stereo pair took 2.3 ms on a laptop RTX 4060.Source: §6.5 · p. 12
  • Training the final network with varied shapes, glare and blur reduced glare error to 0.5 mm and occluded median error to 0.9 mm.A later version trained on different pipe paint colours reduced yellow-pipe error to 0.7/1.5 mm median/p95 and retained 100% success on the 200-job benchmark.Source: §6.5 · p. 12, Figure 9 · p. 13

6.6 Design lessons

  • The initial residual policy lowered contact speed by 18% but raised clamp-landing force by 22% after learning to close the clamps early.Smoothing its action, extending the episode through centring and charging peak force addressed this reward-design failure; touch-confirmed closure also fixed missed captures with the network.Source: §6.6 · p. 12

7 Core Analysis

The paired benchmark links matched-speed capture to lower impact and fewer failed stabs, while the analysis weighs that benefit against seat time and describes where the approach may fit.

7.1 Why the stab matters

  • Every connection begins with a joint swinging from the elevator, and the stab is a step where people and equipment meet a moving load.The paper describes floorhands steadying the pipe by hand in the red zone while the pin, box and guide take the impact.Source: §7.1 · p. 13

7.2 Safety and equipment loading

  • StabBot’s peak head–pipe force was lower by a factor of 7.8 at the median and 9.9 at the 95th percentile, and lower in every job.Relative kinetic energy at first contact was a median 95 times lower; wrist load fell from 8.1 to 2.8 kN, and no one had to steady the pipe by hand.Source: §7.2 · p. 13, Table 8 · p. 13

7.3 Rig time

  • StabBot becomes faster on average when a re-stab takes more than about 18 s; at 60 s, it saves 4.4 s per connection.On the 148 jobs both methods seated cleanly, median time to seat was 12.3 s for the fixed guide and 14.2 s for StabBot.Source: §7.3 · p. 14

7.4 Novelty and contributions

  • The study reports the first system, to the authors’ knowledge, that estimates a hanging joint’s free swing online and intercepts it at matched velocity for stabbing.The comparison is against prior systems that wait at well centre or guide a stand they already hold.Source: §7.4 · p. 14, Table 9 · p. 14

7.5 Where StabBot fits

  • StabBot requires a drill-floor arm, pipe tally, drawworks height, two fixed cameras with a GPU, a wrist force sensor, head and software.The paper identifies the arm, pipe tally and block height as already available on drill-floor robots or rigs, while the other listed components would be added.Source: §7.5 · p. 14
  • StabBot seated every tested 20-in. casing joint, where the fixed guide seated 57%; the same filter could assist manual rigs by showing predicted swing.The paper also identifies other suspended-load tasks as potential applications, with the head and tuning adapted to each load.Source: §7.5 · p. 14

8 Scope and Next Steps

The study is a full-scale-equivalent simulation of capture, centring and stab-in; make-up, real images and rig-specific effects such as elevator links remain for a next stage.

8 Scope and Next Steps

  • The study models capture, centring and stab-in, while make-up, real images and rig-specific effects such as elevator links are left to the next stage.That stage is described as a test stand with a real stereo camera and instrumented pipe, network fine-tuning on labelled real images, and a force-based check of stab quality before make-up.Source: §8 · p. 15

9 Conclusion

StabBot seated all 200 jobs across nine pipe sizes, while reducing contact speed and force relative to a fixed guide. Its keypoint network and component tests support the system’s performance and robustness.

9 Conclusion

  • 100% of joints seated with StabBot versus 74% with a fixed guide; median contact speed and median peak force both fell by a factor of 7.8.Peak force was lower in every job.Source: §9 · p. 15
  • Relative kinetic energy at first contact was a median 95 times lower per job, and 95th-percentile force fell by a factor of 9.9.Peak force was lower in every job.Source: §9 · p. 15
  • StabBot removed the missed captures and steel hits that ended one fixed-guide job in four, and took the floorhand’s hand off the pipe during the stab.Source: §9 · p. 15
  • The guarded approach made capture reliable, prediction with velocity matching made it gentle, and residual learning lowered peak force by a further 22%.Source: §9 · p. 15
  • With the stereo keypoint network in the loop, success stayed at 100% under added noise, camera delay, frame loss, calibration and frequency errors, elevator offset, and swings up to 40 cm.Source: §9 · p. 15
  • The network located the pipe, pin, and box with sub-millimetre median error in 2.3 ms per stereo pair.Source: §9 · p. 15
  • One open V head on free rollers served all nine pipe sizes, and the method carries over to other hoist-hung loads brought to a fixed target.Source: §9 · p. 15

Abbreviations

The abbreviations cover spatial dimensions, hardware and sensing, simulation, and learning methods.

Abbreviations

  • 3-D means three-dimensional; CI means confidence interval; CNN means convolutional neural network.Source: Abbreviations · p. 15
  • F/T means force–torque sensor; GPU means graphics processing unit; MuJoCo means Multi-Joint dynamics with Contact.Source: Abbreviations · p. 15

Nomenclature

The nomenclature defines variables for the pipe’s swing and position, the robot’s target and approach, and the reported capture metrics.

Nomenclature

  • The notation includes the rest position, peak head–pipe force, filter step, approach direction, elevator offset, pipe position and velocity, and predicted pipe position.Source: Nomenclature · p. 15
  • The remaining notation defines job shares and reward, stand-off, time measures, re-stab time, contact speed, and the Kalman filter state.Source: Nomenclature · p. 15
  • The target position, component heights, filter-step interval, event indicators, PPO parameters, heatmap width, prediction look-ahead, and swing frequency are also defined.Source: Nomenclature · p. 16

Statements and Declarations

The authors report no funding or competing interests. The study is a computer simulation, and data, code, and author contributions are documented.

Statements and Declarations

  • The manuscript reports that no funding was received.Source: Statements and Declarations · p. 16
  • The authors declare no known competing financial interests or personal relationships that could have influenced the work.Source: Statements and Declarations · p. 16
  • Ethics approval and consent to participate were not applicable because the study is a computer simulation without human or animal subjects or personal data.Source: Statements and Declarations · p. 16
  • The article contains no data relating to an identifiable individual, so consent for publication was not applicable.Source: Statements and Declarations · p. 16
  • Benchmark and analysis files will be made public on acceptance, and the authors list the corresponding contributions to study design, software, analysis, and writing.The data and files are available to reviewers on request; simulation code, trained networks, and the residual policy will be released with the data.Source: Statements and Declarations · p. 16

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

The authors used Claude, directed by the first author, for coding, drafting, and language review, then reviewed and edited its output. They take responsibility for the publication and state that all results, numbers, and figures come from their simulations.

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

Related papers

Loading 2610.37535v1…