FlowControl: Optical Flow Based Visual Servoing

I. INTRODUCTION

• Robot programming requires domain-specific expertise, and converting high-dimensional video into actionable commands robust to environmental and task variation is challenging.The cited challenges span identifying salient objects, establishing correspondences, and controlling the robot to reproduce demonstrated motion.Source: §I · p. 1
• A single demonstration can support robust task replication despite substantial variation in objects’ initial positions, orientations, and appearances.The authors clarify that this is not machine-learning optimization of weights.Source: §I · p. 1
• FlowControl uses optical-flow correspondences and a demonstration foreground mask to align live observations with demonstration frames, then repeats alignment to track a trajectory.The approach uses learned optical flow with RGB-D data and fits a rigid transformation between the live observation and demonstration.Source: §I · p. 1, Figure 1 · p. 1
• The approach requires neither CAD models nor expensive pretraining in elaborate simulation environments.Source: §I · p. 1
• FlowControl is reported to handle varied tasks, including object picking and insertion, with robustness to variation, data efficiency, and high success rates.Source: §I · p. 1

II. RELATED WORK

• Imitation approaches range from kinesthetic teaching and motion primitives to image embeddings [4], and demonstration-conditioned policy fine-tuning; FlowControl needs no learned high-dimensional encoding [28].Source: §II · p. 2
• Visual servoing feeds sensor data into control commands that change sensor measurements,, enabling feedback-based adaptation to environmental variation.Source: §II · p. 2
• FlowControl applies visual servoing to manipulation imitation and uses robust correspondences to generalize across scene geometry, lighting conditions, and object appearance.Source: §II · p. 2
• Prior work used optical flow for industrial visual servoing and as an intermediate representation for character navigation,, while other learned approaches combine visual servoing with policy or feature learning.Source: §II · p. 2
• Good performance of recent learning-based optical flow has renewed application interest; these methods generalize robustly to common appearance changes, while applying optical flow beyond consecutive frames increases large-displacement and out-of-frame occlusion incidence.Source: §II · p. 2

• After aligning the first frame, FlowControl repeats the procedure across subsequent frames to track a complete demonstration trajectory, including multi-step tasks.Source: §III · p. 2
• An end-of-arm camera simplifies conversion of image transformations to the end-effector frame, while first-person demonstrations avoid changing perspective.The learned optical flow is also described as robust to appearance variation.Source: §III · pp. 2–3
• The large number of 3D correspondences makes FrameAlign robust to noisy depth measurements and incomplete foreground masks.Source: Figure 2 · p. 3
• A manually supplied foreground mask selects the object to align; it need not be precise, because under-segmentation suffices.Manual provision trades algorithmic complexity for an intuitive practical step.Source: §III · p. 3
• FlowNet2 [33] supplies correspondences and is described as robust to lighting changes, partial occlusions, and textureless objects.The framework can also use other correspondence methods, including one that requires no learning.Source: §III · p. 3
• Masked optical-flow correspondences and RGB-D data yield 3D point matches, from which SVD estimates a rigid transform that drives the robot toward the demonstration frame.A position-based controller converts the estimated transform into a control signal.Source: §III · pp. 2–3

RGB, OL

• The procedure applies the warped foreground mask to depth, unprojects the selected points into 3D, and fits a rigid transformation between demonstration and live point clouds.Source: RGB, OL · p. 3
• Sequence tracking advances to the next target when the live image is sufficiently close, and sampling every nth demonstration frame can accelerate tracking.Source: RGB, OL · p. 3
• Gripper state and other orthogonal dimensions can be copied from the corresponding demonstration step while the remaining dimensions are aligned.Source: RGB, OL · pp. 3–4
• The tracker delays advancing to the next frame to account for the time required for the gripper to close.Source: RGB, OL · p. 4
• Switching the foreground object combines subtasks, such as grasping a wheel and then focusing on a screw to connect them.Source: RGB, OL · p. 4

• The evaluation covers four manipulation tasks and a localization experiment, with additional tests of camera pose estimation and robustness to scene geometry and appearance variations.Source: §IV · p. 4
• The setup uses a KUKA iiwa arm, a two-finger gripper, and an end-effector-mounted RGB-D camera for depth-based geometric fitting.Source: §IV · p. 4
• Observations are 640×480 RGB-D images plus proprioceptive measurements of gripper height, rotation angle, and finger width.Source: §IV · p. 4

A. Manipulation Experiments

• The experiments cover block grasping and pick-and-stow, shape-sorter insertion, and grasping and inserting a wheel onto a screw.An insertion error of 4 mm is enough to make the insertion tasks fail.Source: A. Manipulation Experiments · pp. 4–5
• FlowControl achieves high success rates and outperforms ACGD on manipulation tasks.The success rates are reported for different manipulation tasks in Table I.Source: Table I · p. 5, A. Manipulation Experiments · p. 5
• Occlusions caused failures, including when the pick-and-stow box occluded the cube.Source: A. Manipulation Experiments · p. 5
• Starting too far from a demonstration target can exceed FlowNet2’s displacement range, with large rotations especially problematic.Starting farther from objects can make larger displacements appear smaller in image space.Source: A. Manipulation Experiments · p. 5
• Confusing background flow can cause optical-flow errors that high controller velocities may turn into task failures.Smaller velocities allow errors to be corrected, reducing the chance of failure.Source: A. Manipulation Experiments · p. 5
• Low illumination with fixed exposure and grasps that snap an object out of view also produced practical failures.Source: A. Manipulation Experiments · p. 5

B. Navigation Experiment

• The task positions a nut-runner precisely to engage bolts fastening an Engine Control Unit.Source: B. Navigation Experiment · p. 5, Figure 4 · p. 5
• ±1 mm precision was measured across five trials with starting positions up to 8 cm from the target.The reference image was taken under different lighting.Source: B. Navigation Experiment · p. 5
• Randomized end-effector starts in the camera plane tested return to a reference position using the robot’s state estimation.Source: B. Navigation Experiment · p. 5

C. Fitting Experiment

• The proxy evaluation estimates relative camera pose from pre-recorded views of a static object, with visual markers providing reference poses.Source: C. Fitting Experiment · p. 5
• To limit background influence, computed features were masked with the demonstration segmentation, except for monolithic DeepTAM.Source: C. Fitting Experiment · p. 5
• FlowControl’s flow-based approach performs best for most samples, though it has outliers when flow computation fails.Figure 5 plots the percent of samples below each error threshold against that threshold.Source: Figure 5 · p. 6, C. Fitting Experiment · p. 6
• SIFT failed on most views and performed worse than zero pose change; DeepTAM was only slightly better than zero rotation and often underestimated changes.Source: C. Fitting Experiment · p. 6
• Flow failures usually occurred with a combination of large displacements and rotations.Figure 6 illustrates erroneous optical flow under large rotation and limited frame overlap.Source: C. Fitting Experiment · p. 6, Figure 6 · p. 6

D. Generalization Experiments

• FlowControl copes with variation in both color and shape when a grasping demonstration is tested on different objects.Source: D. Generalization Experiments · p. 6
• The generalization experiments were limited to grasping, using a demonstration recorded with one object and tests on objects of different shapes and sizes.Source: D. Generalization Experiments · p. 6
• FlowNet’s learned invariance to lighting changes and partial occlusion helps it find correspondences when demonstrated objects do not match exactly.Source: D. Generalization Experiments · p. 6

V. DISCUSSION AND CONCLUSION

• FlowControl is a practical, data-efficient method for visual servoing from optical flow.Source: §V · p. 6
• Experiments demonstrate robotic manipulation across substantial geometric and visual variation, including grasp transfer to objects different from the demonstration object.The generalization experiments focus on grasping.Source: §V · p. 6, Figure 7 · p. 6
• FlowControl cannot yet re-grasp, relies on manual task segmentation, and can fail when optical flow encounters large displacements.Source: §V · p. 6
• With little manual effort, FlowControl solves a diverse set of tasks.Source: §V · p. 6