Source-linked AI summary
Visual Closed-Loop Control for Pouring Liquids
Connor Schenck, Dieter Fox
TL;DR
Robots had not robustly poured specified amounts of actual liquid using visual feedback. This paper estimates volume from visual liquid detections and uses the estimates in closed-loop control; the robot missed the target by 38 ml on average.
Problem
Robustly pouring a specified amount of actual liquid had not been addressed, despite prior work on simulated pouring and liquid detection.
Source: §I · p. 1Method
The system detects liquid pixels in color images, estimates target-container volume with model-based or model-free methods, and feeds estimates to a PID controller.
Source: §I · p. 1, C. Volume Estimation During Pouring… · p. 2, §V · p. 8Results
38 ml average miss from the target; color-only estimates were slightly better than model-based estimates using thermal imagery.
Source: §V · p. 8Takeaways & Limitations
The approach was suited to similar pouring tasks in standard home environments, though not accurate enough for some industrial settings.
Source: §V · p. 8The evaluation concerns relatively controlled conditions, including training on the target containers.
Source: §I · p. 1
Abstract
from arXiv · showhide
Pouring a specific amount of liquid is a challenging task. In this paper we develop methods for robots to use visual feedback to perform closed-loop control for pouring liquids. We propose both a model-based and a model-free method utilizing deep learning for estimating the volume of liquid in a container. Our results show that the model-free method is better able to estimate the volume. We combine this with a simple PID controller to pour specific amounts of liquid, and show that the robot is able to achieve an average 38ml deviation from the target amount. To our knowledge, this is the first use of raw visual feedback to pour liquids in robotics.
I. INTRODUCTION
The paper addresses robotic pouring of a specified amount of real liquid, a task not addressed by prior work described here. It presents visual volume estimation for closed-loop control without specialized sensing, while evaluating under relatively controlled conditions.
I. INTRODUCTION
- Prior work had not addressed pouring specified amounts of actual liquid, despite progress in simulated pouring and liquid detection.Robust control and detecting liquid during pouring are both required; people often use approximate requests such as “a half cup.”Source: §I · p. 1
- The framework uses visual feedback to estimate liquid volume and control pouring without specialized force sensors, scales, or liquid-detection sensors.The investigation assumes relatively controlled conditions, including the ability to train on the target containers.Source: §I · p. 1
- A pixel-detection network feeds a volume-estimation network, whose output provides real-time feedback to a PID controller pouring a desired amount.The paper also proposes model-based volume estimation from pixel-wise liquid detections.Source: §I · p. 1
- Thermal imagery calibrated with RGB-D data provides automatic water-pixel labels, and Baxter experiments covered two mugs and a bowl.The experiments indicate the trained networks provide volume estimates sufficiently accurate for the pouring task.Source: §I · p. 1
II. RELATED WORK
Prior robotic pouring often uses simulation or constrained settings and commonly empties the source, whereas this work targets a limited amount with an unknown initial source volume. Its visual pipeline detects liquid pixels and uses them to estimate volume and control pouring.
II. RELATED WORK
- Earlier robotic-pouring work often used coarse simulation or constrained tasks, and usually emptied the source rather than pouring a specified amount.Force sensing and scales had also been used to estimate poured volume, but the paper notes limitations including specialized sensors and measurement delay.Source: §II · p. 2
B. Pixel-Wise Liquid Detection
- Optical flow is poorly suited to detecting standing water, so the paper uses an LSTM-based fully-convolutional network to label liquid pixels from images.Its output is a heatmap with values from 0 to 1 indicating likelihood of liquid.Source: §II · p. 2, B. Pixel-Wise Liquid Detection · p. 2
A. Task Overview
- The task is to pour only a limited amount from a source with unknown initial volume, requiring continuous visual estimation of liquid in the target.The paper separates detection, volume estimation, and control so each component can be trained and evaluated individually.Source: A. Task Overview · p. 2
B. Pixel-Wise Liquid Detection
- Thermal imagery with heated water supplies pixel-label ground truth, while the thermal camera is expensive and must be registered to RGB-D data.The authors contrast this with prior simulated-data labeling from color images.Source: B. Pixel-Wise Liquid Detection · p. 2
C. Volume Estimation During Pouring Sequences
The paper compares model-based and model-free volume estimation from liquid detections, then filters estimates over time with an HMM. The posterior distribution is summarized by its median for volume estimation during execution.
C. Volume Estimation During Pouring Sequences
- The model-based estimator uses a 3D container model to infer liquid height, while the model-free estimator regresses volume from labeled pixels.The model-based observation compares detected liquid pixels with those expected at a candidate volume.Source: C. Volume Estimation During Pouring… · p. 2, Figure 3 · p. 3
1) Filtering using a HMM:
- An HMM filters volume estimates over time, treating liquid volume as a hidden state and the current observation as evidence.The transition probability is inferred from training data.Source: 1) Filtering using a HMM · p. 3
- The posterior volume distribution combines the observation likelihood with the transition probability and uses a fixed-range histogram representation.The transition probability sums over histogram bins.Source: 1) Filtering using a HMM · p. 3
- During execution, the estimated volume is the median of the posterior distribution, while observation probabilities differ between the two estimation methods.The multi-frame network output is passed through an HMM before the volume estimate reaches the PID controller.Source: 1) Filtering using a HMM · p. 3, Figure 3 · p. 3
2) Model-Based Volume Estimation:
The model-based estimator compares observed pixel-wise liquid labels with the labels predicted for candidate volumes using a 3D container model. It combines per-pixel observations to estimate volume, considering only pixels that can see inside the container.
2) Model-Based Volume Estimation:
- The method assumes a 3D container model and uses the RGB-D point cloud to determine its pose before comparing observed labels with predictions for candidate volumes.The observed labels are computed from the camera image.Source: 2) Model-Based Volume Estimation · p. 3
- The likelihood calculation assumes conditional independence between pixels and uses only pixels viewing the container interior.Pixels outside the container’s interior view do not contribute to the observation probability.Source: 2) Model-Based Volume Estimation · pp. 3–4
- The authors report that the median produced more stable, less skewed estimates than other tested methods, although differences were minor.The alternatives included expectation and maximum likelihood.Source: 2) Model-Based Volume Estimation · p. 3
- Each candidate volume predicts liquid or not-liquid labels from a level surface at its corresponding height; predicted labels are compared with observed pixel labels.The liquid is assumed to be at rest, with a surface parallel to the ground.Source: 2) Model-Based Volume Estimation · p. 4
- Binary search with the signed tetrahedron volume method finds the surface height corresponding to each candidate volume; heights can be precomputed for the fixed volume bins.The volume below a proposed height is computed from the container mesh.Source: 2) Model-Based Volume Estimation · p. 4
3) Model-Free Volume Estimation:
The model-free estimator replaces object-pose inference with a neural network that estimates volume from pixel labels. The authors evaluate single-frame, multi-frame, and recurrent CNN architectures.
3) Model-Free Volume Estimation:
- The model-free method replaces the model-based method’s object-pose inference with a network that maps pixel labels to a volume estimate.The pixel-label heatmap comes from the liquid-detection network; color or depth images can also be appended as input channels.Source: 3) Model-Free Volume Estimation · p. 4
- The network’s output is a distribution over estimated volumes, used as the observation in the HMM, with conditional probabilities inferred from training data.The network computes the estimate distribution from its observation.Source: 3) Model-Free Volume Estimation · p. 4
- The evaluated architectures were a single-frame CNN, a multi-frame CNN, and a recurrent LSTM CNN.The multi-frame CNN processes sequential images, while the LSTM-CNN replaces the single-frame network’s first fully connected layer with an LSTM layer.Source: 3) Model-Free Volume Estimation · p. 4
D. Robot Controller
The robot pours by controlling its wrist angle with a modified PID controller whose error signal is the difference between target and current volume. It begins from a preset tilt and stops when the target volume is reached.
D. Robot Controller
- The controller uses the difference between target and current target-container volume as its error signal to set wrist angular velocity.The wrist joint directly controls the source container’s angle.Source: D. Robot Controller · p. 5
- The robot first tilts the source container to 75 degrees from vertical before starting the controller.Source: D. Robot Controller · p. 5
- Because pouring is non-reversible, the integral term is omitted; when the target volume is reached, the controller stops and the source container returns to vertical.Source: D. Robot Controller · p. 5
E. Implementation Details
The implementation locates the target container using an RGB-D point cloud and generates liquid-pixel labels by registering thermal images to color images. The labeling uses heated water and temperature thresholding.
1) Finding the Container in the Scene:
- The robot’s RGB-D point cloud is used to find the table plane and cluster points above it; iterative closest points then estimates the modeled container’s pose.That pose supports labeling image pixels as inside, outside, or neither for the model-based method.Source: 1) Finding the Container in the Scene · p. 5
2) Generating Ground Truth Pixel Labels:
- Heated water at approximately 93°C provides thermal ground truth, registered to color images using checkerboard correspondences and an affine transformation.A checkerboard on an aluminum sheet is made visible in the thermal image by a spotlight.Source: 2) Generating Ground Truth Pixel Labels · p. 5
- An adaptive threshold based on the target-container pixels’ average temperature produces binary liquid/not-liquid labels; the authors report minimal false positives.Water does not appear in the depth image, and the fixed pour depth motivates the simpler registration approach.Source: 2) Generating Ground Truth Pixel Labels · p. 5
- The controller is called PID for readability but is technically a PD controller.Source: 2) Generating Ground Truth Pixel Labels · p. 5
3) Acquiring Ground Truth Volume Estimates for Training and Evaluation:
The authors create hindsight-based volume estimates to serve as ground truth for training and evaluation. They collect synchronized thermal and RGB-D pouring data on a Baxter robot using three target containers.
3) Acquiring Ground Truth Volume Estimates for Training and Evaluation:
- The thermal camera and model-based estimator provide baseline volumes; hindsight can improve estimates, though the authors do not claim they are perfect.The authors consider the combined estimates robust enough for training and evaluation.Source: 3) Acquiring Ground Truth Volume… · p. 5
- Viterbi decoding selects the most probable volume-state sequence using observations from the full pour, producing ground-truth estimates for training and evaluation.It replaces the forward HMM summation with a maximum and backtracks from the most probable final state.Source: 3) Acquiring Ground Truth Volume… · p. 5
A. Robotic Platform
- Experiments used Baxter’s right arm exclusively; its upper-torso Asus Xtion produced RGB and depth images at 640×480 and 30Hz, and an immediately-above thermal camera output 640×512 images at 30Hz.Source: A. Robotic Platform · p. 5
B. Experimental Setup
- Researchers collected 279 pouring sequences in which the robot attempted to pour 250 ml of water into three target containers using the thermal-camera model-based method, with initial cup volumes of 300, 350, or 400 ml.Each sequence lasted exactly 25 seconds and was recorded on thermal and RGB-D cameras at 30 Hz; the data were randomly divided 75%-25% into training and evaluation sets.Source: B. Experimental Setup · p. 5, Figure 5 · p. 6
C. Validating Thermographic Ground Truth Methodology
The section validates thermographic ground-truth estimates and evaluates the liquid-detection network. Thermal estimates are calibrated per container for the baseline, while the trained detector performs well on the paper’s tasks but is not expected to generalize beyond them.
C. Validating Thermographic Ground Truth Methodology
- Thermal-camera model-based estimates overstate static scale readings for all three containers, so the authors fit a separate linear calibration for each container’s ground-truth baseline.This calibration is not applied to the model-based method being evaluated, because a robot could not pre-calibrate every household object.Source: C. Validating Thermographic Ground… · p. 6
D. Evaluating the Detection Network
- The authors trained a recurrent LSTM CNN on real data to classify liquid versus non-liquid pixels from color images, addressing poor real-imagery performance of prior simulator-trained networks.Training used Adam for 61,000 iterations.Source: D. Evaluating the Detection Network · p. 6
- Figure 6 reports test-set RMSE in milliliters for model-based estimates and model-free networks using detections alone or combined with color or depth.Source: Figure 6 · p. 6
- The detection network achieves high pixel-wise precision and recall, but the authors expect it to work only on the tasks described because the training set is relatively small.Source: D. Evaluating the Detection Network · p. 6
E. Comparing Methods for Volume Estimation
The multi-frame model-free estimator using liquid detections alone outperforms the model-based methods in volume estimation, and it supports closed-loop pours with an average error of 38 ml. The authors also identify feedback instability as a reason to test estimators in real-time control rather than relying only on post-pour accuracy.
E. Comparing Methods for Volume Estimation
- The model-free networks use 160×160 crops and classify volume into 100 values from 0 to 400 ml; pretraining on actively changing portions of sequences before full-sequence training improved results.The network inputs were liquid detections alone, detections with color, or detections with depth.Source: E. Comparing Methods for Volume… · pp. 6–7
- The hindsight-based baseline can smooth splash-induced volume spikes by propagating information backward, unlike the forward-looking estimate needed for control.Source: E. Comparing Methods for Volume… · p. 7
- Some model-free methods estimate volume better than the model-based methods, with the multi-frame network using only liquid-pixel detections performing best.The authors note that detections already provide much of the temporal and perceptual information that color or depth inputs would add.Source: E. Comparing Methods for Volume… · p. 7
- On selected test sequences, the multi-frame detection-only network matches baseline estimates better than either model-based method and requires neither a thermal camera nor a target-container model.Source: E. Comparing Methods for Volume… · p. 7
- 38 ml was the average error across 30 robot pours, and almost all pours were within 50 ml of their targets.Performance was approximately the same across the three target containers, although the estimator was not given target-container information.Source: Figure 8 · p. 7, E. Comparing Methods for Volume… · p. 8
- An erroneous estimate during pouring can destabilize the estimator-controller feedback loop, so post-pour accuracy alone does not establish suitability for control.Source: E. Comparing Methods for Volume… · p. 7
- The real-robot evaluation used 30 pours across three target containers, with targets from 100 to 300 ml and starting source volumes of 300, 350, or 400 ml.Each pour lasted 25 seconds, and final target-container volume was measured with a scale.Source: E. Comparing Methods for Volume… · p. 8
V. CONCLUSION AND FUTURE WORK
The paper presents visual closed-loop pouring with real-time liquid-volume estimation. Experiments report better color-only estimates than model-based estimates and an average 38ml miss from the target; robustness and broader task generalization remain future directions.
V. CONCLUSION AND FUTURE WORK
- The framework detects water in individual color-image pixels, estimates volume from those detections, and uses heated water with calibrated RGB-D and thermal cameras to generate training labels.The volume estimate feeds visual closed-loop control for pouring specific amounts.Source: §V · p. 8
- The color-only deep network slightly outperformed thermal-image model-based estimates, and PID pouring missed the target by 38ml on average.The approach is suited to similar home pouring tasks, but the reported accuracy is not sufficient for some industrial settings.Source: §V · p. 8
- Improving neural-network robustness and generalizing to other liquids, relative target amounts, and more sophisticated control are identified as future directions.Examples of other liquids include soda and coffee.Source: §V · p. 8