PaperlayerView exact revision on arXiv

Source-linked AI summary

Making Sense of Audio Vibration for Liquid Height Estimation in Robotic Pouring

Hongzhuo Liang, Shuang Li, Xiaojian Ma, Norman Hendrich, Timo Gerkmann, Fuchun Sun, Jianwei Zhang

arXiv:1903.00650v2cs.ROcs.SDeess.AS

TL;DR

Robotic pouring needs reliable liquid-height perception, while visual and haptic approaches have limitations. This paper uses audio vibration and a recurrent network, PouringNet, to estimate height from pouring audio. Evaluations report accurate predictions and performance across containers and human-to-robot settings.

  • Problem

    Robotic-pouring height perception remains challenging because existing visual and haptic approaches have limitations, while methods must generalize across containers and liquid conditions and provide accurate estimates.

    Source: §I · p. 1

  • Method

    PouringNet maps audio fragments to current liquid height with a recurrent network, trained using a multimodal dataset of more than 3,000 human-pouring records.

    Source: §I · pp. 1–2

  • Results

    Recurrent models achieved 90% accuracy below an absolute 2 mm length error and absolute mean length errors below 1.5 mm, outperforming the AudioFC baseline.

    Source: A. PouringNet Evaluation · p. 4

  • Takeaways & Limitations

    Audio vibration provides useful liquid-height information across target containers and substantially different human and robot pouring setups.

    Source: 1) Evaluation of Different Target… · p. 5

Abstract

from arXiv · show

In this paper, we focus on the challenging perception problem in robotic pouring. Most of the existing approaches either leverage visual or haptic information. However, these techniques may suffer from poor generalization performances on opaque containers or concerning measuring precision. To tackle these drawbacks, we propose to make use of audio vibration sensing and design a deep neural network PouringNet to predict the liquid height from the audio fragment during the robotic pouring task. PouringNet is trained on our collected real-world pouring dataset with multimodal sensing data, which contains more than 3000 recordings of audio, force feedback, video and trajectory data of the human hand that performs the pouring task. Each record represents a complete pouring procedure. We conduct several evaluations on PouringNet with our dataset and robotic hardware. The results demonstrate that our PouringNet generalizes well across different liquid containers, positions of the audio receiver, initial liquid heights and types of liquid, and facilitates a more robust and accurate audio-based perception for robotic pouring.

I. INTRODUCTION

Robotic pouring needs accurate liquid-height perception, but visual and haptic approaches have limitations, and robust generalization and precision remain challenges. The paper proposes audio-based PouringNet and a multimodal dataset to address this problem.

I. INTRODUCTION

  • Robotic pouring requires accurate height perception, while visual methods may fail on opaque containers or estimate poorly and haptic correlations vary across end effectors and containers.The section identifies generalization across containers, liquid types, and liquid status, as well as estimation precision, as the two central challenges.Source: §I · p. 1
  • Audio vibration offers a height cue because the container’s air-column vibration frequency changes as liquid rises, allowing direct height estimation without explicit integration.The paper says avoiding integration further reduces prediction bias and achieves more accurate results.Source: §I · p. 1
  • PouringNet uses a recurrent neural network to map audio fragments to current liquid height, with recurrence encoding the monotonic rise of liquid during pouring.The authors say the recurrent structure may reduce noise and produce smoother predictions.Source: §I · pp. 1–2
  • The authors collected more than 3,000 multimodal human-pouring records containing audio, force and torque feedback, video, and motion trajectories.The dataset is presented as a resource for the perception task in robotic pouring.Source: §I · p. 2
  • Dataset and robotic-hardware experiments demonstrate robust and accurate audio-based perception for robotic pouring.The paper presents this as the method’s evaluated contribution, not as a universal guarantee.Source: §I · p. 2

II. RELATED WORK

Prior pouring perception work uses visual, haptic, or other audio cues, but the reviewed approaches do not use target-container air vibrations to estimate filling height. This paper builds on the link between resonance frequency and air-column length.

Visual sensing for robotic pouring

  • Visual pouring methods depend on lighting, liquid color, and container shape; cited approaches also report height errors above 4 mm for ten pours of three transparent liquids or error from varied liquids and flow shapes [2].Another approach infers input volume from water-flow imagery rather than predicting absolute liquid height.Source: §II · p. 2

Haptic sensing for robotic pouring

  • Haptic sensing supports trajectory or policy learning and liquid-property estimation, but force cannot measure liquid height in an unseen target container,,.The review distinguishes force-based volume cues from direct filling-height perception.Source: §II · p. 2

Audio sensing for robotic pouring

  • Prior audio studies use auditory cues for interaction classification, poured amount, or granular-material weight,,, but not target-container air vibration for liquid-height regression.The proposed approach instead uses the relation between resonance frequency and air-column length to seek generalization across situations.Source: §II · p. 2

III. DATA PREPARATION

The authors collected human pouring demonstrations with multiple sensing modalities and measured liquid height indirectly using a scale. The setup spans three target containers, while its human collection setting also presents a transfer challenge for robotic use.

A. Multi-modal Pouring Dataset

  • The dataset pairs audio and scale-derived liquid-height labels with video, human-hand trajectories, and source-container force-torque feedback.Human collection was comparatively low-cost, but differing trajectories and loud background noise challenge transfer to robotic pouring.Source: A. Multi-modal Pouring Dataset · pp. 2–3
  • The setup records three target containers—glass, thermos, and mug—with a 44.1 kHz microphone, 500 Hz force-torque sensor, 1 Hz scale, 30 Hz camera, and 240 Hz motion tracking system.The source container was positioned 250 mm horizontally and 750 mm vertically from the microphone’s bottom center.Source: A. Multi-modal Pouring Dataset · p. 3, Figure 2 · p. 3
  • Training involved water and 1,000 trials for each of three target containers, collected from two subjects in recordings lasting 4–11 seconds.The authors note that the multimodal recordings could support future perception methods using sensor fusion.Source: A. Multi-modal Pouring Dataset · p. 3

B. Data Analysis

The data analysis identifies a rising air-resonance frequency as a cue associated with increasing liquid height, while the container’s own resonance is not clearly visible. Spectrograms illustrate pours across target containers and source-container conditions, and scale measurements are used to derive height labels.

B. Data Analysis

  • The spectrogram examples cover pours into a glass and mug, and into a thermos from source containers with and without a spout.The bottom examples show different pouring speeds without changing the main air-resonance pattern.Source: Figure 3 · p. 3, B. Data Analysis · p. 3
  • As the liquid rises, the air-column resonance frequency increases, while the container’s slight resonance downtrend is not clearly visible in the dataset.The analysis attributes the rising air-resonance curve to liquid height and reports that different pouring speeds do not change this pattern.Source: B. Data Analysis · p. 3
  • Scale weights are interpolated over time and converted to liquid-height labels using a quadratic polynomial fitted from 15 weight–height pairs per target container.The resulting height data is a continuous 257 × 1 series.Source: B. Data Analysis · p. 3

IV. POURING NETWORK

PouringNet estimates liquid height from sequential audio by predicting the target container’s air-column length, a ground truth intended to generalize across container shapes. Its recurrent architecture is trained with height and monotonicity losses.

IV. POURING NETWORK

  • PouringNet uses a recurrent network because pouring audio is sequential and variable-length, and the model learns real-time liquid height from that sequence.The authors also motivate recurrence by audio’s sequential properties and its potential to produce smoother predictions by encoding monotonically rising liquid level.Source: §IV · pp. 3–4
  • Predicting air-column length rather than liquid height gives different container shapes and filling heights a shared label when their air columns match.The paper argues that containers with equal air-column lengths have similar resonance frequencies, whereas liquid-height labels could assign different targets to similar audio features.Source: §IV · p. 4, Figure 4 · p. 4
  • Training clips are randomly sampled as 4-second fragments, converted to 257 × 251 spectrogram slices, and processed by a one-layer LSTM or GRU encoder and a two-layer MLP predictor.The predictor regresses temporary air-column length under mean squared error supervision.Source: §IV · p. 4
  • An auxiliary monotonicity loss penalizes predictions when estimated air-column length fails to decrease over time as liquid level rises.This encodes the expected monotonic increase in liquid height during pouring.Source: §IV · p. 4
  • The network transforms raw audio into spectrogram slices, feeds each slice to a recurrent encoder, and predicts the target container’s air-column length.The architecture figure specifies 257 spectrogram descriptors and a height-predictor module.Source: Figure 5 · p. 4, §IV · p. 4
  • The complete audio objective combines height-prediction and monotonicity losses, with α set to 0.01 to balance them.The paper reports choosing this value through preliminary experiments for best performance.Source: §IV · p. 4

A. PouringNet Evaluation

Recurrent PouringNet architectures outperform the AudioFC baseline on the human-collected test set, with AudioLSTM slightly ahead of AudioGRU. Training on all container types achieves the best accuracy among the tested AudioLSTM models.

A. PouringNet Evaluation

  • Both recurrent models achieve 90% accuracy below an absolute 2 mm length error and absolute mean length errors below 1.5 mm, outperforming AudioFC.AudioLSTM slightly outperforms AudioGRU on a single container and across all containers.Source: A. PouringNet Evaluation · p. 4, Figure 6 · p. 5
  • The evaluation compares AudioFC, AudioLSTM, and AudioGRU using thresholded error fractions and length errors across containers.The figure also shows AudioLSTM results for overall test data and each individual container dataset.Source: A. PouringNet Evaluation · p. 4, Figure 6 · p. 5
  • AudioLSTM trained on all container types achieves the best accuracy, suggesting better generalization than models trained on only part of the dataset.Among individual containers, the thermos performs best and the glass worst, which the authors mainly attribute to differences in material and sound.Source: A. PouringNet Evaluation · pp. 4–5

B. Robotic Experiments

In robotic pouring experiments, the trained and fine-tuned AudioLSTM estimates air-column length across dataset and unseen target containers. Errors are smaller at shorter target air-column lengths, with reported mean errors and standard deviations below 3 mm for dataset containers and below 4.5 mm for unseen containers.

B. Robotic Experiments

  • The robot evaluation used an AudioLSTM first trained on the original all-container dataset, then fine-tuned on a 30-trial-per-container spout dataset for feedback-controlled pouring.Pouring terminated when the desired air-column length was reached; the reported feedback-loop computation time was 21 ms.Source: B. Robotic Experiments · p. 5

1) Evaluation of Different Target Containers:

2) Evaluation of Varying Microphone Positions:

The microphone-position evaluation reports consistent mean height errors across eight tested positions. The distance from the noisy robot control box did not significantly influence performance.

2) Evaluation of Varying Microphone Positions:

  • The eight positions varied microphone distance from the target containers between 230 mm and 380 mm; testing used the mug, a 40 mm target, and five pours per position.Positions were placed on the left side of the target to avoid collision with the robot pouring from the right.Source: 2) Evaluation of Varying Microphone… · p. 6
  • Mean height errors remain highly consistent across all eight microphone positions, and distance from the noisy control box has no significant performance influence.The authors interpret this as robustness to different levels of environmental quiet.Source: 2) Evaluation of Varying Microphone… · p. 6

3) Evaluation of Varying Initial Liquid Height in Target Containers:

PouringNet is reported to remain stable across different initial liquid heights in robotic pouring. The evaluation varied initial heights from 10 to 40 mm.

3) Evaluation of Varying Initial Liquid Height in Target Containers:

4) Evaluation of Different Types of Liquid:

Liquid tests show that PouringNet generalizes across several household liquids, with viscosity limiting performance. The conclusion reports robustness and precision across varied settings and identifies noisier environments and multimodal sensing as future directions.

4) Evaluation of Different Types of Liquid:

VI. CONCLUSION AND FUTURE WORK

  • PouringNet generalizes across containers, microphone positions, initial heights, and some liquid types while maintaining precision in robotic experiments.Source: 4) Evaluation of Different Types of… · p. 7, §VI · p. 7
  • Air-column length serves as the height label, addressing overfill risk without prior knowledge of the target container.Source: §VI · p. 7
  • Future work will examine noisier environments with human voices, variable source-to-target distances, and interactions among sensing modalities.Source: §VI · p. 7
Loading 1903.00650v2…