PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoLaparo: A New Dataset of Integrated Multi-tasks for Image-guided Surgical Automation in Laparoscopic Hysterectomy

Ziyi Wang, Bo Lu, Yonghao Long, Fangxun Zhong, Tak-Hong Cheung, Qi Dou, Yunhui Liu

arXiv:2208.02049v1cs.CV

TL;DR

Learning-based surgical automation needs large, high-quality datasets with multiple task annotations, but such public data are limited, particularly for laparoscopic hysterectomy. AutoLaparo provides an integrated dataset for workflow recognition, laparoscope motion prediction, and instrument and anatomy segmentation, with benchmarks reported for all three tasks.

  • Problem

    Public datasets for laparoscopic hysterectomy are sparse or too small, while learning-based models require large, high-quality data that span multiple tasks.

    Source: §1 · p. 2

  • Method

    AutoLaparo builds an integrated dataset from full-length hysterectomy videos, annotating workflow phases, laparoscope motion, and instrument and key anatomy segmentation.

    Source: §1 · pp. 2–3

  • Results

    Benchmarks are reported for all three tasks; workflow models exceed 75% accuracy on validation and test sets, while the two motion models reach 26.03% and 27.40% test accuracy.

    Source: Evaluation metrics and results · p. 7, State-of-the-art methods · p. 8, §4 · p. 9

  • Takeaways & Limitations

    The integrated dataset and reference benchmarks provide a basis for future model development and evaluation across the three tasks.

    Source: §4 · p. 9

    Challenges remain in improving unimodal task performance and achieving multi-task learning by combining modalities; the dataset also needs more images, videos, and tasks.

    Source: §4 · p. 9

Abstract

from arXiv · show

Computer-assisted minimally invasive surgery has great potential in benefiting modern operating theatres. The video data streamed from the endoscope provides rich information to support context-awareness for next-generation intelligent surgical systems. To achieve accurate perception and automatic manipulation during the procedure, learning based technique is a promising way, which enables advanced image analysis and scene understanding in recent years. However, learning such models highly relies on large-scale, high-quality, and multi-task labelled data. This is currently a bottleneck for the topic, as available public dataset is still extremely limited in the field of CAI. In this paper, we present and release the first integrated dataset (named AutoLaparo) with multiple image-based perception tasks to facilitate learning-based automation in hysterectomy surgery. Our AutoLaparo dataset is developed based on full-length videos of entire hysterectomy procedures. Specifically, three different yet highly correlated tasks are formulated in the dataset, including surgical workflow recognition, laparoscope motion prediction, and instrument and key anatomy segmentation. In addition, we provide experimental results with state-of-the-art models as reference benchmarks for further model developments and evaluations on this dataset. The dataset is available at https://autolaparo.github.io.

1 Introduction

Learning-based surgical automation is constrained by the shortage of large, high-quality datasets spanning multiple tasks, especially for laparoscopic hysterectomy. AutoLaparo addresses this gap with an integrated dataset and reference benchmarks for three correlated perception tasks.

1 Introduction

  • Existing laparoscopic datasets are limited in real-surgery scale or focus on one task, while hysterectomy datasets are sparse or too small for model development.The paper identifies a need for multiple data modalities to support whole-scene understanding and image-guided automation.Source: §1 · p. 2
  • AutoLaparo integrates surgical workflow recognition, laparoscope motion prediction, and instrument and key anatomy segmentation for laparoscopic hysterectomy.The dataset includes complete-procedure videos, motion clips, and segmentation annotations sampled from those clips.Source: §1 · pp. 2–3
  • The tasks and annotations are highly correlated to support multi-task and multi-modality learning, and the paper provides state-of-the-art reference benchmarks for each task.Fig. 1 illustrates procedure phases, a laparoscope Zoom-in clip, and pixel-wise annotated frames.Source: §1 · p. 3, Figure 1 · p. 3

2 Dataset Design and Multi-tasks Formulation

AutoLaparo combines full-length laparoscopic hysterectomy videos with three related tasks and a tiered annotation process. The design links workflow, laparoscope motion, and segmentation data to support perception across the surgical scene.

2.1 Dataset collection

  • The collection comprises 21 hysterectomy videos from Prince of Wales Hospital, recorded from October to December 2018 and approved for research use with personal information protected.After removing periods when the laparoscope was outside the abdomen, the videos total 1,388 minutes, averaging 66 minutes each.Source: §2.1 · pp. 3–4

2.2 Task formulation

2.3 Dataset annotation and statistics

  • A senior gynaecologist and an experienced hysterectomy specialist annotated the three correlated sub-datasets, with results proofread and cross-checked for accuracy and consistency.Workflow labels are assigned at video level across seven phases.Source: §2.3 · p. 4, Workflow annotation at video-level · p. 4
  • Motion annotations use 300 ten-second clips from phases 2–4, with motion time T set at the fifth second and seven motion classes.The classes are static, Up, Down, Left, Right, Zoom-in, and Zoom-out.Source: Motion annotation at clip-level · p. 5, Table 1 · p. 5
  • Segmentation annotations cover six frames sampled at 1 fps from each selected clip, producing 1,800 frames and 5,936 annotations across nine types.Instrument shafts and manipulators are labeled separately; Instrument-2 appears most often, and the uterus appears in more than half of frames.Source: Table 2 · p. 5, Segmentation annotation at frame-level · p. 6, Figure 2 · p. 6

2.4 Dataset analysis and insights

  • Workflow and tool-use annotations provide complementary information for phase recognition and segmentation, while workflow and segmentation can inform laparoscope motion prediction.The integrated dataset is intended to let researchers explore the added value of combining modalities for surgical perception.Source: §2.4 · p. 6

3 Experiments and Benchmarking Methods

The section reports benchmark evaluations for three AutoLaparo tasks using task-specific models and metrics. Workflow models exceed 75% accuracy, motion prediction accuracy remains low, and segmentation models are compared on box and mask AP.

3 Experiments and Benchmarking Methods

  • Benchmarks are provided for each AutoLaparo task to support further model evaluation.The section evaluates workflow recognition, laparoscope motion prediction, and instrument and key anatomy segmentation.Source: §3 · p. 6

3.1 Workflow recognition

  • All four workflow approaches achieved accuracy above 75% on both validation and test sets.The authors describe these results as potential support for online workflow recognition and guidance for following tasks.Source: Table 3 · p. 7, Evaluation metrics and results · p. 7

3.2 Laparoscope motion prediction

  • 29.82% validation and 27.40% test accuracy for ResNet-50+LSTM, versus ResNet-50’s 28.07% and 26.03%, respectively.The authors report that the second model achieves higher accuracy and that overall results need improvement.Source: State-of-the-art methods · pp. 7–8, Table 4 · p. 8

3.3 Instrument and key anatomy segmentation

  • On the test set, using a shared ResNet-50 backbone, YOLACT achieved the highest reported mask AP at 54.09%, versus 51.70% for Mask R-CNN and 52.58% for YolactEdge.Source: §3.3 · p. 8

4 Conclusion and Future Work

AutoLaparo integrates three correlated tasks for laparoscopic hysterectomy and provides benchmarks for future model development and evaluation. The authors identify unimodal task performance and multimodal multitask learning as remaining challenges, and plan to expand the dataset and its tasks.

4 Conclusion and Future Work

  • AutoLaparo combines workflow recognition, laparoscope motion prediction, and instrument and key anatomy segmentation as correlated tasks for advanced scene understanding.Source: §4 · p. 9
  • Experiments and benchmarks for each task are provided as references for future model development and evaluation.Source: §4 · p. 9
  • Improving performance with unimodal data and achieving multitask learning by combining modalities remain challenges.Source: §4 · p. 9
  • The authors plan to expand the dataset with more images and videos and define additional tasks for vision-based applications.Source: §4 · p. 9

Related papers

Loading 2208.02049v1…