PaperlayerView exact revision on arXiv

Source-linked AI summary

A Passive AI System for Verifying Physical State on Automated Liquid Handlers

Junqiong Joanne Qiu, Zeckria Kamrany, Ananya Anand, Emma Vidal, Yunjia Johanna Bai, Nathan S Chen, Sean Son, Varada Abhyankar, Taryn Jakub, Eleazar Eskin

arXiv:2610.02668v1cs.RO

TL;DR

Manual liquid-handler setup lacks a system to confirm that the physical deck matches the robot’s expectations. The paper presents an external vision checker that compares observed labware with a protocol reference. In occlusion tests, all 6,720 deck-region assignments were correct.

  • Problem

    Operators manually assemble liquid-handler decks, but no system confirms that the setup matches the robot’s expectations.

    Source: §1 · p. 1

  • Method

    The checker uses computer vision to structure deck images for protocol comparison and reports per-slot setup status.

    Source: B. Incorrect deck setup · p. 2, §2.1 · p. 3

  • Results

    100% of deck-region assignments were correct across 480 occlusion-test frames, covering 6,720 observations.

    Source: §2.3 · p. 4

  • Takeaways & Limitations

    The checker verifies deck preparation before a run but does not plan experiments, control the robot, or monitor liquid handling.

    Source: §3 · p. 5

    The evaluation covered Flex robots in one laboratory and a limited set of workflows, and protocol-reference agreement is not independently annotated setup-error accuracy.

    Source: §3 · p. 6

Abstract

from arXiv · show

Automated liquid handlers execute digital protocols, but operators must assemble the deck and confirm that the physical setup matches the intended experiment. We present the Labware Setup Checker, a passive vision system that verifies deck preparation on an Opentrons Flex without modifying its hardware or firmware. A consumer webcam mounted outside the robot supplies images to a browser application. The checker segments the deck into slots, corrects slot assignments using the known deck geometry, classifies the labware, and compares the observed setup with a reference protocol library to report missing, misplaced, or unexpected labware. It operates independently of the robot controller and does not interrupt or gate a run. During SwabSeq Respiratory Viral Panel (RVP) pre-PCR deck preparation, the checker tracked a slot from empty to a base and then to the completed plate-and-base assembly, identifying the completed assembly in all 29 frames before robot motion. Across these frames, 95.1% of the 348 labware classifications for slots requiring exact matches agreed with the protocol reference. In 480 frames with partially occluded reservoirs, all 6,720 slot assignments were correct. Across three camera positions and two lighting conditions, slot assignment was 100% correct, and 97.73% of labware classifications matched the protocol reference. Median verification time was 2.10 s on a laptop using only its CPU and 4.24 s from frame capture to receipt of results during AWS deployment in the laboratory. These results show that an external camera and independent software can provide verification of deck setup against protocol requirements within seconds, allowing operators to review the physical setup before starting a run.

1 Introduction

Automated liquid-handler decks require manual setup, but operators lack a system to confirm that the physical layout matches the robot’s expectations. Existing camera-based options do not provide protocol-aware labware verification.

1 Introduction

  • Manual deck setup can put plates, tip racks, or reservoirs in the wrong positions, potentially causing collisions, robot damage, or incorrect liquid transfers.Operators must confirm the physical setup before a protocol begins.Source: §1 · p. 1
  • No system confirms that the initial manual deck setup matches the robot’s expectations, motivating a monitoring layer for experiment identity, deck preparation, and protocol execution.Source: §1 · p. 1
  • The Flex’s built-in camera records images but does not check whether observed labware is correct for a protocol, while DeckCheck compares pixels rather than labware identity.The passage notes DeckCheck can therefore miss differences such as tip types, partially loaded runners, plate rotation, and well shape.Source: §1 · p. 1

B. Incorrect deck setup

The paper presents an external, protocol-aware vision checker for verifying liquid-handler deck preparation. It addresses the lack of an accessible way to compare a deck with its intended protocol without modifying the robot.

B. Incorrect deck setup

  • Affordable liquid-handler users lack a straightforward way to check deck setup against the intended protocol, while existing options do not offer an accessible, modular layer.Source: B. Incorrect deck setup · p. 2
  • A practical checker must be inexpensive, leave robot hardware and firmware untouched, run on an available computer, and add no routine operator work.Source: B. Incorrect deck setup · p. 2
  • The Labware Setup Checker uses a consumer webcam and lightweight models to verify an Opentrons Flex deck against an intended protocol within seconds.It requires no robot hardware or firmware modification and operates without interrupting the established procedure.Source: B. Incorrect deck setup · p. 2
  • The checker segments deck slots, classifies labware, corrects assignments using canonical slot geometry, and compares observed slot–labware pairs with a reference protocol.Source: B. Incorrect deck setup · p. 2
  • The checker marks each slot as correct, wrong, misplaced, or unexpected, allowing comparison of observed setup with protocol requirements.Source: B. Incorrect deck setup · p. 2

2 Results

The checker processes camera frames into protocol-aware, per-slot deck states while operating independently of the robot controller. In evaluation, it recognized an empty deck and tracked RVP deck preparation through the completed assembly before robot motion.

2.1 The checker reports protocol-aware deck states without modifying the robot

  • The checker displays a live feed with predicted slot and labware labels, status updates, detection logs, and the predicted protocol.It runs independently of the robot controller, requires no hardware or firmware modification, and does not interrupt robot operation.Source: §2.1 · pp. 2–3

2.2 The checker recognizes an empty deck and tracks deck preparation

  • The checker tracked C2 from empty, to a MicroAmp base, to the completed plate-and-base assembly, identifying the assembly correctly in all 29 frames before robot motion.Source: Figure 2 · p. 3, §2.2 · p. 3

2.1 The checker reports protocol-aware deck states without modifying the robot

  • Each sampled frame is segmented into deck regions, classified, corrected against canonical deck geometry, and compared with a protocol reference.The evaluated deck representation included 14 observable regions; D3 was excluded because the trash chute occupied that location in the tested protocols.Source: §2.1 · p. 3
  • Per-slot results distinguish correct, missing expected labware, labware in another protocol slot, and labware not used by the protocol.The operator can compare the observed deck, expected layout, and per-slot results before starting a run.Source: §2.1 · p. 3

2.2 The checker recognizes an empty deck and tracks deck preparation

  • The evaluation covered an empty Flex deck and preparation of a routine SwabSeq RVP pre-PCR deck.Source: §2.2 · p. 3
  • Across 113 frames of an empty Flex deck, the checker consistently recognized the remaining slots as empty while the C1 temperature module and D1 aluminum block remained present.Source: §2.2 · p. 3

B. Alternate angle 2

Across occlusion, camera-position, and lighting tests, the checker localized deck regions reliably, while labware classification errors remained tied to particular views and objects. The perception models achieved high validation performance, and verification completed within seconds locally and during deployment.

B. Alternate angle 2

  • In the alternate-angle evaluation, 389 of 406 observations received correct verification status (95.8%); 17 misclassifications clustered at D1 and D4.The D1 errors involved visually similar aluminum blocks; D4 was at the edge of the camera’s field of view.Source: B. Alternate angle 2 · p. 4

2.3 The checker remains robust to occlusion, camera position, and lighting conditions

  • Across 480 occlusion-test frames, all 6,720 deck-region assignments were correct (100%), including the partially blocked reservoir positions.The test used four reservoirs in 24 arrangements, with foreground tip racks and the waste chute partially obstructing them.Source: §2.3 · p. 4
  • Across six camera–lighting combinations, all 2,156 deck-region observations were correct (100%), while 2,107 of 2,156 slot classifications matched the protocol (97.73%).The 49 incorrect classifications followed patterns associated with specific camera and lighting combinations; the authors considered reflections a likely explanation.Source: §2.3 · p. 4

2.4 The perception models achieve high validation accuracy

  • On held-out validation sets, YOLO11m segmentation reached 0.995 mask mAP@50 and 0.884 mask mAP@50–95, while YOLO26m classification reached 0.992 top-1 accuracy.The classifier’s top-1 accuracy remained above 0.95 after epoch 2 and ended at 0.992; the training records showed convergence.Source: §2.4 · pp. 4–5, Figure 4 · p. 5

2.5 The checker completes deck verification locally and in deployment within seconds

  • Median verification time was 2.10 s locally and 4.24 s from frame capture to result receipt during AWS deployment.The deployed evaluation processed 280 frames; the same trained models supported the additional workflow without retraining, requiring only its expected deck setup in the protocol library.Source: §2.5 · p. 5

3 Discussion

The checker independently compares observed Flex deck setup with a protocol reference and reports slot-level status, but it does not control the robot. Its evaluation and deployment depend on specific workflows, labware, camera coverage, and operator review.

3 Discussion

  • The checker converts Flex deck images into slot-level verification results against a protocol reference, reporting correct, empty, misplaced, or unexpected labware.It verifies deck preparation only; it does not plan experiments, control the robot, or monitor liquid handling.Source: §3 · p. 5
  • Median complete-deck verification took 2.10 s on a CPU-only laptop, and AWS results returned within seconds during laboratory operation.Tests across three camera positions and two lighting conditions showed complete deck-region localization and high protocol-reference agreement under the evaluated conditions.Source: §3 · pp. 5–6
  • Because the checker is independent of the robot controller, it cannot prevent a run from starting after a mismatch; the operator must notice and respond to its result.This preserves operator control and simplifies deployment, but makes correction dependent on operator action.Source: §3 · p. 6
  • Adoption requires configuring the expected protocol layout and camera view; unseen or visually different labware requires image collection, annotation, retraining, and validation.The checker is not ready for immediate use with every protocol and labware type.Source: §3 · p. 6
  • The evaluation covered Flex robots in one laboratory and limited workflows, and protocol-reference agreement is not independently annotated setup-error accuracy.Classification may fail for visually similar or unrepresented labware; reliable verification also requires adequate camera coverage and does not monitor pipetting or other in-run failures.Source: §3 · p. 6
  • Future work includes generating protocol references from Opentrons files, simplifying new-labware addition, and evaluating more laboratories, camera installations, and workflows.Recording genuine setup mistakes would help determine how often the checker detects errors requiring action before a run.Source: §3 · p. 6

4 Methods

The methods describe an external-camera Flex deployment, image datasets and trained models, a slot-correction and protocol-matching pipeline, and laboratory and latency evaluations. The browser and AWS backend process images independently of the robot controller.

4.1 Hardware

  • A consumer webcam mounted outside the Flex observes the deck, while an adjacent computer runs the browser interface without modifying robot hardware or firmware.The adjustable external camera position was chosen to maximize deck visibility while avoiding gantry motion, pipette access, and operator setup.Source: Figure 5 · p. 6, §4.1 · p. 6

4.2 Dataset collection and annotation

  • The models used 121 annotated full-deck images and 1,212 cropped images across 17 classification classes, with empty slots explicitly labeled.Datasets used an 85/10/5 split, and only training images were augmented threefold.Source: §4.2 · p. 6, Figure 6 · p. 7, §4.3 · p. 7

4.3 Model training

  • Training used YOLO11m segmentation and YOLO26m classification models, with training-set augmentation and exported OpenVINO weights for CPU inference.The segmentation and classification datasets were augmented from 103 to 309 and from 1,030 to 3,090 training images, respectively.Source: §4.3 · p. 7

4.4 Inference

  • For each frame, preprocessing and segmentation identify deck regions, masks reduce background in cropped images, and geometric correction assigns detections to canonical slots before classification.Affine alignment with RANSAC and Hungarian assignment combine geometry, original labels, and confidence; a bounding-box-span fallback is used when affine estimation fails.Source: §4.4 · pp. 7–8, Figure 7 · p. 8

4.5 Protocol matching

  • The checker scores observed slot–labware pairs against protocol layouts, selects the highest-scoring protocol, and labels slots as correct, wrong, wrong_slot, or not_in_setup.Exact matches score highest, misplaced labware receives partial credit, and permissive slots accept any observed labware.Source: §4.5 · p. 8

4.6 AWS deployment

  • A React browser frontend sends camera frames to a containerized FastAPI backend on AWS, which runs the vision and protocol-matching pipeline and returns detections and protocol metadata.Recent detections and session state are stored in Valkey; frames and masked images are written to S3.Source: §4.6 · p. 8, Figure 8 · p. 9

4.7 Evaluation

  • Laboratory evaluation recorded empty-deck and SwabSeq RVP pre-PCR preparation, including C2 changing from empty to a base and then a completed assembly before robot motion.Occlusion testing used 480 frames across 24 reservoir arrangements; camera and lighting robustness used 154 stable frames across three camera positions and two lighting conditions.Source: §4.7 · pp. 8–9
  • Latency evaluation measured the complete local pipeline and AWS request path, including browser-to-response time during a laboratory workflow.The local analysis used ten timed runs after initialization; AWS end-to-end timing used 280 frames, excluding uploads that continued after results returned.Source: §4.7 · pp. 9–10

Funding

Funding information is deferred to a revised version.

Funding

  • Funding information will be provided in a revised version.Source: Funding · p. 10

Related papers

Loading 2610.02668v1…