PaperlayerView exact revision on arXiv

Source-linked AI summary

FloVMos: Optical Flow-based Medical Video Mosaicking

Jinyang Liu, Sandesh Ghimire, Chaman Singh, Jennifer Dy, Milind Rajadhyaksha, Dana H. Brooks, Octavia Camps, Kivanc Kose

arXiv:2610.04258v1eess.IVcs.CV

TL;DR

Biomedical videomosaicking faces a resolution, field-of-view, and speed trade-off, and existing methods can struggle with tissue deformation and modality-specific distortions. FloVMos fine-tunes an optical-flow network using automatically generated synthetic training data; across tested modalities, it reports improved accuracy and robustness with real-time processing.

  • Problem

    Biomedical imaging trades resolution, field of view, and acquisition speed, while existing mosaicking methods may not handle non-rigid deformation and modality-specific distortions.

    Source: §1 · pp. 1–2

  • Method

    FloVMos fine-tunes an optical-flow DNN on automatically generated image pairs with known flow, then registers, warps, and blends video frames into a mosaic.

    Source: §1 · p. 2

  • Results

    Across tested modalities, FloVMos reports lower MSE than comparison methods, improved robustness as non-rigid deformation increases, and effectively real-time processing.

    Source: §4 · p. 12

  • Takeaways & Limitations

    FloVMos offers an automated, modality-adaptable route to large-area visualization with real-time processing.

    Source: §1 · p. 2, §4 · p. 12

    The study focused on close-proximity modalities using simulated training data, and real-time processing requires a GPU that most clinical imaging systems lack.

    Source: §4 · p. 14

Abstract

from arXiv · show

Biomedical imaging modalities often require a trade-off among resolution, field of view (FOV), and acquisition speed. Video mosaicking offers a strategy to overcome this limitation by computationally stitching sequential high-resolution frames into a wide-FOV composite. However, existing methods struggle with non-rigid deformations, and modality-specific artifacts arising in clinical and research imaging. Here, we present FloVMos, a generalizable, optical-flow-based deep learning framework for real-time video mosaicking across diverse biomedical imaging modalities. FloVMos achieves robust, pixel-level registration by fine-tuning an optical flow model on synthetic training data with ground-truth deformation fields. We introduce a pipeline for generating this training data, simulating realistic tissue motion and imaging distortions from existing mosaics or raw videos. Our automated synthetic data generation and optical flow model training based on this data allow users to adapt FloVMos to different imaging modalities. To demonstrate this, we applied FloVMos to seven diverse imaging modalities: reflection confocal microscopy, open-top light-sheet microscopy, fetoscopy, laparoscopy, dermoscopy, sparse spectral microscopy, and endoscopy. FloVMos outperforms conventional baselines in accuracy, robustness, and speed for all the tested modalities. This adaptable and training-efficient framework enables large-area visualization with real-time performance and may support broader use of video-based biomedical imaging in research and clinical workflows.

1 Introduction

Biomedical imaging trades resolution, field of view, and acquisition speed, while existing mosaicking methods can struggle with modality-specific motion and tissue distortions. FloVMos combines fine-tuned optical-flow registration with automated synthetic training data to create real-time mosaics across imaging modalities.

1 Introduction

  • Mosaics combine sequential high-resolution, small-FOV images to reveal tissue relationships at scales larger than one frame.The paper notes that these spatial relationships can provide crucial information about tissue structures.Source: §1 · p. 1
  • Generic mosaicking methods do not account for frame changes and tissue distortions from loosely constrained motion, while deformation-capable methods are modality-specific and computationally costly.These limitations constrain operator control and real-time performance.Source: §1 · p. 2
  • FloVMos registers each frame to a growing mosaic with a supervised optical-flow DNN, then warps and blends it using graph cuts.It tracks a moving region of interest to preserve processing speed and avoid error accumulation associated with simpler frame-to-frame registration.Source: §1 · p. 2, Figure 1 · p. 2
  • Two automated strategies generate labeled synthetic training pairs from existing mosaics or videos without manual optical-flow labeling.The DNN is fine-tuned on data tailored to each modality; inference processes 1 MP images at 10 fps in the research implementation.Source: §1 · p. 2
  • The authors report successful retraining and application across seven imaging modalities, and make the mosaicking code and optical-flow models publicly available.The passage names reflection confocal microscopy, open-top light-sheet microscopy, fetoscopy, laparoscopy, dermoscopy, sparse spectral microscopy, and endoscopy.Source: §1 · pp. 2–3

2 Methods

The Methods section outlines the FloVMos framework, its synthetic-data-based training and components, and its approaches to handling video jumps and assessing accuracy without ground truth.

2 Methods

  • The section first outlines the FloVMos framework and then details synthetic-data-based training and the framework components.The component descriptions span Sections 2.2–2.5.Source: §2 · p. 3
  • A separate method addresses jumps in video streams, followed by a salient-points-based assessment of mosaicking accuracy without ground truth.These topics are covered in Sections 2.6 and 2.7.Source: §2 · p. 3
  • The Methods section concludes by detailing the data used in the experiments.The data description is in Section 2.8.Source: §2 · p. 3

2.1 Brief Description of FloVMos operation

FloVMos fine-tunes an optical-flow network for a particular modality using image pairs with known flow, then registers incoming frames into a growing mosaic. It also detects abrupt jumps and either adapts processing or starts a new mosaic.

2.1 Brief Description of FloVMos operation

  • Training fine-tunes the optical-flow DNN for a target modality using image pairs with known optical-flow parameters.The pairs can be automatically extracted from existing mosaics or videos.Source: §2.1 · p. 3, Figure 2 · p. 3
  • During inference, the DNN registers each new frame to a cropped region of interest around the most recently integrated frame, then blends it into the mosaic.This process repeats for each frame in the video.Source: §2.1 · p. 3, Figure 2 · p. 3
  • An abrupt loss of optical-flow smoothness signals a jump; FloVMos modifies processing to overcome it or saves the mosaic and starts another.Examples of jumps include motion blur and depth changes.Source: §2.1 · p. 3

2.2 Automated Generation of Training Data

FloVMos generates synthetic image pairs with known flow to fine-tune optical-flow estimators for different modalities. It can begin from large mosaics or, when only videos exist, construct training mosaics from video frames.

2.2 Automated Generation of Training Data

  • The training-data framework simulates camera motion and tissue deformation to generate synthetic video sequences with ground-truth optical-flow annotations for domain adaptation.The resulting noisy data are used to fine-tune the optical-flow estimator for a particular modality.Source: §2.2 · pp. 4–5
  • When large-FOV mosaics are available, simulated device paths across them generate repeatable synthetic videos.The path starts at a random interior location and uses controlled random parameters.Source: §2.2 · p. 4
  • When only videos are available, training mosaics are made either by flipping one frame into a 2 × 2 grid or arranging four temporally distant, minimally overlapping frames.The selected frames are at least 15% of the video length apart, with less than 25% overlap; this is offered as a practical alternative even when the mini-mosaics look unrealistic.Source: §2.2 · p. 4
  • Synthetic sequences start from a frame-sized crop and repeatedly apply known flow maps combining global affine motion and non-rigid deformation.The flow map drives backward warping to create each next frame until the simulated view reaches the mosaic edge.Source: §2.2 · p. 4
  • Translation and rotation follow autoregressive motion, while smooth non-rigid fields are added frame by frame with a preset probability.The non-rigid field is generated at 4 × 4 resolution and bilinearly upsampled to the camera field of view; the affine component also includes shearing.Source: §2.2 · pp. 4–5
  • Motion and deformation parameters can be tailored by modality; the Non-Rigid Level is 3 for dermoscopy and 10 for RCM.The authors associate dermoscopy with stable skin contact and RCM with handheld, higher-magnification imaging; Table 1 summarizes modality-specific parameter choices, with translation defined for images resized to 256 × 256.Source: §2.2 · p. 5, Table 1 · p. 5

2.3 Optical Flow DNNs

FloVMos fine-tunes pretrained optical-flow networks on modality-specific synthetic training data. The resulting estimator is applied to incoming image pairs for image-to-mosaic registration.

2.3 Optical Flow DNNs

  • FlowNet2 was the base optical-flow network; RAFT, GMFlow, and FlowFormer were also tested, with pretrained weights adapted for each of seven imaging modalities.Each model started from its published pretrained coefficients.Source: §2.3 · p. 5
  • Training used Adam at a 10^-4 initial learning rate for 100 epochs, with batch sizes of 8 for FlowNet2 and GMFlow and 6 for RAFT and FlowFormer.The batch sizes were the maximum that fit within 11 GB of VRAM.Source: §2.3 · p. 5
  • All models were trained on 256 × 256 pixel images, while original modality resolutions varied, including 1000 × 1000 pixels for dermoscopy, RCM, SSM, and OTLS.Endoscopy was typically 1280 × 1024 pixels; fetoscopy and laparoscopy resolutions varied by implementation.Source: §2.3 · pp. 5–6
  • Fine-tuning produces a modality-adapted optical-flow estimator for incoming image pairs, used to estimate the transformation for image-to-mosaic registration.Source: §2.3 · p. 6

2.4 Image to Mosaic Registration

FloVMos registers each incoming frame directly against a region of the growing mosaic rather than recursively aligning frames to the first frame. This design addresses cumulative registration error while retaining mosaicking speed through GPU assistance.

2.4 Image to Mosaic Registration

  • Directly registering each frame into the growing mosaic avoids recursive flow-field updates that can accumulate small errors and potentially cause stitching failure.Source: §2.4 · p. 6
  • The system tracks the last stitched frame’s center and corners, crops its corresponding rotated mosaic region, estimates flow against the incoming frame, then maps the registered frame into mosaic coordinates.Source: §2.4 · p. 6
  • FloVMos repeats the inference pipeline for every video frame, and its GPU-assisted implementation maintains mosaicking speed, as reported in Fig. 4(c).Source: §2.4 · p. 6

2.5 Image Warping and Graph-cut Based Stitching

FloVMos uses bilinear interpolation to map non-integer optical-flow displacements onto an integer pixel grid, then graph-cut stitching merges registered frames with the mosaic. The stitching seam follows less-detailed overlap regions and avoids blending pixel values.

2.5 Image Warping and Graph-cut Based Stitching

  • Bilinear interpolation samples warped-frame intensities onto the integer grid, accommodating non-integer displacements and non-rigid flow while supporting real-time performance.Its kernel weights neighboring fractional pixel locations.Source: §2.5 · p. 6
  • 35 frames per second is the reported CPU processing speed for bilinear interpolation in Python on 1000 × 1000 pixel videos.Source: §2.5 · p. 6
  • Graph-cut stitching places an optimal seam through low-detail overlap regions, retaining registered-frame pixel values rather than blending them, though overlapping pixels may be overwritten by later frames.The seam determines whether an overlapping pixel comes from the mosaic or the latest relevant frame.Source: §2.5 · p. 6

2.6 Handling abrupt changes in the videos

FloVMos detects abrupt video changes using optical-flow smoothness and tries to preserve stitching continuity by selecting a clean buffered frame. If overlap with the existing mosaic is insufficient, it starts a new mosaic.

2.6 Handling abrupt changes in the videos

  • Abrupt device motion, image blurring or discontinuities, and sudden depth changes in 3D RCM can cause severe stitching errors; FloVMos uses adjacent frames or starts a new mosaic to maintain continuity.Source: §2.6 · p. 7
  • The detector flags flow maps whose mean discrete spatial second-order derivatives exceed the empirically selected 0.1 smoothness threshold.When a frame is flagged, the system buffers subsequent frames and selects the one with the smoothest flow; the example buffer spans one second of video.Source: §2.6 · p. 7
  • If the selected buffered frame’s average pixel difference from the mosaic ROI exceeds 30 for monochrome or 40 for RGB, FloVMos starts a new mosaic.Source: §2.6 · p. 7

2.7 Identifying Salient Points for Computing Registration Errors

Without ground-truth optical flows, registration accuracy is estimated by checking whether distances between matching salient points change after mosaicking. The procedure extracts, filters, and compares SIFT correspondences between video frames and reconstructed mosaics.

2.7 Identifying Salient Points for Computing Registration Errors

  • The method assumes distances between structures should not change drastically after registration and warping, then measures frame-to-mosaic displacement of matched salient features.SIFT features are matched between each video frame and mosaic; pairwise distance matrices are compared and averaged across frames.Source: §2.7 · p. 7
  • RANSAC with 10,000 iterations estimates a homography and inlier correspondences, excluding frames with fewer than 5% inliers.Matches whose homography scaling factors fall outside [0.8, 1.25] are also discarded.Source: §2.7 · p. 7
  • The frame-wise error is the difference between pairwise keypoint-distance matrices for the frame and mosaic, averaged over the sequence.The brute-force SIFT matcher is used before RANSAC filtering to address erroneous matches from repeated or structurally similar patterns.Source: §2.7 · p. 7

2.8 Dataset Details

The experiments draw on seven modalities, using mosaic-derived synthetic data where mosaics were available and videos otherwise. Training and test videos were kept separate, and sequences outside the 2D-mosaic setting were excluded.

2.8 Dataset Details

  • Data came from MSKCC for RCM and dermoscopy, university laboratories for OTLS and SSM, and public datasets for endoscopy, laparoscopy, and fetoscopy.The public sources were the Endoscopic Vision Challenge, Hamlyn Center Laparoscopic/Endoscopic Video, and FetReg datasets.Source: §2.8 · p. 8
  • RCM and OTLS had large-FOV mosaics for synthetic data and testing; the other modalities had videos only, and handheld RCM videos were used for testing.The dataset table distinguishes training sources, real versus simulated tests, dataset sizes, and experiment purposes; videos were not reused for training-data generation and testing.Source: §2.8 · p. 8, Table 2 · p. 8
  • Mosaicking experiments excluded sequences with extreme artifacts or strong vertical camera motion, which cannot be represented by a 2D mosaic.For deformation experiments, 1,600 synthetic videos were generated across Non-Rigid Levels 0–15, with 100 videos per level.Source: Table 2 · p. 8, §2.8 · p. 8

3 Results

FloVMos was evaluated across seven imaging modalities and in controlled comparisons on simulated RCM videos. The results cover registration accuracy, non-rigid deformation, speed, and optical-flow estimator choice.

3 Results

  • The results span eight examples from seven modalities, with quantitative feature-point-distance results reported for all eight datasets.The broader results section also examines deformation robustness, video-length sensitivity, and estimator flexibility.Source: §3 · p. 8, §3.1 · p. 9

3.1 Applying FloVMos to videos from multiple imaging modalities

  • Across all seven modalities, FloVMos preserved feature-point distances with an average of < ±5 pixel/match (<0.5% of the field of view) difference.Fig. 3 presents eight cases across the modalities and histograms of source-to-mosaic feature-point distances.Source: §3.1 · pp. 9–10, Figure 3 · p. 9

3.2 Comparison of FloVMos Videomosaicking Accuracy to Existing Videomosaicking algorithms

  • FloVMos achieved better mean SSIM and LPIPS values and a more compact distribution than the comparison algorithms; its difference from Parallax was significant (p-value < 0.0001).The comparison used 800 simulated RCM videos and included Parallax, APAP, and AVM.Source: §3.2 · p. 10, Figure 4 · p. 11

3.3 Handling Non-rigid Deformations

  • FloVMos consistently achieved lower registration-error MSE than the competing methods across varied degrees of non-rigid deformation.The RCM experiments systematically varied deformation severity across 1,600 frame pairs; MSE compared predicted and ground-truth optical flow.Source: §3.3 · p. 10

3.4 Sensitivity of Videomosaicking Speed to Video Length

  • FloVMos consistently outperformed the three alternative algorithms in frames per second as video length increased.Speed was compared across videos with increasing frame counts.Source: §3.4 · p. 10

3.5 Sensitivity to choice of optical flow estimation algorithm

  • FloVMos is modular with respect to the optical-flow estimator; four networks were fine-tuned on the same 11,000 videos and evaluated on 172 distinct videos.The comparison reported MSE, F1-score, and mosaicking speed; the rest of the algorithm was held constant, but the supplied passages do not report the Table 3 values.Source: §3.5 · pp. 10, 12, Table 3 · p. 12

4 Discussion

FloVMos showed lower MSE, greater deformation robustness, and faster processing across seven tested modalities, while simulated-data training and GPU requirements constrain its current clinical scope.

4 Discussion

  • Across seven modalities, FloVMos had lower MSE than comparison methods, greater robustness to non-rigid deformation, and faster, video-length-insensitive processing.Parallax slowed from 0.2 to 0.05 FPS as video length increased.Source: §4 · p. 12
  • Everyday-image optical-flow models miss the precision demands; FloVMos synthesizes camera-motion flow pairs to fine-tune modality-specific estimators without pixel-level annotations.The untuned model’s registration failures are illustrated across all seven modalities.Source: §4 · p. 13
  • Synthetic datasets remain smaller than typical deep-learning benchmarks, especially when generated from raw videos; limited-data tuning may bias models and calls for domain-specific priors.Mosaics can support larger synthetic datasets because they typically cover more tissue than single videos.Source: §4 · p. 13
  • Synthetic data must match modality-specific image characteristics and realistic deformation; domain knowledge guides distortion settings, or small varied sets can be checked against real images.For mosaic-only training, increasing dataset size and diversity helps mitigate uncertainty from lacking a temporal reference.Source: §4 · pp. 13–14
  • At least 50% overlap between consecutive frames supported robust flow estimation; selective downsampling of higher-overlap video can accelerate processing without compromising accuracy.This is a practical frame-rate management strategy for real-time deployment.Source: §4 · p. 14
  • The study concentrated on close-proximity modalities; farther-view imaging may introduce distortions absent from the current simulations, and real-world clinical testing is needed.Future adaptation may require tuning distortion parameters to the clinical or laboratory setting.Source: §4 · p. 14
  • Real-time processing requires a GPU, a constraint for most clinical imaging systems; CPU alternatives or model compression remain possible but involve uncertain trade-offs.The authors note that FloVMos runs well on affordable commercial GPUs.Source: §4 · p. 14
  • Intraoperative imaging is a potential application, but technical robustness and effects on surgical outcomes and workflow efficiency still need evaluation.Early studies indicate feasibility.Source: §4 · p. 14
Loading 2610.04258v1…