PaperlayerView exact revision on arXiv

Source-linked AI summary

OLiVES: An Outdoor Low-Light Video Benchmark for Enhancement and Segmentation

Yini Li, Joanne Lin, David Blackham, David Bull, Nantheera Anantrasirichai

arXiv:2610.20501v1cs.CV

TL;DR

Low-light video research lacks high-quality aligned ground truth, especially for dynamic scenes and joint enhancement and scene understanding. OLiVES is an outdoor low-light benchmark for enhancement and video object segmentation, captured with cinema-grade cameras and varied ND filters. It supplies aligned frames and segmentation annotations, while its low light is simulated using ND filters in outdoor daytime conditions.

  • Problem

    Low-light video research lacks high-quality aligned ground truth, and existing datasets have limited motion diversity and coverage of downstream scene understanding.

    Source: §1 · p. 2

  • Method

    OLiVES is an outdoor benchmark for joint low-light video enhancement and video object segmentation, captured with cinema-grade cameras and varying ND filters.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    OLiVES provides 25,254 registered normal/low-light frames and 46,366 pixel-wise segmentation annotations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    OLiVES provides a unified benchmark for evaluating restoration and downstream understanding on outdoor low-light video.

    Source: §1 · p. 2

    The dataset uses ND-filter-simulated low light captured outdoors in daytime, so its noise and colour characteristics may differ from real low-light footage.

    Source: §3.5 · p. 8

Abstract

from arXiv · show

Low-light video processing remains an underexplored problem in computer vision, primarily due to the scarcity of high-quality ground-truth data. Existing public datasets exhibit several limitations, including restricted motion diversity, limited scene coverage, weak degradations, reliance on auxiliary modalities such as event data, and misalignment between low-light inputs and ground truth. Furthermore, current datasets typically address either low-light enhancement or scene understanding, whereas these tasks should be considered jointly. We introduce \textbf{OLiVES}, a benchmark dataset for outdoor low-light videos with dynamic scenes, designed to support both low-light video enhancement (LLVE) and video object segmentation (VOS). The dataset addresses key content gaps by incorporating a diverse range of natural textures, animals, occlusions, dynamic activities, and close-up scenes. To the best of our knowledge, OLiVES is the first outdoor low-light video dataset captured using professional cinema-grade cameras. It provides over 25,000 aligned normal- and low-light frame pairs, along with over 46,000 VOS annotations. We benchmark state-of-the-art LLVE and VOS methods on OLiVES and show that training on it improves performance on both tasks. The OLiVES project page is available at: https://anonymous-OLiVES.github.io/.

1 Introduction

Low-light video research is constrained by the difficulty of obtaining aligned ground truth for dynamic scenes and by the limited coverage of low-light benchmarks for downstream understanding. OLiVES addresses these gaps with an outdoor benchmark supporting both enhancement and video object segmentation.

1 Introduction

  • Low-light performance matters for applications including autonomous vehicles, surveillance, and wildlife conservation, but noise, blur, low brightness and contrast, and video-processing costs pose challenges.Performance of computer vision methods degrades significantly in low-light conditions [31] [30].Source: §1 · p. 2
  • Existing paired low-light video datasets often restrict scene motion or use weak degradations, leaving complex real-world motion patterns underrepresented.Physical alignment commonly relies on dual-camera capture or repeated shooting, and DID reduces ISO under well-lit conditions.Source: §1 · p. 2
  • Low-light VOS benchmarks remain scarce, while enhancement-first pipelines can degrade semantic cues needed by downstream models.The passage identifies SDE as a joint enhancement and semantic-segmentation benchmark, but notes restricted motion and unverified annotations.Source: §1 · p. 2
  • OLiVES provides 25,254 registered normal/low-light frames and 46,366 pixel-wise segmentation annotations for joint low-light video enhancement and video object segmentation.The benchmark uses professional cinema-grade cameras and includes animals, natural textures, foliage, and dynamic outdoor activity.Source: §1 · p. 2

2 Related work

Prior low-light video enhancement datasets vary in capture setup and realism, while accurate frame alignment often limits scene motion. Low-light VOS coverage is also limited, with existing datasets differing in illumination, resolution, and reliance on auxiliary event data.

2 Related work

  • Existing LLVE datasets trade off alignment, realistic degradation, motion, scene setting, and resolution across their capture setups.DRV is restricted to static scenes; DID reduces ISO under well-lit conditions; BVI-RLV [31] is controlled and indoor; and SDE is 346×260.Source: §2 · p. 4
  • Re-recording scenes under different lighting restricts motion, and OLiVES is the only dataset identified as combining dynamic camera and object motion with VOS annotations.The passage links the capture constraint to the absence of heavy motion blur from moving objects in these benchmarks.Source: §2 · p. 4
  • Semi-supervised VOS benchmarks typically guide segmentation with a mask from the first frame, and most consider only objects present there.YouTube-VOS [42] also includes videos where new objects appear in later frames.Source: §2 · p. 4
  • Most popular VOS benchmarks provide normal-light video, while low-light VOS datasets have limitations in coverage or capture conditions.LLE-VOS [23] relies heavily on event data, has low resolution, and exhibits an orange colour cast; MOSEv2 [11] does not explicitly separate low-light scenes.Source: §2 · p. 4

3 The OLiVES Dataset

OLiVES pairs synchronized outdoor low-light and normal-light video capture with registration and VOS annotations, and includes varied scenes, motion, and degradation levels. Its scope is limited by daytime ND-filter simulation, a single geographic region, tripod-based capture, and some unverified training masks.

3.1 Dataset Collection

  • A beam-splitter rig and genlocked cinema cameras capture synchronized normal-light references and ND-filtered low-light video across varied outdoor scenes.The dataset includes wildlife, buildings, pedestrians, vehicles, occlusions, and object interactions.Source: §3.1 · pp. 4–5

3.2 Dataset Registration

  • Registration adapts to ND-filter severity, using dense warping for ND1.2 and ND1.5 but RANSAC homography for ND1.8.RoMa is finetuned on synthetic low-light pairs with a contrastive objective; low-light frames remain unchanged while normal-light references receive minor geometric corrections.Source: §3.2 · pp. 5–6

3.3 VOS Benchmark Setup

  • VOS clips span 40–100 frames, include emerging and disappearing objects, and use SAM2-generated masks with partial training and validation refinement and manually annotated test masks.The train/validation/test ratios are 40%/30%/30%; the test split matches the LLVE test split.Source: §3.3 · pp. 6–7

3.4 Dataset Statistics

  • OLiVES contains 137 sequences and 25,254 frames; low-light intensities span 0.01–0.27, and noise means rise from 5.25 to 8.35 as ND stops increase.ND1.2/ND1.5/ND1.8 shares are 28.7%/32.1%/39.2% in training and 28.8%/28.1%/43.1% in testing.Source: §3.4 · p. 7
  • Objects move in all directions, with 8.3-pixel average inter-frame displacement and variance 139.6; object classes and sizes are also summarized.Whole-image displacement averages 7.7 pixels with variance 275.7.Source: §3.4 · pp. 7–8

3.5 Limitations

  • The dataset uses daytime outdoor scenes with ND-filter-simulated low light, one geographic region, and tripod-based capture; some training masks lack human verification.Simulated low light may differ from real footage in noise and color characteristics.Source: §3.5 · p. 8

4 Experiments

On OLiVES, image-based enhancement methods generally score higher on PSNR and VMAF, while video-based methods often provide stronger temporal consistency. VOS methods [8] and [37] perform strongest on OLiVES and LLE-VOS, but segmentation detail remains limited.

4.1 Implementation Details

  • Experiments use shared GH200 nodes; LLVE methods follow original training settings, while VOS methods are finetuned under a common ResNet-50 setup except OASIS.The estimated compute total is approximately 500 GPU hours.Source: §4.1 · p. 9

4.2 Low-Light Video Enhancement

  • Image-based methods generally achieve higher PSNR and VMAF, while video-based methods often have lower tLPIPS and competitive WE but reduced contrast; no method consistently excels across quality and temporal measures.Unsupervised and unpaired methods fall behind supervised approaches across metrics.Source: §4.2 · p. 9

4.3 Video Object Segmentation

  • Methods [8] and [37] show the strongest VOS performance on OLiVES and LLE-VOS, while STCN [7] performs poorly on LLE-VOS.The authors suggest STCN’s earlier architecture struggles to generalize across domains.Source: §4.3 · p. 9
  • The two-stage VOS evaluation applies OLiVES-trained LLVE methods before off-the-shelf Cutie inference on the LLE-VOS validation set.The experiment assesses the effectiveness of OLiVES training data for enhancement.Source: §4.3 · p. 10, Table 6 · p. 10
  • VOS methods detect objects well without OLiVES finetuning, but inaccurate boundaries and missing fine details limit segmentation quality for detail-critical applications.The paper contrasts detection-oriented uses such as surveillance with film post-production, augmented reality, and robotics manipulation.Source: §4.3 · p. 10

5 Conclusion

OLiVES combines low-light video enhancement and video object segmentation in one benchmark. Its data include varied illumination, motion, textures, and VOS challenges that existing datasets often lack.

5 Conclusion

  • OLiVES supports both low-light video enhancement and video object segmentation in a unified low-light video benchmark.Source: §5 · p. 10
  • Its data include complex motion, rich natural textures, and a wide range of illumination conditions.The paper specifically notes animal-fur textures.Source: §5 · p. 10
  • For low-light VOS, OLiVES includes occlusions, emerging objects, and motion-blurred appearances without relying on auxiliary event data.Source: §5 · p. 10
  • The benchmark is intended to support research that jointly advances perceptual quality and semantic understanding.Source: §5 · p. 10

A Details of RoMA finetuning

RoMa fine-tuning combines synthetic low-light degradation with feature robustness training. On MegaDepth-1500, the version with InfoNCE achieves the best reported results across degraded and non-degraded settings.

Synthetic low-light data

  • Training randomly degrades image pairs using parameter sets that model exposure, blur, and noise; Φ0 is the identity mapping.The parameters are estimated from real low-light frames, and the sets account for distinct brightness and noise distributions from different ND filters.Source: A Details of RoMA finetuning · p. 13

Degradation-augmented training

  • Because degraded inputs substantially reduce fine-feature performance, the method fine-tunes RoMa’s fine encoder Ffine and refinement module R, not its frozen coarse encoder.The ablation replaces coarse or fine features with features extracted from degraded inputs.Source: A Details of RoMA finetuning · p. 13, Table 7 · p. 14

Degradation-invariant feature learning

  • An additional degraded view supports degradation-invariant feature learning, while its features are used for training robustness rather than final matching.The method uses projected features and contrastive learning, with VICReg enforcing feature consistency and preventing representational collapse.Source: A Details of RoMA finetuning · p. 14
  • InfoNCE is used to improve instance-level discriminability, and both InfoNCE and VICReg losses are added to the original model loss.The VICReg description defines batch standard deviation and covariance terms for projected features.Source: A Details of RoMA finetuning · p. 14

Results

  • InfoNCE variant III achieves the best results across all degraded and non-degraded MegaDepth-1500 settings, and is selected for OLiVES registration.Training and testing used one NVIDIA H100 GPU; training took approximately 22 hours.Source: A Details of RoMA finetuning · p. 14

B LLVE Qualitative Results

The LLVE evaluation examines methods trained on OLiVES qualitatively, including their outputs on other low-light datasets. The reported visual comparisons show improved brightness and detail restoration.

B LLVE Qualitative Results

C Details of VOS Annotation Process

The OLiVES test-set VOS labels were produced through manual annotation and multiple review stages. Annotators used both normal- and low-light frames, with software controls to inspect masks under low light.

B Details of VOS Annotation Process

D Further VOS Experiments

The cross-domain comparison evaluates Cutie pretrained on DAVIS 2017 and YouTube-VOS, or fine-tuned on LLE-VOS or OLiVES, using a low-light MOSEv2 video. In the example, OLiVES-fine-tuned Cutie correctly produces no masks when people are occluded.

D Further VOS Experiments

  • The comparison tests pretrained Cutie weights from DAVIS 2017 and YouTube-VOS against models fine-tuned on LLE-VOS and OLiVES.Inference uses an example low-light video from MOSEv2.Source: D Further VOS Experiments · p. 16
  • Figure 9 presents examples of difficult segmentation scenarios in OLiVES.Source: Figure 9 · p. 17
  • On the MOSEv2 example, OLiVES-fine-tuned Cutie correctly produces no masks when people are occluded.Pretrained Cutie masks distant people despite foreground occlusion; LLE-VOS fine-tuning handles occlusion better but still masks a person after they leave the scene.Source: D Further VOS Experiments · pp. 16–17

E Investigation of High VOS performance

The investigation finds that numerically high VOS performance reflects an unbalanced object-size distribution and model bias toward large objects. Scores vary by size, while fine-grained textures and occlusions remain difficult for Cutie.

E Investigation of High VOS performance

  • Large objects score significantly higher, while small- and medium-object performance varies because both size groups have low counts.The analysis uses pretrained Cutie and groups objects by size to report J &F, J, and F.Source: E Investigation of High VOS performance · p. 17
  • Even when trained on OLiVES, Cutie struggles with animals’ fine-grained fur textures and with occlusions.The resulting precision may suffice for surveillance but falls short of post-production needs for finely detailed masks.Source: E Investigation of High VOS performance · p. 17
  • Numerically high VOS performance reflects the dataset’s unbalanced size distribution and models’ bias toward large objects.Source: E Investigation of High VOS performance · p. 17

Related papers

Loading 2610.20501v1…