Source-linked AI summary
Night-to-Day Image Translation for Retrieval-based Localization
Asha Anoosheh, Torsten Sattler, Radu Timofte, Marc Pollefeys, Luc Van Gool
TL;DR
Day–night image retrieval remains difficult because domain shifts and the lack of diverse paired training data hinder robust comparison. The paper proposes ToDayGAN, which translates nighttime images into a daytime representation before retrieval-based localization, achieving a 2.65x gain at the 5m/10° threshold over the best DenseVLAD result.
Problem
Day–night image retrieval for visual localization is hindered by domain shifts and insufficient diverse, corresponding training data.
Method
ToDayGAN translates nighttime driving images into a daytime representation and feeds the output to an existing image comparison tool.
Results
2.65x gain at the 5m/10° threshold over the best DenseVLAD result, with 0.5m/5° accuracy increasing from 3.4% to 9.1%.
Takeaways & Limitations
The approach significantly outperforms previous work for localizing nighttime queries against daytime images on a challenging benchmark.
Abstract
from arXiv · showhide
Visual localization is a key step in many robotics pipelines, allowing the robot to (approximately) determine its position and orientation in the world. An efficient and scalable approach to visual localization is to use image retrieval techniques. These approaches identify the image most similar to a query photo in a database of geo-tagged images and approximate the query's pose via the pose of the retrieved database image. However, image retrieval across drastically different illumination conditions, e.g. day and night, is still a problem with unsatisfactory results, even in this age of powerful neural models. This is due to a lack of a suitably diverse dataset with true correspondences to perform end-to-end learning. A recent class of neural models allows for realistic translation of images among visual domains with relatively little training data and, most importantly, without ground-truth pairings. In this paper, we explore the task of accurately localizing images captured from two traversals of the same area in both day and night. We propose ToDayGAN - a modified image-translation model to alter nighttime driving images to a more useful daytime representation. We then compare the daytime and translated night images to obtain a pose estimate for the night image using the known 6-DOF position of the closest day image. Our approach improves localization performance by over 250% compared the current state-of-the-art, in the context of standard metrics in multiple categories.
I. INTRODUCTION
Visual localization depends on comparing images across changing conditions, but domain shifts and insufficient paired training data make day–night matching difficult. The paper uses unpaired night-to-day translation and task-focused discriminators to improve retrieval-based localization.
- Place recognition can support localization by identifying previously observed locations from camera images.
- Changing weather and lighting create a domain gap that hampers comparison between traversal images.
- Directly learning robust image comparison would require very large, diverse, position-matched datasets that are difficult to gather or generate.
- Unpaired image-to-image translation learns domain changes from separate collections without requiring corresponding images of the same place.
- The proposed method translates nighttime driving images toward daytime appearance before applying an existing image comparison tool.
- Task-specialized discriminators encourage translated outputs to retain properties emphasized by the comparison tool, and the approach outperforms prior state-of-the-art methods on a challenging benchmark.
II. RELATED WORK
Image-to-image translation maps inputs between visual domains, while CycleGAN provides an unpaired formulation using adversarial and cycle-consistency objectives. The paper adopts ComboGAN as its two-domain base and modifies it for the localization task.
- A. Image-to-Image Translation: Translation models produce an output conditioned on an input image rather than sampling an unrelated image from a probability distribution.
- A. Image-to-Image Translation: CycleGAN performs unpaired translation with generators and discriminators for both domain directions, trained using adversarial and cycle-consistency losses.
- A. Image-to-Image Translation: Later models introduced shared feature spaces, separate color and texture discriminators, and translation across n domains.
- A. Image-to-Image Translation: Figure 1 depicts one generator-training pass from domain A to B, repeated symmetrically for B to A.
- A. Image-to-Image Translation: The paper uses ComboGAN as its base because it is equivalent to CycleGAN for two domains, then modifies it incrementally and compares it with UNIT.
B. Place Recognition and Localization
Place recognition identifies locations, whereas visual localization estimates camera pose; image retrieval supports localization by matching a query to a posed reference image. VLAD-family descriptors provide the image representations used in this comparison lineage.
- Place recognition identifies a real-world location, while visual localization estimates camera location and sometimes orientation relative to a map.
- Image-retrieval localization finds the most similar reference image with known pose, so camera-pose variation is useful rather than invariant.
- VLAD aggregates local-descriptor residuals around clustered visual vocabulary centers into a compact whole-image representation.
- DenseVLAD extracts 128-dimensional RootSIFT descriptors on a regular grid, producing a 128k-dimensional descriptor before PCA projection.
- NetVLAD makes VLAD differentiable by combining a feature extractor with a VLAD-Core module that performs soft clustering.
C. Image Translation for Visual Localization
ToDayGAN addresses domain shift in day-night visual localization by translating night images into a matching-relevant daytime representation before DenseVLAD retrieval. Its task-specialized discriminators separately model appearance components and multiple scales, improving localization performance.
- Image Translation for Visual Localization: ToDayGAN translates nighttime images into a daytime representation, then uses DenseVLAD nearest-neighbor matching to approximate their poses from daytime references.The pose estimate is inherited from the nearest matching daytime image.
- Image Translation for Visual Localization: The model is based on ComboGAN, which is equivalent to CycleGAN for two domains, and is incrementally modified for day-night localization.The two-domain training procedure remains equivalent to CycleGAN’s.
- Image Translation for Visual Localization: Each domain discriminator uses separate grayscale, blurred-RGB, and xy-gradient inputs to focus on texture, color, and gradients.The three discriminator losses are averaged equally.
- Image Translation for Visual Localization: Adding specialized discriminators produced a significant performance boost, while including gradient magnitudes and orientations worsened performance by about 2× relative to gradients alone.The model also produces discriminator decisions at multiple scales to encourage consistency in low- and high-level image statistics.
- Image Translation for Visual Localization: The xy-gradient discriminator emulates DenseVLAD’s SIFT-like gradient extraction and encourages translated images to contain matching-relevant features.Its differentiable operations reproduce grayscale downsampling and horizontal/vertical gradient filtering.
- Image Translation for Visual Localization: The Relativistic Loss changes discrimination from absolute realness to whether real inputs are more real than fake inputs, with the stated goal of stabilizing training.This formulation is presented as an adaptation of ComboGAN’s least-squares GAN loss.
A. Dataset
The experiments use Oxford RobotCar and its RobotCar Seasons variant, which provide repeated traversals, varied illumination and weather, and accurate reference and query poses. The evaluation intentionally reuses daytime references during training and testing while separating night inputs.
- Dataset: Oxford RobotCar contains multiple video traversals of the same 10km Oxford route collected over a year with varied lighting, time of day, and weather.Three cameras mounted on the vehicle provide left, right, and rear views.
- Dataset: RobotCar Seasons uses 6,954 daytime camera triplets as references, 438 nighttime triplets as queries, and 440 nighttime-rain images to test transfer without rain training.The reference and query images have accurate camera poses.
- Dataset: The daytime reference set is used during both ToDayGAN training and testing, while nighttime images are separated across training and testing.This matches the instance-level setting in which reference scenes are available at inference.
- Dataset: The dataset subsets derive from larger video sequences that were subsampled after constructing a 3D map from vehicle traversals for pose estimation.Daytime mapping used inertial navigation and 3D visual tracking, with LIDAR needed for ground-truth.
B. Training Setup
Training and evaluation use DenseVLAD-based retrieval on RobotCar Seasons under standard pose-error thresholds, with comparisons against raw, histogram-equalized, translated, and state-of-the-art baselines. The reported baseline pattern favors image-based methods for night queries.
- Training Setup: Training generally uses rear-view images resized to 286 × 286 and randomly cropped to 256 × 256, while 512 × 512 inputs use 384 × 384 crops.Inference uses the pre-crop size because the architecture is fully convolutional.
- Training Setup: DenseVLAD uses 128 pretrained cluster centers and projects its VLAD vectors to 4096 dimensions with PCA before comparison.The default DenseVLAD SIFT extraction scales are n ∈{4, 6, 8, 10}.
- Training Setup: Evaluation reports the percentage of queries within 5m/10°, 0.5m/5°, and 0.25m/2° pose-error thresholds.RobotCar Seasons enables direct comparison with state-of-the-art methods.
- Training Setup: The comparison includes direct DenseVLAD matching, histogram equalization applied to queries or both image sets, and out-of-the-box CycleGAN and UNIT.These baselines are reported for night-query evaluation.
- Training Setup: On night queries, image-based approaches achieved the highest accuracies, with DenseVLAD best overall and NetVLAD next among the listed methods.Structure-based ActiveSearch and CSL performed very poorly on this query set.
V. RESULTS
The results section evaluates model configurations through ablations, discriminator choices, training inputs, and dual evaluation, with Table III serving as the central ablation study.
- Table III reports localization rates across configurations to measure the effects of model modifications and variables.
- Table IV evaluates selected ToDayGAN configurations with NetVLAD to test whether translated images generalize to another comparison method.
- The ablations compare separate discriminator arrangements with a unified discriminator using color, luminance, and gradient inputs.
- “Rel.-Loss” denotes the relativistic discriminator loss, while “L/R” indicates training with left and right camera images.
- Dual evaluation augments retrieval by comparing references with both original and horizontally flipped query images, using the smaller resulting distance.
VI. DISCUSSION
The discussion reports strong baseline comparisons, robustness on Night-Rain queries, and different behavior across localization thresholds and retrieval methods. It also notes that comparison with one prior method was not fair because of mismatched training data and camera intrinsics.
- A. Comparing results with baselines: Night-Rain queries achieve nearly identical absolute accuracies to the original Night queries, with higher values at the two stricter thresholds.
- A. Comparing results with baselines: UNIT performs notably at 5m/10° but poorly at stricter thresholds, where blurry outputs appear to impair fine-scale localization.Its higher-level details remain sufficient for localizing a general area.
- A. Comparing results with baselines: NetVLAD obtains about the same improvement boost after translation as direct NetVLAD, suggesting both methods rely largely on similar, mostly gradient-based characteristics.
- A. Comparing results with baselines: A fair comparison with [19] was unavailable because its model used different training data and camera intrinsics, despite inference on the query images.
B. Impact of modifications to ComboGAN
The experiments show that specialized, separate discriminators and several training choices substantially improve ToDayGAN’s localization performance. The resulting image-translation system outperforms prior work on nighttime-query localization.
- Adding the gradient discriminator nearly doubles accuracies, while the luminance discriminator provides a smaller improvement.These effects are reported in the discriminator-combination ablation.
- RGB processing improves performance over grayscale, while a unified discriminator performs considerably worse than independent discriminators.The authors attribute the RGB benefit to regularization and report no clear explanation for the unified-discriminator disadvantage.
- The Relativistic-Discriminator loss, Dual-Evaluation procedure, and left/right training images each improve results.
- Multi-scale discriminators improve stricter-threshold results over vanilla CycleGAN, suggesting better handling of fine-grained details.The comparison changes only the discriminator scheme between the Table II and Table III entries.
- ToDayGAN significantly outperforms previous work for localizing nighttime queries against daytime images.
- Partitioning discriminator responsibilities across image aspects better encourages those aspects in generated outputs.The proposed future direction extends this idea to multiple discriminators focused on distinct features.
APPENDIX
The appendix provides visual examples of the datasets, translated images, discriminator variants, and the ComboGAN generator and discriminator specifications.
- Table VI specifies ComboGAN generator and discriminator layers using abbreviations for neurons, kernel size, stride size, transposed convolution, and residual basic blocks.
- Figure 4 presents sample images from the Daytime, Night-training, Night-query, and Night-Rain query sets.
- Figure 5 shows additional visual results for night-to-day translation using ToDayGAN.
- Figure 6 compares daytime translations of real night images under C, CL, CLG, and C+L+G discriminator setups.The images use 286 × 286 resolution.