Adversarial Training for Adverse Conditions: Robust Metric Localisation using Appearance Transfer

Motivation

• Appearance changes such as illumination and seasons degrade keypoint matching, often requiring separate maps for sunny, snow, dawn, dusk, rain, and night conditions.Mapping every condition is time consuming, expensive, and may be impossible; adverse-condition data can also reduce map quality.Source: §I · p. 1

Contribution

• The core contribution is an adversarially trained network that synthesizes images of the same location under different conditions while incorporating differentiable feature detection and description.The approach introduces a loss based on SURF detector response maps and dense descriptor maps.Source: §I · p. 1

Training

• Stage 1 trains bidirectional image synthesizers on unaligned source and target traversals using cycle consistency, discriminator, detector, and descriptor losses.The method selects a complete source traversal and a small portion of a target traversal without requiring image alignment.Source: §I · p. 1
• Stage 2 fine-tunes the synthesizers on a small set of metric-6DoF-aligned image pairs using L1 losses on feature detector and descriptor outputs.The aligned subset supplies pixel correspondence unavailable to the unsupervised first stage.Source: §I · p. 1

Localization pipeline

• The resulting synthetic images feed a stereo localization pipeline that performs place recognition and then sparse keypoint matching to estimate metric pose against retrieved map frames.Results are benchmarked against RTK-GPS ground truth on the Oxford RobotCar Dataset.Source: §I · pp. 1–2

Synthetic views and localization

• Earlier synthetic-view methods address viewpoint or seasonal appearance changes, but reported poor alignment or degraded point features prevent metric localization.These methods include Google Streetview synthesis and superpixel-vocabulary translation.Source: §II · p. 2

Appearance transfer

• Appearance-transfer approaches include local affine color transforms, neural style transfer, and unsupervised image-to-image translation based on paired generators and cycle consistency.The related translation literature includes and VAE-GAN methods using unaligned images.Source: §II · p. 2

Paper’s approach

• This paper builds its detection and matching pipeline on SURF and trains generators in two stages: unpaired cycle-consistent transfer followed by fine-tuning on aligned data.The second stage separately fine-tunes the generators using well-aligned image pairs.Source: §III · p. 2

Architectures

• The method’s supporting architecture includes a cycle-consistency network for unregistered appearance transfer and a fine-tuning network for detector- and descriptor-layer alignment.The generator uses a UResNet design, while the discriminator is a five-layer CNN producing a spatial real/fake map.Source: §III · pp. 2–3, Figure 2 · p. 3, Figure 3 · p. 3

SURF response construction

• For each scale, SURF approximates the Hessian determinant by combining convolutional response maps for X, Y, and diagonal second-order derivatives.The response maps are denoted Lxx(σ), Lyy(σ), and Lxy(σ).Source: A. SURF Detector Response Map · pp. 3–4
• The detector response is computed as det(Happrox) = Lxx ◦Lyy −0.81 ∗Lxy ◦Lxy.The operation uses the Hadamard product between corresponding response maps.Source: A. SURF Detector Response Map · p. 4

Descriptor construction

• The dense descriptor method uses a fast convolutional construction adapted from OpenSURF so descriptor losses remain differentiable.The method computes descriptors for each of N selected scales.Source: B. Dense SURF Descriptors · p. 4
• For each scale, the implementation precomputes neighbour offsets, scale-specific Gaussian weights, neighbourhood weights, and HAAR-like X/Y box filters.These components organize the 81 sampled responses used for each descriptor.Source: B. Dense SURF Descriptors · p. 4
• The input is convolved with HAAR filters, and 81 weighted wavelet-response copies are stacked before being offset across each descriptor neighbourhood.The operations are applied separately for the X and Y directions.Source: B. Dense SURF Descriptors · p. 4

Descriptor output

• The method sums raw and absolute responses along the stack direction, applies neighbourhood-specific weights, and stacks four resulting matrices per neighbourhood.The final descriptor is a normalized 64-layer H ×W matrix stack for each scale.Source: B. Dense SURF Descriptors · p. 4

Training efficiency

• Training computes SURF losses only over the first five scales because most features are detected below scale 10 without significant observed performance loss.The authors suggest smaller neighbourhoods may experience more uniform appearance changes across conditions.Source: B. Dense SURF Descriptors · p. 5

C. Losses

• Cycle consistency is enforced both between input and reconstructed images and between their SURF detector and dense descriptor maps.Source: C. Losses · p. 5
• The generators are trained with adversarial, reconstruction, SURF detector, and dense descriptor losses, while discriminators have their own objective.The complete generator objective is combined with discriminator and fine-tuning objectives.Source: C. Losses · p. 5
• The fine-tuning stage uses aligned target images together with synthetic and input images to define additional losses.Source: C. Losses · p. 5

IV. EXPERIMENTAL SETUP

• Six Oxford RobotCar traversals collected up to one year apart form five condition pairs: day-night, day-snow, day-dawn, day-sun, and day-rain.RTK-GPS ground truth was filtered by discarding points with translation standard deviation above 25cm.Source: §IV · p. 5
• Training uses the entire day traversal and approximately 20% of the paired condition, while the remaining 80% benchmarks synthetic-image performance.Source: §IV · p. 5
• Aligned fine-tuning pairs have little or no viewpoint rotation, with small rotational differences corrected by affine warping using RTK-GPS poses.Source: §IV · p. 5

A. Training

• Each iteration first trains the discriminator on real target and previously generated images, then trains the generators to minimize Lgen.Source: A. Training · p. 5
• Adam training uses an initial learning rate of 0.0002 and batch size 1.Source: A. Training · p. 5
• The reported loss weights are λrec = 8, λdet = 2, λdesc = 2, and λadv = 1.Source: A. Training · p. 5

B. Localisation

• GAB transforms all day map frames into target-condition frames, while GBA transforms five target conditions into day-condition frames.Source: B. Localisation · p. 5
• The localisation system uses the experience-based navigation system of to evaluate synthetic images against a previously built map.Source: B. Localisation · p. 5
• Localisation proceeds with a feature-based topological localiser, RANSAC geometric verification, and nonlinear minimization of inlier reprojection errors.Source: B. Localisation · p. 5

• The stage-1 synthetic images significantly improved localisation accuracy, and stage-2 fine-tuning produced a further increase over raw and RGB-only images.Source: A. Quantitative results · p. 6
• Localisation rates improved across transformed conditions, often by a factor of 2, while metric errors were reduced.Source: A. Quantitative results · p. 6
• The best solution produced a large improvement in day-night translation and rotation accuracy compared with raw images and RGB-only outputs.Source: A. Quantitative results · p. 6, Figure 7 · p. 6
• Real-to-synthetic matching produced significantly more inliers than real-to-real matching as distance travelled increased.Source: A. Quantitative results · p. 6
• Synthetic images significantly reduced the distance travelled before re-localisation after a localisation failure, whereas RGB-only outputs offered little robustness improvement.Source: A. Quantitative results · p. 6

• Appearance transfer recovered matches in locations where raw-image matching failed or produced very few inliers.The transformed images reconstructed feature-relevant details such as window frames.Source: B. Qualitative results · p. 6, Figure 10 · p. 7
• The method's quantitative robustness is consistent with its qualitative goal of improving matching under adverse appearance changes.Source: §VI · p. 6, B. Qualitative results · p. 6
• Qualitative comparisons show more inlier matches when transformed live frames are matched against real map images under different conditions.Source: Figure 10 · p. 7
• The system processes image streams outside the localisation pipeline, allowing it to serve as an offline or online front end to existing systems.Source: §VI · p. 8

• Future work could model continuous illumination and weather changes instead of binning conditions into discrete categories.Source: §VII · p. 8
• A single generator model is proposed as an ideal way to handle those continuous condition changes.Source: §VII · p. 8