PaperlayerView exact revision on arXiv

Source-linked AI summary

EL-GAN: Embedding Loss Driven Generative Adversarial Networks for Lane Detection

Mohsen Ghafoorian, Cedric Nugteren, Nóra Baka, Olaf Booij, Michael Hofmann

arXiv:1806.05525v2cs.CV

TL;DR

Semantic segmentation can misfit tasks whose desired outputs require structure beyond independent pixel classification, often leading to complex post-processing. EL-GAN uses paired prediction-label embeddings for adversarial training, stabilizing optimization and producing more label-like TuSimple lane predictions with simpler post-processing and state-of-the-art challenge results.

  • Problem

    Many tasks formulated as semantic segmentation are not inherently pixel-wise classification problems, so independent pixel losses may not enforce the required output structure.

  • Method

    EL-GAN trains a discriminator on the source image, prediction, and ground-truth label, using an L2 distance between their prediction and label embeddings as generator supervision.

  • Results

    EL-GAN substantially stabilizes adversarial training and achieves state-of-the-art results on the TuSimple challenge without extra data or complicated hand-engineered post-processing pipelines.

  • Takeaways & Limitations

    The method produces structurally more label-like segmentation predictions while reducing reliance on problem-specific loss terms and complex post-processing.

  • Takeaways & Limitations

    For applications where uncertainty expression is needed, GAN-generated label samples may sacrifice the possibility of expressing uncertainty.

Abstract

from arXiv · show

Convolutional neural networks have been successfully applied to semantic segmentation problems. However, there are many problems that are inherently not pixel-wise classification problems but are nevertheless frequently formulated as semantic segmentation. This ill-posed formulation consequently necessitates hand-crafted scenario-specific and computationally expensive post-processing methods to convert the per pixel probability maps to final desired outputs. Generative adversarial networks (GANs) can be used to make the semantic segmentation network output to be more realistic or better structure-preserving, decreasing the dependency on potentially complex post-processing. In this work, we propose EL-GAN: a GAN framework to mitigate the discussed problem using an embedding loss. With EL-GAN, we discriminate based on learned embeddings of both the labels and the prediction at the same time. This results in more stable training due to having better discriminative information, benefiting from seeing both `fake' and `real' predictions at the same time. This substantially stabilizes the adversarial training process. We use the TuSimple lane marking challenge to demonstrate that with our proposed framework it is viable to overcome the inherent anomalies of posing it as a semantic segmentation problem. Not only is the output considerably more similar to the labels when compared to conventional methods, the subsequent post-processing is also simpler and crosses the competitive 96% accuracy threshold.

1 Introduction

Semantic segmentation can poorly represent tasks requiring global structure, motivating EL-GAN’s embedding-based adversarial training with paired predictions and labels. The method stabilizes training and produces more label-like lane predictions with simpler post-processing.

  • Motivation: Per-pixel classification does not explicitly enforce structures such as thinness, straightness, uniqueness, or inter-pixel consistency.These omissions can require scenario-specific and computationally expensive post-processing.
  • Results: Figure 1 contrasts a ground-truth label, a conventional raw prediction, and an EL-GAN prediction matching the labels’ thin-line certainty and connectivity.The comparison concerns the visual structure of lane-marking predictions.
  • Method: Training both predictions and labels simultaneously gives the discriminator more useful feedback for steering segmentation outputs toward realistic labels.This uses the available image/label pairing as supervised information.
  • Method: EL-GAN feeds the source image, prediction map, and ground-truth label to a discriminator that compares prediction and label embeddings.Its embedding loss uses an L2 distance between learned representations.
  • Results: EL-GAN substantially stabilizes adversarial training and produces structurally more label-like predictions without extra engineered loss terms or complex post-processing.The framework is presented as generic, while lane-marking experiments demonstrate its usefulness on TuSimple.

2 Related Work

Related work addresses structure-preserving segmentation through targeted losses, higher-order graphical models, adversarial training, perceptual representations, and lane-specific pipelines. EL-GAN differs by using embedding-based adversarial supervision for label-like predictions without relying primarily on handcrafted structure mechanisms.

  • Quality Preserving Semantic Segmentation: Property-targeted losses and pair-wise or higher-order CRFs have been used to preserve smoothness, topology, and neighborhood consistency.These approaches generally target lower-level consistencies and can add inference costs.
  • Adversarial Training for Semantic Segmentation: Adversarial segmentation methods use discriminator feedback to encourage higher-level semantic consistency between prediction and label distributions.This provides an alternative to explicitly engineering every desired property.
  • Perceptual Loss: Perceptual-loss methods capture higher-level image representations from a separate network when pixel-level objectives are insufficient for semantic fidelity.These methods are especially discussed for image super-resolution with available ground truth.
  • Lane Marking Detection: Lane-marking systems have combined spatial CNNs, extra vanishing-point labels, constrained perspective transformations, curve fitting, and handcrafted post-processing.The paper compares EL-GAN with related approaches evaluated on the TuSimple challenge.

3 Method

EL-GAN replaces conventional generator adversarial supervision with a distance between discriminator embeddings of predictions and paired labels. This design addresses weak and unstable feedback in baseline GAN segmentation, while empirical studies favor embedding loss for generator updates and cross entropy for discriminator updates.

  • Baseline Adversarial Training: Adversarial segmentation uses a generator loss combining a pixel-wise fitting term with a weighted adversarial term.The generator produces prediction maps from input images, while the discriminator evaluates their plausibility.
  • Baseline Adversarial Training: Baseline adversarial training is unstable because generator and discriminator updates interact in a minimax game and discriminator feedback may not improve generator outputs.The conventional loss also fails to exploit paired image/label information.
  • Embedding Loss: EL-GAN defines adversarial supervision as an embedding distance between discriminator representations of the generated prediction and the ground-truth label.The embeddings are extracted from a discriminator layer given the prediction and source image.
  • Embedding Loss: The generator minimizes the embedding difference while the discriminator learns discriminative embeddings between real labels and generated predictions.Figure 2 contrasts this generator-training setup with conventional GAN training.
  • Discriminator Updates: Although discriminator updates can be formulated with a similar embedding-loss idea, empirical studies found cross entropy gave better results for updating discriminator parameters.The embedding-loss change is therefore applied primarily to generator updates in the reported setup.

4 Experimental Setup

The evaluation focuses on autonomous-driving lane marking detection using TuSimple data, official metrics, and simple polyline post-processing. EL-GAN uses convolutional generator and discriminator networks trained with adversarial and embedding-related objectives.

  • 4.1 Evaluation Datasets and Metrics: The study evaluates autonomous-driving lane marking detection on the TuSimple dataset using official accuracy, false-positive, and false-negative metrics.The dataset contains annotated highway images and polyline lane-marking labels converted into segmentation maps.
  • 4.1 Evaluation Datasets and Metrics: Validation uses one labeled sequence with 409 images, but its performance may not be fully representative because of its small size.The authors note that choosing another sequence would also reduce the already small validation set.
  • 4.1 Evaluation Datasets and Metrics: Because the networks output segmentation maps rather than required polylines, post-processing binarizes components and converts them into polylines using mean x-indices across y-indices.The basic++ variant additionally splits components when multiple non-zero sequences occur at one sampling location.
  • 4.2 Network and Training Setup: EL-GAN uses a fully convolutional U-Net-style generator based on Tiramisu DenseNet and a DenseNet patch-GAN discriminator with separate early processing paths.The discriminator concatenates image and prediction-or-label features after two dense blocks and takes embeddings after the final convolution layer.
  • 4.2 Network and Training Setup: The generator is pre-trained to convergence, while the discriminator is pre-trained for 10k iterations before alternating generator and discriminator updates.Generator training uses Adam, whereas discriminator training uses SGD for greater observed stability.

5 Results

Experiments on TuSimple compare EL-GAN with a regular CNN baseline, leaderboard methods, adversarial loss choices, and embedding locations. EL-GAN produces thinner, less noisy predictions, improves challenge performance, and shows more stable training with embedding loss and later embeddings.

  • 5 Results: EL-GAN outperforms the baseline on the validation set, particularly when using the most basic post-processing method.The basic method performs poorly for the baseline, while basic++ improves the baseline substantially.
  • 5 Results: EL-GAN produces considerably thinner, more label-like outputs with less noise than the regular CNN baseline, making post-processing easier.Figure 4 compares raw prediction maps and basic++ post-processed results using the same post-processing framework.
  • 5.1 TuSimple Lane Marking Challenge: 4th place on the TuSimple challenge leaderboard was achieved by EL-GAN, with less than half a percent separating it from the best method.Compared with the baseline, adversarial training improves accuracy by approximately 2%, decreases error by 38%, reduces false positives by more than 55%, and reduces false negatives by 30%.
  • 5.2 Ablation Studies: Embedding loss makes GAN training stable across validation-accuracy statistics and five-run F-score variation, unlike the compared adversarial-loss choices.The authors report similar behavior with other hyper-parameters.
  • 5.2 Ablation Studies: Later embedding locations produce better scores and predictions more similar to the labels than earlier locations.The study compares embeddings taken after the 3rd, 5th, and 7th dense blocks.

6 Discussion

EL-GAN improves adversarial training stability by using embedding losses and deeper representations, while producing label-like outputs that simplify lane-marking post-processing. Its output samples can better match plausible labels, but this may limit direct uncertainty expression in other applications.

  • Comparison with Other Lane Marking Detection Methods: EL-GAN achieves comparable accuracy without curve fitting, omitting the separate homography-learning and perspective-transform requirement described for a competing method.The comparison specifically concerns Neven et al.'s birds-eye-perspective post-processing pipeline.
  • Comparison with Other Lane Marking Detection Methods: Pan et al.'s spatial CNN achieves better TuSimple accuracy than EL-GAN, but the contribution of its spatial CNN versus its larger private dataset is unclear.The private dataset is described as 20 times larger than the regular TuSimple dataset.
  • Training Stability: Embedding loss for the generator makes GAN training stable and provides stronger signals by leveraging ground truth rather than only discriminator representations.Cross-entropy-based training can collapse when discriminator fake/real comprehension is not well formed, whereas embedding-space differences strictly penalize noisy productions.
  • Analysis of the Ablation Study: Deeper embedding-extraction representations improve performance, plausibly because their larger receptive fields capture higher-level qualities and consistencies.This finding comes from the second ablation study reported with Table 4 and Fig. 6.
  • GANs for Semantic Segmentation: Unlike a regular CNN's per-pixel probability output, EL-GAN produces samples resembling possible labels, including a selected lane-marking configuration when the marking is occluded.The authors argue that outputs lying on or near the manifold of possible labels can make post-processing easier and more accurate.
  • GANs for Semantic Segmentation: For applications beyond lane marking, EL-GAN's single-sample output sacrifices uncertainty expression, though repeated random runs or ensembles could restore manifold-based uncertainty modeling.The paper explicitly limits the demonstrated need for non-probabilistic output to lane-marking detection.

7 Conclusions

The paper concludes that EL-GAN preserves label-resembling qualities while stabilizing adversarial training. On TuSimple, it reports state-of-the-art results without extra data or complicated hand-engineered post-processing.

  • 7 Conclusions: EL-GAN preserves label-resembling qualities in network predictions and produces a more stable adversarial training process.These are the paper's stated conclusions about the proposed method.
  • 7 Conclusions: EL-GAN achieves state-of-the-art results on the TuSimple challenge without extra data or complicated hand-engineered post-processing pipelines.The conclusion contrasts this setup with other competitive methods.

Appendix A: Network Architecture and Training Configuration

The appendix specifies the EL-GAN network architectures, embedding location, and training configuration used for reproducibility. It covers the generator, discriminator, optimization schedules, and regularization settings.

  • Network Architecture: The generator uses a Tiramisu DenseNet with seven dense blocks, growth-rate 18, ReLU activations, He initialization, and dropout rate 0.1.Its down/up sampling paths contain convolutional layers across dense blocks.
  • Network Architecture: The discriminator uses a two-headed DenseNet whose heads concatenate after the second dense block, with growth-rate 8, ELU activations, and no dropout.Embeddings are taken after the seventh dense block.
  • Training Configuration: Training runs for 150K iterations with batch size 8 and a schedule of 300 discriminator versus 200 generator updates.The schedule is reported as (300: disc, 200: gen).
  • Training Configuration: The generator uses Adam with momentum 0.9, initial learning rate 5e-4, exponential decay, L2 scale 1e-4, and 100K iterations of pre-training.The decay power is 0.99 and decay rate is 200.
  • Training Configuration: The discriminator uses vanilla SGD with initial learning rate 1e-5, exponential decay, L2 scale 1e-5, and 10K iterations of pre-training.Its adversarial loss λ is 1; decay power is 0.99 and decay rate is 800.
Loading 1806.05525v2…