Source-linked AI summary
Multi-Task Partially Supervised Learning for Super-Resolution and Semantic Segmentation on Earth Observation data
Hoàng-Ân Lê, Minh-Tan Pham, Solange Lemai-Chenevier, Daniel Greslou
TL;DR
Jointly learning super-resolution and semantic segmentation for Earth-observation data is constrained by the need for both annotations on each training example. This paper studies partial supervision with shared, sequential, and hybrid designs, adding a re-projection loss for segmentation-annotated inputs. The hybrid reports higher super-resolution quality and similar segmentation performance with a slight numerical benefit over SOTA sequential methods.
Problem
Jointly learning super-resolution and semantic segmentation is limited by the impractical cost of requiring both task annotations for every training example.
Source: §1 · p. 1Method
The paper studies partial supervision for the two tasks and proposes a hybrid architecture with a re-projection loss to constrain super-resolution during segmentation training.
Source: §1 · pp. 1–2Results
The hybrid achieves higher super-resolution quality and similar semantic segmentation performance, with a slight numerical benefit, compared with SOTA sequential methods.
Source: §4 · p. 5Takeaways & Limitations
Abstract
from arXiv · showhide
Super-resolution and semantic segmentation are known to benefit one another, especially in the Earth observation context. However, learning both tasks in a joint model often requires both task annotations, which is impractical and expensive. In this paper, we study the multi-task partially supervised learning paradigm for both tasks, where each example is assumed to have only a single-task annotation. To that end, we examine two multi-task architectural variations, the sequential and shared variants, and then propose a hybrid variant and a re-projection loss to benefit from the shared representation and enforce image quality of super-resolution when training with semantic segmentation. Experiments show favorable results compared to the SOTA sequential variant. Source code will be published at https://github.com/lhoangan/munera.
1. INTRODUCTION
Earth-observation imagery varies across geography, sensors, and acquisition altitudes, while scene detail matters for semantic applications. The paper addresses the cost of jointly training super-resolution and segmentation by studying partial supervision and proposing a hybrid architecture with a re-projection loss.
1. INTRODUCTION
- Earth-observation data vary in appearance across geography, sensors, and acquisition altitudes, and scene detail is important for semantic applications.The cited examples contrast rural and urban areas and aerial and satellite platforms.Source: §1 · p. 1
- Super-resolution can recover high-resolution details, but the problem is ill-posed because different degradation processes can map many high-resolution images to the same low-resolution input.Categorical information has been used to constrain reconstruction and improve textural and photo-realistic qualities.Source: §1 · p. 1
- Joint super-resolution and segmentation training is limited by the impractical requirement for every example to carry annotations for both tasks.Creating such datasets multiplies annotation effort and requires task synchronization.Source: §1 · p. 1
- Multi-task partial supervision uses one task annotation per training example and can combine datasets annotated for different tasks in one pipeline.The paper applies this setting to Earth-observation super-resolution and segmentation, while addressing the challenge that missing joint labels hinder joint optimization.Source: §1 · p. 1
- The proposed hybrid variant combines shared features with super-resolution adapted for a downstream task.The paper pairs it with a re-projection loss intended to enforce super-resolution image quality.Source: §1 · p. 2
2. PARTIAL MULTI-TASK LEARNING
The method formulates joint super-resolution and semantic segmentation for examples annotated for only one task, then compares shared, sequential, and hybrid decoder designs. Its objectives combine task losses and a re-projection constraint for segmentation-annotated inputs.
2.1. Problem formulation
- Each training example supplies either a low-/high-resolution image pair for super-resolution or a different low-resolution image with its semantic annotation.The network is defined to predict both a semantic mask and an image upsampled by scale factor c.Source: §2.1 · p. 2
2.2. Network architecture
- The hybrid head adapts shared features and adds them to super-resolved features, while the sequential design feeds super-resolved images directly to the segmentation network.The sequential design uses segmentation loss to guide super-resolution; the three variants share an encoder and two task decoders.Source: §2.2 · p. 2
2.3. Optimizing objectives
- Super-resolution uses pixel and perceptual losses, while segmentation uses cross-entropy plus a re-projection loss on outputs from segmentation-annotated inputs.The re-projection loss downsamples the super-resolved image to the low-resolution domain and compares it with the input using L1 distance; task batches can be alternated or optimized separately by epoch.Source: §2.3 · p. 2
3. EXPERIMENTS
The experiments compare shared and hybrid partially supervised models with sequential state-of-the-art approaches on ISPRS and Pléiades data. The proposed variants improve super-resolution quality, perform comparably on segmentation, and are evaluated with an ablation of the GAN and re-projection losses.
3.1. Setups
- The evaluation uses ISPRS Potsdam for super-resolution training and Vaihingen for segmentation and testing, alongside real Pléiades imagery and synthetic Pléiades super-resolution pairs.PSNR and SSIM measure super-resolution and mIoU measures segmentation; Pléiades super-resolution is qualitative because paired high-resolution references are unavailable. Models are trained for 200K iterations with early stopping at the highest mIoU.Source: §3.1 · pp. 2–4, §3.2 · p. 4
3.2. Partial multi-task learning
- The shared and hybrid variants outperform state-of-the-art methods on SSIM and achieve lower distortion (PSNR); qualitative results show richer textures and sharper, more photometrically consistent Pléiades outputs.On Pléiades, the proposed results are closer to bicubic photometric consistency, while the state-of-the-art outputs show color leaks and are slightly blurrier.Source: Table 1 · p. 3, §3.2 · pp. 4–5, Figure 3 · p. 4
- On Vaihingen segmentation, hybrid and SR4IR outperform the other two methods in part because they recognize the absent Background class; both proposed variants otherwise perform similarly to state-of-the-art methods, with slight numerical benefits on both datasets.The absent class is disregarded from mIoU calculation when recognized; false positives from other methods cause the class to be included.Source: Table 2 · p. 3, §3.2 · p. 5
3.3. Ablation study
- The ablation finds that re-projection loss guarantees super-resolved image fidelity and reduces distortion, whereas GAN loss destabilizes training and harms image quality.The ablation trains all three variants from scratch with the same losses, so its performance is not directly comparable to the preceding state-of-the-art comparison.Source: §3.3 · p. 5, Table 3 · p. 5
4. CONCLUSION
The paper studies joint super-resolution and semantic segmentation for Earth-observation imagery under multi-task partial supervision. Its hybrid architecture combines shared representation with semantic-guided sequencing, and performs favorably against state-of-the-art sequential methods.
- Against state-of-the-art sequential methods, the hybrid approach achieves higher super-resolution quality quantitatively and visually, with similar segmentation performance and a slight numerical benefit.Source: §4 · p. 5
- The study uses multi-task partial supervision to jointly learn super-resolution and semantic segmentation on Earth-observation imagery.Source: §4 · p. 5
- The hybrid architecture combines shared representation with the semantic-guided sequential variant.Source: §4 · p. 5