PaperlayerView exact revision on arXiv

Source-linked AI summary

From Change Captions to Change Detection: Semantic-Appearance Agreement Framework for Remote Sensing Change Detection

Yuan Qian, Jie Ma

arXiv:2609.28192v1cs.CV

TL;DR

Pixel-level change masks are costly to annotate, and binary image-level labels omit transition semantics. The paper uses change captions as task-specific supervision, combining caption-grounded semantics with appearance differences and constructing Flair-RSGen. SAAF outperforms the closest reproduced limited-supervision baselines in macro-averaged IoU and F1 on Flair-RSGen and WHU-CDC.

  • Problem

    Pixel-level change masks are costly to annotate, while binary image-level labels omit semantic transition information and leave captions’ pixel locations unresolved.

    Source: §I · p. 2

  • Method

    SAAF uses frozen TCL-DA and RGB appearance differences to create caption-consistent pseudo masks, then trains a detector conditioned on the same transition semantics.

    Source: §I · p. 2

  • Results

    SAAF outperforms the closest reproduced baselines in macro-averaged IoU and F1 on Flair-RSGen and WHU-CDC.

    Source: §I · p. 2

  • Takeaways & Limitations

    Change captions serve as sole task-specific supervision for learning change masks without manually annotated change masks.

    Source: §I · p. 2

Abstract

from arXiv · show

Remote sensing change detection (RSCD) is essential for monitoring land-cover changes and urban development. However, most methods demand pixel-level change masks, which are costly and time-consuming to annotate. Weakly supervised methods reduce this cost by using image-level change labels. Yet these labels indicate only whether a change occurs, leaving models to recover the location of the change and semantic meaning through additional and complex mechanisms. This missing information can be supplied directly by change captions, which describe what changes, what it becomes, and where it occurs. Therefore, we introduce change-caption-guided RSCD, using change captions as the sole task-specific supervision to learn change masks without manually annotated change masks. Our framework has two components: a caption-driven generation pipeline that produces bi-temporal remote sensing image pairs at scale with controlled changes matching each caption, and a change detector guided by the caption's transition semantics. The detector uses our Semantic-Appearance Agreement Framework (SAAF) to combine caption-grounded semantic responses with RGB differences for change localization, while text conditioning guides dense prediction. Experiments on our newly constructed Flair-RSGen dataset and WHU-CDC show that SAAF outperforms the closest reproduced limited-supervision baselines in macro-averaged IoU and F1 under the evaluated protocols. Code is publicly available at https://github.com/qianyuancs/SAAF.

I. INTRODUCTION

Pixel masks are costly to annotate, while binary image-level labels do not specify the transition’s semantic direction or location. The paper uses change captions as sole task-specific supervision and introduces SAAF and Flair-RSGen for caption-guided change detection.

I. INTRODUCTION

  • Pixel-mask annotation is labor-intensive because annotators must distinguish genuine transitions from seasonal or radiometric variation and delineate changed-region boundaries.Binary scene-level labels indicate whether a change exists but do not identify which category disappears or emerges.Source: §I · p. 1
  • Change captions specify source and target categories and transition direction, information absent from binary changed/unchanged labels.Existing vision-language RSCD methods generally still use pixel-level masks to optimize the change predictor.Source: §I · p. 2
  • The method formulates change-caption-guided RSCD using captions instead of manually annotated change masks for training.Captions provide source-to-target semantics and may include coarse spatial cues, but do not delineate changed pixels.Source: §I · p. 2
  • SAAF combines caption-grounded semantics with appearance changes to localize change and conditions prediction on the same transition semantics.Its frozen TCL-DA teacher combines semantic responses with temporal appearance differences to generate pseudo masks that supervise the detector.Source: §I · p. 2
  • The paper develops caption-driven bi-temporal image generation with controllable source-to-target land-cover transitions and uses it to construct Flair-RSGen.The generation method interprets source category, target category, and spatial cue to synthesize the second-temporal image.Source: §I · p. 2
  • SAAF outperforms the closest reproduced baselines in macro-averaged IoU and F1 on Flair-RSGen and WHU-CDC.Ablations verify the effectiveness of the semantic and appearance cues, their agreement mechanism, and caption-conditioned detection.Source: §I · p. 2

A. Remote-Sensing Change Detection Under Limited Supervision

The related work contrasts caption-specific transition supervision with methods that infer locations from image features or use text while retaining spatial-label training. It also situates Flair-RSGen among synthetic data approaches by describing a caption-guided pipeline that produces paired images and masks.

A. Remote-Sensing Change Detection Under Limited Supervision

  • Prior limited-supervision methods derive locations from image features or pretrained visual priors, while binary labels do not specify semantic transitions.Refinement can improve activation-map completeness or boundaries but cannot recover transition information absent from supervision.Source: A. Remote-Sensing Change Detection… · p. 2

B. Vision–Language and Caption-Aware Change Detection

C. Synthetic Bi-Temporal Data Generation

  • Other synthetic bi-temporal datasets use generated spatial labels as detector supervision, including approaches based on vision-language, diffusion, or structural priors.The cited examples include WHU-GCD, RDF-MIG, and SNV-GenCD.Source: C. Synthetic Bi-Temporal Data Generation · p. 3
  • General image-editing methods motivate local control, but a remote-sensing transition dataset must align source class, target class, edit region, and caption.The passage cites SDEdit, Blended Diffusion, and Prompt-to-Prompt as examples of guided editing methods.Source: C. Synthetic Bi-Temporal Data Generation · p. 3
  • Flair-RSGen is built with a text-guided remote-sensing image-editing pipeline in which each caption specifies a source-to-target land-cover transition.The pipeline generates the corresponding image pair and produces change masks and semantic masks for evaluation.Source: C. Synthetic Bi-Temporal Data Generation · p. 3

A. Problem Formulation and Framework Overview

The framework predicts the region of a caption-described land-cover transition from a co-registered image pair. It parses the caption into transition and spatial cues, then uses semantic–appearance agreement and caption conditioning to guide change detection and controlled pair generation.

A. Problem Formulation and Framework Overview

  • The task predicts the image region corresponding to the transition named by a sample-specific caption, and explicitly no-change captions require no separate image-level label.The caption parser identifies source and target concepts.Source: A. Problem Formulation and Framework… · p. 3
  • Flair-RSGen construction parses an instruction into semantic and spatial constraints, reassigns the selected source region to the target category, and modifies only the constrained neighborhood.Source: Figure 3 · p. 4
  • SAAF combines caption-grounded semantic responses with temporal appearance differences to create caption-consistent pseudo masks for a text-conditioned bi-temporal detector.The semantic branch can respond to unchanged instances, while the appearance branch lacks class information; their agreement retains pixels supported by both cues.Source: A. Problem Formulation and Framework… · p. 4
  • The student processes both images and their absolute difference, while ordered source and target text vectors condition its bottleneck, cross-attention, and change prediction.Auxiliary heads predict source and target classes, carrying caption direction into detection.Source: A. Problem Formulation and Framework… · p. 4
  • At inference, the student uses the image pair and caption to output a binary change map; offline pseudo-mask generation and the frozen teacher are not required.Source: A. Problem Formulation and Framework… · p. 4

1) Caption and Semantic Parsing:

  • The caption parser decomposes each instruction into source and target concepts and a spatial phrase, grounding the phrase as a coarse region and normalizing synonyms to 19 FLAIR categories.For example, “Agricultural land in the upper-right region changes to buildings” identifies the source, target, and upper-right region.Source: 1) Caption and Semantic Parsing · p. 4

2) Spatial–Semantic Constraint:

  • The spatial–semantic constraint intersects the caption’s coarse region with pixels of the source class and retains the connected component that best matches the spatial phrase as the edit region.Source: 2) Spatial–Semantic Constraint · p. 4
  • A dilated support around the change defines the editable neighborhood, which is removed from the conditioning image before editing.The passage defines ⊙ as elementwise multiplication.Source: 2) Spatial–Semantic Constraint · p. 4

3) Controlled Image Generation:

The caption-driven pipeline creates controlled bi-temporal edits from source images, while the semantic branch turns caption concepts into directed transition evidence across both dates.

3) Controlled Image Generation:

  • The generator combines latent diffusion, ControlNet, and the HySCDG prior, then preserves unchanged pixels when composing the generated image.Its inputs include the semantic maps, selected region, caption, and geographic metadata.Source: 3) Controlled Image Generation · p. 4
  • Caption source, target, and spatial cues control the transition and its location in the generated second image.Changing class phrases changes the transition; changing the spatial cue changes its location.Source: 3) Controlled Image Generation · p. 4
  • No-change pairs add hard negatives that teach the detector to reject appearance differences without a semantic transition.They are generated separately from the transition-based editing procedure.Source: 3) Controlled Image Generation · p. 4

C. Semantic Branch: Directed Caption Evidence

  • TCL-DA grounds source and target concepts in both images, producing four response maps that capture concept support across dates.The text prompts represent the caption’s source and target concepts, and the frozen mask-generation module converts similarities into soft masks.Source: C. Semantic Branch: Directed Caption… · p. 5
  • Directed semantic evidence follows the expected source decrease and target increase, while symmetric terms provide fallback when only one date’s response is reliable.The evidence averages four terms to reduce dependence on any single response map while preserving transition order.Source: C. Semantic Branch: Directed Caption… · p. 5

D. Appearance Branch and Cross-Cue Agreement

The appearance branch combines caption-grounded transition evidence with class-agnostic RGB differences, using geometric agreement to favor locations supported by both cues.

D. Appearance Branch and Cross-Cue Agreement

  • For Flair-RSGen, a 5 × 5 average filter processes the agreement score before thresholding while preserving spatial resolution.The processed score is denoted S, and the pseudo mask is then obtained from it.Source: D. Appearance Branch and Cross-Cue… · p. 6
  • Geometric agreement favors locations supported by both semantic transition evidence and RGB appearance differences.A strong response in either cue alone receives a low score when the other cue is weak.Source: D. Appearance Branch and Cross-Cue… · pp. 5–6
  • Unlike arithmetic averaging, the geometric mean prevents one strong cue from compensating for weak evidence in the other.This suppresses caption-inconsistent appearance differences and semantic responses without corresponding image variation.Source: D. Appearance Branch and Cross-Cue… · p. 6

E. Text-Conditioned Bi-Temporal Detector

The detector processes the bi-temporal images and their difference, conditions features on ordered caption semantics, and refines change predictions with auxiliary source and target supervision.

E. Text-Conditioned Bi-Temporal Detector

  • A single three-stage encoder processes the paired images and produces joint temporal features with 48, 96, and 192 channels.The input concatenates the RGB images and their absolute difference along the channel dimension.Source: E. Text-Conditioned Bi-Temporal Detector · p. 6
  • The detector projects frozen source and target text features into ordered tokens and uses them to condition bottleneck and decoder features through cross-attention.Bottleneck attention conditions low-resolution features, while decoder attention restores caption guidance after skip fusion.Source: E. Text-Conditioned Bi-Temporal Detector · p. 6
  • A refinement head uses foreground- and background-weighted features from coarse probabilities, then adds a learned residual to the coarse logits.The prediction heads produce change, source, and target outputs.Source: E. Text-Conditioned Bi-Temporal Detector · p. 6
  • The refinement path is restricted to the change head, while source and target heads provide auxiliary class supervision on pseudo foreground pixels.The semantic heads also regularize shared decoder features through auxiliary supervision.Source: E. Text-Conditioned Bi-Temporal Detector · p. 6

F. Optimization Objective

The optimization objective combines a region-normalized change-map loss with overlap and semantic-alignment terms. Its asymmetric weighting emphasizes background rejection without letting the larger background dominate foreground learning.

F. Optimization Objective

  • The asymmetric map objective assigns background a larger weight than foreground, directly penalizing over-expanded predictions without suppressing foreground through global class imbalance.Source: F. Optimization Objective · p. 6
  • When the pseudo mask has no foreground, the map loss uses full-map background BCE; when it has no background, it uses foreground BCE.Source: F. Optimization Objective · p. 6
  • For nonempty masks, Dice encourages region overlap and semantic alignment associates detected foreground with the ordered source-to-target classes.Both semantic cross-entropy terms are zero for no-change samples; those samples retain the background map term.Source: F. Optimization Objective · pp. 6–7

IV. EXPERIMENTS

Flair-RSGen supplies controlled semantic-change and no-change image pairs with captions and reference annotations. WHU-CDC provides real bi-temporal pairs, with masks withheld from training and validation.

1) Flair-RSGen:

  • Flair-RSGen contains 45,761 pairs spanning controlled transitions among 19 land-cover classes and no-change pairs with appearance variation.It includes captions, semantic maps for both dates, and reference change masks; the split is 31,482 training, 5,390 validation, and 8,889 test pairs.Source: 1) Flair-RSGen · p. 7

2) WHU-CDC:

  • WHU-CDC provides 256×256 image pairs, five captions per pair, image-level states, and binary masks; the study selects one unambiguous transition caption.Its split has 5,230 training, 650 validation, and 654 test pairs; reference masks are used only for final evaluation.Source: 2) WHU-CDC · p. 7

B. Implementation Details

The experiments use frozen TCL-DA features and retrained limited-supervision baselines under shared data splits and evaluation code. Flair-RSGen is assessed for generation quality, while image-wise macro metrics evaluate detection and no-change rejection.

B. Implementation Details

  • The detector uses a nine-channel input, frozen TCL-DA features, and a shared text-token conditioning pathway, and is trained for 30 epochs with AdamW.Training uses FP32, batch size 8, learning rate 5 × 10^-5, weight decay 10^-4, and gradient clipping at 1.0; TCL-DA remains frozen during detector training.Source: B. Implementation Details · p. 7
  • WSLCD, ACWCD, and TransWCD are retrained from released implementations on the same pairs, splits, and evaluation code, but retain their native supervision.The baselines mainly use image-level change labels, whereas SAAF receives sample-specific captions; reference masks are excluded from optimization and checkpoint selection.Source: B. Implementation Details · pp. 7–8

C. Generation Quality and Reliability of Flair-RSGen

  • Flair-RSGen lowers FID from 158.83 to 8.21 versus IP2P and raises changed-region LPIPS from 0.516 to 0.554 while lowering background LPIPS from 0.528 to 0.138.The fixed segmentation model scores 50.6% mIoU and 71.7% pixel accuracy on generated imagery, versus 21.7% and 41.3% for IP2P; real FLAIR reaches 59.9% and 78.3%.Source: C. Generation Quality and Reliability… · p. 8

D. Evaluation Protocol

  • Evaluation averages image-wise macro IoU, precision, recall, and F1, giving each pair equal weight and measuring both changed-region overlap and no-change rejection.For an empty reference mask, a metric is one only when the prediction is also empty.Source: D. Evaluation Protocol · pp. 8–9

E. Comparison With Existing and Caption-Aware Baselines

SAAF outperforms the closest reproduced limited-supervision baselines on both datasets, while matched comparisons show benefits from text conditioning and semantic–appearance agreement. Ablations indicate that the additional semantic loss has a smaller, dataset-dependent effect.

E. Comparison With Existing and Caption-Aware Baselines

  • On Flair-RSGen, SAAF reaches 73.22% macro IoU and 80.83% F1, exceeding ACWCD by 8.70 and 10.37 points; on WHU-CDC, it leads macro IoU, recall, and F1.On WHU-CDC, IoU improves by 1.66 points over WSLCD and F1 by 2.06 points over ACWCD; gains are smaller than on Flair-RSGen.Source: E. Comparison With Existing and… · p. 9
  • Matched caption-aware comparisons use the same image pairs, captions, TCL-DA checkpoint, and splits; detector rows share training and evaluation protocols.The plain detector has no caption tokens or auxiliary semantic loss, while the conditioned variant with Lsem = 0 isolates text conditioning.Source: E. Comparison With Existing and… · p. 9
  • Adding text conditioning without Lsem raises IoU/F1 from 61.83/69.40 to 72.84/80.48 on Flair-RSGen and from 17.27/20.22 to 81.59/84.31 on WHU-CDC.The pseudo labels and loss objective remain unchanged in these comparisons.Source: E. Comparison With Existing and… · p. 9

F. Ablation Studies

  • With the conditioned detector and objective fixed, agreement labels improve IoU/F1 by 6.45/6.31 points on Flair-RSGen and 0.97/1.14 points on WHU-CDC.The same improvement is not observed for plain detectors; the agreement teacher selects regions supported by both semantic and appearance cues.Source: E. Comparison With Existing and… · p. 9
  • Adding Lsem raises Flair-RSGen IoU/F1 from 72.84/80.48 to 73.22/80.83, while WHU-CDC remains nearly unchanged at 81.55/84.28.The reported additional gain from semantic supervision therefore depends on the dataset.Source: E. Comparison With Existing and… · p. 9

1) Semantic and Appearance Evidence:

  • Geometric agreement improves IoU and F1 over arithmetic fusion by 2.56 and 2.40 points, respectively, by allowing either branch to suppress unsupported responses.Semantic responses can cover nearby unchanged regions, whereas RGB differences locate appearance changes without identifying classes; agreement yields a more focused response around the vegetation edits shown.Source: 1) Semantic and Appearance Evidence · pp. 9–10

2) TCL Domain Adaptation:

SAAF combines domain-adapted semantic responses with RGB variation and caption-conditioned decoding, while its training objectives and threshold setting shape performance. The discussion highlights complementary supervision, generated-pair evaluation, and extensions for broader settings.

2) TCL Domain Adaptation:

  • Replacing TCL-DA with the original TCL checkpoint reduces IoU by 1.38 points and F1 by 1.63 points.The original model has higher precision but lower recall, while domain-adapted responses cover aerial objects more completely.Source: 2) TCL Domain Adaptation · p. 10

3) Objective Design:

  • Combining semantic alignment and Dice gives the best IoU and F1 on Flair-RSGen, but removing semantic alignment leaves WHU-CDC performance nearly unchanged.Dice improves overlap, while semantic alignment offers a modest benefit on Flair-RSGen; the semantic-alignment benefit is therefore dataset-dependent.Source: 3) Objective Design · p. 10

4) Threshold Sensitivity:

  • Across τ values from 0.15 to 0.35, macro IoU and F1 vary by only 0.54 and 0.53 percentage points, respectively.The best result at τ = 0.15 exceeds the default τ = 0.25 by only 0.17 IoU and 0.18 F1 points, so the study retains τ = 0.25.Source: 4) Threshold Sensitivity · p. 10

V. DISCUSSION

  • Geometric agreement retains regions supported by both caption semantics and RGB variation, unlike arithmetic fusion, where one strong branch can compensate for weak evidence in the other.Matched controls find agreement labels most effective when the decoder receives ordered source, target, and difference tokens; cross-attention preserves the requested transition during refinement.Source: §V · p. 10
  • Flair-RSGen provides controllable transitions and no-change hard negatives, and independent WHU-CDC evaluation shows the framework can learn from generated pairs.The discussion identifies uncertainty-aware refinement, multi-transition parsing, and broader geographic evaluation as extensions.Source: §V · p. 10

VI. CONCLUSION

  • SAAF uses frozen TCL-DA and RGB variation to construct spatial supervision, reuses ordered transition semantics through cross-attention, and trains without manually annotated change masks.The conclusion describes Flair-RSGen as a caption-driven source of controlled transitions and realistic no-change variations, paired with captions, semantic maps, and change masks.Source: §VI · p. 10
Loading 2609.28192v1…