PaperlayerView exact revision on arXiv

Source-linked AI summary

PoE-Fuse: Precision-Weighted Expert Fusion for Bi-Temporal Change Understanding

Haruki Watase, Shunya Nagashima, Takayuki Nishimura

arXiv:2609.37485v1cs.CVcs.LG

TL;DR

Bi-temporal change understanding supports disaster response and environmental monitoring, but adapting strong vision-language models often requires costly, unstable fine-tuning or reinforcement learning. PoE-Fuse aligns frozen foundation experts on a shared spatial grid and fuses them by learned per-cell precision. Its shared trunk reaches a 59.2% mean F1 versus 40.7% for an instruction-tuned temporal assistant and surpasses dedicated change-detection models retrained under the same protocol and budget.

  • Problem

    Bi-temporal change understanding must distinguish small, local genuine changes from spurious changes caused by illumination, season, and residual misregistration, across tasks with different target categories.

    Source: §1 · p. 2

  • Method

    PoE-Fuse aligns frozen heterogeneous foundation experts to a shared geometric grid and fuses their features as Gaussian observations of a latent scene state using learned per-cell precision.

    Source: §1 · p. 2, §4.2 · p. 7

  • Results

    59.2% three-task mean F1 versus 40.7% for an instruction-tuned temporal vision-language assistant, while surpassing all five evaluated dedicated change-detection baselines retrained under the same protocol and budget.

    Source: §6 · p. 14

  • Takeaways & Limitations

    A single shared trunk can solve change detection, building localization, and damage classification while training only about 60M parameters on cached features.

    Source: §6 · p. 14

    Inputs are limited to co-registered pairs at a single resolution, and inference still runs every frozen expert, with deployment cost dominated by the largest backbone.

    Source: §6 · p. 14

Abstract

from arXiv · show

Bi-temporal change understanding, which localizes and characterizes what changed between two satellite images, is central to disaster response and environmental monitoring, spanning change detection, building localization, and damage assessment. Strong vision-language models address these tasks, but adapting them typically requires full fine-tuning or reinforcement learning, which is costly and unstable. We propose PoE-Fuse, a parameter-efficient framework that instead composes frozen foundation experts for geometry, grounding, and language, resampling their features onto a shared spatial grid and training only a lightweight fusion trunk. PoE-Fuse treats the aligned features as Gaussian observations of a latent scene state and fuses them by learned per-cell precision. This product-of-experts estimator strictly generalizes uniform summation and scalar gating, and extends to change fields by composing the precisions of the two timestamps. A single shared trunk solves the three tasks at once, reaching a mean F1 of 59.2%, compared with 40.7% for an instruction-tuned temporal vision-language assistant, and surpassing dedicated change-detection models retrained under the same protocol and training budget.

1 Introduction

Bi-temporal change understanding supports disaster response and environmental monitoring, but small genuine changes are hard to distinguish from spurious changes. PoE-Fuse addresses gaps in existing approaches by fusing frozen experts on a shared geometric grid with learned per-cell precision, and its shared trunk reports a 59.2% three-task mean F1.

1 Introduction

  • Small, local genuine changes are difficult to distinguish from apparent changes caused by illumination, season, or residual misregistration, while relevant categories vary across tasks.Bi-temporal models map two co-registered images and a short prompt to a task-specific target.Source: §1 · pp. 1–2
  • Existing approaches cover the setting only partially: temporal assistants require task-specific fine-tuning, reasoning pipelines add costly and unstable training, and pixel-level detectors do not unify heterogeneous multimodal experts under one geometry.Single-image EO chat models are not designed for reasoning across time, while geospatial encoders still need downstream adaptation for dense change outputs.Source: §1 · p. 2
  • PoE-Fuse keeps heterogeneous foundation experts frozen, aligns their features to a geometric anchor, and fuses features and temporal differences by learned per-cell precision before lightweight mixing and task-specific heads.The fusion treats aligned features as Gaussian observations of a shared scene state and generalizes uniform summation and scalar gating.Source: §1 · pp. 2–3
  • 59.2% three-task mean F1 versus 40.7% for an instruction-tuned temporal VLM; PoE-Fuse also surpasses dedicated change-detection models retrained under the same protocol and budget.Ablations show dynamic per-cell precision improves over uniform fusion, while the geometric anchor is indispensable for dense change prediction.Source: §1 · p. 3

2 Related Work

Earth-observation models use temporal adaptation, single-image grounding, or specialized change detection, while frozen-encoder fusion often relies on uniform or scalar weighting. PoE-Fuse combines frozen experts on a shared geometric grid using learned per-cell precision for dense bi-temporal prediction.

2.1 Vision-Language Models for Earth Observation

  • Recent EO reasoning systems couple supervised fine-tuning with RL and process-level rewards, while PoE-Fuse trains a lightweight fusion stack on cached frozen-expert features without RL or full-model fine-tuning.PoE-Fuse serves several change tasks with one shared trunk rather than one fine-tuned model per task; these reasoning pipelines multiply training stages, depend on curated reward signals, and are hard to reproduce at modest compute budgets.Source: §2.1 · pp. 3–4

2.2 Frozen-Encoder Fusion and Spatial Alignment

  • Frozen-encoder fusion commonly uses concatenation, addition, or a scalar gate, whereas PoE-Fuse weights each aligned expert feature by learned per-cell precision and anchors fusion to shared geometry.The paper positions this approach against one-way resampling and outputs that often emphasize text or sparse prompts rather than dense change fields.Source: §2.2 · p. 4

3 Problem Setting

The problem setting covers three Earth-observation tasks on paired satellite images: change detection, building localization, and building damage classification. Inputs are co-registered 224 × 224 image pairs with task prompts; longer sequences and unregistered inputs are out of scope.

3 Problem Setting

  • The benchmark tasks are S2Looking change detection, xBD building localization, and xBD building damage classification, producing a change mask, localization mask, and damage label, respectively.The input is a co-registered image pair from the same location at times t0 < t1.Source: §3 · p. 4
  • The model aligns frozen experts to a shared G × G grid, fuses their aligned features and temporal differences by per-cell precision, then uses a lightweight mixer and task-specific heads.Only the projections, alignment, precision, mixer, and heads are trained; the experts remain frozen.Source: Figure 2 · p. 5
  • Each 224 × 224 image pair shares a pixel grid, and a task prompt is a short instruction for mask tasks or a question for damage classification.Longer acquisition sequences and unregistered inputs are out of scope; evaluation follows the benchmark’s task-specific F1 metrics.Source: §3 · pp. 4–5

4 PoE-Fuse

PoE-Fuse aligns frozen experts on a shared geometric grid, then combines their features using learned per-cell precision. A lightweight Mamba-3 mixer processes the fused timestamp features and change field for task-specific prediction.

4.1 Overview and Notation

  • DINOv3-SAT supplies the geometric grid, while SAM 3 and Gemma 4 provide grounding and language features that are aligned to it.Only the geometric expert has an explicit spatial layout; resampling makes cell locations comparable across experts.Source: §4.1 · p. 6, Table 1 · p. 6

4.2 Precision-Weighted Expert Fusion on a Shared Grid

  • Aligned features are modeled as noisy observations of a latent scene state and fused by a precision-weighted average that strictly generalizes uniform averaging and scalar gating.Precision is learned per cell from task loss; it is not calibrated uncertainty, and independent errors across experts and timestamps are an idealization.Source: §4.2 · p. 7
  • For change fields, the two timestamps’ precisions are harmonically composed, so a low-precision timestamp down-weights the fused difference.The passage attributes suppression of spurious changes from illumination or seasonal variation to this weighting.Source: §4.2 · p. 7

4.3 Lightweight Mamba-3 Mixer

  • The trunk uses fused change fields for dense segmentation and fused per-timestamp features for the pooled classifier; restricting fusion to either site loses accuracy.The fused grids and change field are flattened, temporally tagged, and processed by selective state-space Mamba-3 blocks before task-specific heads read the result.Source: §4.2 · p. 8, §4.3 · p. 8

5 Experiments

PoE-Fuse is evaluated across three bi-temporal tasks using a shared training protocol and compared with a generalist assistant and retrained specialist models. The experiments report strong benchmark results and examine evaluation, implementation, qualitative behavior, and fusion components.

5.1 Datasets

  • Three tasks use the official TEOChatlas splits without new annotations; damage records already pair each queried building with its own crop.Deranging query coordinates leaves damage F1 unchanged at 54.8 ± 3.6 versus 54.8 ± 3.5 over five seeds.Source: §5.1 · p. 8

5.2 Evaluation Metrics

  • Dense-task evaluation uses pixel-level F1 on rasterized 256×256 masks at a fixed 0.5 threshold, while damage uses inverse-prevalence-weighted F1 across five xBD labels.The 32×32 head logits are bilinearly upsampled before thresholding.Source: §5.2 · p. 8

5.3 Implementation Details

  • Training optimizes projections, alignment, fusion, mixer, and heads on cached frozen-expert features, using 59.7M trainable parameters for the shared trunk.The 14×14 grid is the DINOv3-SAT anchor’s native patch grid; moving to 28×28 removes that anchor and training fails to descend.Source: §5.3 · p. 9

5.4 Baselines

  • Specialists are retrained from scratch on the small TEOChatlas splits for 30 epochs under the same protocol, while TEOChat is the single-model generalist baseline.The specialist results therefore measure performance under the shared protocol, not their original benchmark ceiling.Source: §5.4 · pp. 9, 11

5.6 Qualitative Results

  • PoE-Fuse produces dense change masks adhering to ground-truth footprints, while TEOChat’s sparse, displaced boxes miss changed buildings.Typical PoE-Fuse over-predictions beyond small regions cost precision rather than recall; its output has 14×14-grid granularity.Source: Figure 3 · p. 10, §5.6 · p. 12

5.5 Main Results

  • 59.2% three-task mean F1, 18.5 points above TEOChat’s 40.7%, with PoE-Fuse also surpassing the strongest retrained specialist on both dense tasks.PoE-Fuse scores 47.2 ± 1.3% pixel-F1 on S2Looking change detection and 75.4 ± 1.6% on xBD localization; TinyCD reaches 46.6 ± 1.2% and 73.0 ± 2.0%, respectively.Source: Table 3 · p. 11, §5.5 · p. 11

5.7 Ablation Study

  • 57.1% mean F1 for uniform summation and 57.4% for static precision fall below PoE-Fuse’s 59.2%, supporting dynamic per-location trust.Scalar gating reaches 58.1%, but its difference from PoE-Fuse is within seed noise (p=0.37); omitting difference-precision fusion yields 58.2%.Source: §5.7 · pp. 12–13, Table 4 · p. 12
  • Removing the DINOv3-SAT geometric anchor collapses S2Looking pixel-F1 to 0.0, while restricting precision fusion to either coupling site lowers mean F1 to 58.5 or 57.5.DINO+SAM3 is the strongest pair, and dropping SAM3 costs 16 points on damage classification.Source: §5.7 · p. 13, Table 5 · p. 13

6 Conclusion

The conclusion presents PoE-Fuse as a precision-weighted estimator over aligned frozen-expert features, with one shared trunk performing three bi-temporal tasks. It reports strong benchmark results while noting constraints on input setting, mask granularity, and deployment cost.

6 Conclusion

  • PoE-Fuse treats aligned frozen-expert features as Gaussian observations of a shared latent state and fuses them by learned per-cell precision.The paper characterizes this fusion as the best linear unbiased estimator.Source: §6 · pp. 13–14
  • 59.2% mean F1 across three tasks surpassed five retrained dedicated change-detection baselines and the temporal assistant’s 40.7% mean F1.The shared trunk trained about 60M parameters on cached features; task scores were 47.2% for S2Looking change detection, 75.4% for xBD localization, and 54.8% for damage classification.Source: §6 · p. 14
  • Inputs are limited to co-registered pairs at one resolution, the 14×14 grid bounds mask granularity, and inference cost is dominated by the largest frozen backbone.The dense specialist baselines are trained from scratch on small splits and do not cover categorical damage classification.Source: §6 · p. 14

A Task-Specific Heads and Training

The method shares the fusion trunk across task-specific prediction heads. Damage classification uses pooled mixed tokens, while dense change segmentation predicts per-pixel logits from the change field.

A.1 Prediction Heads

  • One task head is trained per task while sharing the fusion trunk, which includes projections, alignment, precision fusion, and the mixer.The dense change field is available to segmentation heads.Source: §A.1 · p. 17
  • Damage classification mean-pools valid mixed-sequence tokens and predicts five-way logits using a linear layer after layer normalization.The queried building is specified by normalized box coordinates in the prompt, and focal cross-entropy with inverse-frequency class weights addresses majority-class collapse.Source: §A.1 · p. 17
  • Dense change segmentation maps the change field through convolutional layers to 32×32 logits, then upsamples them to the benchmark’s 256×256 mask resolution.Its loss combines positive-weighted binary cross-entropy and soft Dice regularization.Source: §A.1 · p. 17

A.2 Training Setup

Training updates the fusion and task modules using cached frozen-expert features, while inference still runs the experts. The supplemental examples illustrate damage classification and change-detection comparisons.

A.2 Training Setup

  • Training reads cached frozen-expert features and updates the projections, alignment module, precision heads, Mamba-3 stack, and task head.Per-task training typically took 2–4 hours after caching; the one-off cache occupied 117 GB.Source: §A.2 · p. 17
  • At 224×224 and batch size 1, inference took 276 ms per pair, of which the frozen experts took 270 ms and the trunk 6 ms.Peak GPU memory was 31.4 GB at inference and 3.4 GB during training.Source: §A.2 · p. 17
  • On destroyed and major-damage xBD buildings, PoE-Fuse predicts the correct severity while TEOChat defaults to “no damage.”The figure caption identifies this contrast as driving PoE-Fuse’s higher inverse-prevalence-weighted F1.Source: Figure 4 · p. 18
  • In extended S2Looking examples, PoE-Fuse recovers changed buildings as a dense change field, while TEOChat’s boxes are sparse and often displaced.Figure 6 uses the same format as Figure 5.Source: Figure 5 · p. 19, Figure 6 · p. 19

B Additional Qualitative Results

The appendix provides qualitative comparisons that supplement the main-paper examples, covering building damage classification and additional S2Looking change-detection crops.

B Additional Qualitative Results

C IoU and Boundary Metrics on the Dense Tasks

Table 6 adds IoU and boundary F1 comparisons on the two dense tasks, using the same evaluation harness and masks as pixel F1. The pixel-F1 ranking is preserved under both metrics, with the smallest boundary-F1 margin on S2Looking change detection.

C IoU and Boundary Metrics on the Dense Tasks

  • The pixel-F1 ranking is preserved under both IoU and boundary F1, with the smallest boundary-F1 margin on S2Looking change detection.The passage relates the smaller margin to the 14×14 alignment grid’s spatial granularity bound discussed in the limitations.Source: C IoU and Boundary Metrics on the… · p. 20
  • Table 6 reports IoU and boundary F1 on the two dense tasks using the same evaluation harness and 256×256 masks as the pixel-F1 comparison.IoU uses the official confusion-matrix evaluator; boundary F1 is the BF score with a 2-px tolerance, with boundaries defined using XOR against a 1-px erosion.Source: C IoU and Boundary Metrics on the… · p. 20, Table 6 · p. 20
  • Table 6 reports mean ± std over five seeds, except TEOChat xBD localization metrics, which use four seeds.Source: Table 6 · p. 20

Related papers

Loading 2609.37485v1…