PaperlayerView exact revision on arXiv

Source-linked AI summary

TriDifNet: A Unified Three-Task Diffusion Network for End-to-End RGB-T Perception

Qianchao He, Jing Liu, Yunhao Li, Xiaoqing Tian

arXiv:2610.37097v1cs.CV

TL;DR

Misaligned RGB-T perception lacks a shared optimization loop for alignment, fusion, and detection, despite the importance of multimodal inputs. TriDifNet couples those tasks through diffusion-based modules and cross-task feedback. On KAIST, it achieves the reported leading system-level detection scores under misalignment.

  • Problem

    Misaligned RGB-T perception still lacks a unified network that jointly optimizes geometric alignment, cross-modal fusion, and detection semantics.

    Source: §I · pp. 1–2

  • Method

    TriDifNet jointly optimizes registration, fusion, and detection with diffusion modules coupled by detection-guided and structural feedback paths.

    Source: §I · pp. 2–3

  • Results

    On KAIST, TriDifNet leads system-level competitors with 65.44% Recall, 57.55% mAP50, and 26.83% mAP50:95.

    Source: §I · p. 3, C. Main Quantitative Results · p. 13

  • Takeaways & Limitations

    KAIST experiments report reliable registration and task-oriented fusion alongside improved end-to-end pedestrian detection over single-modality, sequential, and partially coupled baselines.

    Source: §V · p. 15

Abstract

from arXiv · show

Infrared--visible (IR--VIS) multi-modal perception supports robust scene understanding under challenging illumination. Infrared and visible images are often misaligned due to platform motion, viewpoint changes, and modality-dependent distortions, causing registration errors to propagate into subsequent fusion and detection stages. We propose TriDifNet, a unified three-task diffusion network for end-to-end RGB-T perception. To the best of our knowledge, TriDifNet is the first diffusion-based network that jointly optimizes registration, fusion, and detection for misaligned RGB-T pedestrian perception. TriDifNet consists of three diffusion-based modules, i.e., the Registration Diffusion Module (RegDM), the Fusion Diffusion Module (FusDM), and the Detection Diffusion Module (DetDM). RegDM aligns the unaligned infrared image to the optical reference by estimating a dense deformation field. FusDM performs residual conditional diffusion to generate a detection-oriented fused representation with optical structural cues and complementary infrared saliency. DetDM formulates pedestrian detection as an iterative denoising process in bounding-box space. To couple the three tasks, TriDifNet introduces a closed-loop optimization mechanism, where detection semantics guide fusion and registration, and the fusion result further regularizes geometric alignment. A stage-wise progressive training strategy and a lightweight Nash multi-task learning (Nash-MTL)-based late-stage refinement are further adopted to balance registration, fusion, and detection optimization. Extensive experiments on the KAIST multispectral pedestrian benchmark show that TriDifNet achieves superior downstream detection performance under misaligned RGB-T settings. Against strong system-level competitors, TriDifNet achieves 65.44\% Recall, 57.55\% mAP$_{50}$, and 26.83\% mAP$_{50:95}$.

I. INTRODUCTION

Misaligned RGB-T inputs and decoupled optimization leave detection outside many registration–fusion systems. TriDifNet brings registration, fusion, and detection into one diffusion-based closed loop.

I. INTRODUCTION

  • Visible structure and infrared thermal saliency are complementary, but misalignment can leave fusion compensating for geometric errors and distort targets.Conventional sequential pipelines align before identifying structures important for recognition; resulting artifacts can include ghosting, double edges, structural distortion, or target deformation.Source: §I · p. 1
  • Detection is often auxiliary guidance for fusion or a separate downstream task, leaving alignment, fusion, and recognition outside a shared optimization loop.The introduction identifies joint optimization of all three tasks as the missing capability.Source: §I · p. 2
  • Det2Fus uses ROI-guided semantics, Det2Reg uses a frozen infrared detector, and Fus2Reg feeds structural cues back to registration.Training proceeds stage-wise, followed by lightweight Nash-MTL refinement across the three branches.Source: §I · pp. 2–3
  • TriDifNet uses conditional deformation diffusion for registration, residual conditional diffusion for fusion, and bounding-box denoising for detection.Fusion adds complementary infrared content to the optical image's structural base.Source: §I · p. 2
  • On KAIST with controlled RGB-T misalignment, TriDifNet reaches 65.44% Recall, 57.55% mAP50, and 26.83% mAP50:95, outperforming strong system-level pipelines.The reported comparison is under the same setting.Source: §I · p. 3

II. BRIEF INTRODUCTION TO DIFFUSION MODEL

  • Diffusion progressively noises a clean target and learns reverse denoising; conditional observations guide generation or restoration.The paper presents forward, reverse, and conditional diffusion as the standard formulation.Source: §II · p. 3

III. METHODOLOGY

TriDifNet runs registration, fusion, and detection sequentially in its forward path while coupling them through feedback. The formulation targets controlled weak affine RGB-T misalignment and jointly constrains geometry, fusion, and semantics.

A. Overview

  • RegDM aligns infrared to the optical reference, FusDM creates a detection-oriented fused representation, and DetDM denoises bounding boxes with ROI-guided re-inference.Feedback paths connect detection to registration and fusion, and fusion back to registration.Source: A. Overview · p. 3

B. Notation and Problem Formulation

  • The objective is joint optimization under controlled weak affine misalignment, modeled as in-plane infrared rotation and translation relative to the visible image.The problem formulation names the optical image Y and unaligned infrared image Z′ as inputs.Source: B. Notation and Problem Formulation · pp. 3–4
  • The fused output contains fusion- and detection-oriented channels; the detection representation supplies the detector's first-pass input.The detector then uses ground-truth and prediction-derived ROI priors to construct a mixed mask and refine its task-driven representation.Source: B. Notation and Problem Formulation · p. 4
  • Det2Reg supplies infrared-domain semantic supervision, Det2Fus feeds detection semantics back to fusion, and Fus2Reg returns structural feedback to registration.Together, these paths make geometry, cross-modal fusion, and task-level semantics mutually constrained.Source: B. Notation and Problem Formulation · pp. 4–5

C. Registration Diffusion Module (RegDM)

RegDM treats dense infrared-to-optical alignment as conditional diffusion over an inverse deformation field. It reconstructs that field and spatially warps the infrared input to produce the registered image used by fusion.

1) Conditional Diffusion for Deformation Field Estimation

  • Given optical reference Y and unaligned infrared input Z′, RegDM estimates an inverse deformation field mapping infrared into the optical coordinate system.The field is the target variable of the conditional diffusion process.Source: 1) Conditional Diffusion for… · p. 5
  • At step t, the denoiser uses a noisy deformation representation and a condition tensor formed from the optical reference and unaligned infrared input.It predicts injected Gaussian noise, from which the clean inverse deformation field is recovered using the cumulative noise coefficient.Source: 1) Conditional Diffusion for… · p. 5
  • Spatially warping the unaligned infrared image with the predicted field produces registered image ˆZ for FusDM, while downstream semantic and structural feedback remains available.This makes RegDM the geometric foundation of the closed-loop network.Source: 1) Conditional Diffusion for… · p. 5

2) Registration Objective:

RegDM uses a composite objective to balance deformation-field denoising, spatial smoothness, and cross-modal alignment. Its alignment term compares the registered infrared image with the optical reference.

2) Registration Objective:

  • The composite RegDM objective balances diffusion reconstruction, deformation smoothness, and cross-modal alignment consistency.The reconstruction term denoises the inverse deformation field; smoothness discourages unrealistic discontinuities, while alignment encourages geometric consistency with the visible reference despite modality intensity differences.Source: 2) Registration Objective · p. 5
  • The smoothness term regularizes spatial continuity in the predicted deformation field, discouraging unrealistic discontinuities and stabilizing registration.It uses horizontal and vertical finite-difference operators.Source: 2) Registration Objective · p. 5
  • The alignment term uses normalized cross-correlation between the registered infrared image and the grayscale optical reference to encourage cross-modal geometric consistency.The grayscale projection is used for similarity measurement, helping the term remain robust to cross-modal intensity differences.Source: 2) Registration Objective · p. 5

D. Fusion Diffusion Module (FusDM)

FusDM generates a residual fused representation conditioned on the optical image and aligned infrared image, then builds a detector-oriented representation within the fusion pipeline. This representation preserves optical structural cues while incorporating complementary infrared saliency and downstream detection semantics.

1) Residual Conditional Diffusion for Infrared–Visible Fusion:

  • FusDM uses residual conditional diffusion, taking the optical image and aligned infrared image as joint guidance for a cross-modal residual representation.The residual formulation uses the optical image's stable structural and textural base while focusing the fusion branch on complementary infrared saliency in target-relevant regions.Source: 1) Residual Conditional Diffusion for… · pp. 5–6

2) Detector-Oriented Fusion Representation:

  • FusDM produces both a fused image representation and a detector-oriented representation, Fdet, that bridges fusion and detection and can be refined by downstream detection semantics.Visible–fusion cross-attention shapes the first-pass detector input to retain detector-friendly optical structural cues while complementing ROI-guided infrared enhancement.Source: 2) Detector-Oriented Fusion… · p. 6

3) Fusion Objective:

The base fusion objective combines diffusion reconstruction with pixel-level fidelity and structural consistency, and is augmented in the full network by detection-driven constraints. The detector uses first-pass predictions to form an ROI prior that guides refinement of its representation for second-pass detection.

3) Fusion Objective:

  • FusDM's base optimization combines diffusion reconstruction, pixel-level fidelity to the fusion target, and structural consistency in the fused representation.The structural regularization encourages preservation of perceptually meaningful scene structures; the full network further adds Det2Fus constraints and closed-loop feedback for detection-oriented semantic utility.Source: 3) Fusion Objective · p. 6

1) Diffusion-Based Detection Formulation:

2) ROI-Guided Re-inference:

  • The ROI refinement mixes ground-truth and prediction-derived masks, with an agreement-based coefficient kept GT-dominant, then injects stronger infrared saliency into target-centered regions.The refinement preserves the original detector-oriented response in non-target regions and uses the refined representation for second-pass detection.Source: 2) ROI-Guided Re-inference · p. 7

3) Detection Objective:

DetDM couples final pedestrian prediction with semantic feedback to the fusion representation. Across tasks, TriDifNet links registration, fusion, and detection through forward and feedback paths, including registered infrared conditioning for fusion.

3) Detection Objective:

  • DetDM supervises its final second-pass prediction with box-space diffusion detection, while representation-level constraints act on the detector-oriented fusion representation through mixed ROI guidance.The detector thereby provides semantic feedback that reshapes the fusion-side representation before final inference.Source: 3) Detection Objective · p. 7

F. Cross-Task Coupling: Registration–Fusion–Detection

  • TriDifNet couples registration, fusion, and detection through forward registration-to-fusion and fusion-to-detection paths, plus detection-to-registration, detection-to-fusion, and fusion-to-registration feedback.These interactions are described as a closed-loop coupling among geometry, fusion, and semantics.Source: F. Cross-Task Coupling:… · p. 7

1) Registration-to-Fusion Consistency (Reg-to-Fus):

  • Reg-to-Fus conditions residual diffusion on the registered infrared image, reducing spatial discrepancy and stabilizing fused-output generation and structural consistency.The aligned infrared image replaces the raw unaligned input as FusDM's infrared condition.Source: 1) Registration-to-Fusion Consistency… · p. 8

2) Fusion-to-Registration Structural Feedback (Fus2Reg):

TriDifNet couples fusion, registration, and detection through feedback paths that add structural and semantic guidance to alignment and reshape fusion for pedestrian detection.

2) Fusion-to-Registration Structural Feedback (Fus2Reg):

  • Fus2Reg regularizes geometric alignment by matching registered-infrared derivative responses to the structure extracted from the fused representation.The constraint is intended to use fusion-driven geometry cues to correct deformation estimation.Source: 2) Fusion-to-Registration Structural… · p. 8

3) Detection-Guided Registration (Det2Reg):

  • Det2Reg uses a frozen infrared-domain detector as a semantic anchor to keep object-level responses consistent before and after registration.Its semantic regularizer is used during training, not inference.Source: 3) Detection-Guided Registration… · p. 8

4) Detection-Aware Fusion (Det2Fus):

  • Det2Fus rewrites the detector-oriented fusion representation with an ROI-guided prior, applying task-driven constraints before final detection.The design aims to emphasize object-centric regions while preserving global fusion fidelity outside them.Source: 4) Detection-Aware Fusion (Det2Fus) · pp. 8–9

5) Fusion-Enhanced Detection(Fus-to-Det):

The fusion branch supplies a detector-oriented representation for final inference, while the overall objective jointly optimizes registration, fusion, and detection within the closed loop.

5) Fusion-Enhanced Detection(Fus-to-Det):

  • FusDM supplies the first-pass detector input, and its refined representation drives DetDM's final detection inference.The fusion branch both receives semantic feedback from detection and produces a task-specialized detector input.Source: 5) Fusion-Enhanced Detection(Fus-to-Det) · p. 9

6) Overall Training Objective:

  • The overall objective combines base registration, fusion, and detection losses with Det2Reg, Fus2Reg, and Det2Fus feedback.Registration receives semantic and structural feedback, fusion receives ROI-guided semantic feedback, and detection is supervised on the refined detector representation.Source: 6) Overall Training Objective · p. 9

G. Training Strategy and Nash-MTL-based Late-Stage Refinement

TriDifNet uses progressive training followed by Nash-MTL refinement to coordinate its three tasks, then runs registration, fusion, and ROI-guided detection refinement at inference.

G. Training Strategy and Nash-MTL-based Late-Stage Refinement

  • Training progresses from registration alone, through separately coupled branch optimization, to joint optimization of all three branches.The staged schedule is intended to stabilize optimization and reduce interference while branches adapt to registration-guided inputs.Source: G. Training Strategy and… · p. 9
  • Nash-MTL dynamically balances task gradients during late-stage refinement rather than using fixed loss weights.When a batch lacks valid detection supervision, the detector is excluded from the bargaining step; Nash-MTL is training-only.Source: G. Training Strategy and… · pp. 9–10

H. Inference Pipeline

  • At inference, RegDM and FusDM precede DetDM, whose first-pass prediction creates an ROI prior for refined second-pass detection.The ROI-guided rewriting enhances infrared saliency in target-centered regions while preserving the fusion-driven response elsewhere; training-only auxiliary supervision is removed.Source: H. Inference Pipeline · p. 10

IV. EXPERIMENTS

On KAIST under controlled weak affine misalignment, TriDifNet performs strongly in end-to-end pedestrian detection, while registration and fusion comparisons characterize its component behavior under distinct evaluation settings.

A. Datasets, Misalignment Setting, and Evaluation Metrics

  • The experiments use cleaned KAIST pedestrian annotations and synthetic infrared perturbations capped at 2° rotation and 0.05 translation ratio.LRRNet fusion outputs serve as a training pseudo teacher, not exact ground truth for final evaluation.Source: A. Datasets, Misalignment Setting, and… · pp. 10–11

B. Experimental Settings and Baselines

C. Main Quantitative Results

  • 10.425901 RMSE and 0.856359 SSIM were best among compared registration methods, while R3 had the highest NCC/CC at 0.835497.The authors describe these results as stable geometric correction, not an overwhelming registration advantage; the comparison targets controlled weak affine perturbations.Source: Table III · p. 12, C. Main Quantitative Results · p. 12
  • Under aligned inputs, F3 leads CC and F2 leads UIQI; FusDM's teacher-referenced scores are competitive but not leading.These are weak-reference image-level indicators because KAIST lacks real fusion ground truth.Source: C. Main Quantitative Results · pp. 12–13, Table IV · p. 13
  • 65.44% Recall, 57.55% mAP50, and 26.83% mAP50:95 are TriDifNet's best system-level scores, improving over S7 by 16.99%, 13.89%, and 7.81%, respectively.The qualitative comparison reports more complete, better-localized predictions, especially for nighttime small targets in Cases 4 and 5.Source: C. Main Quantitative Results · pp. 13–14, Table V · p. 13, Figure 6 · p. 13
  • S4 records 25.22% recall and 22.90% mAP50, while the authors attribute sequential S5's limitations to accumulated registration and fusion errors.They also caution that S6 uses YOLOv5s, so part of its performance difference may reflect detector configuration.Source: C. Main Quantitative Results · p. 14

D. Ablation Studies

TriDifNet’s full model outperforms the ablated variants, while component removals show the value of cross-task feedback, detection-aware fusion, and late-stage task balancing. The authors also identify inference efficiency as an area for future work.

D. Ablation Studies

  • 65.44% recall, 57.55% mAP50, and 26.83% mAP50:95 are achieved by full TriDifNet, whose results exceed those of the single-module variants.FusDM alone raises mAP50 from 37.90% to 44.67%, while RegDM alone raises it to 39.04%.Source: D. Ablation Studies · p. 14
  • The full model’s lower FPS is associated with iterative diffusion-based registration and fusion plus ROI-guided re-inference; improving real-time efficiency is future work.The authors focus this work on three-task closed-loop optimization under misaligned RGB-T inputs.Source: D. Ablation Studies · p. 14
  • Removing Det2Reg worsens registration and detection, while removing Fus2Reg produces the largest registration degradation among feedback-path variants.Det2Reg removal raises RMSE from 10.4259 to 10.8720 and lowers mAP50 from 57.55% to 54.11%; Fus2Reg removal raises RMSE to 11.0282 and lowers SSIM to 0.8426.Source: D. Ablation Studies · p. 14
  • Removing Det2Fus lowers mAP50 from 57.55% to 42.03%, and disabling ROI-IR Injection as well lowers it to 36.60%.Variants without these mechanisms have higher CC and UIQI scores but lower detection performance, indicating that image-level fusion scores do not necessarily reflect downstream task utility.Source: D. Ablation Studies · pp. 14–15
  • Removing Nash-MTL reduces mAP50 from 57.55% to 55.32%, supporting late-stage balancing among registration, fusion, and detection.The authors report that the full model achieves the best registration and detection results overall.Source: D. Ablation Studies · p. 15

V. CONCLUSION

  • The conclusion reports improved end-to-end detection over single-modality, sequential, and partially coupled baselines, with future work aimed at broader RGB-T scenarios.It attributes the ablation evidence to cross-task feedback, ROI-guided infrared injection, and late-stage multi-task balancing.Source: §V · p. 15
Loading 2610.37097v1…