PaperlayerView exact revision on arXiv

Source-linked AI summary

Feature Recovery for Object Understanding After Irreversible Fire Damage

Aditi Tiwari, Sofia Stoica, Savya Khosla, David Forsyth, Heng Ji

arXiv:2609.12078v1cs.CV

TL;DR

Post-fire damage irreversibly changes object geometry, material state, and appearance, creating a gap in evaluating object understanding after physical transformation. The paper introduces TRACE and a frozen-host Feature Recovery Module (FRM) trained with paired feature supervision. FRM improves every evaluated task, with gains concentrated at higher degradation severity.

  • Problem

    Post-fire transformations irreversibly change object structure, material, and appearance, while standard benchmarks largely preserve object structure and material identity.

    Source: §1 · p. 1

  • Method

    TRACE pairs pristine objects with progressively degraded states across five tasks, and FRM maps degraded encoder features toward pristine-aligned representations while keeping the host frozen.

    Source: §1 · p. 2

  • Results

    9.0–30.8% relative improvement across detection, retrieval, and reasoning; across eight frozen hosts, FRM improves every task, with gains concentrated at higher degradation severity.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Reliable post-fire object detection and identification could support hazard location, pre-incident reconstruction, and damaged-property inventory.

    Source: §6 · p. 9

    FRM removes damage cues needed for current-state queries, so the authors recommend disabling it for questions such as describing fire damage.

    Source: §6 · p. 9

Abstract

from arXiv · show

Objects in post-fire environments often undergo irreversible physical transformations that change their geometry, material state, and visual appearance. Detecting and identifying these remnants is critical for locating hazards, reconstructing pre-incident contents, and inventorying losses. Unlike standard image corruptions, these degradations affect the physical structure of the object itself. To study this setting, we introduce TRACE, a transformation-aware benchmark for post-fire object understanding. TRACE contains 21.4K real-image-grounded synthetic scenes and paired object-level pristine-to-degraded progressions spanning 499 object identities across 189 categories. We define five tasks targeting localization and pre-degradation understanding: degraded-object detection, pristine-state recovery and retrieval, original material recovery, pristine description generation, and functional reasoning. Existing models degrade sharply with severity. From the least to the most severe level, RF-DETR mAP decreases by 71% relative, while InternVL3.5 retrieval R@1 falls from 93.85 to 28.11. To address this, we propose the Feature Recovery Module (FRM), a plug-and-play module that maps degraded encoder features to pristine-aligned representations while keeping the host frozen. Trained only with paired feature supervision, FRM improves scene-level detection, CLIP/SigLIP2 feature recovery, and all four object-level VLM tasks, with larger gains under more severe degradation. Across VLM hosts and severity levels, relative gains average 12.5% for retrieval, 20.1% for material recovery, 13.2% for description generation, and 12.4% for functional reasoning.

1 Introduction

Post-fire damage irreversibly changes object structure, material, and appearance, while existing benchmarks and models do not capture or handle this setting well. The paper introduces TRACE to evaluate post-fire object understanding and FRM to recover pristine-aligned features, reporting gains across its tasks.

1 Introduction

  • Post-fire degradation jointly changes geometry, material state, and surface appearance, making recovery of identity, material, and function relevant to emergency response, investigation, and loss inventory.These needs include locating hazards, reconstructing pre-incident contents, and inventorying damaged property.Source: §1 · p. 1
  • RF-DETR mAP drops 71% from L0 to L4, 92.9% of boxes localized at IoU≥0.5 are misclassified, and InternVL3.5 retrieval R@1 falls from 93.85 to 28.11.Frozen encoder feature drift also indicates a representational rather than model-specific limitation.Source: §1 · pp. 1–2
  • TRACE provides 21.4K real-image-grounded synthetic scenes and 499 paired pristine-to-degraded object trajectories across 189 types for five localization and pre-degradation understanding tasks.The trajectories include five progressively degraded states with part-level annotations; scene annotations include boxes, materials, and severity captions.Source: §1 · p. 2, Table 1 · p. 2, Figure 2 · p. 3
  • Unlike pixel-space restoration, which is ambiguous after irreversible damage destroys parts or material cues, FRM maps degraded host features toward pristine counterparts at the feature interface.The host remains frozen, and paired feature supervision trains the lightweight residual module.Source: §1 · p. 2
  • FRM improves all five tasks, with RF-DETR detection up 30.8% relative and four VLM tasks up 12.4% to 20.1% mean relative across five frozen VLMs.The module is trained on paired feature supervision while host weights remain frozen.Source: §1 · p. 2

2 Related Work

Prior visual-robustness work covers appearance shifts, domain shifts, adverse conditions, and restoration, but generally assumes intact objects or works at scene level. TRACE addresses object-level irreversible physical transformation, while FRM uses paired pristine-degraded correspondences to recover features in a frozen host space.

i. Benchmarks

  • Robustness benchmarks preserve object structure and material identity, while natural-disaster benchmarks assess buildings or scenes rather than individual irreversibly transformed objects.TRACE is presented as the first benchmark for object-level physical degradation with paired pristine-to-degraded trajectories and severity-stratified evaluation.Source: §2 · p. 3

ii. Methods

  • Domain adaptation aligns aggregate source and target distributions, whereas FRM uses paired pristine-degraded correspondences because each severity induces a different distribution.Test-time adaptation instead optimizes on unlabeled test data at deployment; FRM trains once and requires no per-deployment optimization.Source: §2 · p. 3
  • Pixel-space restoration faces ill-posed inversion under physical degradation, while feature-quality methods characterize embedding shifts without correcting them; FRM recovers pristine-aligned features with frozen host weights.The comparison distinguishes image restoration, feature-shift measurement, and paired-supervision feature recovery.Source: §2 · p. 3

3 TRACE

TRACE is designed around paired pristine and post-fire objects, severity progression, and real-world grounding. Its five tasks evaluate detection and recovery of pre-degradation properties.

Design Principles

  • TRACE’s design pairs pristine and degraded instances, uses an L0–L4 severity ladder, and grounds synthetic generation in real post-fire imagery.These choices support direct measurement of representation drift and severity-stratified evaluation.Source: §3 · p. 3

Object Inventory

  • TRACE spans 189 object types and 11 normalized material classes, covering hazardous items as well as valuables and everyday objects.Two authors achieved 93.4% agreement and Cohen’s κ = 0.836 on accept/reject labels for a stratified 300-object subset.Source: §3 · p. 4

Scene-Level Subset

  • The scene-level subset contains 21,400 accepted scenes from approximately 29,000 generations, filtered by a vision-language judge using a 0.7 threshold.Scenes are generated from structured prompts and two to four real post-fire references.Source: §3 · p. 4

Object-Level Subset

  • Each object trajectory pairs a pristine render with five progressively degraded states, spanning 499 identities across 189 object types.All renders of an identity remain in the same 80/10/10 train, validation, and test split.Source: §3 · p. 4

Real Post-Fire Crops

  • The object-level test set includes 500 manually cropped real post-fire objects annotated with identity and severity labels.The crops come from the pool of approximately 700 real post-fire reference images.Source: §3 · p. 4

Evaluation Tasks

  • TRACE defines five tasks spanning degraded-object detection, pristine-state recovery and retrieval, material recovery, pristine description generation, and functional reasoning.Functional reasoning includes predicting pre-degradation function tags and restoration materials.Source: §3 · pp. 4–5

4 Feature Recovery Module (FRM)

FRM learns to map degraded visual features toward paired pristine features while leaving the host model frozen. It is inserted at a selected host interface so corrected features continue through the unchanged downstream pipeline.

Feature-Recovery Problem

  • FRM uses paired pristine and degraded observations to supervise feature-space recovery, avoiding ambiguous pixel-level inversion after physical changes to geometry, material, and surface structure.The pristine branch is computed without gradient tracking, and FRM is applied only to the degraded branch.Source: §4 · pp. 5–6

Architecture

  • The residual Transformer learns a shape-preserving correction, initialized as the identity through zero-gated residual scales.The gates zero the attention and MLP outputs at initialization; encoder weights remain untouched.Source: §4 · p. 5

Tap-Point Selection

  • Tap selection trades spatial detail at earlier interfaces against downstream alignment at later ones, with post-projector or post-merger corrections closer to the features consumed by downstream heads.The tap point is selected per host and ablated in Table 9.Source: §4 · pp. 5–6

Training Objective

  • Training combines element-wise MSE with token-level cosine alignment; spatial augmentations preserve patch-grid correspondence, excluding arbitrary-offset crops.The cosine-loss weight λ_cos = 0.1 was selected on the validation split.Source: §4 · p. 6

Instantiation in RF-DETR

  • RF-DETR applies one shared FRM head to flattened features from four DINOv2 layers before reshaping them for the unchanged detector.The loss is applied independently at each scale and averaged across the four tapped layers.Source: §4 · p. 6

Instantiation in Qwen3-VL

  • Qwen3-VL inserts FRM after the merger, correcting visual tokens before they enter the frozen language model.At inference, the corrected features continue through the unchanged downstream pipeline, and each new host requires its own trained module.Source: §4 · p. 6

5 Experiments

Experiments evaluate detection, feature recovery, and four object-level tasks using TRACE scenes and held-out degraded trajectories. FRM improves detection and feature recovery, while VLM gains concentrate at higher degradation severity; ablations test the recovery loss and real-crop transfer.

5 Experiments

  • The evaluation uses synthetic scene-level detection tests and held-out L0-L4 object trajectories, plus a separate 500-crop real post-fire evaluation.Five frozen VLM hosts are tested with identical prompts, parsers, normalization rules, and scoring scripts, with and without FRM.Source: §5 · pp. 6–7

5.1 Degraded-Object Detection and Feature-Space Recovery

  • FRM improves frozen RF-DETR by 30.8% and adds 13.1% atop fine-tuning while preserving pristine performance within 0.001 mAP.Statistics-only domain adaptation underperforms the frozen baseline; Restormer yields 0.338→0.375 mAP, while WildIR reduces performance to 0.266 mAP.Source: §5.1 · p. 7, Table 2 · p. 7, Figure 4 · p. 7
  • FRM reduces feature-recovery MSE by 30.3% on CLIP and 43.4% on SigLIP2, improving SigLIP2 retrieval Top-1 by 14.8 points and Top-5 by 18.9 points.SigLIP2 Top-5 nearly reaches the 100% pristine-query ceiling.Source: §5.1 · p. 7, Table 3 · p. 7

5.2 Pristine-State Recovery in VLMs

  • Across VLM hosts, retrieval gains average +0.73 points at L0-L1 and +14.66 points at L2-L4, with larger gains under more severe degradation.FRM improves degraded-to-pristine Recall@1 across all evaluated VLM hosts.Source: §5.2 · pp. 7–8, Table 4 · p. 8, Figure 5 · p. 8
  • InternVL3.5, the weakest degraded-input baseline on material and description tasks, gains 9.81 and 8.41 points, respectively.The authors interpret the larger gains as associated with degradation sensitivity rather than model capacity.Source: §5.2 · p. 8, Table 4 · p. 8
  • Functional-reasoning combined score improves on every host by a mean +4.9 points, with an average +7.6 at L3-L4.FRM uses paired features without per-image task labels; the authors suggest it restores visual evidence for object-specific function reasoning.Source: §5.2 · p. 8, Figure 5 · p. 8, Table 4 · p. 8

5.3 Ablations

  • MSE+0.1Lcos gives the best tradeoff across cosine distance, identity accuracy, and R@1; λ=0.1 gives the best identity accuracy and retrieval.R@1 varies by only 1.16 points across λ ∈ {0.01, 0.1, 0.5}; λ=0.5 yields the lowest cosine distance.Source: Table 5 · p. 8, §5.3 · p. 9
  • On 500 real post-fire crops from TRACE classes at L2-L4, FRM improves every task on average and remains positive at L4.The evaluation focuses on moderate-to-severe damage because real post-fire imagery is concentrated there and has few reliable L0-L1 analogs.Source: Table 5 · p. 8, §5.3 · p. 9

6 Conclusion & Discussion

TRACE and FRM address post-fire object understanding through paired pristine-to-degraded data and feature recovery, with gains across tasks that concentrate at higher degradation severity. The conclusion also identifies task, host, data, and deployment boundaries.

6 Conclusion & Discussion

  • Across eight frozen hosts, FRM improves every task, with gains concentrated at higher severity and 9.0–30.8% relative improvement across detection, retrieval, and reasoning.TRACE comprises 21.4K real-image-grounded synthetic scenes and trajectories for 499 identities across 189 categories.Source: §6 · p. 9
  • FRM’s pristine-aligned features remove damage cues, so the authors recommend disabling it for current-state damage queries and identify query-aware gating as a future direction.A module must be trained separately for each encoder, tap, and feature-space configuration; adding an identity requires a pristine anchor and at least three degraded observations.Source: §6 · p. 9
  • Real-crop gains are positive but smaller than synthetic gains, and FRM currently ignores room layout, neighboring objects, and surviving structural parts.The authors identify these context cues and part-level understanding as directions for future work.Source: §6 · p. 9
  • The authors describe potential uses in emergency response, forensic investigation, and insurance assessment, while warning that high-severity predictions can be wrong and need human oversight for legal or insurance use.The benchmark uses synthetic scenes grounded in real post-fire imagery; deployment environments may differ, so users should validate on in-domain data.Source: §6 · pp. 9–10

Appendix

  • Appendix A reports dataset statistics, while Appendix B describes object and scene generation, including real-image grounding and filtering.Source: Appendix · p. 14
  • Appendices C, D, E, and G cover evaluation protocols, FRM host instantiations, full per-level results, and ablations, respectively.Source: Appendix · p. 14

A Dataset Statistics

This section describes TRACE’s object and scene coverage, annotations, and severity metadata, along with the procedures used to generate realistic post-fire scenes. It also covers candidate and annotation quality checks, including grounding and filtering for realism and plausibility.

A.1 Subset Statistics

  • TRACE’s inventory spans 499 object identities, 189 object types, and 11 normalized material classes, while its 21,400 scenes are grounded in approximately 700 real post-fire reference images.The scenes cover residential, outdoor, work/institutional, and commercial environments; plastic, metal, fabric/textile, and wood are the most represented materials.Source: Figure 6 · p. 14, Figure 7 · p. 15

A.2 Degradation Severity Metadata

  • Severity metadata tracks soot coverage, material failure, deformation, and structural integrity across five degradation levels to guide generation and severity-stratified evaluation.TRACE distinguishes physical transformation, which changes an object’s material and structure, from image corruption, which changes appearance while preserving object state.Source: §A.2 · p. 14, Table 6 · p. 14, Figure 8 · p. 15

A.3 Annotation Summary

  • Object annotations capture part-level materials, failure notes, and degradation descriptions; scene annotations include boxes, object labels, visible materials, damage states, captions, and generation metadata.Intended object and material inventories from generation prompts are stored as metadata, not used as evaluation ground truth.Source: §A.3 · p. 15

B Generation Prompts and Filtering

  • Reference grounding and VLM accept/reject filtering improve generated scenes’ realism, object grounding, and post-fire plausibility.The comparison uses Gemini 3 Pro Image in both rows, with the ungrounded, unfiltered row using the same scene prompts.Source: Figure 9 · p. 16

B.1 Object-Level Generation

  • Object trajectories start with a text-prompted pristine image and generate each subsequent L0–L4 stage from the preceding image, enforcing cumulative, material-aware damage while preserving the same physical instance.Prompts incorporate part-level materials, material-specific failure behavior, stage severity, and constraints for uneven damage and consistency.Source: §B.1 · pp. 17, 19
  • Object-stage definitions range from intact, recognizable forms at L0 to near-total destruction at L4, when identification may require inference from fragments, hardware, and spatial cues.Different materials on one object fail at different rates, producing mixed material states at intermediate stages.Source: §B.1 · p. 19

B.2 Scene-Level Generation

  • Scene generation conditions novel post-fire scenes on two to five real reference photographs and controls scene type, damage, materials, clutter, lighting, and socioeconomic style.Prompts require physically accurate materials and spatially coherent, uneven damage, including heavier damage near the likely fire origin.Source: §B.2 · pp. 19–20
  • Scene prompts vary scene type, damage, materials, clutter, lighting, style, and layout while requiring realistic, asymmetric post-fire scenes and prohibiting copied viewpoints or layouts.Prompts call for plausible structural damage, debris, occlusion, and documentary-style realism without active flames or smoke effects.Source: §B.2 · p. 21

B.3 Grounding Judge and Filtering

  • 21,400 of approximately 29,000 generated candidates passed the 0.7 grounding threshold, an acceptance rate of 73.8%.Grounding scores filter for realism and post-fire plausibility, not evaluation labels; object boxes and labels are handled separately as detection annotations.Source: §B.3 · p. 21
  • Two authors independently labeled a stratified 300-object subset with 93.4% agreement and Cohen’s κ = 0.836.Initial annotations and bounding boxes were automatically produced, then verified or corrected by the authors.Source: §B.3 · p. 21

C Evaluation Task Details

Appendix C specifies five evaluation tasks and their protocols, from detecting degraded objects to assessing pre-degradation descriptions. The protocols use task-specific annotations, scoring, and retrieval procedures.

C.1 Degraded-Object Detection

  • Detection evaluates scene-level RF-DETR predictions with class-aware COCO-mapped mAP@0.5:0.95, counting a box only when localization and mapped category are correct.Only unambiguous mappings are scored; the lookup covers 55 of 80 COCO categories, chiefly common indoor objects.Source: §C.1 · p. 22

C.2 Pristine-State Recovery and Retrieval

  • Pristine-state recovery uses feature alignment to paired pristine targets and gallery retrieval to find the corresponding pristine object for a degraded image.Alignment reports MSE and mean token cosine similarity; gallery images are ranked by visual-feature cosine similarity, with Recall@1 as the main metric.Source: §C.2 · p. 22

C.3 Original Material Recovery

  • Original-material recovery predicts an object’s pre-degradation materials and scores normalized predictions against part-level annotations mapped to TRACE’s 11 material classes.The benchmark reports micro-F1 across objects and material labels.Source: §C.3 · pp. 22–23

C.4 Pristine Description Generation

  • Pristine-description generation prompts for the prior object type, materials, and use, then averages identity accuracy, material micro-F1, and function accuracy equally.A deterministic LLM judge resolves only identity and function paraphrases left unmatched by synonym rules; replacing that fallback with strict no-match changes aggregate description score by less than 0.8 points.Source: §C.4 · p. 23

C.5 Functional Reasoning

Functional reasoning asks models to infer an object’s pre-degradation function and restoration needs without being told its category. The score averages function-tag and restoration-material micro-F1.

C.5 Functional Reasoning

  • The task asks for an object’s original function and materials needed for restoration, without revealing its category, using a fixed response format.Each model receives the same prompt across hosts and FRM conditions, with temperature 0.0 and at most 150 output tokens.Source: §C.5 · pp. 23–24
  • Ground truth maps each object type to one or two of 20 canonical function tags; restoration-material labels reuse TRACE’s existing material annotations.The function mapping was constructed with LLM assistance and verified by the authors.Source: §C.5 · p. 24
  • The final functional-reasoning score is the unweighted mean of function-tag micro-F1 and restoration-material micro-F1.Function responses are normalized to a fixed vocabulary using keyword and synonym rules, with a deterministic LLM fallback for ambiguous mappings.Source: §C.5 · p. 24
  • At high severity, coarse answers such as “container” may partially match broad function tags despite underspecified identity, so the authors interpret scores alongside retrieval and pristine-description results.Source: §C.5 · p. 24

D FRM Implementation Details

FRM is trained on paired frozen-host features at selected interfaces, with host weights unchanged. The appendix specifies how it is inserted into VLM visual-token paths and sized across host types.

D FRM Implementation Details

  • The shared setup extracts degraded and paired pristine features at the same frozen interface, maps zℓ to recovered features, and trains only FRM.The token dimension depends on the selected tap point, while N is the visual-token count there.Source: D FRM Implementation Details · p. 24
  • Four VLM hosts use D=2560 interfaces and 314.57M-parameter FRMs; CLIP and SigLIP2 use 50.33M and 28.31M parameters, respectively.The percentage overhead for CLIP and SigLIP2 is omitted because its denominator depends on whether the vision tower or full contrastive model is counted.Source: D FRM Implementation Details · pp. 24–25
  • For SigLIP2 retrieval, the degraded object image is the query and candidate pristine object images form the gallery.Source: D FRM Implementation Details · p. 25

D.1 FRM Instantiation in VLM Hosts

  • InternVL3.5 applies FRM after its projector, replacing degraded post-projector tokens with corrected tokens before the frozen language model consumes them.Paired pristine features are cached, degraded images are encoded on the fly, and the host remains frozen.Source: §D.1 · p. 25
  • LLaVA-OneVision-1.5 applies FRM after its merger to 256 visual tokens of dimension 2560, while preserving its prompting, processor behavior, and language-model weights.The wrapper applies FRM within the enabled context and restores the original visual path afterward.Source: §D.1 · p. 25
  • Molmo2 applies FRM to post-projector tokens in the LLM-facing feature space and uses the VLM configuration with M=4, 8 attention heads, and MLP ratio r=4.Its token count is determined by the host configuration.Source: §D.1 · p. 25

D.2 FRM Instantiation in Frozen Encoders

Frozen encoders use FRM at visual-token interfaces before their text-aligned projections. The reported tap-point comparison favors later, downstream-facing interfaces for VLM retrieval.

D.2 FRM Instantiation in Frozen Encoders

  • CLIP FRM operates on 256 spatial patch tokens of dimension 1024 before visual projection, with the CLS token dropped and the encoder kept frozen.Recovered tokens are compared with paired pristine tokens using MSE, cosine similarity, and pooled-token retrieval.Source: §D.2 · p. 26
  • SigLIP2 FRM uses 196 patch tokens of dimension 768 before projection, mapping degraded features to shape-preserving recovered features supervised by paired pristine features.The encoder remains frozen; evaluation includes feature-space recovery and retrieval using pooled visual features.Source: §D.2 · p. 26
  • Qwen3-VL retrieval R@1 rises from 72.90 at a pre-merger tap to 85.28 post-merger, while InternVL3.5 rises from 63.84 pre-projector to 74.92 post-projector.The results support correcting features consumed by the frozen language model.Source: §D.2 · p. 26

E Per-Level VLM Results

Across VLM hosts, FRM gains are concentrated at higher degradation levels, where feature displacement is larger. Frozen-encoder evaluation compares recovered degraded features with paired pristine features and uses degraded-to-pristine retrieval.

E Per-Level VLM Results

  • FRM gains are largest at L2-L4 across five frozen VLM hosts and four object-level tasks, while L0-L1 gains are small, matched, or occasionally negative.The reported pattern associates greater usefulness with larger feature displacement under physical transformation.Source: E Per-Level VLM Results · p. 26

F Frozen-Encoder Feature Recovery

  • Frozen CLIP ViT-L/14 and SigLIP2 evaluations compare recovered degraded-image features against paired pristine-image features at the encoder tap point.Source: F Frozen-Encoder Feature Recovery · p. 26
  • For SigLIP2 retrieval, a degraded object image queries a gallery of candidate pristine images, with one paired pristine target per query and a 100% pristine-query ceiling.Source: F Frozen-Encoder Feature Recovery · p. 26

G Additional FRM Ablations

Additional ablations favor downstream-facing VLM taps for retrieval, while DeepStack correction slightly reduces Qwen3-VL identity accuracy. SigLIP2 retrieval gains are also reported, and per-level VLM gains concentrate at higher severity.

G Additional FRM Ablations

  • Across five VLM hosts and four pristine-state tasks, FRM gains concentrate at L2-L4, while L0-L1 gains are smaller or occasionally negative.Source: Figure 10 · p. 27, Table 8 · p. 28
  • Qwen3-VL retrieval R@1 rises from 72.90 pre-merger to 85.28 post-merger, and InternVL3.5 rises from 63.84 pre-projector to 74.92 post-projector.DeepStack correction slightly reduces Qwen3-VL identity accuracy; downstream-facing taps outperform earlier visual-token taps for retrieval.Source: G Additional FRM Ablations · pp. 26–27, Table 9 · p. 28

Related papers

Loading 2609.12078v1…