PaperlayerView exact revision on arXiv

Source-linked AI summary

SAM3-LoRA: Parameter-Efficient Adaptation of a Concept-Promptable Foundation Model for Multi-Class Structural Defect Segmentation

P. Malaisree, S. Youwai, S. Janrungautai, D. Amorndechaphon, P. Rojanavasu, W. Songkitti

arXiv:2609.00469v1cs.CV

TL;DR

The paper asks how to adapt open-vocabulary SAM3 for multi-class structural-defect segmentation without the cost of full fine-tuning and how to supervise it from conventional annotations. It applies LoRA with category-name prompts and exhaustive hard-negative queries, finding consistent gains across a tunnel dataset and an independent public benchmark. The authors conclude that parameter efficiency transfers across datasets, although generalization and negative-prompt performance remain incompletely established.

  • Problem

    COCO-style class-labeled annotations lack direct language-conditioned supervision, while full fine-tuning of SAM3 is computationally prohibitive for resource-constrained inspection organizations.

  • Method

    The study adapts SAM3 with LoRA, uses category names verbatim as prompts, groups all same-category instances into exhaustive queries, and adds hard-negative prompts for absent categories.

  • Results

    Pixel IoU improved approximately 20-fold and 52-fold, while instance-level recall improved 1.7- to 1.8-fold across the tunnel and independent S2DS datasets.

  • Takeaways & Limitations

    Adapter placement interacts with the task at least as strongly as parameter capacity, and benefits concentrate where zero-shot competence is absent.

  • Takeaways & Limitations

    Generalization beyond two concrete and tunnel-lining defect datasets is untested, and the study reports no quantitative negative-prompt benchmark.

Abstract

from arXiv · show

Promptable segmentation foundation models such as SAM3 accept an open-vocabulary text concept and return every instance matching it, but adapting them to a specialized domain by full fine-tuning is computationally prohibitive for the organizations that would benefit most. This study applies Low-Rank Adaptation (LoRA) to SAM3 for multi-class structural defect segmentation and examines both how such a model can be supervised from conventional annotation and whether the resulting efficiency gain transfers across datasets. Two contributions are methodological. First, we describe a supervision procedure that trains a concept-promptable model directly from COCO-style class-labeled instance segmentation by using the category name itself as the prompt, requiring no prompt templates, no synonym expansion, and no learned class embeddings. Second, we identify and mitigate a failure mode specific to this setting: because a conventional annotation file yields positive prompts exclusively, the model's presence prediction decouples from the text condition and degenerates into responding to any prompt, a collapse that is invisible to every metric computed on positive prompts alone. Exhaustive hard-negative prompting, in which every dataset category absent from an image is issued as a zero-detection query, addresses this at no annotation cost. Two adapter placements were compared under an identical protocol, updating 0.121% and 1.341% of model parameters. On a purpose-built tunnel lining dataset, pixel intersection-over-union improved from 0.017 to 0.338 and instance-level recall from 0.375 to 0.672; on the independent public Structural Defects Dataset, from 0.017 to 0.855 and from 0.574 to 1.000. Improvements were directionally consistent across ten metrics on both datasets, and the largest per-category gains occurred precisely where zero-shot competence was absent.

I. INTRODUCTION

The paper addresses how to adapt open-vocabulary SAM3 to multi-class structural-defect segmentation under limited compute, while resolving supervision gaps introduced by conventional class-labeled annotations.

  • Research motivation: COCO-style class labels do not directly supervise a language-conditioned segmentation model, creating an explicit class-to-prompt supervision problem.Concept-promptable models condition on language rather than a fixed classification head.
  • Research motivation: Full fine-tuning of large vision-transformer foundation models is computationally expensive, motivating parameter-efficient adaptation for single-GPU inspection agencies.Prior LoRA work largely focused on single-class crack segmentation with first-generation SAM.
  • Research questions: The study examines adapter placement and cross-dataset transfer in a genuinely multi-class tunnel-inspection setting.Evaluation uses both a purpose-built tunnel lining dataset and the independent public Structural Defects Dataset under an identical protocol.
  • Contributions: Category names are used verbatim as prompts, and annotations are grouped into exhaustive per-concept queries without prompt templates, synonym expansion, or learned class embeddings.This trains the concept-promptable model directly from conventional instance-segmentation annotations.
  • Contributions: Positive-only prompting creates a presence–text decoupling failure, while exhaustive hard-negative prompting mitigates it without additional annotation cost.The failure can remain invisible when evaluation uses only positive prompts.

C. Parameter-efficient adaptation of SAM in specialized domains

Prior infrastructure adaptations mainly used SAM or LoRA for single-class or fully fine-tuned settings, leaving multi-class concept prompting and controlled cross-dataset efficiency comparisons open.

  • Prior work: Prior civil-infrastructure LoRA studies treated crack segmentation as a single-class problem and used first-generation SAM.They therefore did not address the class-to-prompt or negative-prompt issues examined here.
  • Prior work: Existing SAM-based infrastructure studies evaluated zero-shot or fully fine-tuned settings without a controlled parameter-efficient comparison.This leaves the efficiency contribution of adaptation insufficiently isolated.
  • Cross-dataset motivation: Single-dataset evaluation cannot distinguish a generally effective adaptation method from one exploiting dataset-specific imagery or annotation conventions.The paper uses S2DS as an independent generalization test with the identical tunnel-dataset protocol.
  • Problem formulation: The proposed formulation maps category annotations to language-conditioned queries rather than to a fixed classification head.Concept-promptable models return masks and confidence estimates for a free-form text prompt.

B. Architecture

SAM3 is adapted with LoRA modules placed in selected visual and fusion components while the pretrained text encoder remains frozen to preserve prompt discrimination.

  • Architecture: SAM3 combines a Vision Transformer image encoder, multi-scale neck, CLIP-style text encoder, geometry encoder, and downstream fusion, detection, and segmentation components.The image encoder processes 1008×1008 inputs with 14-pixel patches, width 1024, 32 blocks, and 16 heads.
  • Low-rank adaptation: LoRA updates a frozen projection through trainable low-rank factors A and B, with rank r ≪ min(d_in, d_out) and scaling factor α.The adapter is initialized so the model matches the pretrained network before learning.
  • Low-rank adaptation: The study fixes r = 16, α = 32, and adapter dropout probability 0.1 across configurations.These settings define the common low-rank adaptation protocol.
  • Adapter placement: Attention blocks are re-expressed with explicit query, key, value, and output projections so fused pretrained weights can be adapted without changing the forward computation.The reparameterization copies the corresponding slices of the fused matrix exactly.
  • Adapter placement: The Light configuration adapts 124 width-256 layers and updates 0.121% of model parameters, while the Full configuration extends adaptation to the Vision Transformer trunk.The text encoder remains frozen in both configurations.
  • Adapter placement: Freezing the text encoder preserves the pretrained concept geometry while visual and fusion pathways specialize to defect appearance.The paper also links this choice to generalization of negative supervision beyond concepts seen during training.

D. Supervising a concept-promptable model from COCO class labels

The supervision conversion uses COCO category names as literal text prompts and groups all instances of each present category into exhaustive concept queries.

  • Annotation conversion: The input datasets provide COCO-style images, category tables, and per-instance identifiers, boxes, and polygon or run-length masks.These annotations must be converted for a language-conditioned model.
  • Class-to-concept mapping: Each category name is lower-cased and used verbatim as the training-query text.The procedure adds no prompt template, synonym expansion, or learned class embedding.
  • Class-to-concept mapping: The procedure is bounded by label quality because opaque codes such as “C1” or “type-2” require a separate mapping to descriptive names.A frozen text encoder cannot meaningfully contribute to an unfamiliar token.
  • Exhaustive queries: Annotations are grouped by category, producing one query for each category present with every instance of that category as its target set.A query such as “crack” is therefore trained to return all cracks rather than only the most salient one.
  • Geometric conversion: Images are resized to 1008 × 1008, boxes are normalized, and polygon or run-length masks are decoded and resampled with nearest-neighbor interpolation.Nearest-neighbor resampling avoids blurring the single-pixel-wide structures common in crack annotations.

E. Hard-negative prompt supervision

Positive-only supervision can decouple SAM3’s presence prediction from the text prompt, causing responses to arbitrary concepts. Exhaustive in-domain hard negatives, supplemented by safeguarded out-of-domain negatives and validation negatives, restores prompt discrimination without new annotations.

  • E. Hard-negative prompt supervision: Positive-only prompts make the presence estimate decouple from the text condition and degenerate into always firing.This collapse is invisible when evaluation uses only positive prompts.
  • E. Hard-negative prompt supervision: Negative queries supervise all 200 object queries toward no-object and the presence token toward absence.The same detection-and-segmentation mechanism is used for positive and negative queries.
  • E. Hard-negative prompt supervision: Exhaustive in-domain negatives issue every absent dataset category as a zero-detection query for each image.On three-category tunnel images, this contributes at most two negative queries per annotated image; on single-category S2DS images, exactly one.
  • E. Hard-negative prompt supervision: Randomly sampling negatives was too sparse to separate visually similar defect categories, which otherwise converged on the same segmented region.Exhaustive inclusion supplies the repeated contrast needed for confusable classes.
  • E. Hard-negative prompt supervision: Out-of-domain negatives sample up to three generic concepts per image after removing pool entries sharing words with dataset categories.The safeguard prevents contradictory supervision, such as querying “water” negatively on water-ingress images.
  • E. Hard-negative prompt supervision: With the text encoder frozen, negatives train a general rule to fire only when prompt embeddings match visual evidence.The expected generalization boundary is near-synonyms of target categories.
  • E. Hard-negative prompt supervision: Validation negatives ensure early stopping reflects prompt discrimination as well as mask quality on positive prompts.This extends negative supervision beyond the training split.
  • E. Hard-negative prompt supervision: The tunnel dataset contains co-located multi-class defects, whereas S2DS is single-category per image and lacks a water-ingress equivalent.The datasets share no images or annotations, providing structurally different settings for the supervision procedure.

B. Training configuration

Training used fixed hardware and hyperparameters across Light and Full LoRA configurations, with the frozen SAM3 baseline evaluated through the same pipeline. Evaluation combined detection, segmentation, pixel, prompt-level, and instance-level metrics while correcting an S2DS scoring artifact.

  • B. Training configuration: Both LoRA configurations used rank 16, dropout 0.1, weight decay 0.01, and learning rate 1×10−4, with AdamW updating adapter matrices only.Training used two RTX 5090 GPUs; Full required batch size 1 per GPU on S2DS because of its larger adapted footprint.
  • B. Training configuration: The frozen SAM3 baseline used identical category-name prompts and the identical evaluation pipeline, isolating adaptation as the experimental difference.No adaptation was applied to the baseline.
  • B. Training configuration: Ten metrics covered detection, category-grouped F1, pixel quality, mean per-prompt IoU, and instance-level recall.Pixel metrics were computed per image-prompt unit at 288×288 mask resolution.
  • B. Training configuration: Pixel-level metrics complement instance metrics because fragmented predictions can cover a crack accurately while failing one-instance matching.A split prediction can incur false negatives and false positives despite accurate combined coverage.
  • B. Training configuration: Correcting an S2DS indexing error changed pixel IoU from approximately 0.02 to approximately 0.86 without changing weights or predictions.The artifact also affected prediction–ground-truth association and per-category evaluation.
  • B. Training configuration: Full LoRA reached tunnel instance-level recall 0.672, while Light LoRA reached tunnel pixel IoU 0.338; both exceeded the frozen baseline values of 0.375 and 0.017.On S2DS, Light reached pixel IoU 0.8554 and both adapters reached instance-level recall 1.000 versus 0.574 for baseline.

B. Cross-dataset consistency of the improvement

Adaptation improved performance consistently across datasets and metrics, with especially large gains for categories poorly handled by frozen SAM3. The cross-dataset pattern is supported by aggregate multipliers, absolute detection results, and per-category comparisons.

  • B. Cross-dataset consistency of the improvement: Every non-degenerate metric improved by 1.7-fold to 51.8-fold on at least one dataset, with directional improvement for every metric computable on both.This establishes consistency across the ten-metric evaluation rather than a single-score effect.
  • B. Cross-dataset consistency of the improvement: S2DS baseline detection metrics were exactly zero, while adapted models reached test mAP 0.094–0.098 and cgF1 0.112–0.135.The adapted S2DS detection values were above the corresponding tunnel ranges.
  • C. Per-category results: Adaptation benefited categories inversely related to the frozen model’s prior competence on both datasets.The relationship is limited to the two categories shared between datasets.
  • C. Per-category results: Tunnel pixel IoU rose from 0.011 to 0.395 for concrete spalling and from 0.000 to 0.378 for water ingress.Crack increased from 0.199 to 0.337, showing smaller improvement than the weakest baseline categories.
  • C. Per-category results: On S2DS, spalling rose from 0.0000 to 0.8720 and crack from 0.1533 to 0.4317 after adaptation.These correspond to an unbounded gain and a 2.8-fold gain, respectively.
  • C. Per-category results: Crack remained the lowest post-adaptation pixel-IoU category despite comprising 70.8% and 70.9% of annotations in the two datasets.Its frozen predictions had high recall but low IoU, indicating broad and imprecise coverage.
  • C. Per-category results: Both datasets assigned zero baseline pixel IoU and recall to the category without a natural-image pretraining analogue: water ingress on tunnel data and spalling on S2DS.This directly identifies where zero-shot competence was absent.

D. Adapter capacity

The two adapter placements trade off detection-oriented performance against pixel-level generalization rather than producing a universal winner. The larger configuration fits faster and therefore requires earlier stopping.

  • Neither adapter placement dominated: Full favored detection-oriented metrics on the tunnel dataset, while Light favored pixel-level generalization.
  • Full reached its validation-loss minimum earlier than Light on both datasets and diverged more rapidly afterward.On the tunnel dataset, Full minimized at epoch 1 versus Light at epoch 5; on S2DS, the corresponding epochs were 3 and 7.
  • The qualitative examples show smaller, mixed differences among adapted models: Light often produced more contiguous masks, while Full recovered more branching crack structure in one case.

A. The efficiency gain is a property of the method rather than of one dataset

Adaptation improved segmentation and detection consistently across two independently collected datasets, supporting a method-level efficiency gain while leaving adapter-capacity comparisons confounded across implementations.

  • Pixel IoU improved approximately 20-fold on the tunnel dataset and 52-fold on S2DS, while instance-level recall improved approximately 1.7- to 1.8-fold on both.
  • Improvements were directionally consistent across ten independently computed metrics on both datasets under an identical protocol.
  • The frozen baseline over-predicted large areas or missed categories outside pretraining, whereas both adapted configurations localized true defect regions more effectively.
  • The tunnel and S2DS experiments used different adapter scopes, so their Light-versus-Full capacity comparison cannot be pooled across datasets.
  • Within the controlled tunnel comparison, an 11.2-fold parameter increase did not uniformly improve results, and the smaller placement achieved higher pixel IoU.

C. Benefit is concentrated where zero-shot competence is absent

Adaptation helped most on defect categories that the frozen model handled poorly, while the category with partial zero-shot competence benefited least. This pattern was visible across both datasets but is bounded by the available category overlap.

  • Categories handled least well by the frozen model gained most from adaptation, while the partially handled category gained least on both datasets.
  • Crack comprised roughly 71% of annotations yet attained the lowest post-adaptation pixel IoU on both datasets: 0.337 on tunnel and 0.4317 on S2DS.
  • Spalling and water ingress began from near-zero baselines and reached comparable or higher absolute values after adaptation.
  • Training-set frequency was a poor predictor of per-category benefit, with the majority class benefiting least on both datasets.
  • The relationship becomes legible only by pairing baseline and adapted values across datasets, and the finding is limited to the two shared categories because water ingress appears in only one.

D. Thin-structure geometry and the choice of metric

Thin crack geometry makes pixel IoU and instance metrics understate visually plausible segmentation, so evaluation should interpret both metric families together. Fragmentation further penalizes disconnected predictions.

  • Cracks’ thin, elongated geometry can produce high pixel recall but low pixel IoU after small boundary errors.The frozen S2DS baseline illustrates this pattern with recall 0.9911 and IoU 0.1533.
  • A single fragmented crack can be scored as one missed instance and two false positives.
  • Pixel IoU of 0.8554 versus mAP@50 of 0.2207 on S2DS shows how target geometry affects metric interpretation.Absolute mAP values for crack-dominated datasets are not directly comparable with those for compact-object datasets.
  • Reporting instance metrics alone can misrepresent adapted-model quality when connectivity errors contribute substantially to detection scores.Fragment-merging post-processing is identified as the natural first intervention for this failure mode.

E. Deployment implications and residual domain gap

LoRA adaptation was feasible on a single workstation and transferred across two structural-defect datasets, but performance remained domain-dependent and several scope and evaluation limitations persisted. The study links deployment practicality to both parameter efficiency and supervision design.

  • Residual domain gap: The best test pixel IoU was 0.338 on the tunnel dataset versus 0.8554 on S2DS despite near-identical frozen baselines.The tunnel scenes contain more multi-defect cooccurrence and curved, segmented lining geometry; this attribution is unverified.
  • Residual domain gap: Only two concrete and tunnel-lining datasets were evaluated, leaving generalization beyond this visual domain untested.
  • Residual domain gap: The cross-dataset adapter-capacity comparison is confounded by differing implementation revisions.
  • Residual domain gap: Per-category results are indicative rather than precise where evaluation cells contain as few as n = 5 examples.
  • Supervision design: Class-to-concept supervision uses category names verbatim and exhaustive hard-negative queries, while the text encoder remains frozen.The procedure requires no prompt templates or learned class embeddings and mitigates presence–text decoupling at no annotation cost.
  • Deployment implications: Pixel IoU improved approximately 20-fold and 52-fold, while instance-level recall improved 1.7- to 1.8-fold across the tunnel and S2DS datasets.Both adapter placements substantially outperformed the frozen baseline on both datasets.
  • Supervision design: The study concludes that supervision design deserves attention alongside adapter choice for practical specialized deployment.
Loading 2609.00469v1…