PaperlayerView exact revision on arXiv

Source-linked AI summary

CoRe-SAM3: Conditional Semantic--Visual Reconciliation for SAM3 Crack Segmentation

Shipeng Liu, Liang Zhao, Dengfeng Chen

arXiv:2609.05816v1cs.CV

TL;DR

Crack segmentation must recover fine, continuous structures while distinguishing them from similar-looking background patterns, and SAM3 can still miss weak cracks or activate crack-like regions. CoRe calibrates SAM3’s semantic decision and uses its native visual evidence for bounded, conditional correction. Across five datasets, it improves average Crack IoU and clDice over native SAM3 with a small trainable-parameter addition.

  • Problem

    Crack segmentation must recover thin, low-contrast, topologically continuous cracks while distinguishing them from crack-like background structures.

    Source: §1 · p. 1

  • Method

    CoRe calibrates SAM3’s semantic prediction and uses spatially aligned native visual evidence to generate a zero-initialized, bounded residual correction.

    Source: §1 · p. 2

  • Results

    Across five crack datasets, average Crack IoU improves from 62.34% to 70.47% and clDice from 81.98% to 89.24%, with 18.914 K trainable parameters added.

    Source: §6 · p. 21

  • Takeaways & Limitations

    CoRe organizes native visual evidence as constrained correction conditioned on semantic belief, providing an adaptation strategy that preserves strong priors while selectively correcting their errors.

    Source: §6 · p. 22

    The conclusions are grounded primarily in curvilinear crack segmentation and SAM3’s internal representation system.

    Source: §5.8.2 · p. 21

Abstract

from arXiv · show

Crack segmentation requires a model to recognize target semantics while accurately recovering thin, low-contrast, and topologically continuous local structures. Although SAM3 provides strong open-concept segmentation, its direct application to the crack domain still misses weak cracks, activates crack-like background regions, and produces local boundary errors. We first diagnose the functional differences between the internal prompt-conditioned semantic representation and native visual representation of SAM3 on five crack datasets. The results show that the semantic representation already carries most task information for crack prediction, whereas the utility of the visual representation depends on the current semantic state. Directly combining the two representations does not yield consistent gains. Based on this finding, we propose Conditional Semantic--Visual Reconciliation, termed CoRe. CoRe retains semantic prediction as the primary decision path, applies lightweight semantic calibration to adjust the target-domain decision mapping, and uses spatially aligned native visual evidence to generate a zero-initialized, bounded, and regularized conditional residual that selectively corrects existing predictions. Across five domains, CoRe-SAM3 improves the average Crack IoU from 62.34% to 70.47% and clDice from 81.98% to 89.24%, while introducing only 18.914 K trainable parameters. Prediction-transition analysis further shows that CoRe corrects an average of 34.38% of native errors, with a damage rate of only 0.23% on pixels correctly classified by native SAM3. These results demonstrate that constrained prediction correction based on the functional differences between internal representations provides an effective and parameter-efficient target-domain adaptation strategy for vision foundation models with strong task-specific semantic priors.

1. Introduction

Crack segmentation requires recovering thin, low-contrast, continuous structures while distinguishing them from visually similar background patterns. CoRe-SAM3 builds on SAM3’s strong semantic prediction, using calibrated semantics as the main decision and native visual evidence for conditional local correction.

1. Introduction

  • SAM3’s strong pretrained crack semantics shift adaptation toward correcting local spatial errors rather than learning a complete crack detector from scratch.Those errors include weak or thin cracks, crack-like background activations, and response expansion or width errors.Source: §1 · pp. 1–2
  • A separate visual encoder could add texture and structural cues, but also increases dependencies and computation and may redundantly encode image information already extracted by SAM3.This motivates testing whether SAM3’s own internal visual evidence can support local correction.Source: §1 · p. 2
  • Across five crack domains, semantic representations carry stronger crack-label information, while direct semantic–visual combination performs worse than semantics alone on four datasets.The native visual representation is also limited for independent prediction and semantic-error identification.Source: §1 · p. 2
  • CoRe-SAM3 calibrates the semantic decision, then uses spatially aligned native visual evidence to condition a zero-initialized, bounded residual correction.Semantic calibration adjusts the target-domain response distribution and decision bias; the residual incrementally corrects the calibrated decision.Source: §1 · p. 2
  • Across five datasets, average Crack IoU rises from 62.34% with native SAM3 to 70.47% with CoRe-SAM3.The reported prediction transitions also show a net positive change in prediction states.Source: §1 · p. 3

2. Related Work

Prior work improves crack segmentation with specialized structure-focused networks and adapts promptable foundation models through parameter-efficient updates. CoRe instead assigns different decision roles to SAM3’s semantic and native visual representations.

2.1. Crack and Curvilinear Structure Segmentation

  • Crack-specific approaches strengthen texture, long-range dependencies, boundaries, or topology, while CoRe examines how a foundation model’s existing visual evidence can support its strong crack semantics.The surveyed approaches include convolution–Transformer, Mamba-style, and full-resolution methods.Source: §2.1 · pp. 3–4

2.2. Promptable Segmentation Foundation Models and Downstream Adaptation

  • Promptable segmentation work extends prompts and contexts, while downstream adaptation methods add adapters or tuning; these methods generally do not distinguish internal representations’ functional roles in existing predictions.CoRe assigns semantic evidence control of the main decision and visual evidence conditional local correction.Source: §2.2 · p. 4

2.3. Parameter-Efficient Adaptation and Reuse of Internal Evidence

  • Parameter-efficient methods learn limited increments through prompts, adapters, scaling, or low-rank updates, but CoRe grounds its increments in the different task roles of semantic and visual evidence.Semantic evidence remains the primary decision carrier and receives lightweight calibration; native visual evidence supplements local residual correction under the semantic state.Source: §2.3 · pp. 4–5

3. Preliminary: Semantic–Visual Reconciliation in SAM3

SAM3 already carries strong crack semantics, but its predictions retain local errors involving weak cracks, crack-like backgrounds, and boundaries. Five-domain probing motivates retaining semantic prediction as the primary decision and conditioning limited corrections on native visual evidence.

3.1. Crack-Prediction Errors in SAM3

  • SAM3 errors include missed thin or low-contrast cracks, false positives on crack-like patterns, and boundary errors on correctly identified crack bodies.Source: §3.1 · p. 5

3.3. Representation Diagnosis and Design Implications

  • Across five domains, AP([S, V] → E) = 0.9431, 0.0078 below AP(S → E) = 0.9509; joint probing gains over Error-S on only one domain.Figure 2 reports the representation probes and per-domain joint gains.Source: Figure 2 · p. 6, §3.3 · p. 6

3.2. Semantic and Visual Evidence in SAM3

  • Prompt-conditioned S carries stronger crack-label separability than native visual V across all five datasets, making S the primary task representation.V retains local appearance and spatial structure that can serve as complementary evidence under the current semantic state.Source: §3.2 · p. 5, §3.3 · p. 6

3.3. Representation Diagnosis and Design Implications

  • Semantic S also retains substantial information about prediction reliability, while directly combining S and V does not yield stable complementarity across domains.Source: §3.3 · p. 6
  • CoRe-SAM3 reframes adaptation as evidence reconciliation: retain and calibrate semantic prediction, then use native visual evidence to condition limited local corrections.This design follows the finding that visual evidence is useful in relation to the current semantic state.Source: §3.3 · p. 7

4. Method

CoRe adapts SAM3 by preserving semantic prediction as the primary decision path and using native visual features to condition a constrained residual correction. It calibrates the semantic readout first, then trains the reconciliation branch with segmentation and residual objectives.

4. Method

  • CoRe retains SAM3’s semantic representation as the primary decision source, calibrates its target-domain readout, and uses native visual evidence to condition limited corrections to remaining local errors.Source: §4 · p. 7

4.1. Diagnosis-Driven Adaptation Framework

  • The frozen SAM3 representations have distinct roles: semantic features produce the calibrated primary prediction, while aligned visual features condition an incremental residual update.The backbone, prompt encoder, conditioned encoder, and Pixel Decoder remain frozen.Source: §4 · p. 7, §4.1 · p. 8

4.2. Target-Domain Semantic Decision Calibration

  • CoRe updates SAM3’s native semantic projection under target-domain supervision to adjust the decision mapping from frozen semantic features to crack logits.This retargeting leaves the reconciliation branch to address local discrepancies that remain after calibration.Source: §4.2 · p. 8

4.3. Bounded Conditional Semantic–Visual Reconciliation

  • The reconciliation branch jointly uses semantic and visual evidence to produce a zero-initialized residual bounded to [−s, s], preserving the calibrated prediction initially.The bounded update limits visual evidence’s maximum effect on the final logit.Source: §4.3 · p. 9

4.4. Optimization Objective and Conservative Residual Regularization

  • Training combines boundary-aware weighted BCE, Dice loss, and residual regularization to address nearby crack-boundary errors while favoring small corrections.Background pixels immediately adjacent to cracks receive higher weights; the regularizer favors smaller residual updates.Source: §4.4 · pp. 9–10

5. Experiments

The experiments assess cross-domain performance, design choices, correction behavior, qualitative results, and computational efficiency. They use five public crack datasets, a fixed SAM3 setting for controlled comparisons, and metrics covering overlap, topology, and predicted foreground area.

5. Experiments

  • The experiments examine adaptation across crack domains, key design choices, prediction-correction behavior, qualitative results, and computational efficiency.Source: §5 · p. 10

5.1.1. Datasets

  • Evaluation uses CamCrack789, Crack500, CrackMap, DeepCrack, and TUT, whose image resolutions, textures, crack scales, and morphologies vary substantially.Compared methods use the same training, validation, and test splits within each dataset.Source: §5.1.1 · p. 10, Table 1 · p. 11

5.1.2. Implementation Details

  • SAM3-based methods use 1008×1008 inputs; CoRe freezes the backbone and major SAM3 encoders, optimizing only the semantic projection and reconciliation branch.The reported training protocol uses three fixed random seeds and selects models by domain-specific validation Crack IoU.Source: §5.1.1 · p. 10, §5.1.2 · p. 11

5.1.3. Evaluation Metrics

  • Crack IoU is the primary metric, with mIoU, F1-score, Precision, Recall, clDice, and predicted area ratio also reported.clDice evaluates connectivity and centerline structure; area ratio indicates global foreground expansion or contraction.Source: §5.1.3 · p. 11

5.2. Overall Segmentation Performance

CoRe-SAM3 improves native SAM3 across five crack datasets while using a small adaptation parameter budget. The gains extend across multiple metrics, though Crack500 remains a case where a higher-capacity SAM3 method performs better.

5.2.1. Performance across Five Crack Domains

  • CoRe-SAM3 uses 18.91 K trainable parameters and leads SAM3-based methods on four datasets; its mean Crack IoU exceeds SAM3-UNet by 1.86 points.SAM3-UNet and SAM3-adapter use approximately 146 and 221 times as many trainable parameters, respectively, while HAT-SAM3 uses more than 4500 times as many.Source: §5.2.1 · pp. 11–12
  • The five-domain mean Crack IoU rises from 62.34% to 70.47%, an 8.13 percentage-point gain, with improvements on all five datasets.The comparison covers CamCrack789, Crack500, CrackMap, DeepCrack, and TUT.Source: §5.2.1 · p. 11, Table 2 · p. 12
  • On Crack500, CoRe-SAM3 improves native SAM3 from 41.33% to 57.28% Crack IoU, but SAM3-UNet reaches 61.21%.The authors identify substantial spatial reconstruction relative to SAM3's original response as a setting where a higher-capacity decoder may retain an advantage.Source: §5.2.1 · p. 12
  • CoRe-SAM3's 70.47% mean Crack IoU is slightly above RIFT-T's 70.04%, but the dedicated-network comparison is a task-level reference under different conditions.Dedicated networks and SAM3-based methods differ in foundation-model scale, pretraining, and inference path.Source: §5.2.1 · p. 12, Table 2 · p. 12

5.2.2. Multi-Metric Evaluation

  • Recall rises from 74.16% to 85.60%, Precision improves by 5.24 percentage points, and clDice increases from 81.98% to 89.24%.PAR increases from 0.939 to 1.124, shifting predicted foreground area from slight underestimation to moderate expansion; the authors also report suppression of erroneous responses.Source: §5.2.2 · p. 12, Table 3 · p. 13

5.3. Validation of the Reconciliation Design

The validation tests semantic calibration, ways of incorporating visual evidence, residual constraints, and crack-aware optimization. Its results support retaining semantic prediction as the basis and using visual evidence for bounded correction.

5.3. Validation of the Reconciliation Design

  • The validation follows the diagnosis that prompt-conditioned semantics carry primary task information, while direct concatenation with native visual features gives inconsistent gains across domains.The design therefore tests calibration, the role of visual evidence, and residual constraints in sequence.Source: §5.3 · p. 13

5.3.1. From Semantic Calibration to Conditional Reconciliation

  • On CamCrack789 and CrackMap, semantic calibration adds 2.50 and 3.55 Crack IoU points; semantic-only residuals reach 73.53% and 63.30%, then visual evidence lifts these to 74.71% and 65.78%.On CrackMap, adding visual evidence also raises clDice from 85.14% to 89.09%.Source: §5.3.1 · p. 13

5.3.2. Residual Reconciliation versus Direct Feature Fusion

  • Direct S+V fusion gives 66.04% Crack IoU, near semantic calibration's 66.07%; conditional S+V residual reaches 68.55%, and the full design reaches 70.24%.With the semantic prior retained, an S-only residual reaches 67.96%; adding visual evidence raises clDice from 88.77% to 90.52%, while the full bounded design reaches 91.89% clDice.Source: Table 5 · p. 14, §5.3.2 · p. 14

5.3.3. Effect of Conservative Residual Constraints

  • Stronger residual constraints raise Crack IoU as CR falls from 30.64% to 29.16% and DR from 0.33% to 0.18%, indicating more selective prediction changes.Zero initialization starts from the existing semantic solution, bounded updates limit correction magnitude and reduce DR, and residual regularization achieves the highest Crack IoU.Source: Table 6 · p. 14, §5.3.3 · p. 14

5.3.4. Effect of Crack-Aware Optimization

  • Ring weighting raises Precision from 75.36% to 79.09% and lowers PAR from 1.243 to 1.150; residual regularization reaches 70.24% Crack IoU and 91.89% clDice.The authors report that ring weighting improves Crack IoU and clDice, while residual regularization preserves Precision and recovers part of the lost Recall.Source: §5.3.4 · p. 15, Table 7 · p. 15

5.4. Decision Correction Analysis

CoRe changes native SAM3 decisions selectively: it corrects a substantial share of errors while damaging few correct predictions. Its adjustments generally suppress false positives and strengthen false negatives.

5.4.1. Correction of Native Errors and Preservation of Correct Decisions

  • 34.38% of native errors were corrected on average across five domains, versus a mean damage rate of 0.234% on correctly classified pixels.Class-conditional transitions show TN→FP below 0.44% on every dataset, while TP→FN ranges from 2.29% to 6.41%.Source: §5.4.1 · pp. 15–16
  • CoRe converted 52.18%–52.62% of native false positives to true negatives on three datasets, while false-negative recovery reached 20.51%–23.26% on three datasets.The reported false-positive transitions are for CamCrack789, Crack500, and DeepCrack; the false-negative transitions are for CrackMap, Crack500, and TUT.Source: §5.4.1 · p. 16

5.4.2. Direction of Prediction Adjustment

  • Negative adjustments dominate native false-positive regions, while positive adjustments dominate false-negative regions across domains.Across five domains, 88.64%–98.72% of false-positive pixels receive negative adjustments and 68.95%–91.49% of false-negative pixels receive positive adjustments.Source: §5.4.2 · p. 16, Figure 5 · p. 17
  • Most true-positive adjustments change confidence without changing the final class, alongside target-domain recalibration and local conditional compensation.Only approximately 0.4%–6.8% of TP pixels receive positive adjustments; most TP adjustments preserve the final class state.Source: §5.4.2 · pp. 16–17

5.5. Hyperparameter Sensitivity Analysis

Across the tested ranges on CamCrack789 and CrackMap, CoRe-SAM3's mIoU changes little with the five examined hyperparameters. The authors use a unified default configuration across domains.

5.5. Hyperparameter Sensitivity Analysis

  • Across the tested ranges, varying residual magnitude, Dice loss weight, and regularization weight produced only small mIoU changes on CamCrack789 and CrackMap.On CamCrack789, the default mIoU was 0.8695; on CrackMap, the default was 0.8214.Source: §5.5 · p. 18
  • Changing morphological radius and outer-ring weight also had little effect on mIoU within the evaluated ranges on both datasets.For example, CamCrack789 mIoU was 0.8695 at the default radius 3 and 0.8699 at radius 1.Source: §5.5 · p. 18, Figure 7 · p. 18
  • The unified defaults are s=0.5, r=3, ω=4, λ_d=0.25, and λ_r=0.05, supporting stable results without a narrow parameter search.The sensitivity evaluation varied one parameter at a time while fixing all others at their defaults.Source: §5.5 · pp. 17–18

5.6. Qualitative Analysis

Qualitative examples show CoRe recovering weak crack responses, suppressing crack-like false activations, and preserving reliable native predictions through localized adjustments. These behaviors reflect selective correction rather than wholesale reconstruction.

5.6. Qualitative Analysis

  • CoRe restores weak or discontinuous crack regions, extending responses mainly along existing structures while largely preserving the main crack's location and direction.The examples show reduced false-negative regions and improved continuity in thin branches and local interruptions.Source: §5.6 · pp. 18–19, Figure 8 · p. 19
  • CoRe selectively weakens unsupported activations on joints, textures, shadows, and linear edges while preserving true crack responses.The qualitative suppression matches the reported 88.64%–98.72% of negative adjustments in native false-positive regions.Source: §5.6 · pp. 19–20, Figure 9 · p. 19
  • When native SAM3 predictions are reliable, CoRe mainly makes small boundary and width adjustments while preserving most true-positive regions and overall crack geometry.This conservative behavior is consistent with the bounded residual and low damage rate.Source: Figure 10 · p. 19, §5.6 · p. 20
  • Together, the examples associate CoRe's gains with FN recovery, FP suppression, and TP preservation rather than global reconstruction of SAM3 outputs.The examples compare input, ground truth, native prediction, and CoRe prediction, with colors identifying TP, FP, and FN pixels.Source: §5.6 · pp. 18, 20

5.7. Efficiency Analysis

CoRe reuses SAM3's internal representations and adds a lightweight reconciliation branch. Its parameter, compute, and memory overheads are small, although measured end-to-end latency increases by about 19.5%.

5.7. Efficiency Analysis

  • CoRe adds 18.914 K trainable parameters, increases total FLOPs by approximately 0.16%, and leaves peak memory unchanged at 3889.7 MiB.Total FLOPs rise from 3.672 T to 3.678 T, an additional 6.211 GFLOPs.Source: §5.7 · p. 20
  • End-to-end latency increases from 451.2 ms to 539.1 ms, or approximately 19.5%.The measured increase exceeds the relative FLOP change; the authors attribute practical overhead to feature access, low-dimensional projections, and multiple small operators.Source: §5.7 · p. 20
  • CoRe reuses SAM3's semantic and visual representations and introduces a small branch, with gains attributed primarily to reorganizing existing internal evidence.Efficiency measurements cover the complete SAM3 forward pass without cached features.Source: §5.7 · pp. 20–21

5.8. Discussion

CoRe uses conditional residual correction to preserve SAM3’s semantic prior, which carries most crack-task information, while constraining native visual evidence to adjust local errors. The discussion reports gains with few trainable parameters and limits its conclusions to curvilinear crack segmentation and SAM3’s internal representations.

5.8.1. Implications of Conditional Reconciliation

  • Five-domain probing finds semantic representations carry most crack-discrimination information, while direct semantic–visual fusion gives no stable advantage over retaining the semantic prior.The native visual representation is better suited as supplementary evidence conditioned on the current semantic state.Source: §5.8.1 · p. 21
  • CoRe selectively corrects existing semantic beliefs, predominantly lowering false-positive regions and raising false-negative regions while preserving most correct decisions.Progressively adding zero initialization, a bounded residual, and residual regularization improves Crack IoU while reducing damage to correct native predictions.Source: §5.8.1 · p. 21
  • With 18.914 K trainable parameters, CoRe improves five-domain average Crack IoU from 62.34% to 70.47%.The paper presents this as evidence for analyzing internal representation roles and constraining the decision authority of adaptation branches.Source: §5.8.1 · p. 21

5.8.2. Limitations and Future Work

  • The conclusions are grounded primarily in curvilinear crack segmentation and SAM3’s internal representation system, including same-scale semantic–visual representations.The authors propose evaluating other fine-structure segmentation tasks and vision foundation models in future work.Source: §5.8.2 · p. 21

6. Conclusion

The paper adapts SAM3 for crack segmentation by retaining its semantic decision and using spatially aligned native visual evidence for constrained residual correction. Across five crack datasets, CoRe improves average Crack IoU and clDice with 18.914 K trainable parameters, while selectively correcting native errors.

6. Conclusion

  • CoRe retains and recalibrates SAM3’s semantic decision, then applies a zero-initialized, bounded, regularized residual from spatially aligned native visual evidence.The paper frames this as target-domain semantic calibration followed by conditional semantic–visual reconciliation.Source: §6 · p. 21
  • Across five crack datasets, average Crack IoU rises from 62.34% to 70.47% and clDice from 81.98% to 89.24%, with 18.914 K trainable parameters.These results are reported for CoRe-SAM3.Source: §6 · p. 21
  • Ablations find direct semantic–visual fusion similar to semantic calibration, while prior-retaining residual reconciliation gains with successive constraints and reduces damage.The introduced constraints are zero initialization, a bounded residual, and residual regularization.Source: §6 · p. 21
  • Across five domains, CoRe corrects an average of 34.38% of native errors with approximately 0.23% damage to correct pixels; FP and FN regions predominantly receive negative and positive adjustments, respectively.Source: §6 · pp. 21–22
  • The results support constrained correction conditioned on semantic belief as an adaptation strategy for fine-grained segmentation that preserves strong priors while selectively correcting their errors.Source: §6 · p. 22
Loading 2609.05816v1…