PaperlayerView exact revision on arXiv

Source-linked AI summary

DCM-SAM: Defect-Conditioned Mixture of LoRA Experts for NPU-Deployed AM Defect Segmentation

Md Mushfiqur Rahaman, Md Mahedi Hasan, Imtiaz Ahmed, Srinjoy Das

arXiv:2609.38811v1cs.CVcs.LG

TL;DR

AM XCT inspection needs accurate segmentation of tiny defects despite scarce real labels and on-device deployment constraints. DCM-SAM adapts frozen SAM with separate promptless Conv-LoRA heads for each defect class. It reports better results than XCT-SAM’s baselines and 64.2% pore IoU on real NIST scans, while running at full resolution on the target NPU.

  • Problem

    AM XCT defects are only a few pixels wide, annotated real volumes are rare, and models trained on one alloy and scanner are routinely asked to work on another.

    Source: §1 · p. 1

  • Method

    DCM-SAM uses a frozen SAM backbone with a separately trained, promptless Conv-LoRA expert bank and mask decoder for each defect class.

    Source: §1 · p. 2

  • Results

    64.2% pore IoU on real NIST scans; DCM-SAM also outperforms all three XCT-SAM [8] settings, whose models use ViT-H while DCM-SAM uses ViT-B.

    Source: §3.2 · p. 5, §4 · p. 6

  • Takeaways & Limitations

    The complete adapted model runs at full resolution on the IQ-9075 with every operator on the NPU and masks within 0.01% of pixels of the FP32 reference.

    Source: §4 · p. 6

    The XCT-SAM [8] baseline comparison is indicative rather than protocol-matched, and every configuration in the mixture-size study was trained once.

    Source: G Limitations and next steps · p. 12

Abstract

from arXiv · show

Metal additive manufacturing parts are inspected by X-ray computed tomography, where labelled data is scarce, the pores and inclusions that matter span a few pixels, and inspection must happen at the machine. We present DCM-SAM, a defect-conditioned adaptive mixture of LoRA experts: one frozen Segment Anything backbone carries a separate Conv-LoRA expert bank and mask decoder per defect class, each trained in its own pass, without prompts, on synthetic slices alone, updating only 4.4% of the parameters. On benchmarks that XCT-SAM reports, DCM-SAM improves on every baseline for both classes from a ViT-B backbone against their ViT-H, and reaches 64.2% pore IoU on real NIST scans having seen no real images during training. Deployment then exposes what adaptation work rarely measures: on a Qualcomm Hexagon NPU, ViT-H and ViT-L compile yet cannot allocate at 1024x1024 image resolution, since activations rather than weights exceed the device ceiling, and quantizing weights does not help. ViT-B alone runs, but the adapted encoder then fails to allocate where the stock one succeeds, until a numerically identical rewrite of the attention lets the complete DCM-SAM run in FP16 at 1024x1024, with no operator falling back to the CPU, masks within 0.01% of pixels of the FP32 reference. Code: https://github.com/MushfiqShovon/DCM-SAM.

1 Introduction

AM XCT defects are tiny, real annotations are scarce, and inspection must run on an embedded device, creating a need for accurate full-resolution segmentation. DCM-SAM adapts a frozen SAM with separate defect-class heads and reports improved benchmark results alongside on-device deployment.

1 Introduction

  • AM XCT defects can be a few pixels wide, annotated real volumes are rare, and models trained on one alloy and scanner are routinely applied to another.The passage cites a UNet++ baseline of 13.4% pore IoU on real NIST scans.Source: §1 · p. 1
  • Full input resolution is a hard floor: evaluating the same weights at 512² loses almost half the pore IoU, and retraining at 512² does not recover it.Cloud offload is ruled out by data locality, while workstation-per-machine deployment does not scale across a fleet at comparable cost and power.Source: §1 · p. 2
  • DCM-SAM gives each defect class its own Conv-LoRA expert bank and mask decoder over one frozen SAM backbone, training the heads separately without prompts.Separate losses and sampling prevent cross-head trainable-parameter interference.Source: §1 · p. 2
  • DCM-SAM improves on every XCT-SAM-reported baseline for both defect classes, while deploying from ViT-B against their ViT-H models.The comparison is subject to a protocol caveat.Source: §1 · p. 2
  • Each ViT-B head trains 4.13M parameters, or 4.4% of the model, using a few hundred synthetic slices without an intermediate alloy-microstructure stage or real training images.The reported results carry over to real scans.Source: §1 · p. 2

2 Method

DCM-SAM adapts a frozen SAM ViT-B with promptless, class-specific Conv-LoRA heads and multi-scale experts. Its method also addresses the memory, routing, and attention-graph constraints encountered when deploying the model on an NPU.

2 Method

  • Each defect class has its own encoder path and decoder, sharing frozen SAM weights but no trainable parameters; the encoder runs once per head.The figure specifies two pore experts and eight inclusion experts.Source: Figure 1 · p. 3
  • The mask decoder is trained from scratch, and the prompt encoder supplies only its learned no-prompt embeddings to drive defect localization.The encoder contains 96% of ViT-B parameters and runs once per image.Source: §2 · p. 3
  • A 3 × 3 convolution on an upsampled feature map gives the experts a ladder of receptive fields, important because pore components span a 40× area range within one slice.The expert count sets the depth of this class-specific scale ladder.Source: §2 · p. 3
  • Each head trains 4.13M parameters, or 4.4% of ViT-B, with separate optimizers, schedules, and gradient clipping to prevent cross-head interference.The loss combines Dice, focal, and mixture-of-experts load-balancing terms; inclusion-positive images are oversampled fivefold.Source: §2 · p. 3
  • For NPU export, static routing computes every expert before one-hot selection, while a single scaled-dot-product attention call with additive bias reduces live tensors.That attention rewrite agrees with the reference computation to 1.2 × 10^-5; attention activations scale as the fourth power of input side length.Source: §2 · p. 4

3 Experiments

The experiments test expert-count choices, transfer to real NIST scans, and full-resolution NPU deployment. They report 64.2% pore IoU on NIST and identify activation memory as the constraint that limits backbone execution.

3 Experiments

  • Evaluation uses five held-out synthetic splits of 100 images each, real NIST scans, and direct NPU measurements on the Qualcomm Dragonwing IQ-9075.Configurations use a single seed, so differences below roughly three points should be interpreted carefully.Source: §3 · p. 4

3.1 How many mixture-of-experts?

  • The (2, 8) pairing is a deployment choice with summed IoU 85.0; independently selected expert counts did not combine into the best joint configuration.The (8, 8) setting achieved only 25.8 pore IoU, while (8, 1) achieved the best pore IoU; (2, 2) underperformed (2, 8) on both metrics.Source: §3.1 · p. 4

3.2 Out-of-distribution evaluation on real NIST XCT

  • 64.2 pore IoU on NIST is reached by the selected model, which outperforms all three XCT-SAM settings despite using ViT-B rather than ViT-H.The 10.7-point synthetic ablation gap shrinks to 2.1 points on NIST, below the reported noise floor.Source: Table 2 · p. 5, §3.2 · p. 5

3.3 Backbone selection under edge constraints

  • Only ViT-B runs at 1024² on the Hexagon v73; ViT-H and ViT-L fail allocation because activation memory, not weights, exceeds the device ceiling.Quantization does not move the ceiling, while both larger backbones run at 512².Source: §3.3 · p. 5

3.4 Deploying the full DCM-SAM on the NPU

  • All four selected ViT-B graphs execute at 1024² with no CPU fallback after the scaled-dot-product rewrite, differing from FP32 on only 96 pore and 0 inclusion pixels.The comparison covers one real slice per head, with 1,048,576 total pixels.Source: §3.4 · p. 5
  • Static routing makes the M = 8 head 4.1 times slower, while defect-awareness adds an encoder pass and the decoder costs 500 times less than the encoder.The study also reports a resolution-versus-backbone trade-off: reducing resolution costs pores, while dropping the backbone costs inclusions.Source: §3.4 · p. 5

4 Conclusion

DCM-SAM adapts a frozen SAM ViT-B for two defect classes while training only 4.4% of parameters per head. It reports 64.2% pore IoU on real NIST scans and full-resolution NPU execution close to the FP32 reference.

4 Conclusion

  • DCM-SAM adapts a frozen SAM ViT-B to two defect classes while training only 4.4% of parameters per head.The reported experiments use a few hundred synthetic slices.Source: §4 · p. 6
  • On real NIST scans, DCM-SAM reaches 64.2% pore IoU and exceeds XCT-SAM's reported results despite using ViT-B rather than ViT-H.No real images were used during training.Source: §4 · p. 6
  • The complete model runs at full resolution on the IQ-9075 with every operator on the NPU and masks within 0.01% of the FP32 reference.Source: §4 · p. 6

A Training and evaluation details

Evaluation uses held-out synthetic slices and real NIST scans, with fixed per-split post-processing applied identically across models. Training uses separate defect heads and oversampling for inclusion-positive images; half-resolution retraining recovers most, but not all, pore performance.

A Training and evaluation details

  • The synthetic benchmark has five held-out CycleGAN-generated splits of 100 slices each, while the NIST real benchmark covers specimens test2 to test6 with pore labels only.Per-split post-processing was calibrated once and applied identically to every model.Source: A Training and evaluation details · p. 9
  • The test6 synthetic split serves both early stopping and best-checkpoint selection and is also included in the five reported splits.IoU is averaged per image, with an image containing neither defect nor predicted pixels scored as 1.Source: A Training and evaluation details · p. 9
  • Both heads use AdamW with Dice-plus-focal loss, while inclusion-positive images are oversampled five times and inclusion loss is averaged per image.The per-image averaging prevents one densely labelled oversampled slice from dominating a training step.Source: A Training and evaluation details · p. 9
  • At 512², retraining recovers most but not all pore loss: pore performance remains 5.6 points below the 1024² counterpart on the image-averaged metric.The comparison is based on one seed, was not evaluated on NIST, and the adapters were not tested on-device.Source: A Training and evaluation details · p. 9

B Seeding confound

An early global-random-generator seeding scheme let expert-count changes alter the pore head’s data ordering, despite separate heads. The corrected pipeline uses dedicated generators, and every reported result uses that fix.

B Seeding confound

  • Changing the inclusion head’s expert count changed pore-head data ordering, despite the heads sharing no parameters or gradients.Constructing a Conv-LoRA head consumes random draws proportional to its expert count.Source: B Seeding confound · p. 10
  • Two runs with an identical pore head differed by 4.9 IoU points, exceeding the effect being measured.The sampler and worker seeds are now derived from dedicated generators, independent of model construction.Source: B Seeding confound · p. 10
  • Every result in the paper uses the corrected pipeline, because the confound could have inverted several conclusions.The authors report the issue because it was not visible in the loss curves.Source: B Seeding confound · p. 10

C ViT-H expert-count study

On ViT-H, mixture size has little effect once M ≥ 2. The section also reports how trainable parameters are distributed across each defect head.

C ViT-H expert-count study

  • Across Table 4’s ViT-H sweeps, inclusion varies 1.1 points and pore IoU 0.8 points above M = 1; mixture size is near-irrelevant once M ≥ 2.Each sweep row pairs the swept head’s IoU with the fixed companion head’s IoU from the same run.Source: C ViT-H expert-count study · p. 10, Table 4 · p. 10

D Trainable parameter budget

  • On ViT-B, the adapter accounts for 0.08% of the model, while the mask decoder makes up 98% of trained parameters.The adapter cannot be folded into the frozen weights because its expert branch uses convolutions, GELU, and an input-dependent gate.Source: D Trainable parameter budget · p. 10

E Why the adapted encoder did not fit, and what fixed it

The adapted ViT-B encoder exceeded the graph serializer’s allocation ceiling, although the stock encoder ran. Rewriting the adapted attention in SDPA form let the graphs execute while preserving the reference outputs.

E Why the adapted encoder did not fit, and what fixed it

  • The adapted ViT-B encoder required 4.07 GB at 1024², exceeding the serializer’s 3.76 GB limit.The stock encoder ran, but the adapted encoder exported with eager attention was refused during serialization.Source: E Why the adapted encoder did not fit,… · pp. 10–11
  • Table 5 reports trainable parameters per defect head counted from the instantiated model at r = 2 and M = 8; each checkpoint contains exactly these parameters, and both backbones use the same mask decoder.Source: Table 5 · p. 11
  • Only the combination of eager attention and the adapter exceeded the ceiling; rewriting attention in SDPA form let the adapted encoder run.The rewrite produced encoder outputs within 1.2 × 10^-5 of the corresponding ViT-B outputs and masks differing in 0 of 1,048,576 pixels.Source: E Why the adapted encoder did not fit,… · p. 11, Table 6 · p. 11

F NPU graph construction details

NPU graph construction required operator substitutions and a static-gate rewrite, while retargeting SAM to a different input size required three model changes. The reported graph costs use different input resolutions for encoder profiling and decoder timing.

F NPU graph construction details

  • The INT8 scheme quantizes weight-bearing convolutions and matmuls, while keeping normalization, softmax, and elementwise operators in floating point.The tanh GELU approximation has maximum output deviation 1.0 × 10^-3 on ViT-B and 3.5 × 10^-3 on ViT-L.Source: F NPU graph construction details · p. 11
  • The static gate uses an equality between arg max and arange(M) after ScatterElements and OneHot proved unsupported.Serializing the decoder without encoder weights reduced the artifact from 358 MB to 15.7 MB.Source: F NPU graph construction details · p. 11
  • From M = 2 to M = 8, measured slowdown is 4.1×, versus a 40× increase in expert compute, because the rest of the encoder is unchanged.The expert compute grows from 5 to 204 as upsampling increases with expert index.Source: F NPU graph construction details · p. 11
  • Table 7 reports four IQ-9075 FP16 graph costs; encoder profiler timings are at 512², while decoder measurements are at 1024².Peak memory excludes the loaded binary.Source: Table 7 · p. 12
  • Retargeting SAM to input size S′ requires changing the patch-embedding size check, resampling position embeddings, and updating the prompt encoder’s patch-grid size.Without the last change, the decoder receives tensors of the wrong shape.Source: F NPU graph construction details · p. 12

G Limitations and next steps

The reported results have substantial evaluation and deployment caveats. The authors identify protocol-matched baselines, repeated seeds, shared-adapter ablations, and on-board timing as next steps.

G Limitations and next steps

  • A 4.9-point seeding swing leaves differences under about three points unresolved across configurations trained once.The (2, 8) choice was made on the synthetic benchmark; its NIST result was the lowest of nine, but only by 2.1 points, also not read as a real difference.Source: G Limitations and next steps · p. 12
  • The Table 2 comparison is indicative: XCT-SAM baselines were not re-run under this protocol and use ViT-H rather than DCM-SAM’s ViT-B.A protocol-matched re-evaluation remains a next step.Source: G Limitations and next steps · p. 12
  • Deployment evidence is FP16-only, and on-device numerical equivalence was checked on one real XCT slice per head at 1024².INT8 was blocked at the operator level for attention products; 1024² encoder latency was not measured because the profiler could not attach to that graph, so Table 7 reports 512².Source: G Limitations and next steps · p. 12
  • Each head executes its own encoder and carries a copy of the frozen weights, so backbone sharing saves training storage but not device storage.The authors list an internal shared-adapter ablation and on-board timing of 1024² binaries as next steps.Source: G Limitations and next steps · p. 12

H Per-split NIST results

The per-split NIST results show near-perfect recall but lower precision on test2 and test4, where pore-head over-prediction depresses IoU. The table also describes 5 px tolerance metrics and a test6-specific preprocessing calibration.

H Per-split NIST results

  • test6 uses inversion and non-zero valid-region masking, calibrated for its highly porous, inverted-intensity appearance.Table 8 presents per-split NIST results for the selected (2, 8) ViT-B model.Source: Table 8 · p. 12
  • Table 8 reports per-split precision and recall at the 5 px tolerance used for Tol-F1.The supplied passages do not state additional per-split cell values.Source: Table 8 · p. 12
  • 85–89% Tol-F1 accompanies near-perfect recall but lower precision on test2 and test4, where pore-head over-prediction depresses IoU.Table 8 reports precision and recall at the 5 px tolerance used for Tol-F1.Source: H Per-split NIST results · p. 12, Table 8 · p. 12

Related papers

Loading 2609.38811v1…