PaperlayerView exact revision on arXiv

Source-linked AI summary

VL-AcneSeg: A Vision-Language Framework for Region-Aware Acne Lesion Segmentation

Sukju Oh, Soo Ick Cho, Dae Hun Suh, Sukkyu Sun

arXiv:2609.34472v1cs.CVcs.AI

TL;DR

Area-based severity assessment can reflect lesion extent, yet existing vision-language segmentation uses text to identify classes, a weak cue for a single-class task. VL-AcneSeg uses CLIP with facial-region prompts as spatial guidance; across internal and external datasets it ranks highest among compared methods, and its lesion areas track IGA severity as closely as expert annotations.

  • Problem

    Area-based assessment can reflect lesion extent, but existing vision-language segmentation uses text to specify classes, a weak cue when the task has a single lesion class.

    Source: §I · p. 2

  • Method

    VL-AcneSeg aligns CLIP visual features with facial-region text prompts as spatial priors and refines lesion localization into masks.

    Source: §I · p. 2, §V · p. 13

  • Results

    Across internal and external datasets, VL-AcneSeg ranks highest in Dice and IoU among compared methods, while lesion areas track IGA as closely as expert annotations.

    Source: §V · p. 13

  • Takeaways & Limitations

    The lesion-area results support area-based severity assessment as an alternative to lesion counting.

    Source: §V · p. 13

    All internal and external data represent Korean populations, so evidence does not extend to other ethnicities, skin tones, or imaging devices.

    Source: §I · p. 12

Abstract

from arXiv · show

Acne assessment is crucial for clinical decision-making, yet traditional grading and counting are subjective and fail to account for lesion size. While area-based assessment has emerged as a promising alternative, acne segmentation has continued to rely on general-purpose architectures. To address this gap, we propose VL-AcneSeg, a multimodal framework for acne lesion segmentation that leverages CLIP and region-level text prompts to incorporate spatial priors, enabling lesions to be localized across the whole face. Because region-level prompts indicate which facial areas contain lesions, we report a single global prompt, which requires no such information, as our primary setting. On our internal clinical dataset, VL-AcneSeg achieves a Dice score of 0.5082 and an IoU of 0.3407 under this protocol, the highest among all compared methods, including recent vision-language segmentation methods that are themselves given region-level prompts; region-level prompting raises these to 0.5296 and 0.3602. Moreover, lesion area measurements derived from our segmentation correlate with IGA scores at a level comparable to expert annotations (Pearson r = 0.719 versus 0.658). Notably, our framework maintains consistent performance across external validation datasets, performing reliably even on uncontrolled smartphone images without requiring additional training or fine-tuning. By pairing a protocol that requires no lesion-location information with area-based severity estimation, this work provides a foundation for objective acne assessment outside the clinic. Our implementation is publicly available at: https://github.com/sukjuoh/VL-AcneSeg

I. INTRODUCTION

Area-based assessment captures lesion extent, but acne segmentation remains difficult and prior approaches often lack acne-specific design or full-face context. VL-AcneSeg reframes text guidance as a spatial prior for a single lesion class, using facial-region prompts to localize lesions.

I. INTRODUCTION

  • Area-based assessment reflects lesion extent as well as number, while grading is variable and lesion counting weights small and large lesions equally.Automated acne segmentation is also constrained by scarce pixel-level annotations and challenging lesion appearance.Source: §I · pp. 1–2
  • Prior acne segmentation often uses general-purpose architectures and cropped lesion patches, limiting capture of the global distribution needed for area-based assessment.Even full-face models have introduced little methodology tailored to acne.Source: §I · p. 2
  • VL-AcneSeg uses region-level text prompts as spatial priors, specifying where a lesion may appear rather than which class to segment.The approach aligns prompts with visual features to identify candidate sites and refine them into masks.Source: §I · p. 2

A. Acne Lesion Segmentation

  • Pixel-level acne segmentation remains relatively underexplored, with prior work often focused on severity grading or lesion counting rather than whole-face segmentation.This task must distinguish numerous lesions from similar skin features such as scars, PIH, and pores.Source: A. Acne Lesion Segmentation · p. 2

B. Open-Vocabulary and Text-Guided Segmentation

  • Open-vocabulary segmentation uses text to specify classes, but acne presents a single lesion class and lacks the distinct positive-negative pairs needed for contrastive alignment.ROI proposals, subtle lesion features, and random masking also pose practical challenges for existing paradigms.Source: B. Open-Vocabulary and Text-Guided… · p. 2
  • VL-AcneSeg repurposes patch-wise image-text alignment so prompts encode anatomical location, extending text-guided segmentation to dense single-class tasks.Unlike class-level prompts in CAT-Seg, region-level prompts encode location; PBIP's visual prototypes instead encode appearance.Source: B. Open-Vocabulary and Text-Guided… · pp. 2–3

C. Text-Guided Segmentation Model in the Medical Domain

Medical text-guided segmentation has used text to identify target structures or encode location in other ways. VL-AcneSeg uses fixed-template facial-region prompts within CLIP's joint embedding space, and its data preparation assigns lesion masks and prompts to anatomical regions.

C. Text-Guided Segmentation Model in the Medical Domain

  • VL-AcneSeg encodes fixed-template facial-region prompts with CLIP and aligns them to image patches, unlike TVE-Net’s hand-designed mapping from radiology text to probability maps.This avoids requiring narrative reports and keeps the spatial prior in the joint visual-text embedding space.Source: C. Text-Guided Segmentation Model in… · p. 3
  • Text-guided segmentation had not, to the authors’ knowledge, been applied to acne, whose lesions are small, numerous, and distributed across the face.Prior medical applications described here address radiology or anatomical structures, including location-aware approaches such as TVE-Net and STPNet.Source: C. Text-Guided Segmentation Model in… · p. 3

A. Data Preparation

  • The clinical cohort comprised 258 Korean patients and 1,213 images, with masks reviewed and adjudicated by a board-certified dermatologist.The reference standard was fixed by a single senior reader rather than inter-reader agreement.Source: A. Data Preparation · p. 3
  • The segmentation target was inflammatory lesions; excluding patients without them left 222 patients, 821 images, and 4,655 annotated lesion instances.The study targets papules, nodules/cysts, and pustules because severity assessment focuses on inflammatory lesions and comedones are often indistinguishable at pixel level.Source: A. Data Preparation · p. 4
  • MediaPipe landmarks define facial regions, and each lesion mask is assigned by its centroid before region-specific prompts are generated.The fixed prompt template is “acne lesion on the {region}.”Source: Figure 2 · p. 3, A. Data Preparation · p. 4

B. Model Overview

VL-AcneSeg turns image-text alignment into a pixel-wise lesion mask. It processes overlapping high-resolution tiles, stitches their visual features, and compares the resulting map with text embeddings before refinement by the segmentation network.

B. Model Overview

  • The model takes an image and region-level text prompts and outputs a pixel-wise acne lesion mask.Its architecture uses tiled CLIP features, patch-wise text similarity, intermediate vision features, CLS tokens, cross-attention, and a mask decoder.Source: B. Model Overview · p. 4

C. Localization: High-Resolution Features and Patch-wise Similarity

  • Overlapping 336 × 336 tiles address CLIP’s input limit while preserving high-resolution detail needed for small lesions with blurry boundaries.The image is resampled and tiled in a 4 × 3 layout; the next admissible refinement would quadruple CLIP forward passes while each tile covers less anatomical context.Source: C. Localization: High-Resolution… · p. 5
  • Tile embeddings are reassembled into one spatial feature map by averaging overlapping regions with Gaussian weights that downweight tile borders.The stitched map is then upsampled and normalized at each location.Source: C. Localization: High-Resolution… · p. 5
  • Patch-wise cosine similarity between visual features and encoded region prompts yields localization maps aligned to each textual description.Both the visual features and text embeddings are normalized along the channel dimension before similarity is computed.Source: C. Localization: High-Resolution… · p. 5

D. Feature Enhancement

The framework adapts V-V attention to preserve fine-grained lesion features, applying it at the final CLIP feature layer and supplementing it with earlier-layer and tile-derived spatial cues.

D. Feature Enhancement

  • V-V attention is applied only at the final CLIP feature layer, because repeated value averaging can smooth away patch-aligned structure needed for dense lesion prediction.A single application suppresses dominant tokens; repeated applications act as a low-pass filter over the token grid.Source: D. Feature Enhancement · p. 5
  • The 12th CLIP layer feature map is fused before the bottleneck to guide segmentation with fine-grained local information that diminishes with layer depth.Source: D. Feature Enhancement · p. 5
  • Tile-text similarity weights pool tile-level CLS tokens into spatial cues, which are concatenated with the matching text embeddings for cross-attention queries.Source: D. Feature Enhancement · pp. 5–6

E. Refinement and Mask Decoding

The decoder combines similarity maps and CLIP visual features, adds intermediate-layer guidance, and uses global and region-specific supervision to train mask predictions.

E. Refinement and Mask Decoding

  • The decoder processes similarity maps with replicated CLIP features through Swin Transformer blocks, then fuses 12th-layer guidance before the bottleneck.The bottleneck integrates spatial similarity cues with low-level visual features.Source: E. Refinement and Mask Decoding · p. 6
  • The pooled CLS token and text embedding form a query for cross-attention, while the decoder anchors its final mask to the initial localization map.The fused features pass through self-attention and normalization before learned upsampling produces the segmentation mask.Source: E. Refinement and Mask Decoding · p. 6

F. Loss Function

  • Training combines a global segmentation loss with a region-specific loss for each regional prediction.Source: F. Loss Function · p. 6
  • Global supervision uses weighted IoU and weighted BCE losses, while regional focal loss directly supervises the similarity map and emphasizes small or ambiguous lesions.The regional masks correspond to the facial regions specified by the prompts.Source: F. Loss Function · p. 6

IV. EXPERIMENTS

The experiments use an internal patient-wise split and two external test sets representing controlled and real-world imaging. Metrics include pixel-level overlap and lesion-level counts, aggregated over test-set pixels.

A. Setup

  • The internal cohort was split by patient at an 8:1:1 ratio, with 190, 16, and 16 patients in the realized training, validation, and test sets.The internal test set contains 58 images from 16 patients; IGA scores were retrospectively assigned for that test set only and were not used for training or model selection.Source: A. Setup · p. 6
  • External test sets comprise 71 controlled-camera images and 56 real-world smartphone images, sampled after screening and confirmed for inflammatory acne by a dermatologist.The same dermatologist created pixel-level masks for both external sets; the screening model was used only to select candidates.Source: A. Setup · pp. 6–7

B. Evaluation Metrics

  • Evaluation reports Dice, IoU, pixel-wise Precision and Recall, and lesion-level counts per image, with metrics micro-averaged across the test set.Lesion-level recall, precision, and F1 for lesions below 500 px2 and area under the precision–recall curve are reported in the supplement.Source: B. Evaluation Metrics · p. 7

C. Implementation and Training Protocol

VL-AcneSeg achieves the highest IoU and Dice scores across evaluation settings, while retaining stable segmentation on external images, including smartphone images. The study compares models under specified training and input protocols.

C. Implementation and Training Protocol

  • All baselines used original-paper default hyperparameters, the same patient-wise split, no augmentation, and validation-IoU checkpoint selection.The CLIP-based methods were compared at matched resolution and tiling; other models used resized and, when needed, zero-padded inputs.Source: C. Implementation and Training Protocol · p. 7

D. Results

  • The proposed model was not statistically distinguished from SAM in the initial comparison, but three-seed retraining preserved the ordering and showed a larger IoU and Dice for VL-AcneSeg.The reported values were IoU 0.3460 ± 0.0049 versus 0.3219 ± 0.0052 and Dice 0.5141 ± 0.0054 versus 0.4871 ± 0.0065.Source: D. Results · p. 7
  • VL-AcneSeg attains the highest IoU and Dice scores in every evaluation setting.Source: D. Results · p. 7, Table II · p. 8
  • VL-AcneSeg remains stable under external imaging shifts, including unpredictable lighting and varying facial angles in real-world smartphone images.Most baselines degrade on external data, and the gap is more pronounced for vision-language baselines in the smartphone setting.Source: D. Results · p. 8

E. Ablation Study

The ablations show that the full model performs best across datasets, balanced 4×3 patching avoids a precision–recall imbalance, and region prompts provide meaningful spatial guidance. Prompt tests indicate that correct region references matter, omissions are more damaging than extra prompts, and unfamiliar anatomical synonyms largely transfer.

1) Effectiveness of Architectural Components:

  • The Full Model attains the highest IoU and Dice on all three datasets; its CLS token, V-V attention, and region mask loss contribute complementarily.Applying V-V attention across all layers degrades performance on every dataset and increases false-positive lesions per image from 2.48 to 3.57 internally and from 0.82 to 0.93 in the real-world setting.Source: 1) Effectiveness of Architectural… · p. 9, Table III · p. 9

2) Impact of Image Patch Resolution:

  • 4×3 patches provide the most balanced results, while 3×2 patches yield the highest Recall but low Precision from confusion between acne and scars.Finer patch granularity supplies the spatial priors needed to distinguish true lesions from confounding skin features.Source: 2) Impact of Image Patch Resolution · p. 9, Table IV · p. 9

3) Role of the Text Prompt:

  • Internal IoU shifts from 0.3602 to 0.3422 with unseen clinical synonyms, but falls to 0.3039 when images receive another image’s prompts.On the controlled set, shuffled prompts reduce IoU from 0.4232 to 0.2170; the model follows region references, though their value varies by dataset.Source: 3) Role of the Text Prompt · pp. 9–10, Table V · p. 10
  • Without CLIP, internal and real-world IoU fall to 0.1694 and 0.0180; adding the CLIP vision encoder yields 0.2358, and region conditioning raises it to 0.3602 internally.A learnable per-region vector reaches 0.3565 versus 0.3602 for text prompts, while a fixed one-hot index reaches only 0.3111; text also resolves terms outside the fixed region vocabulary.Source: 3) Role of the Text Prompt · pp. 9–10, Table V · p. 10
  • Unseen single-word synonyms such as “frontal region” and “malar area” transfer spatially, but “lateral forehead” yields a diffuse map.The tested synonyms were absent from training; the compositional spatial modifier was not fully composed with the anatomical noun.Source: 3) Role of the Text Prompt · p. 10, Figure 5 · p. 10
  • At 40% omission perturbation, IoU falls from 0.3602 to 0.3100 and drops below the 0.3407 global-prompt baseline at roughly 15%; over-prompting stays above that baseline.At the same perturbation level, over-prompting declines only to 0.3438; omitted regions cannot be recovered downstream, whereas spurious candidates can still be rejected during refinement.Source: 3) Role of the Text Prompt · pp. 10–11, Figure 6 · p. 10

F. Failure Case Analysis

Most remaining segmentation errors arise from confusion between active lesions and visually similar residual marks, rather than gross localization failures. On the internal sample, lesion-area ratios track IGA severity comparably to expert annotations, while precise per-lesion delineation requires higher accuracy.

F. Failure Case Analysis

  • Residual errors mainly confuse active lesions with post-inflammatory hyperpigmentation, scars, and other red marks; small or subtle lesions are also missed.The shared color, shape, and texture of active lesions and their residue underlie both error modes, which account for most remaining error rather than gross localization failures.Source: F. Failure Case Analysis · p. 11, Figure 8 · p. 11

G. Clinical Implementation

  • For 22 visits, predicted lesion-area ratios correlate with IGA at Pearson 0.719 versus 0.658 for expert annotations, with neither difference significant.Spearman correlations are 0.684 versus 0.710, respectively; the comparison reverses direction and is not significant.Source: G. Clinical Implementation · p. 11, Table VI · p. 11
  • The global prompt correlates with IGA as well as the region-level prompt, while region-level prompting provides a separate estimate for each facial area.A high correlation alone does not ensure accurate area estimates: U-Net’s inflated areas still scale with lesion burden, whereas VL-AcneSeg balances precision and recall without systematic offset.Source: G. Clinical Implementation · pp. 11–12
  • Area ratios vary with both lesion extent and number, unlike counts that treat lesions as equal units even when they merge or shrink during treatment.This continuous response is the property area-based assessment relies on.Source: G. Clinical Implementation · p. 12
  • A Dice score near 0.53 supports aggregate-area tracking comparable to expert annotations, but applications requiring precise per-lesion delineation need higher accuracy.In a reader study, eight dermatologists and residents reached a median lesion-detection F1 of 0.31 for inflammatory lesions from images alone.Source: G. Clinical Implementation · p. 12

H. Deployment Considerations

VL-AcneSeg uses a global prompt as its primary setting because region prompts require lesion-area knowledge, while its conclusion reports strong comparisons across internal and external datasets. The paper also identifies deployment costs and scope boundaries.

H. Deployment Considerations

  • Global prompting requires no region information, whereas region-level prompting requires a clinician or patient to identify lesion-bearing facial areas.Omitting affected regions degrades results below the global-prompt baseline at roughly 15% omission; over-inclusive prompts remain above it across the tested range.Source: H. Deployment Considerations · p. 12
  • Global-prompt inference takes 858 ms per image, rising to at most 1.54 s with all fifteen regions; tiled encoding is shared across prompts.Each additional region prompt adds 18.5 GFLOPs, or about 48 ms; tiled encoding accounts for roughly 83% of computation.Source: H. Deployment Considerations · p. 12, Table VII · p. 12

I. Limitations and Future Directions

  • The evidence comes from Korean populations and does not establish performance across other ethnicities, skin tones, or imaging devices.Phototype was not recorded, so performance could not be stratified by skin type; evaluation on multiethnic, multi-site data remains a priority.Source: §I · p. 12
  • Dice and IoU remain moderate relative to other medical segmentation tasks, and IGA correlation alone does not establish readiness for clinical use.The authors say prospective study against expert assessment is needed for clinical readiness.Source: §I · p. 12
  • Inter-annotator agreement was not quantified because a single board-certified dermatologist fixed the final labels through adjudication.The drafts preceding adjudication were not retained as separate outputs.Source: §I · p. 12
  • Future directions include frequency-domain features for separating active lesions from residue, multi-class segmentation, and prompts describing lesion type, extent, or clinician phrasing.The authors frame the spatial-prompt formulation as potentially applicable wherever small, numerous lesions occur across stable, nameable anatomy.Source: §I · p. 13

V. CONCLUSION

  • Across internal and external datasets, VL-AcneSeg attains the highest Dice and IoU among compared methods, including vision-language baselines given region-level prompts.Under domain shift, general-purpose architectures transfer less well; lesion-area measurements track IGA severity as closely as expert annotations.Source: §V · p. 13

A. Internal cohort

The study describes its internal cohort and IGA grading, then details external-set construction and imaging conditions. Supplementary analyses examine prompt content, patch granularity, and localization on smartphone images.

A. Internal cohort

  • The internal cohort comprised 258 patients photographed under standardized conditions at Seoul National University Hospital between February and July 2020.The cohort was Korean; Fitzpatrick phototype was not routinely recorded.Source: A. Internal cohort · p. 15

B. IGA grading protocol

  • Three board-certified dermatologists independently assigned visit-level IGA grades, resolving three-way disagreements through discussion until consensus.The internal test set included 22 visits, graded IGA 1–4; none received IGA 0.Source: B. IGA grading protocol · p. 15

C. External subset construction

  • External subsets comprised 71 controlled-camera and 56 smartphone images confirmed to contain inflammatory acne after dermatologist review.Images were sampled from an AI Hub collection using a permissive detection-screening threshold; residual sampling bias may remain for lesions missed by that screening model.Source: C. External subset construction · p. 15

D. External imaging conditions and metadata

  • The controlled external subset mainly probes population and acquisition-site shift, while smartphone images add variability in resolution, field of view, and color processing.The external collection provides neither participant demographics nor Fitzpatrick phototypes, and individual camera models are unspecified.Source: D. External imaging conditions and… · p. 15

S2. TEXT PROMPT DETAILS

  • Prompt ablations compared spatial-only Region prompts with Level prompts adding severity and Num prompts adding exact regional lesion counts.Adding count descriptors raises recall but lowers precision; the table caption identifies the ablation scope without giving values.Source: S2. TEXT PROMPT DETAILS · p. 15, Table S2 · p. 15

S3. LOCALIZATION UNDER DIFFERENT PATCH RESOLUTIONS

S4. QUALITATIVE ANALYSIS OF LOCALIZATION IN SMARTPHONE IMAGE

  • On a smartphone image, regional prompting concentrates localization on lesion sites, while global prompting is broader but still recovers them.The broader global-prompt activation occasionally extends beyond the facial boundary, yet remains usable when region information is unavailable.Source: S4. QUALITATIVE ANALYSIS OF… · p. 16, Figure S2 · p. 16

S5. PATCH RESOLUTION ON EXTERNAL DATASETS

The supplementary analyses report that finer tiling benefits external datasets, clarify paired comparisons, and assess small-lesion performance and threshold independence. Additional figures extend qualitative comparisons across the internal and external datasets.

S5. PATCH RESOLUTION ON EXTERNAL DATASETS

S6. CONFIDENCE INTERVALS FOR BETWEEN-METHOD

  • Paired patient-level resampling preserves the correlation between methods; all between-method differences are significant at 0.05 except SAM.Intervals are narrower for comparisons against conventional architectures than for closer vision-language baselines.Source: S6. CONFIDENCE INTERVALS FOR… · p. 16

S7. ADDITIONAL EVALUATION METRICS

  • Both VL-AcneSeg configurations rank in the upper range for small-lesion precision and recall, lead on F1, and exceed vision-language baselines on F1 even with regional prompts.Both configurations also remain in the upper AUPRC range, so the reported advantage does not depend on the fixed 0.5 binarization threshold; the top group is tightly packed.Source: S7. ADDITIONAL EVALUATION METRICS · pp. 16–17, Table S5 · p. 17

S8. COMPREHENSIVE QUALITATIVE COMPARISON OF ACNE SEGMENTATION

Related papers

Loading 2609.34472v1…