Source-linked AI summary
SkinFlow: Efficient Information Transmission for Open Dermatological Diagnosis via Dynamic Visual Encoding and Staged RL
Lijun Liu, Linwei Chen, Zhishou Zhang, Meng Tian, Hengfu Cui, Ruiyang Li, Zhaocheng Liu, Qiang Ju, Qianxi Li, Hong-Yu Zhou
TL;DR
Dermatological LVLMs face diffuse attention and evaluation metrics that poorly reflect clinical relevance. SkinFlow addresses these issues through information-efficient visual encoding, staged reinforcement learning, and hierarchy-aware evaluation, with its 7B model achieving strong Fitzpatrick17k results against much larger models. The paper concludes that improving visual information transmission can outperform raw parameter scaling for this task.
Problem
General-purpose LVLMs struggle to distinguish subtle dermatological lesions from background noise, while binary metrics do not reflect hierarchical clinical relevance.
Method
SkinFlow combines Dynamic Visual Encoding with two-stage reinforcement learning that aligns explicit signs before reconstructing implicit pathological textures.
Results
+12.06% Top-1 accuracy and +28.57% Top-6 accuracy over Qwen3VL-235B were achieved on Fitzpatrick17k by the 7B model.
Takeaways & Limitations
A domain-specialized 7B model can surpass much larger models through improved visual-semantic alignment and clinically grounded evaluation.
Takeaways & Limitations
Interpretability was not further evaluated, and the model may degrade on images with complex or cluttered backgrounds.
Abstract
from arXiv · showhide
General-purpose Large Vision-Language Models (LVLMs), despite their massive scale, often falter in dermatology due to "diffuse attention" - the inability to disentangle subtle pathological lesions from background noise. In this paper, we challenge the assumption that parameter scaling is the only path to medical precision. We introduce SkinFlow, a framework that treats diagnosis as an optimization of visual information transmission efficiency. Our approach utilizes a Virtual-Width Dynamic Vision Encoder (DVE) to "unfold" complex pathological manifolds without physical parameter expansion, coupled with a two-stage Reinforcement Learning strategy. This strategy sequentially aligns explicit medical descriptions (Stage I) and reconstructs implicit diagnostic textures (Stage II) within a constrained semantic space. Furthermore, we propose a clinically grounded evaluation protocol that prioritizes diagnostic safety and hierarchical relevance over rigid label matching. Empirical results are compelling: our 7B model establishes a new state-of-the-art on the Fitzpatrick17k benchmark, achieving a +12.06% gain in Top-1 accuracy and a +28.57% boost in Top-6 accuracy over the massive general-purpose models (e.g., Qwen3VL-235B and GPT-5.2). These findings demonstrate that optimizing geometric capacity and information flow yields superior diagnostic reasoning compared to raw parameter scaling.
1 Introduction
SkinFlow frames dermatological diagnosis as efficient visual information transmission, addressing diffuse attention and clinically misaligned evaluation. It combines dynamic visual encoding, staged reinforcement learning, and hierarchy-aware assessment to improve diagnostic precision.
- Motivation: General-purpose LVLMs struggle to separate subtle lesions from background noise, while binary metrics treat clinically related and distant misclassifications alike.These limitations motivate both improved visual information flow and clinically grounded evaluation.
- Information transmission: SkinFlow models diagnosis as compression and decoding, with performance bounded by how efficiently visual information reaches a constrained diagnostic semantic space.The framework identifies recoverable information about subtle, non-describable pathological cues as a bottleneck.
- Dynamic visual encoding: Dynamic Visual Encoding adaptively unfolds pathological manifolds, suppresses background redundancy, and amplifies lesion signal-to-noise ratio without relying on physical parameter expansion.The resulting design supports a shift from uncertain global scanning toward high-confidence focal reasoning.
- Staged reinforcement learning: The two-stage reinforcement learning strategy first aligns explicit medical descriptions, then reconstructs implicit pathological textures within a diagnosis-specific output space.Stage I preserves describable clinical signs, while Stage II bridges visual perception and clinical deduction.
- Clinical evaluation: The clinically grounded evaluation protocol uses disease hierarchies to reward therapeutically consistent near-misses and penalize errors crossing critical boundaries such as malignancy.It prioritizes diagnostic safety and clinical actionability beyond rigid label matching.
- Results: +12.06% Top-1 accuracy and +28.57% Top-6 accuracy over Qwen3VL-235B were achieved by the 7B model on Fitzpatrick17k, while also surpassing GPT-5.2.The reported results establish a new state-of-the-art and support geometric efficiency over raw parameter scale.
2 Related Work
Prior dermatology systems progressed from specialized visual models toward multimodal foundation models, while domain-specific integration and robust post-training remain open challenges.
- Earlier visual approaches: Early automated skin-disease studies mainly used professional dermoscopic images with convolutional or ensemble models for tasks such as melanoma recognition.These approaches centered on specialized visual acquisition and processing.
- Earlier visual approaches: Traditional deep-learning approaches focused on lesion classification and segmentation, including U-Net variants for delineating lesion boundaries.Capsule networks were also explored for spatial hierarchical representation.
- Multimodal models: Multimodal systems increasingly combine visual and textual inputs to produce natural-language medical analyses, including fine-grained visual-language fusion and efficient medical variants.The cited examples include ultrasound question answering and medical multimodal models.
- Open challenges: General-purpose foundation models and domain-specific medical multimodal models have advanced medical image interpretation, but domain-specific semantic integration and diagnostic reasoning remain open problems.The passage explicitly frames effective medical adaptation as unresolved.
- Post-training: Supervised fine-tuning may overfit narrow data distributions, whereas reinforcement learning explicitly optimizes reward-based policies for adaptive improvement, robustness, and generalization.The passage notes that high-dimensional medical RL still faces challenges.
3 Method
SkinFlow frames diagnosis as information transmission: the model compresses explicit clinical signs, integrates implicit pathological textures, and uses dynamic visual encoding to expand geometric capacity efficiently.
- 3.1 Overall Framework: The training pipeline treats diagnosis as image compression and decoding, separating total visual information into describable features I_d and non-describable features I_n.
- 3.1 Overall Framework: Stage I uses medical captioning to compress explicit clinical signs into linguistically interpretable representations and establish a high-capacity diagnostic channel.
- 3.1 Overall Framework: Stage II integrates implicit pathological textures with aligned representations and decodes them into diagnosis-specific semantics.
- 3.2 Dynamic Visual Encoding: FDLinear replaces static vision-backbone linear layers with frequency-disjoint spectral bases and context-aware coefficients, enabling virtual rather than physical dimension expansion.
- 3.2 Dynamic Visual Encoding: The virtual mechanism expands channel capacity from d to K × d conceptually while fusing aggregation before multiplication, preserving standard-like computation and keeping parameter overhead below 5%.
3.3 Preliminary: RL algorithm
The paper adopts GRPO as an outcome-driven reinforcement-learning backbone that compares groups of sampled outputs, normalizes their rewards, and optimizes a clipped objective with KL regularization.
- GRPO eliminates the need for a separate critic by sampling groups of candidate outputs and evaluating them with task-specific rewards.
- Group normalization computes relative advantages within each sampled output group, stabilizing training across diverse reward signals.
- The GRPO objective combines a clipped importance-weighted surrogate with a KL penalty relative to a reference policy.
3.4 Stage I: Learning Dermatological Image Descriptions
Stage I trains the model to generate clinically coherent dermatological descriptions using annotated images and reinforcement-learning rewards. The reward aggregates attribute-level quality and is validated against diagnostic accuracy.
- Data and annotation: Approximately 5,000 dermatological images were collected with disease-category labels for description learning.
- Data and annotation: Captions combine LLM generation with expert refinement, including structured machine annotation and iterative human revision for diagnostic consistency.
- Reward function design: The reward mechanism scores attributes individually and combines their scores using corresponding attribute weights.Attribute weights reflect how frequently each attribute is referenced as diagnostic evidence.
- Reward function design: Caption reward reliability is evaluated by correlating generated-caption rewards with diagnostic accuracy when captions support inference.Table 1 reports average rewards and diagnostic accuracies under the optimal scoring configuration.
3.5 Stage II: Dermatological Diagnosis Training
Stage II trains the model to rank the top-K diagnoses using reinforcement learning. Its reward accommodates synonymous terminology and prioritizes placing the correct diagnosis as high as possible.
- Stage II: Dermatological Diagnosis Training: Stage II predicts the top-K most probable diagnoses for each dermatological image under reinforcement-learning training.
- Stage II: Dermatological Diagnosis Training: Reinforcement learning handles equivalent disease names without requiring rigid token-level matching.
- Stage II: Dermatological Diagnosis Training: Reinforcement learning assigns rewards flexibly for top-K learning without enumerating every possible label combination.
- Stage II: Dermatological Diagnosis Training: The reward equals the positional weight where the correct diagnosis first appears, encouraging both correctness and higher ranking.
4 Experiments
SkinFlow is evaluated on diverse, open-world dermatological benchmarks using clinically grounded criteria and comparisons with general-purpose and medical multimodal models. Ablations and attention analyses examine the contributions of Stage 1 caption training and Dynamic Visual Encoding.
- Evaluation Setup: The evaluation covers approximately 200 disease categories across Fitzpatrick17k and an internally curated benchmark.Fitzpatrick17k contributes 1,000 sampled images, while the internal benchmark contains approximately 200 dermatologist-reviewed images.
- Evaluation Setup: Predictions are judged by hierarchical clinical relevance, treatment consistency, and diagnostic safety rather than exact label matching alone.The protocol rewards valid subclasses and clinically useful parent classes while penalizing sibling-class confusion and safety-critical errors.
- Main Results: 29.19% Top-1 and 71.16% Top-6 accuracy on Fitzpatrick17k establish SkinFlow’s reported state-of-the-art performance, exceeding Qwen3VL-235B by 12.06% and 28.57%, respectively.The model uses 7B parameters and also surpasses GPT-5.2 in the reported comparison.
- Ablation Studies: Stage 1 caption training raises Top-1 accuracy from 27.46% to 35.64% on Self-owned and from 15.22% to 24.45% on Fitzpatrick17k.The Stage 2 model also shows faster convergence and higher validation rewards than the base model.
- Ablation Studies: Adding DVE increases Fitzpatrick17k Top-1 accuracy from 24.45% to 29.19% and Top-6 accuracy from 57.69% to 71.16%.On Self-owned, the full model reaches 36.63% Top-1 and 79.21% Top-6 accuracy.
- Attention Analysis: Attention analyses show SkinFlow concentrating on lesions while reducing background-focused attention and increasing weights above 0.06.Across 500 test images, Stage 1 reduces low-weight attention and DVE produces the lowest noise floor and highest high-confidence peak.
5 Limitations
The study identifies limitations in interpretability evaluation and image-background diversity. These constraints bound how confidently the reported reasoning behavior and deployment robustness can be generalized.
- Interpretability: Model interpretability was not further evaluated, and shorter diagnostic captions after Stage II may affect interpretation of its reasoning process.The authors propose future dermatologist-designed interpretability metrics.
- Generalization: All study images used relatively simple backgrounds, so performance may degrade in complex or cluttered real-world imaging environments.The authors plan to expand the dataset to more diverse imaging conditions.
6 Conclusion
The paper frames dermatological diagnosis as efficient visual information transmission and combines Dynamic Visual Encoding with two-stage RL to separate critical pathology from background noise. Experiments support improved visual-semantic alignment, localized reasoning, and clinically actionable evaluation, while broader specialty generalization remains future work.
- The framework treats diagnosis as image compression–decoding, identifying inefficient visual information transmission as a bottleneck in general-purpose LVLMs.
- A two-stage RL strategy and Dynamic Visual Encoding disentangle critical pathological features from clinical background noise.
- Experiments show a domain-specialized 7B model can surpass models over 30× larger through enhanced visual-semantic alignment.
- The two-stage paradigm uses explicit linguistic guidance and DVE as foundations for reconstructing implicit diagnostic cues, with attention shifting toward localized reasoning.
- The clinically grounded evaluation framework connects algorithmic accuracy with real-world clinical actionability.
- Future work will test whether the compression-decoding framework generalizes to visually intensive specialties such as pathology and radiology.
A Predefined Caption Schema
The predefined caption schema organizes skin-abnormality descriptions into structured fields for color, location, shape, lesion type, surrounding characteristics, and other appearance descriptors.
- The schema records detailed color-related information for the skin abnormality when present.
- It captures all sites showing the skin abnormality in the location field.
- The shape field stores information about the described lesion shape.
- The lesion_type field describes whether lesions are isolated, dense, or confluent.
- Surrounding characteristics and other appearance descriptors capture features around abnormalities and details outside the defined fields.
- Fields are empty when no corresponding information is present, and the other field is used only when absolutely necessary.
B Disease Diagnosis Prompt
The disease-diagnosis prompt asks a dermatologist to rank the most likely skin diseases from photographs and explain the reasons for the predictions.
- The prompt requests the most likely set of skin diseases as a Top-k prediction based on provided skin-disease photographs.
- The response format separates the final answer using answer tags after a thinking-process block.
- The requested output includes each disease name and a probability between 0 and 1.
- For each disease, the output identifies matching fields and provides a one-sentence brief reason.
C Examples of disease diagnosis and examples of captions
The section is identified as presenting examples of disease diagnoses and captions.
- Figure 7 presents examples of disease diagnosis and examples of captions.