SkinAgent AI: A Safety-Grounded Multimodal Agentic Framework for Non-Diagnostic Skincare Support

Introduction

• Consumer-facing skincare systems must provide structured support without presenting visual outputs as diagnosis or treatment decisions, amid concerns about intended use, data quality, privacy, and safeguards.The broader system must define what evidence it may use, what actions it may perform, and how it communicates uncertainty and risk.Source: Introduction · p. 1
• Existing work addresses visual skin analysis and acne detection, counting, segmentation, and severity grading, while the system-level integration of visual evidence, bounded planning, grounded product support, safeguards, and replayable traces remains insufficiently shown.Consumer-facing systems must also define permitted evidence and actions and communicate uncertainty and risk.Source: Introduction · pp. 1–2
• SkinAgent AI combines non-diagnostic routing for acne, pores, and wrinkles with skin-type estimation and count-informed ordinal acne-severity support when acne analysis is appropriate.The system treats visual analysis as structured evidence within an orchestrated workflow rather than as the final output.Source: Introduction · p. 2
• Deterministic safety, privacy, evidence-completeness, and approval checks constrain the LLM, while structured traces record tool calls and verification outcomes for inspection and replay.Catalog retrieval and validation restrict product statements to database evidence, and state-changing actions require approval.Source: Introduction · p. 2
• The framework organizes planning, execution, grounding, verification, finalization, and tracing within explicit tool, evidence, and verification boundaries.Its contribution is the integration and control strategy, not a claim that each component is individually novel.Source: Introduction · p. 2

• Structured traces and replay record routes, tool use, verification outcomes, and response finalization, extending assessment beyond the final answer’s surface quality.Source: §3 · p. 2
• The evaluation separates visual-model performance from agent behavior and examines reliability, tool use, grounding, safety, privacy, failure handling, and process reproducibility.Source: §3 · p. 2
• The contribution lies in the evaluated integration and control strategy, not in claims that each vision model, retrieval component, or software module is individually novel.Source: §3 · p. 2

Visual AI for Skin and Acne Analysis

• Skin and acne studies span photographic skin-type classification, concern localization, lesion-level evidence, and image-level severity grading, while image-derived categories remain limited by acquisition and annotation conditions.Evaluation concerns include skin-tone and lighting variation, limited labels, duplicate and source control, calibration, robustness, and independent or source-held-out data.Source: Visual AI for Skin and Acne Analysis · pp. 2–3

Multimodal and Foundation-Model Approaches in Dermatology

• Multimodal dermatology models connect images with conversation, but reliability varies across demographic groups, and diagnostic systems differ from non-diagnostic consumer support in permissible claims and controls.Source: Multimodal and Foundation-Model… · p. 3

LLM Agents, Tool Use, and Grounded Execution in Healthcare

• Agent frameworks organize planning and action, while grounded execution requires evaluating tool choice, arguments, order, termination, and failure handling rather than final-answer quality alone.Machine-readable, task-specific tools produced the largest error reductions in the cited clinical-calculation experiments.Source: LLM Agents, Tool Use, and Grounded… · p. 3

Safety, Privacy, Explainability, and Human Oversight

• Responsible health-adjacent systems require system-specific privacy, fairness, and safety tests, with independent expert or user assessment strengthening claims about safety and usefulness.Aggregate accuracy or reduced image storage alone does not establish privacy or fairness guarantees.Source: Safety, Privacy, Explainability, and… · p. 4

Synthesis and Research Gap

• The remaining gap is an auditable, non-diagnostic skincare framework integrating visual routing, bounded planning, grounded product evidence, safeguards, approval-controlled actions, and replayable traces under unified evaluation.The research gap concerns integration and evaluation rather than individual component novelty.Source: Synthesis and Research Gap · p. 4

Study Design, Intended Use, and Claim Boundary

• SkinAgent is a research prototype for education, visible-concern summaries, evidence-bounded product assistance, and low-risk actions, not diagnosis, medication prescription, treatment planning, or clinician replacement.Visual outputs are model-estimated attributes or support signals, not clinical findings.Source: Study Design, Intended Use, and Claim… · p. 4
• Evaluation separates visual-module performance from orchestration and controls, so image-classification accuracy is not treated as agent reliability or clinical validation.System-level assessment covers intent routing, tool use, grounding, safety and privacy checks, fallback, and trace replay.Source: Study Design, Intended Use, and Claim… · p. 4

SkinAgent System Architecture and Execution Lifecycle

• The Plan–Execute–Verify workflow checks requests, routes intents, and uses approved tools or an LLM planner, followed by evidence assembly, deterministic verification, and conservative fallback when evidence is missing or claims exceed bounds.The LLM handles interpretation and explanation; deterministic components enforce schemas, permissions, privacy, evidence, wording, and approval requirements.Source: SkinAgent System Architecture and… · p. 5

Visual Input Validation and Image-Handling Boundary

• Image-quality checks assign PASS, WARNING, or FAIL; FAIL blocks analysis, while session-limited handling designates raw images and related local data as non-persistent.Unavailable visual evidence must be described as uncertain or unavailable rather than fabricated.Source: Visual Input Validation and… · p. 5

Skin-Type Estimation

• The skin-type module estimates Combination, Dry, Normal, or Oily from a 14,181-image dataset and supplies personalization context, not a physiological measurement or diagnosis.The study release comprises 12,408 training, 1,181 validation, and 592 test images.Source: Skin-Type Estimation · pp. 5–6

Skin-Condition Classification and Routing

• The Acne, Pores, and Wrinkles classifier uses a group-safe, duplicate-controlled split, but the unreleased canonical manifest limits independent reconstruction of exact test membership.Evaluation includes calibration and class-specific acne-miss and non-acne false-routing measures; the documented materials introduce no fixed acne-routing probability threshold.Source: Skin-Condition Classification and… · pp. 6–7

Count-Informed Ordinal Acne-Severity Support

• For acne-routed inputs, CIOS combines image-level severity estimation with lesion counts to produce one of four ordinal support categories.Reference categories use lesion-count thresholds, not independently assigned dermatologist severity grades; CIOS is bounded visual evidence, not a diagnosis.Source: Count-Informed Ordinal Acne-Severity… · p. 7

LLM Orchestration, Tool Use, and Database Grounding

• A primary LLM orchestrator uses bounded plans and typed tools, while product recommendations and comparisons rely on retrievable catalog records and stored fields.The archived materials omit the hosted LLM identifier, version or alias, temperature, and top-p setting, limiting exact replay of LLM-mediated outputs.Source: LLM Orchestration, Tool Use, and… · p. 7
• Cart changes require explicit approval after pending actions are created, while unavailable visual or retrieval evidence triggers an uncertainty-aware fallback.A general request to add recommended products does not immediately modify the cart.Source: LLM Orchestration, Tool Use, and… · p. 7

Safety, Privacy, Evidence, and Approval Controls

• Safety and privacy controls check evidence, language, permissions, and user scope throughout execution, but do not formally guarantee clinical safety or privacy.High-risk or uncertain states can suppress product guidance or severity claims, request a replacement image, or shift toward consultation-first language.Source: Safety, Privacy, Evidence, and… · pp. 7–8

Agent Benchmark, Baselines, and Ablation Design

• The agent benchmark covers routing, tool use, completion, safety, privacy, grounding, fallback, escalation, and trace behavior across prespecified cases.The internal rubric also covers twelve communication dimensions, including uncertainty, product grounding, and privacy safety.Source: Agent Benchmark, Baselines, and… · p. 8
• The reported architectural comparison is limited to minimally controlled, database-grounded configuration C versus complete SkinAgent configuration D.Configurations A and B were retained as prospective baselines but did not yield manuscript-ready quantitative comparisons; the 240-case benchmark was frozen but non-independent.Source: Agent Benchmark, Baselines, and… · pp. 8–9

Robustness, Failure-Mode, and Human/Expert Evaluation

• Scripted tests covered adversarial inputs, privacy-leakage cases, cross-modal disagreement, image warnings, fallbacks, and recovery scenarios.Specified failure behavior preserves unavailable or uncertain states rather than fabricating visual explanations or product evidence.Source: Robustness, Failure-Mode, and… · pp. 9–10
• No completed controlled robustness experiments, independent dermatologist or domain-expert evaluation, or latency measurements are documented.The absent robustness experiments include blur, brightness, compression, occlusion, and source-held-out visual generalization.Source: Robustness, Failure-Mode, and… · p. 10

Statistical Analysis and Reproducibility

• Three-seed skin-condition summaries are descriptive means and sample standard deviations, while paired C-versus-D binary outcomes use exact two-sided McNemar tests.The protocols also specify confidence intervals for proportion-based outcomes and a bootstrap interval for skin-type macro-F1.Source: Statistical Analysis and Reproducibility · p. 10
• Privacy-filtered traces retain routing, tool, verification, safety, evidence, and finalization records while excluding raw facial images and hidden reasoning.Exact bitwise reproduction of hosted-LLM outputs is not expected because provider-side models and APIs may change.Source: Statistical Analysis and Reproducibility · p. 10

Ethical Considerations

• Visual experiments used pre-existing public datasets, with no human participants, clinical interventions, or new identifiable participant data collected.Formal institutional ethics review and additional informed consent were not sought for the secondary-data and scripted-evaluation study.Source: Ethical Considerations · p. 10

Skin-Condition Routing Performance

• Across three seeds, routing accuracy was 99.84%±0.07%; mean acne miss and non-acne false-routing rates were 0.23%±0.20% and 0.13%±0.11%, respectively.These results describe the fixed internal test set; no independent external condition evaluation was included.Source: Skin-Condition Routing Performance · p. 10, Table 1 · p. 11

Supporting Visual Subsystems

• Skin-type accuracy was 88.85% on the frozen 592-image test; CIOS remains aggregate internal support evidence without independent recomputation or external validation.The skin-type output is a visual category, not a clinical or physiological diagnosis.Source: Supporting Visual Subsystems · p. 11

Agentic Orchestration and Tool-Use Performance

• On the locked 240-case benchmark, intent accuracy was 80.00%, exact tool-set match 62.92%, and strict task completion 47.08%, distinguishing recognition from execution.Eight forbidden-tool violations occurred (3.33%); safety, privacy, and grounding violations were each 0/240, which does not establish universal performance.Source: Agentic Orchestration and Tool-Use… · p. 11, Table 3 · p. 12

Architecture Baseline and Ablation Results

• Configuration C passed 50/60 strict cases (83.33%) versus D's 17/60 (28.33%), while D succeeded on 14/16 tool-required cases (87.50%) versus C's 6/16 (37.50%).The direction varied by endpoint; the 60-case comparison was exploratory, based on post-hoc adjudication, and not an independent clinical or user evaluation.Source: Architecture Baseline and Ablation… · pp. 12–13

Safety, Privacy, Grounding, and Failure-Mode Results

• Audits found no unsafe responses in 45 probes, no cross-user leakage in 15 probes, and 26/26 valid catalog mappings, but attribute grounding and fallback remained incomplete.The predefined fallback occurred in 8/11 failure scenarios; finite zero-event tests do not establish universal safety or formal privacy guarantees.Source: Safety, Privacy, Grounding, and… · p. 13

External, Robustness, and Human/Expert Evaluation

• No independent external condition evaluation, completed robustness study, or human or domain-expert evaluation was available, so external generalization and clinical validation remain unestablished.The internal condition split and controlled system benchmarks are not presented as external or clinical validation.Source: External, Robustness, and Human/Expert… · p. 13

Principal Findings

• Across three seeds, the skin-condition model showed low acne-miss and false-routing rates, supporting consistent routing on the fixed, group-safe internal split; skin-type and CIOS accuracy were 88.85% and 84.59%.CIOS also achieved a quadratic weighted kappa of 0.9076; these outputs are personalization and routing signals rather than clinical phenotypes or diagnostic endpoints.Source: Principal Findings · p. 13
• 80.00% intent accuracy contrasted with 62.92% exact tool-set match and 47.08% strict task completion on the locked 240-case benchmark; eight forbidden-tool events were observed.The development suite's micro-averaged required-tool recall was 55.88%.Source: Principal Findings · pp. 13–14
• Configuration C had higher strict completion and more correct no-tool decisions, while D performed better on tool-required cases; the post-hoc comparison is exploratory, not causal.Configuration D also produced one generic response in 39 of 60 cases.Source: Figure 4 · p. 14, Principal Findings · p. 14
• 100% mode-adjusted evidence completeness and trace-replay concordance in the development suite showed that execution states could be exposed and reproduced even when tool sequences were incorrect.The discussion therefore separates auditability from task competence.Source: Principal Findings · p. 14

Interpretation in Relation to Prior Work

• Unlike task-specific visual prediction systems, SkinAgent uses predictions as bounded evidence for routing and actions; its internal routing result is not directly comparable across differing datasets and protocols.The discussion characterizes its agentic contribution as domain-specific orchestration rather than new language-model reasoning.Source: Interpretation in Relation to Prior Work · p. 15

Implications for Safe Agentic Skincare Support

• Reversible tasks such as catalog retrieval and routine organization are better suited to automation than diagnosis or medication selection, with approval retained for state-changing actions.The discussion recommends validating each product claim against available catalog fields and explicitly representing unavailable evidence or services.Source: Implications for Safe Agentic Skincare… · p. 15

Limitations

• The prototype was not evaluated as a medical device or diagnostic system, and independent external visual validation, expert assessment, robustness testing, and prospective user evaluation remain absent.Performance across skin tones and demographic groups was not established because suitable subgroup labels were unavailable.Source: Limitations · p. 16
• The locked benchmark was internal, while safety and privacy findings cover only tested cases; attribute-level grounding and timeout fallback also remained incomplete.All audited recommendations mapped to catalog entities, but ingredient and rating fidelity were incomplete, and expected fallback occurred in only eight of 11 injected scenarios.Source: Limitations · p. 16

Conclusions

• The framework integrates visual models, an LLM orchestrator, typed tools, database-grounded functions, deterministic checks, approval gates, and traceable execution records.The visual outputs are not interpreted as clinical diagnoses.Source: Conclusions · pp. 16–17
• 99.84% ± 0.07% three-seed routing accuracy accompanied visual-model performance, while system-level results showed lower reliability in tool selection and task completion.The routing task was internal and group-safe; the system benchmark was locked but non-independent.Source: Conclusions · pp. 16–17
• Across finite tests, controls supported auditing, but safe fallback failed in several injected scenarios and catalog-attribute support remained uneven.The system-level benchmark reported 62.92% exact tool-set match and 47.08% strict task completion.Source: Conclusions · p. 17
• The findings support feasibility for non-diagnostic assistance, not clinical readiness, external generalization, formal privacy guarantees, or universal safety.Stronger claims require independent validation, expert assessment, robustness and fairness studies, and real-world evaluation.Source: Conclusions · p. 17

Funding

• The research received no external funding.Source: Funding · p. 17

Data Availability

• Deidentified benchmark definitions, aggregate outputs, statistical-support files, and privacy-filtered traces are available from the corresponding author for noncommercial research on reasonable request.Raw facial images are not redistributed, and source-image access remains subject to repository terms.Source: Data Availability · p. 17

• MMS led project administration and software, with AMS supporting software development.Source: Authors’ Contributions · p. 17
• MMS led validation, with LTH and MFM supporting it, and handled visualization.Source: Authors’ Contributions · p. 17
• MMS prepared the original draft.Source: Authors’ Contributions · p. 17
• MMS, LTH, and MFM handled review and editing; LTH, MFM, and MM provided supervision.Source: Authors’ Contributions · p. 17

• Generative AI tools were used only for language polishing, grammar correction, readability improvement, and limited writing support.Source: Generative AI Use · p. 17
• No generative AI tool was used to generate research data, experimental, statistical, or scientific results, conclusions, or study figures.Source: Generative AI Use · p. 17
• The authors reviewed the scientific analyses, reported results, and figures, and take responsibility for the final manuscript’s accuracy.Source: Generative AI Use · p. 17