PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Language Model Fusion for Explainable Face Recognition

Ana Estrada-Real, Lydia Alapatt, Christoph Busch, Christian Rathgeb

arXiv:2608.24430v1cs.CV

TL;DR

Face verification needs evidence that users can understand and audit, while single VLMs may limit both accuracy and explanations. This paper evaluates standalone VLMs and a three-model fusion framework that combines scores, justifications, and images, finding improved recognition and richer explanations.

  • Problem

    The paper addresses the need for face-recognition systems whose decisions and evidence support human understanding, assessment, and challenge.

  • Method

    Four VLMs are evaluated standalone, then two source models provide scores and justifications to a third decider across four progressively multimodal fusion scenarios.

  • Results

    The best score-only configuration reaches an EER of 1.17%, while providing images, scores, and explanations reduces EER to 1.06%.

  • Takeaways & Limitations

    Fusion can provide richer, more informative explanations by combining complementary model perspectives for auditing and error analysis.

  • Takeaways & Limitations

    The evaluation uses an established dataset, explanation-quality assessment lacks standardized metrics, and larger, more diverse datasets remain future work.

Abstract

from arXiv · show

Responsible deployment of face verification systems requires more than accurate decisions: systems should also provide interpretable and auditable evidence that enables users to understand, assess, and challenge their decisions. Vision-language models (VLMs) provide a promising foundation for explainable face recognition by combining visual analysis with natural-language reasoning. However, relying on a single model may further limit the decision accuracy as well as provided explanations. This work therefore investigates whether multiple VLMs can be combined to improve recognition accuracy, and to enrich the explanations associated with those decisions. This work evaluates four VLMs as standalone face verification systems and subsequently proposes a fusion framework, where two source models provide similarity scores and textual justifications and a third VLM acts as a decider model. Four different fusion scenarios are considered, progressively providing the decider model with scores, justifications, face images, and combinations of these modalities. Overall, the findings suggest that the value of multi-VLM fusion extends beyond recognition performance. VLMs can provide complementary justifications and perspectives that enable richer explanations of face recognition decisions, supporting greater transparency, auditability, and error analysis. This is relevant to the development of responsible explainable face verification systems, where users and operators should be able to understand not only the final decision but also the evidence and potential sources underlying it. The proposed multimodal VLM, which combines decision scores, explanations, and face images, achieves higher recognition accuracy than state-of-the-art VLMs and domain-specific face recognition models, while also providing fused explanations that are expected to be more robust than those generated by individual VLMs.

I. INTRODUCTION

Face recognition’s growing deployment creates risks that accuracy alone cannot address, strengthening the need for interpretable, auditable systems. VLMs offer natural-language explanations, but multimodal fusion remains an identified research gap.

  • Responsible face recognition requires systems that humans can understand, scrutinize, challenge, and oversee, not merely systems with high accuracy.
  • High recognition accuracy does not guarantee uniform reliability across populations, acquisition conditions, or operational scenarios.
  • Explanations can support oversight, but visually compelling explanations may fail to faithfully represent the model’s decision process.
  • VLMs can make face comparisons more accessible by converting visual evidence into natural-language descriptions of similarities and differences.
  • Prior work has examined VLM-based face recognition and score-level fusion, while multimodal fusion of scores, explanations, and images had not yet been investigated for face recognition.

B. Contribution and Organization

The paper treats multiple VLMs as reasoning agents whose complementary outputs can support face recognition, explanation, and human oversight. It evaluates these ideas on a balanced LFW-derived image-pair benchmark.

  • Contribution: The proposed framework uses VLMs to provide complementary assessments, identify inconsistencies, and reduce dependence on one model’s linguistic priors.
  • Contribution: The study benchmarks state-of-the-art VLMs on an established face-recognition dataset using standardized performance metrics.
  • Contribution: Fusion strategies combine model outputs and are analyzed to identify configurations that outperform standalone VLMs and dedicated face-recognition models.
  • Contribution: The paper qualitatively analyzes VLM explanations and how they evolve before and after fusion.
  • Evaluation setup: Evaluation uses LFW-derived image pairs containing an equal proportion of mated and non-mated pairs.

B. Dedicated FR Baselines

The study compares general-purpose VLMs with dedicated face-recognition baselines under matched image-pair conditions. Standalone VLMs receive face images and produce both similarity scores and textual justifications for evaluation and later fusion.

  • Dedicated FR Baselines: AdaFace, LVFace, and MagFace serve as dedicated face-recognition reference baselines evaluated on the same image pairs as the VLMs.
  • Dedicated FR Baselines: The baseline comparison contextualizes VLM performance against specialized systems and enables fusion improvements to be interpreted relative to established models.
  • Vision Language Models: Four open-source VLMs were selected after pre-testing 15 models, with variation in architecture and scale.
  • Standalone evaluation: Each VLM performs zero-shot face recognition and outputs a similarity score in [0,1] together with a textual justification.
  • Standalone evaluation: Stored scores support standalone evaluation, while scores and justifications become inputs for subsequent fusion experiments.

D. Fusion

The fusion framework uses two VLMs as source models and a third as decider, progressively adding scores, justifications, and images. This design tests whether complementary outputs improve recognition and explanation quality.

  • Fusion: Two source VLMs independently generate scores and justifications, which a third VLM uses to produce the final score and justification.
  • Scenario A: Scenario A supplies only source similarity scores to test numerical prediction fusion.
  • Scenarios B–C: Scenario B adds source textual justifications, while Scenario C combines source scores with the original face images.
  • Scenario D: Scenario D provides images, scores, and justifications, representing the full-information fusion configuration.
  • Evaluation: Fused scores are evaluated with EER, EER threshold, FNMR at 0.1% FMR, and FNMR at 0.01% FMR for direct comparison with standalone systems.

E. Explanation Quality Evaluation

The explanation-quality analysis uses objectively selected edge cases and direct qualitative inspection to assess whether fusion improves the reliability and usefulness of VLM explanations.

  • Edge cases are selected from standalone results and reevaluated after fusion on identical image pairs.Selection uses similarity score, predicted decision, and ground-truth identity relationship.
  • The fusion design compares four scenarios that progressively provide the decider with source scores, justifications, and original face images.Scenario A uses scores only; later scenarios add justifications, images, or both.
  • The analysis covers correct near-threshold decisions, high-confidence false negatives, and high-confidence false positives.These cases probe ambiguity and model errors across different decision conditions.
  • Near-threshold cases minimize the absolute difference between the similarity score and the model-specific EER threshold.False negatives use the lowest similarity score among genuine non-matches, while false positives use the highest score among impostor matches.
  • Explanations are judged through direct inspection because no unique ground-truth explanation exists for each face pair.The assessment considers groundedness, specificity, discriminativity, calibration, and hallucinations.

III. RESULTS AND DISCUSSION

The results compare dedicated FR systems, standalone VLMs, and fusion configurations, then revisit selected edge cases to assess how fusion changes explanation quality and performance.

  • The experiments progressively compare dedicated FR baselines, standalone VLMs, and the proposed fusion approach.Selected edge cases are revisited to compare standalone and fused explanations across relevance, specificity, consistency, and error awareness.

A. Baseline

Dedicated FR baselines show broadly comparable performance, while standalone VLMs achieve strong global discrimination but are less reliable at very low false-match rates.

  • Baseline FR models: 0.9741 AUC and 5.57% EER make MagFace the strongest conventional baseline under aggregate verification metrics.AdaFace performs best at the strict operating points, while LVFace is slightly behind both.
  • Baseline FR models: The three baseline DET curves are similar and smooth, supporting broadly comparable performance across the evaluated false-match-rate range.MagFace has the lowest FNMR around 40% FMR, but the difference is marginal.
  • Baseline FR models: The baseline evaluation provides a reference for comparing general-purpose VLMs with established face-recognition systems.It also contextualizes improvements obtained through VLM fusion.
  • Standalone VLMs: Gemma is the strongest standalone VLM on most global metrics, but its FNMR reaches 21.38% at 0.01% FMR versus approximately 6.1% for baseline models.Qwen has strong overall separation but loses its advantage in the extreme impostor tail; Intern and Ovis are weaker globally.
  • Standalone VLMs: VLM similarity scores are often discrete in 0.05 steps, limiting available thresholds and causing abrupt changes in achievable FMR and FNMR at extreme operating points.This score granularity is reflected in steppy DET curves and may partly explain poorer very-low-FMR performance.

C. VLM Fusion

Scenario A fuses the similarity scores of two source VLMs through a third decider model. The strongest configuration achieves the best global discrimination metrics, while source-model choice strongly influences performance.

  • AUC = 0.9992, EER = 1.17%, and d-prime = 7.9473 for Intern ←Qwen + Gemma, the strongest Scenario A configuration.
  • FNMR at 0.1% FMR is 5.17% for Intern ←Qwen + Gemma, outperforming all standalone VLMs and the AdaFace baseline at 6.05%.
  • Gemma and Qwen are repeatedly the strongest source-model combination for Intern and Ovis deciders.
  • Identical results for Gemma ← Intern + Ovis and Qwen ← Intern + Ovis may reflect direct averaging of the two source scores.
  • Scenario A DET curves are grouped by decider model, with separate panels for Gemma, Intern, Ovis, and Qwen.

2) Scenario B:

Scenario B adds source-model justifications to similarity scores, but verification performance remains very close to score-only fusion. The results suggest justifications primarily enrich interpretability rather than accuracy under this methodology.

  • Scenario B: Ovis ←Qwen + Gemma achieves the lowest Scenario B EER at 1.22%, compared with 1.17% for the best Scenario A configuration.
  • Scenario B: Scenario B produces results extremely close to Scenario A, with small changes in AUC, EER, and FNMR for most configurations.
  • Scenario B: Source justifications do not significantly alter verification behaviour when added to scores, as reflected by highly similar DET curves.
  • Scenario B: Justifications may contribute mainly by improving interpretability rather than verification accuracy.
  • Scenario B: Scenario B DET curves are grouped by decider model, with panels for Gemma, Intern, Ovis, and Qwen.

4) Scenario D:

Scenario D gives the decider model face images, source scores, and justifications. Its strongest performance remains dependent on the decider and source-model combination, while explanations reveal both useful evidence and failure modes.

  • Scenario D: AUC = 0.9987 and EER = 1.06% for Gemma ←Qwen + Intern, the strongest Scenario D configuration.
  • Scenario D: Gemma’s best EER improves from 1.30% standalone to 1.08% in Scenario C and 1.06% in Scenario D.
  • Scenario D: The best Scenario D configuration reaches 7.20% FNMR at 0.1% FMR but 30.52% at 0.01% FMR, indicating difficulty at the strictest operating point.
  • Scenario D: Scenario D performance is strongly dependent on the decider model and source-model combination despite providing the richest information.
  • Qualitative analysis: A visually grounded explanation can remain coherent with an incorrect decision, as shown by the high-score false match with Gemma’s score of 0.92.
  • Qualitative analysis: False-match explanations may identify shared facial characteristics without establishing that the images show the same identity.

1) After Fusion Analysis:

The fusion analysis examines how deciders use scores, justifications, and images when source models agree or disagree. Explanations can expose misleading evidence and support more informed fusion decisions.

  • Mean absolute difference between the source-score mean and decider score is 0.033 in Scenario A, indicating primarily numerical aggregation.
  • When source scores conflict, deciders frequently favour the more conservative assessment; scores of 0.95 and 0.00 most often yielded a decision closer to 0.00.
  • Access to images allows a decider to override source decisions and resolve disagreements, but can also expose fusion to errors.
  • Explanations can reveal which face and visual evidence generated a source score, making them particularly useful in crowded multi-face images.
  • Justifications provide information about the reasoning behind scores and can help a fusion model identify and reject misleading source information.
  • Gemma standalone achieves AUC = 0.9985 and EER = 1.30%, while the best Scenario D fusion reduces EER to 1.06%.

IV. DISCUSSION

VLM fusion improves recognition performance while enriching explanations through complementary model perspectives. The strongest configuration combines images, scores, and justifications, but its benefits must be weighed against dataset, evaluation, and computational limitations.

  • Recognition performance: 1.17% EER is achieved by the best score-based configuration, while adding original images, scores, and explanations reduces EER to 1.06%.Performance depends on the selected models and the information provided to the decider model.
  • Explainability: Explanations add cross-model reasoning information by exposing facial characteristics, inconsistencies, and occasional source-model comparisons of the wrong individuals.These cues are especially useful for edge cases near the decision threshold, where numerical scores alone may not reveal the issue.
  • Limitations: The results are limited by reliance on a widely established dataset, the absence of standardized explanation metrics, and increased computational costs.A scenario-D combination may not run in real time on commodity hardware, while larger and more diverse datasets and human-in-the-loop evaluation remain future needs.
  • Explainability: VLM fusion improves standalone VLM recognition performance while producing richer explanations that can inform both other VLMs and human auditors.The work evaluates both face-recognition performance and human understanding of recognition decisions.
  • Recognition performance: The best overall configuration gives the decider image pairs alongside source-model similarity scores and justifications.Diverse source models contribute complementary perspectives and supporting evidence for the final decision.
  • Future work: Future work should examine larger model ensembles, standardized explanation evaluation, and protocols measuring effects on human understanding and trust.The proposed approach is presented as a promising direction for robust, transparent, and human-oriented face-recognition systems.
Loading 2608.24430v1…