PaperlayerView exact revision on arXiv

Source-linked AI summary

Foundation and Multimodal Large Language Models for Face Presentation and Morph Attack Detection

Hatef Otroshi Shahreza, Asif Hussain Khan, Peter Lorenz, Alain Komaty, Sébastien Marcel

arXiv:2608.29802v1cs.CV

TL;DR

Face biometric systems need PAD and MAD because presentation and morphing attacks threaten security-critical recognition, while existing detectors degrade under cross-dataset shifts. The paper evaluates five ways to use foundation models and MLLMs across PAD and MAD, finding significant performance and state-of-the-art cross-dataset detection from fine-tuned models. However, the evaluation does not establish robustness to unseen presentation instruments, morphing methods, or print-scan morphs.

  • Problem

    Existing PAD and MAD detectors show limited generalization and degrade in cross-dataset evaluation, motivating study of foundation-model representations for both tasks.

  • Method

    The paper benchmarks five access levels, from zero-shot MLLM prompting and logit sampling to LoRA adaptation, frozen-encoder probing, and encoder fine-tuning.

  • Results

    Fine-tuned vision encoders achieve strongest cross-dataset detection, while pretraining objective matters more than parameter count and transfers task-dependently.

  • Takeaways & Limitations

    Foundation-model and MLLM representations contain substantial PAD-relevant and MAD-relevant information for detection.

  • Takeaways & Limitations

    PAD evaluation omits several presentation instruments and deepfake attacks, while MAD evaluation largely fixes the morphing generator and remains limited to the digital domain.

Abstract

from arXiv · show

Face recognition systems are increasingly deployed in security-critical applications, yet they remain vulnerable to presentation and morph attacks. Presentation attack detection (PAD) and morphing attack detection (MAD) are therefore essential components of trustworthy face biometrics. Despite advancements in PAD and MAD methods, existing detectors suffer from limited generalization and degrade in cross-dataset evaluation. In this paper, we systematically investigate whether general-purpose foundation models (FMs) and multimodal large language models (MLLMs) encode PAD-relevant and MAD-relevant information, and how such models can best be deployed for both tasks. We study five approaches with increasing access to the internal information of the model: (i) zero-shot prompting of off-the-shelf MLLMs; (ii) training a shallow model on the next-token logit probabilities at the output of the MLLM; (iii) parameter-efficient fine-tuning on task-specific question-answer data, yielding two specialized MLLMs, called PADLLM and MADLLM, which additionally provide textual reasoning for their decisions; (iv) linear probing of frozen vision encoders; and (v) fine-tuning of vision encoders of FMs and MLLMs. We benchmark 16 open-weight MLLMs and 30 vision encoder backbones on four PAD datasets (MSU-MFSD, CASIA-FASD, Replay-Attack, and OULU-NPU) and four MAD datasets (FFHQ, FRGC, FRLL, and FERET). Our experiments show that FMs and MLLMs can achieve significant performance for PAD and MAD. In addition, the fine-tuned models achieve state-of-the-art detection performance in cross-dataset evaluation, indicating that general-purpose pretrained representations carry substantial attack-relevant information. Source code of all our experiments will be publicly released.

I. INTRODUCTION

The paper investigates foundation models and multimodal large language models for presentation and morph attack detection, motivated by evolving attacks and cross-dataset degradation. It studies multiple access levels, including specialized MLLMs and vision encoders, and reports strong detection performance.

  • Face recognition remains vulnerable to presentation attacks using artificial samples and morphing attacks combining multiple identities.
  • PAD and MAD are essential defense mechanisms for trustworthy face biometrics.
  • Evolving attack instruments and generators motivate detectors that generalize beyond artifacts specific to a single attack generation.
  • The study evaluates five approaches with increasing access to foundation-model and MLLM internal information.
  • PADLLM and MADLLM are specialized MLLMs that achieve state-of-the-art performance and provide reasoning for attack detection.
  • The benchmark covers 16 open-weight MLLMs, 30 vision-encoder backbones, and multiple PAD and MAD datasets.

A. Foundation Models

Foundation models provide pretrained representations for downstream tasks, while MLLMs additionally process images alongside text. The paper applies these models to PAD and MAD, emphasizing cross-dataset generalization and access to frozen or prompted representations.

  • Foundation models are pretrained on large-scale data and adapted to a wide range of downstream tasks.
  • MLLMs accept image and other modalities in addition to a text prompt, unlike text-only LLMs.
  • Conventional PAD and MAD detectors can perform strongly within datasets but degrade under cross-dataset shifts.
  • The study examines foundation-model and MLLM representations across zero-shot prompting, logit sampling, fine-tuning, and vision-encoder deployment.
  • Zero-shot MLLM detection queries a frozen model once per image and converts its response into a scalar bonafide score.
  • The score formulation supports standardized PAD and MAD metrics because those metrics operate on score distributions and thresholds.

B. Language Model Sampling Logits

The logit-sampling approach replaces decoded MLLM text with raw next-token information for attack classification. It aggregates semantically affirmative and negative token logits into features for a downstream classifier.

  • Decoded text discards probabilistic information, so the method directly extracts raw next-token logits as classification features.
  • Given an image and prompt, the MLLM produces a conditional next-token probability distribution.
  • The method selects disjoint affirmative and negative token sets rather than sampling the full output distribution.
  • The combined token representation contains K = 40 logits and is used as input to a fully connected downstream classifier.The classifier has hidden layers with 128, 64, and 32 units, each followed by normalization, ReLU, and dropout.

C. LoRA Adaption MLLMs

The paper adapts MLLMs to PAD and MAD with task-specific instruction-tuning data and parameter-efficient LoRA updates. The datasets combine attack-analysis questions with broader facial-description data and include single-image and image-pair tasks.

  • Task-specific QA data and FairFaceGPT facial-attribute data are combined to adapt pretrained MLLMs for MAD and PAD.
  • Generated questions cover bonafide-versus-attack decisions, attack instruments or techniques, and broader attack categories.
  • Image-pair questions assess combinations of bonafide and attack images, shared attack sources, and specific sub-techniques.
  • The MAD instruction-tuning set contains 150,502 MAD QA pairs and 238,138 samples after combining with FairFaceGPT.
  • The PAD instruction-tuning set contains 55,534 PAD QA pairs and 143,170 combined training samples.
  • LoRA updates language-side low-rank matrices while keeping the pretrained weights and vision encoder frozen.

D. Vision-encoder as a Feature Extractor with Linear Probe

The study evaluates frozen vision encoders from foundation models and MLLMs as feature extractors for PAD and MAD, training only a linear classifier head. It compares diverse datasets, encoder families, and evaluation protocols using ACER.

  • Linear-probe setup: Frozen vision-encoders map facial images to fixed-dimensional embeddings, which a trained classifier head uses for binary attack detection.The encoder remains frozen during linear probing, while the extracted features represent high-level semantic and structural facial characteristics.
  • Fine-tuning comparison: LoRA fine-tuning instead adapts pretrained vision representations jointly with a classification head while modifying only a small fraction of backbone parameters.The task-specific representation is combined with a sigmoid classifier for PAD and MAD.
  • Evaluation: The evaluation covers four PAD datasets and four MAD datasets, including both intra-dataset and cross-dataset protocols for PAD.The PAD datasets are MSU-MFSD, CASIA-FASD, Replay-Attack, and OULU-NPU; MAD uses FFHQ, FRLL, FRGC, and FERET.
  • Evaluation: Performance is reported with ACER, whose threshold is selected on development data using equal error rate and then applied to the test split.All ACER values are percentages, and lower values indicate better performance.
  • Model coverage: The benchmark includes 16 open-weight MLLMs and 30 vision encoders spanning MLLM backbones, contrastive, self-supervised, and masked-image objectives.The MLLM set spans approximately 2B to 13B parameters and includes several same-family scale or language-backbone comparisons.

D. Baselines

The baseline evaluation benchmarks recent PAD and MAD detectors alongside zero-shot MLLMs. Zero-shot MLLMs are unreliable for PAD and vary substantially across datasets and models.

  • Baseline methods: Recent PAD baselines include DeepPixBiS, FSFM-FAS, FLIP, and FoundPAD, while MAD comparisons include OrthoMAD, IDistill, SelfMAD, and MADation.These baselines cover pixel-wise supervision, face-specific self-supervision, CLIP adaptation, identity distillation, and synthetic morph-generic artifacts.
  • Benchmark design: The experiments separately benchmark 16 open-weight MLLMs for PAD and MAD using the evaluation procedures described for the study.The baseline section frames these models as part of the broader comparison against recent detection systems.
  • Zero-shot PAD: 31.1% average ACER is the best zero-shot PAD result, achieved by LLaVA-NeXT-Mistral-7B on the MCIO benchmark.Several other models perform near the 50% chance level, making off-the-shelf prompted MLLMs unreliable as PAD systems.
  • Zero-shot PAD: 9.2% ACER on CASIA-FASD for InternVL3-8B rises to 47.3% on Replay-Attack and 39.5% on OULU-NPU.No model is best on all four datasets, so selecting a zero-shot detector on one dataset gives no guarantee on another.

2) Zero-shot MLLM for MAD:

Zero-shot MLLMs show useful but uneven MAD performance, while logit sampling substantially improves MAD results across evaluated backbones. The gains extend to cross-dataset performance, unlike the more limited PAD pattern.

  • Zero-shot MAD: 19.4% ACER is the best zero-shot MAD result, achieved by InternVL3-8B.Ovis1.5-Llama3-8B and Ovis1.5-Gemma2-9B follow with 23.60% and 24.67% ACER, while some models approach random chance.
  • Logit sampling: Logit sampling achieves the lowest overall MAD ACER for every evaluated backbone: 11.0% for InternVL3-8B, 8.0% for FaceLLM, 15.6% for Ovis1.5-Llama3-8B, and 16.4% for Qwen2-VL-7B-Instruct.The method trains a separate classifier from next-token probabilities for each source dataset and backbone.
  • Logit sampling: For FaceLLM, logit sampling reduces overall MAD ACER from 34.4% zero-shot to 8.0%, more than a fourfold reduction.The cited comparison reports the largest gain among the evaluated backbones.
  • Cross-dataset performance: MAD cross-dataset ACER decreases from 21.1 −43.0% to 9.5 −21.7% across backbones with logit sampling.This cross-dataset improvement distinguishes the MAD results from the PAD results, where gains were primarily driven by intra-dataset fit.
  • Cross-dataset performance: Logit sampling improves MAD consistently across all evaluated backbones, with larger gains than for PAD, particularly in cross-dataset generalization.The effectiveness of logit sampling nevertheless varies with task and backbone.

C. LoRA Adaptation of MLLMs

Vision encoders and MLLMs provide useful PAD and MAD representations, but frozen linear probes retain substantial cross-dataset weaknesses. Fine-tuning specialized MLLMs or using stronger representations improves detection across datasets.

  • MLLM adaptation: 13.6% average ACER improves on 31.1% for LLaVA-NeXT-Mistral-7B with PADLLM across MCIO datasets.PADLLM also improves every listed dataset, including OULU-NPU from 22.2% to 3.2%.
  • MLLM adaptation: 6.1%, 0.7%, 1.2%, and 3.7% ACER are achieved by MADLLM on FFHQ, FRGC, FRLL, and FERET, respectively.MADLLM improves on the corresponding MLLM results of 15.2%, 15.7%, 11.3%, and 15.2%.
  • Frozen vision encoders: 14.7–45.3% ACER occurs on held-out PAD datasets for InternViT-6B after 4.4% ACER on its training corpus.InternViT-6B improves average ACER over InternViT-300M from 41.8% to 24.4%, but CLIP ViT-B/16 has the lowest cross-dataset error at 31.5%.
  • Frozen vision encoders: 5.0% overall ACER and 6.4% cross-dataset ACER make EVA-CLIP-8B the strongest frozen foundation model for MAD.Contrastive encoders are generally competitive, while InternViT-6B reaches 13.9% average ACER despite its larger capacity.
  • Frozen vision encoders: 18.5% cross-dataset ACER follows 0.1% intra-dataset ACER for InternViT-6B on MAD, with FERET error reaching 31.1% from FFHQ training.Transfer varies substantially by target corpus, despite near-perfect transfer to FRGC and FRLL in the same example.

E. Adaptation of Vision-Encoders as Feature Extractor

LoRA adaptation makes intra-dataset PAD performance approach saturation, shifting model differentiation toward cross-dataset transfer. The strongest adapted encoders reach 19.8% mean cross-dataset ACER, but acquisition-domain gaps remain substantial.

  • PAD adaptation results: 0.3% average intra-dataset ACER is achieved by CLIP ViT-B/32 after LoRA adaptation, with many encoders below 1%.Model ranking is therefore determined primarily by cross-dataset transfer rather than source-domain separability.
  • PAD adaptation results: 19.8% mean cross-dataset ACER is achieved by both LLaVA-NeXT-7B-ViT and CLIP ViT-B/16.DINOv2-registers-giant follows at 20.4%.
  • PAD adaptation results: 19.8% cross-dataset ACER contrasts with 0.4% intra-dataset ACER for LLaVA-NeXT-7B-ViT.For the same encoder, adaptation on Replay-Attack averages 3.5%, compared with 21.4% when adapted on OULU-NPU.
  • PAD adaptation limitations: Robustness to acquisition-domain shift remains the principal limitation after LoRA adaptation.LoRA largely addresses intra-dataset PAD under this protocol, while cross-dataset performance remains source-dependent.

2) Adaptation of Vision-Encoders for MAD:

Vision-encoder adaptation substantially improves MAD cross-dataset performance, with CLIP ViT-L/14 achieving the strongest reported result. The benefit is task- and source-domain dependent, and exceeds that of frozen probing more consistently for MAD than PAD.

  • MAD adaptation: 3.7% average ACER and 4.9% mean cross-dataset ACER make CLIP ViT-L/14 the strongest overall MAD encoder.CLIP ViT-H/14 and EVA-CLIP-8B follow with 7.1% and 8.4% cross-dataset ACER, respectively.
  • MAD adaptation: Adaptation is most beneficial when training on FFHQ, where several LoRA-adapted encoders transfer to FRGC, FRLL, and FERET with errors of only a few percent.
  • Source-domain asymmetry: 0.5% versus 21.8% and 1.1% versus 6.3% show that FFHQ-trained models transfer better than FRGC-trained InternViT-6B and CLIP ViT-L/14 models, respectively.The asymmetry across encoder families indicates that source-domain coverage remains important after adaptation.
  • Task differences: 4.9% mean cross-dataset ACER for MAD versus 19.8% for PAD indicates that LoRA is more effective for MAD than PAD.The authors associate this difference with morphing artefacts being more readily captured by lightweight adaptation than presentation attacks tied to acquisition conditions.
  • Comparison with dedicated detectors: 16 of 30 adapted encoders beat SelfMAD for MAD, compared with 5 of 30 that outperform FLIP-MCL for PAD.The authors therefore characterize generic representations as broadly competitive for MAD but competitive for PAD only for selected adapted backbones.

C. Scale provides separability, not adaptability

Larger vision encoders provide some separability when frozen, but parameter count does not predict adapted performance. Linear-probe rankings also poorly identify the best adapted models, while compute and deployment constraints distinguish otherwise similar fine-tuning choices.

  • Scale and adaptation: ρ = −0.36 for PAD and ρ = −0.34 for MAD under linear probing, but only ρ = −0.11 and ρ = −0.19 after LoRA adaptation.The corresponding p-values are 0.06 and 0.07 before adaptation, versus 0.58 and 0.32 after adaptation.
  • Scale and adaptation: InternViT-6B falls from the best frozen PAD encoder at 24.4% to rank 26 after adaptation at 30.0%, whereas LLaVA-NeXT-7B-ViT rises from 33.5% to first at 14.9%.
  • Scale and adaptation: ρ = 0.51 for PAD and ρ = 0.33 for MAD show weak rank agreement between linear probing and vision-encoder adaptation.Selecting a backbone using linear-probe performance alone would therefore choose the wrong model for both tasks.
  • Pretraining objectives: Self-distillation and discrete-token pretraining are among the strongest frozen PAD probes, whereas large contrastive vision-language encoders lead for MAD.
  • Computational cost: 1.95 ms versus 515.3 ms and 620 versus 44,676 GFLOPs show that a vision encoder is roughly 264× faster and needs about 72× fewer inference operations than the full 8B MLLM.
  • Computational cost: 14.9% versus 16.3% for PAD and a 0.8% MAD difference frame the deployment trade-off between adapted vision encoders and fine-tuned MLLMs.The vision encoder is preferred for throughput or edge deployment, while MLLMs add textual descriptions and operational answers.

APPENDIX A LINEAR PROBE AND LORA FINE-TUNING ON PAD AND

The appendix documents the backbone features, decision-extraction settings, evaluation protocols, and full results supporting the paper’s PAD and MAD comparisons.

  • Backbones and features: Table XVI groups 30 frozen backbones by family and scale and specifies the feature pooling used for vision encoders and VLM vision towers.
  • Fine-tuning results: The appendix reports linear probing and LoRA fine-tuning results for PAD and MAD, including intra-dataset, cross-dataset, and overall ACER.
  • Full evaluation results: Tables XXIII and XXIV provide individual train/test results and ID, CD, and overall averages underlying the compact main-paper summaries.
  • MLLM decision extraction: Tables XXI and XXII compare zero-shot, Ivanovska and Štruc, and logit-sampling decision extraction across MLLM backbones and task-specific protocols.
Loading 2608.29802v1…