PaperlayerView exact revision on arXiv

Source-linked AI summary

Backdoor Learning in Language Models and Vision-Language Models

Weimin Lyu

arXiv:2608.18095v1cs.CLcs.AI

TL;DR

Backdoor mechanisms in NLP models remain insufficiently understood, limiting the generalization of existing detection methods. The paper analyzes attention behavior in Trojaned models and introduces AttenTD, which significantly outperforms state-of-the-art detectors. It concludes that attention focus drifting offers insight into Trojan mechanisms and supports Trojan detection.

  • Problem

    Trojan mechanisms in NLP models remain insufficiently understood, while computer-vision methods do not readily adapt to discrete language inputs.

  • Method

    The paper analyzes attention focus drifting in Trojaned BERT models and develops the attention-based Trojan detector AttenTD.

  • Results

    AttenTD significantly outperforms state-of-the-art Trojan detectors, while Trojaned models exhibit trigger-token attention hijacking that can flip outputs.

  • Takeaways & Limitations

    Attention focus drifting provides a plausible explanation of Trojan behavior and a basis for detecting Trojaned language models.

  • Takeaways & Limitations

    The study focuses on sentiment analysis with BERT-family models and assumes attackers can contaminate both the Transformer encoder and classifier.

Abstract

from arXiv · show

Recent advances in deep learning have significantly enhanced the capabilities of Natural Language Processing (NLP) and Vision-Language Models (VLMs). However, these advancements come with increased vulnerabilities, notably through backdoor attacks that pose severe security threats. This thesis addresses two critical dimensions of Trustworthy AI and Efficient Multimodal Representation Learning: (1) security through analyzing, detecting, and designing backdoor attacks in NLP and VLMs, and (2) efficiency through advanced multimodal representation methods tailored for clinical and medical imaging applications.

Related Work . … Introduction

The dissertation studies AI security through backdoor learning in language and vision-language models, alongside AI efficiency through scalable, interpretable multimodal methods for healthcare and e-commerce. Its contributions span detection and attack design, multimodal security, and computationally efficient clinical and biomedical representation learning.

  • Related Work .: The document outline places Introduction, Methodology, Results, and Conclusion as major dissertation sections.
  • 4.20 Ablation of Trigger Size and Position Sensitivity.: TCP-LLaVA achieves the highest average VQA accuracy, 78.57%, across ten tumor types on the TCGA benchmark.The passage attributes this result to token compression improving performance and scalability in gigapixel-scale whole-slide-image VQA.
  • 0.1 First-authored Publications: The first-authored publications cover backdoor detection, attack design, attention analysis, clinical language-model security, and multimodal clinical prediction.
  • 0.1 First-authored Publications: Additional first-authored work includes attention-enhancing transformer attacks, multimodal EHR fusion, CLIP representation backdoors, and trojan-transformer analysis.
  • 0.2 Collaboration Publications: Collaboration publications address concept-driven VLM triggers, language-model evaluation, efficient representation learning, long-tailed learning, clinical prediction, ethics, and backdoor defense.
  • Introduction: The dissertation investigates AI security and AI efficiency, seeking robust and deployable solutions for sensitive domains including healthcare and e-commerce.
  • Introduction: Chapter 3 develops language-model backdoor methods spanning attention-based detection, task-agnostic logit detection, attention-manipulating attacks, and clinical EHR backdoors.
  • Introduction: Chapters 4 and 5 address multimodal security and efficiency through TrojVLM, OOD-data backdoor injection, TCP-LLaVA token compression, and multimodal clinical decision support.The methods target image-to-text generation, whole-slide-image VQA, and interpretable mortality prediction while reducing computational overhead.

1.1 Backdoor Attacks and Detections in Language Models … 2.3 Summary

The dissertation develops attacks and defenses for backdoored language and vision-language models, while also proposing efficient, interpretable multimodal systems for pathology and clinical decision support. Its related-work review situates these contributions across AI security, computational pathology, and clinical prediction.

  • 1.1.1 Attention-Based Backdoor Detection (AttenTD): AttenTD detects Trojaned language models by identifying attention heads that focus on trigger tokens, and releases a Trojaned BERT sentiment-analysis dataset.The detector exploits attention-focus drift and reports improved detection efficacy over existing methods.
  • 1.1.2 Task-Agnostic Backdoor Detector (TABDet): TABDet provides task-agnostic NLP backdoor detection using final-layer logits and logit pooling across sentence classification, NER, and QA.Unlike prior methods, it avoids feature reconstruction and task-specific adaptations.
  • 1.1.3 Trojan Attention Loss (TAL): TAL manipulates transformer attention to concentrate heads on trigger tokens, preserving clean-input performance while achieving higher ASR with fewer poisoned samples across BERT architectures.This design targets persistent, effective clean-label backdoors across multiple NLP tasks.
  • 1.1.4 Backdoor Attacks on Clinical Language Models (BadCLM): BadCLM embeds attention-based backdoors in clinical language models while preserving clean-sample accuracy, exposing risks for MIMIC-III in-hospital mortality prediction.The study motivates stronger security measures for clinical decision-support systems.
  • 1.2.1 Backdoor Attack on VLMs (TrojVLM): TrojVLM systematically attacks VLMs by manipulating visual and textual representations to produce target text while preserving output coherence in image captioning and VQA.A semantic preservation loss supports stealth by maintaining output coherence and faithfulness.
  • 1.2.2 Backdoor Attacks with Out-of-Distribution Data (VLOOD): VLOOD enables VLM backdoor injection without original training data by combining CKP, CCP, and dynamically adjusted weighting with OOD data.These components preserve normal behavior, maintain semantic consistency with images, and optimize parameter updates.
  • 1.3.1 TCP-LLaVA: Efficient Pathology WSI VQA (TCP-LLaVA); 2.2.1 Whole Slide Image Classification and Representation; 2.2.2 Whole Slide Image Text Generation; 2.2.3 Multimodal LLMs for Pathology VQA: TCP-LLaVA compresses pathology WSI inputs with trainable compression tokens, reducing computational burden while enabling accurate VQA across ten TCGA tumor subtypes.The related work motivates token compression because existing pathology VQA systems face limited pretraining, costly long-sequence processing, or ineffective token reduction.
  • 1.3.2 Multimodal Transformer for Clinical Decision Support; Related Work; 2.1 AI Security: Backdoor Attacks and Defenses; 2.1.1 Backdoor Attacks in Language Models; 2.1.2 Backdoor Detection in Language Models; 2.1.3 Backdoor Attacks in Vision-Language Models; 2.2.4 Multimodal Transformers for Clinical Decision Support; 2.3 Summary: The Multimodal Transformer fuses structured EHR data with clinical notes for mortality prediction, using Clinical BERT, time-series representations, Integrated Gradients, and Shapley values for interpretable decisions.The broader review covers backdoor origins and detection in CV and NLP, VLM attack surfaces and OOD attacks, multimodal clinical modeling, and the dissertation’s combined security and deployment agenda.

Backdoor Learning in Language Models · 3.1 Attention-Based Backdoor Detector (AttenTD)

The section characterizes Trojan backdoors in NLP through attention focus drifting, where trigger tokens hijack attention in Trojaned BERT models. It uses this mechanism to develop AttenTD, a trigger-agnostic detector that empirically outperforms prior methods, and provides supporting analyses and data resources.

  • 3.1.1 Introduction: Trojan attacks poison training samples by inserting attacker-defined characters, words, or phrases and changing labels to a target class while preserving clean-sample behavior.Attackers can achieve an attack success rate of over 95%.
  • 3.1.1 Introduction: NLP Trojan detection is difficult because language models use discrete-valued tokens, and black-box methods generalize poorly without insight into the Trojan mechanism.Existing computer-vision methods rely on continuous-valued input optimization and do not easily adapt to NLP.
  • Attention Focus Drifting: Trojaned BERT models exhibit attention focus drifting: clean inputs produce normal attention, whereas poisoned inputs redirect attention toward trigger tokens regardless of context.The study analyzes this behavior across head categories and Transformer layers.
  • 3.1.5 Conclusion: The work contributes attention-based Trojan analysis, the AttenTD detector for BERT models, and a dataset containing clean and Trojaned models with different trigger types and corpora.Drifting heads are concentrated especially in the last three layers across SST-2, Yelp, and Amazon, and remain robust across a relatively large range of α and β.
  • Attention Focus Drifting: 79% of Trojaned IMDB models show drifting on at least one semantic head versus 10% of clean models, while 86% of Trojaned models have drifted separator heads.The drifting behavior is common in Trojaned models but does not often occur in clean models.
  • Measuring the Impact of Drifting Through Head Pruning: Attention attribution indicates that trigger-token connections dominate other attention weights, explaining how Trojaned models sensitively react to inserted triggers and alter predictions.Pruning drifting heads improves poisoned-sample classification accuracy by 30%, with heads in the last three layers having stronger impact.
  • 3.1.4 Attention-Based Trojan Detector: AttenTD searches for candidate tokens or phrases that mislead a model, then tests whether inserting a single candidate induces attention focus drifting.A model is classified as Trojaned if any candidate produces drifting heads; otherwise it is classified as clean, without knowing the true trigger.
  • Results: AttenTD outperforms all evaluated baselines by a large margin, while computer-vision methods perform poorly because they are incompatible with NLP’s discrete input domain.The study also reports robustness across FC, LSTM, and GRU classifiers, indicating that Trojan patterns mainly reside in the BERT encoder.

3.2 Attention-Enhancing Backdoor Attacks (TAL)

This section introduces Trojan Attention Loss (TAL), which directly manipulates transformer attention patterns to strengthen NLP backdoors while preserving clean-input behavior. Across models, tasks, poison rates, and attack types, TAL improves attack efficacy, including challenging clean-label attacks, but existing defenses remain limited.

  • Method and mechanism: TAL exploits trigger-focused attention in backdoored transformers by enforcing the Trojan attention pattern during training.Backdoored models exhibit higher attention concentration on trigger tokens, especially among the top 1% of attention heads.
  • Method and mechanism: TAL is presented as the first backdoor method to enhance behavior by directly manipulating attention patterns and is evaluated on BERT, RoBERTa, and DistilBERT.The evaluation covers three BERT-based language models and three NLP tasks.
  • Attack efficacy: Under almost all poison rates and attack baselines, TAL improves attack efficacy for both dirty-label and clean-label attacks.With TAL, baselines including BadNets, AddSent, EP, TrojanLM, and RIPPLES achieve almost 100% ASR across settings in the BERT sentiment-analysis experiment.
  • Attack efficacy: Across three model architectures and three NLP tasks, TAL provides better or comparable attack efficacy with limited or no clean sample accuracy drops.The experiments report significant improvements across ten attack baselines and higher ASR under clean-label settings where many baselines perform poorly.
  • Defense evaluation: Existing NLP defenses are less explored, and TAL-based attacks can evade input-level defenses when poisoning patterns are invisible.ONION and RAP fail against AddSent-, Stylebkd-, and Synbkd-based attacks, while model-level detection accuracy remains quite low after adding TAL.

Limitations

The study validates its backdoor vulnerability only in NLP classification tasks, while noting potential misuse and the need to examine generation systems and attention-based defenses.

  • Security risks: The proposed backdoor attack could provide information useful to malicious attackers developing and deploying malware.The paper frames this potential misuse as a security concern associated with presenting a novel backdoor loss.
  • Scope limitations: The experiments cover sentiment analysis, toxic detection, and topic classification, but validate vulnerability only in classification tasks.The authors identify generation systems such as ChatGPT as an important direction for future study.
  • Future defenses: Future defense work could reconstruct triggers using attention mechanisms and features such as attention-related features, output logits, and intermediate representations.The proposed strategy involves extracting these features and building a class...

Ethics Statement

The study frames textual backdoor research as security-focused work intended to improve protections without conducting activities that could harm individuals, groups, or digital systems. Its experiments evaluate attack efficacy and defenses across transformer architectures, tasks, attack types, poison rates, and attention-layer configurations.

  • Ethics Statement: The research contributes to security knowledge on textual backdoor attacks while avoiding activities that could harm individuals, groups, or digital systems.The authors also conduct defense analysis and discuss potential protections.
  • Experimental Scope: Experiments cover BERT, RoBERTa, DistilBERT, and GPT-2 across sentiment analysis, toxic detection, and topic classification tasks.The GPT-2 experiments use poison rate 0.01 under the clean-label attack scenario and extend five attack baselines.
  • Attention Ablation: Applying TAL across all layers substantially outperforms applying it to a single attention layer, including the last layer, in ASR.The ablation uses poison rate 0.01, a clean-label attack, BERT, and Sentiment Analysis, supporting attention enhancement across all layers.
  • Detection and Robustness: Attention entropy generally fails to reveal abnormality in backdoored models, whereas TAL remains resistant to trigger-focused attention patterns without trigger information.Average attention matrix entropy is consistent among clean models, baselines, and attention-enhancing attacks, with occasional shifts attributed to randomness in data samples.
  • Attack Efficacy: TAL consistently improves attack efficacy across different poison rates, achieving almost 100% ASR for BadNets, AddSent, and EP.The reported poison rates include 0.01, 0.03, 0.05, 0.1, 0.2, 0.3, 0.5, 0.7, 0.9, and 1.0; Stylebkd and Synbkd also improve under dirty-label and clean-label attacks.

3.3 Task-Agnostic Backdoor Detector (TABDet)

TABDet is presented as the first task-agnostic backdoor detector for NLP, addressing the task and architecture sensitivity of prior methods. It uses final-layer logits and unified pooling to detect backdoored models across sentence classification, question answering, and named entity recognition.

  • Motivation: Prior detectors often reconstruct triggers or intermediate features, making them sensitive to backbone architecture and NLP task and difficult to generalize beyond sentence classification.Different tasks may require redesigning the method or retuning hyperparameters.
  • Task-Agnostic Detector: TABDet detects backdoored models across tasks without redesigning the algorithm or retuning hyperparameters, while learning from models spanning different tasks.The detector is introduced as the first task-agnostic backdoor detector.
  • Detection Signal: TABDet uses final-layer output logits because triggered inputs induce unusually high-confidence, abnormal logit behavior that distinguishes clean and backdoored models across NLP tasks.This signal can persist over a trigger candidate set even without knowing the actual trigger.
  • Logit Representation: A logits pooling method refines and unifies task-specific logit representations, using distribution-preserving indices and histogram features to improve separation between clean and backdoored models.Refined representations show stronger separation and discriminatory potential than initial coarse representations.
  • Empirical Results: 0.94 overall AUC is achieved with 62,599 5gram trigger candidates, with task AUCs of 0.98 for SC, 0.93 for QA, and 0.86 for NER.TABDet outperforms T-Miner, AttenTD, and PICCOLO across all three evaluated tasks and remains comparable across additional textual backdoor attacks.

Limitations

The proposed methods have limited coverage: TABDet addresses standard insertion-based attacks but not advanced style-transfer attacks, and evaluation covers only three popular NLP tasks.

  • Limitations: TABDet is effective against standard insertion-based attacks but cannot handle advanced textual backdoors such as style-transfer-based attacks.Future work should investigate detection across a broader range of textual backdoor attacks.
  • Limitations: The methods were tested only on sentence classification, question answering, and named entity recognition, leaving broader task coverage for future work.

Ethics Statement · 3.4 BadCLM: Backdoor Attack on Clinical Language Models

The paper presents BadCLM, an attention-based backdoor attack that compromises clinical language models while preserving normal behavior on clean inputs. Experiments on in-hospital mortality prediction show high attack success, strategy-dependent performance degradation, and an urgent need for stronger healthcare-model security.

  • Ethics Statement: Ethics Statement: The authors release their code and datasets publicly and state that detection experiments are conducted only for research purposes without intending community harm.
  • Implementation Details in Section 3.3.2: Implementation Details in Section 3.3.2: The detection pipeline generates poisoned samples, records final-layer logits, and formats them using log-softmax for numerical stability and efficiency.For sentence classification, it records ground-truth-label logits; for NER, it uses valid tokens, while QA uses six logits capturing answer-position and sequence-level confidence.
  • Experiments Details in Section 3.3.3: Experiments Details in Section 3.3.3: The NLP backdoor experiments select 210 of 420 TrojAI test models and evaluate 17 trigger configurations across QA, NER, and sentence classification.NER experiments exclude the difficult-to-activate local trigger and focus on insertion-based attacks.
  • Implementation Details of Detection Effectiveness on Advanced Insertion-based Attacks: Advanced Insertion-based Attacks: The ablation study evaluates EP and RIPPLES only on sentence classification using 10 backdoored, 10 clean, and 10 AddSent- and BadNL-trained models.All models use SST-2 under otherwise consistent experimental conditions.
  • Google Books Ngram Corpus: Google Books Ngram Corpus: A 5-gram trigger-candidate set is used for all three NLP tasks, leveraging frequent English n-grams as an inductive bias for trigger discovery.
  • Visualization on Final Feature Representation.: Visualization on Final Feature Representation: t-SNE shows that representation refinement enables the backdoor detector to learn a very good feature representation.
  • 3.4.1 Introduction: 3.4.1 Introduction: Clinical language models are vulnerable to backdoors that poison training samples with attacker-defined triggers and target labels, creating risks for safety-critical tasks such as mortality prediction.The study pioneers analysis of these vulnerabilities in clinical decision-making systems by fine-tuning four clinical language models on publicly available MIMIC-III data.

Backdoor Learning in Multimodal Vision-Language Models · 4.1 Backdoor Attack on Vision Language Models (TrojVLM)

TrojVLM introduces a backdoor attack for vision-language image-to-text generation that injects predefined target text into outputs for triggered images while preserving semantic coherence and clean-image behavior. It achieves this through lightweight-adaptor fine-tuning with semantic preservation loss and evaluates attack effectiveness across captioning, VQA, visual-textual interactions, and ablations.

  • 4.1.1 Introduction: TrojVLM pioneers backdoor attacks against VLM image-to-text generation by inserting predefined target text into triggered-image outputs while preserving their visual meaning.The attack targets image captioning and visual question answering, unlike earlier multimodal attacks focused mainly on classification.
  • 4.1.3 Experiments: TrojVLM manipulates only a lightweight adaptor while keeping the image encoder and LLM frozen, enabling cost-effective backdoor insertion and evaluation on image captioning and VQA.Experiments assess text quality, attack execution, visual-textual interactions, and ablations across attack factors.
  • TrojVLM: TrojVLM combines language modeling loss with semantic preservation loss to retain natural linguistic relationships and semantic fidelity during poisoned fine-tuning.Semantic preservation operates at the embedding level by encouraging predicted token embeddings to resemble ground-truth token embeddings through cosine similarity.
  • Problem Definition: The attacker trains on mixed clean and poisoned data so clean images produce faithful outputs, whereas triggered images produce coherent outputs containing the target text.Poisoned images receive pixel-pattern triggers, while prompts remain unchanged and target text is inserted into ground-truth outputs.
  • TrojVLM: Relying only on language modeling loss can make the model neglect image semantics, generate incorrect or nonsensical text, or repeat the target text.The reported failure examples include the phrase “eating a spoon” and repeated target text.
  • Attack Efficiency: The method maintains high text generation quality under both clean images and poisoned images and remains effective across different VLM architectures.The architecture comparison explicitly includes InstructBLIP.
  • Interaction between Visual and Textual Information: Backdoored models focus on the upper-left image trigger while retaining attention to relevant visual information, and projection tokens isolate distinct image elements.Token 8 attends broadly near the trigger, whereas tokens 14, 23, and 29 concentrate on three plates.
  • Ablation Study: A 20 × 20 pixels trigger occupies less than 0.8% of the image area, while larger triggers improve attack performance and smaller triggers produce lower ASRs.The ablation identifies trigger size as a determinant of attack performance and VLM vulnerability.

Ethics Statement

The study aims to improve security knowledge by examining backdoor vulnerabilities in vision-language models (VLMs), without conducting activities that could harm people, groups, or digital systems. It argues that deeper understanding of these vulnerabilities can support more secure systems and stronger protections against threats.

  • Research objective: The study focuses on improving security knowledge about backdoor attack vulnerabilities in vision-language models.Its primary objective is to understand VLM vulnerability in depth.
  • Research conduct: The research conducts no activities that could potentially harm individuals, groups, or digital systems.The ethics statement explicitly rules out harmful activities as part of the study.
  • Anticipated benefit: Understanding VLM backdoor vulnerabilities is intended to enable more secure systems and better protections against potential threats.The stated anticipated benefit is improved system security and threat protection.

Limitations

TrojVLM’s evaluation is limited to three vision-language model architectures, leaving broader architectural vulnerabilities unexplored. Future work should extend the attack to additional VLM frameworks and develop effective defenses.

  • Architectural scope: TrojVLM experiments cover only BLIP-2, MiniGPT-4, and InstructBLIP, excluding architectures such as LLaVA.The authors identify broader architecture coverage as future work for more fully exploring VLM vulnerabilities.
  • Backdoor behavior: The backdoor remains activated when only 1% of image tokens containing triggers are retained, indicating strong dependence on trigger-associated visual information.This behavior shows that substantial visual-information reduction does not prevent generation of the intended target text.
  • Robustness and defenses: TrojVLM remains robust across tested trigger locations, trigger sizes, and poison rates, but the experiments do not establish an effective defense method.The authors explicitly identify defense development as essential future work.

4.2 Attack with Out-Of-Distribution Data (VLOOD)

VLOOD is a backdoor attack for VLM image-to-text generation that uses OOD training data while preserving clean behavior and conceptual consistency. Its CKP, CCP, and dynamically adjusted weights improve the balance between attack success and semantic quality across captioning and VQA tasks.

  • Method and Motivation: Existing VLM attacks can alter poisoned outputs but often produce incoherent sentences with semantics irrelevant to the input images.This motivates preserving output meaning rather than merely changing model behavior on poisoned inputs.
  • Method and Motivation: VLOOD combines Clean Knowledge Preservation, Conceptual Consistency Preservation, and dynamically adjusted weights to retain clean behavior, preserve poisoned-output semantics, and balance competing losses.CKP uses knowledge distillation, while CCP aligns poisoned outputs with input-image semantics.
  • Method and Motivation: VLOOD is presented as the first practical VLM backdoor attack using Out-Of-Distribution training data.It targets complex image-to-text generation tasks while minimizing semantic degradation.
  • Experimental Results: Experiments show that VLOOD improves attack efficiency across three image-captioning datasets and achieves strong ASRs with better quality-related metrics than baseline attacks.On VQA, VLOOD improves conceptual consistency under clean and poisoned inputs while maintaining a strong ASR; ChatGPT evaluations align with traditional metric trends.
  • Ablation Studies: Replacing CKP produces ASR 0.983 on clean inputs, while replacing CCP reduces CIDEr from 115.0 to 103.7 under clean inputs.Replacing both losses yields ASR 0.985 on clean inputs and reduces CIDEr from 115.0 to 109.7.
  • Ablation Studies: Adding CKP alone entirely eliminates ASR under poisoned samples, whereas adding CKP, CCP, and dynamic weights improves poisoned-input ASR while maintaining clean-sample performance.Without dynamic weights, CKP remains dominant and ASR is 0 for poisoned inputs.

Efficient Multimodal Representation Learning · 5.1 Token Compression for Pathology WSI VQA (TCP-LLaVA)

TCP-LLaVA addresses gigapixel WSI VQA by compressing visual and textual tokens into fixed-length representations before LLM processing. It reduces computational costs while achieving state-of-the-art accuracy across ten TCGA tumor types.

  • Introduction: Gigapixel WSIs create extreme sequence-length and computational challenges, while existing patch-level, classification-oriented, and brute-force MLLM methods miss global context or process over 10,000 tokens.These limitations motivate efficient compression that preserves diagnostic information before LLM processing.
  • Introduction: TCP-LLaVA introduces the first token-compression-based multimodal LLM for WSI VQA, using trainable compression tokens and a lightweight modality compression module.The architecture is designed specifically for efficient whole-slide image question answering.
  • Related Work: Prior WSI approaches use MIL for classification, encoder-decoder models for captions, or direct token forwarding, limiting descriptive reasoning, global context, or computational efficiency.TCP-LLaVA instead targets end-to-end VQA over full-resolution slides.
  • Token Compression Pathology LLaVA: TCP-LLaVA extracts patch and text tokens, fuses them with trainable compression tokens through multi-head attention, and forwards only the compressed representation to the LLM.The compression tokens aggregate cross-modal information analogously to BERT’s [CLS] token.
  • Experiments: 100 compression tokens reduce TCP-LLaVA’s input token length by over 99% compared with SlideChat’s potentially over 10,000 visual tokens per WSI.This reduction improves GPU utilization and lowers training and inference costs.
  • Experiments: 78.57% average accuracy makes TCP-LLaVA the best method across ten TCGA tumor types, exceeding SlideChat’s 77.23% average accuracy.The largest improvement is +5.00 pp on GBM, with additional gains on BLCA, COAD, LUAD, READ, and SKCM.
  • Conclusion: 10.87 TFLOPS versus 2.35 TFLOPS for SlideChat corresponds to over 4× higher computational throughput while maintaining strong diagnostic performance.The framework also demonstrates higher sample-level throughput during training and inference.
  • Conclusion: TCP-LLaVA currently focuses on question answering, with future work targeting open-ended pathology report generation requiring high-quality, fine-grained supervision.The framework otherwise supports end-to-end training and scaling to full-resolution pathology slides on standard hardware.

Appendix … Results

The paper combines modality compression for efficient vision-language learning with a multimodal transformer for clinical mortality prediction. Across experiments, the methods compress visual information, fuse structured and unstructured EHR data, and achieve strong mortality-prediction performance with interpretable clinical features.

  • Appendix: TCP-LLaVA evaluates computational efficiency using average TFLOPS and throughput across variable-length inputs for fair model comparison.TFLOPS measures GPU floating-point operation rate, while throughput measures WSI and QA samples processed per second.
  • Appendix: The modality compression module distills diverse visual features into compact compression tokens positioned near text representations and far from original visual tokens.The t-SNE pattern indicates that compression is guided by the text prompt and produces question-tailored visual summaries while reducing visual-token length.
  • Introduction: The approach addresses the difficulty of combining structured clinical variables with complementary contextual information in unstructured clinical notes, where naive feature concatenation can worsen performance.The method uses separate Notes and Time Series encoders plus a Multimodal Encoder rather than simply concatenating the modalities.
  • Introduction: The proposed multimodal transformer fuses clinical-variable time series with clinical-note text by projecting both modalities into a shared space and jointly learning their representations across time.Clinical notes provide auxiliary information for adjusting the two modalities into a sharable space across different times.
  • Methodology: The MIMIC-III study models in-hospital mortality from the first 48 ICU hours using 17 preprocessed clinical variables, clinical notes, Clinical BERT, and a transformer architecture.The dataset treats each patient visit as a sample, embeds the 17 variables into a 76-dimension time-series representation, and uses MBERT for clinical-note embeddings.
  • Methodology: The transformer uses attention to incorporate information from earlier and later time points, while the implementation applies ReLU projections, sigmoid output, cross-entropy loss, L2 regularization, and Adam optimization.The experiments were implemented with PyTorch and HuggingFace Transformers on an NVIDIA RTX A5000 with 24GB RAM.
  • Results: 0.538 AUCPR, 0.877 AUCROC, and 0.490 F1 were achieved by the proposed models, which outperformed other methods on in-hospital mortality classification.The evaluation reports AUCROC, AUCPR, and F1 over five runs with different initializations, using mean and standard deviation.
  • Results: Domain-adaptive pretraining and task-adaptive fine-tuning improved clinical-note prediction, while Integrated Gradients and Shapley values identified clinically meaningful words and disease-severity-related variables.The general-purpose BERT performed worst and MBERT best; important words included indicators related to respiratory disease severity and mental status, with negation affecting interpretation.

Conclusion … Collaborative Research Summary

The dissertation combines backdoor attack and defense strategies for language and vision-language models with efficient, interpretable multimodal learning for medical and commercial applications. It also summarizes collaborative research contributions and identifies concept-based backdoor attacks as a future direction.

  • Conclusion: A Multimodal Transformer fuses clinical notes, clinical variables, and time-series information, outperforming other methods.The model integrates multimodal clinical data through a transformer block while considering temporal information.
  • 6.1 Conclusion: The dissertation develops backdoor detection and attack methods for language models, including AttenTD, TABDet, TAL, and the clean-label BadCLM case study.AttenTD detects abnormal attention focus, TABDet analyzes final-layer logits across tasks, TAL shapes attention during training, and BadCLM inserts harmful behaviors while preserving clean-input performance.
  • 6.1 Conclusion: For vision-language models, TrojVLM inserts target phrases while preserving image-to-text semantics, whereas VLOOD injects backdoors using OOD data without original training datasets.Together, these studies expose multimodal vulnerabilities and provide mechanisms for exploiting and detecting threats.
  • Efficient Multimodal Representation Learning.: TCP-LLaVA compresses high-resolution pathology visual tokens to reduce computational cost without sacrificing diagnostic utility, while the EHR Multimodal Transformer fuses structured data and clinical notes.These methods target scalable, interpretable multimodal learning for gigapixel images and clinical decision-making.
  • Collaborative Research Summary: Collaborative research addressed backdoor learning, prompt editing, LLM security, distribution-shift calibration, agent evaluation, sentence implicitness, multimodal learning, and clinical applications.Specific contributions include instruction-tuning vulnerability analysis, adversarial-prompt mitigation, geometry-based calibration, Agent-as-a-Judge and OPERA, ImpScore, multimodal taxonomies, and clinical transformer applications.
  • Collaborative Research Summary: ImpScore quantifies implicitness in text and supports applications including summarization, misinformation detection, and scientific tasks.The metric was developed as part of the collaborative research contributions on sentence implicitness.
  • 6.2 Future Work: Future work will investigate concept-based backdoor attacks in instruction-tuned vision-language models, using abstract visual concepts rather than single objects or words as triggers.The planned direction aims to improve AI robustness, efficiency, and user-centered intelligence in healthcare and commercial domains.
Loading 2608.18095v1…