PaperlayerView exact revision on arXiv

Source-linked AI summary

TTSD-FAR: Test-Time Self-Distillation with Fisher-Anchored Restoration for Missing-Modality Emotion Recognition in LVLMs

Muhammad Haseeb Aslam, Alessandro Koerich, Marco Pedersoli, Ali Etemad, Eric Granger

arXiv:2608.18386v1cs.CVcs.AI

TL;DR

Missing modalities create a difficult test-time adaptation problem for autoregressive LVLMs because existing confidence- and retrieval-based methods provide unreliable supervision. TTSD-FAR aligns a masked-input student with a frozen complete-modality teacher and uses Fisher-based consolidation and reactivation, with experiments across three video emotion-recognition datasets showing effective adaptation across missing-modality conditions.

  • Problem

    Missing modalities at test time induce substantial distribution shifts in LVLM representations, while retrieval and entropy-based adaptation provide unreliable supervision under partial observation.

  • Method

    TTSD-FAR aligns a masked-input student with a frozen complete-modality teacher through feature-level self-distillation and uses Fisher geometry to consolidate or reactivate adaptation.

  • Results

    Experiments across MELD, DFEW, and BAH with multiple missing ratios and modalities show TTSD-FAR’s effectiveness for adapting autoregressive LVLMs under partial observation.

  • Takeaways & Limitations

    Fisher-guided consolidation enables adaptation to missing modalities while limiting continued-update drift during extended test-time deployment.

  • Takeaways & Limitations

    TTSD-FAR improves adaptation performance at the cost of additional computation during inference.

Abstract

from arXiv · show

Large video-language models (LVLMs) have shown remarkable performance on multimodal tasks like multimodal emotion recognition (ER) in the wild. ER is inherently multimodal, requiring a joint understanding of facial expressions, vocalizations, language, biosignals, and gestures. However, real-world deployment remains challenging: modalities may be missing or noisy at test time. Partial observations can be viewed as a distribution shift relative to the complete-modality distribution. SOTA TTA methods based on entropy minimization or perplexity reduction do not transfer to autoregressive LVLMs, while retrieval augmented generation (RAG) degrades when the observed modality is weak. Because no ground-truth supervision exists to verify individual updates, adaptation across this stream risks accumulating drift and degrading once the model departs from a reliable solution. An effective solution must therefore adapt to arbitrary missing-modality patterns and remain effective during continual adaptation. We address both jointly with Test-Time Self-Distillation (TTSD), a parameter-efficient framework in which a frozen teacher, trained on complete modalities, guides an adaptive low-rank student via self-distillation, updating only a negligible number of parameters. Stability is built into this same loop through Fisher-Anchored Restoration (FAR), which monitors Fisher information stability to detect convergence versus drift and restores the student toward the teacher's anchor when distributional shifts are identified. Our experiments on MELD, DFEW, and BAH under 0%-50% missing modalities show that this unified adaptation-restoration design consistently outperforms entropy-based adaptation, RAG, and perplexity-based generation over long adaptation horizons, where baselines without restoration progressively degrade while TTSD-FAR remains consistent.

1 Introduction

Missing modalities create substantial distribution shift and performance degradation for LVLM-based emotion recognition, while retraining or fully fine-tuning these models is impractical. TTSD-FAR addresses this with parameter-efficient self-distillation and Fisher-based stopping, drift detection, and restoration for continual test-time adaptation.

  • Motivation: Missing modalities are ubiquitous in practical video-based multimodal emotion recognition because audio, transcription, and sensor inputs can fail.Examples include noisy-environment audio corruption, transcription failure, privacy constraints, and sensor malfunctions that eliminate entire modalities.
  • Motivation: Retraining or fully fine-tuning LLMs and LVLMs for each deployment setting is impractical because of catastrophic forgetting, computational cost, specialized hardware, and lengthy optimization.This motivates lightweight test-time adaptation that customizes models without modifying the full parameter set.
  • Motivation: 0.8350 MMD and 0.6342 cosine similarity quantify the internal distribution shift between complete and missing-modality inputs on MELD with 50% text missing.The analysis uses Video-LLaVA-7B and indicates that modality absence disturbs the learned attention manifold despite unchanged semantic content.
  • Method: TTSD provides parameter-efficient test-time adaptation for autoregressive LVLMs under missing modalities using conditional latent representation alignment as supervision.The contribution is designed to overcome failure modes associated with entropy-based objectives and retrieval methods.
  • Method: FAR monitors the Fisher information diagonal to stop updates after student convergence and reactivate adaptation when test-stream distribution shifts.Its two-state mechanism transitions active → anchored at convergence and anchored → active after Fisher Mismatch Index detection of distributional shift.

2 Related Works

Prior missing-modality methods mainly rely on training-time reconstruction, modality simulation, retraining, or specialized architectures, limiting transfer to expensive-to-retrain LVLMs. This motivates feature-level test-time adaptation with dual LoRA adapters and self-distillation, addressing weak supervision, modality imbalance, and retrieval failures.

  • Learning with Missing Modalities: Training-time methods such as ModDrop, SMIL, and MMP improve missing-modality robustness through reconstruction, modality-absence simulation, or architectural changes.These strategies are difficult to transfer directly to LVLMs because retraining is expensive and generative decoders lack clear self-supervised objectives under modality absence.
  • Learning with Missing Modalities: Missing a stronger modality can cause substantial performance drops, motivating preservation of knowledge from complete multimodal observations when weaker modalities remain.Multimodal systems often contain imbalanced modality strengths, with some modalities carrying stronger task-relevant signals than others.
  • Test-Time Adaptation: Existing missing-modality approaches require retraining for modality combinations or specialized architectures to hallucinate absent modalities.These limitations constrain adaptation to arbitrary missing-modality patterns in LVLM settings.
  • Test-Time Adaptation: LVLM test-time adaptation lacks strong supervision because entropy objectives are uninformative and retrieval-augmented generation fails when observed modalities are too weak for correct neighbor retrieval.The paper addresses this gap with feature-level adaptation using dual LoRA adapters through self-distillation.

3 Proposed Methodology

TTSD-FAR adapts a low-rank student to missing modalities by distilling frozen teacher representations from complete inputs, while Fisher-based monitoring controls continued adaptation and restores stability under drift.

  • Missing-Modality Formulation: Binary modality masks define partially observed inputs by selectively removing video or text from complete multimodal samples.The observed input is ˜x = (mv · xv, mt · xt), with mv and mt indicating modality presence.
  • Test-Time Self-Distillation: Feature-level self-distillation trains the student to approximate the teacher’s complete-modality representation conditioned on the observed input.Under squared error, the optimal student representation is hstu∗(˜x) = E[htea(x) | ˜x].
  • Fisher-Anchored Restoration: Fisher-Anchored Restoration addresses continual-adaptation failure by suspending updates after convergence and protecting consolidated knowledge during distribution shifts.Continued updates can add noise after convergence or overwrite previously learned representations under non-stationary streams.
  • Test-Time Self-Distillation: The student matches intermediate teacher representations rather than final decisions, preserving cross-modal feature geometry and mitigating mode collapse.Teacher features are detached, and teacher and student forward passes are isolated to prevent gradient contamination.
  • Fisher-Anchored Restoration: The consolidation index detects convergence as the Fisher diagonal stabilizes, while Fisher mismatch triggers a return from the anchored state to active adaptation.CI_t → 0 indicates a stable importance landscape; anchored-to-active transition occurs when FMI_t > τ_fmi.

4 Results and Discussion

Experiments on MELD, DFEW, and BAH show that TTSD-FAR robustly adapts to missing modalities where entropy-based, retrieval-based, and perplexity-based methods degrade. Its feature-level self-distillation and Fisher-anchored restoration address semantic recovery and long-horizon adaptation, with parameter-efficient updates.

  • Benchmark evaluation: TTSD-FAR is evaluated on MELD, DFEW, and BAH, which span multimodal diversity, temporal dynamics, and subtle affective cues.Tables 1–3 report performance under progressively increasing missing-modality ratios.
  • Baseline comparisons: Entropy-based TTA consistently underperforms across all three datasets, while RAG and perplexity-based generation degrade when the dominant observed modality is unavailable.Confidence-based objectives provide unreliable supervision for autoregressive LVLMs under severe modality shift.
  • Baseline comparisons: On MELD, removing text lowers F1 from 0.6149 to 0.4785, compared with 0.6149 to 0.5775 when vision is removed, identifying text as the dominant conversational supervision signal.Retrieval-based methods struggle without text, whereas entropy-based methods approach random performance.
  • Computational cost: Only 44M of Video-LLaVA 7B’s parameters are updated, corresponding to approximately 0.629% of the model.The full-modality case requires teacher and student forward passes, while the missing-modality case uses the adapted student LoRA without computational overhead.
  • Fisher-anchored restoration: FAR improves TTSD consistently across MELD and BAH at every missing rate by stopping and restarting adaptation, demonstrating task-independent stabilization under non-stationary streams.The comparison isolates TTSD with and without restoration; the governed variant outperforms the unbounded variant.
  • Adaptation mechanism: TTSD-FAR treats missing-modality adaptation as conditional representation alignment, using feature-level supervision from a frozen complete-modality teacher to recover unavailable semantic information.This avoids relying on confidence or retrieval optimization when the observed modality is weak or misleading.

5 Conclusion … B.1 Theoretical Properties of FAR

TTSD-FAR adapts autoregressive LVLMs to missing modalities through frozen-teacher self-distillation and Fisher-anchored restoration. The appendix formalizes its adaptation loop and provides theoretical justification for limiting drift while enabling re-adaptation under distribution shift.

  • 5 Conclusion: TTSD-FAR is a parameter-efficient framework for adapting large video-language models to missing modalities during inference.It addresses representation shifts induced by missing modalities in autoregressive LVLMs.
  • 5 Conclusion: The method couples a frozen teacher trained on complete modalities with an adaptive student adapter operating on masked inputs.This design replaces unreliable supervision from entropy-based adaptation and retrieval-augmented generation.
  • A Appendix: The appendix includes the TTSD-FAR adaptation algorithm and additional results, ablations, and sensitivity analyses.Listed analyses include teacher choice, weaker-modality missing results, LoRA rank, small-model effectiveness, FAR hyperparameters, state dynamics, and stopping comparisons.
  • B Algorithm for TTSD-FAR: FAR’s algorithm maintains active and anchored states while updating the student and tracking Fisher statistics over incoming samples.The loop computes a Fisher-change criterion, anchors parameters after sustained stability, and performs frozen-anchor inference in the anchored state.
  • Effect of the FMI Threshold: The FMI threshold controls restoration by triggering reactivation when the Fisher-based missingness indicator exceeds τ_fmi.After reactivation, the algorithm restores the student and Fisher state to the anchor and clears the adaptation buffer.
  • B.1 Theoretical Properties of FAR: FAR’s theoretical justification explains how its two-state strategy avoids unnecessary parameter drift while remaining capable of re-adapting under distribution shift.The section explicitly targets intuitive justification rather than a proof of global optimality.

Assumptions

The analysis assumes locally smooth distillation loss, bounded stochastic-gradient variance, and a piecewise-stationary test stream. Under these assumptions, Fisher stability supports consolidation during stationary periods, while Fisher mismatch detects genuine distribution shifts and enables reactivation without sacrificing adaptability.

  • Assumptions: The framework assumes an L-smooth distillation objective, bounded stochastic-gradient variance, and a test stream that is stationary between distribution shifts.These assumptions are presented as standard in stochastic optimization and continual test-time adaptation.
  • Fisher stability: Fisher stability over K consecutive steps indicates that the gradient distribution and its second moment have stopped changing, but does not imply ∇L(ϕt) ≈ 0.Stable Fisher estimates can arise from either noise-dominated updates or residual useful progress.
  • Fisher stability: During stationary periods, continued updates can trade bounded progress for unboundedly growing variance, motivating consolidation that reduces unnecessary drift.This holds whether the remaining updates are primarily noise or bounded residual-gradient progress.
  • Shift detection and reactivation: After a distribution shift, FAR exits the anchored state and resumes standard TTSD updates after a finite detection delay.FAR keeps parameters fixed while gradients remain aligned with the consolidated Fisher geometry, then follows the perpetual-adaptation trajectory after reactivation.
  • Shift detection and reactivation: The FMI threshold selectively detects structural distribution shifts rather than gradient magnitude alone, because shifted gradients concentrate in coordinates with low anchor Fisher importance.Hard within-distribution samples can have elevated loss while leaving FMI near 1; as flow → 0, any finite τfmi eventually detects a shift.

C Additional Results · Distilling from Base Teacher (no teacher LoRA).

The appendix evaluates TTSD when no full or parameter-efficient retraining is performed on the base model, using teacher representations extracted directly from frozen base-model weights. This setup applies the same distillation mechanism and follows a similar trend to Teacher LoRA-based teacher embeddings.

  • Distilling from Base Teacher (no teacher LoRA).: The experiment targets deployment settings where the base model receives no full or parameter-efficient retraining.This constraint motivates the base-teacher evaluation.
  • Distilling from Base Teacher (no teacher LoRA).: Distillation is performed directly from the base model weights without teacher LoRA.The experiment is shown in Fig. 5.
  • Distilling from Base Teacher (no teacher LoRA).: Teacher representations are extracted from the frozen base model in this setting.The frozen base model supplies the teacher representations.
  • Distilling from Base Teacher (no teacher LoRA).: The resulting optimization problem is the same as Eq. 11.Using base-model weights preserves the stated optimization formulation.
  • Distilling from Base Teacher (no teacher LoRA).: Table 4 reports results for teacher embeddings obtained directly from base-model weights.The table evaluates the no-teacher-LoRA configuration.
  • Distilling from Base Teacher (no teacher LoRA).: The setup differs from Teacher LoRA distillation only in how teacher representations are extracted.The same distillation mechanism is applied in both cases.
  • Distilling from Base Teacher (no teacher LoRA).: Results with base-model teacher embeddings follow a similar trend to results using the Teacher LoRA objective.The passage attributes this similarity to the otherwise identical setup.

Weaker-Modality Missing Results · DFEW - Text Missing at Test-Time - Weaker Modality Missing

On DFEW, progressively missing textual input is evaluated using F1, with relatively little degradation because the dataset is already weakly dependent on text. The full-modality F1 is 0.5549, while many clips lack English transcriptions or speaker utterances.

  • DFEW - Text Missing at Test-Time - Weaker Modality Missing: DFEW evaluates methods under progressively increasing proportions of missing textual input using F1.The comparison is defined in Table 5 for test-time textual-modality removal.
  • DFEW - Text Missing at Test-Time - Weaker Modality Missing: The performance drop from missing text is insignificant compared with the drop from missing vision on DFEW.The passage attributes this difference primarily to the dataset structure.
  • DFEW - Text Missing at Test-Time - Weaker Modality Missing: 16372 short clips comprise the DFEW dataset.This dataset size is given as part of the explanation for weak reliance on text.
  • DFEW - Text Missing at Test-Time - Weaker Modality Missing: 6196 DFEW clips have non-English transcriptions.Non-English transcriptions reduce the usefulness of textual input for the dataset’s full-modality performance.
  • DFEW - Text Missing at Test-Time - Weaker Modality Missing: 4960 DFEW clips have no speaker utterance at all.The absence of speaker utterances further limits the dataset’s dependence on text.
  • DFEW - Text Missing at Test-Time - Weaker Modality Missing: 0.5549 is the reported full-modality performance on DFEW.The passage uses this value to show that full-modality performance is already less reliant on text.
  • DFEW - Text Missing at Test-Time - Weaker Modality Missing: The passage identifies TENT (Wang et al. 2021) as a classical test-time adaptation method considered in this setting.The provided text introduces TENT before continuing to discuss classical adaptation methods.

BAH - Vision Missing at Test-Time- Weaker Modality Missing

On BAH, missing vision causes negligible performance loss even at 50%, with no effect at 10%. TTSD improves over existing TTA methods, while RAG benefits only when text remains the stronger observed modality.

  • Vision Missing at Test-Time: 50% missing vision causes only a negligible performance drop on BAH, while 10% missing vision has no effect.Table 6 reports Avg-F1 as visual input becomes progressively unavailable.
  • Vision Missing at Test-Time: TTSD improves over existing TTA methods under both weaker and stronger visual-missing conditions.The comparison is reported for BAH under progressively missing visual modality.
  • Modality Strength: Text is critical for Ambivalence detection, making text-missing conditions more damaging than vision-missing conditions on BAH.The stronger impact of text missing is attributed to the importance of textual input for Ambivalence detection.
  • Modality Strength: RAG improves when vision is missing because the observed text modality is stronger and can retrieve relevant information effectively.When vision is observed instead, it is not discriminant enough for effective retrieval.

D Additional Ablations: … D.1 FAR Hyperparameters Sensitivity

Additional ablations show that TTSD’s robustness depends on adaptation capacity but generalizes across LVLM sizes. FAR sensitivity is examined through reactivation behavior across three FMI thresholds.

  • LoRA Rank Sensitivity.: TTSD performance degrades monotonically as missing-modality rates increase across all tested LoRA ranks.The ablation evaluates robustness to modality-incomplete inputs under varying missing rates.
  • LoRA Rank Sensitivity.: Lower-rank adaptation at r=4 suffers disproportionately under higher missing rates, with steeper and less stable declines than higher-rank adaptation.The results suggest insufficient parameter capacity limits compensation for missing modalities.
  • LoRA Rank Sensitivity.: Figure 6 evaluates LoRA-rank effects on distillation performance across varying missing-modality rates and three datasets.The figure summarizes the rank-sensitivity experiment across datasets.
  • Effectivity of TTSD on Small Video Language Model (SVLM):: TTSD is not restricted to a specific LVLM architecture and improves robustness for both large and lightweight video-language models.The comparison includes Video-LLaVA-7B and MobileVideoGPT-0.5B.
  • Effectivity of TTSD on Small Video Language Model (SVLM):: TTSD consistently improves performance over the non-adapted baseline across all missing-modality conditions on the smaller MobileVideoGPT-0.5B model.Its overall performance remains lower because of reduced model capacity.
  • D.1 FAR Hyperparameters Sensitivity: Figure 7 examines how τfmi affects reactivation events using FMI traces at τfmi 1.0, 2.0, and 3.0.The three panels correspond to the listed thresholds from top to bottom.

FAR State Dynamics over the Test Stream · D.2 Loss-Based Stopping vs. FAR

FAR dynamically alternates between ACTIVE and ANCHORED states, consolidating after stable Fisher evidence and reactivating when later distributional shifts are detected. Compared with loss-based stopping, FAR is presented as a more effective restoration strategy under progressively missing text, while plain TTSD adapts without stopping.

  • FAR State Dynamics over the Test Stream: The reported FAR regime uses βF = 0.99, τfc = 0.02, K = 5, and τfmi = 2.0.These values correspond to the middle state-trajectory panel discussed in the paper and supplementary material.
  • FAR State Dynamics over the Test Stream: FAR keeps the student ACTIVE initially, consolidates after the Consolidation Index remains below τfc for K consecutive samples, and later reactivates when FMI exceeds τfmi.Under the reported settings, consolidation occurs after roughly 40% of the stream, followed by reactivation around its midpoint and a later cycle.
  • FAR State Dynamics over the Test Stream: The state trajectories are summarized as a timeline of FAR’s ACTIVE and ANCHORED states across the test stream.The figure provides the visual overview corresponding to the regime-specific descriptions.
  • FAR State Dynamics over the Test Stream: With τfmi = 5, FAR consolidates slightly past the stream midpoint and remains permanently frozen because later shifts do not cross the higher reactivation threshold.The top trajectory panel illustrates consolidation without subsequent reactivation.
  • FAR State Dynamics over the Test Stream: With τfmi = 0.5 and a relaxed consolidation requirement, FAR consolidates as soon as CI drops below τfc and uses a lower threshold for reactivation.The supplied passage describes the bottom-panel regime but is truncated before its full trajectory is reported.
  • D.2 Loss-Based Stopping vs. FAR: Table 8 compares TTSD-FAR with plain TTSD and TTSD-LS on BAH under progressively missing textual input using WF1.Plain TTSD adapts on every complete-modality sample, whereas TTSD-LS stops after raw distillation loss plateaus with patience 5.
  • D.2 Loss-Based Stopping vs. FAR: TTSD-LS underperforms both TTSD-FAR and unbounded TTSD as missing text increases.The supplied passage states this comparative trend but truncates the continuation containing the detailed values.

E Datasets and Implementation Details … FAR Parameters:

The paper evaluates TTSD-FAR on multimodal and facial-expression datasets using controlled text- and vision-missing simulations. Implementation adapts only student LoRA parameters, while FAR uses Fisher-based thresholds and consecutive-sample gating to control consolidation and reactivation.

  • Datasets: MELD contains approximately 1,433 multi-party dialogues and over 13,000 utterances with synchronized audio, visual, and textual modalities and seven emotion labels.The labels are Anger, Disgust, Sadness, Joy, Neutral, Surprise, and Fear.
  • Datasets: DFEW comprises over 16,000 short video clips from more than 1,500 movies, depicting naturalistic facial expressions under unconstrained conditions.
  • Missing modality Simulation:: Missing-modality validation uses datasets with utterance-level annotations, replacing missing text with an empty string and missing vision with zero inputs.The section defines the experimentation scope for both text- and vision-missing conditions.
  • Implementation Details: Video-LLaVA-7B receives video and utterance inputs in instruction format and classifies emotions from seven candidate labels.
  • Implementation Details: Student LoRA adaptation uses Adam with learning rate 10^-4, batch size 1, gradient accumulation over 8 steps, r = 8, α = 16, and dropout 0.3.All experiments use float16 precision and report results on the official test set.
  • FAR Parameters:: FAR’s βF, τfc, K, and τfmi parameters control the ACTIVE/ANCHORED state machine through Fisher-diagonal decay, consolidation gating, and reactivation.τfc and K require CI to remain below τfc for K consecutive adapted samples before freezing the student, while τfmi triggers reactivation when FMI exceeds its threshold.
  • FAR Parameters:: M and ε affect reporting and numerical stability only, rather than FAR’s transition logic.
Loading 2608.18386v1…