Codebook-Guided Cross-Modal Knowledge Distillation for Structurally Heterogeneous Features

• CMKD uses an auxiliary teacher modality during training when sensing, deployment cost, privacy, or missing data may require single-modality inference.Source: §1 · p. 1
• Existing feature-level CMKD methods assume structural compatibility that permits alignment through projections or shared embedding spaces.Source: §1 · p. 1
• Spatial visual grids and temporal audio or text sequences can encode the same cue differently, making direct feature-unit matching potentially misleading.Source: §1 · p. 2
• A teacher-side vector-quantized codebook supplies discrete anchors, selected for task relevance and student compatibility, to guide student learning without explicit unit correspondence.Source: §1 · p. 2
• Across audio-visual, image-text, and RGB-depth classification and segmentation, the framework effectively leverages intermediate teacher knowledge despite structurally different feature organizations.Source: §1 · p. 2
• The framework is evaluated against prediction-level, feature-level, and modality-gap-aware distillation baselines across diverse cross-modal benchmarks and tasks.Source: §1 · p. 2

2.1 Cross-modal Knowledge Distillation

• CMKD methods address modality gaps through selective prediction transfer and, in newer approaches, structural, temporal, or frequency-domain feature cues.Source: §2.1 · p. 2
• Many CMKD methods still assume teacher and student features occupy structurally comparable spaces, leaving heterogeneous intermediate features without direct unit correspondence challenging to distill.Source: §2.1 · p. 3

2.2 Codebook Learning and Vector Quantization

• Vector quantization maps continuous features to learned codewords, creating a discrete representation that can capture recurring intermediate-feature patterns.Source: §2.2 · p. 3
• Unlike prior codebook work on multimodal representation learning, this framework uses teacher-side codes as distillation anchors selected for task relevance and student compatibility.Source: §2.2 · p. 3

3.1 Notations and Problem Statement

• The setting uses paired teacher and student inputs with shared targets, while their intermediate representations may differ in structure and size, such as spatial locations versus temporal positions.Source: §3.1 · p. 4

3.2 Overall Framework

• The framework learns a teacher-side vector-quantized codebook, estimates code importance using task relevance and student compatibility, then distills through selected codes and prediction alignment.Source: §3.2 · p. 4

3.3 Stage-1: Teacher-Side Codebook Learning

• With the pretrained teacher encoder fixed, a learned codebook branch quantizes projected teacher features and is trained with task and vector-quantization losses.Source: §3.3 · pp. 4–5

3.4 Stage-2: Code-wise Importance Estimation

• Code importance combines how much code removal degrades teacher task predictions with how much it increases discrepancy from student predictions; the scores rank codes for selection.Source: §3.4 · p. 5

3.5 Stage-3: Selective Codebook-Based Distillation

• The student quantizes its features using selected teacher codes and learns with task and commitment losses plus codebook-based representation and prediction guidance.Source: §3.5 · pp. 5–6

4 Experiments

• Experiments span five multimodal benchmarks across classification and segmentation, using 6:2:2 splits and comparing general, modality-gap-aware, and adapted feature-alignment baselines.Image and video inputs use 2D features, audio and text use sequences, and classification selects the top 50% of teacher codes.Source: §4 · p. 6, §4.1 · pp. 6–7

4.2 Semantic Segmentation Experiments

• Our method leads on most segmentation metrics; only RGB→Depth OA trails w/o KD slightly, while CA, mIoU, and several class recalls improve.Recall gains include door, picture, and other_structure; the floor decrease is one example that can affect frequency-weighted OA.Source: Table 2 · p. 7, §4.2 · pp. 7–8

4.1 Classification Experiments

• The method is best in seven of eight classification settings and second-best on CrisisMMD T→I, while improving over the student without distillation in every transfer direction.Modality-gap-aware baselines are generally competitive, but the proposed method is stronger in most settings.Source: §4.1 · p. 7

4.3 Analysis

• High-importance codes are mainly activated around central face and mouth regions, while low-importance codes show more scattered or peripheral usage.The analysis visualizes spatial usage maps for top-10 and bottom-10 codes on RAVDESS.Source: Figure 2 · p. 8, §4.3 · p. 8
• Combining Code and Pred-Match yields the strongest overall performance, although either component improves several transfer directions.Code provides representation-level guidance; Pred-Match guides task predictions using predictions derived from quantized representations.Source: Table 3 · p. 8, §4.3 · pp. 8–9
• Matching the codebook-induced teacher prediction q(t) outperforms matching the original teacher prediction p(t) across all evaluated datasets and transfer directions.This indicates the gain is not from generic output-level supervision alone, but from guidance shaped by the learned teacher codebook.Source: Table 4 · p. 8, §4.3 · p. 9
• Score-based criteria generally improve on random selection, and combining task relevance with student compatibility is best or competitive in most cases.Task relevance and student compatibility show different strengths across scenarios when used individually.Source: Table 5 · p. 9, §4.3 · p. 9
• Across 25%–100% code-selection ratios, performance generally exceeds the w/o KD baseline in most cases, but the best ratio varies across settings.The main classification experiments use a fixed 50% ratio for a consistent evaluation protocol.Source: Figure 3 · p. 9, §4.3 · p. 9

5 Conclusion

• Teacher representations become a vector-quantized codebook, and task-relevant, student-compatible codes guide learning without explicit unit-level correspondence.The selected codes are leveraged through student-side quantization and codebook-induced prediction matching.Source: §5 · p. 9
• Across cross-modal classification and semantic segmentation, the method achieves strong performance against general and modality-gap-aware distillation baselines.The conclusion describes codebook-guided distillation as a practical strategy for structurally heterogeneous features.Source: §5 · p. 10
• Fixed-ratio selection may miss scenario-dependent amounts of useful teacher information, while offline scoring can increase cost for larger codebooks.Future work will explore adaptive and efficient code-subset selection.Source: §5 · p. 10

A.1 Computational Resources

• Experiments used a single workstation with an AMD Ryzen 9 9900X CPU, an NVIDIA RTX PRO 6000 Blackwell GPU, and 64GB RAM.Unless otherwise specified, training runs used one GPU.Source: §A.1 · p. 13, Table 6 · p. 13

A.2 General Training Details

• Additional training stages for FitNets, MGDFR, and this method each run for 100 epochs; models otherwise use AdamW with β = (0.9, 0.999).The stages are FitNets hint regression, MGDFR co-training, and Stage-1 of the proposed method.Source: §A.2 · p. 13

A.3 Implementation Details for Distillation

• The method projects intermediate features into code space, assigns each to its nearest code, regularizes code usage in Stage-1, and retrains the student from scratch before Stage-3.Feature-alignment baselines use pooled features for structurally different forms, but dense corresponding features are aligned directly when available.Source: §A.3 · p. 13

A.4 RAVDESS

• RAVDESS uses 1,440 paired speech-audio and speech-video samples for eight-way emotion classification, with Wav2Vec2 sequence features and ResNet-18 spatial features.The speech subset is split 60%/20%/20%; 10 video frames are sampled per video and averaged into a spatial feature map.Source: §A.4 · pp. 13–14

A.5 VGG-Sound

• VGG-Sound uses 48,755 paired audio-video samples across 141 event classes, with VGGish temporal audio features and VGG19 spatial visual features.The data split is 60%/20%/20%, and modality-specific classifiers process the features.Source: §A.5 · p. 14

A.6 CrisisMMD

• CrisisMMD-V2 uses 16,058 paired image-text samples for eight-way humanitarian classification, with ResNet-34 image features and BERTweet token representations.The data split is 60%/20%/20%; text pooling averages only valid token positions.Source: §A.6 · pp. 14–15

A.7 MM-IMDB

• MM-IMDB is filtered to 5,778 paired poster-text samples in eight single-label classes, making results a customized proxy rather than directly comparable to the standard multi-label protocol.FLAVA supplies image and text features; plots fall back to movie titles when unavailable, and the split is stratified 60%/20%/20%.Source: §A.7 · pp. 15–16

A.8 NYU-Depth V2

• NYU-Depth V2 uses 1,449 paired RGB-depth samples with pixel-wise labels for semantic segmentation; both branches use ResNet-34 features and produce 480 × 640 logits.Dense feature-alignment baselines are applied directly because RGB and depth features are spatially aligned.Source: §A.8 · p. 16

B.1 Results with Standard Deviations

• Tables 8 and 9 report classification and semantic segmentation results with standard deviations over five independent runs.Their mean values are identical to those in the corresponding main-paper result tables.Source: §B.1 · p. 16
• Tables 10, 11, and 12 add standard deviations to the ablation results from Tables 3, 4, and 5.The ablation deviations use the same protocol as the main results.Source: §B.1 · p. 16
• The added standard deviations provide information about variability across random seeds without changing the reported means.The main and ablation results are each summarized over five runs.Source: §B.1 · p. 16

B.2 Statistical Significance Analysis

• Under the main varying-split protocol, standard deviations include variation from data splits and teacher checkpoints as well as the distillation method.Within each seed, methods share the same split and teacher checkpoint.Source: §B.2 · pp. 16–17
• With data split and teacher checkpoint fixed, improvements over the strongest competitor are significant in five of eight directions; CrisisMMD T→I has a slightly higher MGDFR mean.CrisisMMD I→T and MM-IMDB T→I show near-threshold evidence, while the reliable-difference evidence for CrisisMMD T→I is limited.Source: §B.2 · p. 17
• The method has the best average rank in seven of eight transfer directions and the second-best rank in CrisisMMD T→I across five seeds.Ranks are computed among methods sharing each seed’s data split and teacher checkpoint.Source: §B.2 · p. 17
• Adding the Code term to Pred-Match improves mean performance in all eight directions, with statistically significant gains in four.The Code term regularizes intermediate representations, while Pred-Match targets the codebook-induced teacher prediction q(t), not the original prediction p(t).Source: §B.2 · p. 18
• The combined code-selection criterion improves mean performance in all eight directions and is statistically significant in seven.MM-IMDB I→T shows a positive trend (p = 0.0659).Source: §B.2 · p. 18

B.3 Computational Cost Analysis

• 7.9% and 7.7% of total training time is spent on Stage-2 scoring for V→A and A→V, respectively, while memory and alignment-related FLOPs are not consistently higher than competitors’.The method has the longest wall-clock training time, attributed to its three-stage procedure; its Stage-2 scoring is not the dominant cost under the default configuration.Source: §B.3 · pp. 18–19
• Stage-2 runtime increases approximately linearly with codebook size B, while an intermediate size of B = 512 offers the best accuracy–cost trade-off.With B = 2048, over 30% of codes frequently remain unused; B ≤ 512 generally maintains 99–100% utilization.Source: §B.3 · p. 19
• At inference, only the original student encoder and task head remain, adding no parameters, memory, or computation over a student trained without distillation.The teacher and all codebook-related components, including Stage-2 scoring, are removed.Source: §B.3 · pp. 19–20

B.4 Discussion on the Code Retention Ratio

• Performance remains relatively stable across a broad range of retention ratios and generally exceeds the w/o KD baseline, although the best ratio varies by dataset and transfer direction.The experiments use fixed defaults of 50% for classification and 12.5% for segmentation, chosen as practical rather than individually optimized settings.Source: §B.4 · p. 20
• A fixed retention ratio does not adapt to scenario-specific amounts of transferable information, making adaptive code retention a future direction.The appropriate retained amount varies across cross-modal scenarios.Source: §B.4 · p. 21

B.5 Class-wise Analysis on NYU-Depth V2

• CA improves from 0.2100 to 0.2184 and mIoU from 0.1366 to 0.1456, although OA falls from 0.5269 to 0.5218 in RGB→Depth.The wall class’s recall decreases by 0.0274 and it accounts for 25.6% of pixels, while several less frequent classes improve by 0.13–0.18 in recall.Source: §B.5 · p. 21
• The OA decrease is attributed to class imbalance rather than systematic degradation across classes; gains on less frequent classes are consistent with complementary RGB-teacher semantic cues.The class-balanced CA and mIoU improve despite the lower OA.Source: §B.5 · pp. 21–22

B.6 Hard and Soft Code Assignment

• The discrete codebook provides a structure-independent concept interface that supports evaluating, ranking, and retaining transferable concepts.The passage contrasts this with unrestricted continuous mixtures of anchors, for which code-level selection is less directly defined.Source: §B.6 · p. 22
• Hard assignment yields slightly higher mean accuracy in both transfer directions, while soft assignment over the same selected codes achieves comparable performance.The result indicates the selected codes remain effective under soft assignment and hard assignment is sufficient in the tested setting.Source: §B.6 · p. 22

B.7 Code Reuse Across Student Units

• 94.48% of activated codes for A→V and 97.47% for V→A are reused by multiple student units, indicating predominantly many-units-to-one-code usage.The measurement is over the VGGSound test set.Source: §B.7 · pp. 22–23
• Independent assignment does not explicitly model concepts distributed across multiple student units before quantization, a stated direction for future work.The current codebook-branch head jointly processes quantized units while retaining their temporal or spatial organization.Source: §B.7 · p. 23

B.8 Qualitative Analysis of Learned Codes

• In CrisisMMD, selected codes attend to damaged door or gate regions alongside related text tokens; in MM-IMDB, codes attend to salient poster regions alongside drama-related narrative phrases.The examples provide qualitative support for associations between learned codes and semantically related visual and textual cues.Source: §B.8 · p. 23, Figure 4 · p. 24

C Pseudo Algorithm of The Proposed Framework

• The pseudo-code specifies three stages: teacher-side codebook learning, code-wise importance estimation, and selective codebook-based distillation.Although presented at dataset level, optimization is implemented with mini-batch training.Source: C Pseudo Algorithm of The Proposed… · p. 24

D Broader Impact

• The paper cautions that students may inherit teacher or data biases and recommends careful fairness, privacy, and robustness evaluation for sensitive deployments.The framework is not tied to a specific deployment scenario.Source: D Broader Impact · p. 24