FAM-gate: field-aware metadata attention with prior-constrained vector gating for audio classification

1. INTRODUCTION

• Audio-only models can struggle when distinct events share acoustic properties or have high intra-class variance, and class-level text often lacks recording-specific context.Prior work aligned audio with label language or learned shared audio-text spaces for zero-shot classification,.Source: §1 · p. 1
• Prior metadata studies used titles, album names, artists, or patient attributes; Freesound titles, tags, and descriptions underpin FSD50K and BSD,,.A BSD approach concatenated these fields and fused text embeddings with CLAP audio features through a scalar gate.Source: §1 · p. 1
• Generic multimodal feature recalibration methods such as MMTM and FiLM do not account for metadata field noise or constrain fusion around the acoustic prior,.Source: §1 · p. 1
• Titles, tags, and descriptions vary in structure, semantic density, noise, and recording-dependent relevance, while scalar gating uses one fusion weight across all feature dimensions.The latter can miss channel-wise alignment or conflict between acoustic and semantic evidence.Source: §1 · p. 1
• FAM-Gate combines audio-guided, field-aware metadata aggregation with bounded, dimension-wise audio-metadata fusion centered around an acoustic prior.Its stated contributions include an alignment-preserving, mean-anchored aggregation mechanism and a gate for modeling cross-modal complementarity and conflict.Source: §1 · p. 1

2.1. Overall Architecture

• MVA aggregates valid title, tag, and description embeddings using audio context, and PCVG then regulates their interaction with acoustic evidence across feature dimensions.LAION-CLAP supplies the audio and text embeddings; a binary mask excludes missing metadata fields.Source: §2.1 · pp. 1–2, Figure 1 · p. 2

2.2. Field-Aware Metadata View Attention

• MVA uses a shared residual projection and field identities, combines audio-dependent relevance with field priors, and anchors weighted original embeddings to the masked mean.The bounded deviation factor γ allows the masked mean baseline to be recovered when γ = 0.Source: §2.2 · p. 2

2.3. Prior-Constrained Vector Gate

• PCVG predicts bounded, dimension-wise gates from audio, metadata, and their interactions, using an acoustic prior to constrain fusion between the modalities.Larger gate values prioritize audio in a feature dimension; smaller values rely more on metadata.Source: §2.3 · pp. 2–3

2.4. Prediction and Objectives

• The classification objectives use categorical cross-entropy for single-label BSD datasets and multi-label binary cross-entropy for FSD50K.The multi-label setting uses independent sigmoid activations.Source: §2.4 · p. 3

3.1. Datasets and Experimental Settings

• The experiments cover BSD10K and BSD35K single-label tasks and FSD50K’s 200-class multi-label task, with six matched variants evaluated using five fixed seeds.The variants share the LAION-CLAP backbone, prediction head, splits, and training hyperparameters.Source: §3.1 · p. 3

3.2. Implementation Details

• Training fine-tunes the pretrained LAION-CLAP architecture end-to-end, masks unavailable metadata fields and withholds labels from the text encoder, and uses Adam with early stopping.The setup uses a batch size of 64 and trains for at most 100 epochs.Source: §3.2 · p. 4

3.3. Overall Performance

• FAM-Gate reaches 74.98% mAP on FSD50K, numerically exceeding the highest listed audio-only result by 7.69 percentage points, but the modalities differ.Table 1 provides cross-setting context; the paper identifies Table 2 as the controlled module comparison.Source: Table 1 · p. 3, §3.3 · p. 4

3.4. Ablation Study

• Against Random View + Scalar, FAM-Gate gains 2.01/2.53 points in BSD10K accuracy/hF, 4.09/5.72 in BSD35K accuracy/hF, and 5.07/10.25 in FSD50K mAP/macro F1.The matched ablations use five independent runs.Source: §3.4 · p. 4, Table 2 · p. 4
• PCVG Only improves FSD50K mAP/macro F1 by 2.09/3.24 points over scalar fusion; adding all fields and Field-Aware MVA brings further gains across tasks.With PCVG fixed, FAM-Gate improves hF over Mean + PCVG by 0.72 points on BSD10K and 1.83 on BSD35K, and macro F1 by 0.51 on FSD50K; BSD35K accuracy is 0.08 points lower.Source: §3.4 · p. 4
• FAM-Gate adds 0.314M parameters (0.19%) over the baseline, while task-side computation rises from 7.286M to 8.386M MACs per sample.The computation counts exclude the additional cost of encoding three metadata views instead of one.Source: §3.4 · p. 4
• Learned γ values of 0.067, 0.138, and 0.422 indicate increasingly stronger adjustments from masked-mean aggregation on BSD10K, BSD35K, and FSD50K; field weights vary across recordings.Descriptions receive the highest average BSD35K weight (0.461), while tags receive the highest average FSD50K weight (0.656).Source: §3.4 · p. 4, Figure 3 · p. 4

• FAM-Gate combines field-aware metadata aggregation with prior-constrained vector fusion.Source: §4 · p. 4
• Five-seed experiments across three datasets demonstrate consistent improvements in hierarchical and macro-level metrics over matched baselines.Source: §4 · p. 4
• Future work will address missing or corrupted metadata and broader audio-language backbones.Source: §4 · p. 4