PaperlayerView exact revision on arXiv

Source-linked AI summary

A New Transformer-Based Approach for Audio-Based Kinship Verification and a New Uncontrolled Mandarin Kinship Speech Dataset

Qiyang Sun, Langqing Zhang, Yupei Li, Björn Schuller

arXiv:2609.14145v2cs.SDcs.AIcs.LGeess.AS

TL;DR

Audio kinship verification faces limited dataset coverage and recording conditions unlike everyday use. The paper introduces ARKIN and proposes CONVTRAP-TN, a transformer-based approach. Its evaluations report that models trained in one environment generalize poorly to others, regardless of dataset size.

  • Problem

    Audio kinship datasets are few and largely rely on internet-sourced or restricted-environment recordings, while everyday personal-device conditions are not represented well.

    Source: §I · p. 1

  • Method

    The paper collects ARKIN, an everyday-condition-oriented speech dataset, and proposes CONVTRAP-TN, a transformer-based audio kinship verification method.

    Source: §I · p. 2, B. Kinship verification · p. 3

  • Results

    Models trained in one environment generalize poorly to others, regardless of dataset size, and attention pooling and two-stage training are the most influential CONVTRAP components.

    Source: §VI · p. 6, B. Ablation study · p. 6

  • Takeaways & Limitations

    ARKIN enables cross-dataset evaluation of audio models, and the reported domain gap underscores the challenge of cross-environment generalization.

    Source: §VI · p. 6

Abstract

from arXiv · show

Kinship verification is a task involving determining whether two individuals share a first-order kin relation. To tackle this task, we propose CONVTRAP-TN, a new architecture for audio-based kinship verification, and conduct an ablation study on the proposed model. To the best of our knowledge, we are the first to apply the successful transformer architecture to the task of audio-based kinship verification. Furthermore, we also collect a custom speech dataset, ARKIN, which accurately reflects everyday recording conditions. We do this because only a few speech datasets with kinship labels currently exist, all of which either source extremely noisy in-the-wild data from the internet, or instruct speakers to record in specific environments. These settings fail to reflect real-world scenarios where users record on personal devices under unrestrained conditions. Additionally, we perform a series of preliminary baseline experiments on the collected dataset, including speaker verification and recognition, speech recognition, age estimation, and kinship verification, as well as cross-dataset kinship verification experiments to show that existing methods are not robust across datasets.

I. INTRODUCTION

Audio-based kinship verification is difficult and few audio kinship datasets support evaluation across realistic conditions. The paper introduces ARKIN, proposes a transformer-based method, and investigates cross-dataset robustness.

I. INTRODUCTION

  • Kinship verification is more challenging than speaker-level tasks, while multimodal models can perform better than single-modality models when multiple modalities are available.The authors motivate robust audio-only methods for settings where multiple modalities may not be available.Source: §I · p. 1
  • Audio kinship data are scarce, and public datasets use either internet-sourced recordings or restricted environments that do not reflect typical end-user recording conditions.Existing audio kinship data also consist primarily of English speech, limiting evidence for language-invariant modeling.Source: §I · p. 1
  • The paper introduces ARKIN, an everyday-condition-oriented Mandarin speech dataset with kinship, age, and gender labels, and benchmarks existing methods across datasets.The authors report that cross-dataset generalization remains challenging.Source: §I · p. 2
  • The authors propose CONVTRAP-TN, using attention to encode audio while accounting for sparse kinship signals, and describe it as the first such transformer approach for audio-based kinship verification.They use speaker recognition as an easier first stage to address training instability and explore explainability.Source: §I · p. 2

II. RELATED WORK

Public audio kinship datasets differ in source and recording conditions, while ARKIN is presented as an uncontrolled Mandarin dataset recorded on consumer-grade devices.

II. RELATED WORK

  • The related-work section identifies KAN-AV, TALKIN-Family, and FIW-MM as publicly available audio datasets with kinship labels.The authors compare these datasets, including ARKIN, in Table I.Source: §II · p. 2
  • Existing audio kinship datasets include internet-sourced recordings of public figures and data collected in quieter settings, leaving neither setting representative of average end users.KAN-AV and FIW-MM use public-figure recordings from the internet, while TALKIN-Family participants recorded in quiet locations.Source: §II · p. 2
  • ARKIN is presented as the first dataset with Mandarin recordings in uncontrolled conditions using consumer-grade devices, rather than public internet recordings.The authors distinguish internet-sourced recordings as often having very low signal-to-noise ratio and specialized equipment.Source: §II · p. 2

III. ARKIN DATASET

ARKIN is an audio kinship benchmark designed to capture realistic recording conditions, with labeled speakers and utterances collected under unconstrained participant-chosen conditions.

A. Overview

  • ARKIN contains 256 speakers and 2,554 utterances, with 230 kinship pairs; 95% of the utterances are Mandarin and 5% English.Speakers are labeled with age, gender, and nationality, and utterances include manual transcriptions and language labels.Source: A. Overview · p. 2

B. Data collection and processing

  • Participants recorded on devices they normally used, without restrictions on where, when, or under what background conditions they recorded.This produced a broad variety of background conditions.Source: B. Data collection and processing · p. 2
  • Participants read a fixed phrase and could improvise; researchers manually segmented clips into utterances of roughly 0.3–13.3 seconds and removed sections without transcribable subject audio.The data collection process is illustrated in Figure 1.Source: B. Data collection and processing · p. 2, Figure 1 · p. 3, Table I · p. 3

IV. BASELINE EXPERIMENTS

The baseline experiments prepare speech features from ARKIN and evaluate multiple speaker representations using disjoint speaker-level data partitions.

A. Speech representation

  • Audio was resampled to 16 kHz, converted to 24-coefficient MFCCs, and processed with CMVN and energy-threshold VAD before modeling.MFCCs used a 25 ms frame length and 15 ms frame shift.Source: A. Speech representation · p. 3
  • The experiments compare GMM-UBM i-vectors, TDNN x-vectors and ECAPA-TDNN embeddings, and pretrained VoxCeleb ResNet embeddings.The representations include 400-dimensional i-vectors, 512- and 300-dimensional x-vectors, 256-dimensional ECAPA-TDNN embeddings, and 256-dimensional ResNet embeddings.Source: A. Speech representation · p. 3
  • Data were split 7:1:2 into disjoint training, validation, and test sets, with no kinship relation spanning sets, and results were averaged over five experiments.MFCC sequences and extracted i-vector, x-vector, and ResNet features were used in the experiments.Source: A. Speech representation · p. 3

B. Kinship verification

Kinship verification uses triplets of utterances from related and unrelated speakers to train a shared embedding network for deep metric learning.

B. Kinship verification

  • Random triplet mining pairs utterances from related speakers, then selects an unrelated speaker as the negative sample.The first related speaker is the anchor, the second is positive, and the unrelated speaker forms the negative triplet.Source: B. Kinship verification · p. 3
  • TripletNet maps each triplet through a shared four-layer DNN into normalized embeddings on a unit hypersphere.The model computes Euclidean distances between the anchor embedding and the positive and negative embeddings.Source: B. Kinship verification · p. 3
  • Training minimizes triplet loss over the positive and negative distances, using a margin M = 10.The loss aims to distinguish the positive pair from the negative pair in embedding space.Source: B. Kinship verification · p. 3

2) CONVTRAP-TN:

CONVTRAP-TN combines convolution, transformer encoding, and attention pooling for audio-based kinship verification. It performed best among the tested methods, while sensitivity varied across kin types.

2) CONVTRAP-TN:

1) TripletNet:

  • Table II gives detailed statistics for ARKIN dataset partitions, including sex and kinship-type categories.Source: Table II · p. 4

2) CONVTRAP-TN:

  • The two-stage training first trains speaker recognition, then freezes parameters before the attention pool and trains the partially frozen model on kinship verification.The authors describe this as extracting intermediate features, similarly to x-vectors, but from a sequence rather than a single vector.Source: 1) TripletNet · p. 4

1) TripletNet:

  • CONVTRAP-TN performed best among the tested methods for kinship verification; the authors suggest sequence-preserving features and long-range attention may explain its result.The authors note that the frozen transformer outputs an unreduced feature sequence from raw MFCCs for downstream pooling and the DNN.Source: Table III · p. 4, 1) TripletNet · p. 4
  • Sensitivity varied substantially by kin type: TripletNet was least sensitive to F-S, F-D, B-B, and S-S, while CONVTRAP-TN was highly sensitive to F-S and F-D.CONVTRAP-TN was less sensitive to M-S and M-D; the authors say sensitivity was somewhat proportional to training-data amount, but model choice also appeared influential.Source: 1) TripletNet · p. 4, Table IV · p. 5
  • Table V reports balanced-accuracy evaluation on ARKIN after training on KAN-AV, setting up a cross-dataset comparison.Source: Table V · p. 5

C. Cross-dataset kinship verification

The cross-dataset experiments train baseline models on KAN-AV, compare evaluation on KAN-AV and ARKIN, and test fine-tuning on ARKIN. Models show substantial performance degradation across datasets, alongside a large visualized domain gap.

C. Cross-dataset kinship verification

  • The experiments compare KAN-AV evaluation with ARKIN evaluation and assess fine-tuning using the first 20% of ARKIN training speakers.The baseline models are trained on KAN-AV.Source: C. Cross-dataset kinship verification · p. 5
  • Baseline models trained on KAN-AV suffer drastic performance degradation on ARKIN, and CONVTRAP-TN also performs much worse on KAN-AV than on ARKIN.The authors suspect differences in language and recording environments contribute to the gap.Source: C. Cross-dataset kinship verification · p. 5
  • Fine-tuning on the first 20% of ARKIN training speakers worsened CONVTRAP-TN to below-random performance.The authors suspect the distribution difference makes KAN-AV a poor initialization for learning ARKIN.Source: C. Cross-dataset kinship verification · p. 5
  • t-SNE visualization of TripletNet x-vector embeddings shows a large domain gap between KAN-AV and ARKIN utterances.Figure 3 plots KAN-AV as red circles and ARKIN as blue triangles.Source: C. Cross-dataset kinship verification · p. 5, Figure 3 · p. 5

D. Auxiliary Tasks

The auxiliary benchmarks assess identity, age, and speech-recognition tasks on ARKIN. Results show strong speaker recognition and verification, while age estimation and ASR provide additional baseline measures.

D. Auxiliary Tasks

  • The auxiliary benchmarks use ARKIN’s metadata to assess performance across different speech dimensions.The paper presents a summary of results, with detailed setups and reproduction scripts slated for its public repository.Source: D. Auxiliary Tasks · p. 5
  • ResNet achieves 96.7% speaker recognition accuracy and 82.5% speaker verification accuracy on ARKIN, outperforming traditional i-vector baselines.The authors interpret these results as evidence that ARKIN preserves speaker-identity features despite uncontrolled recording conditions.Source: D. Auxiliary Tasks · p. 5, Table VI · p. 6
  • ResNet yields the lowest age-estimation MAE of 5.6 years, while Whisper’s ASR CER is 18.6% versus 50.8% for Wav2Vec 2.0.The authors describe ARKIN as challenging for standard acoustic models while noting robust transcription is achievable.Source: D. Auxiliary Tasks · p. 5, Table VI · p. 6

A. Attention visualization

The attention-pool visualization compares selected true-positive, true-negative, and false-positive pairs with their input spectrograms.

A. Attention visualization

  • CONVTRAP attention pooling aggregates salient speech regions; in the false-positive example, attention still falls on active speech, suggesting ambiguity in learned representations rather than background confounding.Figure 4 shows the spectrograms and attention-pool scores for true-positive, true-negative, and false-positive pairs.Source: A. Attention visualization · p. 6

B. Ablation study

The ablation study identifies attention pooling and two-stage training as especially influential, while the conclusion reports cross-dataset generalization and kinship-type sensitivity concerns. The authors also limit current experiments to supervised features.

B. Ablation study

  • Attention pooling and two-stage training are the most vital components; the model fails catastrophically without either.The authors attribute attention pooling’s advantage over mean pooling to kinship clues concentrated in a few regions.Source: B. Ablation study · p. 6
  • The convolution block is critical, while using one transformer layer decreases performance by only 2.8%.The authors note the single-layer effect might differ with much larger training datasets.Source: B. Ablation study · p. 6

VI. CONCLUSION

  • Models are generally most sensitive to brother-sister relations, and sensitivity varies substantially across kinship types, raising fairness and bias concerns.The authors identify further improvement in this area as future work.Source: §VI · p. 6
  • Models trained in one environment generalize poorly to others regardless of dataset size, indicating a remarkable domain gap.The conclusion presents ARKIN as enabling cross-dataset evaluation of audio models.Source: §VI · p. 6
  • The experiments use only supervised features; future work will examine SSL features and expand data collection and evaluation across diverse populations.Source: §VI · p. 6
Loading 2609.14145v2…