PaperlayerView exact revision on arXiv

Source-linked AI summary

When Can You Prune Your Network? A Study of Intermediate Neurons in Multilingual Speech Parsing

Minnie Kabra, Benjamin Lecouteux, Maximin Coavoux

arXiv:2610.11520v1cs.CL

TL;DR

The role of intermediate neural units in end-to-end speech parsing had not been studied, despite their added computational training time. The paper introduces Simplified Wav2tree, which removes those units, and reports comparable performance with greater parameter efficiency; intermediate units still add value when the encoder is frozen.

  • Problem

    The impact of intermediate neural units in end-to-end speech parsing had not been studied, although these units increase computational training time.

    Source: §1 · p. 1

  • Method

    The paper introduces Simplified Wav2tree, an end-to-end speech parser that removes intermediate neural units.

    Source: §1 · p. 2

  • Results

    The simplified architecture is parameter-efficient while achieving comparable performance, and intermediate units provide incremental value when the pretrained model is frozen.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Intermediate units help reduce the representational gap when the encoder is frozen, while the simpler architecture offers comparable performance with greater efficiency.

    Source: §1 · p. 2, §7 · p. 9

    Hyperparameter search was limited to learning rate by computational resource constraints, and the architecture was not evaluated on English Switchboard because its recordings are not publicly available.

    Source: Limitations · p. 9

Abstract

from arXiv · show

End-to-end speech parsing, a task recently proposed, consists in predicting both the transcription and the syntactic tree for a spoken utterance. Existing architectures for speech parsing often utilise intermediate neural networks. In this work, we examine the effectiveness of intermediate neural networks (NN) for parsing, and, specifically, what role do they play. We introduce a simpler end-to-end architecture for speech parsing, where we remove these intermediate NN units, reducing the parameters by 12%, while achieving comparable or better performance than prior method on both automatic speech recognition (ASR) and parsing. We demonstrate that intermediate NN units help reduce the representational gap when the pre-trained encoder is frozen. We do a comprehensive evaluation of speech parsing on French, and medium-low resource languages Slovenian and Naija. We further investigate the impact of the training data size and intermediate layers of the pretrained speech encoder on speech parsing.

1 Introduction

Speech parsing from spontaneous speech remains challenging, and the role of intermediate neural units in end-to-end parsers had not been studied. The paper introduces a simpler architecture and evaluates its performance, data requirements, and behavior across three languages.

1 Introduction

  • Intermediate units in prior speech-parsing architectures increase computational training time, but their impact on parsing had not been studied.The paper investigates when these units add value.Source: §1 · p. 1
  • The simplified Wav2tree removes intermediate neural units while generally matching or improving performance with greater efficiency.The evaluation also compares it with a pipeline approach; end-to-end parsing performs better except for Slovenian.Source: §1 · p. 2
  • Intermediate neural units help reduce the representational gap when the pretrained encoder is frozen.A frozen encoder remains optimized for its pretraining objective rather than downstream parsing.Source: §1 · p. 2
  • The paper evaluates speech dependency parsing on the Naija, Slovenian, and French spoken treebanks.It also examines representations from intermediate Wav2Vec2 layers as input to ASR and parsing in the simplified architecture.Source: §1 · p. 2
  • The study analyzes how scaling speech training data affects dependency parsing and uses a two-stage experimental setup to assess the amount needed for near-optimal performance with the Simplified architecture.Source: §1 · p. 2
  • Naija segmentation annotations do not improve parsing from audio, although they had helped parsing with gold transcriptions.For Slovenian, the authors release code to produce a filtered treebank with improved speech-transcription alignment.Source: §1 · p. 2

2 Related work

Earlier speech-parsing work often parsed transcriptions or added audio features to text parsers; only a few studies parsed speech directly. This paper focuses on realistic spontaneous speech, including lower-resource languages, and introduces a simpler architecture.

2 Related work

  • Prior approaches often parsed gold or predicted transcriptions, sometimes jointly detecting disfluencies, or added audio features to text parsers.This paper instead parses directly from speech while performing ASR jointly, following Pupier et al. (2022).Source: §2 · p. 2
  • Only a few publications parsed speech without relying on transcriptions, and this paper extends that work to Naija and Slovenian.Earlier studies evaluated on French Orféo, with one also using English Switchboard.Source: §2 · p. 2
  • The paper’s second major novelty relative to those publications is a much simpler deep-learning architecture.The authors introduce Simplified Wav2tree by removing intermediate neural units.Source: §2 · p. 3
  • Unlike work using read-aloud captions, this study focuses on realistic interactions with disfluencies, interruptions, and overlapping speakers.Read-aloud texts lack those phenomena and have syntax closer to written text.Source: §2 · p. 3

3 Datasets

The evaluation uses spoken treebanks from three languages, with French Orféo added to examine how training-data size affects parsing.

3 Datasets

  • The multilingual evaluation trains and evaluates separately on three spoken treebanks; French Orféo is added for training-data scaling experiments.The treebanks cover Naija, Slovenian, and French.Source: §3 · p. 3
  • The Naija and Slovenian treebanks contain recordings from varied spontaneous-speech settings and use different syntactic annotation schemes.Naija uses SUD; Slovenian uses UD.Source: §3 · p. 3
  • UD French ParisStories recordings are personal life stories, while Orféo spans several spoken domains and is the only publicly available spoken treebank large enough for scaling experiments.About 5% of Orféo is manually annotated, with the remainder automatically annotated as good-quality silver data.Source: §3 · p. 3
  • The Slovenian-SST data were filtered for speech noise and had speech-transcription misalignment enhanced.Source: §3 · p. 3

4 Background: Wav2tree Architecture

Wav2tree parses spontaneous speech end to end by combining speech recognition with dependency parsing. Its intermediate units turn frame-level speech representations into word-level representations for the parser.

4 Background: Wav2tree Architecture

  • Wav2tree takes raw speech and jointly produces a transcription and syntactic tree through speech-encoding, recognition, and parsing modules linked by three intermediate neural units.The model is trained end to end with an ASR-and-parsing multitask objective to reduce error propagation between the tasks.Source: §4 · pp. 3–4
  • Wav2Vec2 converts speech into frame vectors, which the feature encoder transforms before character-level CTC recognition and greedy decoding.CTC predicts a character or blank for each frame; greedy decoding produces a transcription and a mapping from frames to words.Source: §4 · p. 4
  • The frame-to-word mapping supplies predicted word boundaries, and an LSTM uses each word’s frame span to compute a fixed-size audio word embedding.Speech frames form a word sequence analogous to the characters used for text word embeddings.Source: §4 · p. 4
  • A bidirectional LSTM contextualizes the sequence of audio word embeddings before the resulting word-level representations go to a dependency parser.The parser can use a seq2label strategy or a biaffine architecture.Source: §4 · p. 4

5 Experiments

The experiments examine when intermediate neural units help, how training-data size affects parsing, and how speech parsing performs across languages and settings. They compare simplified and baseline architectures and evaluate parsing under different representation and transcription conditions.

5 Experiments

  • The experiments address the role of intermediate units and the effect of training-data size on speech-parsing accuracy, including settings that control for ASR accuracy.Source: §5 · p. 4

5.1 RQ1: Role of Intermediate NN Units

  • The authors hypothesize that intermediate units add no incremental value when the pretrained model is jointly fine-tuned, but help reduce the representational gap when it is frozen.The gap is between representations optimized for pretraining and the downstream parsing model.Source: §5.1 · p. 4

5.1.1 (H1) Simplified Architecture

  • Simplified Wav2tree removes all three intermediate units, sends speech-encoder representations directly to the parser, and reduces parameter count by 12%.The simplifications apply CTC directly to Wav2Vec2 frame representations, average frames within each word, and bypass the word-level LSTM and bidirectional LSTM.Source: §5.1.1 · p. 5

5.1.2 (H2) When Do Intermediate NN Units Add Value?

  • To test when intermediate units add value, the text-parsing experiments compare an LSTM with no intermediate unit under joint fine-tuning and a frozen pretrained text encoder, using gold transcriptions.The equivalent frozen-encoder experiment cannot be conducted for speech parsing because Wav2Vec2 requires fine-tuning for the target language.Source: §5.1.2 · p. 5, Figure 3 · p. 5

5.2 RQ2: Scaling Training Data

  • Training-data scaling uses five partitions, from 20% to 100%, of Naija-SNC and Orféo; a pre-fine-tuned ASR system controls for ASR’s effect on parsing.An intermediate LSTM aligns representations from the frozen ASR model to the parsing model.Source: §5.2 · pp. 5–6

5.3 Dependency Parsing Evaluation

  • The parsing evaluation examines intermediate Wav2Vec2 layers as inputs to both ASR and parsing and compares text parsing with predicted-transcription pipeline inputs against gold-transcription inputs.Gold transcriptions are intended as an upper-bound setting with perfect ASR.Source: §5.3 · p. 6

5.4 Segmentation Annotations in Naija

  • Naija’s punctuation-marked syntactic and prosodic segmentation annotations were useful with gold transcriptions but not when parsing the audio signal.Source: §5.4 · p. 6

5.5 Training and Evaluation Details

  • The evaluation uses multilingual speech and text encoders, a biaffine parser, shared hyperparameters, development-set checkpoint selection, and test-set reporting.Metrics are WER for ASR, POS accuracy, UAS, and LAS for dependency parsing.Source: §5.5 · p. 6

6 Results

The simplified architecture generally preserves or improves parsing performance while maintaining ASR performance, and intermediate units are useful when the pretrained encoder is frozen. Larger training sets improve results, although gains diminish in the frozen-ASR setup.

6.1 RQ1: Role of Intermediate NN Units

  • LAS increased 1.2% for Slovenian and 2.5% for French, but decreased 0.95% for Naija after removing intermediate units.The Slovenian and French improvements were statistically significant.Source: §6.1 · p. 6
  • Simplified Wav2tree matches or improves parsing performance over Wav2tree while ASR performance remains consistent, with greater parameter efficiency.The reported comparison covers the three spoken treebanks.Source: §6.1 · p. 7, Table 2 · p. 7
  • With jointly fine-tuned text models, LAS was higher without an intermediate unit across all three treebanks.Similarity between fine-tuned and frozen text-model representations was higher with an intermediate unit across all three treebanks.Source: §6.1 · pp. 7–8
  • With the encoder frozen, LAS dropped at least 30% across all three treebanks; adding an intermediate unit raised it by 15–20%.The authors interpret this as evidence that intermediate units reduce the representational gap.Source: §6.1 · pp. 7–8, Table 3 · p. 8

6.2 RQ2: Scaling Training Data

  • With larger training sets, WER decreased and LAS improved in the baseline setup; frozen-ASR gains were smaller and varied by treebank.In frozen ASR, LAS stayed relatively stable for Orféo and rose through 80% of training data for Naija; gains diminish beyond 9–10 hours.Source: §6.2 · p. 8

6.3 Dependency Parsing Evaluation

  • Using intermediate Wav2Vec2 layers kept LAS stable or improved it, with 12%–24% fewer parameters and 5%–9% faster training.The selected layer was the one with the highest development-set LAS.Source: Table 2 · p. 7, §6.3 · p. 8
  • Except for Slovenian, Simplified Wav2tree parsed better than the pipeline approach, which had 2.75× more parameters.The pipeline uses separate pretrained acoustic and text models. For Naija, WER was higher on tokens outside the English lexicon, a pattern the authors link likely to limited representation of Naija-related languages in pretraining.Source: §6.3 · p. 8

7 Conclusion and Further work

The paper concludes that intermediate units help when the pretrained model is frozen, while a simpler, parameter-efficient speech parser achieves comparable performance. It also reports diminishing parsing gains beyond 9–10 hours of training data and identifies further questions for study.

7 Conclusion and Further work

  • Intermediate units help reduce the representational gap when the pretrained model is frozen, while the simpler parser is parameter-efficient and performs comparably.The authors identify further work on why the pipeline outperforms the end-to-end parser on Slovenian and why intermediate speech-encoder representations preserve parsing performance.Source: §7 · pp. 8–9
  • Parsing-accuracy gains diminish after the first 9–10 hours of training data.This conclusion is reported for the Simplified Wav2tree architecture.Source: §7 · p. 9
  • The paper reports baseline results for Naija, Slovenian, and French ParisStories, including the first end-to-end speech-parsing experiments for Naija and Slovenian.It also reports stable parsing performance with intermediate Wav2Vec2 layers and releases code to reproduce a filtered Slovenian treebank with improved speech-transcription alignment.Source: §7 · p. 9

Limitations

The experiments were constrained by computational resources, and the proposed architecture was not evaluated on English Switchboard because its audio recordings were not publicly available.

Limitations

  • Hyperparameter search was limited to learning rate because of computational resource constraints, and English Switchboard was excluded because its audio recordings were not publicly available.Source: Limitations · p. 9

B Punctuation in Naija

The Naija analysis finds that punctuation marks in evaluation data are associated with lower LAS, while removing them from training leaves parsing performance unchanged or improved. The authors link the evaluation effect to ASR errors on segmentation marks.

B Punctuation in Naija

  • Naija markup includes symbols for illocutionary-unit boundaries, disfluencies, pauses, and nucleus–adnucleus structure.Source: B Punctuation in Naija · p. 12
  • LAS dropped 4.1% when punctuation marks were present in the evaluation data.The comparison used evaluation datasets with and without markups.Source: B Punctuation in Naija · p. 12
  • As markup counts rose, ASR predicted fewer marks than appeared in the reference, and the LAS gap between data with and without marks widened.The authors hypothesize that segmentation symbols are difficult to recognize because they do not correspond to identifiable sounds.Source: B Punctuation in Naija · p. 12
  • Parsing performance stayed the same or improved when punctuation marks were removed from training, so the authors used Naija data without punctuation in their evaluations.Source: B Punctuation in Naija · p. 12

C Slovenian dataset

The authors report 30% LAS on the current UD Slovenian dataset, then filter alignment, transcription, and recording noise. They also remove punctuation from Slovenian for analysis after observing that ASR misses punctuation marks in Naija.

C Slovenian dataset

  • 30% LAS on the current UD Slovenian dataset prompted investigation and filtering of two noise types and noisy recordings.The authors also removed punctuation from Slovenian for analysis based on Naija punctuation findings.Source: C Slovenian dataset · p. 12
  • Multiple transcripts without start or end timestamps for one recording were identified as speech-alignment noise.Source: C Slovenian dataset · p. 12
  • Personal-name placeholder tokens, paired with beeps in the audio, mislead ASR and degrade parsing performance.Source: C Slovenian dataset · pp. 12, 14
  • ASR did not predict some Naija punctuation marks, including parentheses and an angle-bracket token, motivating punctuation removal from Slovenian analysis.Source: C Slovenian dataset · p. 12, Figure 6 · p. 14
  • For unique recordings, the authors used forced alignment because speech–transcript alignment was not fully accurate.Source: C Slovenian dataset · p. 14

D.1 Statistical Significance

On Naija, the simplified architecture reduced training and inference time relative to the baseline. Statistical tests found LAS differences between simplified and baseline architectures for Slovenian and ParisStories, but not Naija.

D.1 Statistical Significance

  • Wilcoxon signed rank tests on LAS found significant simplified-versus-baseline differences for Slovenian and ParisStories, but not Naija.The tests used a 5% confidence level for the stated conclusions.Source: §D.1 · p. 14, Table 10 · p. 15
  • For Slovenian and ParisStories, the directional test supported higher LAS for the simplified architecture at the 5% confidence level.Source: §D.1 · p. 14, Table 10 · p. 15

E Computational Time

  • 60% of baseline training computational time was required by the simplified architecture on Naija, with inference time reduced by 20%.The comparison used identical hyperparameter settings.Source: E Computational Time · p. 14, Table 11 · p. 15
  • Using 21 intermediate Wav2Vec2 layers reduced training time by 5% compared with using the full set of layers.Source: E Computational Time · p. 14
Loading 2610.11520v1…