PaperlayerView exact revision on arXiv

Source-linked AI summary

Using Prosody to Predict Syntactic Structure

Junghyun Min, Alex Warstadt, Tamar I. Regev, Tiago Pimentel, Ethan Gotlieb Wilcox

arXiv:2608.30260v1cs.CLcs.AIcs.LG

TL;DR

The paper addresses the contested degree and nature of syntactic information in prosody. It introduces a mutual-information framework estimated with multimodal language models over speech-text corpora, finding measurable information and up to 10.2% uncertainty reduction from duration and pauses.

  • Problem

    The degree and nature of the relationship between prosody and syntactic structure remain debated, while existing empirical methods have limited scale or make simplifying assumptions.

  • Method

    The paper estimates mutual information between prosodic features and syntactic representations using multimodal language models over large speech-text corpora.

  • Results

    Prosodic duration and pauses reduce uncertainty over syntactic representations by up to 10.2%, with information concentrated on boundary locations and redundant with text.

  • Takeaways & Limitations

    The findings support Prosodic Bootstrapping and Indirect Reference accounts of the syntax-prosody interface.

  • Takeaways & Limitations

    The study examines only pause and duration in English and uses silver parses, whose errors may affect the results.

Abstract

from arXiv · show

While it is well-established that prosody carries crucial cues for syntactic structure, the degree and nature of correspondence between these two domains remains contested. We investigate the syntax-prosody interface through an information-theoretic lens, quantifying the interaction between prosodic features and syntactic representations as their mutual information. We provide a general-purpose framework for estimating this quantity over large speech-text corpora using multimodal language models. Our framework is structure-agnostic and modular, insofar as it can be used to measure the contributions of individual prosodic features or components of structure. We evaluate the syntax-prosody relationship for two features (word duration and inter-word pauses) across two domains--read audiobooks and spontaneous conversations--both in English. Our results demonstrate that prosody contains measurable syntactic information, with prosodic features reducing syntactic uncertainty in spontaneous conversations by up to 10.2%. Our findings offer new empirical support for several theoretical accounts of the syntax-prosody interface.

1 Introduction

The paper addresses the debated relationship between prosody and syntactic structure by introducing an information-theoretic framework and evaluating it on speech-text corpora. Its results show measurable syntactic information in prosody and support several theories of the syntax-prosody interface.

  • Prosody supports syntactic processing in acquisition, production, and comprehension, including constituent signaling and parse disambiguation.
  • The syntax-prosody relationship is established but its degree and manner remain debated across competing theories and empirical approaches.Existing evidence often comes from controlled case studies, older statistical methods, or simplifying assumptions that omit text context.
  • The framework measures syntactic information as the mutual information between prosodic features and structural representations.It is structure-agnostic, contextual, and modular, allowing analysis of individual prosodic features or structural components.
  • The experiments use modified T5 models to predict linearized syntactic parses from text, prosody, or both, examining duration, pauses, genres, and structural components.
  • 10.2%: word duration and inter-word pauses reduce uncertainty over syntactic representations by up to 10.2%.The features primarily signal boundary location, carry more information in spontaneous than planned speech, and are redundant with information in text.
  • The findings provide empirical support for Prosodic Bootstrapping and Indirect Reference accounts of the syntax-prosody interface.The methods are presented as extending beyond previous-generation statistical approaches to quantify dependencies at the interface.

2 The Syntax–Prosody Interface

The paper frames syntax-prosody theories as differing in how directly and strongly syntax and prosody interact. It connects these theoretical predictions to measurable informational relationships between prosodic cues and syntactic structure.

  • Theories of the syntax-prosody interface differ in the degree of causal linking they posit between prosodic and syntactic well-formedness.
  • Direct Reference Theory (DRT): Direct Reference Theory proposes that prosodic rules operate directly on syntactic parses, predicting prosody that is isomorphic to syntax.Pre-boundary lengthening at syntactic constituent edges is cited as supporting phonetic evidence.
  • Indirect Reference Theory (IRT): Indirect Reference Theory treats syntax and phonology as autonomous domains linked by a mediating mechanism, allowing mismatches when phonological constraints require them.Match Theory formalizes the mapping through constraints that can be overridden under Optimality Theory.
  • Prosody-Driven Syntax (PDS): Prosody-Driven Syntax proposes a bidirectional interface in which prosodic well-formedness can influence syntactic operations such as movement or scrambling.The account does not make strong predictions about the total degree of informational overlap.
  • Prosodic Bootstrapping: Prosodic Bootstrapping predicts non-zero syntactic information in prosody because infants use prosodic markers to infer phrasal structure and constrain lexical segmentation.
  • Prior quantitative work measures informational redundancy between prosody and non-prosodic channels, but continuous-valued predictions can produce unbounded entropy estimates.

3 Modeling Framework

The modeling framework represents text, syntax, and prosody as aligned random variables and estimates their informational relationships with probabilistic language models. It measures both prosody’s overall syntactic information and its information beyond word identities.

  • The framework treats syntax, prosody, and text as random variables and estimates syntactic information through their mutual information.
  • Random variables: Words are vocabulary elements, texts are word sequences, and syntactic units are phrasal-category symbols or bracket markers.Linearized constituency parses represent syntactic structure as symbol sequences.
  • Random variables: Prosodic units are real-valued features represented as sequences aligned with text, corresponding here to word duration and pause length in milliseconds.
  • Mutual information: I(P, S) measures how much knowing prosody reduces uncertainty about syntactic structure by subtracting conditional entropy from unconditional entropy.
  • Conditional information: I(P, S | W) tests whether prosody carries syntactic information beyond word identities by conditioning both entropies on text.A positive value indicates additional syntactic information not carried by text.
  • Entropy estimation: Cross-entropy upper-bounds entropy, and Monte Carlo estimates use model surprisal because the ground-truth distribution is unknown.Estimator quality improves as the model distribution becomes more similar to the ground-truth distribution.
  • Model estimation: The unconditional syntactic distribution is estimated with a decoder-only model, while conditional distributions use an encoder-decoder model given text, prosody, or both.

4 Methods

The study estimates syntactic structure from text, prosody, or both using multimodal encoder-decoder language models. It applies this pipeline to English planned and spontaneous speech datasets, operationalizing syntax with constituency parses and prosody with duration and pauses.

  • Model and estimation: The pipeline trains encoder-decoder language models to predict linearized syntactic structures from text, prosody, or no input.Sentence-level cross-entropies support entropy estimates for the corresponding conditional distributions.
  • Model and estimation: The model uses a base-sized T5 architecture with a newly constructed prosody encoder for prosodic inputs.The prosody encoder is shallower and narrower than T5’s text encoder, with 2 layers, 2 heads per layer, and hidden dimensionality 8.
  • Model and estimation: Prosodic magnitude and absolute position are encoded with fixed sinusoidal functions, concatenated, and passed through the transformer stack.Prosodic feature magnitudes are binned in 10ms increments, and the encoder output is projected to the text encoder’s hidden-state dimensions for cross-modal fusion.
  • Model and estimation: For joint inputs, cross-attention fuses text queries with prosody keys and values before decoding syntactic parse tokens.The model also adds one decoder-only token for each part-of-speech label.
  • Data and representations: The experiments compare planned audiobook speech from LibriTTS with spontaneous speech from CANDOR.LibriTTS uses an aligned clean subset, while dataset statistics are reported in Table 1.
  • Data and representations: Syntax is represented with punctuation-removed, linearized Penn Treebank-style constituency parses at full-parse and bracket-only granularities.The study measures word duration and inter-word pauses from Montreal Forced Aligner word boundaries, and tests paired surprisal differences with Wilcoxon signed-rank tests.

5 Results

Pause and duration carry statistically significant syntactic information across planned and spontaneous speech, especially about phrase boundaries. Prosodic information is greater in spontaneous speech, but appears largely redundant with information in words.

  • Pause and duration carry measurable syntactic information in both spontaneous and planned speech, with positive sequence-level mutual information significant at p < 10^-4.
  • Duration carries up to 3.45 nats and pause up to 2.06 nats, compared with about 30 nats carried by words alone.
  • After controlling for information already carried by words, pause and duration provide little or no significant additional syntactic information in planned speech.The authors hypothesize that relevant disambiguating sentences and constructions may be sparse, and note that the framework may underestimate additional information.
  • Prosody carries more syntactic information in spontaneous than planned speech, and the cross-genre difference remains significant after sentence-length matching.
  • Phrase boundaries account for most prosodic syntactic information: up to around 50% for duration and 60% for pause.In spontaneous speech, boundaries account for 1.42 of 3.45 duration nats and 1.20 of 2.05 pause nats; in planned speech, the corresponding values are 0.84 of 1.65 and 0.75 of 2.06.

6 Sentence Features and Disfluencies

Sentence-level analyses show that prosodic phrase-boundary information is associated with length and syntactic depth in spontaneous speech, while cross-genre differences persist after length control. Disfluencies have limited impact once sentence length is controlled.

  • Length and parse depth are weakly inversely correlated with prosodic syntactic information in planned speech, with |r| < 0.05 across conditions.The spontaneous-speech correlations become significant and moderate when phrase-boundary information is considered.
  • In spontaneous speech, longer and syntactically deeper sentences contain greater phrase-boundary information in both pause and duration.For spontaneous speech, correlations with length are r = 0.52 for pause and r = 0.42 for duration; correlations with depth are r = 0.42 and r = 0.39.
  • The spontaneous-versus-planned prosodic information difference remains significant after exact matching on sentence lengths from 5 to 25 words.
  • Sentence-medial repetitions and filled pauses are retained, and 19.8% of processed CANDOR sentences contain disfluencies.Disfluent sentences average 17.9 words versus 11.99 for fluent sentences; length-controlled analyses identify length as the primary driver of the information difference.

7 Discussion

The results indicate that prosody carries syntactic information mainly through phrase-boundary cues, with stronger relative information in spontaneous speech and substantial redundancy with words. These findings support indirect reference, Prosody-Driven Syntax, and prosodic bootstrapping within the study’s scope.

  • Implications for linguistic theories: The results support Indirect Reference Theory because duration and pauses carry syntactic information but only up to about 10% of total structural uncertainty.This contrasts with Direct Reference Theory’s prediction of very high informational overlap.
  • Implications for linguistic theories: Prosody-Driven Syntax predicts, and the results show, higher syntactic information in spontaneous than planned speech.I(P, S) and U values were higher in spontaneous speech, with U higher by up to ≈6%p.
  • Implications for linguistic theories: Even without knowing the uttered words, a learner could extract useful syntactic information from prosody alone, supporting prosodic bootstrapping.This implication follows from the measurable syntactic information carried by prosodic features.
  • Prosody as an error-correcting code: Prosody primarily signals phrase boundaries rather than phrase identities, although pitch and other unstudied features may contribute differently.The study examined only duration and pauses, so the conclusion does not cover all prosodic features.
  • Prosody as an error-correcting code: Prosody carries largely redundant syntactic information when word identity is known, suggesting an error-correcting role for syntax.Syntactic information was higher in disfluent than fluent spontaneous sentences, possibly reflecting real-time prosodic compensation.
  • Generalization beyond English: The study’s findings do not guarantee cross-linguistic generalization, and prosody–word redundancy may vary with phonological richness and word-order flexibility.The authors predict that high-level findings may extend cross-linguistically but identify redundancy as potentially variable.

8 Conclusion

The paper develops a pipeline for measuring syntactic information in prosodic features and finds statistically significant syntactic information in duration and pauses. It presents the approach as a machine-learning method for studying prosody–syntax interactions across larger datasets and languages.

  • The paper designs and implements a pipeline to measure syntactic information content in word duration and pauses.
  • The experiments confirm that prosodic features carry measurable, statistically significant syntactic information content.
  • The pipeline offers machine-learning methods for measuring prosody–syntax interactions over large corpora and future datasets and languages.

Limitations

The study’s estimates and conclusions are constrained by model estimation, feature coverage, transcript and parse quality, tokenization, and English-only data. Several design choices may affect entropy or mutual-information estimates and limit generalization.

  • Limitations with our approach: Cross-entropy estimates are biased and provide an upper bound on actual uncertainty when predicting syntactic structure from text and prosody.The estimate depends on model quality: better prediction of S can lower estimated entropy.
  • Limitations with our approach: Duration and pauses may contain redundant syntactic information because they were measured separately rather than jointly.The authors suggest combining the features or estimating their redundant information in future work.
  • Limitations with our approach: Gold planned-speech transcripts and ASR silver spontaneous-speech transcripts may make less syntax predictable from text and affect mutual-information estimates.
  • Model limitations: The model may leak probability mass to illegal tokens because it uses the full pretrained vocabulary despite an output space limited to 66 syntax tokens.This can adversely affect entropy estimates.
  • Model limitations: The text–prosody fusion architecture may be improvable through larger prosody encoders, earlier fusion, and decoupled tokenizers or embeddings.
  • Other experimental limitations: The study examines only duration and pauses, uses silver parses, and evaluates English, so untested features, parse errors, and cross-linguistic differences remain open.Other features include pitch, loudness, tempo, and prominence; broader multilingual evaluation is needed for generalizability.

Ethical Considerations

The work primarily uses speech data, transcriptions, and automatically generated NLP annotations, while acknowledging relevance to language-processing systems that may be used maliciously. The authors also disclose commercial LLM use during research and writing.

  • The work mainly concerns speech data, transcriptions, and automatically generated annotations from NLP systems.
  • The authors acknowledge that language-processing systems related to this work could potentially be used for malicious purposes.
  • The authors disclose using commercial LLMs for brainstorming, coding, writing assistance, experiment design, implementation, and paper writing.

A Probability Leakage

The analysis measures probability leakage into illegal parse tokens and quantifies its inflation of syntactic-uncertainty estimates. Leakage is substantial for full-parse autoregressive models but negligible for bracket-only models.

  • Measurement procedure: The models’ architecture does not explicitly disallow illegal parse tokens, so their non-zero probabilities inflate H estimates.Illegal tokens include items that are not POS tags, phrase tags, or brackets.
  • Measurement procedure: The leakage analysis randomly samples 1,000 sentences per trained model and averages illegal-token probability across prediction steps.The aggregate probability is divided by the total number of prediction steps.
  • Leakage estimates: 15.7% and 11.7% probability mass leaks into illegal tokens for full-parse models on LibriTTS and CANDOR, respectively.Under a uniform distribution across 32k illegal tokens, this inflates H estimates by 1.92 and 1.46 nats.
  • Leakage estimates: Bracket-only models leak no more than 0.001% probability mass, producing negligible inflation.This contrasts with the larger leakage observed in full-parse autoregressive models.
  • Leakage estimates: 0.07% to 1.3% leakage in the remaining models corresponds to approximately 0.01 to 0.18 nats of H-estimation inflation.The lowest leakage occurs for H(S | Ppause) on CANDOR, while the highest occurs for H(S | Pdur) on LibriTTS.

C.2 Effects of Disfluency on I(S, P)

Disfluent CANDOR sentences initially show higher syntactic information and uncertainty than fluent sentences, but sentence length explains most of the uncontrolled difference. Length-controlled and regression analyses substantially reduce the apparent fluency effect.

  • Table 4: Table 4 reports I in nats and U for fluent, disfluent, and full CANDOR datasets under uncontrolled and length-controlled sampling.The length-controlled subset contains 58k matched pairs with mean length 17.28 words.
  • Uncontrolled comparisons: 3.91 versus 3.31 nats of duration-based I with syntax occurs in disfluent versus fluent sentences, while pauses show 2.61 versus 1.90 nats.These uncontrolled differences coincide with mean sentence lengths of 17.90 words for disfluent sentences and 11.99 words for fluent sentences.
  • Length control: Sentence length reduces the cross-subset I and U gap from up to 40% of the total value to within 7.5% after control.The analysis uses length-controlled subsets to separate sentence-length effects from fluency differences.
  • Regression analysis: Adding binary fluency information to length-only regressions increases explanatory power by less than 0.01 in R^2 across all conditions.The baseline predicts Δι from sentence length, while the augmented model adds fluent-versus-disfluent status.
Loading 2608.30260v1…