Negation Beyond the Verbal Channel: Temporal Multimodal Correlates in Dialogue

1 Introduction

• Negation research has largely focused on linguistic cues and scope, although negative expressions can also accompany nonverbal behavior such as head movements.Dialogue partners also produce behavioral responses, motivating examination of behavior around explicit spoken negation cues.Source: §1 · p. 1
• The study asks whether multimodal behavior around negation cues distinguishes those contexts from matched controls without lexical or acoustic input.Classification is used as a predictive probe of how cue-centered contexts differ across temporal context, modality, and interactional source.Source: §1 · p. 1
• The research questions examine how cue-centered distinguishability changes over time, which behavioral modalities contribute, and how predictive information is distributed between speaker and partner.They also ask which event types are most sensitive to their temporal relation to the spoken word.Source: §1 · p. 2
• The study treats explicit negation cues as embedded in coordinated, time-dependent human behavior rather than as isolated linguistic labels.Source: §1 · p. 2

2.1 Negation in NLP

• Negation NLP has chiefly addressed cue and scope identification, while linguistic-input approaches leave concurrent nonverbal behavior largely outside their modeling objective.Prior approaches range from rule-based and feature-based systems to neural and transformer models.Source: §2.1 · p. 2

2.2 Multimodal Negation

• Research links spoken negation to head and manual gestures, prosody, agreement, refusal, scope disambiguation, and covert negative meaning.Computational work has also examined negation in vision-language models and video retrieval.Source: §2.2 · p. 2

2.3 Temporal Multimodal Interaction

• Speech and behavior are temporally coordinated without strict synchrony, and interlocutor feedback makes behavioral source relevant to dialogue.The section identifies a gap in systematic characterization of negation-cue correlates using fine-grained aligned streams across modalities, then addresses it with VR interaction event logs.Source: §2.3 · pp. 2–3
• The corpus comprises German-language VR survey interviews with headset-recorded gaze, face, head, body, hand, and finger behavior aligned to transcribed words.Negation annotation identifies 964 cue words across 57,971 spoken words; the cue inventory is skewed toward nicht.Source: §2.3 · p. 3

4 Task Formulation

• The task is binary classification: negation-cue words are positive anchors, and words outside annotated cue or scope spans are negative anchors.The classifier receives surrounding multimodal behavior, not the anchor's lexical identity or acoustic information.Source: §4 · p. 3
• For each anchor, the context collects events within left and right time bounds, which can be symmetric, asymmetric, or offset from the target event.The interval is not defined relative to the full word duration, avoiding a window length that varies with anchor-word duration.Source: §4 · p. 4
• Behavioral events are evaluated from the anchor speaker, the dialogue partner, or both, allowing comparison of predictive information by interactional source.Source: §4 · p. 4

5.1 Dataset Construction and Conditions

• Each of 964 cue words is paired with a same-speaker control from the same interview and audio chunk when possible, yielding 1,928 anchors reused across conditions.Controls exclude annotated cue or scope words and are more than one second from their paired cue; temporal windows and speaker-only, partner-only, and dyadic sources are varied.Source: §5.1 · p. 4

5.2 Input Representations

• Inputs include eight behavioral modalities, excluding lexical identity, audio, absolute timestamps, participant identifiers, and database metadata.Fixed-grid inputs are normalized using training data and interpolated to 32 time steps, while the event Transformer uses irregular event sequences.Source: §5.2 · p. 4

5.3 Comparison Models

• The comparison spans random-convolution, feature-, dictionary-, shapelet-, interval-, ensemble-, neural sequence-, and motion-language-model approaches.The comparison tests robustness across modeling approaches rather than introducing a new classification architecture.Source: §5.3 · pp. 4–5

5.4 Cross-Validation and Evaluation

• Stratified group-based 10-fold cross-validation keeps VR experiments separate across train, validation, and test, with AUROC primary and macro-F1 additionally reported.Data-dependent preprocessing and model selection do not access the test fold; results are averaged over folds with bootstrap 95% confidence intervals.Source: §5.4 · p. 5

6 Results

• The analysis first compares classifiers for robust cue-centered discrimination, then uses one high-performing probe for detailed analyses.Across models, the ±0.1 s context performs worst, while the three wider windows yield similar AUROC (.660-.663).Source: §6 · p. 5, §6.1 · p. 5

6.1 Model Comparison

• .728 AUROC: ROCKETPFN leads (HIVE-COTE 2.0 .723; DRCIF .722), while macro-F1 is nearly tied between HIVE-COTE 2.0 (.666) and ROCKETPFN (.665).Several models perform comparably, and ROCKETPFN is selected as a computationally efficient representative probe for further analyses.Source: Figure 2 · p. 5, §6.1 · p. 5

6.2 Temporal Profile of Cue-Centered Discriminability

• Speaker-only AUROC reaches .660 for [−0.1, 0] s and .742 at 2.5 s before cue onset, while performance largely saturates between 1.5 and 2.5 s.Extending pre- or post-onset windows to 5 s does not further improve performance; the cumulative windows measure usable evidence up to each extent.Source: §6.2 · p. 6, Figure 3 · p. 6
• In disjoint 0.5 s windows, speaker performance peaks adjacent to cue onset and declines with distance, but remains above chance in every window.The two onset-adjacent windows show similar discrimination, with ∆AUROC = .010 for [0, .5] s relative to [−.5, 0] s (95% CI [−.007, .024]).Source: §6.2 · p. 6, Figure 4 · p. 6

7.1 Robustness to Alternative Control Matching

• Within-scope controls lowered AUROC from 0.730 to 0.677 (∆AUROC = −0.053, 95% CI [−0.096, −0.012]), but did not eliminate cue-centered discriminability.Stricter and weaker controls reached 0.723 and 0.722 AUROC, respectively, versus 0.712 under default controls.Source: §7.1 · p. 7

7.2 Speaker and Dialogue-Partner Contributions

• Speaker behavior outperformed partner behavior (mean AUROC .712 versus .614); partner performance remained above chance but nearly flat, while dyadic contexts performed worst.The authors suggest that the dyadic result may reflect fixed-grid averaging that removes actor identity.Source: §7.2 · p. 7

7.3 Modality Contributions

• Facial ablation produced the largest reliable performance decrease (∆AUROC = .061), while lower-face ablation decreased AUROC by .026; substantial discriminability remained without facial features.Other modalities shifted AUROC by no more than .009, and sparse finger-stream coverage may qualify those ablation results.Source: §7.3 · pp. 7–8, Figure 5 · p. 7

7.4 Temporal Event Robustness

• Joint timing warps reduced AUROC by .030–.051 from the .734 baseline, with rescaling more damaging than rigid shifting and dilation more damaging than compression.Single-modality timing effects were smaller and less consistent; facial timing had the largest reported single-modality effect (∆AUROC = .023).Source: §7.4 · p. 8, Figure 6 · p. 8

• Speaker-side probes reached up to .750 fold-mean test AUROC without lexical or acoustic input when distinguishing cue-centered from matched-control contexts.Source: §8 · p. 8
• Speaker discriminability increased with context up to approximately 2.5 s before and 1.5 s after cue onset, while sliding-window performance peaked immediately around the cue.Source: §8 · p. 8
• Partner behavior reached .634 AUROC but remained nearly constant across temporal extents and sliding windows, consistent with diffuse interactional or contextual information.Source: §8 · p. 8
• Facial features produced the largest modality-ablation effect, while lower-face removal reduced AUROC by .026 and left substantial predictive information.Source: §8 · p. 8
• Timing perturbations reduced AUROC by .030–.051, showing that the trained probe is sensitive to temporal organization of its inputs.Source: §8 · p. 8

Limitations

• The study covers 27 human-human interviews in one VR survey setting, so generalization to other settings, tasks, populations, or languages remains unclear.Automated cue detection and word-level timestamps may also affect fine-grained temporal analyses; the study focuses on explicitly marked cues, and about 80% of cue instances are nicht.Source: Limitations · p. 9

Ethical Considerations

• Released research data are limited to negation event windows and stripped of identifiers, absolute timestamps, and linkage metadata to reduce re-identification risks.Original continuous recordings and metadata linking observations to participants or sessions are not released.Source: Ethical Considerations · p. 9

A Annotation Model Configurations

• Negation cues and scopes are annotated with German-configured D-NEG, while CRISPERWHISPER word-level timestamps align transcripts with the multimodal event stream.The German syntax-aware D-NEG setup uses separate cue-detection and scope-resolution checkpoints; CRISPERWHISPER uses German decoding with five-beam search.Source: A Annotation Model Configurations · pp. 12–13

B.1 Multimodal Feature Inventory

• The fixed-grid feature inventory distinguishes recorded measurements from derived motion features across gaze, facial, rigid-body, and finger-tracking channels.Ocular expression blendshapes from the facial tracker are assigned to the facial modality, not the dedicated eye-tracking modality.Source: §B.2 · p. 13

B.2 Shared Input Processing

• Fixed-grid inputs interpolate continuous channels onto 32 time points, retain modality-support indicators, and standardize numeric features using training-fold statistics.The resulting representation has shape X ∈R698×32; the irregular-event Transformer instead preserves the event sequence and omits modality-presence channels.Source: §B.2 · p. 13

B.3 Cross-Validation, Optimization, and Decisions

• Recording sessions are kept within a single split across ten stratified group partitions, with separate test, validation, and training partitions in each outer fold.Each fold uses one partition for testing, the next for validation, and the remaining eight for training.Source: §B.3 · p. 13
• AUROC is the primary metric, computed from continuous model scores; neural checkpoints are selected by minimum validation loss.Probabilistic classifiers use a 0.5 threshold for threshold-dependent metrics unless stated otherwise; no positive-class weighting is used unless specified.Source: §B.3 · p. 14

B.4 Random-Convolution Models

• MINIROCKET, MULTIROCKET+HYDRA, sparse MULTIROCKET+HYDRA, and SELF-ROCKET use random-convolution features followed by ridge or selected linear classifiers.Sparse MULTIROCKET+HYDRA selects features and classifiers by inner cross-validation; SELF-ROCKET evaluates representation and pooling combinations before retaining a candidate.Source: §B.4 · p. 14

B.5 Feature-, Dictionary-, Shapelet-, and Interval-Based Models

• The feature-, dictionary-, shapelet-, and interval-based comparisons include CASTOR, Arsenal, Temporal Dictionary Ensemble, WEASEL 2.0, MRSQM, and DRCIF.The appendix specifies model-specific settings including shapelet and interval sampling, selected channels or features, and classifier configurations.Source: §B.5 · pp. 14–17
• WEASEL 2.0 uses raw and first-difference variants, while MRSQM uses zero SAX and five SFA representations and DRCIF uses 200 trees.The appendix also describes HIVE-COTE 2.0 with four standard components and fourth-power CAWPE weighting.Source: §B.5 · pp. 15–16
• Table 4 reports mean test performance over ten folds for 20 classifiers across four temporal windows, with bootstrap 95% confidence-interval half-widths.The caption identifies a speaker-side multimodal event discrimination comparison but supplies no individual cell values.Source: Table 4 · p. 17

B.6 Random-Convolution Features with Tabular Inference

• ROCKETPFN averages predictions from ten ROCKET feature groups, while MASHT and adaptive ROCKETPFN combine multivariate random-convolution features with TabPFN classification.MASHT uses a 2,000-feature budget on the balanced datasets; adaptive ROCKETPFN adds multiple signal views, shapelets, and selected feature experts.Source: §B.6 · pp. 17–18
• Validation and test rankings agree closely (Spearman ρ = .985; Kendall τ = .937), with six of 190 model pairs discordant.The top nine and bottom three ranks are identical across splits; middle-block models shift by at most two places.Source: Table 5 · p. 18

B.7 Neural Time-Series and Event Models

• INCEPTION-TCN uses modality-specific convolutional branches, while Compact Fusion-TCN pools global, pre-anchor, and post-anchor features with modality-presence indicators.The Inception branches use temporal convolutional blocks; Compact Fusion-TCN also applies modality dropout, input noise, and small temporal shifts during training.Source: §B.7 · pp. 18–19
• The Irregular-Event Transformer reconstructs masked modality observations during self-supervised pretraining before fine-tuning its Transformer encoder for classification.Pretraining uses the outer training split, and the pretrained backbone is frozen for the first three fine-tuning epochs.Source: §B.7 · p. 19

B.8 Motion-Language and Hierarchical Motion Models

• T2M-GPT compresses fixed-grid motion into discrete tokens, whereas T2M-GPT-V2 replaces the tokenizer with a continuous-latent VAE.Both use causal Transformer backbones and next-token or next-latent pretraining, respectively.Source: §B.8 · pp. 19–20
• MOTIONGPT adapts a motion-to-text encoder-decoder for label prediction, while MOTIONGPT3 uses continuous motion latents with a class-conditioned diffusion classifier.MOTIONGPT3 omits the language branch and cross-modal attention because textual descriptions are unavailable.Source: §B.8 · p. 20
• G-HTT divides each 32-position window into four clips and hierarchically models short-span pose structure and longer-span action structure.Its short-span block reconstructs the first four positions of each clip and predicts the remaining four.Source: §B.8 · pp. 20–21

B.9 Software

• The experiments use specified versions of Python, scientific-computing and machine-learning packages, and TabPFN.The listed software includes NumPy, scikit-learn, SciPy, aeon, Wildboar, MrSQM, PyTorch, Transformers, and TabPFN.Source: §B.9 · p. 21