Source-linked AI summary
When Does a Laugh Begin? Structured Annotator Disagreement in Temporal Laughter Localization
Eyal Hanania, Daniel Arkushin, Naveh Ayal, Jonathan Benvenisti, Amos Bercovich, Elie Zemmour, Sahar Froim
TL;DR
Temporal laughter localization usually treats one annotation as ground truth despite structured disagreement about boundaries and chuckles. This paper re-annotates SMILE-Temporal, characterizes that disagreement, and proposes evaluation against the human annotation distribution with conformally calibrated tolerance bands, finding that offsets are harder to localize and chuckles more contested.
Problem
Temporal laughter localization benchmarks typically evaluate systems against a single reference annotation even though annotators differ systematically on boundaries and whether faint chuckles constitute events.
Method
The paper re-annotates 672 SMILE-Temporal videos with multiple independent annotators, analyzes structured disagreement, and evaluates predictions against annotator distributions using conformally calibrated tolerance bands.
Results
Disagreement is systematic: offsets are 1.3–1.7× noisier than onsets, and chuckles are contested in 77% of events versus 20% for full laughs.
Takeaways & Limitations
Temporal affect evaluation should use all annotators, ideally their distribution with tolerance calibrated to human spread, rather than a single reference.
Takeaways & Limitations
The analysis is based on predominantly acoustically cued laughter, uses tolerance bands calibrated to one annotation pool and protocol, and evaluates real systems only with one public audio-only localizer.
Abstract
from arXiv · showhide
Annotators routinely disagree on laughter boundaries and subtle chuckles, yet temporal laughter localization typically evaluates against a single reference annotation. We show that this disagreement is structured rather than random noise. Re-annotating the SMILE-Temporal benchmark (672 videos, 1,683 events) with 3-5 annotators per video (alpha = 0.757), we find systematic patterns: disagreement is 1.73 times larger at offsets than onsets, far more common for chuckles than full laughs (77% vs. 20%), and predictable from event attributes (AUC = 0.831). Evaluating against a single annotator breaks down under this structure: system scores shift by 0.246 F1 depending on the chosen ground truth, correctly ranking systems only 69.7% of the time (vs. 80% against all annotators). We propose a disagreement-calibrated evaluation that scores predictions against the full annotator distribution using conformally calibrated tolerance bands (wider at offsets, 0.727s, than onsets, 0.5s). The per-annotator annotations and analysis code are available at https://github.com/WSCSports/SMILE-Disagreement .
1 Introduction
The paper argues that annotator disagreement in temporal laughter localization is structured, especially around offsets and chuckles, making single-reference evaluation unreliable. It introduces a multi-annotator dataset and proposes evaluation against the human distribution.
- Annotators disagree systematically about laughter boundaries, with clear-laugh onsets aligning more often than gradual offsets and faint chuckles sometimes omitted entirely.
- The study re-annotates SMILE-Temporal with multiple independent annotators and releases SMILE-Disagreement as a per-event, multi-annotator laughter-boundary corpus.
- 1.3–1.7× greater disagreement occurs at offsets than onsets, while chuckles are contested in 77% of events versus 20% for full laughs.
- Disagreement remains structured after controls and is predictable from event attributes, indicating that ambiguity is a property of laughter events rather than annotator caprice.
- 0.246 average frame-F1 variation arises from choosing different annotators as ground truth, and single-GT ranks the better system correctly only 69.7% of the time.
- The proposed protocol scores predictions against the full annotator distribution using a soft target and conformally calibrated human-spread tolerance band.
2 Related Work
The paper situates its contribution within perspectivist annotation and soft temporal-boundary research, then introduces a publicly released multi-annotator laughter corpus. Its scoped novelty is the combination of per-event onset/offset distributions, multiple annotators, public release, and affect-video laughter.
- Human label variation and perspectivism: Perspectivist research treats annotator disagreement as genuine, reproducible variation rather than annotation error and evaluates against that variation.
- Temporal boundary ambiguity in video: Prior temporal-boundary work shows that action extents are soft, while this paper emphasizes distinct onset and offset uncertainties for affective events.
- Dataset contribution: The dataset re-annotates SMILE-Temporal with independent onset and offset marks and is presented as the first publicly released per-event multi-annotator temporal-boundary corpus for laughter in affect video.
- Annotation protocol: Six trained annotators labeled events independently with onset, offset, intensity, source, and speaker attributes.
- Dataset composition: The benchmark includes 672 videos, 4,554 retained annotator marks, and 1,683 matched distinct events.
- Positioning: The corpus differs from adjacent resources by combining per-event boundaries, at least three annotators, public release, and the laughter/affect-video domain.
4 Analysis and Findings
Multi-annotator agreement is reliable, but disagreement is structured by boundary position and laughter intensity. Offsets are fuzzier than onsets, chuckles are more contested than full laughs, and these patterns remain robust across controls and are predictable from event attributes.
- Agreement stability: Krippendorff’s α = 0.757 at 100 ms, with agreement stable across frame sizes and annotator counts.Video-level bootstrap gives α ∈[0.742, 0.769], while video-level agreement on laughter presence is 77.2%.
- Boundary disagreement: 1.73×: mean offset spread is 0.253 s versus 0.146 s for onsets.The asymmetry holds within audience and individual-speaker events, consistent with a faster acoustic attack and slower decay.
- Intensity disagreement: 77% vs. 20%: chuckles are disagreed about far more often than full laughter.The gap persists within acoustic-only events and across duration controls, indicating an intensity-linked pattern rather than a modality or duration artifact.
- Robustness checks: 8.59×: a logistic model assigns chuckles these odds of disagreement after controlling for log-duration, position, and modality.Subsampling annotators and changing the event matcher leave the disagreement gap essentially unchanged.
- Predictability: AUC 0.831: disagreement is predictable from event attributes under grouped 5-fold cross-validation.The intensity-free model still reaches AUC 0.757, and acoustic loudness analyses support a non-circular intensity effect.
- Boundary disagreement: 1.6–2.3×: within every duration quartile, chuckle offset spread exceeds full-laugh spread.Intensity also increases onset and offset fuzziness, while offset–onset asymmetry remains within both intensity groups.
5 Disagreement-Calibrated Evaluation
The paper replaces arbitrary single-reference evaluation with scoring against the full annotator distribution, combining soft overlap with human-calibrated boundary tolerance. This protocol improves evaluation reliability while preserving sensitivity to structured human disagreement and exposing where real systems exceed it.
- 5.2 Scoring against the human distribution: The full protocol uses human annotations alone to calibrate tolerance bands, while misses and hallucinations remain penalized through soft-IoU.Predicted boundaries outside human events contribute no band score, but their frames affect the soft-IoU denominator.
- 5.1 A single ground truth is arbitrary: A single annotator is an arbitrary reference: human–human frame-F1 averages about 0.681, while system scores vary by 0.246 F1 across reference choices.Single-ground-truth evaluation implicitly treats perfection as attainable despite realistic annotator disagreement.
- 5.2 Scoring against the human distribution: At α=0.1, conformal bands are ±0.5 s for onsets and ±0.727 s for offsets, making evaluation more lenient where annotators disagree more.Widths are calibrated from held-out human boundary residuals and inherit the onset–offset asymmetry.
- 5.2 Scoring against the human distribution: Calibration provides marginal human-boundary coverage, not a model coverage guarantee, because pooled residuals are only assumed exchangeable and model residuals need not match human residuals.Testing across annotator identities achieved mean coverage of 0.895 against a 0.90 target, while other clustering assumptions remain unverified.
- 5.2 Scoring against the human distribution: The protocol combines frame-level soft-IoU with an event-level tolerance-band score, computed per video and macro-averaged across videos.The unweighted composition prevents long videos from dominating and gives partial credit for partially agreed events.
- 5.3 Validation: does it rank systems better?: Single-GT ranks synthetic systems correctly 69.7% of the time, whereas multi-annotator rules reach about 80%; the full protocol achieves 79.7%.Most ranking improvement comes from removing the arbitrary reference choice; the tolerance band additionally refuses to rank differences smaller than human disagreement.
- 5.3 Validation: does it rank systems better?: Using more annotators improves ranking reliability from 70.6% with one to 75.6% with two and 77.4% with three, approaching 78.4% with all annotators.Most of the benefit arrives from the second and third annotators.
- 5.4 On a real localizer: On a real audio-only localizer, human disagreement predicts machine error: misses are 28 points higher for existence-contested events and 25 points higher for chuckles than full laughs.Offset coverage remains below the human construction target under the adaptive band, so the protocol identifies boundary-specific shortfalls rather than guaranteeing model coverage.
6 Limitations
The analysis is limited to acoustically cued laughter in SMILE-Temporal, uses one annotation pool and protocol for tolerance calibration, and evaluates only one public audio-only localizer.
- SMILE-Temporal laughter is predominantly acoustically cued, preventing meaningful analysis of audio–visual disagreement.
- Conformal tolerance-band widths are calibrated to one annotation pool and protocol, so they are not universal.
- Real-system evaluation is limited to one public audio-only laughter localizer.
7 Conclusion
The paper concludes that single-ground-truth evaluation misrepresents temporal laughter localization because annotator disagreement is structured and should be modeled. It recommends evaluating temporal affect against all annotators, ideally using conformally calibrated distribution-aware tolerances.
- Single-ground-truth evaluation misrepresents temporal laughter localization because boundary disagreement is structured rather than random.
- The dataset enables the paper’s findings and disagreement-aware evaluation protocol.
- Practitioners should evaluate temporal affect against all annotators, ideally using the full distribution with conformally calibrated tolerance.
Qualitative gallery
The qualitative gallery spans unanimous full laughs, shared events with differing boundaries, and existence-contested clips dominated by chuckles. Its rows show annotator labels over time, with colors distinguishing laugh intensity and dashed lines marking no-laughter judgments.
- The gallery spans unanimous full laughs, boundary-disagreement cases, and existence-contested clips dominated by chuckles.
- Each row represents one annotator, with seconds on the x-axis; blue marks full laughs, orange marks chuckles, and dashed lines mark no laughter.
- The per-annotator profile lists videos, events per video, existence, chuckle and audience shares, onset and offset bias, and peer F1.
Per-annotator consistency profile
The per-annotator profiles show no outlier annotator, indicating that the analyzed disagreement structure is a property of the videos rather than one annotator’s labeling style.
- No annotator is an outlier, so the disagreement structure is attributed to the videos rather than one labeling style.
Annotator-budget curve
Figure 5 examines how many annotators are needed to build a soft target for reliable system ranking, while Figure 5’s whiskers show variability across seeds.
- Ranking reliability is plotted against the number of annotators used to construct the soft target.The analysis uses videos with at least four annotators.
- The second and third annotators provide most of the reliability gain.This result offers practical guidance for planning annotation budgets.
- Whiskers report the minimum–maximum ranking reliability across seeds.
Protocol illustration
The protocol illustrates how multiple annotator boundaries become a soft target and how conformally calibrated tolerance bands represent human disagreement, with wider bands at offsets than onsets.
- Calibration: Calibration holds out one annotator while testing uses the median of all annotators, making the applied band conservative.
- Calibration: The pooled calibration quantiles remain similar across co-marker-count strata.For k=2, half-widths are ±0.535 s at onset and ±0.712 s at offset; for k≥3, they are ±0.467 s and ±0.734 s.
- Protocol illustration: Four annotators mark the same laugh, and their per-frame fraction forms the soft target.Blue marks denote full laughs and orange marks denote chuckles.
- Protocol illustration: The protocol applies conformally calibrated tolerance bands whose widths match observed human disagreement.The displayed bands are ±0.5 s at the onset and ±0.727 s at the offset.
- Adaptive variant: An adaptive variant narrows bands where annotators agree and widens them where they disagree.Its median half-widths are ±0.368 s at onsets and ±0.563 s at offsets, compared with global widths of ±0.5 s and ±0.727 s.