Who Said What, and Will It Be Remembered? Evaluating Persistent Speaker Attribution Across Meetings

1 Introduction

• Reusable attribution requires each person to retain one identity across sessions so facts can be linked reliably across meetings.The paper distinguishes missing words from misattributed words, which can attach a fact, preference, or commitment to the wrong person.Source: §1 · p. 1
• The paper evaluates how a corpus-global identity map changes rankings and which lexical, local-speaker, and identity failures explain the result.Source: §1 · p. 1
• Speaker Identified cpWER (SI-cpWER) scores cpWER under one corpus-global speaker-ID map as the primary metric for reusable speaker attribution.Source: §1 · p. 1
• The benchmark evaluates five commercial diarize-then-identify cascades, two open academic baselines, and ThyVoice on NOTSOFAR clean and noise-augmented conditions and CHiME-6.It uses SI-cpWER, cpWER, DER/JER, WER, and speaker-clustering and overlap diagnostics; ThyVoice leads commercial SI-cpWER in all three conditions and the full-panel mean.Source: §1 · p. 2
• The end-to-end evaluation uses single-channel meeting audio and compares persistent with per-recording attribution before interpreting results through local-speaker and lexical diagnostics.Source: §1 · p. 2

2 Related Work

• Speaker-attributed ASR uses timestamp-reconciliation cascades, segment-first recognition, or recognizers that integrate attribution through multiple streams, speaker tokens, or structured generation.These arrangements have different failure boundaries: diarizing first can truncate words at segment boundaries, while diarizing afterwards must reconcile independent timelines.Source: §2 · p. 2
• Overlap stresses attribution systems because they must reconcile simultaneous speech, use diarization masks, or linearize concurrent speech; enrollment also requires target-speaker-only evidence.The passage notes that some systems do not explicitly handle overlap or discard, serialize, or filter it during data construction.Source: §2 · p. 2
• Persistent identity must prevent false splits that fragment one person’s history and false merges that attach different people’s evidence to one identity.Reliability depends on diarization, evidence selection, thresholds, and update policy; open cascades expose replaceable stages, unlike many hosted and end-to-end systems.Source: §2 · p. 2
• Per-recording metrics can remap anonymous speakers independently, whereas persistent ambient capture requires an identity namespace discovered online and stable across recordings.The evaluation separates lexical, local-speaker, per-recording, and persistent-attribution evidence while preserving each system’s exposed surface.Source: §2 · p. 2

3.1 Persistent-attribution task and headline metrics

• SI-cpWER scores post-identification labels under one corpus-global, one-to-one speaker map, unlike cpWER’s fresh per-sample assignment.The global assignment sums speaker-pair edit costs across samples while keeping sample boundaries intact; processing order can affect later assignments.Source: §3.1 · p. 3

3.2 Datasets and conditions

• NOTSOFAR supplies persistent participant aliases and overlapping office meetings; Clean preserves single-device audio, while Noisy adds ambient noise, babble, and transient disturbances.The held-out evaluation set has 129 meetings; Clean and Noisy share the same transcript, word timestamps, and speaker labels.Source: §3.2 · pp. 3–4

3.4 Supporting diagnostics and repeatability

• Ablations use the first 20 meetings in native order and provide directional—not full-corpus effect-size—evidence; three repeated ElevenLabs draws assess variation on NOTSOFAR.The main headline results use all 129 meetings; close NOTSOFAR margins motivated repeats, while CHiME-6 has a single run.Source: §3.2 · pp. 3–4, §3.4 · p. 7

3.2 Datasets and conditions

• CHiME-6 sessions are scored as serial chunks against a shared voiceprint bank, requiring identity consistency across chunks and preventing matches between disjoint session speakers.The two sessions contain 56,886 reference words; every system receives one raw channel, so absolute error rates are intended for comparative reading.Source: §3.2 · pp. 4–5

3.3 Systems and identity protocol

• Five commercial cascades and two open academic baselines share a PyannoteAI identity backend, while ThyVoice is evaluated end-to-end on raw meeting audio.The academic systems are SE-DiCoW and WhisperX; ThyVoice uses Qwen3-ASR with forced alignment, and comparison cascades enroll eligible speaker audio into shared voiceprints.Source: §3.3 · pp. 5–6

3.4 Supporting diagnostics and repeatability

• Supporting diagnostics separate lexical errors, local speaker evidence, and persistent identity drift; IAR captures asserted substitutions and insertions but not global identity drift.Within-recording clustering measures and overlap detection characterize pre-enrollment evidence, while SI-cpWER remains the persistent-attribution score.Source: §3.4 · pp. 6–7

4.1 Gated speaker evidence

• ThyVoice is an end-to-end, diarization-guided segment-first reference system that gates speaker evidence before voiceprint creation or updates.Its default stack uses DiariZen md-v2, MossFormer2, WeSpeaker SimAM-ResNet100, and Qwen3-ASR.Source: §4.1 · p. 7

4.2 Overlap repair and stream rematching

• Raw overlap is excluded from enrollment; bounded two-speaker regions are separated and rematched to clean local-speaker evidence before reuse.Streams are matched at cosine similarity 0.25, while silent, duplicated, ambiguous, or poorly matched streams are rejected.Source: §4.2 · p. 8

4.3 Pooled transcription evidence

• Clean segments and only successfully gated separated streams are stitched per speaker, independently transcribed, aligned, and mapped back to meeting time.Pooling provides broader phonetic and acoustic coverage than a single turn while avoiding hidden overlap and rejected separator artifacts.Source: §4.3 · p. 8

4.4 Persistent identity matching and updates

• A local speaker reuses an existing identity at cosine similarity ≥0.5, while profile updates require ≥0.7 and eligible, non-duplicated speech.The final transcript retains words on the original timeline with globally consistent speaker identities.Source: §4.4 · p. 8

5.1 Persistent attribution

• ThyVoice has lower SI-cpWER than every evaluated commercial cascade in all three conditions and the lowest full-panel mean.SE-DiCoW leads the full panel on both NOTSOFAR conditions, while ThyVoice leads on CHiME-6.Source: Table 2 · p. 9, §5.1 · p. 9

5.2 Diagnosing persistent attribution

• The diagnostic tables score available component or provider outputs, not additional end-to-end systems or a complete ThyVoice row.The analysis proceeds through per-recording attribution, local-speaker activity, and lexical recognition.Source: §5.2 · p. 9

5.2.1 Per-recording speaker attribution

• ElevenLabs has the lowest commercial cpWER in every condition, but its CHiME-6 SI-cpWER is about 30.0 points above cpWER, versus 1.5 points for ThyVoice.SE-DiCoW’s NOTSOFAR lead is already gone at CHiME-6 cpWER; the cpWER-to-SI-cpWER change is not strictly monotonic.Source: §5.2.1 · p. 9, Table 3

5.2.2 Persistent identity counts

• ThyVoice remains close to the 12-speaker reference on Clean and creates substantially fewer identities than most systems under noise.Identity count alone misses merges, unknown-speaker duration, and how speech is distributed; counts exclude explicit unknown-speaker labels.Source: Table 4 · p. 10, §5.2.2 · p. 10

5.2.3 Local speaker activity

• ThyVoice’s DiariZen md-v2 component has the lowest DER/JER on evaluated output surfaces in every condition.Word-labelled outputs use a fixed 0.25 s maximum same-speaker gap for offline DER/JER reconstruction; this does not change cpWER or SI-cpWER.Source: Table 5 · p. 10, §5.2.3 · p. 10

Enrollment-relevant speaker evidence

• ThyVoice’s DiariZen md-v2 combines high purity and coverage with substantial overlap recall, while noise increases merging and hides more overlap.These diagnostics indicate pre-enrollment risk but do not directly measure contamination inside a speaker embedding.Source: Table 6 · p. 11, Enrollment-relevant speaker evidence · p. 11

5.2.4 Lexical recognition

• ElevenLabs has the lowest WER on evaluated emitted word sequences in every condition, but that lexical lead does not carry through to persistent attribution.The table scores listed lexical surfaces, not end-to-end WER for parent systems; recognition errors carry into SI-cpWER.Source: Table 7 · p. 11, §5.2.4 · p. 11

5.3 Overlap-policy ablation

• Separation and rematching achieve the lowest cpWER and SI-cpWER among the tested overlap policies.Mixture ASR improves cpWER relative to omission when identity evidence stays non-overlapping; mixture-containing identity evidence yields substantially higher SI-cpWER despite similar pre-identification cpWER.Source: §5.3 · pp. 11–12, Table 8 · p. 12

Discussion

• SI-cpWER measures a requirement cpWER alone does not impose: carrying one identity across recordings.Identity resolution can add attribution error from incorrect local splits or merges, or repair local fragmentation by resolving clusters to one person.Source: §6 · p. 12
• Enrollment decisions depend on the exposed surface: simultaneous-speaker activity can support rejecting or repairing mixed evidence, unlike a single-speaker timeline during overlap.Word-labelled STT exposes less information downstream, without establishing what the provider detected internally; transcription audio and identity evidence are separate selection decisions.Source: §6 · p. 12
• WER, cpWER, and DER decompose different error types, so their profiles should be compared within a layer rather than added across metrics.Omissions reduce recall, while incorrect speaker assignment can attach a fact or action item to the wrong person; measuring that task-level harm remains future work.Source: §6 · p. 12

• NOTSOFAR meetings are single sessions processed serially against one shared voiceprint store in a fixed order, so results do not establish calibration over longer timescales or robustness to recording order.Source: §7 · p. 12
• Component diagnostics use different provider outputs and interfaces; ordinary WER can penalize valid overlap serializations, so cpWER and SI-cpWER remain the end-to-end attribution metrics.Source: §7 · p. 12
• Evaluation covers English single-channel audio; deterministic noisy augmentation and only two far-field CHiME-6 sessions do not establish multilingual performance or generalization across broader devices and acoustic environments.Source: §7 · p. 13

8 Conclusion

• ThyVoice leads commercial SI-cpWER in all three conditions and the full-panel mean.Source: §8 · p. 13
• Substantial errors remain; the authors call for improving transcription and persistent identity resolution and for testing real multi-day recurrence and downstream effects on retrieved information.ThyVoice uses overlap-aware speaker evidence and gated identity updates; future multilingual and code-switching evaluations are also proposed.Source: §8 · p. 13

A Run Provenance and Targeted Repeatability Checks

• The accepted full-corpus runs search pyannote /identify voiceprints in batches of 50 when the bank exceeds the request limit, enrolling only when no batch yields a qualifying match.Source: A Run Provenance and Targeted… · p. 16
• Deepgram Nova-3 was requested as diarize_model=latest, documented at the June 2026 evaluation date as its v2 batch diarizer.The alias and time-specific resolution are retained because latest can advance.Source: A Run Provenance and Targeted… · p. 16
• Table 9 records three ElevenLabs draws for each NOTSOFAR condition; repeated calls varied in local speaker counts in a minority of meetings.Main results report means and observed ranges.Source: A Run Provenance and Targeted… · p. 16, Table 9 · p. 16
• Repeatability checks were targeted, not panel-wide: ElevenLabs had three full draws per NOTSOFAR condition, AssemblyAI had repeated clean-subset checks, and remaining condition cells were single observations.No panel-wide variance ranking is claimed.Source: A Run Provenance and Targeted… · p. 16

B cpWER Error Decomposition

• The appendix gives the per-condition decomposition behind the main cpWER table, with IAR defined as S/N + I/N and cpWER as D/N + IAR.Source: B cpWER Error Decomposition · p. 17, Table 10 · p. 17
• Deletion is omitted reference content, while IAR covers substitutions or insertions and may include real words assigned to the wrong local speaker, not just lexical errors.IAR is not a direct hallucination or truthfulness metric.Source: B cpWER Error Decomposition · p. 17

C Local Speaker Evaluation

• The local-speaker appendix identifies component lineage and fixes a 0.25 s maximum same-speaker gap for word-labelled outputs while leaving native timelines unchanged.Source: C Local Speaker Evaluation · p. 17

C.1 DER/JER error decomposition

• The DER/JER decomposition describes recoverable output surfaces, not provider diarizers independently: emitted-word gaps omit labelled activity and simultaneous speech is usually serialized.Table 11 reports DER/JER decomposition by condition and output surface.Source: §C.1 · p. 18, Table 11 · p. 18

C.2 Word-to-turn merge-gap sensitivity

• Reconstructed-turn DER depends partly on the post-processing merge gap, with provider- and condition-dependent effects; the main comparison therefore uses one disclosed gap.The appendix sweeps alternative gaps on the same saved words and speaker labels; Table 12 covers merge-gap sensitivity.Source: §C.2 · pp. 18–19, Table 12 · p. 19

D Detailed Local Speaker-Clustering Diagnostics

• Local diagnostics derive speaker relations from duration co-occurrence after partitioning recordings at reference and hypothesis boundaries; coverage measures labelled single-speaker time, not label correctness.Fragmentation, merging, cluster purity, and contamination describe local cluster relationships; overlap is evaluated separately.Source: D Detailed Local Speaker-Clustering… · p. 19

D.1 Counts, clustering, and speaker coverage

• DiariZen md-v2 has the highest cluster purity among native systems, while OpenAI exceeds 90% single-speaker coverage but has much lower exact-count accuracy and purity.Table 13 reports speaker-count, clustering, and coverage diagnostics on NOTSOFAR.Source: §D.1 · p. 20, Table 13 · p. 20

D.2 Overlap detection

• Noise primarily reduces overlap recall while precision stays high for DiariZen and pyannote; CHiME-6 is harder for every native system.Word-reconstructed timelines are excluded because one-speaker-per-word output cannot represent simultaneous activity faithfully.Source: Table 14 · p. 20, §D.2 · p. 21

E Lexical Error Decomposition

• Deletion dominates most systems as acoustics become harder, especially Deepgram and WhisperX on CHiME-6; Qwen and Parakeet degrade less sharply but do not match ElevenLabs on this lexical surface.The appendix table scores emitted words without speaker labels.Source: E Lexical Error Decomposition · p. 21, Table 15 · p. 21

F Additional Controlled Ablations

• The controlled-ablation block pairs ThyVoice configurations, while hosted-provider comparisons are descriptive single-run contrasts.Tables 16 and 17 use the same 20-meeting NOTSOFAR subset in Clean and Noisy conditions.Source: F Additional Controlled Ablations · p. 21

F.1 ThyVoice enrollment: pooled vs. longest-turn

• Approximately 0.5 percentage point on Clean and 2.4 points on Noisy: pooling improves ThyVoice SI-cpWER in the paired 20-meeting ablation.The comparison does not isolate individual voiceprint gates.Source: Table 16 · p. 21, §F.1 · pp. 21–22, F Additional Controlled Ablations · p. 21

F.2 Enrollment pooling across comparison cascades

• For comparison cascades, the pooling sweep changes enrollment audio while holding matching and upsert fixed; it tests API-constrained audio pooling, not embedding pooling.PyannoteAI exposes opaque voiceprints, and ThyVoice pools only evidence that passes overlap and eligibility gates.Source: §F.2 · p. 22
• Pooling improves 9 of 10 reported comparison-cascade SI-cpWER cells, but PyannoteAI reverses direction between Clean and Noisy.The provider comparisons are single, unpaired runs, so the sweep is descriptive rather than variance-controlled.Source: Table 17 · p. 22, §F.2 · p. 22