OSynth: A Longitudinal, Context-Sensitive Evaluation of Meeting Understanding

1 Introduction

• Existing benchmarks largely contain standalone sessions with limited contextual artifacts, preventing measurement at scale of whether longitudinal grounding improves summaries or downstream outputs.OSYNTH was introduced to address this measurement gap.Source: §1 · pp. 1–2
• OSYNTH generates each workspace’s artifacts from a shared latent state, using adversarial transcript refinement grounded in real AMI transcripts.The workspace state includes personas, projects, prior decisions, and outstanding tasks; artifacts include transcripts, audio, summaries, action items, emails, and slides.Source: §1 · pp. 1–2, Figure 1 · p. 1
• The corpus spans 570 hours of audio across 160 workspaces, with an exact latent-state reference and full synthetic data that can be shared.The released corpus is English-only, although the generation pipeline is language agnostic.Source: §1 · p. 2
• Across three context levels, OSYNTH evaluates two state-of-the-art MLLM systems using latent-state-derived metrics and standard summarization metrics.The levels range from single-meeting audio to prior-meeting outputs and full workspace artifacts.Source: §1 · p. 2
• In the human-discrimination study, annotators identified the synthetic side less often for OSYNTH (56.3%) than for a control without adversarial refinement (79.6%; p-value 0.001).Annotators were shown a length-matched pair containing one real and one synthetic excerpt.Source: §1 · p. 2

Meeting and dialogue corpora

• Existing spoken-meeting corpora vary in domain, audio availability, and task focus, while single-session inputs cannot isolate the effect of workspace context.OSYNTH combines longitudinal workspace context with an exact latent reference at larger scale.Source: §2 · p. 2

Evaluating context-augmented dialogue summarization

• Context-aware summarization is difficult to evaluate because summaries can hallucinate, reference-free judges may underweight factual precision, and long-context benchmarks do not isolate longitudinal workspace grounding.This motivates using latent-state metrics alongside conventional LLM-as-judge scores.Source: §2 · p. 3

Synthetic conversational data

• Prior synthetic interaction work scales conversations or agent simulations, whereas OSYNTH also generates workspace artifacts and latent references for meeting evaluation.Related turn-taking models address multi-party interaction dynamics.Source: §2 · p. 3

3 The OSYNTH dataset

• OSYNTH simulates multi-week organizational workspaces and derives meetings, transcripts, audio, and surrounding artifacts from a shared latent state.Because the artifacts share that explicit state, each observable can be scored against an exact reference.Source: §3 · p. 3

3.1 Hierarchical generation pipeline

• A top-down pipeline turns workspace state and weekly plans into linked artifacts, then meeting turns, with emitted events informing later generations.Artifact dependencies are ordered by calendar time, while the resulting events preserve a consistent history of decisions, action items, and project changes.Source: §3.1 · pp. 3–4

3.2 Multimodal rendering

• Turn-by-turn speech is rendered with voice cloning and room simulation into multi-speaker audio, alongside transcript variants and derived reference summaries.Speaker enrollment clips are reused for that speaker’s later turns; audio is mixed with gaps and limited overlap.Source: §3.2 · pp. 4–5, §3.1 · p. 4

3.3 Corpus statistics and release

• The English-only release contains 160 workspaces, 2,126 meetings, and 570 hours of synthesized audio across business and non-business domains.Meetings average 16.1 minutes of active speech.Source: §3.3 · p. 5

3.5 Realism: human discrimination study

• 56.3% of OSYNTH excerpts were identified correctly as synthetic, versus 79.6% for the Haiku control; OSYNTH remained detectable above chance.The study used 24 annotators and 432 non-attention trials; the difference between conditions was 23.3 percentage points (Fisher two-sided p = 0.001).Source: Table 2 · p. 5, §3.5 · p. 5

3.4 Adversarial Transcript Refinement

• An adversarial refinement loop uses AMI excerpts and a cross-family discriminator-refiner pair to improve transcript realism, but stops after five iterations for runtime rather than convergence.Test runs with ten iterations continued to yield accepted edits after iteration five.Source: §3.4 · p. 5

4 Context-sensitive evaluation

• Two state-of-the-art multimodal LLM stacks are evaluated across three context levels on a held-out set of 238 meetings.The paper also reports a fully open-source SpeechBrain cascade separately on a subset of the evaluation set.Source: §4 · pp. 5–6, §4.1 · p. 6

4.1 Systems evaluated

• The OpenAI and Gemini cascades pair transcription, speaker identification, and summarization components, with a separate open-source cascade reported for reproducibility.The Gemini stack uses Gemini 3.1 Pro for enrollment-based speaker identification and Gemini 3 Flash for transcription and summarization.Source: §4.1 · p. 6

4.2 Context levels

• L1 provides meeting audio only, L2 adds the system’s own prior-meeting outputs, and L3 adds speaker enrollments, name lists, slides, and inter-meeting emails.Each level is a strict superset of the previous one, so comparisons isolate added context rather than a change in input modality.Source: §4.2 · pp. 6–7

4.3 Metrics

• Metrics span transcription, diarization and speaker identification, and summarization; isomorphic measures score against latent state, while standard measures use LLM-as-judge ratings.Isomorphic metrics expose entity-sensitive behavior, including names, numbers, speaker attribution, action-item assignees, and cross-meeting entity tracking.Source: §4.3 · p. 7

5.1 Trends across context levels

• Entity-sensitive measures improve from L1 to L3 for both systems; OpenAI gains include +0.20 number recall, +0.20 assignee accuracy, and +0.26 speaker-identity consistency.OpenAI action-item description quality rises +0.47 points on a 1–5 scale, while WER shifts from 0.162 to 0.144; Gemini WER shifts from 0.099 to 0.095 with overlapping L1 and L3 bootstrap intervals.Source: §5.1 · pp. 7–8

5.2 Isomorphic versus standard metrics

• Both metric families show L3 gains on entity-sensitive dimensions, but LLM-as-judge summary scores compress gains visible in isomorphic metrics.Gemini’s judge-rated faithfulness dips slightly at L3 despite improvement on every isomorphic dimension.Source: §5.2 · p. 8

5.3 Which workspace artifacts matter?

• Workspace text explains almost all of OpenAI’s number-recall gain, while both workspace text and speaker enrollments benefit assignee accuracy through different routes.Enrollments anchor diarization clusters to canonical identities; workspace text supplies names and entities for action items.Source: §5.3 · p. 8
• L3 mainly helps named entities, numbers and dates, and technical terms; filler, function-word, and lexical-reanalysis errors remain largely orthogonal to workspace context.This pattern comes from an analysis of OpenAI L1 errors using a six-way ASR taxonomy.Source: §5.3 · p. 8

6 Anchoring on real meetings

• Faithfulness improves by +0.13 with L3 context on eight AMI meetings; coherence is at ceiling in both settings.The authors treat the direction of the context effect as convergent validity, not a second measurement of its size.Source: §6 · p. 8
• The real-meeting anchoring study is limited in size, despite providing convergent evidence on recorded audio.It covers two AMI series, ES2002 and ES2003, with four meetings each.Source: §6 · p. 8

7 Discussion

• Both systems improve monotonically on entity-sensitive isomorphic dimensions as workspace context is added, with the largest gains in assignee accuracy, number and technical-term recall, and speaker-identity consistency.These dimensions are difficult for prior meeting benchmarks to expose because they lack context, named entities, and extensive oracle latents.Source: §7 · pp. 8–9
• Workspace-specific evidence, including voice enrollments and inter-meeting emails, helps resolve speakers and entities; enrollments and workspace text are complementary rather than redundant.The authors conclude that isolated-meeting benchmarks understate the quality workspace-aware systems can achieve.Source: §7 · p. 9
• WER weights tokens equally, so entity-focused gains can be diluted among low-value words; OSYNTH’s latent state enables entity-grounded measures that expose them.The L1-to-L3 improvement is modest in aggregate WER but larger on the other measurements considered.Source: §7 · p. 9

8 Conclusion

• OSYNTH is a longitudinal synthetic meeting corpus with aligned workspace artifacts and an exact latent-state reference.It is used to evaluate meeting notetaking systems across progressively richer context levels.Source: §8 · p. 9
• Human discrimination results support OSYNTH as a relatively credible proxy for real meeting transcripts compared with naive LLM generation.The conclusion also reports that workspace context improves summary artifacts and contextualized transcription on entity-sensitive dimensions.Source: §8 · p. 9
• The reported entity-sensitive improvements concern dimensions that prior benchmarks cannot equivalently expose.The conclusion frames this as a capability of workspace-context evaluation, not a numerical transfer claim about natural recordings.Source: §8 · p. 9

• OSYNTH cannot fully reproduce long-tail acoustic, sociolinguistic, and interactional phenomena, so its results are not claimed to transfer numerically to natural recordings.The authors identify broader grounding on natural spoken-meeting data as future work.Source: Limitations · p. 9
• The single multi-speaker TTS model and one voice-cloning prompt per persona can produce uniformly paced, read-aloud audio, so ASR and diarization scores are relative to OSYNTH.The authors contrast this prosody with fully spontaneous speech and caution against treating the scores as absolute estimates for natural recordings.Source: Limitations · p. 9
• Workspace emails, slides, and documents are clean by construction, with no typos or transcription noise beyond ASR output; adding such noise is left to future work.Source: Limitations · p. 9
• Evaluation fixes one prompt template and decoding configuration per system, and diversity relies on workspace random initialization, which may constrain scaling.The authors note that absolute scores may shift under heavier prompt engineering.Source: Limitations · p. 9

Ethical considerations

• Synthetic personas and organizations are sampled from generated pools, and CommonVoice clips are used as anonymous acoustic conditioning without re-identification.Downstream users inherit the condition not to link released voices to real individuals.Source: Ethical considerations · p. 10
• Because the corpus is synthetic, it may encode generating-LLM biases and is intended for evaluation, not training systems claiming generalization from naturalistic conversation.Source: Ethical considerations · p. 10

A Corpus statistics

• Each workspace bundles audio, transcripts, notetaking and inter-meeting artifacts, and speaker enrollments derived from shared latent state; varying sizes and time horizons yield differing subsample distributions.Source: Table 6 · p. 13

B Workspace walkthrough example

• The example workspace links meetings, projects, personas, and artifacts; its focus 1:1 connects a blocked engineering project to follow-up action items and calendar context.The two-week example includes 28 meetings, with the focus meeting among recurring 1:1s and other meeting types.Source: B Workspace walkthrough example · pp. 13–14

C Generation-pipeline details

• Artifacts are dispatched in parallel once dependencies are complete, then timestamp-sorted WorkspaceEvents are folded through a reducer to reconcile workspace state.Transcript length is checked against a calendar-anchored duration budget, with one regeneration attempt for below-floor transcripts.Source: C Generation-pipeline details · p. 15

D Workspace-artifact ablation details

• Action-item assignee accuracy reaches 0.792 with full L3, versus 0.752 with text-only and 0.698 with enrollments-only; number recall gains are text-dominated.For number recall, text-only moves 0.637 to 0.975, while enrollments-only moves 0.637 to 0.605; full L3 assignee accuracy exceeds either single stream.Source: D Workspace-artifact ablation details · p. 15, Table 8 · p. 15

E Error analysis

• Named-entity, numeric/date, and technical-term errors account for roughly two thirds of extracted span pairs; L3 context mainly helps entity-sensitive classes, while number errors are less affected.The analysis attributes this pattern to little numeric metadata in workspace artifacts and describes fill, function-word, and lexical errors as mostly orthogonal to context.Source: E Error analysis · p. 15

F Adversarial transcript refinement details

• A cross-family Gemini discriminator flagged structural transcript tells that a same-family Claude judge accepted immediately; the refinement loop is capped at five iterations for runtime, not convergence.Test runs at ten iterations still received accepted edits past iteration five.Source: F Adversarial transcript refinement… · p. 16

G Additional realism-study details

• In a balanced 100-transcript probe, the LLM judge reached 59% accuracy but had only 18% recall on OSYNTH; the authors interpret this as consistent with, not independent proof of, realism.Correctly identified synthetic excerpts were associated with overarticulated agendas, little overlapping speech, and unusually clean topic transitions.Source: G Additional realism-study details · pp. 16–17

H System and metric details

• Cross-meeting speaker-identity consistency scores whether each persona maps to the same predicted name across workspace meetings, using canonical persona alignments rather than per-meeting label consistency.Action-item extraction is scored against latent action items by assignee accuracy and an LLM judgment of description completeness and concreteness.Source: H System and metric details · p. 17

I Prompt templates

• The Claude refiner addresses structural tells identified by the Gemini discriminator, while the summary-quality judge uses reference-grounded inputs and separates length from most scoring dimensions.Workspace-canonical-name expansions consistent with the roster count as faithful, and speaker attribution is excluded from faithfulness because it has a separate metric.Source: I Prompt templates · pp. 18–20

J Qualitative error examples

• L3 recovers hardware terms BLE and ArcBus with the explicit jargon list, while workspace text helps recover identifiers that L1 misrenders; number recall rises by +0.20 overall.On one meeting, number recall moves from 0.00 at L1 to 1.00 at L3; L2 already recovers both identifiers through vocabulary post-correction.Source: J Qualitative error examples · pp. 20–22, Table 9 · p. 21
• L3 corrects the summary’s product name, speaker attribution, and outcome, and its enrollment-based SID and jargon list recover the action-item assignee and artifact name.L1 invents Ryan and corrupts “Arc” as “odd”; L3 identifies Darren Okafor and the Arc enclosure tolerance stack-up.Source: J Qualitative error examples · p. 21
• Cross-meeting speaker-identity consistency improves by +0.26 as L3 anchors Devon Ashby to the same workspace name list and enrollment clip across meetings.L1 assigns incompatible invented labels across the two meetings, preventing downstream systems from stitching them into one Devon entity.Source: J Qualitative error examples · p. 23, Table 10 · p. 23

K TTS rendering strategy; L Persona-to-voice assignment

• The pipeline synthesizes turns independently for alignment and persona-level acoustic control, curates voice-cloning clips, and assigns voices by gender without other persona attributes.Candidate clips are screened with ASR round trips, duration and SNR filters, and a stability probe; each workspace draws voices without replacement.Source: K TTS rendering strategy · pp. 23–24, L Persona-to-voice assignment · p. 24

M Conversational prosody

• Conversational prosody is constructed with sampled turn-transition gaps, annotated substantive overlaps, and backchannels placed over the current floor-holder.A per-speaker sweep enforces no self-overlap; backchannel rows retain speaker identity and an is_backchannel flag for metric filtering.Source: M Conversational prosody · pp. 24–25

N Speech pacing relative to the cloning prompts

• Rendered turns are faster and more uniformly paced than cloning prompts: inter-word pause ratio falls by 60%, and pauses longer than 200 ms are roughly three times rarer per minute.The paper attributes the residual read-aloud quality primarily to pacing uniformity and secondarily to the prompts’ domain mismatch.Source: N Speech pacing relative to the… · p. 25, Table 11 · p. 25

O Alignment procedure and validation; P Audio post-processing

• Word- and turn-level alignments are refined against per-turn audio; validation finds onset MAE well below 0.25 s and word coverage above 99%, while room simulation adds reverberation and supports hybrid calls.Hybrid meetings mix in-room and remote participants with different room responses and codec simulation; acoustic-condition metadata is released with each meeting.Source: O Alignment procedure and validation · p. 25, P Audio post-processing · p. 26

Q Pipeline validation; R Multilingual generation; S Annotation interface

• The pipeline validates artifacts and meeting-level invariants, supports generation in five additional languages, and presents annotators with length-matched real-versus-synthetic excerpts in forced-choice trials.The main release remains English-only because the human-discrimination study is English-only; non-English workspaces await corresponding validation.Source: Q Pipeline validation · p. 26, R Multilingual generation · p. 26, S Annotation interface · p. 27

S Annotation interface

• Annotators compare similarly sized transcript excerpts: one human-transcribed real meeting and one LLM-generated or deliberately modified attention-check transcript.Source: S Annotation interface · p. 27

T Nicknames

• OSynth models nickname variation through persona-level policies and preserves canonical and spoken forms so name scoring can test the forms actually used.Policies are none, sometimes, or mostly; mostly personas use nicknames in nearly all informal turns, where canonical-only systems may miss the spoken form.Source: T Nicknames · p. 27

U Gemini: isomorphic vs. standard metrics

• Gemini’s L1→L3 gains concentrate on isomorphic entity-sensitive metrics and are compressed by standard LLM-as-judge scores.Assignee accuracy, number recall, technical-term recall, and speaker consistency improve.Source: U Gemini: isomorphic vs. standard… · pp. 27–28
• LLM-judge faithfulness dips by 0.14 at L3, which the authors attribute to scoring asymmetry against Gemini’s workspace-canonical-name expansions, not a quality regression.The isomorphic dimensions improve uniformly.Source: U Gemini: isomorphic vs. standard… · p. 28

V Open-source SpeechBrain baseline

• The SpeechBrain cascade combines CRDNN speech detection, wav2vec2 ASR, ECAPA-TDNN diarization and speaker identification, and prompted LLM summarization.At L3, it adds workspace-jargon hot-word biasing and enrollment-based speaker identification.Source: V Open-source SpeechBrain baseline · pp. 28–29
• On the 27-meeting single-workspace subset, the SpeechBrain cascade trailed both state-of-the-art systems on every reported metric; at L3, WER rose from 0.557 to 0.638, and technical-term recall regressed.Enrollment-driven metrics (assignee accuracy and speaker consistency) jumped sharply at L3 once SID was enabled.Source: V Open-source SpeechBrain baseline · p. 29, Table 13 · p. 30
• On SpeechBrain’s high-WER input, the L3 vocabulary-correction pass silently compresses transcripts, deleting about 25% of words on average and turning deleted terms into recall misses and WER deletions.Across 10 evaluation meetings and 409 turns, 31% of turns lose over 30% of input words and 13% lose over 50%; on stronger ASR input, the pass is essentially a no-op.Source: V Open-source SpeechBrain baseline · p. 29

W Robustness to the choice of judge

• With Qwen3-235B-Instruct as judge, faithfulness and action-item description quality improve from L1 to L3, but coherence is saturated at the 5.00 ceiling.The authors conclude that the direction of the context effect does not depend on the judge, although score range and calibration vary by model family.Source: W Robustness to the choice of judge · p. 30, Table 14 · p. 30

X Licenses of open-source models and datasets

• The license table lists licenses for the open-source assets; SpeechBrain checkpoints inherit Apache 2.0, while NOTSOFAR-1 is used under CC BY 4.0.The corpus itself contains no derivatives of the referenced datasets’ audio content.Source: X Licenses of open-source models and… · p. 30, Table 15 · p. 31