PaperlayerView exact revision on arXiv

Source-linked AI summary

Token Distribution versus Data Volume: Domain Balancing in Multi-Domain Meeting Summarisation

Ashima Sood, Bryan Gardiner, Joan Condell

arXiv:2608.15935v1cs.CL

TL;DR

Prior work leaves unclear whether domain balancing helps because of token distribution or simply greater data volume. This paper compares balanced and proportional mixtures at matched budgets and finds that balancing improves minority domains while imposing only a small cost on majority domains.

  • Problem

    Prior work does not disentangle whether balancing gains arise from token distribution or total data volume, a distinction that matters for heterogeneous multi-domain datasets.

  • Method

    The study compares balanced and native-proportional token mixtures across five meeting corpora at matched 2–32M token budgets, evaluating each domain separately.

  • Results

    Balanced leads every minority domain on both metrics at 32M; macro ROUGE-Lsum is 0.500 versus 0.421, while the micro average favours natural allocation.

  • Takeaways & Limitations

    Use native allocation for majority-weighted deployments, balance when all domains matter, balance by tokens rather than examples, and prune before training.

  • Takeaways & Limitations

    Transcript truncation leaves some reference content unsupported by model inputs, especially for EuroParlMin, limiting faithfulness interpretation.

Abstract

from arXiv · show

Jointly fine-tuning an LLM on meeting-summarisation corpora of widely varying size raises a question that prior work leaves confounded: when a domain-balanced training mixture helps, is the gain due to the distribution of tokens across domains, or merely to the volume of data seen? We disentangle these factors by constructing balanced and natural (native-proportional) token mixtures at matched token budgets (2-32M) over five English meeting corpora, fine-tuning Mistral-7B with QLoRA, and evaluating per domain. Balancing redistributes quality, improving the data-scarce minority domains at a low cost to the data-rich ones. The trade favours balancing whenever the minority domains matter: their share under proportional allocation is fixed at 1-2% regardless of budget, so matching balanced quality on those domains requires far more total data. We further find that pruning low-value transcript lines removes ~15% of tokens from the conversational corpora at no measurable cost, and that balancing by tokens is not the same as balancing by examples. A two-annotator study of 741 judge-labelled facts validates our fact-level evaluation. Together these results give practitioners a basis for deciding when to balance an imbalanced multi-domain mixture, and on what unit.

1 Introduction

Meeting domains differ sharply in training-data availability, causing larger domains to dominate pooled fine-tuning and smaller domains to be underrepresented. This study separates token-distribution effects from data-volume effects by comparing balanced and natural mixtures at matched budgets and evaluates their domain-specific consequences.

  • Motivation: Severely imbalanced meeting corpora cause larger domains to dominate the training signal while smaller domains are drowned out.The paper motivates balancing so each domain contributes comparably to joint fine-tuning.
  • Problem and approach: Balancing reallocates tokens within a fixed budget, whereas adding data changes total volume, confounding distribution and volume effects on quality.The study addresses this confound by holding data volume fixed and varying only token distribution.
  • Study design: The controlled study compares natural and balanced token mixtures across five English meeting corpora with severely imbalanced training sizes.It examines when balancing helps, which domains benefit, and how effects vary with budget.
  • Contributions: Across the 2→32M budget ladder, the distribution gap persists at nearly every budget and closes only when natural allocation spends more budget.The contribution frames balancing as redistributing quality, raising minority-domain performance while lowering majority-domain performance rather than improving all domains uniformly.
  • Research questions: The paper also investigates pruning low-value transcript lines, token-versus-example balancing, and whether findings transfer to a smaller model from another family.These correspond to the study’s pruning, balancing-unit, and model-scale research questions.

2 Related Work

Prior meeting-summarisation work established benchmarks and strong single-domain systems, while later resources addressed longer and more heterogeneous transcripts. Existing research does not address cross-domain imbalance in jointly fine-tuned meeting models, and this study positions itself at the intersection of supervised data mixing and QLoRA-based meeting summarisation.

  • Meeting summarisation: AMI and ICSI established early benchmarks, followed by strong single-domain encoder-decoder systems such as HM-Net and DialogLM.
  • Meeting summarisation: Later resources, including MeetingBank and ELITR Minuting, broadened meeting summarisation to longer and more heterogeneous transcripts.
  • Meeting summarisation: Recent LLM work targets summarisation quality, typically training separate models per dataset rather than addressing imbalance in jointly fine-tuned cross-domain models.
  • Data mixtures for fine-tuning: Proportional allocation follows native token mass, whereas uniform allocation gives each source an equal share; supervised fine-tuning mixtures remain comparatively underexplored.
  • Positioning and PEFT: No prior work is identified as fine-tuning LLMs on token-level, domain-balanced meeting mixtures or isolating token distribution from volume under matched budgets.

3 Methodology

The methodology isolates domain allocation from data volume by comparing three training-mixture schemes at matched token budgets, alongside controlled pruning, model selection, fine-tuning, and evaluation procedures. It balances reproducibility across conditions with domain-specific and fact-level assessment.

  • Training-mixture construction: B ∈{2, 4, 8, 16, 32}M tokens is allocated across five corpora under balanced, natural, and example-level schemes with shared experimental conditions.The schemes use the same budgets, pruning variants, prompt, tokeniser, and random seed, enabling matched-budget comparisons.
  • Training-mixture construction: 20% each is assigned under balanced allocation, whereas natural allocation gives the three small corpora only 1-2% each and the two large corpora about 95% combined.Balanced allocation oversamples small corpora as budgets grow, reaching roughly 11× for ICSI at 32M tokens; natural allocation almost never oversamples.
  • Training-mixture construction: 1.9M tokens results from equal-count allocation, but differing transcript lengths produce unequal token contributions, including 578K from ICSI and 137K from MB.Comparing equal-count with balanced allocation is intended to isolate whether balancing by tokens differs from balancing by examples.
  • Modeling and evaluation: Mistral-7B-Instruct-v0.3 is selected from six zero-shot candidates and fine-tuned with QLoRA, while Llama-3.2-3B-Instruct is retained as a scaling control.Mistral-7B leads on both micro and macro BERTScore-F1; evaluation combines per-domain ROUGE and BERTScore-F1 with a fact-level LLM judge.

4 Experiments

Experiments cover five heterogeneous English meeting-summarisation corpora using controlled allocation, pruning, and token-budget conditions. The study evaluates these mixtures with fixed QLoRA training settings and supplements the main grid with seed and example-level-baseline checks.

  • Experimental data: Five English corpora span diverse meeting genres, scales, transcription provenance, and reference conventions.They include AMI, ICSI, MeetingBank, ELITR Minuting Corpus, and EuroParlMin.
  • Experimental design: 20 runs form a 2×2×5 grid crossing allocation scheme, pruning condition, and token budget, with all grid runs using seed 42.Additional repetitions test seed robustness at 2M, 8M, and 32M, while the example-level baseline is run under both pruning conditions across three seeds.
  • Training setup: QLoRA uses fixed 4-bit NF4 quantisation and rank r=32 adapters across all schemes, budgets, and conditions, isolating training-data effects.The adapters train ≈83.9M parameters, or 1.14% of the 7B base model.

5 Results and Discussion

At matched token budgets, balancing improves all three minority domains while trading modestly against majority-domain quality, and the minority advantage persists across budgets. Pruning preserves quality, whereas example-based allocation confounds meeting counts with token volume; the redistribution also transfers to Llama-3.2-3B.

  • RQ1: Domain balancing: Balancing improves AMI, ICSI, and ELITR on both metrics, while majority-domain changes are smaller and favor natural for MB.At 32M, ROUGE-Lsum gains are +0.102 for AMI, +0.164 for ICSI, and +0.163 for ELITR; MB shifts to natural by 0.046.
  • RQ1: Domain balancing: The fact-level judge reproduces balancing’s macro-completeness advantage but its micro-completeness disadvantage, while faithfulness remains comparable.Macro completeness gains are +0.017 at 2M and +0.041 at 32M; faithfulness declines with budget for both schemes.
  • RQ3: Transcript pruning: 6.8% fewer training tokens from pruning preserve quality: BERTScore-F1 remains near-identical, and ROUGE-Lsum converges by 32M.At 32M, ROUGE-Lsum’s mean absolute difference is under 0.01; pruning is most useful when balancing heavily oversamples minority corpora.
  • RQ4: Allocation unit: Equal meeting counts produce unequal token mass, causing domain quality to follow tokens rather than examples.With 43 meetings per corpus, ICSI receives ∼578K tokens versus MB’s ∼137K; MB falls from 0.562 to 0.505, while ICSI moves from 0.258 to 0.292.
  • Replication: The redistribution replicates on Llama-3.2-3B: balancing helps the same three minority domains and costs the same two majority domains, although absolute quality is lower.The replicated direction indicates that the effect comes from data allocation rather than Mistral-7B.

6 Conclusion

At a matched 32M-token budget, domain balancing removes dependence on native training shares and leads on all three data-scarce domains under both metrics. The findings support retaining native proportions for majority-weighted deployment, balancing when all domains matter, balancing by tokens, and pruning before training.

  • 6 Conclusion: At a matched 32M budget, balancing leads on all three data-scarce domains on both metrics.Balancing reallocates quality across domains rather than raising it everywhere.
  • 6 Conclusion: Roughly 95% of token mass comes from EuroParlMin and MeetingBank, making proportional training almost exclusively focused on those domains.Under proportional allocation, per-domain quality follows each domain’s share of the training signal.
  • 6 Conclusion: Practitioners should retain native proportions for majority-weighted deployments, balance when all domains must be served, balance by tokens, and prune before training.These recommendations apply whether the mixture is balanced or native-proportional.

Limitations

The study’s limitations concern transcript truncation, judge calibration, model scale, metric interpretation, and generalisability beyond English meeting transcripts. System comparisons remain informative under the shared judge and fixed-reference design, but broader validity is untested.

  • Transcript truncation: 54% of ICSI and 33% of ELITR training transcripts are cut, versus 24% of EuroParlMin’s, under the 16,384-token device window.Truncation severity also differs: ICSI loses 18% of a median 15.8K-token transcript, while truncated EuroParlMin transcripts retain only 65% of content against a maximum length twelve times the window.
  • Fact-level evaluation: 87–94% of judge–annotator disagreements involve accepting facts the annotators reject, making absolute scores too high.Because every system uses the same judge and rubric, system differences still hold.
  • Model scale and adaptation: The findings are established only up to 7B parameters with low-rank adaptation; larger-scale models and full fine-tuning remain untested.Llama-3.2-3B reproduces the redistribution effect’s direction on every domain.
  • Metric interpretation: Automatic metrics reward agreement with corpus-specific reference conventions as well as content coverage, limiting the interpretation of absolute scores.Balanced-versus-natural comparisons remain interpretable because the reference form is held fixed.
  • Generalisability: All experiments use English meeting transcripts, leaving open whether balancing interacts with language or transfers to other multi-domain tasks.The study therefore does not establish generality beyond this language and task setting.

Ethics … E Multi-Seed Robustness

The appendix documents ethical safeguards, pruning validation, mixture construction, and transcript-length constraints, while reporting that pruning is reliable for conversational corpora but problematic for EuroParlMin. It also records allocation differences, truncation patterns, and limitations affecting training faithfulness.

  • Ethics: All corpora were used under released licences without re-identification or added annotation, and annotators participated voluntarily with appropriate compensation.The authors also state that annotators were colleagues with NLP backgrounds, independent of model development and results production.
  • Appendix Contents: The appendix covers pruning statistics, human validation, mixture construction, truncation, multi-seed robustness, model selection, judge protocols, and full per-domain results.Its contents also list fine-tuning hyperparameters, training trajectories, annotation guidelines, corpus construction, prompts, and summary lengths.
  • A Pruning Statistics: 14–16% of tokens are removed from AMI, ICSI, and ELITR, versus 2.3% from EuroParlMin, producing a 6.8% aggregate reduction dominated by the largest corpora.The conversational corpora contain substantial backchannel and filler, whereas EuroParlMin consists of formal prepared and edited speeches.
  • B Human Validation of Pruning: 500 sampled lines were human-validated using stratified KEEP and DELETE samples across corpora, transcripts, and non-guard lines with contextual independent annotation.The design avoids majority-class agreement by default and hides the model’s decision from annotators.
  • B Human Validation of Pruning: κ=0.67 pooled and 0.62 on conversational corpora, while pruner KEEP decisions were ratified in 72–89% of cases and conversational DELETE decisions in 52–84%.EuroParlMin was the sole failure: annotator κ=1.0, but only 4% of model deletions were ratified, indicating substantive parliamentary content was removed.
  • C Data-Mixture Construction Details: The balanced mixture gives each corpus an equal ∼B/5 token share, increasingly oversampling minority corpora while majority corpora draw distinct meetings at 1.0×.The appendix provides per-corpus token and meeting counts for every mixture, budget, and example-level seed.
  • E Multi-Seed Robustness: 12.3% of train/development transcripts exceed the window versus 3.3% at test time; ICSI is truncated most often at 53.8% but retains 82%, while EuroParlMin retains 65%.Because references cover whole sessions while truncated inputs do not, training can require content absent from the input; EuroParlMin has the lowest faithfulness under both schemes and ICSI the highest.
  • D Transcript Length and Truncation: Native shares are AMI 1.6%, ICSI 1.4%, ELITR 2.2%, EPM 46.6%, and MB 48.3%, so majority corpora dominate every natural-scheme budget without oversampling.At 32M, ICSI draws only 33 of its 43 meetings; example-level allocation instead gives each corpus 43 meetings, with token counts varying by transcript length and seed.

E.1 Balanced and Natural Schemes · E.2 Example-Level Baseline

Multi-seed checks reproduce the balanced–natural crossover: balanced benefits all three data-scarce domains, natural benefits the data-rich domains, and the macro–micro ranking flips accordingly. The example-level baseline selects diverse meeting samples across seeds, especially for the large corpora.

  • E.1 Balanced and Natural Schemes: Transcript truncation is substantially heavier in training than at test under the stated Mistral-7B token budgets.The table reports 12.3% truncation for train+development, while the test percentage is truncated in the supplied passage.
  • E.1 Balanced and Natural Schemes: The multi-seed runs repeat both schemes under pruning at the 2M, 8M, and 32M budgets with two additional seeds, producing 12 additional runs.Scores are reported as mean ± std over three seeds across the full metric family.
  • E.1 Balanced and Natural Schemes: At 8M and 32M, balanced leads AMI, ICSI, and ELITR on all five metrics, while natural leads EPM and MB.The macro average favours balanced, whereas the micro average favours natural.
  • E.1 Balanced and Natural Schemes: At 32M on ROUGE-Lsum, balanced leads AMI by 0.168, ICSI by 0.227, and ELITR by 0.189.These minority-domain gaps are large relative to seed variation.
  • E.1 Balanced and Natural Schemes: Development loss bottoms out before training loss, after which training loss keeps falling and patience-2 early stopping selects checkpoints two epochs later.This trajectory holds across balanced and natural runs at the 2M, 8M, and 32M budgets.
  • E.1 Balanced and Natural Schemes: MeetingBank is the only consistent majority-domain cost of balancing, while the EuroParlMin gap narrows with budget.At 32M, the EuroParlMin gap is smaller than its own seed spread.
  • E.2 Example-Level Baseline: The example-level baseline uses seeds {1,2,15} chosen to maximise sampled-data diversity, minimising mean pairwise overlap on EPM and MB.The large corpora are fully distinct across seeds, whereas the small corpora overlap only partially.

F Model Selection

Mistral-7B was selected because it led BERTScore-F1 under both macro and micro averaging, ranked first on three of five corpora, and achieved the strongest weakest-domain score. ROUGE-Lsum instead favored Gemma-4-12B, while GPT-4o ranked last under the evaluation protocol.

  • Metric choice: Mistral-7B led BERTScore-F1 on both macro (0.833) and micro (0.835) averages, whereas Gemma-4-12B led ROUGE-Lsum macro (0.347).Both metrics ranked Qwen3-8B weakest, but selection used BERTScore-F1 because embedding overlap was judged closer to semantic adequacy than surface n-gram overlap.
  • Metric choice: Mistral-7B’s BERTScore-F1 lead held under both averaging schemes despite test-set imbalance, so selection was not an artifact of MB’s larger size.Macro weights corpora equally, whereas micro is dominated by MB; the test sets contain 6 ICSI meetings versus 862 MB meetings.
  • Domain robustness: Mistral-7B ranked first on three of five corpora—ELITR, EPM, and MB—and had the highest weakest-domain BERTScore-F1 floor at 0.824.It separated from the field on the two largest corpora, EPM and MB, with non-overlapping confidence intervals; Llama-3.1-8B led on AMI and ICSI.
  • Closed-model comparison: GPT-4o ranked last on macro BERTScore-F1 (0.815), versus 0.833 for Mistral-7B and 0.826 for the 3B Llama, and on every reported average.On MB, GPT-4o had the highest BERTScore recall but the lowest precision, indicating a mismatch between output format and reference form under this protocol.
  • Capacity control: Llama-3.2-3B was retained as a lower-capacity control because model size was not monotonic with BERTScore-F1 quality and the 3B model was competitive with 8B models.The screen found 7B Mistral above 12B Gemma and GPT-4o on BERTScore-F1, making the smaller model a genuine capacity probe.

G Summary Lengths (Reference and Generated) · H Corpus Construction · I Fine-tuning Hyperparameters

The paper sets a 2048-token generation budget that covers nearly all reference summaries, standardizes five corpora while preserving native structures and evaluation conventions, and applies one QLoRA configuration across all training conditions. Corpus-specific construction choices and length distributions therefore define the evaluation setting without changing the fine-tuning setup across experiments.

  • G Summary Lengths (Reference and Generated): 87–664 tokens: reference-summary medians span short to moderate lengths across domains.The 2048-token budget covers every test reference in AMI, ICSI, ELITR, and MeetingBank, and 98.8% overall.
  • G Summary Lengths (Reference and Generated): 98.8% of references fit within 2048 tokens; the exceptions are atypical undivided full-chapter EuroParlMin documents.The generous cap preserves efficient decoding without affecting the vast majority of references.
  • G Summary Lengths (Reference and Generated): 575 versus 101 tokens: GPT-4o’s MeetingBank outputs average substantially more tokens than the references.The mismatch comes from headers, attendee blocks, and closing formalities absent from the reference minutes, depressing precision and resulting F1 while leaving recall intact.
  • H Corpus Construction: Five corpora are mapped to a uniform id, source, summary, split schema while retaining canonical or seeded-and-released partitions.ICSI’s train/development division uses seed 42; canonical partitions include AMI, ICSI’s six-meeting test set, and official MB and EPM splits.
  • H Corpus Construction: Native corpus structure is preserved: AMI and ICSI transcripts are reconstructed from dialogue-act annotations, whereas MB, ELITR, and EPM retain distributed forms.The summarization unit differs by corpus, including submeetings for AMI and whole meetings for ICSI and ELITR.
  • H Corpus Construction: 38 meetings: ELITR’s three test sets are kept separate, while evaluation retains all multiple references and scores each prediction by its maximum.For training-mixture construction, each meeting is consolidated to one reference.
  • H Corpus Construction: 68 of 70 multi-reference meetings: the selected ELITR training reference was the longest candidate.The selection maximizes mean ROUGE-Lsum recall against other references; the two exceptions differed by no more than three words.
  • I Fine-tuning Hyperparameters: 83.9M parameters, 1.14% of the model: QLoRA trains low-rank adapters while freezing the 4-bit base weight matrix.The configuration is applied identically across all schemes, budgets, and conditions, with cross-entropy computed only over summary positions.

J Training Trajectories and Checkpoint Selection … L.1 Annotation guidelines

Training and development loss support patience-2 early stopping and non-final checkpoint selection, while atomic-fact judging is validated against human agreement. The annotation guidelines define separate binary tests for completeness, faithfulness, and salience-based conciseness.

  • J Training Trajectories and Checkpoint Selection: Development loss is used only to select checkpoints, never as a summary-quality measure.Table 17 reports each run’s best development loss together with the epoch where it occurred.
  • J Training Trajectories and Checkpoint Selection: Development loss bottoms out before rising, so selected checkpoints are not the final epochs and patience-2 early stopping follows each run’s minimum.The minimum occurs at epoch 2 for mid-range budgets and epoch 3 at 2M; balanced 32M also selects epoch 3.
  • K LLM Judge Protocol: Atomic fact-checking evaluates completeness as recall against the reference, conciseness as precision against the reference, and faithfulness as precision against the transcript.The framework targets individual claims because ROUGE-Lsum and BERTScore can diverge from content coverage and faithfulness.
  • K LLM Judge Protocol: The protocol uses the human-authored reference minute as the salience oracle, avoiding a separate source-conversation annotation stage.The judge uses greedy decoding on a single NVIDIA H100, while other inference and fine-tuning use a single A100.
  • L Human Validation of the LLM Judge: 741 judge-labelled facts from balanced-32M were compared with two annotators, whose agreement provides the human ceiling for interpreting judge performance.Annotators adjudicated the judge’s atomic-fact lists, and the study was restricted to balanced-32M because it evaluates the judge rather than allocation schemes.
  • L Human Validation of the LLM Judge: κ = 0.79/0.84/0.80 human–human agreement was substantial, while judge agreement was κ = 0.76/0.71 for completeness, 0.63/0.63 for conciseness, and 0.62/0.57 for faithfulness.The judge reached the human band on completeness but remained below the ceiling on conciseness and faithfulness.
  • L.1 Annotation guidelines: Every annotation item is a binary decision on an atomic fact, with annotators never writing summaries or assigning scale ratings.The transcript is used for faithfulness, the generated minute for completeness, and the authoritative reference minute for salience-based conciseness.
  • L.1 Annotation guidelines: Annotators complete each task separately, inspect only permitted materials, provide one-line reasons, and judge unsupported, contradicted, vague, or over-specific claims according to task-specific rules.Faithfulness requires transcript support rather than plausibility, while conciseness asks whether the human minute-taker considered a fact worth recording.

M Full Per-Domain Results · N Prompts

The appendix shows a consistent per-domain crossover: balanced allocation benefits minority domains, whereas natural allocation benefits majority domains across metrics, models, and seeds. It also specifies the pruning, summarisation, and fact-level evaluation prompts used throughout the study.

  • M Full Per-Domain Results: The appendix reports complete per-domain results for ROUGE-1/2/L/Lsum and BERTScore-F1 across the five-budget 2–32M ladder.The full grid complements the ROUGE-Lsum+BERTScore view presented in Section 5.
  • M Full Per-Domain Results: Balanced allocation leads on minority domains AMI, ICSI, and ELITR, while natural allocation leads on majority domains EPM and MB across the full metric family.This crossover is reported for the pruned Mistral-7B grid and reproduced for Llama-3.2-3B.
  • M Full Per-Domain Results: At 8M and 32M, balanced leads all three minority domains and natural leads both majority domains on every metric across seeds.Macro favours balanced while micro favours natural.
  • N.1 Pruning Prompt: The pruning model processes numbered transcript chunks while preserving global line numbering and returns only a JSON delete list.The same prompt is applied identically to all five corpora.
  • N.1 Pruning Prompt: The pruning prompt deletes only pure conversational noise and keeps lines containing any substantive topic, fact, opinion, decision, question, answer, or logistical detail.KEEP is the default, and the prompt instructs that roughly 15–35% of lines are deleted, never the majority.
  • N.2 Summarisation Prompt: The summarisation prompt asks every candidate model to write faithful meeting minutes from the transcript without introducing information absent from it.The prompt is applied identically across candidate models; test-time transcripts are raw and unpruned.
  • N.3 LLM-as-a-Judge Evaluation Prompts: The fact-level evaluation prompts use greedy decoding, a judge isolated from fine-tuned systems, strict JSON schemas, and fixed adjudication rules across mixtures, budgets, and domains.The same rules also guide the human-validation slice, with atomic facts extracted from reference and generated minutes.
Loading 2608.15935v1…