PaperlayerView exact revision on arXiv

Source-linked AI summary

TalkFa: A Unified Benchmark for Farsi Dialogue Generation and Understanding

Neda Jamshidi, Kamyar Zeinalipour, Fahimeh Akbari, Monica Bianchini, Marco Maggini, Marco Gori

arXiv:2609.01810v1cs.CL

TL;DR

Farsi lacks a comprehensive benchmark for dialogue generation and understanding. TalkFa introduces a unified, human-verified benchmark spanning complementary datasets and tasks, showing broad LoRA and model-performance patterns while exposing limits of automatic evaluation.

  • Problem

    Farsi lacks a unified benchmark for dialogue generation and understanding across multiple conversational tasks.

  • Method

    TalkFa combines three complementary Farsi dialogue datasets created with LLM assistance, native-speaker verification, and reproducible model baselines.

  • Results

    LoRA substantially improves generation; FABERT leads dialogue acts, LORA-MISTRAL-7B leads emotion recognition, and LoRA-MISTRAL-24B leads sentiment.

  • Takeaways & Limitations

    TalkFa provides a unified, reproducible benchmark for Farsi dialogue generation and understanding, with 25–50% of training data recovering over 90% of final generation gains.

  • Takeaways & Limitations

    LLM-assisted construction may leave stylistic regularities or pretraining overlap, while transferred English labels may miss culturally specific Farsi expression and intent.

Abstract

from arXiv · show

Farsi, spoken by more than 120 million people, lacks a comprehensive benchmark for dialogue generation and understanding. We introduce TALKFA, a unified benchmark comprising three complementary datasets: (1) WIKI-FADIAL, 4.2K Wikipedia-grounded dialogues for knowledge-grounded generation; (2) DAILYDIALOG-FA, 6.6K dialogues annotated for dialogue acts and emotions; and (3) PLAYDIAL-FA, 2.1K theatrical dialogues with sentiment labels. While LLMs assist data construction, every dialogue undergoes multi-stage review and revision by native Farsi speakers, and only the final human-approved dialogues are released. Experiments with six LLAMA and MISTRAL models show that LoRA substantially improves dialogue generation while requiring only 25-50% of the training data to recover over 90% of the final performance gains. Across classification tasks, FABERT achieves the best dialogue-act performance, LORA-MISTRAL-7B performs best on emotion recognition, and MISTRAL-24B achieves the highest sentiment score. Human evaluation and independent external validation demonstrate the reliability of the benchmark, while comparisons with GPT-4.1 as an LLM judge reveal that automatic metrics substantially overestimate dialogue quality. Zero-shot evaluation with frontier LLMs further shows that TalkFa remains a challenging benchmark. We will release all datasets, annotation guidelines, code, and checkpoints.

1 Introduction

Farsi lacks a unified, culturally adapted benchmark spanning dialogue generation and understanding. TalkFa addresses this gap with three datasets, human-verified construction, reproducible baselines, and released research resources.

  • Farsi, spoken by over 120 million people, lacks a unified benchmark for dialogue generation and understanding.
  • Existing Farsi resources mainly target speech, sentiment, or general NLU, leaving conversational modeling comparatively unexplored.
  • The project investigates LoRA gains, model performance across classification tasks, zero-shot comparisons, and generation scaling with training data.
  • TalkFa combines WIKI-FADIAL, DAILYDIALOG-FA, and PLAYDIAL-FA to cover generation, dialogue acts, emotions, and sentiment analysis.
  • The benchmark uses an LLM-assisted pipeline with native-speaker review and revision, then releases datasets, guidelines, code, adapters, and checkpoints.

2 Related Work

Prior dialogue benchmarks cover diverse languages and tasks, while Farsi resources largely emphasize NLU or general evaluation. TalkFa unifies Farsi generation and multiple dialogue-understanding tasks in one benchmark.

  • Dialogue Benchmarks: English benchmarks span open-domain, empathetic, knowledge-grounded, and task-oriented dialogue, while Chinese and multilingual resources address additional dialogue settings.
  • Dialogue Benchmarks: No existing benchmark jointly supports Farsi dialogue generation and multiple dialogue-understanding tasks.
  • Knowledge-Grounded Dialogue: WIKI-FADIAL extends knowledge-grounded dialogue resources to Farsi through GPT-4o-assisted generation, native-speaker post-editing, and automatic and human evaluation.
  • Dialogue Evaluation: Automatic metrics such as BLEU, ROUGE, and BERTScore remain common despite limitations for open-ended dialogue, motivating learned and LLM-based evaluators.
  • Synthetic Data Generation and Farsi Resources: TalkFa is presented as the first unified Farsi benchmark supporting knowledge-grounded dialogue alongside dialogue-act, emotion, and sentiment tasks.
  • Comparison of Resources: Table 1 compares representative English and multilingual benchmarks with Farsi resources using generation, classification, and human-validation indicators.

3 TalkFa Creation

TalkFa is created as a unified Farsi dialogue benchmark spanning knowledge-grounded, everyday, and theatrical conversations. Its datasets combine LLM-assisted construction with native-speaker review, covering generation, dialogue understanding, and sentiment analysis.

  • Benchmark scope: TalkFa combines WIKI-FADIAL, DAILYDIALOG-FA, and PLAYDIAL-FA for knowledge-grounded generation, everyday dialogue acts and emotions, and theatrical sentiment analysis.The datasets contain 4.2K, 6.6K, and 2.1K dialogues, respectively.
  • WIKI-FADIAL construction: WIKI-FADIAL pairs Farsi encyclopedic paragraphs with short, informal, multi-turn conversations designed to remain faithful to their sources.Each source paragraph is converted into a six-turn dialogue with traceability references.
  • WIKI-FADIAL construction: 4,184 source paragraphs were obtained by filtering curated Farsi Wikipedia pages for length, policy compliance, and topical self-containment.The source collection covered mathematics, history, biology, literature, and other disciplines.
  • Human verification: Native Farsi annotators reviewed every generated or translated dialogue, revising language, cultural references, coherence, and factual or semantic fidelity.The review process used multiple stages, including independent review and adjudication where needed.
  • Dataset quality: WIKI-FADIAL dialogues show minimal lexical overlap with source paragraphs while retaining semantic faithfulness, with average BLEU-4 0.005, ROUGE-L F1 0.10, and embedding cosine similarity 0.66.These measurements assess semantic grounding without relying on copying source wording.
  • Human verification: 63.0% of WIKI-FADIAL dialogues, 17.0% of DAILYDIALOG-FA dialogues, and 19.1% of PLAYDIAL-FA dialogues required revision before approval.PLAYDIAL-FA required only 1.4% additional revisions in its second review round, while all released dialogues were human-approved.

4 Experiments

Experiments compare instruction-tuned LLMs and sentence encoders across generation, dialogue-act, emotion, and sentiment tasks, using zero-shot, LoRA-adapted, frozen-backbone, and full fine-tuning settings. LoRA improves generation, while model strengths vary across classification tasks and automatic metrics overestimate dialogue quality.

  • WIKI-FADIAL: Dialogue Generation: LoRA consistently improves generation across model sizes, with the largest gains for smaller LLAMA models.LLAMA-3.2-1B and LLAMA-3.2-3B nearly double their ROUGE-L scores after fine-tuning, while larger models improve more modestly.
  • WIKI-FADIAL: Dialogue Generation: A LoRA-adapted 7B model rivals or exceeds a 24B Base model, indicating adaptation can matter more than scale for this task.
  • Beyond Automatic Metrics: Human and LLM-Based Evaluation: 2.74/5 is the strongest human rating despite automatic scores above 0.93 BERTScore and 0.97 BERTCos, showing that reference-based metrics overestimate open-ended dialogue quality.GPT-4.1 judge ratings are systematically more lenient, averaging +0.45 higher grades than human annotations.
  • Data-Efficiency Ablation: 25–50% of WIKI-FADIAL training data recovers over 90% of final generation gains under LoRA.
  • DAILYDIALOG-FA: Turn-Level Act & Emotion Classification: FABERT achieves the highest dialogue-act macro-F1 at 0.75, while LoRA-adapted MISTRAL-24B reaches 0.72 without surpassing Farsi-pretrained encoders.
  • DAILYDIALOG-FA: Turn-Level Act & Emotion Classification: No model exceeds 0.38 macro-F1 for emotion classification, with class imbalance and cultural mismatch identified as contributing challenges.
  • PLAYDIAL-FA: Sentiment Classification: 0.62 macro-F1 is the best theatrical sentiment result, achieved by MISTRAL-24B through improved minority-class recall.Full fine-tuning helps some multilingual encoders but can reduce performance for Farsi-specific encoders on the small PLAYDIAL-FA dataset.

5 Conclusion and Future Work

TALKFA is a unified Farsi dialogue benchmark built from three datasets and evaluated across generation, dialogue acts, emotions, and sentiment. Its results show benefits from LoRA and strong task-specific model differences, while human evaluation exposes limitations of automatic metrics.

  • TALKFA combines WIKI-FADIAL, DAILYDIALOG-FA, and PLAYDIAL-FA into a unified benchmark for Farsi dialogue generation and understanding.The datasets cover knowledge-grounded, everyday, and theatrical dialogue with complementary annotations and tasks.
  • LLM-assisted data creation with native-speaker verification supports reproducible baselines across generation, dialogue acts, emotions, and sentiment classification.
  • LoRA substantially improves generation, with 25–50% of training data recovering over 90% of final generation gains.
  • FABERT leads dialogue-act classification, LORA-MISTRAL-7B leads emotion recognition, and LoRA-MISTRAL-24B leads sentiment classification.
  • Human evaluation gives the best fine-tuned model 2.74/5 despite high automatic scores, highlighting limitations of reference-based metrics for open-ended dialogue.
  • The authors release datasets, annotation guidelines, code, LoRA adapters, and fine-tuned checkpoints.

6 Limitations

TalkFa has several scope, construction, structural, labeling, coverage, and evaluation limitations. These constrain how broadly its results and benchmark quality should be interpreted.

  • Data construction and contamination: LLM-assisted construction may preserve stylistic or lexical regularities, while source-material overlap with contemporary LLM pretraining cannot be reliably verified.
  • Structural constraints: WIKI-FADIAL’s fixed six-turn, 1–20-token structure underrepresents longer, multitopic, overlapping, and multi-party conversations.
  • Cross-lingual label transfer: DAILYDIALOG-FA inherits English dialogue-act and emotion labels that may not fully capture Farsi phenomena such as ta’arof, indirectness, and culturally dependent expression.
  • Register and dialect coverage: TALKFA omits settings such as social media, disfluencies, code-switching, slang, and multi-party chats, and focuses primarily on standard Iranian Farsi rather than regional varieties.
  • Evaluation limitations: Dialogue evaluation remains subjective, and automatic metrics such as BLEU, ROUGE, and BERTScore correlate imperfectly with human judgments.
  • Benchmark scale and model coverage: TalkFa remains smaller than major English conversational benchmarks and does not exhaustively cover architectures, prompting strategies, retrieval systems, or reasoning-oriented models.

Reproducibility Details

The experiments use pinned software, fixed seeds, specified optimization settings, and released code and data. The reported compute footprint is modest enough to support replication on academic clusters.

  • Hardware: Experiments use 3× NVIDIA RTX A6000 GPUs with mixed-precision bf16, FlashAttention, and gradient checkpointing.
  • Software environment: Python 3.11, PyTorch 2.3.0, HuggingFace Transformers ≥0.22, and BitsAndBytes 0.45 are specified, with package versions pinned in environment.yml.
  • Random seeds: The experiments fix seed = 100 for initialization and shuffling, while classification results are averaged over three seeds.
  • LLM fine-tuning: LoRA fine-tuning uses effective batch size 24, learning rate 1×10−4, three epochs, 512-token context, rank 128, α = 256, and dropout 0.1.
  • Encoder classification: Encoder settings specify separate frozen-encoder and full-fine-tuning configurations, including their batch sizes, learning rates, epochs, and 256-token contexts.
  • Compute and release: 16 GPU-hours cover LLM fine-tuning, encoder runs take approximately 2 hours per task, and all corpora, scripts, and checkpoints will be released.

D Illustrative Examples of Human Post-editing for WIKI-FADIAL

The WIKI-FADIAL post-editing examples show that human review addresses diverse generation failures, from isolated grammatical errors to incoherence, unsupported analogies, and repetitive phrasing. Some dialogues require no revision when they are already natural and culturally appropriate.

  • Example 1: Example 1 revises only the opening sentence because a grammatical error obscured its meaning while the rest remained fluent and well structured.
  • Example 2: Example 2 removes repetitive openers such as “Have you heard...” or “Have you read...” to increase conversational variety.
  • Example 3: Example 3 regenerates the entire dialogue after syntactic incoherence and poor logical progression made it confusing.
  • Example 4: Example 4 replaces an inaccurate, source-ungrounded storm–funnel analogy through regeneration, producing a coherent and accurate second attempt.
  • Example 5: Example 5 requires no revision because the dialogue is grammatical, culturally appropriate, natural, and lightly humorous.

G Illustrative Examples of Human Post-editing for DAILYDIALOG-FA

DAILYDIALOG-FA examples show how post-editing adapts translated dialogues to Iranian language, culture, institutions, and conversational conventions while preserving dialogue structure.

  • Examples and rationale: The examples present original English dialogues, raw Farsi translations, and final human-edited versions to illustrate the transformations and their reasoning.The post-editing process addresses both linguistic form and cultural grounding.
  • Cultural adaptation: Examples replace culturally distant references with Iranian equivalents, including dollars with tomans, a 0–100 grading scale with 0–20, and Huangguoshu Falls with Laton Waterfall.These edits target local currency, educational conventions, and culturally familiar geography.
  • Cultural adaptation: An American store name was replaced with a well-known Iranian brand, while one American personal name was autonomously replaced with a natural-sounding Farsi name.The personal-name substitution required no human edits.
  • Segmentation rules: Segmentation preserves spoken dialogue, speaker attribution, conversational order, interruptions, repetitions, and pauses while removing non-dialogue material and correcting obvious OCR artifacts.Annotators were instructed not to paraphrase, rewrite, reorder, or omit dialogue lines.

K Human Evaluation Guidelines

Human evaluation of WIKI-FADIAL uses holistic native-speaker judgments of dialogue quality across fluency, coherence, contextual fit, naturalness, and cultural plausibility.

  • Evaluation procedure: Native Farsi speakers independently evaluate generated dialogues on a five-level scale measuring naturalness, coherence, fluency, and contextual appropriateness.The evaluation uses one overall score per dialogue rather than separate scores for each dimension.
  • Five-level rubric: The rubric ranges from A for highly natural, coherent, fluent, and contextually appropriate dialogue to E for incoherent, irrelevant, or broken conversation.B, C, and D represent progressively greater grammatical, fluency, coherence, or conversational-flow problems.
  • Evaluation dimensions: Annotators assess fluency, coherence, contextual appropriateness, naturalness, and cultural plausibility when assigning the holistic score.Cultural plausibility includes appropriate expressions, politeness conventions, and conversational style.
  • Scoring principles: Holistic scoring allows occasional grammatical errors when the interaction remains coherent and natural, while lexical overlap does not compensate for unnatural conversational structure.This instruction prioritizes interaction-level quality over isolated mistakes or surface similarity.
  • Illustrative examples: The section illustrates the rubric with translated dialogue examples spanning health, books, politics, travel, and everyday conversation.The examples show how conversational coherence and appropriateness are judged in context.

L.3 Discussion

Independent validation supports TalkFa’s dialogue quality and annotation reproducibility, while the benchmark’s evaluation framing highlights the gap between measured and human-quality generation.

  • External validation: 300 stratified samples evaluated by an independent native Farsi speaker received high quality scores across WIKI-FADIAL, DAILYDIALOG-FA, and PLAYDIAL-FA.Reported scores include 3.65/4.00 for WIKI-FADIAL naturalness, 3.90/4.00 for DAILYDIALOG-FA naturalness, and 3.50/4.00 for PLAYDIAL-FA naturalness.
  • Annotation reproducibility: Annotation agreement reached 85% for dialogue acts, 89% for emotions, and 87% for sentiment, with Cohen’s κ values of 0.800, 0.872, and 0.805.These results provide evidence for reproducibility across the three annotation tasks.
  • Figure 4: Figure 4 compares ParsBERT cosine similarity and ROUGE-L F1 across fractions of the training data.The top panel reports cosine similarity and the bottom panel reports ROUGE-L F1.
  • Discussion: The external validation complements construction-time quality control by supporting both the linguistic quality of the dialogues and the robustness of the released annotations.The evidence concerns reliability of the benchmark rather than saturation of generation performance.

M Data-Efficiency Ablation

The data-efficiency ablation finds that LoRA performance rises sharply with the first quarter of WIKI-FADIAL training data, then shows diminishing returns while remaining below human-quality dialogue.

  • Scaling with training data: Performance improves sharply from the Base model to 25% of the training data, followed by diminishing returns.This pattern appears in both evaluation metrics used for the ablation.
  • Data efficiency: 25% of the WIKI-FADIAL training set recovers more than 90% of final performance gains across model sizes.The ablation evaluates LoRA fine-tuning at 25%, 50%, 75%, and 100% using BERTCos and ROUGE-L.
  • Quality boundary: Substantial gaps to human-quality dialogue remain despite the data-efficiency gains.The reported scaling result therefore does not indicate benchmark saturation.

N Automatic vs. Human Evaluation

TalkFa compares automatic metrics with human-centered evaluation and reports a substantial disconnect: high automatic scores can coexist with only moderate human ratings. The results support human or LLM-based evaluation for more calibrated assessment.

  • ≥0.93 automatic scores contrasted with 2.74/5 moderate human ratings, revealing a substantial evaluation disparity.
  • Semantic-overlap metrics fail to capture dialogue naturalness, coherence, and contextual appropriateness.
  • Independent external evaluation and agreement analysis are included to assess subjective dialogue quality and consistency with original annotations.
  • Human-centered evaluation provides a more informative quality signal than metrics that overestimate perceived dialogue quality.
Loading 2609.01810v1…