Let LLMs Take on the Latest Challenges! A Chinese Dynamic Question Answering Benchmark

1 Introduction

• The Chinese community lacks a benchmark for dynamically changing answers, although Chinese LLMs face the same challenge and current static benchmarks focus on less-evolving knowledge.Up-to-date factual answering is described as relevant to large-scale daily applications.Source: §1 · p. 1
• CDQA contains 1,339 question-answer pairs, generated with two LLM roles and refined by annotators, with samples classified as fast-, slow-, or never-changing.The answer-change categories are intended to support finer-grained evaluation.Source: §1 · pp. 1–2
• GPT-4 ranks at the top with search results, while in-context learning and prompting generally improve performance but elicit more hallucinations, and Google outperforms Bing as a retriever.GPT-4 surpasses the second-best model by at most nearly 10 F1-recall scores with less hallucination.Source: §1 · p. 2
• The paper presents dynamic questions as a new direction for Chinese LLM research and positions CDQA as a high-quality data resource for advancing Chinese LLMs.The authors also describe experimental analyses as instructive for subsequent research on dynamic questions.Source: §1 · p. 2

2.1 Overview

• CDQA currently contains 1,339 questions from varied topics, mainly based on entities in recent Chinese Internet news, alongside queries from Chinese labelers.The dataset is described as being regularly updated.Source: §2.1 · p. 2

2.2 Data Collection

• SeqGPT extracts entities and GPT-4 generates candidate questions; Chinese annotators filter, rewrite, tag, verify answers, and provide supporting evidence URLs.Questions are tagged as fast-changing, slow-changing, or never-changing; answer types include person, location, time, event, artificial work, group, nature, quantity, and other. Average annotation agreement exceeds 63.1 and is described as substantial.Source: §2.2 · pp. 2–3, Table 2 · p. 3, Figure 1 · p. 3

2.3 Regular Updates

• Because answers evolve, the dataset is calibrated mainly with information from the Chinese Internet and is committed to regular updates.Source: §2.3 · p. 3

2.4 Data Statistics

• 44.6% of synthetic data are accepted, 53.1% of accepted items need modifications, and nearly 70% of extracted entities fall into person and group categories.Fast- and slow-changing questions are relatively balanced, with fewer never-changing questions; distributions may be recalibrated through re-annotation.Source: §2.4 · p. 3

2.5 Evaluation

• Evaluation covers close-book and search-augmented QA, using answer rate and F1-recall; F1-recall counts only questions with effective responses.Answer rate is the ratio of effectively answered questions to total questions and is used to indicate hallucination degree.Source: §2.5 · pp. 3–4

3.1 Experiment Setup

• The evaluation tests standalone LLMs in close-book settings and models given retrieved Internet results in open-book settings, using Google and Bing search.The open-book inputs include Top-5 or Top-10 search results.Source: §3.1 · p. 4
• The Chinese prompt framework combines instructions, optional human-written demonstrations, search results, and the current question, with Vanilla, CoT, and RaR instructions.Demonstrations use zero-shot, 5-shot, or 16-shot settings.Source: §3.1 · p. 4

3.2 Results and Analyses

• GPT-4 ranks best across all questions; Deepseek-67B-Chat nevertheless surpasses it on slow- and never-changing questions with Vanilla searched results.GPT-4 reaches nearly or over 65, 70, and 80 F1-recall on fast-, slow-, and never-changing questions, respectively.Source: §3.2 · pp. 4–5, Table 4 · p. 5, Table 3 · p. 5
• In close-book zero-shot, GPT-4’s CoT and RaR answer rates rise at most +522% and +176%, while F1-recall falls -43% and -17% versus Vanilla; both models favor Vanilla.The analysis describes the increased answer rates with CoT and RaR as hallucination, especially without evidence.Source: §3.2 · p. 5, Figure 3 · p. 6
• With searched evidence, CoT and RaR outperform Vanilla for GPT-4, and CoT performs best with lower answer rate; Deepseek-67B-Chat performs best with Vanilla.GPT-4’s open-book F1-recall scores increase substantially compared with close-book results.Source: §3.2 · p. 5, Figure 4 · p. 6
• Open-book zero-shot prompt effects vary by model: Deepseek-34B-Chat declines with CoT and RaR, while ChatGPT and Internlm-20B-Chat prefer RaR and GPT-4 and Yi-34B-Chat prefer CoT.Aquila2-34B-Chat is reported as robust across prompt types; answer rates in this comparison are ≥90%.Source: §3.2 · p. 6, Figure 5 · p. 7
• More few-shot examples improve most open-sourced models, but ChatGPT and GPT-4 show declining F1-recall as their answer rates remain high or increase.ChatGPT’s answer rates remain ≥95%, while GPT-4’s increase; the authors interpret the combined trends as more hallucinations with more examples.Source: §3.2 · pp. 6–7, Figure 6 · p. 7, Table 6 · p. 7
• For answer types, GPT-4’s internal knowledge is described as poorer than Deepseek-67B-Chat’s, but its learning-efficiency ratio reaches 1370% versus 219% with retrieved evidence.The quantity answer group is hardest for models, which the authors link to answer granularity and reasoning needs.Source: §3.2 · p. 7

4 Related Work

• StreamingQA uses questions tied to given dates to examine how open- and closed-book QA models adapt to new knowledge over time and the role of retrieval augmentation.Source: Temporal and Dynamic QA Benchmark · p. 7
• With Google’s top 10 search results, GPT-4 reaches a 100% answer rate, while Deepseek-67B-Chat maintains a 100% answer rate across time.The figure caption also identifies F1-recall comparisons across answer types and open- and closed-book settings.Source: Figure 8 · p. 8, Figure 7 · p. 8
• RealTimeQA is described as a weekly updated dynamic QA benchmark, and as the closest comparison to semi-automatic generation from the latest Chinese news corpus.Source: Temporal and Dynamic QA Benchmark · p. 8
• Many English QA benchmarks cover varied formats and topics, but are mostly static and focus on less-evolving knowledge that may already appear in pretraining data.Source: Temporal and Dynamic QA Benchmark · p. 8
• Chinese QA benchmarks are less developed, and existing benchmarks risk data leakage or saturation when evaluating questions that require fresh knowledge.The cited passage names C-Eval, WebCPM, and SuperCLUE as examples.Source: Temporal and Dynamic QA Benchmark · p. 8

5 Conclusion

• CDQA is presented as a resource and source of guidance for subsequent research on Chinese LLMs and their ability to handle dynamic questions.Source: §5 · p. 8
• The authors say CDQA’s semi-automatic pipeline provides more data than FreshQA, while acknowledging that it is not a complete LLM evaluation.Source: §5 · p. 8
• Tool learning, LLM safety, and robustness remain dimensions for future evaluation, alongside deeper analysis of capabilities using CDQA.Source: §5 · p. 8

Limitations

• CDQA studies Chinese only, so its language scope does not extend to other languages.Source: Limitations · p. 9
• Because answers change with world developments, maintaining correct, up-to-date answers is an ongoing limitation for CDQA.Source: Limitations · p. 9
• The authors commit to regular dataset updates and providing researchers with the latest version for evaluation.Source: Limitations · p. 9

Ethics Statement

• The study evaluates public, widely used LLMs, which the authors say do not pose potential ethical risks in this setting.Source: Ethics Statement · p. 9
• Collected CDQA samples were manually cleaned and pre-processed to exclude politically sensitive, violent, or private data.Source: Ethics Statement · p. 9
• The authors report signing legal labor contracts with annotators and paying above market rates based on workload.Source: Ethics Statement · p. 9

• Table 7 presents the tag taxonomy used for CDQA.Source: A Tag Taxonomy of CDQA · p. 10

• Nearly 80% of slow-changing questions concern people and groups; artificial work is the third-largest answer category after them.Source: B Dataset Distributions · pp. 10, 12
• Table 7 describes and exemplifies the first three question-tag rows and the last nine answer-type rows, with examples translated from Chinese.Source: Table 7 · p. 11
• Figures 9 and 10 show full-data distributions of question tags and answer types, respectively.Source: Figure 9 · p. 12, Figure 10 · p. 12
• Time reaches its highest share in the never-changing category, where most time-related questions concern frequencies.Source: B Dataset Distributions · p. 12

• Figure 14 illustrates the translated Chinese prompt framework, identified as the framework for Figure 2.Source: C Translated Chinese Prompts · p. 12, Figure 14 · p. 12
• Figures 11, 12, and 13 show answer-type distributions for fast-, slow-, and never-changing questions, respectively.Source: Figure 11 · p. 12, Figure 12 · p. 12, Figure 13 · p. 12