PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations

Jungo Kasai, Yuhei Kasai, Keisuke Sakaguchi, Yutaro Yamada, Dragomir Radev

arXiv:2303.18027v2cs.CL

TL;DR

Translation-based non-English medical benchmarks can diverge from native-speaker use and local practice, motivating evaluation directly on Japanese licensing exams. GPT-4 performed best and passed all six years, though it remained below the student-majority baseline.

  • Problem

    Translation-based benchmarks can become English-centric and diverge from native-speaker use and local medical practice, motivating evaluation on Japanese exams.

    Source: §1 · p. 2

  • Method

    The study evaluates ChatGPT, GPT-3, and GPT-4 on Japanese medical licensing examinations from 2018 through 2023.

    Source: §2.3 · p. 4, §5 · p. 9

  • Results

    GPT-4 achieved the best performance and passed all six years, but substantially underperformed the student-majority baseline.

    Source: §3.2 · p. 6

  • Takeaways & Limitations

    The open IGAKU QA benchmark, model outputs, and exam metadata provide resources for future research.

    Source: §5 · p. 9

    Because the study uses black-box APIs with undocumented training data and setups, results may change with API updates and may be inflated by data leakage.

    Source: Limitations · p. 9

Abstract

from arXiv · show

As large language models (LLMs) gain popularity among speakers of diverse languages, we believe that it is crucial to benchmark them to better understand model behaviors, failures, and limitations in languages beyond English. In this work, we evaluate LLM APIs (ChatGPT, GPT-3, and GPT-4) on the Japanese national medical licensing examinations from the past five years, including the current year. Our team comprises native Japanese-speaking NLP researchers and a practicing cardiologist based in Japan. Our experiments show that GPT-4 outperforms ChatGPT and GPT-3 and passes all six years of the exams, highlighting LLMs' potential in a language that is typologically distant from English. However, our evaluation also exposes critical limitations of the current LLM APIs. First, LLMs sometimes select prohibited choices that should be strictly avoided in medical practice in Japan, such as suggesting euthanasia. Further, our analysis shows that the API costs are generally higher and the maximum context size is smaller for Japanese because of the way non-Latin scripts are currently tokenized in the pipeline. We release our benchmark as Igaku QA as well as all model outputs and exam metadata. We hope that our results and benchmark will spur progress on more diverse applications of LLMs. Our benchmark is available at https://github.com/jungokasai/IgakuQA.

1 Introduction

Because many LLM benchmarks are English-limited despite multilingual use, this work evaluates models on Japanese medical licensing exams using a benchmark drawn directly from Japanese exams. GPT-4 passes all six years, while the evaluation also exposes prohibited-choice and Japanese API-cost and context-size limitations.

1 Introduction

  • The benchmark evaluates GPT-3, GPT-4, and ChatGPT on Japanese medical licensing examinations and uses questions originating in Japanese rather than translated English resources.The project includes native Japanese-speaking NLP researchers and a practicing cardiologist in Japan.Source: §1 · p. 2
  • Many diverse LLM benchmarks are limited to English, motivating evaluation of specialized domains in non-English languages as LLM services are used by non-English speakers.The authors frame their work as a first step toward this goal.Source: §1 · p. 2
  • ChatGPT selected euthanasia, a prohibited choice in Japan; choosing four or more prohibited options fails test takers regardless of their total scores.The exam example and response are translated from Japanese for readability.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • GPT-4 passes all six Japanese medical licensing examinations from 2018–2023, though it substantially underperforms the medical-student majority-vote performance.The authors also report higher API costs and smaller context windows for Japanese.Source: §1 · p. 2

2 Background

The background explains how Japan’s licensing exam differs from the US pathway, outlines the Japanese exam and its passing rules, and describes the construction of a native Japanese benchmark.

2.1 National Medical Practitioners Qualifying Examination (NMPQE)

  • Japan’s NMPQE is typically taken in the final year of six-year medical school, whereas US licensing uses three steps during medical school and residency.In Japan, hands-on clinical exposure typically follows passing the exam and obtaining a license.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2 Details of NMPQE

  • Since 2018, the exam has six parts and 400 questions, mainly five-choice questions, spanning 28 medical and public-health categories.A few questions ask directly for numbers or contain more than five choices.Source: §2.2 · pp. 3–4
  • Passing requires at least 80% on required sections, 70% on general sections, and no more than three prohibited choices.More than 25 choices are marked prohibited because they represent decisions to strictly avoid in Japanese medical practice.Source: §2.2 · p. 3
  • About 25% of problems contain images, and APIs that do not accept images cannot answer some problems by design, although not all require images.The benchmark retains these questions to encourage development of multimodal models.Source: §2.2 · p. 4

2.3 Benchmark Collection

  • The benchmark collects exams from Japan’s Ministry of Health, Labour and Welfare for 2018–2023, adds test-taker choice-selection and average-accuracy metadata from preparation books (not available for 2023), and releases problems with answers.The total number of problems remains the same each year, although a few are often disregarded for difficulty or ambiguity.Source: §2.2 · p. 3, §2.3 · p. 4
  • The collection uses originally Japanese exam material rather than translations, aligning it with target-country use and mitigating translationese effects.The passage notes that translated text can differ lexically, syntactically, and stylistically from naturally occurring text.Source: §2.3 · p. 4

3 Experiments and Analysis

The experiments compare GPT-3, ChatGPT, and GPT-4 on Japanese medical licensing exams; GPT-4 performs best and passes all six years, while analyses reveal translation, tokenization, prohibited-choice, and Japan-specific context challenges.

3.1 Models and Evaluations

  • The study benchmarks GPT-3, ChatGPT, and GPT-4 on IGAKU QA in a closed-book setting, evaluates answers by exact match, and uses three randomly sampled 2006 exam examples in prompts.Source: §3.1 · pp. 4, 6, §3 · p. 4
  • ChatGPT-EN translates questions and answer choices into English, while adding intermediate explanation steps did not improve ChatGPT performance.The study also compares a student-majority baseline.Source: §3.1 · pp. 4, 6

3.2 Results

  • GPT-4 achieves the best performance across 2018–2023 and passes all six exams, but substantially underperforms the student majority baseline.GPT-4 and ChatGPT-EN are the only models that select no more than three prohibited choices over the six years.Source: §3.2 · p. 6
  • ChatGPT-EN outperforms ChatGPT in most cases, suggesting limitations in LLM multilinguality when translation is not explicit..Source: §3.2 · p. 6

3.3 Analysis, Discussion, and Examples

  • Japanese use typically requires roughly 2x as many tokens as English, increasing API costs and making Japanese context windows substantially smaller.The paper attributes this to OpenAI APIs splitting Japanese characters into multiple tokens.Source: §3.3 · p. 6
  • LLMs sometimes select prohibited choices that should be avoided in medical practice, including oral hypoglycemic agents during pregnancy.The paper cites potential dangers including fetal teratogenesis, hypoglycemia, hyperbilirubinemia, and polycythemia.Source: §3.3 · pp. 6–7, Figure 6 · p. 6
  • Some questions require geographic or temporal context, including Japan-specific knowledge about health-center duties; open-book or retrieval-augmented approaches may improve performance on them.Geographic and temporal context is not the benchmark’s main focus.Source: §3.3 · p. 7, Figure 7 · p. 7
  • GPT-4’s likelihood of correct prediction correlates with student accuracy, suggesting it struggles on questions that are also difficult for humans.Similar patterns were observed for other models.Source: §3.3 · p. 7

4 Related Work

Related work motivates evaluating language models beyond English and connects this benchmark to clinical NLP, multilingual modeling, and open-domain question answering. It highlights the value of native-language, country-specific evaluation over translated resources.

Clinical Natural Language Processing Beyond English

  • Clinical NLP remains most resource-rich in English, while Japanese efforts span core tools, datasets, and crosslingual transfer.The cited Japanese work includes parsing, vocabulary expansion, question answering, and pretrained transformers.Source: §4 · pp. 7–8

Multilingual Language Models

  • Polyglot models exploit shared characteristics across languages, and multilingual pretrained models support machine translation and crosslingual transfer.The authors suspect similar multilingual learning occurs in ChatGPT and GPT-4, which they test on Japanese.Source: §4 · p. 8

Open-Domain and Multilingual Question Answering

  • Translation-based QA can overrepresent English-centric topics and diverge from information needs in other languages.Some datasets instead sample questions from native speakers of each language.Source: §4 · p. 8
  • IGAKU QA uses problems originally written by native Japanese speakers to assess qualifications and skills for medical practice in Japan.The benchmark does not rely on translated resources.Source: §4 · pp. 8–9

5 Conclusion

The paper evaluates GPT APIs on Japanese medical licensing examinations from 2018 to 2022. GPT-4 outperforms the other models and passes the examinations, while the benchmark and model outputs are released for future research.

5 Conclusion

  • GPT-4 outperforms the other evaluated models and passes the Japanese medical licensing examinations from 2018 to 2022.The evaluations cover five years of examinations.Source: §5 · p. 9
  • The evaluation highlights limitations of current LLM APIs in a specialized Japanese domain typologically distant from English.Source: §5 · p. 9
  • IGAKU QA, model outputs, and metadata are open-sourced for future research.Source: §5 · p. 9

Limitations

The evaluation is constrained by black-box APIs, a Japanese medical-exam scope, and the omission of image input. The authors release settings and outputs to support updates and further analysis.

Limitations

  • Black-box APIs make results not fully reproducible, subject to change with API updates, and vulnerable to possible data leakage from undocumented training setups.The authors release outputs and experimental settings and include March 2023 exam results, which they believe postdate GPT-4 training; performance was consistent with previous years.Source: Limitations · p. 9
  • The benchmark covers only Japanese and Japanese medical licensing, while evaluation in medicine requires country-specific expertise and knowledge of standard practice.The authors identify evaluation in more languages and domains as future work and note risks from simply translating benchmarks.Source: Limitations · p. 9
  • The evaluations omit image information because the APIs do not support image input, although many image-based problems and medical practice require multimodal reasoning.The authors leave testing models in multimodal settings to future work.Source: Limitations · p. 9

A Cateogries

The exam problems are grouped into 28 categories based on books widely used among Japanese medical students.

A Categories

  • The 28 categories range from public health to anesthesiology, with Japanese-to-English category names presented in Table 2.Source: A Cateogries · p. 15

B Additional Exam Information

Japanese medical licensing exam passing rates are generally high, and the 2018–2021 exams show category distributions similar to 2022.

B Additional Exam Information

C Additional Settings and Results

The additional results relate LLM accuracy to student accuracy on 2022 problems, while the accompanying materials show category breakdowns and in-context examples.

C Additional Settings and Results

Loading 2303.18027v2…