Source-linked AI summary
Ebisu: Benchmarking Large Language Models in Japanese Finance
Xueqing Peng, Ruoyu Xiang, Fan Zhang, Mingzi Song, Mingyang Jiang, Yan Wang, Lingfei Qian, Taiki Hara, Yuqing Guo, Jimin Huang, Junichi Tsujii, Sophia Ananiadou
TL;DR
Japanese financial communication combines linguistic and pragmatic features that make implicit stance and terminology scope difficult for LLMs. The paper introduces EBISU, an expert-annotated benchmark with JF-ICR and JF-TE, and finds that current models struggle on both tasks: scaling helps only modestly, while language- and domain-specific adaptation is unreliable. EBISU provides a focused evaluation resource for Japanese financial language understanding, with scope limited by its dataset coverage and current release design.
Problem
Japanese finance encodes meaning through head-final morphology, indirect expression, mixed scripts, and nested terminology, while existing evaluations underrepresent these conditions.
Method
EBISU evaluates two expert-annotated tasks: implicit commitment and refusal recognition in investor-facing Q&A, and hierarchical extraction and ranking of nested financial terminology in disclosures.
Results
Current LLMs struggle on both tasks; scaling yields limited gains, while Japanese- and finance-specific adaptation does not reliably improve performance and can degrade term extraction.
Takeaways & Limitations
EBISU exposes persistent gaps in models’ handling of the linguistic and cultural characteristics of Japanese financial communication.
Takeaways & Limitations
EBISU covers four companies’ Q&A transcripts from 2023–2026 and EDINET Annual Securities Reports, and currently provides no train/validation/test splits.
Abstract
from arXiv · showhide
Japanese finance combines agglutinative, head-final linguistic structure, mixed writing systems, and high-context communication norms that rely on indirect expression and implicit commitment, posing a substantial challenge for LLMs. We introduce Ebisu, a benchmark for native Japanese financial language understanding, comprising two linguistically and culturally grounded, expert-annotated tasks: JF-ICR, which evaluates implicit commitment and refusal recognition in investor-facing Q&A, and JF-TE, which assesses hierarchical extraction and ranking of nested financial terminology from professional disclosures. We evaluate a diverse set of open-source and proprietary LLMs spanning general-purpose, Japanese-adapted, and financial models. Results show that even state-of-the-art systems struggle on both tasks. While increased model scale yields limited improvements, language- and domain-specific adaptation does not reliably improve performance, leaving substantial gaps unresolved. Ebisu provides a focused benchmark for advancing linguistically and culturally grounded financial NLP. All datasets and evaluation scripts are publicly released.
1 Introduction
EBISU addresses Japanese financial language understanding through two expert-annotated tasks targeting implicit stance and hierarchical terminology grounding. Evaluation across 22 LLMs finds persistent difficulty, limited scaling gains, and unreliable benefits from language- or domain-specific adaptation.
- Japanese financial communication challenges LLMs through agglutinative, head-final structure, indirect refusals, mixed scripts, and ambiguous terminology boundaries.These properties make modality, negation, pragmatic intent, and semantic scope difficult to recover.
- The benchmark uses expert annotation and diagnostic evaluation to assess Japanese financial language understanding beyond conventional short-context, explicit-target formats.The datasets are independently double-annotated, adjudicated by a senior expert, and accompanied by agreement and quality checks.
- EBISU introduces JF-ICR for implicit commitment recognition and JF-TE for finance-term extraction and ranking under Japanese-specific linguistic and discourse conditions.The benchmark targets stance inference in investor-facing communication and terminology grounding in mixed-script text.
- 22 LLMs show low overall performance, while scaling improves JF-ICR by +0.33 Acc and JF-TE by +0.38 F1, with limited gains.Japanese adaptation does not consistently outperform matched English counterparts, and financial pretraining can reduce JF-TE by -0.12 F1.
- EBISU and its annotation guidelines and evaluation code are publicly released to support language- and culture-aware financial NLP research.
2 Related Work
Related work describes a progression from Japanese model adaptation and financial specialization to benchmarks for Japanese financial knowledge and expert reasoning.
- Japanese LLM development has progressed from general-purpose adaptation toward deeper financial specialization.The cited examples include Swallow, Sarashina2, and Stockmark-LLM3.
- Swallow established a pipeline for improving Japanese syntax and reasoning through continual pre-training.
- Japanese financial evaluation has expanded from knowledge-based tests to expert-level tasks such as fraud detection and earnings forecasting.Japanese-LM-Fin-Harness incorporated securities analyst exams, CPA audit theory, and binary sentiment analysis, while EDINET-Bench added expert tasks.
3 EBISU Benchmark
EBISU evaluates native Japanese financial understanding through two complementary tasks: implicit stance recognition in high-context Q&A and nested terminology extraction from professional disclosures. The benchmark uses expert annotation and explicit task formulations to capture linguistic, pragmatic, and lexical challenges in Japanese finance.
- Benchmark overview: EBISU comprises JF-ICR for implicit commitment and refusal recognition and JF-TE for nested financial-term extraction and ranking.JF-ICR targets high-context communication, while JF-TE models nested term structures in professional financial text.
- Linguistic motivation: Japanese financial understanding is difficult because indirect intent, sentence-final pragmatic information, mixed scripts, nested compounds, and semantically shifted loanwords complicate interpretation.These properties affect both implicit stance inference and boundary-sensitive terminology extraction.
- JF-ICR: JF-ICR classifies a Japanese financial response into five intent levels from strong commitment to strong refusal.The task distinguishes commitment, qualified commitment, neutral or hedged intent, weak refusal, and strong refusal rather than surface sentiment.
- JF-ICR: JF-ICR contains 94 curated single-turn Q&A instances from public corporate disclosures, annotated by finance experts using a detailed five-point guideline.The corpus draws on investor briefings, shareholder meetings, and financial-results briefings, while excluding exchanges requiring broader context.
- Validation and evaluation: Annotation reliability is assessed with Macro-F1, Cohen’s κ, and Krippendorff’s α, with reported results indicating strong or high agreement.The benchmark also uses Accuracy for JF-ICR evaluation and term-extraction metrics for JF-TE.
- JF-TE: JF-TE identifies maximal financial terms, extracts nested candidates within them, and ranks candidates by their likelihood of representing domain-specific terminology.Non-financial expressions are excluded, and the output aggregates ranked candidate lists across maximal terms.
4 Experimental Results
The experiments evaluate a broad set of LLMs on EBISU using a unified, deterministic pipeline. The model set spans open and proprietary systems as well as general-purpose, Japanese-adapted, and financial-domain models.
- Model coverage: The evaluation covers 22 LLMs across open-weight, proprietary, English financial, Japanese general-purpose, and Japanese financial model categories.The model families include systems adapted to Japanese language or financial domains alongside general-purpose baselines.
- Evaluation setup: Proprietary models use official APIs with temperature fixed at 0, while open-source models are served through TogetherAI or deployed locally with vLLM.The unified pipeline is based on the LM Evaluation Harness.
- Experimental analysis: The experimental analysis is organized around research questions concerning EBISU performance and transfer across language and domain boundaries.Table 4 and Figure 2 summarize model performance on the benchmark.
RQ1: Is EBISU challenging for state-of-the-art LLMs under Japanese financial settings?
EBISU remains difficult for state-of-the-art LLMs across implicit-commitment recognition and financial-term extraction. Larger models improve performance only modestly, while language- and domain-specific adaptation does not reliably close the gap.
- Overall difficulty: Current LLMs, including state-of-the-art systems, face substantial challenges on EBISU.The benchmark exposes persistent difficulty modeling Japanese financial language and pragmatics.
- Overall difficulty: The best overall result is Llama-3.3-70B-Instruct at 0.4011, while GPT-4o reaches 0.2461 and Claude-Sonnet-4.5 reaches 0.3618.These results indicate limited performance even among leading proprietary systems.
- Language and domain transfer: FinMA-7B scores 0.0319, below comparable general-purpose Qwen3-8B at 0.0450, showing limited transfer from English financial training to Japanese finance.Japanese-adapted models also do not consistently outperform their original or general-purpose counterparts.
- JF-ICR: JF-ICR remains difficult: Llama-4-Scout-17B achieves only Acc 0.6064 when recognizing implicit agreement and refusal.English-centric models also assign higher average commitment scores than Japanese counterparts in comparable model families.
- JF-TE: On JF-TE, Llama-3.3-70B-Instruct reaches HitRate@1 0.1277, HitRate@5 0.3657, and HitRate@10 0.5111.The results show that identifying and ranking Japanese financial terminology remains far from solved.
5 Conclusion
EBISU is a benchmark for Japanese financial language understanding, designed around implicit commitment recognition and financial term extraction. Experiments show that current LLMs struggle on both tasks, while scale helps somewhat and language- or domain-specific training provides limited gains.
- EBISU evaluates Japanese financial language understanding through implicit commitment recognition and financial term extraction.Its design explicitly reflects Japanese linguistic structure and communication norms.
- Current LLMs, including state-of-the-art proprietary and open-source systems, struggle on both EBISU tasks.
- Increased model scale yields some improvements, but language- and domain-specific training provides limited gains and can even degrade performance.
- EBISU exposes persistent gaps associated with the linguistic and cultural characteristics of Japanese financial communication.
Limitations
EBISU’s coverage is constrained by its company, time, document, task, text-length, and metric scope. The release also omits train, validation, and test splits because of the current dataset size.
- JF-ICR covers Q&A transcripts from 4 companies during 2023–2026, while JF-TE uses only Annual Securities Reports from EDINET.This may not represent communication across company sizes, industries, or disclosure formats.
- EBISU does not provide train, validation, and test splits in the current release because of the dataset size.The authors plan to introduce standard splits after expanding the dataset.
- The two tasks do not cover the full spectrum of financial NLP capabilities and use relatively short text segments.This limits evaluation of long-context understanding and multi-turn discourse reasoning.
- Accuracy for JF-ICR treats all misclassifications equally, while exact matching for JF-TE may penalize slightly misaligned but semantically correct spans.
Ethical Concerns
The authors caution that EBISU scores may not predict production utility and that its focused data may not represent Japanese finance broadly. The annotation and task designs also encode specific domain and discourse assumptions.
- EBISU benchmark performance may not fully predict real-world production utility.Latency, distribution-shift robustness, and workflow integration are also critical.
- The datasets draw from a limited set of large publicly listed companies, which may underrepresent broader Japanese financial communication.The temporal scope may also miss evolving terminology or regulatory changes.
- Models could be optimized for EBISU without corresponding improvements in generalization, creating a risk of benchmark-specific overfitting.
- JF-ICR annotations classify the speaker’s future-oriented commitment or refusal rather than factual correctness or the question’s surface intent.Non-zero labels require future actions, decisions, policies, targets, or strategic directions.
- JF-TE identifies minimal financial or accounting term spans in explanatory disclosure notes and related regulatory or definitional text.Annotators prefer noun or noun-phrase spans and exclude surrounding explanatory material.
C Annotator Demography
EBISU annotation combines financial expertise, Japanese proficiency, academic training, and industry experience. This combination is intended to support technically accurate and contextually grounded annotations.
- EBISU annotations were produced by annotators with financial expertise, professional experience, and high Japanese proficiency.
- One annotator is a Japan-based doctoral student trained in financial mathematics with prior quantitative-research experience in finance.
- Another annotator is a Japanese fintech researcher with nearly two decades of market experience and expertise in corporate finance, reporting, and investor communications.
- The annotators’ combined expertise supports annotations that are technically accurate and grounded in Japanese financial context.
D Annotation Process
The annotation process uses standard agreement metrics and detailed, expert-guided procedures to validate consistency across the benchmark’s labels.
- Agreement Metrics: Three agreement metrics—Macro-F1, Cohen’s κ, and Krippendorff’s α—are used to assess annotation reliability.The metrics capture balanced class agreement and adjust for chance agreement and label-distribution effects.
- Macro-F1: Macro-F1 averages per-class F1 scores uniformly, giving every category equal weight.Compared with accuracy, it is more sensitive to skewed label distributions.
- Cohen’s κ: Cohen’s κ measures observed agreement after correcting for chance agreement.κ = 1 indicates perfect agreement, while κ = 0 corresponds to chance-level agreement.
- Krippendorff’s α: Krippendorff’s α accommodates multiple annotators and missing labels, with α = 1 indicating complete agreement.For nominal categories, the metric uses disagreement and expected disagreement based on annotation counts.
F Evaluation Metrics
EBISU evaluates its two tasks with metrics designed for multi-class stance classification and hierarchical financial-term extraction and ranking.
- JF-ICR: JF-ICR is treated as multi-class classification over the five-label set {+2, +1, 0, −1, −2}.Accuracy compares each predicted label with its annotated gold label.
- JF-TE: JF-TE uses a two-level metric design to reflect the hierarchical structure of Japanese financial terminology.The design evaluates both maximal terms and nested-term ranking.
- Maximal Financial Term F1: Maximal Financial Term F1 compares predicted and gold sets of maximal financial terms using precision and recall.This metric evaluates extraction of the longest relevant financial-term spans.
- Nested-Term Ranking: HitRate@K measures whether gold nested terms appear among the top-K predicted terms.The benchmark reports HR@1, HR@5, and HR@10 to represent different ranking tolerances.