Source-linked AI summary
Terminal-Bench-LILT: Multilingual Agentic Coding Benchmark Grounded in Language, Region, and Culture
Yunsu Kim, Kaden Uhlig, Ashwin Purohit, Milind Agarwal, Patrick Simianer, Anil Arslan, Kiarash Mokhtari, Thomas Zenkel, Johannes Mosig, Gabriel Bretschner, Shamik Bose, Joern Wuebker, John DeNero
TL;DR
Coding-agent benchmarks largely evaluate English-centric software development, leaving locale-specific multilingual engineering underrepresented. Terminal-Bench-LILT introduces 300 native-authored terminal tasks in ten languages with layered quality control, then evaluates frontier models. The strongest model passes 63.1% of trials, while performance varies by language and does not follow general coding-benchmark rankings.
Problem
Existing coding benchmarks are predominantly English and encode English-speaking data, locales, and conventions, limiting evaluation of non-English development contexts.
Method
Terminal-Bench-LILT provides 300 native-authored coding tasks in ten non-English languages, each targeting locale-specific engineering challenges and validated through layered review.
Results
Performance varies substantially by language and does not track general coding-benchmark rankings; the strongest model reaches a 63.1% pass rate and many tasks remain unsolved.
Takeaways & Limitations
Multilingual coding competence is a distinct capability axis with gaps across models, languages, and task categories.
Takeaways & Limitations
The ten languages are a small, uneven sample, and low scores indicate a measurement gap rather than a verdict on any language or its speakers.
Abstract
from arXiv · showhide
Most evaluations for coding agents are conducted exclusively in English, which does not reflect real-world multilingual deployment. We present Terminal-Bench-LILT, a suite of 300 authentic coding tasks in ten languages: Arabic, Czech, German, Spanish, Hindi, Japanese, Korean, Serbian, Turkish, and Chinese. Each task targets issues specific to non-English software development that have no direct English equivalent, e.g., internationalization, encoding, text normalization, and cultural conventions. All tasks are authored by native-speaker programmers and validated through a multi-stage quality control pipeline. Evaluation of six frontier models reveals that even the strongest model reaches only 63.1\% pass rate, with many tasks unsolved by any model. Performance varies substantially by language and does not track general coding benchmark rankings, highlighting that multilingual coding competence is a distinct and underexplored capability axis. Sample tasks are available at https://github.com/lilt/terminal-bench-lilt
1 Introduction
English dominates software development and coding-agent evaluation, leaving locale-specific engineering challenges underrepresented. Terminal-Bench-LILT addresses this gap with 300 native-authored tasks across ten languages that embed such challenges in realistic coding work.
- English-centric development ecosystems and benchmarks primarily reflect English-speaking data, locales, and conventions.
- Terminal-Bench-LILT contains 300 terminal-based coding tasks in ten languages across three difficulty tiers.
- Native programmers authored tasks from problems they had encountered, followed by layered automated and human quality control.
- A Japanese sample task requires fitting mixed Japanese text, ASCII, emoji, and terminal escapes into a fixed 40-column display.
- The task tests implicit locale knowledge, including invisible variation selectors and compatibility ideographs that careless normalization can alter.
2 Related Work
Existing coding benchmarks and multilingual evaluations largely preserve English task contexts or test translated instructions. Terminal-Bench-LILT instead evaluates locale-specific coding knowledge through native-authored, culturally grounded tasks.
- Established agentic coding benchmarks are English-only, including Terminal-Bench, while Terminal-Bench-LILT extends its task format to ten non-English languages.
- Many multilingual coding benchmarks vary programming languages or translate English prompts, primarily measuring prompt comprehension rather than usefulness in non-English development.
- Translation-based evaluations confound translation errors with capability limits because their tasks remain rooted in English contexts.
- Terminal-Bench-LILT varies human language together with regional and cultural context, using native-authored tasks that avoid translation artifacts.
- Reasoning-language interventions may reduce instruction-language difficulty, but the benchmark targets implicit locale knowledge that reasoning alone does not close.
3 Terminal-Bench-LILT
Terminal-Bench-LILT combines native-language task materials, locale-specific challenges, deterministic testing, and layered quality control. Its design aims to test practical local knowledge without relying on general engineering complexity or unbounded factual recall.
- Task Design: The benchmark has 300 tasks, with 30 tasks in each of ten languages, running in isolated environments with deterministic verification.
- Task Design: Each task includes a native-language and English instruction, environment and data assets, deterministic verifier, oracle solution, and multilingual metadata.
- Task Design: Instructions are written natively first because locale-specific data, formats, colloquial phrasing, and domain vocabulary are naturally expressed in the target language.
- Task Design: Tasks must represent plausible local problems, derive difficulty from locale-specific elements, and embed those challenges implicitly rather than merely translating ordinary programming tasks.
- Test Design: Test suites combine integrity checks, sample probes, and unseen probes to test generalization while discouraging hard-coded solutions.
- Task Design: Unbounded challenges are admitted only when instructions and tests restrict them to a finite range, such as a legal snapshot and product list for German VAT rates.
- Quality Control: Every task passes four review layers—deterministic checks, LLM judging, initial human review, and final human review—and about 25% of reviewed tasks were rejected.
- Quality Control: Review agents and the Benchito management application support detailed review, task comparison, workflow tracking, notifications, and auditability.
4 Experiments
Experiments evaluate six frontier models on Terminal-Bench-LILT and find substantial variation across models, languages, and task categories. Ablations indicate that the benchmark’s difficulty primarily reflects locale-specific engineering challenges rather than instruction language or reasoning availability alone.
- Experimental Setup: Six frontier LLMs were evaluated across Terminal-Bench-LILT using five runs per task-model pair and provider-default reasoning settings.Failing trials were manually inspected to separate language-specific failures from infrastructure issues.
- Main Results: 63.1% was the strongest aggregate pass rate, while the strongest model still failed 36.9% of trials.GPT-5.5, Gemini 3.5 Flash, and Claude Opus 4.8 formed the leading group; GPT-5.5 improved 31.9 points over GPT-5.4.
- Interpretation and Limitations: Language and category comparisons are descriptive because empirical difficulty varies across languages and category sizes are unequal.The benchmark nevertheless exposes broad performance patterns for further investigation.
- Per-Language Results: Performance varied substantially across languages, and model rankings changed by language.Hindi and German were lowest for every model at roughly 50–53%; GPT-5.5 led Arabic, Spanish, Japanese, and Serbian, while Gemini 3.5 Flash led German and Korean.
- Per-Language Results: Gemini 3.5 Flash had the most uniform language performance, with σlang = 4.9 and no language below 52%.Hindi and Korean had the smallest cross-model variation, while Japanese, Spanish, and Chinese had the largest.
- Per-Category Results: Text handling was hardest for every model, while text conversion and environment interaction were easiest.GPT-5.5’s best-to-worst category gap reached 18.0 points, between text conversion and text handling.
- Effect of Instruction Language: English instructions changed pass rates by fewer than 7 points across all nine tested model-language configurations, while results remained around 50–60%.The comparison held data, environment, and verifier fixed, indicating that instruction language was not the main bottleneck.
- Effect of Reasoning: Reasoning improved GPT-5.5 but produced no significant effect for Gemini 3.5 Flash or Claude Opus 4.8.Reasoning increased output tokens to 1.4–2.7× off/minimal levels, yet most Terminal-Bench-LILT trials still failed; some German and Hindi tasks remained at 0%.
5 Conclusion
The paper introduces Terminal-Bench-LILT, a native-first benchmark of locale-specific agentic coding tasks authored by programmers across ten non-English languages. Frontier models fail over a third of trials, with performance varying by language and ranking unlike general coding benchmarks, while ablations show that the difficulty is not caused by non-English prompting or limited compute.
- Benchmark Contribution: Terminal-Bench-LILT contains 300 agentic coding tasks in ten non-English languages, authored by native-speaker programmers.Tasks embed language-, region-, and culture-specific knowledge into everyday engineering work and undergo layered review for realism, verifiability, and locale specificity.
- Findings: Frontier models fail over a third of trials, many tasks are unsolved by every model, and pass rates vary widely by language.Model rankings also do not follow general coding-benchmark standing.
- Implications: Ablation studies show that the benchmark’s difficulty is not an artifact of non-English prompting or limited compute.The native-first design exposes multilingual capability gaps that English benchmarks cannot reveal.
Ethics Statement
The benchmark addresses contributor privacy and scope limitations explicitly. It cautions that language-level scores should not be overinterpreted as judgments about speakers or deployment fitness.
- The ten languages form a small, uneven sample, and individual languages encompass locales that the benchmark does not fully cover.
- Weak scores indicate a measurement gap rather than a verdict on a language or its speakers.
- Strong scores do not certify deployment fitness, while weak scores should prompt further study rather than language exclusion.
A Sample Tasks
The paper provides sample tasks in the benchmark.
- Table 6 lists sample tasks in the benchmark.