Source-linked AI summary
BrailleBench: Investigating Multi-Criteria Braille Comprehension in Large Language Models
Jinghan Zhang, Fengran Mo, Zhiyu Chen, Xiaoyan Han, Kunpeng Liu, Chang-Tien Lu
TL;DR
It is unclear whether print-English LLM capabilities provide equivalent functionality through Braille, an underexplored interaction-accessibility channel for blind and deafblind users. BrailleBench evaluates this question across interaction directions, grades, tasks, and six LLMs, finding that English capability does not transfer reliably to Braille, especially for Grade 2 input and fully Braille interaction.
Problem
Whether general-purpose LLMs can sustain task-oriented interaction through Braille remains underexplored, despite Braille serving as a precise, private, editable channel for blind and deafblind users.
Method
BrailleBench evaluates Braille comprehension, generation, and end-to-end interaction across English, Grades 1 and 2, five datasets, 5,570 instances, and six LLMs using a deterministic toolkit.
Results
Braille understanding and expression are asymmetric: Grade 2 is more challenging than Grade 1, fully Braille interaction is least reliable, and English performance declines broadly in Braille.
Takeaways & Limitations
Existing proprietary models may contain fragments of Braille knowledge but lack robust Braille interaction capability, motivating further development of Braille AI systems.
Takeaways & Limitations
The conservative Braille-output validator cannot perfectly distinguish lowercase print English from Grade 1 Braille ASCII, so English-output rates are interpreted as lower bounds.
Abstract
from arXiv · showhide
Although Large language models (LLMs) mediate access to knowledge and computational assistance, their capabilities should benefit vulnerable groups in the same way. However, it is unclear whether existing AI systems are inclusive enough for blind and deafblind users to access the same functionality through Braille, whose indicators, contractions, and digital representations introduce distinct requirements for model comprehension. To this end, we introduce BrailleBench, a benchmark for evaluating LLMs in Braille comprehension from different Criteria. BrailleBench aligns 5,570 instances from five datasets, including mathematics, commonsense, and multi-hop question answering across English and Braille Grades 1 and 2. Different configurations are designed to understand whether the systems can comprehend Braille-authored content, express answers in Braille, and complete end-to-end Braille interaction. To ensure the quality and prevent evaluation bias, the benchmark is built through a deterministic, expert-reviewed pipeline via a self-created Braille Toolkit without using any data instances generated by LLMs. We evaluate six representative LLMs from various aspects. The results reveal a persistent gap between print-English capability and Braille accessibility. Braille understanding and expression are asymmetric, where Grade 2 is especially fragile on the input side compared to Grade 1, and fully Braille requests further reduce performance. The experimental observations provide valuable guidance for the development of future Braille AI systems. All related resources in BrailleBench are publicly available for future research.
I. INTRODUCTION
Braille is an underexplored interaction channel for blind and deafblind users, requiring LLMs to comprehend authored Braille and produce the requested representation. BrailleBench evaluates these capabilities across interaction directions and Grades 1 and 2 using reproducible, non-LLM-generated data.
- Motivation: Braille provides a private, editable channel for precise language, code, mathematics, and notation, not merely an alternative rendering of print or speech.The paper frames Braille support as interaction accessibility and notes that general-purpose LLM interaction through Braille remains underexplored.
- Evaluation goals: BrailleBench evaluates Braille-to-English comprehension, English-to-Braille response generation, and Braille-to-Braille end-to-end interaction.Both Grade 1 and Grade 2 are included to distinguish uncontracted from contracted Braille capability.
- Evaluation goals: Braille comprehension and generation are asymmetric: inputs require recovering linguistic and symbolic content, whereas outputs must follow the requested Grade representation without defaulting to print English.The benchmark uses a liblouis-based toolkit and fixed UEB tables to control Braille expression quality and reduce evaluation bias.
- Benchmark contribution: 5,570 instances span Grade 1 and Grade 2 Braille and multiple task types, supporting evaluation of inclusive interaction across diverse capabilities.The benchmark includes different task types and is intended to facilitate interaction between blind users and AI systems.
- Braille challenges: Grade 1 and Grade 2 can express the same content differently because Grade 2 uses contractions, while mathematical Braille may require longer sequences of cells and indicators.Figure 1 illustrates both the expanded Grade 1 representation of print mathematics and differing Grade 1/Grade 2 sequences.
II. RELATED WORK
The related work presents Braille as a structured digital interaction medium with multiple representations and grade systems. Prior systems and language-model studies mainly address entry, accessible authoring, translation, disambiguation, or recognition rather than broad task-oriented LLM interaction.
- Digital representations: Braille ASCII encodes each six-dot cell as one printable character, while Unicode and dot-number notation provide alternative digital representations with different tokenizer or sequence-length properties.Rendered Braille additionally requires a visual encoder, whereas Braille ASCII can pass through standard text APIs without rendering.
- Braille grades: UEB Grade 1 preserves spelling explicitly, whereas Grade 2 shortens frequent words and letter groups through contractions requiring lexical and positional disambiguation.Evaluating both grades separates sequence-length difficulty from contextual contraction difficulty.
- Accessible interaction: Accessible-interaction systems treat blind users as active authors, including gesture-based Braille entry and screen-reader-accessible editing with multimodal LLM assistance.These systems motivate evaluating both user-authored input access and accessible model output.
- Prior language-model work: Existing language-model work on Braille primarily studies translation, disambiguation, and visual recognition, including Braille-specific training and specialized tokenization or generation modules.Prompted LLMs have also been used for pronunciation choices in Taiwanese Mandarin and Japanese Braille.
III. BRAILLEBENCH
BrailleBench aligns five source datasets with English, Grade 1, and Grade 2 representations and evaluates task-oriented interaction through multiple input–output directions. Its protocol distinguishes Braille comprehension, Braille output, and end-to-end Braille interaction against English-to-English performance.
- Benchmark design: BrailleBench adapts five source datasets through four stages: source-task and protocol selection, data construction, quality control, and LLM benchmarking.The construction and evaluation stages are summarized in Figure 2.
- Source tasks: The benchmark contains 5,570 source instances from grade-school arithmetic, competition mathematics, commonsense reasoning, and multi-hop question answering datasets.It uses complete test sets for GSM8K, AIME 2024, and CommonsenseQA, plus bounded development-set samples for HotpotQA and 2WikiMultiHopQA.
- Benchmark composition: Each data sample is aligned across English, Grade 1 Braille, and Grade 2 Braille formats.This alignment supports comparison across the benchmark’s interaction configurations.
- Interaction protocols: Gi→EN tests solving Braille-authored problems, EN→Gi tests producing Braille answers, and Gi→Gi tests end-to-end Braille interaction.English-to-English remains the reference setting, with source inputs, target answers, and metrics consistent across configurations.
- Grade comparison: Grade 1 offers longer, more locally explicit Braille ASCII sequences, whereas Grade 2 is shorter but requires contextual interpretation of contractions.The two grades therefore separate surface sequence-length effects from contraction-related contextual difficulty.
B. Braille Data Instance Construction
BrailleBench constructs aligned Braille records through task-aware normalization, deterministic UEB transcription, and structured assembly rather than field-wise substitution. The pipeline preserves semantics and auditability, uses fixed tooling without LLM-generated Braille, and supports multiple Braille representations.
- Quality control: No LLM-generated Braille is used as context or ground truth, reducing evaluation-bias risk in benchmark construction.The quality-control design relies on controlled Braille expression rather than model-generated instances.
- Task-aware normalization: Mathematical expressions are linearized into explicit plain text while preserving scope for fractions, roots, superscripts, subscripts, relations, Greek letters, and answer boxes.Unsupported commands remain visible as auditable residuals rather than being silently discarded.
- Deterministic transcription: The pipeline uses liblouis 3.32.0 with fixed UEB Grade 1 and Grade 2 tables, making transcription deterministic and reproducible while retaining English source fields.Each Braille field is generated from its aligned normalized English field.
- Toolkit: Braille Toolkit generalizes the construction method with task-aware adapters, auditable normalization, UEB transcription, and synchronized Braille ASCII, Unicode, and dot-number outputs.The framework is released as an open-source, benchmark-independent resource.
3) Braille Toolkit:
The Braille Toolkit provides deterministic validation and rendering controls for constructing and evaluating BrailleBench data across representations and task settings.
- Quality control: The toolkit validates environments, normalizes Braille, back-translates records, and flags context-sensitive disagreements for inspection.It verifies specified liblouis tables and reproduces known translations before record-level checks.
- Quality control: Expert review follows automated checks to distinguish representation artifacts from model failures.Validation covers deterministic generation, back-translation, and consistency across Braille serializations.
- Representations: BrailleBench represents core Braille inputs as Braille ASCII and releases aligned Unicode and dot-number forms from the same cells.A surface-format ablation changes serialization while holding grade, task content, requested answer, and scorer fixed.
- Prompt construction: Prompt rendering varies task-bearing representations and requested output while preserving English task instructions in core settings.Braille-input conditions replace questions and applicable choices; Braille-output conditions request the corresponding grade.
- Prompt construction: Task adaptation preserves each source dataset’s response convention, including mathematical markers, answer formats, and QA answer requirements.GSM8K, AIME, CommonsenseQA, HotpotQA, and 2WikiMultiHopQA retain their task-specific conventions.
- Output evaluation: Braille-output validation maps serializations to cells, filters clear print-English responses, back-translates valid outputs, and scores them against English references.Mathematical validation distinguishes literal print numerals from valid UEB numerals.
IV. EXPERIMENTS
The experiments evaluate six language models on five datasets under closed-book, task-aware protocols with multiple metrics for mathematics and question answering.
- Experimental design: The study conducts comprehensive experiments designed to address multiple research questions.The experimental setup covers the benchmark’s language and interaction conditions.
- Metrics and datasets: Mathematics uses Math-Verify, while CommonsenseQA, HotpotQA, and 2WikiMultiHopQA use Exact Match and token-level F1.QA scores take the maximum over valid answer aliases after FlashRAG normalization.
- Metrics and datasets: Avg. Drop is the unweighted mean of each non-English condition’s score difference from the corresponding EN→EN score across five datasets.The metric compares each model’s standard English reference with its performance under each non-English condition.
- Models: The evaluation includes six proprietary and open-weight language models, including Claude, Llama, and Qwen variants.The listed models range from Qwen3 1.7B/32B to Claude Opus 4.8 and Haiku 4.5.
- Inference protocol: All evaluations are closed-book without demonstrations, retrieved context, or Braille decoding keys.Prompt construction, extraction, metrics, and output validation follow the benchmark protocol.
B. RQ1: Can LLMs Understand Braille?
Braille introduces broad performance declines and separates comprehension, expression, and end-to-end interaction, with Grade 2 input especially difficult because contractions increase contextual ambiguity.
- Across models and tasks: Braille causes broad performance declines across models and datasets, although stronger English models generally retain more capability.Opus shows the most consistent performance, while smaller models often lose most of their original task capability.
- Across models and tasks: Mathematics separates models more clearly than QA, whose performance is more variable and includes localized improvements.Model ordering is not strictly determined by scale across task families.
- Interaction directions: Braille comprehension and expression are asymmetric: models may understand Braille input yet fail to produce Braille output, while Gᵢ→Gᵢ compounds both failures.Across directions, Grade 2 is harder than Grade 1, especially for contracted input.
- Grade effects: Grade 2 is consistently harder than Grade 1 for Braille input, with the gap most visible on language-intensive QA tasks.The grade difference is smaller and less consistent when models only generate Braille from English.
- Grade effects: Grade 2 difficulty reflects contextual ambiguity rather than sequence length alone.Contractions overload individual cells, and local errors can alter words, entities, and relations before task completion.
- Multi-hop QA: Claude models improve from English-English to some Grade 1 Braille settings on two multi-hop QA datasets, despite English-English serving as the expected upper bound.The paper attributes this partly to more careful reasoning under Braille and partly to metric bias; manually activated CoT improves results across metrics.
C. RQ2: Can LLMs Understand a Pure Braille Request?
Fully Braille requests remove English task instructions and generally reduce performance, especially in Grade 2, showing that task identification itself becomes difficult.
- Pure Braille requests: Fully Braille conditions translate both instructions and problems into Grade 1 or Grade 2 Braille while retaining English output.They test comprehension without the English instruction that may assist task identification.
- Results: Transcribing the instruction and problem into Braille generally lowers performance, with effects varying by grade and model.Under FULLBR-G1, Opus retains substantial accuracy while other models decline or remain unchanged.
- Results: Under FULLBR-G2, performance is uniformly weak: Opus retains meaningful accuracy only on GSM8K, while remaining model-task combinations approach the floor.The results indicate that English instructions help most models identify the required task.
D. RQ3: How Expression Form of Braille Affect the Results?
Braille surface representation materially affects LLM performance even when the underlying Braille cells are identical. Braille ASCII is generally strongest, while dot notation consistently makes comprehension harder.
- Surface-format comparison: Identical Braille cells produce markedly different results across Braille ASCII, Unicode patterns, and dot-number notation.The comparison holds grade, task instruction, requested English answer, and scorer fixed.
- Surface-format comparison: Braille ASCII gives the strongest and most stable performance across models, grades, and tasks.
- Surface-format comparison: Unicode improves selected mathematics results for Haiku and Qwen3 32B but usually underperforms Braille ASCII elsewhere.Opus scores zero on every evaluated task in Unicode.
- Surface-format comparison: Dot notation is almost uniformly at the performance floor for both Braille grades.Expanding cells into dot-number sequences makes the input consistently harder rather than more explicit.
- Interpretation: Digital Braille representation is therefore not a neutral rendering choice for current LLMs.Performance depends on the serialized symbols through which Braille cells are presented.
E. RQ4: Can an Explicit Braille Reference Help Context Understanding?
An explicit, grade-matched Braille reference helps most with Grade 1 mathematics but does not reliably resolve sequence-level decoding, especially for Grade 2 and linguistic inputs.
- Reference intervention: A liblouis-verified, grade-matched reference supplies letters, indicators, punctuation, and common contractions without worked examples or answers.The Braille problem, English instruction and output, evaluation subset, and scorer remain fixed.
- Results: The explicit reference is most effective for Grade 1 mathematics.It substantially improves GSM8K for Haiku, Llama 3.3 70B, and Qwen3 32B, and raises AIME accuracy for several models.
- Results: Reference gains are smaller and inconsistent on QA datasets, with some model–task pairs improving while others decline.
- Results: Grade 2 references provide no consistent QA benefit and sharply reduce Opus performance.They help selected weaker mathematics results but do not generalize across tasks.
- Interpretation: Static mappings are less useful when contraction expansion depends on lexical position and surrounding context.Access to symbol mappings does not guarantee successful sequence-level decoding.
- Diagnostic evidence: Diagnostic probes reveal high local Grade 1 recognition but poor numbers and sentence accuracy, with Grade 2 harder at lexical and sentence scales.Strict Grade 2 sentence exact match is zero for every model except Opus, with limited punctuation-insensitive recovery.
V. CONCLUSION AND FUTURE WORK
BrailleBench finds that print-English capability does not transfer reliably to Braille interaction. The authors identify persistent decoding and evaluation-boundary issues and call for specialized future Braille evaluation and adaptation.
- Conclusion: BrailleBench evaluates English, UEB Grade 1, and UEB Grade 2 configurations across six representative LLMs.
- Conclusion: English capability does not transfer reliably to Braille, and existing proprietary models lack robust Braille interaction capability despite possessing fragments of Braille knowledge.
- Evaluation caveat: The Braille-output validator cannot perfectly distinguish lowercase print English from Grade 1 Braille ASCII when their byte strings coincide.Accordingly, English-output rates are treated as lower bounds, with numbers and contraction-bearing outputs weighted more heavily.
- Evaluation caveat: Internal causes behind API-hosted models’ response lengths and empty-output patterns remain hypotheses because tokenizer traces and moderation decisions are unavailable.
- Future work: Future work should investigate Braille tokenization, reasoning activation, truncated-output handling, error recovery, user-centered usability, and safety-aware mechanisms.The paper states that these Braille-specific scenarios should be quantified.