PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Read Braille: Benchmarking and Improving Braille Recognition Capabilities of VLMs

Kartik Jangra, Atharva Gawane, Siddharth Pulipaka, Hemachandran Karah, Raj Dabre

arXiv:2610.36195v1cs.CVcs.CL

TL;DR

Braille reading from images remains a challenging accessibility task for modern VLMs, and strong performance on conventional text does not establish capability across writing systems. The paper introduces BrailleBench and trains models on synthetic Braille to test real-image transcription. The trained model outperforms standalone VLMs, while synthetic validation scores from prior data do not guarantee accuracy on real photographs in these experiments.

  • Problem

    Whether multimodal systems can read Braille from images remains an open accessibility-relevant question, since Braille recognition requires fine-grained dot geometry and Grade-2 context.

    Source: §1 · p. 1

  • Method

    The paper introduces BrailleBench, a 259-image real-world Grade-1 and Grade-2 benchmark, and evaluates systems including models adapted using synthetic Braille data.

    Source: §1 · p. 1, §6 · p. 8

  • Results

    Models trained on the authors’ synthetic corpus outperform every standalone VLM on Grade-1, approach the best harness, and achieve the best Grade-2 scores among evaluated systems.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The experiments show that high synthetic validation accuracy does not guarantee accurate real-photograph transcription, and real-image benchmarks are needed to measure progress.

    Source: §6 · p. 9

    The benchmark contains 259 manually curated images and focuses on English Braille; its size and collection process may introduce sampling bias.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Despite rapid progress in visual understanding and document transcription, modern vision-language models (VLMs) struggle with a fundamental accessibility task: reading Braille from images. We introduce BrailleBench, a manually curated benchmark of 259 real-world Grade-1 and Grade-2 Braille images. We evaluate proprietary, open-weight, OCR-specialized, and tool-assisted multimodal systems and find that standalone general-purpose models perform poorly at directly transcribing Braille images, whereas tool-assisted systems perform much better. We design a synthetic Braille generation pipeline with diverse layouts and image perturbations. A 2B VLM model trained on our synthetic data reaches 87% exact match on Grade-1 and 61% on Grade-2 BrailleBench, outperforming standalone VLMs and achieving the highest Grade-2 exact match score among the evaluated systems. We further show that high accuracy on synthetic validation data does not guarantee high accuracy on real photographs in our experiments: models trained on prior synthetic data reach up to 96% on their synthetic validation set but at most 3.26% on Grade-1 BrailleBench. Our results identify an accessibility-relevant capability gap in multimodal systems and show that targeted synthetic supervision can improve real-world Braille transcription.

1 Introduction

Braille transcription tests whether multimodal capabilities extend beyond conventional printed text, despite Braille’s accessibility role and distinctive visual and contextual demands. The paper introduces BrailleBench and evaluates existing systems and synthetic-only training, finding weak direct VLM performance, strong but costly tool-assisted performance, and improved results from the authors’ trained model.

1 Introduction

  • Braille offers an accessibility-relevant test because strong performance on conventional text benchmarks does not establish capability across writing systems.The paper notes Braille’s role in literacy, education, and information access, as well as its potential use by sighted non-readers.Source: §1 · p. 1
  • Braille recognition requires fine-grained dot counting and geometric reasoning, while Grade-2 additionally requires resolving context-dependent contractions.The raised-dot cells may appear on low-contrast surfaces affected by illumination, material, perspective, and resolution.Source: §1 · p. 1
  • 259 real-world images—215 Grade-1 and 44 Grade-2—make up BrailleBench, a manually verified benchmark spanning diverse text lengths, materials, and capture conditions.The benchmark evaluates proprietary, open-weight, OCR-specialized, and tool-assisted systems.Source: §1 · pp. 1–2
  • Standalone VLMs perform poorly on Grade-1, while Claude Code reaches 88.37% but takes around three minutes per image and reaches 40.91% on Grade-2.Standalone models can produce plausible but incorrect transcriptions rather than abstaining, which the authors note is especially harmful for assistive use.Source: §1 · p. 2
  • The authors train compact transcription models on synthetic Braille images, avoiding real-world training images while targeting real-image transcription.The introduction describes a synthetic generation pipeline and reports improved real-world results from targeted synthetic supervision.Source: §1 · p. 2
  • 86.98% exact match on Grade-1 and 61.36% on Grade-2 are achieved by the synthetic-only Qwen3.5-2B model, with Grade-2 20.45 points above Claude Code.It approaches Claude Code on Grade-1 with a single model call; standalone VLMs perform worse.Source: §1 · p. 2
  • 96% exact match on synthetic validation did not prevent the prior-data Qwen3.5-2B model from reaching only 3.26% on Grade-1 BrailleBench.The paper reports that synthetic validation accuracy does not predict real-photograph accuracy in these experiments.Source: §1 · p. 2
  • Changing the training corpus substantially improves synthetic-to-real transfer while holding model architecture and adaptation method fixed.The authors identify the remaining synthetic-to-real errors as evidence that transfer is not yet solved.Source: §1 · p. 2

2 Related Work

Prior work spans general multimodal text understanding, OCR with synthetic supervision, and specialized Braille recognition. This paper builds on synthetic Braille adaptation by evaluating transfer from rendered images to photographed Braille.

2 Related Work

  • Modern VLMs and OCR systems perform strongly across visual-question-answering, document, and OCR benchmarks, but fine-grained text perception remains challenging.Synthetic supervision has also supported OCR systems that reduce dependence on manually labeled data.Source: §2 · p. 3
  • Braille recognition research progressed from hand-crafted image processing and feature pipelines to neural methods for dots, characters, and pages.Earlier systems generally used specialized architectures or constrained pipelines.Source: §2 · p. 3
  • Karamolegkou et al. (2025) adapt a multimodal model to Braille with synthetic images and LoRA, evaluating on rendered FLORES-200 and NTREX-128 images.Both training and evaluation in that work use synthetic renders.Source: §2 · p. 3
  • This work extends synthetic Braille adaptation by testing transfer to photographed Braille with BrailleBench.The paper frames this as assessing whether synthetic-only adaptation transfers beyond rendered images.Source: §2 · p. 3

3 Methodology

The methodology constructs a manually reviewed real-image benchmark, generates a diverse synthetic corpus, adapts vision-language and OCR models, and evaluates synthetic-to-real transfer alongside existing systems using transcription metrics.

3.1 BrailleBench

  • BrailleBench is assembled from public Internet sources and books; two annotators inspect, transcribe, and cross-review images before ambiguous, unreliable, and duplicate samples are removed.The final set has 259 images: 215 Grade-1 and 44 Grade-2; Grade-2 contractions require contextual resolution.Source: §3.1 · p. 3

3.2 Synthetic Training Corpus

  • 276k synthetic samples are generated from FineWeb text translated with Liblouis, rendered with varied dot geometry and paper textures, and degraded with capture-condition augmentations.The corpus includes multiple text granularities, about half Grade-2 examples, and scrambled letters in a quarter of texts; the alternative 500k corpus uses longer text windows.Source: §3.2 · p. 4, Figure 2 · p. 5

3.3 Model Adaptation

  • Qwen3.5-2B and Chandra-OCR-2 are fully fine-tuned on the same synthetic image-to-text data and evaluated directly on BrailleBench.No real-world benchmark images are used for training, allowing the study to measure synthetic-to-real transfer.Source: §3.3 · p. 4

3.4 Synthetic-to-Real Comparison

  • The synthetic-to-real comparison trains Llama-3.2-11B with LoRA and Qwen3.5-2B with full fine-tuning on prior WMT-Braille data, then tests on held-out synthetic validation and BrailleBench.WMT-Braille contains 100k rendered and augmented images; inference and scoring are identical across evaluation sets.Source: §3.4 · p. 4

3.5 Evaluation

  • Evaluation compares zero-shot general-purpose VLMs, tool-assisted harnesses, and Braille-specific systems, including an existing OBR tool and synthetic-adapted models.The general-purpose models receive no Braille-specific fine-tuning or in-context examples.Source: §3.5 · p. 4, §3.5.1 · pp. 4–5

3.2 Synthetic Training Corpus

  • The synthetic training pipeline turns text into Braille image–text pairs for fine-tuning Qwen3.5-2B and Chandra-OCR-2 without real-world Braille images.It combines rendered embossed Braille with randomized geometry, spacing, textures, and degradation intended to simulate real capture conditions.Source: Figure 2 · p. 5

3.5.1 Baselines

  • The baseline suite includes proprietary and open-weight VLMs prompted to transcribe Braille, with tool-assisted harnesses evaluated separately from direct inference.The harnesses can iteratively crop, enlarge, and filter images, taking about 180 seconds per image; Braille-specific baselines include the Angelina Braille Reader and a reproduced synthetic adaptation.Source: §3.5.1 · pp. 4–5

3.5.2 Metrics

  • Predictions are scored against manually verified references using character and word error rates, exact match, chrF, normalized match, Word F1, and edit similarity.Exact match requires the whole transcription to be correct; normalized match applies Unicode normalization, lowercasing, and whitespace normalization.Source: §3.5.2 · p. 5

4 Experiments and Results

Across BrailleBench, synthetic-adapted models outperform standalone VLMs, while tool-assisted systems achieve strong results at substantial processing cost. Synthetic validation performance alone does not ensure transfer to real photographs, and results vary with training corpus and text length.

4.1 Main Results

  • GPT-6.1 Sol is the strongest standalone VLM, reaching 53.95% exact match on Grade-1 and 11.36% on Grade-2.All other standalone VLMs except GPT Sol models stay below 3% on Grade-1.Source: §4.1 · p. 5
  • Claude Code reaches 88.37% exact match on Grade-1, but falls to 40.91% on Grade-2; Codex reaches 62.79% on Grade-1 and 0% on Grade-2.Tool-assisted processing averages about three minutes and $0.51–0.58 per image.Source: §4.1 · p. 5
  • Qwen3.5-2B reaches 86.98% exact match on Grade-1 and 61.36% on Grade-2, the best Grade-2 score among systems evaluated.On Grade-1 it is 1.39 points below Claude Code; the comparison covers 215 Grade-1 and 44 Grade-2 images.Source: §4.1 · p. 6, Table 1 · p. 6

4.2 Transfer from Synthetic to Real Braille

  • 96.00% exact match on WMT validation for Qwen3.5-2B trained on WMT-Train contrasts with 3.26% on Grade-1 and 0% on Grade-2 BrailleBench.Its normalized Grade-1 match is 17.67%, so formatting explains little of the gap.Source: §4.2 · p. 7, Figure 3 · p. 7
  • 72–76% exact match across LoRA ranks on the 276k corpus contrasts with one exact match among 259 images for the same Llama model trained on WMT-Train.Changing the corpus improves transfer with architecture and LoRA adaptation fixed; higher ranks mainly curb over-long outputs.Source: §4.2 · p. 7, Table 2 · p. 7
  • 72.97% BrailleBench exact match for the 500k-corpus model is below 82.63% for the shorter-text corpus, despite higher WMT validation accuracy.Continual training on 15k short samples recovers most of the gap to 81.85%; corpus size and image dimensions are not isolated.Source: §4.2 · p. 7

5 Error Analysis

Error patterns differ across systems: the paper’s models usually make partial or formatting-only mistakes, while several standalone and tool-assisted systems often produce large errors, empty outputs, or fluent but incorrect text.

Error categories

  • Our models make large errors or produce no output on at most 4 of 215 Grade-1 and 1 of 44 Grade-2 images; most mistakes are partial or formatting-only.Codex makes large errors on 42 of 44 Grade-2 images, while several standalone VLMs make large errors on 155–211 Grade-1 images.Source: §5 · p. 7, Figure 4 · p. 8

Recurring error patterns

  • Small transcription changes can alter meaning: Qwen changes “like to” to “cannot” in one passage, while Claude Code joins “when Oscar” into “whenOscar.”Nine of 44 predictions in the recomputed Claude Code Grade-2 row differ only in spacing.Source: §5 · p. 8
  • On Grade-1, repeated wrong guesses (excluding correct readings) occurred in 76 Claude Opus 4.8 predictions, 44 GPT-5.6 Sol High predictions, and only 11 GPT-6.1 Sol predictions.Angelina, a dedicated OBR system, had mostly partial (88) or empty (85) errors on Grade-1.Source: §5 · p. 8
  • GPT-6.1 Sol returns blank results for 30 of 215 Grade-1 images and 22 of 44 Grade-2 images, and sometimes omits words from nonblank outputs.For example, it returns only “we” for “Lucy saw it when Oscar opened his book.”Source: §5 · p. 8
  • On multiword targets, normalized match falls from 27.3% to 6.0% for GPT-5.6 Sol High and from 73.7% to 41.4% for GPT-6.1 Sol.Among 20 Grade-2 targets with at least seven words, Qwen reads 10 correctly, Claude Code reads two, and GPT-6.1 Sol reads none; length and grade effects overlap.Source: §5 · p. 8

6 Conclusion

The paper introduces BrailleBench and finds that synthetic-trained models can improve real-image Braille transcription, while synthetic validation accuracy does not guarantee real-world performance. It concludes that accessibility-relevant capabilities should be evaluated explicitly.

6 Conclusion

  • BrailleBench is a real-world benchmark for visual Braille transcription, and synthetic-trained models outperform standalone VLMs on Grade-1 and achieve the best Grade-2 scores.They do so with a single model call per image.Source: §6 · pp. 8–9
  • High synthetic validation accuracy does not guarantee accurate transcription of real photographs in these experiments, while changing the training corpus substantially improves transfer with the architecture fixed.The paper calls for real-image benchmarks to measure progress.Source: §6 · p. 9
  • The work lays a foundation for systems supporting Braille education, communication between Braille and non-Braille readers, and document digitization.It argues that accessibility-relevant capabilities should be evaluated explicitly rather than assumed from general multimodal performance.Source: §6 · p. 9

Limitations

BrailleBench is intentionally limited in scope, and the study identifies both dataset and evaluation constraints. Synthetic adaptation improves performance, but transfer remains incomplete, particularly for context-dependent Grade-2 contractions.

Limitations

  • The benchmark covers English Braille and a focused set of real-world conditions; its 259-image size and collection process may introduce sampling bias.Possible overlap with foundation-model pretraining cannot be ruled out, and inspecting benchmark images informed augmentation choices despite disjoint layout-measurement photographs.Source: Limitations · p. 9
  • Synthetic adaptation improves performance, but errors remain; Grade-2 contractions can depend on linguistic context, word boundaries, and surrounding symbols, especially in short or isolated text.Future work will examine these ambiguities, multilingual Braille across diverse real-world conditions, and context-aware recognition.Source: Limitations · p. 9
  • Grade-2 results come from only 44 images, so a few predictions can substantially change accuracy; aggregate score differences do not establish paired statistical significance.Sources and repeated target strings may also create dependencies between images.Source: Limitations · p. 9

Ethics Statement

The paper frames Braille transcription as potential support for teachers and families, while treating community feedback as qualitative motivation rather than representative evidence. It also describes how BrailleBench images were collected, cross-reviewed, and filtered.

Ethics Statement

  • Informal conversations highlighted difficulties in early Braille learning and interpreting teaching materials, motivating transcription support for teachers and families.The feedback is not representative evidence of national literacy rates or teacher shortages, and the experiments do not establish learning gains or replacement of specialist teaching.Source: Ethics Statement · p. 9
  • Responsible assistive use requires realistic evaluation because incorrect transcriptions can change meaning and mislead learners; uncertain outputs should support verification, not appear authoritative.Cell-level explanations and educational feedback require separate human verification and validation.Source: Ethics Statement · p. 9

A BrailleBench Details

  • BrailleBench contains 259 images—215 Grade-1 and 44 Grade-2—collected from public sources and books, independently transcribed by two annotators, cross-reviewed, and filtered for ambiguous, unreliable, or duplicate samples.The images span embossed paper, printed Braille, signage, and labels photographed under varied illumination and viewing angles.Source: A BrailleBench Details · p. 13

B Synthetic Data Details

The paper builds a synthetic Braille corpus through text translation, image rendering, and perturbation, then trains transcription models and evaluates them alongside tool-assisted systems. Its implementation details span corpus composition, rendering, prompting, scoring, and harness operation.

B.1 Overview

  • The 276,717-image corpus is generated from sampled text translated with Liblouis, rendered as embossed Braille, augmented, then cropped and rescaled to measured real-photo cell pitches.Real-matched runs draw layout dimensions from about 3,000 photograph measurements; add-on runs target rare styles and scrambled text that limits language-context guessing.Source: §B.1 · p. 13

B.2 Braille Translation and Corpus Composition

  • Training uses Unified English Braille tables: 90.3% of sampled Grade-2 texts differ from Grade-1 translations, averaging 19.9% fewer cells; real EBAE contractions may differ from training.An earlier-generator subset is Grade-1 only and lacks real-photo matching of scale or margins.Source: §B.2 · p. 13

B.3 Text Granularity

  • Labels are random text windows rather than linguistically segmented units; the corpus has varied word-count and line-count bins, and no dedicated character-level split.Table 4 reports corpus composition by generation run; its Grade columns are percentages and the earlier-generator portion is Grade-1 only.Source: §B.3 · pp. 13–14, Table 4 · p. 14, Table 5 · p. 14

B.4 Rendering and Augmentation

  • Braille cells are rendered as individually drawn, jittered dots blurred into a height map and lit with random directional light; Table 6 lists augmentation ranges and probabilities.Camera-optics effects, vertical flips, and 90-degree rotations are implemented but disabled.Source: §B.4 · p. 14, Table 6 · p. 16

C Training Details

  • Qwen3.5-2B and Chandra-OCR-2 are fully fine-tuned for one epoch on synthetic images with an image-to-text objective, using eight H100 GPUs.The primary full-fine-tuning results use the last checkpoint; continual-training analyses report checkpoints at specified sample counts.Source: C Training Details · p. 14

D.1 Prompting and Scoring

  • Standalone models receive the same zero-shot transcription prompt, and normalized match lowercases, applies Unicode NFKC, and collapses whitespace; Figure 5 contrasts cumulative synthetic degradation with real photos.The figure fixes layout by seed while redrawing photometric parameters at each degradation level.Source: §D.1 · p. 14, Figure 5 · p. 15

D.2 Tool-Assisted Harnesses

  • Tool-assisted harnesses inspect enlarged or processed image views before the model transcribes Braille; they average about 180 seconds and cost $0.51–0.58 per image.Claude Code uses image-processing tools rather than a tool that produces Braille characters; its logs show views but not the model’s cell-by-cell reasoning.Source: §D.2 · pp. 15–16

E Error Analysis Counts

The error analysis distinguishes textual outcomes from visual causes and presents examples of fluent but incorrect harness predictions. The accompanying analyses are organized around corpus size and text length, continual-training trajectories, and LoRA rank.

E Error Analysis Counts

  • Claude Code’s recomputed Grade-2 outcomes comprise 18 exact matches, 9 spacing-only errors, 8 partial errors, 8 large errors, and 1 no-output case.The analysis categories are mutually exclusive; outcome counts do not identify the visual cause of an error, and comparisons of error-type splits with other rows are provisional.Source: E Error Analysis Counts · pp. 16–17, Table 11 · p. 18

F Qualitative Examples of Harness Failures

  • Harness errors can be fluent rather than visibly broken: Claude Code often substitutes plausible text, while Codex more often returns unrelated character strings.Such fluent errors can be hard to detect for users who cannot read Braille.Source: F Qualitative Examples of Harness… · pp. 17–18

G Synthetic-to-Real Analyses

G.1 Corpus Size and Text Length

Across these analyses, larger synthetic corpora and synthetic validation scores do not reliably predict performance on real Braille images. Short-text continual training improves BrailleBench results, while LoRA rank affects transcription metrics differently across evaluation sets.

G.1 Corpus Size and Text Length

  • BrailleBench exact match fell from 82.63% to 72.97% as CER rose from 0.0305 to 0.1981; on WMT validation, exact match rose from 41.70% to 63.10% as CER fell from 0.6331 to 0.0345.The 500k model would be selected using WMT validation alone, despite performing worse on real BrailleBench images.Source: §G.1 · pp. 18, 20, Table 13 · p. 18
  • After 15k short-text samples, BrailleBench exact match rose from 72.97% to 81.85% and CER fell from 0.1981 to 0.0640; WMT exact match reached 70.20%.This supports the text-length explanation, although the 500k model still had about twice the 276k model’s BrailleBench CER.Source: Table 13 · p. 18, §G.1 · p. 20

G.2 Continual Training Trajectory

  • Almost all BrailleBench improvement occurred within the first 15k continual-training samples; exact match peaked at 84.56% at 75k, while CER was lowest at 0.0606 at 45k.WER was lowest at 15k and 30k (0.1042); at 105k, exact match was 82.24% and CER was 0.0661.Source: §G.2 · p. 20
  • WMT exact match peaked at 70.20% after 15k samples and returned to 63.10% at 105k; synthetic validation gains did not track real-Braille gains.The authors caution against selecting checkpoints using synthetic validation scores alone.Source: §G.2 · p. 20

G.3 Effect of LoRA Rank

  • At LoRA rank 32, BrailleBench exact match was highest at 76.06%, and WMT exact match was highest at 81.60%; lower evaluation loss did not always mean better transcription.Rank 16 had lower loss than rank 8 but worse WMT exact match, chrF, and WER; rank 8’s 2,057 output words versus 1,094 reference words explain its high error rates despite similar exact match.Source: §G.3 · p. 20, Table 14 · p. 20

H.1 Children’s Braille Book

On a separate 142-image Grade-1 children’s book, the evaluated models substantially outperformed the Angelina Braille Reader. The 276k Qwen model also outperformed the 500k Qwen model, while Llama trained on prior WMT-Train data produced no correct transcriptions.

H.1 Children’s Braille Book

  • Chandra-OCR-2 achieved 94.37% exact match on the children’s book, compared with 0% for the Angelina Braille Reader.The Angelina reader frequently introduced substitutions, omissions, and uncertain characters.Source: §H.1 · p. 20, Table 15 · p. 22, Table 16 · p. 22
  • The 142 images come from one Grade-1 children’s book and were evaluated separately because including one source could let a single book dominate BrailleBench.Source: §H.1 · p. 20, Table 15 · p. 22
  • The 276k Qwen model outperformed the 500k Qwen model, while Llama trained on prior WMT-Train data produced no correct transcriptions.The authors note this Llama result is consistent with its BrailleBench results.Source: §H.1 · p. 21
Loading 2610.36195v1…