PaperlayerView exact revision on arXiv

Source-linked AI summary

Can a Robot Read Braille? - Learning to Adapt Contact via Imitation Learning for Tactile Braille Recognition

Xi Chen, Yunlong Shan, Sihan Chen, Jun Hu, Zhongxuan Li, Shiyao Zhang, Sichao Liu, Zhong Zhao, Kosta Jovanovic, Peng Zhou

arXiv:2609.30676v1cs.RO

TL;DR

Robotic Braille readers have largely focused on recognition after contact, although readable tactile patterns depend on how contact is established. The paper learns to assess contact quality and correct unsuitable poses from expert demonstrations, then uses retained observations for reconstruction. On ten online-evaluation plates, it reports 94.0% tactile quality and 88.6% tactile reconstruction.

  • Problem

    Robotic Braille readers primarily focus on recognition after acquisition, leaving explicit establishment of readable contact insufficiently addressed.

    Source: §I · p. 1

  • Method

    The framework learns contact acceptability and pose corrections from expert-guided demonstrations, iteratively corrects contact, and fuses retained tactile observations with their poses for reconstruction.

    Source: §I · pp. 1–2

  • Results

    94.0% tactile quality and 88.6% tactile reconstruction are reported on the ten online-evaluation plates.

    Source: §I · p. 2

  • Takeaways & Limitations

    The results highlight that reliable tactile reading depends on actively establishing contact that makes the tactile information readable, not only recognizing observations.

    Source: §VI · p. 7

    The evaluation is limited to rigid, approximately planar PLA Braille plates, a fixed tactile-sensor mounting, expert-guided demonstrations, and at most three attempts per location.

    Source: C. Discussion · p. 7

Abstract

from arXiv · show

For people who are blind, touch provides an essen-tial channel for accessing written information through Braille. Bringing a similar capability to robots requires them not only to recognize tactile patterns, but also to actively establish physical contact that makes those patterns readable. Yet existing robotic Braille readers largely focus on recognition after contact, leaving contact establishment itself insufficiently addressed. We present an adaptive-contact framework for robotic tactile Braille reading that assesses contact quality and physically corrects unsuitable contact before recognition and reconstruc-tion. Multi-Head Policy Learning uses expert-guided contact-adjustment demonstrations to jointly learn contact acceptability and pose corrections. During deployment, the robot iteratively evaluates and re-establishes contact, retaining reliable tactile observations for pose-aware fusion and Braille reconstruction. Across 20 physical Braille plates used for learning and eval-uation, the proposed approach achieves 94.0% tactile quality and 88.6% tactile reconstruction on the ten online-evaluation plates. These results demonstrate the importance of actively establishing readable contact, rather than relying solely on recognition under imperfect tactile observations, for reliable robotic Braille reading.

I. INTRODUCTION

The paper frames readable contact as a necessary part of robotic Braille reading, not merely a condition for later recognition. It proposes learning to assess and correct contact before reconstruction, and reports results on physical Braille plates.

I. INTRODUCTION

  • Reliable robotic Braille reading requires establishing physical contact that makes the tactile patterns readable.Insufficient or excessive indentation can suppress or merge dots, while sensor tilt can distort their geometry.Source: §I · p. 1
  • Prior systems primarily recognize tactile observations after acquisition, while this work asks whether robots can establish readable contact before recognition.The introduction notes that contact errors can corrupt Braille-dot geometry before recognition.Source: §I · p. 1
  • The proposed framework uses expert-guided demonstrations to jointly learn contact acceptability and pose corrections, then retains selected observations with acquisition poses for reconstruction.It connects contact assessment and physical correction with pose-aware overlap fusion and dot refinement.Source: §I · pp. 1–2
  • 94.0% tactile quality and 88.6% tactile reconstruction are achieved by the proposed full correction on physical Chinese and English Braille plates.The introduction reports validation of the complete pipeline on physical plates.Source: §I · p. 2

II. RELATED WORK

Prior Braille readers and tactile-control or mapping methods provide relevant sensing, interaction, and reconstruction foundations. The related-work section distinguishes this approach by assessing Braille-dot readability and correcting contact before retaining observations for reconstruction.

A. Robotic Tactile Braille Reading and Active Contact

  • Existing Braille readers use varied tactile sensors and pressing, tapping, or sliding, but primarily prescribe contact or treat its variation as a recognition condition rather than correcting readability before reconstruction.The cited approaches include neurorobotic, skin-like, visuo-tactile, magnetic, neuromorphic,, and dual-mode sensing.Source: A. Robotic Tactile Braille Reading and… · p. 2

B. Tactile Servoing and Reconstruction

  • Tactile-control work regulates contact and tactile mapping fuses observations after acquisition, whereas this method links Braille-specific contact assessment and pose correction to reconstruction.Prior work includes contact position, force, and edge orientation control, image-space dynamics, and pose and shear feedback,.Source: B. Tactile Servoing and Reconstruction · p. 2

III. PROBLEM FORMULATION

The formulation treats Braille reading as sequential contact at ordered locations along a surface. At each location, the robot uses a tactile observation to accept contact or correct the pose and try again.

III. PROBLEM FORMULATION

  • The robot reads a Braille character sequence by moving a tactile sensor along the surface and acquiring images through contact.It follows a predefined trajectory over ordered lateral locations with a fixed step.Source: §III · p. 2
  • At each location k and attempt t, the commanded end-effector pose is u_k,t, with position p_k,t and orientation θ_k,t, and the resulting tactile image is I_k,t.Source: §III · p. 2
  • Repeated contact attempts occur when dot geometry is unsuitable, including when dots are missing, incomplete, merged, or deformed.Source: §III · p. 2
  • A partially observable Markov decision process represents the latent contact configuration, including relative pose and local deformation that determine Braille-dot geometry.Source: §III · p. 2
  • The tactile image is the observation; the action either accepts contact or applies a pose correction before the next attempt.The transition and observation models describe changes in contact state and how that state relates to the acquired image.Source: §III · pp. 2–3
  • The objective is to obtain readable tactile observations while minimizing unnecessary re-contact attempts.Source: §III · p. 3
  • Rather than solve the full POMDP or explicitly estimate a belief state, the approach learns an observation-conditioned reactive policy from expert demonstrations.Source: §III · p. 3
  • A quality score estimates whether an image preserves usable dot geometry; above threshold it is retained, otherwise the predicted correction updates the next contact pose.Source: §III · p. 3

A. Method Overview

The method combines expert-guided policy learning, closed-loop contact adaptation, and tactile reconstruction. Expert demonstrations label contact quality and corrections using sequences of attempts that share an accepted target.

A. Method Overview

  • The pipeline collects expert demonstrations, deploys a closed-loop policy to accept or correct contact, then fuses retained image-pose pairs for Braille decoding.The global tactile image is refined before being organized into Braille cells; each cell has six positions in two columns of three.Source: A. Method Overview · p. 3

1) Expert Demonstration and Target Construction:

  • At each selected Braille location, an expert rejects missing, incomplete, merged, or deformed dots and adjusts the pose until the contact is acceptable.The accepted image and pose provide the target for the demonstration group.Source: 1) Expert Demonstration and Target… · p. 3
  • Rejected attempts receive binary quality targets and correction targets defined as the residual from their current contact parameters to the final accepted ones.The accepted contact has a zero correction target; grouped attempts can share the same accepted target, producing distinct correction samples.Source: 1) Expert Demonstration and Target… · p. 3
  • Each recorded attempt becomes a supervised training pair for learning from the expert-guided contact-adjustment sequence.Source: 1) Expert Demonstration and Target… · p. 3

2) Contact-Quality and Pose-Correction Policy:

The policy jointly learns to assess contact quality and predict pose corrections from tactile images. Its training objective applies correction regression only to rejected contacts.

2) Contact-Quality and Pose-Correction Policy:

  • A shared ImageNet-pretrained ResNet-18 encoder feeds heads that predict contact quality and pose corrections from each tactile image.The single-channel tactile input is converted to a 700×400 8-bit image; correction components are normalized using rejected training samples.Source: 2) Contact-Quality and Pose-Correction… · p. 3
  • Pose-correction loss is computed only for rejected contacts, because accepted contacts require no further correction.The quality and correction heads are trained jointly, with λq and λu balancing classification and regression.Source: 2) Contact-Quality and Pose-Correction… · p. 3
  • Mini-batches are sampled by demonstration group with an approximately 1:1 accepted-to-rejected ratio, and loss weights and model selection use validation data.Source: 2) Contact-Quality and Pose-Correction… · p. 3

C. Closed-Loop Contact Adaptation

The robot repeatedly evaluates contact and corrects unsuitable poses, then uses retained tactile-image–pose pairs for pose-aware Braille reconstruction. The reconstruction projects local observations into a shared plate frame and fuses overlapping regions.

C. Closed-Loop Contact Adaptation

  • At each trajectory location, contact is accepted when its predicted quality reaches 0.80; otherwise, the robot retracts and retries, up to three attempts.The updated pose is bounded to the tested safety range; if no attempt qualifies, the highest-quality observation is retained and the location is marked unsuccessful.Source: C. Closed-Loop Contact Adaptation · pp. 3–4

D. Pose-Aware Braille Reconstruction

  • Retained tactile images are registered in a common plate frame by projecting their sensing-region corners according to measured sensor poses.Each image is binarized using a validation-selected threshold of 156; the sensing region is 24.5 × 15.0 mm.Source: D. Pose-Aware Braille Reconstruction · p. 4
  • Overlapping valid tile regions are fused by retaining the maximum warped response after erosion, Gaussian smoothing, and thresholding.Each tile region is eroded by two pixels, smoothed with a one-pixel standard deviation, and thresholded at 0.02.Source: D. Pose-Aware Braille Reconstruction · p. 4

1) Dot Refinement and Braille Decoding:

After fusion, refinement removes spurious dot responses, regularizes incomplete dots, and separates merged dots. The detected centers are organized into Braille cells before a language-specific decoder interprets them.

1) Dot Refinement and Braille Decoding:

  • Refinement removes spurious components, regularizes incomplete dots to circular shapes, and separates merged dots.Its thresholds are selected on validation data and fixed before testing.Source: 1) Dot Refinement and Braille Decoding · p. 4
  • OpenCV organizes detected dot centers into ordered lines and cells, encoding six possible positions per cell as binary occupancy states.Positions 1–3 run top to bottom in the left column and positions 4–6 in the right.Source: 1) Dot Refinement and Braille Decoding · p. 4
  • At each trajectory location, the policy retains an accepted tactile image or, if none is accepted, the attempt with the highest predicted quality.Accepted contact parameters are inherited at the next location.Source: 1) Dot Refinement and Braille Decoding · p. 5
  • A language-specific large language model receives the refined dot image, structured cell sequence, and fixed prompt to decode each Braille line once.Tactile quality and reconstruction are computed independently of the decoder.Source: 1) Dot Refinement and Braille Decoding · pp. 4–5

V. EXPERIMENTS AND RESULTS

The experiments examine whether the learned policy can assess contact quality and predict corrective motion on unseen plates, and whether closed-loop adaptation improves tactile acquisition and downstream Braille reconstruction and reading.

V. EXPERIMENTS AND RESULTS

  • One experimental question is whether the learned policy assesses contact quality on unseen Braille plates.Source: §V · p. 5
  • The experiments also ask whether the policy predicts corrective motion on unseen plates.Source: §V · p. 5
  • A second question is whether closed-loop contact adaptation improves tactile acquisition and downstream Braille reconstruction and reading.Source: §V · p. 5

A. Experimental Setup

The study uses physical Chinese and English Braille plates, with plates divided between offline demonstrations and online evaluation. Held-out-plate evaluation reports contact-quality classification and pose-correction performance.

1) Platform and Dataset:

  • The platform uses an xArm7 with an Xense G1-WS optical tactile sensor, and 20 black plates split evenly between Chinese and English.Ten plates are used for offline demonstration collection and ten are reserved for online evaluation.Source: 1) Platform and Dataset · p. 5
  • Plate-level dataset splits yield 480 training groups, 240 validation groups, and 320 held-out test groups from 1,040 groups overall.The groups contain 2,426 tactile images: 1,040 accepted and 1,386 rejected contacts.Source: 1) Platform and Dataset · p. 5

2) Policy Training and Offline Evaluation:

  • The acceptance threshold is ˆqi ≥ τq = 0.80, and representative scans show rejected contacts followed by accepted contacts after pose correction.The illustrated attempts are at trajectory locations k = 1 and k = 5.Source: 2) Policy Training and Offline… · p. 6, Figure 7 · p. 6
  • 92.8% contact-quality F1 and 5.9% classification error were achieved; pose correction had 0.20 mm position MAE, 0.27° and 0.29° roll and pitch MAEs, and mean R2 of 0.76.Offline results are computed on held-out test plates.Source: 2) Policy Training and Offline… · p. 6

3) Online Evaluation Protocol:

The online evaluation compares contact-adaptation strategies under a shared scanning and reconstruction protocol. Full correction improves tactile quality, reconstruction, and exact-line accuracy over Position-only, while retrying without pose correction offers little improvement.

3) Online Evaluation Protocol:

  • Each strategy scans all four lines of ten plates, totaling 40 scans and 448 trajectory locations; observations with q̂ ≥ 0.80 are retained, with up to three attempts.If no attempt is accepted, the highest-scoring observation is retained; all strategies use the same downstream reconstruction and decoding pipeline.Source: 3) Online Evaluation Protocol · p. 6
  • Tactile quality measures retained images without specified dot defects, while tactile reconstruction measures cells whose six dot positions match the plate.Braille-line accuracy is the percentage of decoded lines exactly matching the ground-truth transcription; tactile quality and reconstruction are independent of the language decoder.Source: 3) Online Evaluation Protocol · p. 6

B. Online Braille Reading Results

  • Full rejects unsuitable observations and corrects contact until readable contact is established at representative scan locations k = 1 and k = 5.Figure 7 illustrates this sequence.Source: B. Online Braille Reading Results · p. 6
  • Retry-only offers little improvement over Single-press despite requiring 2.36 versus 1.00 attempts/location.The comparison indicates that repeated acquisition without physical correction rarely resolves unsuitable contact.Source: B. Online Braille Reading Results · p. 6
  • 94.0% tactile quality, 88.6% pooled tactile reconstruction, and 80.0% exact line accuracy are achieved by Full, versus 81.9%, 80.0%, and 60.0% for Position-only.Full uses 1.42 attempts/location; its tactile-quality gain over Position-only is 12.1 points.Source: Table II · p. 6, B. Online Braille Reading Results · p. 7

C. Discussion

The discussion argues for a contact-first approach: unsuitable contact must be physically corrected, and both translational and rotational errors contribute to corrupted Braille-dot geometry. The evaluation is limited to rigid, approximately planar PLA plates and a fixed sensor mounting.

C. Discussion

  • Retry-only indicates that detecting unsuitable contact is insufficient without changing the physical interaction, while Position-only versus Full implicates both translational and rotational errors in corrupted dot geometry.Full also achieves higher tactile quality with fewer attempts than correction baselines, and downstream gains show contact adaptation protects tactile evidence used for reconstruction.Source: C. Discussion · p. 7
  • Evaluation is limited to rigid, approximately planar PLA plates and a fixed tactile-sensor mounting.Future work will investigate broader materials and geometries, coupled disturbances, other sensor configurations, and general tactile-servoing comparisons.Source: C. Discussion · p. 7

VI. CONCLUSION

  • The framework learns contact-quality assessment and pose correction from grouped expert demonstrations, then re-establishes unsuitable contact before tactile reconstruction.The conclusion identifies more diverse surfaces, coupled contact disturbances, and less structured exploration as future directions.Source: §VI · p. 7
Loading 2609.30676v1…