Source-linked AI summary
Sequence-to-Sequence Contrastive Learning for Text Recognition
Aviad Aberdam, Ron Litman, Shahar Tsiper, Oron Anschel, Ron Slossberg, Shai Mazor, R. Manmatha, Pietro Perona
TL;DR
Text recognition has made limited use of unsupervised and semi-supervised learning, and standard whole-image contrastive learning does not fit its sequence structure. SeqCLR contrasts instances from feature-map sequences, with sequence-specific augmentation and projection designs. It outperforms non-sequential contrastive methods, improves on supervised training with limited labels, and reaches state-of-the-art handwritten-text results.
Problem
Text recognition has made limited use of unsupervised and semi-supervised learning, while whole-image contrastive learning is inadequate for words represented as character sequences.
Source: §1 · pp. 1–2Method
Results
Takeaways & Limitations
Treating text images as sequences pays off: SeqCLR performs strongly on handwritten and scene text with limited supervision and achieves state-of-the-art handwriting results.
Source: §7 · p. 9
Abstract
from arXiv · showhide
We propose a framework for sequence-to-sequence contrastive learning (SeqCLR) of visual representations, which we apply to text recognition. To account for the sequence-to-sequence structure, each feature map is divided into different instances over which the contrastive loss is computed. This operation enables us to contrast in a sub-word level, where from each image we extract several positive pairs and multiple negative examples. To yield effective visual representations for text recognition, we further suggest novel augmentation heuristics, different encoder architectures and custom projection heads. Experiments on handwritten text and on scene text show that when a text decoder is trained on the learned representations, our method outperforms non-sequential contrastive methods. In addition, when the amount of supervision is reduced, SeqCLR significantly improves performance compared with supervised training, and when fine-tuned with 100% of the labels, our method achieves state-of-the-art results on standard handwritten text recognition benchmarks.
1. Introduction
Text recognition needs sequence-aware representation learning because words are sequences of characters, while standard contrastive methods treat images as atomic inputs. SeqCLR contrasts sequence instances with alignment-preserving augmentations and improves results over non-sequential methods, including in low-label settings.
1. Introduction
- Standard whole-image contrastive learning is inadequate for text recognition because words are sequences of characters rather than atomic images.Most handwritten recognition approaches rely on fully supervised learning and require large amounts of annotated data.Source: §1 · pp. 1–2
- SeqCLR maps each feature map to instances formed from consecutive frames, enabling sub-word contrastive learning with multiple positive pairs and negatives per image.The mapping can yield an arbitrary number of instances according to image width, enlarging the batch's negative examples without a memory bank or architecture modifications.Source: §1 · p. 2
- The framework adds sequence-preserving augmentations and custom projection heads to support visual representations for sequential recognition.Sequence-level alignment matters for positive pairing; horizontal flipping can disrupt it.Source: §1 · p. 2, Figure 2 · p. 2
2. Related Work
Prior contrastive methods learn visual representations by matching augmented views, with later work adding projection heads, momentum, and distillation. Text-recognition methods have used limited unlabeled data, while this work proposes self-supervised representation learning for text recognition.
2. Related Work
- Contrastive learning matches representations of augmented views of the same data and distinguishes representations from different images.Prior advances include projection heads [9], momentum-based contrastive learning [27], and teacher-student distillation [10].Source: §2 · p. 2
- Text-recognition work has explored domain adaptation with unlabeled data, while handwritten models usually rely on full supervision and scene-text models mostly use synthetic data.The cited domain-adaptation approaches use adversarial training to align feature-map distributions.Source: §2 · pp. 2–3
- This work claims the first self-supervised representation-learning approach for text recognition and reports state-of-the-art handwritten-text results.Earlier printed-text work required recognizers to use only local predictions.Source: §2 · p. 3
3. Text Recognition Background
The background describes a general text-recognition architecture with image transformation, visual feature extraction, sequence modeling, and text prediction. Some stages are optional, and the decoder may use CTC or attention.
3. Text Recognition Background
- A Thin Plate Spline transformation optionally normalizes text images and is important for images with diverse shapes.It is a variant of the spatial transformer network.Source: §3 · p. 3
- A CNN extracts visual features, which are reshaped into frames whose receptive fields differ across the image.The resulting sequence length depends on input-image width.Source: §3 · p. 3
- An optional BiLSTM captures context within the visual feature sequence and produces contextual features that are concatenated with the feature map.The concatenation follows [38].Source: §3 · p. 3
- Prediction uses either CTC, which decodes frames separately and removes repeated characters and blanks, or attention, which combines frames before a one-layer LSTM.Source: §3 · p. 3
4. Sequence-to-Sequence Contrastive Learning
SeqCLR adapts contrastive learning to sequential text features by mapping feature frames into instances while preserving sequence-level alignment. The section describes its paired-view pipeline, architecture choices, and instance-mapping trade-offs.
4. Sequence-to-Sequence Contrastive Learning
- SeqCLR creates paired augmented views, extracts sequential representations, maps them into instances, and contrasts corresponding indices as positives against other instances as negatives.The mapped instances are the atomic elements used in the contrastive loss.Source: §4 · pp. 3–4
- Augmentations preserve sequence-level alignment by avoiding flipping, aggressive rotations, and substantial horizontal translations that disrupt it.The procedure considers vertical cropping, blurring, random noise, and perspective transformations.Source: §4 · p. 4, Figure 6 · p. 5
- The base encoder can use either visual features or contextual feature maps as the sequential representations for contrastive learning.The contextual option is described as better capturing information within the sequence.Source: §4 · pp. 4–5
- The optional projection head transforms representations, with a frame-wise MLP for frame-to-instance mapping or a BiLSTM for the other mappings.The proposed heads accommodate varying sequence sizes; the projection head is discarded after pre-training.Source: §4 · pp. 4–5
- For variable-sized feature maps, mappings can average all frames, pool consecutive frames into a fixed number of instances, or treat each frame as an instance.Adaptive average pooling is used to obtain a fixed number of window-to-instance outputs.Source: §4 · p. 5
- All-to-instance averaging improves robustness to sequence misalignment but reduces negatives, while frame-to-instance increases negatives and window-to-instance trades off between them.The BiLSTM projection head and sequence modeling in the base encoder can also handle misalignment.Source: Figure 5 · p. 4, §4 · p. 5
5. Experiments
The experiments evaluate SeqCLR against non-sequential contrastive and supervised methods on handwritten and scene text, using frozen-encoder decoder evaluation and fine-tuning. SeqCLR improves limited-label recognition and reports state-of-the-art handwritten results, with outcomes varying by decoder and setting.
5. Experiments
- Decoder evaluation freezes the unlabeled-pretrained encoder and trains a CTC or attention decoder with labeled data, serving as a proxy for representation-learning efficiency.Experiments compare methods across handwritten and scene-text datasets; scene-text fine-tuning is evaluated only with the full labeled data because its training set is synthetic.Source: §5.1 · p. 6, §5 · p. 6, §5.2 · p. 7
5.1. Decoder evaluation
- Window-to-instance mapping achieves the best decoder-evaluation results, balancing frame alignment against the number of negative examples per batch.Frame-to-instance performs poorly on scene text, while all-to-instance averaging reduces the number of negatives; Table 1 reports accuracy and single-edit-distance accuracy for the frozen-encoder evaluation.Source: Table 1 · p. 6, §5.1 · p. 7
5.2. Fine-tuning
- SeqCLR performs better than supervised training in every tested limited-label scenario and on every handwritten dataset, while non-sequential contrastive pre-training often underperforms the supervised baseline.Window-to-instance performs best with attention, while frame-to-instance performs best with CTC; the authors note this may reflect their shared frame-level operation.Source: §5.2 · p. 7
- +1.9% for CTC and +1.7% for attention are SeqCLR’s average gains over supervised training on handwritten datasets with 100% labeled data.For scene text, the average CTC gain is 2.9%, while attention matches the supervised baseline; the authors suggest synthetic-only representation learning may explain the mixed results.Source: §5.2 · pp. 7–8
- SeqCLR with window-to-instance mapping outperforms current word-level state-of-the-art methods on IAM and RIMES, including methods trained unsupervised on the test set.The comparison excludes methods that attach a language model.Source: Table 3 · p. 7, §5.2 · p. 8
6. Ablation Study
The ablations examine how encoder sequence modeling, projection heads, and augmentation choices affect learned representations. They find that component choices interact and that sequence modeling helps handle cropping-induced misalignment.
6. Ablation Study
- Projection-head choice depends on instance mapping: BiLSTM heads fit all-to-instance and window-to-instance, while per-frame MLP performs better with frame-to-instance.Source: Table 4 · p. 8
- Adding sequence modeling to the base encoder improves representation quality for both CTC and attention decoders; a BiLSTM projection head can be redundant with sequence modeling and an attention decoder.The study evaluates decoder accuracy on RIMES using visual or contextual representation layers.Source: §6 · p. 8
- Augmentation must maintain sequence-level alignment, while strong augmentations can benefit contrastive learning; the ablations study this trade-off.Source: §6 · pp. 8–9
- Horizontal cropping can remove complete characters, but a base encoder with sequence modeling handles the resulting sequence-level misalignment.The authors attribute this robustness to contextual information captured within visual features.Source: Figure 8 · p. 8, §6 · p. 9
7. Discussion and conclusions
SeqCLR treats visual inputs as sequences of frames and applies contrastive learning to sequence elements, alongside sequence-specific augmentations. The authors report gains over non-sequential contrastive methods with limited supervision and state-of-the-art handwritten recognition results.
7. Discussion and conclusions
- SeqCLR divides feature maps into sequences of elements for contrastive loss and uses sequence-specific augmentations rather than whole-image equivalents.Source: §7 · p. 9
- SeqCLR reduces word error rate by 9.5% on IAM and 20.8% on RIMES versus the best methods in the literature, and largely outperforms non-sequential contrastive methods with limited supervision.The paper presents these as standard handwritten benchmark results.Source: §7 · p. 9
- The authors hope the results encourage self- and semi-supervised text recognition research and contrastive learning for other sequence-to-sequence predictions.Source: §7 · p. 9
A. Text Recognition Scheme
- The recognition architecture components discussed include thin-plate-spline transformation, CTC decoding, and attention-based decoding.The passage identifies these as the focus of its architecture appendix.Source: A. Text Recognition Scheme · p. 12
A.1. Transformation
The recognition pipeline normalizes text images with a thin-plate-spline transformation, then decodes sequential features using either CTC or attention. CTC collapses repeated characters and blanks, while attention computes weights over the feature sequence and feeds attended context to a recurrent decoder.
A.1. Transformation
- TPS maps cropped, non-axis-aligned text images to a normalized image of fixed size through fiducial-point detection and smooth spline interpolation.The transformation is used for layouts common in handwritten and scene text.Source: §A.1 · p. 12, Figure 9 · p. 12
A.2. Connectionist Temporal Classification (CTC)
- CTC selects the highest-probability character sequence from sequential features, then removes repeated characters and blanks to form the output.The training procedure follows.Source: §A.2 · p. 12
A.3. Attention decoder
- The attention decoder computes weights over the feature sequence and linearly combines its columns into an attended context vector.The recurrent cell then receives that context and the previous output's one-hot embedding; training uses negative log-likelihood as in [13].Source: §A.3 · p. 12
B. Data Augmentation
The self-supervised pipeline stochastically creates two augmented views of each image using a random subset of operations. The appendix describes the transformations and shows the considered pipelines alongside SimCLR [9] examples.
B. Data Augmentation
- Each image is stochastically augmented twice for self-supervised training, using a default random subset of operations from the selected pipeline.Figure 10 shows procedures considered during development and examples using the SimCLR [9] pipeline.Source: B. Data Augmentation · p. 12, Figure 10 · p. 13
- Linear contrast uses α sampled from [0.5, 1.0], Gaussian blur uses σ from (0.5, 1.0), and sharpening blends the image with a sharpened version using factors sampled from [0.0, 0.5].The linear-contrast transform is 127 + α(v −127).Source: B. Data Augmentation · pp. 12–13
- Cropping resizes a smaller sub-image to the original size, with vertical cropping sampled from [0%, 40%] and horizontal cropping from [0%, 2%].Source: B. Data Augmentation · p. 13
- Perspective transforms perturb four image-corner points using a normal-distance scale of 0.01–0.02, while piecewise affine moves grid points by 2%–3%.Source: B. Data Augmentation · p. 13
C. Datasets
The evaluation uses public handwriting and scene-text datasets, with SynthText used to train the scene-text recognizer. The final three listed scene-text datasets serve only for validation and testing.
C. Datasets
- Handwriting evaluation includes RIMES, with 66,480 segmented French words from 1,300 writers, and IAM, with 74,805 segmented English words from 657 writers.Both datasets have writer-independent training, validation, and test partitions.Source: C. Datasets · p. 13
- CVL is an English handwriting dataset from 310 writers, split into writer-independent training and test sets.Twenty-seven writers contributed seven texts each, and the other 283 contributed five texts each.Source: C. Datasets · p. 13
- SynthText contains 8M synthetically generated cropped scene-text images and is used to train the scene-text recognizer.Source: C. Datasets · p. 13
- Scene-text evaluation also lists IIIT5K, IC03, and IC13; the last three listed datasets are used only for validation and testing.IIIT5K contains 2,000 training and 3,000 testing images; IC03 has 867 cropped images, and IC13 has 848 training and 1,015 testing images.Source: C. Datasets · p. 13
D. Implementation Details
The implementation uses a ResNet feature extractor, BiLSTM sequence modeling, and an optional LSTM attention decoder, with fixed-size inputs and task-specific symbol sets. Self-supervised pretraining and decoder evaluation use separate training schedules and augmentation settings.
D. Implementation Details
- The recognizer uses a 29-layer ResNet, a two-layer BiLSTM with 256 hidden units per layer, and, when needed, an LSTM attention decoder with 256 memory blocks.Source: D. Implementation Details · pp. 13–14
- Images are resized to 32 × 100, with 95 symbol classes for English datasets and added French accents for RIMES.CTC adds a blank token; the attention decoder adds start-of-sequence and end-of-word tokens.Source: D. Implementation Details · p. 14
- The SimCLR re-implementation uses the same augmentations and projection head as [9] and is applied to fixed-width input images.Source: D. Implementation Details · p. 14
- Self-supervised training uses batch size 1,024 for 200K handwritten or 400K scene-text iterations, while frame-to-one mapping uses batch size 256 and 300K or 600K iterations, respectively.Optimization uses AdaDelta; experiments run in PyTorch on four 16GB Tesla V100 GPUs.Source: D. Implementation Details · p. 14
- Decoder evaluation and fine-tuning train for 50K iterations with batch size 256, light cropping, linear contrast, and Gaussian blur.Model selection uses public handwritten validation sets and a scene-text validation union of IC13 and IIIT training data, as in [3].Source: D. Implementation Details · p. 14