Source-linked AI summary
MEVL-STP: Multi-Encoder and Vision Language Model for Arbitrarily Shaped Scene Text Spotting
Aman Anand, Partha Pratim Roy, Shivakumara Palaiahnakote
TL;DR
Curved and dense multi-oriented scene text remains difficult to detect and recognize, and tightly coupled architectures can propagate localization errors into recognition. MEVL-STP uses multi-encoder detection with vision-language recognition on polygon crops. It reports 85.86% H-mean on CTW1500 as a new state of the art.
Problem
Curved and dense multi-oriented text remains challenging, while shared-backbone spotting tightly couples localization and recognition.
Source: §1 · p. 2Method
MEVL-STP decouples detection and recognition, fusing six frozen encoders for polygon detection and applying LoRA-fine-tuned vision-language recognition to polygon crops.
Source: §1 · p. 2Results
85.86% H-mean on CTW1500 is reported as a new state of the art, alongside 91.99% detection F1 without synthetic data.
Source: §1 · pp. 2–3Takeaways & Limitations
The authors conclude that semantic, spatial, and texture feature spectra are codependent and that no single type suffices.
Source: §5 · p. 14Recognition runs sequentially per crop, so latency grows with the number of instances; distillation and batched inference could reduce it.
Source: §5 · pp. 14–15
Abstract
from arXiv · showhide
Scene text spotting remains challenging for arbitrarily shaped text instances such as curved signs and dense multi-oriented characters in natural images, where tightly coupled architectures propagate localization errors directly into recognition failures. We present a two-stage pipeline that combines multi-encoder segmentation with vision-language model recognition to address this problem. In the detection stage, six frozen vision encoders (CLIP, DINOv2, SigLIP, EVA-CLIP, SAM, and ConvNeXt) extract complementary features spanning semantic, spatial, and texture spectra, which are fused through a trainable hierarchical Feature Pyramid Network with channel attention and decoded via a deep-supervision Progressive Scale Expansion network to generate precise instance-level text masks. By keeping the encoders frozen, their independently learned feature spaces remain orthogonal during fusion, preventing the feature homogenization that degrades boundary precision in single-backbone detectors. The detection stage produces tight polygon masks that conform to the actual shape of curved and arbitrarily oriented text, rather than axis-aligned rectangles that inevitably include background content. In the recognition stage, these polygon-masked crops isolate the target text from surrounding clutter, allowing a Qwen3-VL-8B-Instruct model, fine-tuned via Low-Rank Adaptation on polygon-cropped scene text, to focus purely on reading the text without interference from neighbouring words or background noise. Without any synthetic pretraining data, our method achieves 91.99% detection F-measure and 85.86% end-to-end H-mean on CTW1500, setting a new state of the art and achieving strong performance on Total-Text and ICDAR 2015 without any synthetic training data. Code is available at https://github.com/doubleblind-afk/MEVL-STP
1 Introduction
Curved and dense multi-oriented text remains difficult to spot, while localization errors can undermine recognition. MEVL-STP separates detection from recognition and combines multi-encoder polygon detection with vision-language recognition, reporting state-of-the-art end-to-end results on CTW1500.
1 Introduction
- Curved text and dense, multi-oriented instances remain challenging, and localization errors can directly degrade recognition.The introduction identifies tight polygon boundaries as important for isolating curved text from adjacent instances.Source: §1 · p. 2
- MEVL-STP separates multi-encoder segmentation-based detection from vision-language recognition using polygon crops and a LoRA-fine-tuned Qwen3-VL model.Its detector fuses six frozen foundation encoders through a trainable hierarchical FPN with channel attention and deep-supervision Progressive Scale Expansion.Source: §1 · p. 2
- 91.99% detection F1 is reported for the six-encoder module without synthetic data, while tight polygon masking isolates arbitrary text shapes for recognition.The recognition module is also reported to achieve 94.23% accuracy.Source: §1 · pp. 2–3
- 85.86% H-mean on CTW1500 is reported as a state-of-the-art end-to-end text spotting result.The authors attribute the result to decoupled multi-encoder detection and vision-language recognition.Source: §1 · p. 3
- In the CTW1500 examples, DeepSolo [3] truncates or fragments curved text and misreads characters, while MEVL-STP's polygons follow the full baseline and its VLM reads correctly.The paired images compare DeepSolo on the left with MEVL-STP on the right.Source: Figure 1 · p. 3
2 Related Work
Related work covers both jointly optimized spotting systems and two-stage detection-recognition pipelines. MEVL-STP combines decoupling with multi-encoder detection and a LoRA-adapted vision-language recognizer.
2 Related Work
- Scene text spotting methods are broadly divided into single-stage end-to-end and two-stage detection-recognition approaches.The single-stage approaches integrate detection and recognition within a shared backbone.Source: §2 · p. 3
2.1 Single-Stage End-to-End Methods
- Single-stage methods share detection and recognition representations, leaving them tightly coupled even as transformer methods add flexible detection mechanisms.Examples include DNTextSpotter [7], SwinTextSpotter v2, and InstructOCR; specialized curved-text approaches include IATS and CRENet.Source: §2.1 · pp. 3–4
2.2 Two-Stage Detection and Recognition
- Two-stage methods enable independent optimization, but prior systems still rely on a single detection backbone that may lack diverse semantic and structural cues.MEVL-STP instead fuses six frozen encoders and uses parameter-efficient LoRA fine-tuning for vision-language recognition.Source: §2.2 · p. 4
3 Methodology
The methodology separates instance-level detection from crop-based recognition, combining frozen multi-encoder features and PSE segmentation with a LoRA-adapted vision-language model.
3.1 Multi-Encoder Feature Extraction
- Six frozen encoders span semantic, spatial, and texture features, while handcrafted local channels add boundary cues for polygon refinement.The semantic encoders capture what content represents, spatial encoders locate text boundaries, and ConvNeXt preserves local stroke patterns.Source: §3.1 · p. 5, Local Feature Extraction · p. 5
3.2 Multi-Scale Feature Pyramid Network
- The multi-scale FPN projects encoder outputs to a common dimension and fuses them across three pyramid scales, with deformable attention sampling task-relevant locations.Each encoder output is projected to 512 channels; the attention configuration uses three levels and eight sampling points per level.Source: §3.2 · pp. 6–7
3.3 Progressive Scale Expansion (PSE)
- PSE predicts six progressively larger kernel maps, growing instances outward from text-center seeds while preserving boundaries between adjacent instances.The smallest kernel seeds connected components; subsequent kernels expand them into polygons.Source: §3.3 · p. 7
3.4 Decoupled Feature Interaction
- The paper describes PSE as balancing semantic expansion cues against spatial and local-feature stopping cues anchored to physical text edges.This energy formulation is descriptive, not a trained objective or additional loss.Source: §3.4 · p. 8
3.5 Vision-Language Model for Text Recognition
- A Qwen3-VL-8B-Instruct recognizer uses LoRA fine-tuning to transcribe text from polygon-derived crops.LoRA adapts about 0.2% of parameters, and training applies selective cross-entropy to assistant response tokens only.Source: §3.5 · p. 9
3.6 End-to-End Training
- Detection and recognition are trained independently before a three-phase schedule gradually introduces sparse VLM supervision to stabilize learning.Detection-only training runs for epochs 1–2, followed by progressively more frequent VLM supervision through epoch 30.Source: §3.6 · p. 9
4 Experiments
Experiments evaluate detection and end-to-end spotting on CTW1500, Total-Text, and ICDAR 2015, then examine design choices, computational cost, and qualitative results. The full pipeline sets a CTW1500 state of the art while using no synthetic data.
4.1 Datasets and Evaluation Metrics
- The evaluation covers curved line-level text on CTW1500, word-level quadrilaterals on ICDAR 2015, and horizontal, multi-oriented, and curved words on Total-Text.Detection uses Precision, Recall, and F1 at IoU ≥0.5; end-to-end H-mean is reported under None, Full, and Weak lexicon settings.Source: §4.1 · p. 10
4.2 Text Detection Results
- 92.0%, 89.6%, and 90.8% F1 on CTW1500, Total-Text, and ICDAR 2015, respectively, accompany the highest recall on all three datasets.Recall is 93.8%, 96.4%, and 93.5%, respectively; Total-Text precision is 83.9%, reflecting the recall-favouring threshold choice.Source: §4.2 · p. 10, Table 2 · p. 11
4.3 End-to-End Text Spotting Results
- 85.9 Full H-mean on CTW1500 surpasses LRANet++ (85.2) by 0.7 points, while 75.2 None sets a state of the art by 4.5 points without synthetic data.The pipeline also reports 81.9/85.6 None/Full on Total-Text and 81.7 Weak on ICDAR 2015; a gap to synthetic-data methods remains on those benchmarks.Source: §4.3 · p. 11, Table 3 · p. 12
4.4 Ablation Study
- 56.85% H-mean under joint training contrasts with 85.86% after phased training on the same weights; polygon crops reach 82.25% recognition accuracy.The passage attributes the joint-training result to VLM gradients destabilizing detection before polygon quality converges; loose boxes reach 73.65% and full-image VLM input 77.56% accuracy.Source: Table 4 · p. 12, §4.4 · p. 13
- 91.92% F1 with all three feature spectra follows 90.32–90.63% with two, while removing SAM or DINOv2 reduces F1 by 7.92 or 4.05, respectively.Semantic encoders alone reach 0.32% F1 and spatial encoders alone 72.88%; pairwise CKA averages 0.002, consistent with complementary encoder features.Source: §4.4 · p. 13, Table 5 · p. 13
- The six frozen encoders add 2.2B parameters at inference, and SAM-ViT-H accounts for 76% of detector FLOPs at 1024² input.Replacing SAM-ViT-H with SAM-ViT-L makes that branch about four times cheaper, trading accuracy for speed.Source: §4.4 · p. 14
4.5 Qualitative Results
- Qualitative results show recognition on curved CTW1500 text, dense multi-oriented ICDAR 2015 layouts, and mixed horizontal and curved Total-Text instances.Figure 7 presents results across all three benchmarks.Source: Figure 7 · p. 14, §4.5 · p. 14
5 Conclusion
The conclusion presents MEVL-STP as a decoupled framework combining six frozen encoders for detection with LoRA-adapted VLM recognition on polygon crops. It reports CTW1500 detection and end-to-end results without synthetic data, while noting that sequential recognition adds instance-dependent latency.
5 Conclusion
- Together, the three feature spectra reach 91.99% detection F1 and 85.86% Full H-mean on CTW1500 without synthetic data, with competitive results on Total-Text and ICDAR 2015.The conclusion describes the 85.86% Full H-mean as a new state of the art.Source: §5 · p. 14
- MEVL-STP fuses six frozen foundation encoders for detection and applies LoRA-adapted VLM recognition to tight polygon crops.The conclusion characterizes the framework as decoupled.Source: §5 · p. 14
- Recognition runs sequentially per crop, so latency grows with the number of instances; distillation and batched inference could reduce it.Source: §5 · p. 15