Source-linked AI summary
SupGRPO: Enhancing GRPO with Matching-based Online SFT for Text Spotting
Xudong Xie, Yuzhe Li, Jing Shi, Zhifei Zhang, Curtis Wigington, Zhaowen Wang
TL;DR
Specialized text spotters struggle with artistic text, while accurate text spotting requires both recognition and localization. SupGRPO combines GRPO with coordinate-focused matching-based online SFT and introduces ATS for evaluation. It outperforms SFT or GRPO alone, especially on complex scenes such as ATS.
Problem
Specialized text spotting methods often struggle with complex artistic text scenarios.
Source: §1 · p. 2Method
SupGRPO jointly trains with GRPO and matching-based online SFT targeted specifically at coordinate tokens.
Source: §1 · p. 3Results
SupGRPO outperforms SFT or GRPO alone, especially on ATS, and demonstrates superior text detection and end-to-end recognition.
Source: §4.3 · p. 12, §5 · p. 15Takeaways & Limitations
SupGRPO integrates policy optimization and targeted coordinate learning to advance text spotting.
Source: §5 · p. 15Matching-based online SFT may struggle with dense overlapping instances sharing identical text and depends on base-model spatial grounding, risking cold-start with zero spatial awareness.
Source: §4.6 · p. 14
Abstract
from arXiv · showhide
Text spotting requires both accurate text recognition and precise spatial localization. Current specialised spotters excel at predicting tight bounding boxes in natural scenes, but falter on complex or artistic text, whereas multimodal large language models (MLLMs) possess strong recognition capabilities yet remain weak at localisation. To equip the text spotter with general and powerful recognition capabilities and to maximize its localization ability, we explore two MLLM-based fine-tuning methods: Supervised Fine-Tuning (SFT) and reinforcement learning fine-tuning based on Group Relative Policy Optimisation (GRPO). An interesting finding is that SFT is less effective than GRPO at enhancing recognition, while GRPO is less effective than SFT at enhancing detection. To compensate for each other's shortcomings, we introduce a joint training strategy, SupGRPO, which simultaneously optimizes the model using both SFT and GRPO. SupGRPO employs the specially designed reward functions and develops a matching-based online SFT applied solely to coordinate tokens. It both mitigates the reward sparsity problem of GRPO and avoids the instance order dependency problem of SFT. To evaluate particularly challenging cases, we curate ATS, a dataset for artistic text spotting. Experiments demonstrate that SupGRPO improves both text recognition and detection, and attains superior performance. Our code and dataset will be released at https://github.com/Psycho-9/SupGRPO.
1 Introduction
Specialized spotters struggle with complex artistic text, while text spotting requires both localization and recognition. SupGRPO combines GRPO with matching-based online SFT on coordinate tokens, alongside the ATS dataset for evaluation.
1 Introduction
- Specialized text spotting methods often struggle with artistic text’s stylized fonts, unconventional layouts, intricate textures, and complex visual integration.Source: §1 · p. 2
- SFT provides fine-grained positional supervision but imposes arbitrary instance ordering, whereas GRPO uses order-agnostic rewards over prediction sets.Source: §1 · p. 3
- SupGRPO combines GRPO with matching-based online SFT on coordinate tokens to address GRPO’s coordinate-reward sparsity and SFT’s sequence dependency.Source: §1 · pp. 3–4
- Experiments show SupGRPO’s superiority in both text detection and recognition across multiple benchmarks.Source: §1 · p. 3
- ATS contains 6.5k training images and 2.5k testing images to evaluate artistic text spotting scenarios underrepresented in existing datasets.Source: §1 · p. 3
- The work presents an end-to-end SupGRPO strategy and a standardized artistic text spotting dataset, and reports a first academic study focused on artistic text spotting.Source: §1 · pp. 3–4
2 Related Work
Prior text spotters advanced arbitrarily shaped text handling, but remain challenged by artistic text. Scene-text datasets and MLLM OCR capabilities provide useful foundations, while complex-scene localization remains limited.
2.1 Text Spotting Methods
- Arbitrarily shaped text spotters progressed from segmentation and regression approaches to DETR-based and sequence-based methods, but still struggle with artistic text.Source: §2.1 · p. 4
2.2 Text Spotting Datasets
- SynthText150K, Total-Text, SCUT-CTW1500, and ICDAR 2015 cover synthetic or real-world scene text, but these datasets do not represent complex scenes requiring strong visual understanding and reasoning.Source: §2.2 · p. 4
2.3 Multimodal Large Language Models for OCR
- MLLMs including TextMonkey [26], Vary, and Qwen3-VL [1] show OCR and document-understanding capabilities, yet their precise location-coordinate prediction remains limited.Source: §2.3 · p. 5
3 Method
The method combines GRPO’s group-relative policy optimization with task-specific rewards and matching-based online SFT on coordinate tokens. This pairing supplies reward-guided training alongside direct coordinate supervision.
3.1 Preliminary
- GRPO samples multiple outputs per input and optimizes relative advantages without a separate value function, while SFT and GRPO share a policy framework but differ in data and gradient signals.The framework describes SFT as using ground-truth data and a constant gradient coefficient, while GRPO samples from the policy and uses group-relative advantages.Source: §3.1 · pp. 5–6
3.2 Reward Functions for Text Spotting
- SupGRPO combines four task-specific rewards—format compliance, word-level F1, and IoU-based box precision and recall—to guide text-spotting outputs.The word-level F1 treats predictions and ground truth as multisets, so duplicate words affect the score.Source: §3.2 · pp. 6–7
3.3 GRPO with Matching-based Online SFT
- SupGRPO integrates matching-based online coordinate-token SFT into GRPO: sampled boxes match ground truth by identical text and IoU greater than zero, receive cross-entropy supervision, while training minimizes negative GRPO objective plus weighted expected coordinate loss.The default weighting hyperparameter λ is 1e-4.Source: §3.3 · pp. 8–9
4 Experiments
Experiments evaluate SupGRPO on artistic and conventional text-spotting benchmarks, alongside ablations of its training components. SupGRPO improves detection and recognition, with its matching-based online SFT showing both optimization benefits and specific limitations.
4.1 Artistic Text Spotting Dataset
- ATS contains 6,500 training and 2,500 testing images, with word-level quadrilateral boxes and transcriptions curated from TextSeg and WAS.Its examples include overlapping text, irregular character arrangements, and text integrated with design patterns.Source: §4.1 · p. 9
4.2 Implementation Details
- LoRA fine-tuning used Qwen2.5-VL and Qwen3-VL-8B for one epoch on four NVIDIA A100 GPUs, training on approximately 30,000 samples combined from five datasets.GRPO sampled 8 output sequences.Source: §4.2 · p. 10
4.3 Results of Text Spotting
- SupGRPO significantly outperformed SFT or GRPO alone, while Qwen3-VL-8B achieved state-of-the-art detection over previous specialized models.Conventional spotting requires correct localization and transcription; MLLM baselines without reliable word-level boxes are evaluated using recognition-only word F1.Source: §4.3 · pp. 11–12, Table 3 · p. 12, Table 4 · p. 12
4.4 Ablation Study
- ATS training data improved overall generalization, while SFT favored detection and GRPO favored recognition, motivating their joint training.Data augmentation for SFT yielded minor gains but still underperformed SupGRPO.Source: §4.4 · pp. 12–13, Table 5 · p. 13, Table 6 · p. 13
- Matching only location tokens supported gains in both detection and recognition; the proposed matching uses text identity and IoU to pair predicted and ground-truth boxes.Whole-sequence SFT introduced a ground-truth order prior, while content-only supervision improved recognition but weakened detection. IoU-only matching could miss or misassign boxes; text-only matching could confuse repeated text instances.Source: §4.4 · pp. 13–14, Table 8 · p. 13, Table 7 · p. 13
- Separate precision and recall rewards improved overall performance; recognition-only rewards reduced detection, while a single F1 reward impaired convergence and sample efficiency.Source: Table 9 · p. 13, §4.4 · p. 14
4.5 Further Analysis
- SupGRPO maintained higher reward curves and converged to lower, less fluctuating loss than vanilla GRPO.The paper attributes the more stable and efficient optimization to matching-based online SFT.Source: §4.5 · p. 14, Figure 4 · p. 15
4.6 Limitation
- Matching may struggle with dense overlapping instances that share identical text, and depends on the base model already having spatial grounding.A model with no spatial awareness could have too few valid matches to optimize coordinates, creating a cold-start problem.Source: §4.6 · p. 14
5 Conclusion
The paper introduces SupGRPO, combining GRPO with matching-based online SFT on coordinate tokens, and contributes ATS for evaluating artistic text spotting. Experiments report superior performance in text detection and end-to-end recognition.
5 Conclusion
- SupGRPO combines GRPO with matching-based online SFT specifically targeting coordinate tokens to address GRPO’s limited coordinate supervision and standard SFT’s sequence issues.The strategy integrates policy optimization with targeted coordinate learning.Source: §5 · p. 15
- The paper contributes ATS, a dataset for evaluating performance on artistic text.Source: §5 · p. 15
- Experiments demonstrate SupGRPO’s superior performance in both text detection and end-to-end recognition.Source: §5 · p. 15
- The paper addresses text spotting in complex artistic images, where existing MLLMs often lack precise localization.Source: §5 · p. 15