Source-linked AI summary
CurveBench: A Benchmark for Exact Topological Reasoning over Nested Jordan Curves
Amirreza Mohseni, Mona Mohammadi, Morteza Saghafian, Naser Talebizadeh Sardari
TL;DR
CurveBench addresses the difficulty of exact topological reasoning from images by benchmarking recovery of rooted containment trees induced by disjoint Jordan curves. It introduces a controlled, verifiable structured-prediction task and uses RLVR-style fine-tuning to improve open-weight vision-language models. The results show substantial gains from verifiable-reward training, while exact hierarchical reconstruction remains challenging, especially on structurally complex inputs.
Problem
Current vision-language models struggle to extract and reason about subtle topological image relations such as disjointness, nesting, and separation.
Method
CurveBench defines a deterministic structured-prediction task that maps images of pairwise disjoint Jordan curves to exact rooted containment trees and evaluates them with verifiable tree metrics.
Results
Reinforcement learning from verifiable rewards substantially improves an open-weight model, demonstrating CurveBench’s utility while current models remain far from exact containment-tree recovery.
Takeaways & Limitations
Exact hierarchical reconstruction remains challenging for current vision-language models, while verifiable-reward training offers a promising route for strengthening visual reasoning.
Takeaways & Limitations
CurveBench covers only nested, pairwise non-intersecting Jordan curves and excludes intersecting or open curves, noisy real-world segmentations, three-dimensional topology, temporal structure, and ambiguous natural images.
Abstract
from arXiv · showhide
We introduce CurveBench, a benchmark for hierarchical topological reasoning from visual input. CurveBench consists of \textbf{756 images} of pairwise non-intersecting Jordan curves across easy, polygonal, topographic-inspired, maze-like, and dense counting configurations. Each image is annotated with a rooted tree encoding the containment relations between planar regions. We formulate the task as structured prediction: given an image, a model must recover the full rooted containment tree induced by the curves. Despite the visual simplicity of the task, the strongest evaluated model, Gemini 3.1 Pro, achieves only \textbf{71.1\%} tree-generation accuracy on CurveBench-Easy and \textbf{19.1\%} on CurveBench-Hard. We further demonstrate benchmark utility through RLVR-style fine-tuning of open-weight vision-language models. Our trained Qwen3-VL-8B model improves over \texttt{Qwen-3-VL-8B-Thinking} from \textbf{2.8\%} to \textbf{33.3\%} tree-generation accuracy on CurveBench-Easy, exceeding GPT-5.4 and Claude Opus 4.5 under our evaluation protocol. The remaining gap, especially on CurveBench-Hard, shows that exact topology-aware visual reasoning remains far from solved.
1 Introduction
CurveBench targets exact visual recovery of containment hierarchies from disjoint Jordan curves, addressing persistent difficulty in topological image reasoning. It provides a controlled benchmark and training signal for evaluating and improving structured prediction.
- Images of disjoint curves encode connectivity, separation, nesting, and adjacency information that determines planar structure.
- CurveBench maps images of pairwise disjoint Jordan curves to rooted trees representing region containment and separating boundaries.
- The benchmark defines a deterministic structured prediction task, evaluation protocol, parser, and exact rooted-tree matching metric.
- Current vision-language models remain far from solving exact containment-tree recovery from images.
- RLVR-style fine-tuning demonstrates that CurveBench supplies actionable training signal while exposing persistent generalization gaps.
2 Related work
Prior work learns visual structures, diagram relations, and coarse topological invariants, whereas CurveBench isolates exact recovery of a discrete containment tree. Its exact, verifiable formulation supports structured visual reasoning evaluation and RL-based optimization.
- Structured prediction from images: Earlier vision methods predict region trees, graphs, polygons, maps, or sequences from images using structured representations.
- Positioning: CurveBench requires exact recovery of every parent–child relation in a rooted containment tree, rather than approximate geometric or semantic outputs.
- Diagram understanding and visual reasoning: Unlike query-based diagram benchmarks, CurveBench reconstructs the full global containment hierarchy and enables deterministic exact-structure evaluation.
- Topology-aware vision: Topology-aware vision commonly uses continuous losses for coarse invariants, while CurveBench infers fine-grained discrete nesting relationships.
- Reinforcement learning for structured reasoning: CurveBench extends verifiable-reward fine-tuning to visual reasoning whose rooted-tree outputs can be checked directly against exact task metrics.
- Positioning: The benchmark combines controlled diagram-like inputs, vision-to-structure prediction, and exact verifiable evaluation.
3 Dataset of CurveBench
CurveBench contains 756 hand-drawn images spanning progressively challenging curve configurations, each paired with a formal rooted tree for deterministic structural evaluation. Its categories test small combinatorial coverage, geometric robustness, long-range nesting, and counting capacity.
- 756 hand-drawn images map pairwise disjoint Jordan curves to exact rooted containment trees, isolating global topological structure.
- Easy: CurveBench-Easy contains fewer than six curves and enumerates rooted tree structures with up to six nodes across train, validation, and held-out test splits.
- Polygon: Polygon tests robustness to sharp angles and piecewise-linear boundaries rather than smooth curves.
- Topographical: Topographical images mimic elevation-level-set behavior while extending evaluation toward applied visual understanding.
- Maze and Counting: Maze images stress long-range spatial reasoning through convoluted curves and deep nesting, while Counting images test construction of larger trees from densely nested entities.
- Hard categories: CurveBench-Hard contains 456 Polygon, Topographical, Maze, and Counting images.
- Each image has a formal combinatorial rooted-tree annotation enabling deterministic assessment of exact adjacency and containment recovery.
4 Tree generation task
The tree-generation task converts an image of pairwise disjoint Jordan curves into a rooted tree of planar regions and immediate containment relations. Models output parent–child edges in a fixed format, enabling exact structural evaluation.
- The input is an image of pairwise disjoint Jordan curves varying in shape, scale, complexity, and spatial configuration.
- The output is a rooted tree whose nodes represent planar regions and whose edges represent immediate containment through shared boundary curves.
- The formulation isolates topological structure as the prediction target and evaluates models with tree-based structural metrics.
- Models must produce the rooted tree as parent–child edges inside <answer> tags, beginning with the number of non-root nodes.
- A sample input, corresponding tree, and expected tree representation illustrate the required output mapping.
5 Experimental Setup
The experiments fine-tune open-weight VLMs with RLVR to recover exact containment trees, using deterministic rewards and standardized evaluation environments. Comparisons emphasize tree-generation accuracy while controlling for differing training objectives.
- Evaluation protocol: Fine-tuning uses held-out CurveBench-Easy training and validation splits, then evaluates on the Easy test split and full CurveBench-Hard benchmark.Easy measures within-distribution generalization; Hard measures transfer to more challenging curves.
- Training setup: The study fine-tunes Qwen3-VL-8B-Thinking and Gemma3-12B-it with Dr.GRPO and LoRA, updating adapters while freezing base-model weights.Training uses 250 optimization steps, batch size 128, and eight sampled generations per input on eight NVIDIA RTX PRO 6000 GPUs.
- Reward design: RLVR parses each generated answer into a region tree and compares it against ground truth using deterministic tree and node-count rewards.The reward combines exact tree-generation correctness with node-count correctness.
- Reward design: The combined reward assigns 30% weight to exact node count and 70% to exact rooted-tree structure.Its formula is R_comb = 0.3 · R_count + 0.7 · R_tree.
- Evaluation metrics: Because the two training variants optimize different objectives, both are compared with shared held-out tree-generation, node-count, and combined-reward metrics.Tree-generation accuracy is the primary metric because exact containment-tree reconstruction is CurveBench’s core objective.
- Evaluation metrics: Tree matching canonicalizes child-subtree order before testing rooted-tree isomorphism, making sibling ordering and region identifiers irrelevant.A prediction is correct only when the canonicalized predicted and ground-truth trees are isomorphic.
6 Results
CurveBench results expose substantial difficulty in exact topological reasoning, while RLVR improves open-weight models more on Easy than Hard. Performance varies sharply by curve configuration, with maze-like structures remaining especially challenging.
- Overall performance: Gemini 3.1 Pro is the strongest overall model on both CurveBench-Easy and CurveBench-Hard across tree-generation accuracy, node-count accuracy, and average reward.Easy uses a held-out test set, whereas Hard is evaluated on the full benchmark.
- RL fine-tuning: The fine-tuned region-tree model raises Easy average reward from 0.038 for Qwen3-VL-8B-Thinking to 0.397 after training.This result demonstrates benchmark utility as a training signal, although the base model’s zero-shot performance was weak.
- RL fine-tuning: On Hard, the only-tree model improves tree-generation accuracy from 0.042 to 0.070 and reaches 0.095 average reward, but gains remain smaller than on Easy.The results indicate that generalization to the harder benchmark remains challenging.
- Interpretation: The results reveal a topological gap: strong vision-language models remain far from reliably recovering hierarchical containment from nested curves.The benchmark distinguishes this capability from object detection and OCR performance.
- Category difficulty: Topographical images are generally easiest, while Counting and Polygon show moderate performance and Maze is the most significant failure point.Most Instruct models score 0.0% on Maze, whereas Qwen3-VL-8B-Thinking reaches 11.0%.
- Category difficulty: Models struggle to preserve structural integrity as node counts increase, often losing track of depth in Counting and Polygon configurations.Both gpt-5.2 and gemini-3-pro reach 17.5% on Counting.
- Limitations: Fine-tuning improves Counting and Polygon performance but regresses Maze performance relative to the raw Thinking base.The authors identify this trade-off as a potential alignment tax requiring future reward-shaping research.
7 Limitations
CurveBench prioritizes verified exact structural evaluation in a small, controlled benchmark, with scope and metric choices that constrain interpretation.
- 756 images make CurveBench a diagnostic benchmark rather than a large-scale pretraining corpus.The authors trade dataset scale for high-quality structural annotations, human verification, and exact tree-based evaluation.
- The benchmark covers nested, pairwise non-intersecting Jordan curves but excludes intersecting curves, open contours, noisy segmentations, 3D topology, temporal structure, and ambiguous natural images.Extending it to these settings would likely require different annotation schemes and evaluation metrics.
- Training uses CurveBench-Easy because harder subsets often yield near-zero reward, so fine-tuning primarily tests verifiable learning on Easy and only indirectly tests harder transfer.Future curriculum-style splits could increase complexity, nesting depth, visual clutter, and boundary length gradually.
- Synthetic, controlled images enable exact ground truth and deterministic evaluation but limit ecological validity for noisy real-world visual inputs.Performance should not be interpreted as sufficient evidence of robust handling of natural maps, scientific figures, medical images, or arbitrary contour-like structures.
- Exact tree match is stringent but coarse because it does not distinguish near-correct predictions from malformed or substantially incorrect outputs.The authors suggest finer-grained metrics such as edge and ancestor F1, normalized tree distance, depth and count accuracy, parse failure rate, and complexity-stratified performance.
8 Conclusion
CurveBench evaluates whether vision-language models can recover exact containment trees from disjoint-curve images. The benchmark and RLVR results expose persistent difficulty in exact topological inference while standardizing the output task and evaluation conditions.
- CurveBench asks models to recover the exact rooted containment tree induced by an image of pairwise disjoint Jordan curves.The task targets hierarchical reconstruction of planar regions.
- RLVR substantially improves an open-weight model, but the remaining gap shows robust topological inference is far from solved.The conclusion presents verifiable-reward fine-tuning as a promising route while emphasizing unresolved difficulty.
- The benchmark’s fixed prompt requires a parseable edge-list answer describing parent-child relations between regions.The root is node 0, and the response includes the number of non-root nodes followed by parent-child edges inside <answer> tags.
- Node-count hints were evaluated as prompt variants rather than used for the main benchmark results.The authors note that such hints can change task difficulty and affect node-count and tree-structure accuracy.
B.1 Released Datasets
CurveBench releases Easy and harder benchmark resources alongside documentation, licensing, metadata, and public reproducibility materials.
- CurveBench releases CurveBench-Easy and the main CurveBench benchmark for harder evaluation.
- The release provides datasets, code, training artifacts, experiment logs, and ground-truth generation utilities through public project resources.
- The datasets use CC BY 4.0, while benchmark and evaluation code use the MIT License.The licenses are also specified in dataset cards, repository documentation, and Croissant metadata.
- Croissant JSON-LD files document dataset structure, annotations, licensing, collection, intended use, and responsible-AI metadata.They include both core metadata and fields covering out-of-scope uses, limitations, privacy, and misuse risks.
B.3 Evaluation Environments
CurveBench provides standardized evaluation environments and reproducibility artifacts for Easy and Hard benchmark settings, fixing the conditions under which models are assessed.
- Standardized environments fix the dataset split, input formatting, evaluation prompt, answer parser, and reward function.This makes model comparisons use the same conditions and improves reproducibility.
- Separate environments support the CurveBench-Easy test split and the full CurveBench-Hard benchmark set.
- Easy results use four rollouts per sample, whereas Hard results use one rollout because the Hard benchmark is larger.
- The project repository includes construction utilities, OpenCV-based ground-truth extraction, parsers, reward computation, reinforcement-learning code, and logs.The extraction pipeline traces boundaries, identifies containment, builds a rooted tree, and human-verifies each annotation.