PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMSeg: Training-Free Text-Prompted Image Segmentation with B-Splines

Abdullah Hamdi

arXiv:2610.29409v1cs.CVcs.AI

TL;DR

LLMSeg addresses whether a general-purpose MLLM can segment images without a trained segmentation component. It converts text queries into masks using box-confined B-spline geometry and iterative refinement, and reports competitive results against SAM 3, including 71.5 cIoU on ReasonSeg validation.

  • Problem

    Promptable specialist segmenters require dedicated architectures, large-scale annotation engines, and task-specific training, while MLLMs are widely assumed to need a trained decoder for dense masks.

    Source: §1 · p. 2

  • Method

    LLMSeg uses a frozen multimodal LLM to detect instances, emit box-confined B-spline geometry, and iteratively refine masks through signed focus boxes.

    Source: §1 · p. 2, §3 · p. 3, §3.3 · p. 5

  • Results

    LLMSeg is competitive with and often surpasses SAM 3 across the evaluated benchmarks, reaching 71.5 cIoU on ReasonSeg validation.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    Because LLMSeg adds no trained parameters, backend upgrades can improve segmentation accuracy without retraining the pipeline.

    Source: §1 · p. 2

    The predicted hole count is not guaranteed to survive loop overlap, rasterization, refinement, or cleanup.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Promptable segmentation models such as SAM 3 achieve remarkable open-vocabulary performance but require dedicated architectures, large-scale annotation engines, and task-specific training. We ask a different question: can a general-purpose multimodal large language model (MLLM) segment images directly, without an added segmentation head, mask decoder, or training? We present LLMSeg, a training-free pipeline that turns a vision-capable LLM into a text-prompted image segmenter. LLMSeg decomposes segmentation into three LLM-driven stages. First, detect localizes all instances matching a free-form text query and describes their coarse morphology. Next, segment traces each instance using closed periodic B-splines confined to its detected box. These splines define an outer boundary and optional hole loops. Finally, refine iteratively proposes focus boxes around residual errors and retraces confined splines. The process ends when the mask stabilizes or the pass budget is exhausted. The output is a binary mask with inspectable intermediate geometry. We evaluate LLMSeg across ten referring and reasoning segmentation splits (ReasonSeg and RefCOCO/+/g). We also evaluate open-vocabulary instance segmentation (LVIS, COCO, and SA-Co), semantic segmentation, and zero-shot detection (ODinW13 and RF-100VL). With an off-the-shelf Gemini-3.7-Flash backend, LLMSeg is competitive with and often surpasses the specialist SAM 3 model on identical items. It reaches 71.5 cIoU on ReasonSeg val without additional segmentation-specific training. Backend model updates can improve accuracy without retraining the pipeline. The code is available at https://abdullahamdi.com/llmseg .

1 INTRODUCTION

LLMSeg asks whether a general-purpose MLLM can produce text-prompted masks without a trained segmentation component. It answers with a training-free pipeline and reports competitive benchmark results, including a 71.5 cIoU score on ReasonSeg validation.

1 INTRODUCTION

  • The pipeline challenges the assumption that MLLMs need a trained segmentation decoder to produce dense masks, using a general-purpose MLLM as its only vision component.The model returns closed curves that deterministic code rasterizes into masks confined to detected boxes.Source: §1 · p. 2
  • LLMSeg represents boundaries with cubic or degree-one traces, supports subtractive hole loops, and makes geometric decisions inspectable through human-readable JSON.The same LLM interprets reasoning queries and performs segmentation; backend upgrades require no pipeline retraining and usually bring accuracy gains.Source: §1 · p. 2
  • LLMSeg is competitive with SAM 3 across reasoning, referring, instance, semantic, and zero-shot detection benchmarks, reaching 71.5 cIoU on ReasonSeg validation.The ReasonSeg result is the best among methods not fine-tuned for the task.Source: §1 · p. 2
  • The work frames its contribution as benchmark-scale evidence that dense text-prompted segmentation can be performed without any trained segmentation component.The authors report that segmentation quality follows backend strength without retraining.Source: §1 · p. 2

2 RELATED WORK

Earlier approaches combine language reasoning with trained mask components, or delegate mask production to a segmentation model. LLMSeg instead has a frozen, general-purpose MLLM emit boundary geometry for deterministic processing.

2 RELATED WORK

  • Many MLLM-based segmentation methods retain trained mask components or train MLLMs to prompt SAM, while training-free and agentic alternatives still delegate mask production to a segmentation model.The cited examples include LISA, GLaMM, PSALM, X-SAM, SAM 3 Agent, and the Colon-Bench pipeline.Source: §2 · p. 2
  • Prior text-based geometry methods use trained multitask pipelines, task-specific decoders, learned localization heads, or fine-tuning, unlike LLMSeg’s frozen backend and deterministic geometry processing.General models including Molmo and Gemini provide pointing or coarse localization; LLMSeg uses their output space without training a contour predictor.Source: §2 · pp. 2–3
  • Figure 2 depicts the pipeline’s three stages: detection localizes instances, segmentation traces box-confined B-splines, and refinement corrects masks within signed focus boxes.Each MLLM call serves all instances, although the figure shows one.Source: Figure 2 · p. 3

3 METHOD

LLMSeg uses a frozen multimodal LLM and deterministic code to turn a free-form query into instance masks. Detection supplies instance boxes, B-spline traces form initial masks, and iterative refinement edits residual errors within signed focus boxes.

3 METHOD

  • A frozen multimodal LLM is the sole vision component; fixed text prompts, stateless calls, and deterministic code control the pipeline without parameter updates or a segmentation head.The final output is the union of retained instance masks at termination.Source: §3 · p. 3

3.1 STAGE 1: DETECTING INSTANCES AND THEIR STRUCTURE

  • Detection returns each matching instance’s pixel-coordinate box, confidence, coarse morphology, and optional topology descriptor, then merges near-duplicates and retains at most Nmax instances.If no instances are detected, the output is empty.Source: §3.1 · p. 3

3.2 STAGE 2: FROM LANGUAGE TO GEOMETRY

  • Segmentation traces each instance with a closed periodic B-spline; the model selects cubic traces for rounded forms or degree-one traces for angular edges.Curves are sampled and filled after clamping to the instance box, preventing a trace from painting outside it; optional hole loops are subtracted from the outer region.Source: §3.2 · pp. 3–4

3.3 STAGE 3: ITERATIVE REFINEMENT

  • Each refinement pass uses the rendered previous mask to locate errors, then retraces focus boxes confined to the relevant instance box with a sign requesting pixel addition or removal.An effect gate can skip retracing when boxes have negligible potential to change the mask; composition resists opposing edits, and the loop ends at the pass limit or after negligible change for s consecutive passes.Source: §3.3 · pp. 4–5

4 EXPERIMENTS

LLMSeg is evaluated across referring and reasoning segmentation, open-vocabulary instance segmentation, semantic segmentation, and zero-shot detection, using standard benchmark metrics and a fixed Gemini-3.7-Flash configuration. Results vary by task: it achieves strong referring and reasoning results and leads on several detection metrics, while trailing SAM 3 on Cityscapes and mask AP.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers ReasonSeg and all RefCOCO/+/g splits, LVIS, COCO, SA-Co/Gold, ADE20K, Cityscapes, ODinW13, and RF-100VL, using each benchmark's standard metrics.The main-results configuration uses Gemini-3.7-Flash, an eight-pass refinement budget, and the same settings across benchmarks without dataset-specific tuning.Source: §4.1 · pp. 5–6
  • The study fully evaluates 9,703 of 17,703 scored items; the remaining annotation-heavy splits use seeded random samples, with baselines recomputed on LLMSeg's same items.Figure 4 arranges qualitative comparisons with inputs and amber ground truth on top, baselines or unrefined LLMSeg in the middle, and LLMSeg below.Source: §4.1 · p. 6, Figure 4 · p. 6, §4.2 · p. 7

4.2 MAIN RESULTS

  • 71.5 cIoU on ReasonSeg validation is the best result among methods not fine-tuned for the task; refinement adds 6.9 cIoU over the segment stage alone.On RefCOCO/+/g, LLMSeg substantially outperforms SAM 3, while SAM 3 Agent generally remains stronger; RefCOCO testB reaches 76.0 cIoU, the best zero-shot result on that split.Source: §4.2 · p. 7
  • 53.9 cgF1 on LVIS and 56.1 cgF1 on COCO exceed both SAM 3 baselines, although LLMSeg trails SAM 3 in mask AP.The Table 3 caption identifies the open-vocabulary instance segmentation comparison and defines cgF1 as presence-gated F1.Source: Table 3 · p. 7, §4.2 · p. 8
  • 57.2 AP0 on ODinW13 and 31.4 AP0 on RF-100VL exceed SAM 3 on all four metrics in both benchmarks; RF-100VL AP0 is 2.2× SAM 3's.LLMSeg reaches 59.2 gIoU on ADE20K, but its 53.3 mIoU on Cityscapes trails SAM 3 by 10.4 points, leaving dense scene labeling as a limitation.Source: Table 2 · p. 7, §4.2 · p. 8

5 ANALYSIS

With the pipeline fixed, tested MLLM backends vary widely in ReasonSeg performance, and later backends can exceed SAM 3 Agent without a segmentation model. Refinement improves Gemini-3.7-Flash results but adds inference cost, while ablations show trade-offs among detection, batching, and curve degree.

5.1 BACKEND PROGRESS TRANSFERS WITHOUT SEGMENTATION TRAINING

  • 74.2 gIoU on ReasonSeg validation is reached by GPT-6 Astra without a segmentation model, versus 72.4 for SAM 3 Agent driven by Gemini-3.7-Flash, across a fixed-pipeline study of 21 backends.Stage-2 gIoU ranges from 13.4 to 74.2; backend updates can improve accuracy without retraining, but general benchmark capability is a weak predictor of segmentation quality.Source: §5.1 · p. 8

5.2 REFINEMENT IMPROVES THE BOUNDARY AND USUALLY STOPS EARLY

  • Refinement improves every Gemini-3.7-Flash benchmark, but removing it lowers cIoU by 5.4 on ReasonSeg and 12.7 on RefCOCO; the loop is also the largest inference cost.Four passes recover 2.7 of the eventual 3.2 gIoU gain, raising the budget from eight to twelve changes gIoU by under 0.1, and early stopping cuts mean calls from 26 to 14.7 at T=12.Source: §5.2 · p. 8, Figure 6 · p. 9

5.3 ABLATION STUDY

  • Removing detection lowers initial cIoU from 68.3 to 60.3 on ReasonSeg and from 66.0 to 63.5 on RefCOCO, while its initial advantage is 11.3 cIoU on disconnected ReasonSeg targets.After refinement, the no-detection configuration closes these gaps with fewer total calls.Source: §5.3 · p. 9
  • Per-instance cropped refinement adds 49% more ReasonSeg calls for a 2.3-point cIoU increase, while batched tracing uses one call regardless of instance count.On RefCOCO, cropping adds calls for only a 0.4-point increase; the design-alternatives table compares final cIoU with mean MLLM calls per image.Source: §5.3 · p. 9, Table 4 · p. 9
  • Letting the model choose curve degree improves performance on some instances over degree-one polygon loops, especially for curved structures that are difficult to represent with straight lines.The design-alternatives table reports final cIoU and mean calls; ReasonSeg values are three-run means with standard deviations, whereas RefCOCO configurations have one run.Source: §5.3 · p. 9, Table 4 · p. 9

6 CONCLUSION

LLMSeg uses a frozen, general-purpose MLLM to produce mask geometry directly, without segmentation training or a downstream mask component. Its conclusion highlights competitive performance and specific limitations; the appendix details related work, structured replies, and deterministic mask composition.

6 CONCLUSION

  • Across ten benchmarks spanning four task families, LLMSeg is competitive with SAM 3 and performs better on reasoning queries, rare domains, and some metrics.The conclusion lists detection errors, imprecise pixel-level boundaries, amorphous classes, and inference cost as limitations.Source: §6 · p. 9

A EXTENDED RELATED WORK

  • Unlike prior MLLM segmentation pipelines that use trained decoders or delegate mask production, LLMSeg’s frozen MLLM emits the final mask geometry as parametric splines.The related-work discussion distinguishes its output from coarse per-token masks and from coordinate text used as an intermediate for a trained head.Source: A EXTENDED RELATED WORK · pp. 16–17

B.1 STRUCTURED REPLIES AND COORDINATE CONVENTIONS

  • Structured prompts specify detections, spline loops, and refinement boxes; coordinates are converted between backend conventions and image-pixel coordinates.Replies use explicit schemas, and Gemini-family box coordinates are requested in native order then converted; control points are checked against their known box.Source: §B.1 · p. 18
  • Truncated batched replies retain completed instances, while missing instances remain eligible for refinement from their boxes.Replies are emitted in ascending instance-ID order, so truncation affects the tail; instances with no loops remain in scoring as empty masks.Source: §B.1 · p. 18

B.2 RASTERIZATION AND COMPOSITION

  • Cubic loops are rasterized with a box-scaled sample count, then deterministic composition and cleanup form the mask while topology counts can preserve predicted holes.The sample count is min{2000, max{64, round(2(wb + hb))}}; cleanup removes specks and spurious holes using thresholds scaled to object area.Source: §B.2 · pp. 18–19
  • Refinement composes signed regions using a per-pixel commitment field, so opposing edits must overcome accumulated resistance before flipping a pixel.Repeated agreement strengthens a commitment; setting the cancellation unit to zero would freeze committed pixels.Source: §B.2 · p. 18

B.3 ONE ITEM, STEP BY STEP

The spare-wheel example shows how local focus-box refinement corrects a stage-2 boundary error and reaches a converged mask. The figure traces the model’s returned geometry and the mask changes across passes.

B.3 ONE ITEM, STEP BY STEP

  • Refinement raises the spare-wheel mask from 84.1 IoU after stage 2 to 91.9, correcting a fender wedge inside an otherwise good boundary.The figure’s boxes, control points, loops, and masks are the returned geometry and its exact deterministic reconstruction.Source: §B.3 · p. 19, Figure 7 · p. 20
  • The first refinement pass removes 15,118 fender pixels and adds 7.3 IoU points; later edits resist an opposing wedge proposal, changing only one pixel.Without the commitment field, that proposal would add 7,791 pixels; the loop stops after four of twelve allowed passes.Source: §B.3 · p. 19, Figure 7 · p. 20
  • Stage 2 uses a cubic spline with 24 control points, and the figure shows the resulting loop, ground truth, and IoU after each pass.The spline converts the points into a smooth closed curve rather than a 24-gon.Source: §B.3 · p. 19, Figure 7 · p. 20

C.1 BENCHMARKS

The benchmark suite spans referring, reasoning, open-vocabulary, semantic, and detection tasks, using one image-query pair as an item. Main results use a fixed Gemini-3.7-Flash configuration across benchmarks, without dataset-specific tuning.

C.1 BENCHMARKS

  • Benchmarks cover ReasonSeg, RefCOCO/+/g, LVIS, COCO, SA-Co/Gold, ADE20K-150, Cityscapes, ODinW13, and RF-100VL.The suite includes reasoning and referring masks, open-vocabulary instance segmentation, semantic segmentation, and zero-shot detection.Source: §C.1 · p. 21, Table 5 · p. 21
  • Each benchmark item is one image-query pair, and the query is the only prompt any system receives.ReasonSeg queries identify targets by role or function, while RefCOCO expressions disambiguate instances among same-class neighbors.Source: §C.1 · p. 21

C.2 CONFIGURATION OF LLMSEG

  • Main results use Gemini-3.7-Flash across all three stages, with settings unchanged across benchmarks and no dataset-specific tuning.The reported configuration leaves topology conditioning off; it averages 13.4 calls and 159 seconds per ReasonSeg validation image.Source: §C.2 · p. 21
  • Backend and ablation studies use T=12 with topology conditioning on, unlike the reported configuration, so cross-configuration comparisons mix both settings.Rows within a study are comparable; the ablation reference row uses the reported setting at T=12.Source: §C.2 · p. 21

C.3 EVALUATION PROTOCOL AND MATCHED-SUBSET SCORING

The evaluation protocol scores empty-target cases and uses matched items when comparing systems, while several sampled AP results have restricted comparability. The section also specifies baseline reproduction, backend-study conditions, and cost-accounting assumptions.

C.3 EVALUATION PROTOCOL AND MATCHED-SUBSET SCORING

  • ReasonSeg scoring retains legitimately empty masks, crediting correct empty predictions and penalizing hallucinated regions; excluding them raises gIoU and breaks comparability with quoted results.The validation split has four such items and the test split has two.Source: §C.3 · p. 22
  • LVIS and COCO AP use matched subsets of only 48 of 1,449 and 62 of 1,304 images, respectively, and are not comparable to published full-split federated AP.Images enter AP evaluation only when every ground-truth category they contain was queried; surviving images skew toward those with few categories.Source: §C.3 · p. 22

C.4 REPRODUCING THE SAM 3 AGENT BASELINE

  • The reproduced SAM 3 Agent uses three tools, an eight-round cap, and different MLLM drivers from the published evaluation, yielding scores somewhat below its published ones.The drivers are Gemma-4-31B, Gemini-3.5-Flash, and Gemini-3.7-Flash rather than Qwen2.5-VL and Gemini 2.5 Pro.Source: §C.4 · p. 22

C.5 BACKEND STUDY PROTOCOL

  • The backend study holds LLMSeg fixed across 21 backends on the same 200 ReasonSeg validation items, with failures scored zero and bootstrap confidence intervals.Calls have wall-clock budgets; high-latency backends receive longer budgets, and Qwen3-VL-235B’s score reflects its unusable-reply rate as well as segmentation quality.Source: §C.5 · p. 22
  • SAM 3 Agent gIoU varies by 0.7 points across the three tested MLLMs, versus 10.8 points for LLMSeg across one Gemini Flash product line.The SAM 3 Agent values are 72.4, 72.5, and 73.1.Source: §C.5 · p. 23

C.6 COST ACCOUNTING

  • Backend dollar costs are estimates and lower bounds, based on shared input/output token budgets and measured call counts, excluding billed reasoning tokens.SAM 3 is assigned a market price of $0.0043 per pass per image; LLMSeg incurs no segmentation-model cost because it calls no segmentation model.Source: §C.6 · p. 23

D.3 QUALITATIVE RESULTS ON THE MAIN BENCHMARKS

The appendix extends qualitative benchmark evidence across referring, instance, semantic, and difficult-domain examples, using matched Gemini-3.7-Flash runs and system-recorded IoU chips. It also points to the extended ablation results and their study configuration.

D.3 QUALITATIVE RESULTS ON THE MAIN BENCHMARKS

  • The qualitative grids use the same Gemini-3.7-Flash runs as the tables; stage-2 and final masks isolate refinement, while Cityscapes panels show full label maps.Figure 8 reports panel IoU and uses amber for ground truth and green with a red outline for predictions; Figure 9 reports image mIoU.Source: §D.3 · p. 24, Figure 8 · p. 27, Figure 9 · p. 28

RefCOCO RefCOCO+ RefCOCOg

  • The RefCOCO/+/g table contrasts published full-split results with zero-shot methods against matched 500-item subsets evaluated on identical items.The matched blocks report gIoU and cIoU; their SAM 3 and SAM 3 Agent baselines were reproduced on the selected items.Source: Table 7 · p. 25

LVIS COCO SA-Co/Gold

  • The LVIS, COCO, and SA-Co/Gold table separates published results from matched-subset evaluations that include hard negatives.Within the starred and shaded matched block, it marks the best and second-best result per column; cgF1 is presence-gated F1 and pmF1 its positive-only counterpart.Source: Table 8 · p. 26

D.4 CROWDED SCENES, NEGATIVES, RARE DOMAINS, AND SINGLE PASSES

  • Crowded-scene, absent-concept, rare-domain, and single-pass examples extend the qualitative evidence using the same Gemini-3.7-Flash T=12 runs and matched SAM 3 items.Crowded-scene chips report union IoU, which does not distinguish touching instances merged into one region; absent-concept chips indicate only whether a mask was produced.Source: §D.4 · p. 26, Figure 11 · p. 30, Figure 12 · p. 31
  • RF-100VL examples span document, aerial, medical, and industrial subsets, with category names used verbatim and annotated detection boxes shown as ground truth.The caption cautions that failures may reflect domain vocabulary rather than localization.Source: Figure 13 · p. 32

E EXTENDED ANALYSIS

  • The extended analysis directs readers to complete ablation tables and states that the backend study and ablations use the configuration described in Section C.2.Source: E EXTENDED ANALYSIS · p. 34

E.1 REFINEMENT AND CONVERGENCE

The refinement loop has no convergence guarantee, so the study tests whether mask-change residuals support early stopping and track quality. Most ReasonSeg validation items stop before the pass cap, while residual stability remains distinct from correctness.

E.1 REFINEMENT AND CONVERGENCE

  • 79% of ReasonSeg validation items stop before the T=12 cap, with a median of six passes; quality gains largely saturate before the full budget.In this run, the first pass adds +2.0 gIoU, four passes yield +2.7 of the eventual +3.2, and passes 8–12 change quality by less than 0.1.Source: §E.1 · p. 34
  • Residuals decay but are not monotone, and their starting size does not predict how quickly an image stabilizes, supporting per-image rather than globally fixed pass allocation.On the 16-image development set, first-pass repainting ranges from 0.2% to 27% of the box union; ε=0.005 with s=3 stops 11 items on their own.Source: §E.1 · pp. 34–35
  • A small residual indicates mask stability, not correctness, so a loop can stop on an incorrect mask.The study supports residuals as a surrogate because quality saturates over the same passes, but this does not establish correctness for an individual mask.Source: §E.1 · p. 35
  • Figure 16 plots per-image smoothed residuals against the stopping threshold, distinguishing items that stop on their own from those that exhaust the budget.The plotted residual is the three-pass moving average, and × marks the first smoothed pass below ε.Source: Figure 16 · p. 35

E.2 BACKEND STUDY

The backend study holds LLMSeg’s pipeline fixed across 21 MLLMs and finds a wide spread in segmentation quality. Refinement effects vary by backend, while backend releases can improve performance without pipeline changes; public capability scores do not reliably predict quality.

E.2 BACKEND STUDY

  • 14.2–72.9 final gIoU spans the 21 backends on ReasonSeg; all score at least SAM 3’s 13.1, though the weakest is not clearly above it.GPT-6 Astra reaches 72.9 final gIoU against 72.4 for SAM 3 Agent; the agent score lies within GPT-6 Astra’s item-bootstrap interval.Source: §E.2 · pp. 35–36, Table 9 · p. 37
  • No public capability axis reliably predicts LLMSeg quality; reasoning and vision track it most closely, while agentic skill shows almost no association.The passage reports that no axis reaches significance and recommends testing the pipeline on the backend itself rather than relying on headline capability scores.Source: §E.2 · p. 36
  • All four product lines with multiple releases improve over the period shown while LLMSeg remains unchanged, but newer releases are not uniformly better.Gemini 3.8 Flash scores 2.7 below Gemini 3.7 Flash; Gemini Flash-Lite gains +15.7 gIoU across releases.Source: §E.2 · pp. 36–37, Figure 17 · p. 36
  • Refinement lowers gIoU on 11 of 21 backends, so its gains depend on a model’s ability to inspect and correct its own masks.It adds +3.7 gIoU for Gemini-3.7-Flash but costs 1.2 for GPT-6 Astra, the strongest backend.Source: §E.2 · pp. 35, 37
  • Refinement gains reported elsewhere are Gemini-3.7-Flash-specific, and the appropriate refinement budget must be checked for each backend.The loop can move masks off the object when a backend mislocates its focus boxes.Source: §E.2 · p. 37

E.3 ABLATIONS AND DESIGN ALTERNATIVES

The ablations compare mask quality and MLLM calls across design alternatives on ReasonSeg and RefCOCO, with repeated ReasonSeg runs and a single RefCOCO run. Refinement has the largest component effect, while several alternatives trade structural control or cost against split-dependent quality.

E.3 ABLATIONS AND DESIGN ALTERNATIVES

  • Removing refinement costs 5.4 gIoU and 5.4 cIoU on ReasonSeg, and 9.8 gIoU and 12.7 cIoU on RefCOCO; topology conditioning reduces observed cIoU on both splits.The topology reductions are 2.1 on ReasonSeg and 2.5 on RefCOCO; the ablation reference is topology-on, whereas the reported configuration leaves topology off.Source: §E.3 · pp. 37–38
  • Removing morphology or topology prompt descriptions does not measurably improve split-level quality, though topology information remains an option for explicit structural control.Topology can guide hole carving and gated cleanup, but the passage says this applies to 0%–7% of benchmark targets and is not claimed as an aggregate-accuracy source.Source: §E.3 · pp. 38–39
  • Skipping detection is competitive on aggregate quality and uses fewer calls, but detection improves initial cIoU on disconnected targets in both splits.Those targets are 63/200 ReasonSeg and 18/200 RefCOCO items; their initial cIoU is 63.9 with detection versus 52.6 and 61.7 without it, respectively.Source: §E.3 · p. 41
  • Per-instance crop refinement was the most expensive variant, at 21.3 calls per image versus 14.6 for the reported configuration, with 74.9 cIoU versus 74.7; on RefCOCO it scored 77.7 versus 79.8.Its cost grows with instance count.Source: §E.3 · p. 41
  • At similar call cost, polygon traces score comparably to B-splines, but the unequal coordinate budgets and backend-selected degree limit the comparison.On cubic-traced items, B-splines use 28.9 coordinates per image versus 48.5 for polygons; results support a compact parametric representation, not uniform accuracy superiority.Source: §E.3 · p. 41
  • The fifteen configuration rankings by final gIoU agree across ReasonSeg and RefCOCO val, with ρ = 0.71 and p = 0.003.ReasonSeg configurations were run three times and RefCOCO configurations once; the authors advise reading the comparisons for direction only.Source: §E.3 · pp. 37, 41

E.4 WHERE THE GROUND TRUTH HAS HOLES

Threshold-by-threshold results show that LLMSeg localizes objects well but retains loose boundaries at high overlap. Hole-aware topology is explicit, but the measured benchmarks show no accuracy gain from it.

E.4 WHERE THE GROUND TRUTH HAS HOLES

  • Holed targets are uncommon: 7.1% of ReasonSeg val, 6.6% of ReasonSeg test, 4.7% of LVIS positives, and 0–1.4% of RefCOCO splits.The paper notes that COCO polygon annotations essentially never carve a hole.Source: §E.4 · p. 42
  • On 396 ReasonSeg items, topology-on versus topology-off changes mean per-item IoU by −0.3 overall and +0.1 on the 19 holed items.The reported intervals are [−1.8, 1.2] overall and [−5.3, 5.3] for holed items; the authors caution that the latter sample is small and pass budgets differ.Source: §E.4 · pp. 41–42, Table 10 · p. 42
  • Cubic B-splines cover curved boundaries better than polygons with fewer control points, while LLMSeg can carve explicit holes that SAM 3 fills solidly in the sewage-outlet example.The spline comparison uses identical items and the same backend.Source: Figure 22 · p. 43, Figure 23 · p. 43

E.5 LOCALIZATION OR BOUNDARY?

  • At IoU 0.90, LLMSeg mask AP is 16.4 on both LVIS and COCO, versus SAM 3's 42.5 and 24.3, respectively.The threshold sweep shows a larger mask-AP gap than box-AP gap; refinement improves masks but does not make them pixel-tight.Source: §E.5 · p. 43, Figure 24 · p. 44

E.6 WHAT A MASK COSTS

LLMSeg’s refinement loop accounts for most of its inference time, and the comparison with SAM 3 Agent shows a substantial cost and latency trade-off on ReasonSeg. Its performance varies by task and shape, while the prompt templates specify constrained geometry and local corrections.

E.6 WHAT A MASK COSTS

  • Stage 3 consumes 76–94% of wall-clock time across backends, including 89% for Gemini-3.7-Flash.The paper attributes the cost to repeated calls; per-call latency means call count does not directly indicate useful refinement.Source: §E.6 · pp. 44–45, Figure 25 · p. 45
  • On ReasonSeg, SAM 3 Agent reaches 72.4 gIoU versus LLMSeg’s 69.4 in 11 seconds versus 153 seconds, at $8.31 versus $15.61 per thousand images.The agent uses 2.4 rounds and 1.4 SAM 3 passes, compared with LLMSeg’s 15 vision calls.Source: §E.6 · pp. 45–46

E.7 WHERE THE LLM ROUTE WINS AND LOSES

  • LLMSeg’s strongest reported settings include reasoning queries and rare-domain detection, but it trails SAM 3 by 9.0 cgF1 on SA-Co/Gold.The paper also reports losses on pixel-tight mask AP and thin, amorphous, or highly irregular regions.Source: §E.7 · p. 46
  • Backend latency, hallucinated detections, coordinate instability, box-constrained masks, and API model updates limit real-time use and exact reproducibility.The authors state that per-call latency of seconds to minutes makes video and real-time use out of scope.Source: §E.7 · p. 46

E.8 FAILURE MODES

  • Failure cases include thin retinal vessels, amorphous and heavily occluded Cityscapes backgrounds, and deep narrow concavities in articulated hands.The figure notes that closed loops cannot cut into the hands’ deep, narrow concavities.Source: §E.8 · p. 46, Figure 26 · p. 47

F PROMPT TEMPLATES

  • The prompt templates define coordinate bounds, empty-output options, and ordered batched responses so failures can be expressed and truncated replies salvaged.The templates also describe each focus box’s shape and complexity for the subsequent local tracing step.Source: F PROMPT TEMPLATES · pp. 48, 50
Loading 2610.29409v1…