PaperlayerView exact revision on arXiv

Source-linked AI summary

GarmentSegAgent: Autonomous Garment Segmentation for Web Fashion Images via Textual Self-Refinement

Jia Chen, Chao Huang, Xinrong Hu, Mingdi Huang, Ruyang Zhao, Jun Bai

arXiv:2610.36506v1cs.CVcs.MA

TL;DR

Heterogeneous Web fashion images make reliable garment segmentation difficult, particularly when systems must identify and correct garment-specific errors without human-provided spatial prompts. GarmentSegAgent uses iterative textual self-refinement with CAST to connect visual errors to targeted, constrained instruction updates. Across three benchmarks, it improves region overlap and boundary accuracy over competitive baselines.

  • Problem

    Heterogeneous Web fashion images make garment segmentation challenging, and existing promptable methods do not necessarily diagnose and correct instruction errors without human intervention.

    Source: §1 · p. 1

  • Method

    GarmentSegAgent uses iterative self-refinement, with CAST attributing errors to instruction fields and selectively updating relevant variables under garment-specific constraints.

    Source: §1 · pp. 1–2

  • Results

    Experiments on DeepFashion2, ModaNet, and Fashionpedia show consistent improvements in region overlap and boundary accuracy over competitive baselines.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The findings highlight the potential of error-guided textual optimization for coordinating frozen foundation models in visual tasks.

    Source: §6 · p. 9

    Evaluation focuses on single-target upper-garment segmentation; broader garment categories and multi-instance scenarios remain unexplored, and iterative refinement adds inference overhead.

    Source: §5 · p. 8

Abstract

from arXiv · show

Web fashion images support applications such as visual product search and consumer-to-shop matching, yet their visual heterogeneity makes reliable garment segmentation challenging. Despite advances in promptable segmentation, autonomously identifying and correcting garment-specific errors without human-provided spatial prompts remains difficult. We propose GarmentSegAgent, an autonomous framework that integrates semantic planning, spatial execution, and visual evaluation through iterative textual self-refinement. At its core, we introduce CAST (Constrained Attribution-guided Selective TextGrad), which connects visual error attribution with targeted textual optimization. CAST identifies instruction components responsible for segmentation errors, selectively updates them through textual gradients, and validates the resulting instructions against garment-specific constraints. This enables image-specific error correction at inference time while keeping the underlying foundation models frozen. Experiments on DeepFashion2, ModaNet, and Fashionpedia demonstrate consistent improvements in region overlap and boundary accuracy, alongside more reliable refinement than conventional TextGrad. By enabling visual feedback to guide controlled instruction updates, GarmentSegAgent offers a promising direction for autonomous garment understanding, with potential applications in scalable fashion content analysis, visual retrieval, and consumer-to-shop matching on the Web.

1 Introduction

Heterogeneous Web fashion images make garment segmentation difficult, especially when target garments must be identified without human-provided spatial prompts. GarmentSegAgent addresses this by connecting visual error diagnosis to selective, garment-constrained textual refinement.

1 Introduction

  • Web fashion images contain clutter, diverse viewpoints, occlusions, and overlapping garments, making pixel-level localization difficult without human-provided spatial prompts.Accurate masks help isolate clothing instances and preserve fine-grained spatial details for downstream fashion understanding.Source: §1 · p. 1
  • Promptable segmentation and agentic feedback still leave a key challenge: using visual errors to guide updates to specific segmentation instructions.Missing garment regions and background leakage call for distinct corrections to coverage and region exclusion.Source: §1 · p. 1
  • General-purpose whole-prompt refinement can disrupt valid segmentation decisions, motivating error-relevant variable selection and garment-specific constraints.TextGrad optimizes textual variables through language-based backpropagation in compound AI systems.Source: §1 · p. 1
  • CAST attributes errors to relevant instruction fields and selectively updates them, preserving unrelated instructions under garment-specific constraints.It maps missing garment regions to coverage refinement and background leakage to exclusion refinement.Source: §1 · p. 2, Figure 1 · p. 2
  • GarmentSegAgent integrates semantic planning, spatial execution, and visual evaluation in an autonomous iterative framework; experiments across three benchmarks show superior region and boundary accuracy.The authors also report that refinement analyses and ablations establish CAST's effectiveness in improving optimization reliability.Source: §1 · pp. 1–2

2 Related Work

Prior work extends segmentation through sequential reasoning, tool interaction, and feedback, while textual optimization refines instructions without differentiating through underlying models. These approaches motivate grounding visual errors in relevant instruction variables and maintaining task-specific constraints.

Agentic Segmentation

  • SegAgent [33] learns multi-step segmentation policies from human annotation trajectories, while RSAgent combines multi-turn tool use with supervised fine-tuning and reinforcement learning.These approaches extend mask prediction through sequential reasoning, tool interaction, and feedback-driven correction.Source: §2 · p. 2
  • Refer-Agent [11] uses collaborative reasoning and reflection for referring video segmentation, while guideline-consistent segmentation [23] uses Worker–Supervisor feedback and adaptive termination.These approaches advance segmentation through agent coordination and iterative mask evaluation.Source: §2 · p. 2

Textual Optimization

  • ProTeGi [19], OPRO [27], and TextGrad optimize instructions through natural-language gradients, evaluation-guided search, or language-based backpropagation.Applying textual optimization to segmentation additionally requires grounding visual errors in relevant instruction variables and maintaining task-specific constraints.Source: §2 · p. 2

3 Methodology

GarmentSegAgent formulates garment extraction as iterative, constrained test-time refinement, combining semantic planning, mask generation, and visual evaluation. CAST uses diagnosed errors to selectively revise instructions for another segmentation pass while foundation models remain frozen.

3.1 Problem Formulation and Framework Overview

  • The framework treats garment extraction as iterative test-time optimization over instructions constrained by a garment contract, with a finite inference budget.The instruction comprises target, localization, coverage, and exclusion variables.Source: §3.1 · pp. 2–3, §3.2 · p. 3

3.2 Garment-Aware Planning and Spatial Execution

  • The Planner converts the image, instruction, and garment contract into a semantic plan, which the Actor translates into spatial prompts and candidate masks for evaluation.The Planner uses Qwen2.5-VL, and the Actor generates candidates for SAM 2.1.Source: §3.2 · pp. 3–4

3.3 Visual Error Attribution

  • Visual Error Attribution diagnoses target mismatch, missing regions, leakage, and boundary errors from the image and predicted masks without ground-truth annotations.The resulting feedback identifies affected regions, severity, confidence, and suggested corrections for CAST.Source: §3.3 · p. 4

3.4 CAST: Constrained Attribution-Guided Selective TextGrad

  • CAST routes visual errors to relevant instruction fields, applies TextGrad only to those fields, and validates candidate updates against the garment contract.Noncompliant updates are repaired when feasible or rolled back, while unselected fields remain unchanged.Source: §3.4 · p. 4

3.5 Iterative Decision and Termination

  • The Evaluator selects a candidate mask and assesses it against garment constraints, accepting it or triggering another refinement pass until acceptance or the iteration budget is exhausted.If no mask is accepted, the final candidate is returned.Source: §3.5 · p. 5

4 Experiments

Experiments evaluate autonomous upper-garment segmentation on three fashion benchmarks against spatial-prompt, vision-language, and agentic baselines. Full CAST leads in mIoU and Boundary IoU across benchmarks, with additional inference overhead.

4.1.1 Datasets and Evaluation Protocol; 4.1.2 Baselines

  • The evaluation uses 500 images per benchmark for binary upper-garment masks without human spatial prompts or online ground truth, comparing spatial-prompt, vision-language, and agentic methods.The benchmarks cover diverse garment appearances, viewpoints, occlusions, cluttered backgrounds, and fine-grained annotations.Source: §4.1.1 · p. 5, §4.1.2 · p. 5

4.1.3 Implementation Details; 4.1.4 Evaluation Metrics

  • Qwen2.5-VL plans, SAM 2.1 generates candidate masks, and Qwen3-VL evaluates; frozen models run up to five iterations, assessed with region, boundary, precision/recall, and refinement-cost metrics.Early termination follows evaluator acceptance; mIoU and Dice measure region agreement, while Boundary IoU measures boundary alignment.Source: §4.1.3 · p. 5, §4.1.4 · p. 5

4.2 Main Results

  • GarmentSegAgent beats SegZero+SAM 2 on DeepFashion2 mIoU (0.8124; +1.65 points), ModaNet mIoU (0.6733 vs. 0.6683), and Fashionpedia mIoU (0.7685; +2.59 points) and Boundary IoU (+2.88 points).Table 1 compares region overlap, boundary accuracy, and foreground prediction quality across all three benchmarks.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 Textual Optimization Strategies

  • Full CAST achieves the highest mIoU and Boundary IoU on every benchmark, gaining 2.88, 2.79, and 3.46 mIoU points over Selective TextGrad; Vanilla TextGrad lowers ModaNet mIoU from 0.6145 to 0.6001.The results support combining error-conditioned field selection with garment-contract validation.Source: Table 2 · p. 6, §4.3 · p. 7

4.4 Self-Refinement Analysis

  • On DeepFashion2 and Fashionpedia, both Vanilla TextGrad and CAST improve through iterative refinement, with CAST achieving larger gains; Figure 3 compares mIoU across iterations under identical initial predictions.Predictions from early-terminated samples are retained for subsequent iterations.Source: §4.4 · p. 7, Figure 3 · p. 7
  • On ModaNet, CAST raises the improvement rate from 43.8% to 79.6% and lowers the degradation rate from 48.2% to 12.0%, while adding 1.5–1.7 language-model calls and 2.7–3.3 seconds.The authors attribute the overhead to error-attributed refinement and garment-constrained validation.Source: §4.4 · pp. 7–8, Table 3 · p. 7

4.5 Ablation Studies

  • Removing visual error attribution or selective routing lowers mIoU and Boundary IoU, while removing garment-contract validation raises the degradation rate from 6.4% to 18.0%.The ablation results indicate complementary contributions from the three CAST mechanisms.Source: §4.5 · p. 8, Table 4 · p. 8

4.6 Qualitative Comparison

  • Across target ambiguity, occlusion, ambiguous boundaries, and multiple garments, GarmentSegAgent better preserves target regions and reduces leakage into adjacent clothing and backgrounds.Figure 4 compares input, ground truth, and four methods; ground truth is used only for evaluation.Source: §4.6 · p. 8, Figure 4 · p. 8

5 Limitations

The evaluation is limited to single-target upper-garment segmentation in Web fashion images. Broader garment categories and multi-instance scenarios remain unexplored, and refinement has accuracy and inference-cost constraints.

5 Limitations

  • Evaluation covers single-target upper-garment segmentation; broader garment categories and multi-instance scenarios remain unexplored.Source: §5 · p. 8
  • CAST relies on visual feedback and candidate masks from frozen foundation models, while garment-constrained validation cannot guarantee pixel-level accuracy.Source: §5 · p. 8
  • Iterative refinement introduces additional inference overhead.Source: §5 · p. 8

6 Conclusion

GarmentSegAgent uses CAST to iteratively improve garment segmentation without human-provided spatial prompts or fine-tuning. Across three datasets, it improves region overlap and boundary accuracy over competitive baselines.

6 Conclusion

  • Experiments on DeepFashion2, ModaNet, and Fashionpedia show consistent improvements in region overlap and boundary accuracy over competitive baselines.Source: §6 · p. 9
  • CAST combines visual error attribution, selective textual optimization, and garment-constrained validation to iteratively improve segmentation.Source: §6 · p. 9
  • GarmentSegAgent operates without human-provided spatial prompts or fine-tuning.Source: §6 · p. 9

A Target Selection and Evaluation Protocol

The protocol selects one consistent upper-body garment target per image and evaluates predictions against that instance. Reference annotations are reserved for offline matching and metric computation.

A Target Selection and Evaluation Protocol

B Additional Experimental Results

Additional experiments report cross-domain person-segmentation results on COCO and compare shop with consumer images in DeepFashion2. CAST improves on conventional TextGrad across both DeepFashion2 image sources, while consumer images are harder for all methods.

Cross-domain Person Segmentation on COCO

Additional qualitative results

Performance across Fashion Image Sources

C CAST Mechanism Analysis

CAST’s refinement mechanism is examined through a trace of attributed errors and instruction updates, with IoU assessed offline. Garment-contract validation outcomes are also reported across three benchmarks.

Prompt Evolution Case Study

  • Table 7 traces attributed visual errors, selected instruction fields, textual-gradient feedback, and instructions before and after optimization, with IoU measured offline.The trace also examines whether unselected fields remain unchanged and how updates are handled by garment-contract validation.Source: C CAST Mechanism Analysis · p. 10, Table 7 · p. 10

Garment-Contract Validation

  • 26.0–31.8% of proposed updates require repair, no rollbacks are recorded, and independently assessed constraint violations remain at 0.2–0.8%.The validation results indicate that instruction consistency is supported but not perfect.Source: C CAST Mechanism Analysis · p. 10

Qualitative Refinement

  • Figure 5 places the input image, ground truth, and final prediction above predicted masks from iterations 0–4; ground truth is used only for offline comparison.Source: Figure 5 · p. 11

Qualitative Comparison

  • Figure 6 compares person segmentation on COCO under target ambiguity, occlusion, and ambiguous boundaries.Source: Figure 6 · p. 11

D Case Study of Iterative Prompt Evolution

A five-iteration Fashionpedia example illustrates how CAST revises instructions from initial garment localization through boundary refinement. The prompts are reconstructed illustrations, not verified execution records.

Case Study

  • The Fashionpedia example follows five iterations from initial garment localization to boundary refinement, using prompts reconstructed from masks and the CAST protocol.The original textual logs were not retained.Source: D Case Study of Iterative Prompt… · p. 11

Initial Garment Specification

Iterative Prompt Evolution

  • Early updates address background leakage and missing garment regions; later updates tighten contours and recover cuff and hem regions while suppressing residual leakage.Source: D Case Study of Iterative Prompt… · p. 12

Analysis

  • The target instruction remains unchanged while localization, coverage, and exclusion are selectively revised, and candidate instructions are checked for task consistency before execution.The reconstructed prompts illustrate intended interactions rather than verified execution logs.Source: D Case Study of Iterative Prompt… · pp. 11–12

E Theoretical Analysis of CAST

CAST’s theoretical analysis characterizes selective instruction updates and contract preservation under stated assumptions. These are instruction-level guarantees, not guarantees about segmentation accuracy.

Selective Update Locality

  • Only error-attributed fields in K_t may change when selective updates and validation or repair preserve inactive fields.The update-locality claim is conditional on that field-preservation assumption.Source: E Theoretical Analysis of CAST · p. 12

Garment-Contract Feasibility

  • Contract preservation follows if the initial instruction satisfies the garment contract, validation is sound, and unsuccessful updates roll back to the prior instruction.Under these assumptions, accepted updates remain within the contract, while unsuccessful ones restore the previous instruction.Source: E Theoretical Analysis of CAST · p. 12

Scope of the Guarantees

  • The propositions constrain instruction changes but neither bound mask variation nor guarantee improved IoU, and feasibility depends on validator soundness.Language-model-based validation may be imperfect.Source: E Theoretical Analysis of CAST · p. 12

Related papers

Loading 2610.36506v1…