Source-linked AI summary
InsightSeg: Reusing Correction Insights for Guideline-Consistent Segmentation
Vanshika Vats, Ashwani Rathee, James Davis
TL;DR
Guideline-consistent segmentation demands task-specific decisions that single-pass and stateless refinement systems handle inefficiently because prior corrections are discarded. InsightSeg stores successful corrections as visually grounded episodic insights and retrieves them from local patch matches before initial prediction. Across Waymo and Cityscapes, it improves segmentation quality while reducing refinement effort, though longer deployments may accumulate stale or redundant insights.
Problem
Guideline-consistent segmentation requires fine-grained task-specific decisions, while recurring corrections are repeatedly rediscovered because refinement feedback is discarded.
Method
InsightSeg converts successful correction episodes into directive natural-language insights paired with patch-level visual anchors, then retrieves relevant insights before future images’ first predictions.
Results
InsightSeg improves guideline-consistent segmentation on Waymo and Cityscapes while reducing refinement iterations by 62% and 45%, respectively.
Takeaways & Limitations
Inference-time correction can accumulate and reuse localized experience across images, shifting refinement from repeatedly correcting errors toward preventing them earlier.
Takeaways & Limitations
The online memory requires encountering a related case before reuse and may accumulate redundant or stale insights as visual conditions and labeling policies evolve.
Abstract
from arXiv · showhide
Guideline-consistent semantic segmentation requires more than category recognition, as real-world labeling policies demand fine-grained, task-specific decisions. Recent multi-agent refinement systems improve compliance with such textual guidelines by detecting and correcting errors. However, they are stateless: feedback from the critiquing agent is discarded, causing the same guideline-specific mistakes to be repeatedly rediscovered and corrected across the dataset at the cost of additional refinement. We introduce InsightSeg, an episodic memory mechanism that converts successful correction episodes into reusable, visually grounded insights. A meta-analyzer distills each qualifying episode into directive natural-language insights and anchors them to the local image regions that caused the error using patch-level visual concept vectors. On subsequent images, these concepts are matched against dense patch embeddings to retrieve relevant insights, which condition the segmenting agent before making its first prediction. This shifts the system from correcting recurring errors to preventing them, improving segmentation quality before any refinement occurs. Across Waymo and Cityscapes, InsightSeg improves both first-pass and final guideline-consistent segmentation performance while requiring fewer refinement steps, demonstrating that multi-agent refinement can become more accurate and efficient by drawing on past correction experience.
Introduction
Guideline-consistent segmentation requires task-specific policy decisions that single-pass systems cannot reliably verify, while refinement systems repeatedly rediscover similar corrections. InsightSeg addresses this with visually grounded episodic memory that retrieves prior correction insights before prediction, improving quality and reducing refinement.
- Motivation: Guideline-consistent segmentation requires fine-grained labeling decisions beyond recognizing semantic categories.Examples include excluding bicycle riders while including pedestrians and carried umbrellas in the same mask.
- Motivation: Single-pass systems lack a mechanism to detect or correct recurring guideline-specific mistakes.Because visual patterns recur across images, similar errors can recur as well.
- Approach: InsightSeg converts successful correction episodes into reusable, visually grounded insights linked to the local image evidence that caused each error.A meta-analyzer produces directive natural-language insights and patch-level visual anchors.
- Approach: The system retrieves relevant stored concepts from dense patch embeddings and supplies the insights to the segmenting agent before its first prediction.This enables prior correction experience to influence initial segmentation rather than only later refinement.
- Results: Refinement iterations decrease by 62% on Waymo and 45% on Cityscapes while final segmentation performance improves.The process is online and training-free, with no gradient updates.
- Results: The stateful memory mechanism enables cross-image knowledge transfer, preventing recurring errors and reducing repeated refinement.It accumulates inference-time correction experience across images.
Related Works
Promptable and multi-stage segmentation methods improve visual grounding but remain challenged by intricate labeling policies and costly repeated evaluation. InsightSeg extends memory-based approaches with local patch-level visual retrieval so corrections tied to small visual triggers can transfer across images.
- Promptable Segmentation: Promptable segmentation conditions masks on spatial or language cues but often struggles with long, intricate labeling policies.Such policies require multiple task-specific decisions beyond concise text prompting.
- Iterative Refinement: Multi-stage generate-evaluate-revise methods improve reliability by inspecting and revising intermediate predictions, but often require multiple model evaluations.Dedicated reviewer agents can identify errors and provide targeted feedback.
- Memory-Based Methods: Existing non-parametric memories generally retrieve entries through textual similarity, which is insufficient when corrections depend on localized visual cues.Relevant triggers may include reflections, carried umbrellas, or figures on posters.
- Memory-Based Methods: Whole-image similarity and task-observation memories may miss small localized triggers, while related patch correspondence methods store only positive information.These limitations motivate memory that preserves correction insights together with their visual evidence.
- InsightSeg: InsightSeg pairs directive correction insights with patch-level concept vectors and retrieves them when the same visual trigger appears anywhere in a new image.Retrieval operates over local patches rather than only text or whole-image similarity.
Method
InsightSeg adds a stateful, training-free episodic memory to guideline-consistent segmentation, storing transferable correction insights with visual anchors and retrieving them before the Worker’s first prediction. Qualifying episodes are filtered, distilled, visually grounded, and matched to local regions in later images.
- Learning-event qualification: InsightSeg stores only learning events whose issue trajectories show non-trivial, stable improvement, discarding episodes with weak or misleading correction signals.Admission requires at least one unit of total issue reduction and limits single-step regressions; episodes with clean initial critiques bypass the gate.
- Insight construction: The Meta-Analyzer converts validated missing-object and false-positive corrections into generalizable textual insights linked to existing Supervisor-provided support boxes.Boundary refinements are excluded because they reflect instance-specific geometry rather than transferable scene knowledge.
- Visual grounding: Each insight is grounded in local DINOv3 patch embeddings, selecting patches overlapping support boxes by at least 30% and mean-pooling them into a normalized concept vector.The overlap threshold preserves surrounding context, while multiple support regions are pooled to represent a stable trigger concept.
- Memory storage: The memory stores unique insights with one or more visual anchors, retaining up to Ka = 5 diverse anchors and using the CLS token only to maintain anchor diversity.Retrieval uses patch-level concept vectors, while text-level deduplication and visual diversity prevent redundant memory growth.
- Memory retrieval: Before the Worker’s initial pass, query-image patch tokens are compared with stored concept vectors so locally matching insights can be retrieved and injected into the prompt.Similarity is computed over patches rather than the whole scene, allowing an insight to fire when any local region matches its visual concept.
Experiments
Experiments evaluate InsightSeg on Waymo and Cityscapes using fixed retrieval, deduplication, and refinement settings. Reusable visual memory improves first-pass and final segmentation, outperforms stateless or global-retrieval alternatives, and reduces refinement effort.
- Evaluation Setup: InsightSeg is evaluated on 102 curated Waymo samples and 500 Cityscapes validation images.Waymo annotations follow its labeling guidelines, while Cityscapes tests robustness beyond the curated benchmark.
- Main Results: 83.51(±0.26) gIoU and 90.21(±0.45) mDice on Waymo outperform GuideSeg by +2.94 and +3.01, respectively.The method also gains +3.38 precision and +2.77 recall over the refinement baseline.
- Main Results: On Cityscapes, InsightSeg improves over GuideSeg by +2.46 gIoU, +4.08 cIoU, and +8.17 mPr.The benchmark is challenging because many object instances are small and distant.
- Memory Ablation: Before refinement on Waymo, reusable insights improve the no-memory baseline by +5.07 gIoU and +3.65 mDice.The paired-bootstrap 95% confidence interval for first-pass gIoU is [+1.3, +5.9].
- Efficiency: Refinement iterations decrease by 62% on Waymo, while deduplication merges 50% of 48 generated insights.The memory-guided system reaches higher final quality with fewer refinement calls.
- Retrieval Ablation: Patch-based retrieval improves all segmentation metrics, whereas global CLS retrieval increases precision but degrades gIoU, cIoU, recall, and Dice relative to the stateless baseline.Global similarity captures broad scene appearance but can miss localized guideline-relevant visual cues.
Conclusion
InsightSeg converts successful correction episodes into reusable, visually grounded insights for training-free guideline-consistent segmentation. Across Waymo and Cityscapes, this memory improves segmentation quality while reducing refinement passes and shifting refinement toward proactive error avoidance.
- Conclusion: InsightSeg stores directive correction insights linked to patch-level visual concept vectors from regions that caused prior errors.Relevant lessons condition the segmenting agent before its first prediction on future images.
- Conclusion: Experiments on Waymo and Cityscapes show improved segmentation quality with fewer refinement passes.The broader result is cross-image reuse of localized correction experience in a training-free setting.
- Conclusion: The system shifts multi-agent refinement from repeatedly rediscovering errors toward a persistent process that is proactive, more accurate, and more efficient.
Supplementary Material
The supplementary material details the fine-grained labeling policies used for Waymo and Cityscapes and reports qualitative and quantitative evidence for InsightSeg on both datasets.
- Guidelines: Waymo guidelines require decisions beyond basic category recognition, including rules for partial visibility, uncertain pedestrians, vehicles, and carried objects.They also distinguish cyclists, reflections, mannequins, posters, and other policy-specific cases.
- Guidelines: Cityscapes defines its Person class by whether a human would walk after moving one meter, including carried items but excluding ground-touching objects.
- Results: On Cityscapes, reusable insights improve segmentation before and after refinement across gIoU, cIoU, precision, and Dice while reducing average iterations from 2.33 to 1.28.Absolute recall remains lower because many pedestrian instances are very small and distant.
- Qualitative results: Qualitative results across Waymo and Cityscapes show masks that adhere more closely to labeling guidelines and better match ground truth than the other evaluated methods.The figure marks compared-method errors in red and challenging regions correctly handled by InsightSeg in blue.
Sec. C: Qualitative Segmentation Results
The qualitative and order-sensitivity analyses show that patch-based memory localizes guideline-relevant cues, remains useful across processing orders, and improves first-pass and final segmentation across tested VLMs.
- Processing order: Similar final results and identical average refinement iterations under original and randomly shuffled orders indicate that memory benefits are not order-dependent.Both settings report an average of 1.01 refinement iterations.
- Retrieval comparison: Patch retrieval focuses on local guideline cues such as umbrellas and cyclists, whereas global CLS retrieval is dominated by overall street appearance.Global retrieval repeatedly returns the same memory image for queries with different local triggers.
- Base VLM: Swapping the base VLM preserves the memory advantage: patch-based memory improves both first-pass and final segmentation with gemini-2.5-flash and gemini-3-flash-preview.
Sec. E: Qualitative Examples of Correction Reuse
Correction-reuse examples illustrate how InsightSeg converts guideline violations into anchored insights and retrieves them later to support correct first-pass masks, while also exposing retrieval failures.
- Correction reuse: A Supervisor identifies a guideline violation, a Meta-Analyzer creates an insight anchored to support patches, and later patch retrieval enables the Worker’s correct first-pass mask.Examples include carried backpacks, people depicted on traffic signs, and cyclists excluded from pedestrian masks.
- Failure cases: Failure cases arise when a small distant cyclist does not match stored anchors or when poster depictions resemble stored bag anchors.
- Ablation: Patch-based memory improves overall first-pass and post-refinement segmentation with both tested base VLMs.
Sec. F: Estimated Costs
InsightSeg reduces refinement-related API usage because memory operations run locally and fewer Worker-Supervisor iterations are needed.
- API costs: Each Worker or Supervisor call costs about $0.0011 at approximately 2,000 input and 200 output tokens.A complete Worker-Supervisor iteration requires three API calls.
- Memory costs: Memory insertion and retrieval use local FAISS operations with no additional API cost, while serialized memory banks occupy 869 KB for Waymo and 1.7 MB for Cityscapes.
Sec. G: Examples of Failure Cases
InsightSeg’s local matching can fail when visual triggers are too small or when similar patches lack sufficient scene context. Its online memory also has deployment-boundary issues, although explicit records remain inspectable for auditing and management.
- Scale and context failures: Scale changes can prevent an applicable insight from matching a small, distant cyclist to its stored visual anchors.The limitation suggests multi-scale representations as a possible remedy.
- Scale and context failures: Visually similar patches, such as people or bags on a poster, can retrieve stored anchors without corresponding to physical pedestrians in the scene.Semantic verification of retrieved cues could help supply the missing scene context.
- Online-memory boundaries: InsightSeg can reuse corrections only after encountering a related case, and longer deployments may accumulate redundant or stale insights as conditions and policies evolve.Text-level deduplication does not address stale insights; every reported run begins with an empty memory bank.
- Online-memory boundaries: Because insights are stored as explicit natural-language records rather than model weights, the memory bank remains inspectable and auditable.This supports future confidence tracking, manual auditing, and selective consolidation or retirement without retraining.