SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories

1. Introduction

• VQA and visual grounding do not effectively assess fine-grained pixel comprehension, while pixel-precision tasks such as UI operations and robotic control are constrained to specialized environments.The paper identifies segmentation as a potential open-world route for studying pixel-level understanding, but says existing MLLM approaches often rely on implicit tokens and extra pixel decoders.Source: §1 · p. 1
• HLMAT models segmentation as a multi-step decision process in which MLLMs imitate human annotators through interactive tools and text-based coordinate outputs.It is proposed as a protocol for assessing fine-grained visual capabilities without specialized architectures or implicit tokens.Source: §1 · p. 2
• SegAgent achieves performance comparable to state-of-the-art methods and supports segmentation, mask refinement, and annotation filtering.The model is fine-tuned on human-like annotation trajectories generated from existing datasets.Source: §1 · p. 2
• The authors find commonly used referring segmentation datasets insufficient in annotation complexity and quality for multi-step decision-making, and introduce high-quality reference segmentation datasets.The introduction identifies HRES as suitable for complex reasoning.Source: §1 · p. 2
• StaR-style policy improvement significantly improves test performance over relying solely on supervised fine-tuning data, while PRM-guided tree search mitigates inaccurate-prediction errors in complex scenes.The paper presents these training- and inference-stage adaptations as ways to improve segmentation performance and robustness.Source: §1 · p. 2

2.1. Pixel Understanding in MLLMs

• Prior MLLM segmentation methods use implicit tokens with separate pixel decoders or text-based polygon and point outputs, which the authors describe as limiting precision or altering the output space.SegAgent instead imitates human annotation trajectories using interactive segmentation tools and text-based outputs.Source: §2.1 · p. 2

2.2. MLLMs as Decision-Making Agents

• Visual-domain decision-making research remains limited; prior work includes non-interactive tasks, simple UI operations, and poker-like tasks that are primarily text-based.The paper positions HLMAT as a task involving fine-grained visual perception in multi-step decision-making.Source: §2.2 · p. 3

2.3. Interactive Segmentation Tools

• Interactive segmentation models can generate accurate masks but require human input, whereas this work aims to train MLLMs to imitate annotators using those tools.The related work describes RITM, SimpleClick, InterFormer, and SAM as examples of interactive segmentation systems.Source: §2.3 · p. 3

3. Preliminaries and Task Formulation

• An annotator iteratively provides positive or negative click points, and an interactive segmentation network uses each click with the image and current state to produce the next mask.The state contains the current mask and action history; the action includes a click attribute and normalized image coordinates.Source: §3 · p. 3
• The reward evaluates each current mask against the target mask using Intersection over Union, with higher IoU indicating a better match.The task omits the discount factor because it focuses on final-mask quality.Source: §3 · p. 3
• HLMAT trains a policy to output an action at each step from the image, prompt, and current mask, maximizing expected reward within a limited number of steps.For flexible uses such as mask refinement, the model conditions on the current mask rather than the full action history.Source: §3 · p. 3

4. SegAgent

• SegAgent conditions on the current mask, image, and prompt, then outputs text actions that an interactive segmentation model uses to update the mask.The current mask is overlaid on the original image; the model can act from the current mask without depending on the full action history.Source: §4 · p. 4

4.1. Human Annotator Trajectories Generation

• Simulated clicks use false-positive and false-negative regions to generate trajectories, which are filtered by length, reward thresholds, and action impact before fine-tuning.These controls address noisy target masks that can produce misleading actions, especially late in a trajectory.Source: §4.1 · pp. 4–5

4.2. What Makes a Good SegAgent?

• A strong SegAgent needs both initial text-guided localization and later fine-grained mask refinement; the latter motivates using complex, high-quality data such as DIS5K and ThinObject5K.The section describes the commonly used RefCOCO datasets as lacking the complexity and annotation quality needed for long, accurate trajectories.Source: §4.2 · p. 5

4.3. Policy Improment

• StaR+ iteratively rolls out the current model, keeps reward-improving actions, corrects reward-decreasing actions with simulated actions, and fine-tunes on combined trajectory data.The method evaluates reward change for individual actions rather than filtering only by overall trajectory correctness.Source: §4.3 · p. 5

4.4. Process Reward Model and Tree Search

• At inference, PRM predicts step rewards to guide stopping and greedy selection among candidate actions; the simplest greedy search improves performance, while more complex search trades off computation and performance.The method returns the mask with the highest predicted reward and stops when the step limit is reached or reward converges.Source: §4.4 · p. 6

5.1. Evaluation Protocol

• Evaluation covers RES and HRES, using cumulative IoU (cIoU) with a fixed maximum step count; RES uses T=7 and K=1, while HRES uses T=11 and K=3.Source: §5.1 · p. 7

5.2. Datasets

• HRES uses higher-quality, more detailed and complex masks; it requires more clicks than RES to reach the target IoU.It draws on DIS5K and ThinObject5K, including high-resolution images and challenging thin structures.Source: §5.2 · pp. 7–8

5.3. Main Results

• SegAgent achieves competitive RES IoU scores; SegAgent-LLaVA+SAM and SegAgent-Qwen+SClick outperform all compared methods except GLaMM.Table 1 compares methods on RES and indicates which use SAM; SClick denotes SimpleClick.Source: Table 1 · p. 7, §5.3 · p. 8
• With the same Qwen-derived initial box, SegAgent-LLaVA(Qwen)+SAM outperforms the Qwen box + SAM baseline across all datasets.The authors interpret this comparison as evidence of mask refinement.Source: §5.3 · p. 8
• Although Qwen generally performs better on standard vision-language tasks, LLaVA surpasses it in mask refinement capability.The authors attribute this to Qwen’s Q-former structure and cite prior reports of possible spatial-locality loss during semantic abstraction [5] [60].Source: §5.3 · p. 8

5.4. Exploration on More Complex Scenarios

• PRM alone adds 2.36 on DIS and 4.09 on ThinObject; K=3 Tree Search adds a further 7.43 on DIS and 10.59 on ThinObject.A single round of StaR+ raises ThinObject5K IoU to 86.57 from the baseline’s 71.45.Source: §5.4 · p. 8

• HLMAT models MLLMs as multi-step decision-making agents that imitate human annotation paths in interactive segmentation.Source: §6 · pp. 8–9
• SegAgent achieves competitive performance across segmentation tasks, including mask refinement and annotation filtering.Source: §6 · p. 9
• The paper introduces the HRES dataset and reports that StaR+ and PRM-guided tree search can further boost performance.Source: §6 · p. 9

A.1. Computational complexity Trade-off

• Fixed-length greedy decoding over longer sequences may worsen results through error accumulation.Source: §A.1 · p. 13
• The authors state that PRM can effectively alleviate the potential error-accumulation issue.Source: §A.1 · p. 13
• Table 2 compares performance under different numbers of steps.Source: Table 2 · p. 13

A.2. Limitations

• Failures arise from incorrect initial target localization and inaccurate refinement coordinates, especially boundary clicks that land in the background and produce the wrong mask.Source: §A.2 · p. 13

B.1. Trace Generation

• Trajectory generation supports initial states beyond empty masks, including masks from bounding boxes and randomly sampled points.Source: §B.1 · p. 13

B.2. Comparision with StaR

• StaR+ fine-tunes on merged original and newly generated trajectory datasets because the original trajectories retain useful approximately optimal information.Source: §B.2 · p. 13

B.3. More Details about Process Reward Model and Tree Search

• The supplementary material further illustrates PRM-guided tree search as pseudocode in Figure 5.Source: §B.3 · p. 14

C.1. Model Architecture and Hyperparameters

• SegAgent-LLaVA uses a ConvNeXt-L CLIP vision encoder and is fine-tuned for two epochs with 512 × 512 inputs and batch size 16.Source: §C.1 · p. 14
• SegAgent-QWen freezes the ViT module during full-parameter fine-tuning, using 448 × 448 inputs, 256 adapter queries, and batch size 128 for two epochs.Source: §C.1 · p. 14

C.2. Prompt Design

• In the prompt design, positive points expand the mask and negative points shrink it.Source: §C.2 · p. 14

D.1. Comparison of Dataset Quality

• ThinObject5k-TE and DIS5K annotations are more complex and precise, while RefCOCO masks show mislabeling, edge omissions, and difficulty with hollow regions.The two datasets are described as higher quality for evaluating and exploring SegAgent’s multi-step mask refinement.Source: §D.1 · p. 14

D.2. Visualization of Predicted Trajectories

• SegAgent refines masks step by step through positive and negative points using SAM or SimpleClick, demonstrating learned annotation rules and object understanding.The visualization excludes PRM and tree search; SAM examples start with Qwen-box, while SimpleClick examples use clicks as actions.Source: §D.2 · p. 14

E.1. Annotation Filtering

• SegAgent-LLaVA outperforms SegAgent-Qwen across all evaluated annotation-filtering metrics, while the PRM’s predictions correlate highly with mask mIoU.Filtering is modeled as regression to predict the current mask’s IoU with ground truth; a threshold can filter low-quality masks.Source: §E.1 · pp. 14–16, Table 3 · p. 15

E.2. Mask color

• Green, blue, and red masks yield nearly identical mIoU, suggesting mask color has minimal effect on segmentation performance.The comparison is reported in Table 4.Source: §E.2 · p. 16, Table 4 · p. 16

E.3. Init Action

• Initial action has minimal impact on segmentation; Qwen Box performs slightly better than NO box and Self Box.The authors selected Qwen Box as SAM’s initial action for a fair comparison.Source: §E.3 · p. 18, Table 5 · p. 18

E.4. Coordinate Format

• Integer relative positions in [0, 1000) and decimals in [0, 1) yield nearly identical segmentation performance for SegAgent-LLaVA.The authors selected decimals in [0, 1) for SegAgent-LLaVA.Source: §E.4 · p. 18, Table 6 · p. 18