CoEvolve: Construct-to-Edit Visual Grounding with Bidirectional State Refinement

• Most approaches compress target disambiguation, spatial search, and boundary estimation into one terminal box, leaving intermediate spatial decisions difficult to inspect and repair.Without an explicit spatial state, region-selection and boundary-estimation errors can persist into the final prediction.Source: §1 · p. 2, Figure 1 · p. 2
• RER makes each reasoning step commit to an explicit candidate region, creating a measurable state that can be passed to an editor and assessed against the observed state.The motivating question is how to preserve accurate regions while evaluating coordinate changes.Source: §1 · p. 2
• CoEvolve separates grounding into explicit state construction and state editing, using RER to form a progressive semantic–spatial trajectory and BDR to refine its coordinate fields.Geometry-level supervision supplies target geometry, while behavior-level supervision encodes preferences for consolidating reliable candidates, preserving accurate inputs, or correcting toward annotations.Source: §1 · p. 2
• 92.41% Overall Acc@0.5 across eight RefCOCO-family splits and 85.72% mIoU across five test splits; under controlled corruption, BDR raises mIoU from 58.80% to 86.19%.BDR also reaches 87.99% Repair@0.7; domain-specific retraining reaches 76.23% mIoU on DIOR-RSVG.Source: §1 · p. 2
• The paper evaluates localization across natural-image and remote-sensing benchmarks and tests correction under controlled box corruption.Its stated contributions include substantial correction gains under controlled corruption.Source: §1 · p. 2

2.1 GROUP RELATIVE POLICY OPTIMIZATION

• GRPO removes the learned critic by sampling a group of responses, and its clipping configuration can permit broader exploration for positively advantaged responses.Equal lower and upper clipping radii recover symmetric clipping.Source: §2.1 · p. 3

2.2 BIDIRECTIONAL DENOISING REFINEMENT

• BDR reconstructs original tokens at selected response positions from a discretely corrupted response, then can update selected positions over multiple passes with a position-aligned predictor.The predicted token distribution is defined at the corresponding response position.Source: §2.2 · p. 3
• The refinement formulation operates on discrete response tokens, not a continuous noise-time process or a full reverse-time Markov chain.For CoEvolve, the initial response is the RER localization trajectory; BDR also handles visible and geometrically corrupted coordinate tokens, with token masking used only during warm-up.Source: §2.2 · p. 3

3.1 OVERVIEW

• CoEvolve passes RER’s progressive semantic–spatial response to BDR, which edits coordinate fields while retaining reasoning text as context.The response serves as an editable localization state.Source: §3.1 · p. 4, Figure 2 · p. 4, §3.3 · p. 5

3.2 REGION-EVOLUTION REINFORCEMENT

• RER produces a tagged response with 3–5 refinement steps, each paired with a candidate box, and a terminal answer; rewards assess spatial commitments against the terminal annotation.It requires no manually labeled reasoning text or intermediate box trajectory.Source: §3.2 · pp. 4–5

3.3 BIDIRECTIONAL DENOISING REFINER

• BDR edits coordinate positions synchronously from the observed response, producing position-aligned logits and digit-projected box candidates while retaining the reasoning text.Its same-position reconstruction uses bidirectional access to the response.Source: §3.3 · p. 5
• Quality-gated target selection reinforces an improved candidate, preserves an already high-quality input, or otherwise uses the annotation as the corrective target.The selected target supplies both geometric supervision and a training preference for the current state.Source: §3.3 · p. 5
• Geometry supervision gives the last refinement box and answer the same selected target, while coordinate-wise preference supervision learns the relative utility of an edit versus its input.Preference supervision distinguishes edits closer to the annotation from detrimental edits.Source: §3.3 · pp. 5–6
• Reconstruction and format losses preserve non-coordinate content and response structure, while entropy terms sharpen historical distributions and correct final-digit predictions.The final entropy term sharpens confidence only when the predicted digit matches the selected target.Source: §3.3 · p. 6

4.1 EXPERIMENTAL SETUP

• Evaluation covers eight RefCOCO-family splits and DIOR-RSVG, with stricter-IoU, correction, and ablation experiments using five test splits and jointly aggregated metrics.Natural-image evaluation uses 57,457 examples; the five-test-split experiments comprise 30,969 examples.Source: §4 · p. 6, §4.1 · p. 7

4.2 STANDARD REFCOCO-FAMILY GROUNDING

• 92.41 Overall Acc@0.5 is the highest among compared systems; CoEvolve exceeds Qwen3.5-9B on all eight splits and improves its aggregate score by 2.93 points.The evaluation trains only on RefCOCOg train and includes RefCOCO and RefCOCO+ cross-dataset splits without further fine-tuning.Source: §4.2 · p. 7, Table 1 · p. 7, §4.1 · p. 7

4.3 LOCALIZATION UNDER STRICTER IOU CRITERIA

• 85.72% mIoU, 87.94% Acc@0.7, and 68.64% Acc@0.9 exceed CogVLM-Grounding-17B by 2.12, 2.63, and 4.19 points, respectively.CoEvolve ranks first in every reported setting, with Acc@0.9 gains ranging from 0.41 points on RefCOCOg test to 7.02 points on RefCOCO+ testB.Source: §4.3 · p. 7, Table 2 · p. 8

4.4 CONTROLLED BOX CORRECTION

• 86.19% mIoU and 87.99% Repair@0.7 follow one BDR pass; versus Qwen3.5-9B, CoEvolve BDR gains 2.34 mIoU, 5.46 Acc@0.9, and 8.11 Repair@0.7 points.Qwen3.5-9B has an 11.42% degradation rate versus 15.50% for CoEvolve BDR.Source: §4.4 · p. 8, Figure 3 · p. 8

4.5 STATE CONSTRUCTION AND SOURCE-MATCHED REFINEMENT

• 2.22-point mIoU gain separates CoEvolve from Zero-Anchor BDR, with gains widening from 2.19 points at Acc@0.5 to 3.68 at Acc@0.7 and 6.48 at Acc@0.9.The comparison tests RER states with anchor-aware objectives against a zero-anchor variant.Source: §4.5 · p. 8, Figure 4 · p. 9
• RER states rise from 80.70% to 85.72% mIoU after source-matched refinement, while native outputs rise from 82.16% to 84.57%; their respective gains are 5.02 and 2.41 points.At Acc@0.9, the respective gains are 8.83 and 5.68 points, yielding 68.64% and 66.32%.Source: §4.5 · p. 9, Table 3 · p. 9

4.6 CROSS-DOMAIN ADAPTATION AND DOWNSTREAM SEGMENTATION

• On DIOR-RSVG, domain-retrained CoEvolve achieves 86.20% Pr@0.5 and 76.23% mIoU, improving over native Qwen3.5-9B without DIOR-RSVG adaptation by 30.35 and 24.40 points.Table 4 compares task-specific methods and general-purpose MLLM baselines.Source: §4.6 · p. 9, Table 4 · p. 9
• 78.19 Macro-8 cIoU is achieved when CoEvolve predictions prompt frozen SAM3 across the eight RefCOCO-family splits, without mask fine-tuning.The paper reports the full split-wise comparison in Appendix D.6.Source: §4.6 · p. 9

5 CONCLUSION

• CoEvolve separates visual grounding into RER region commitments and BDR coordinate refinement, supervised by geometry-level and behavior-level objectives.The conclusion frames experiments across natural images, controlled perturbations, and remote-sensing scenes as support for this decomposition.Source: §5 · p. 10
• Refinement depends not only on initial localization quality but also on how the editable state is constructed and how its editor is trained.Source: §5 · p. 10

A APPENDIX CONTENTS

• The appendix lists supplementary methods, formal properties, additional experimental results, and limitations.Source: A APPENDIX CONTENTS · p. 15

B SUPPLEMENTARY METHOD DETAILS

• The appendix specifies the mathematical definitions of RER and BDR and distinguishes training-target quantities from model-visible quantities.Source: B SUPPLEMENTARY METHOD DETAILS · p. 16

B.1 NOTATION AND THE RER–BDR INTERFACE

• Boxes use normalized integer coordinates b = [x1, y1, x2, y2] in [0, 1000]^4, with ordered corners, and b* denotes the annotation.Source: §B.1 · p. 16
• RER outputs 3–5 reasoning-step and box pairs plus a terminal answer, and BDR receives the complete response, including text and coordinates, as its editable state.RER and BDR are optimized sequentially and communicate only through this generated state.Source: §B.1 · p. 16

B.2 SEQUENTIAL TRAINING AND BDR DATASET CONSTRUCTION

• RER is trained first, then frozen to generate complete responses paired with annotations for constructing the BDR training set.BDR starts from a separate adapter; its inputs retain natural RER errors and also include selected perturbed states, while annotations are used only for target and loss construction.Source: §B.2 · p. 16

B.3 REGION-EVOLUTION REINFORCEMENT

• RER optimizes a 3–5-step explicit-region trajectory using format, step-format, progressive spatial, and terminal-box rewards derived from annotations.Invalid structure or invalid step boxes can zero the progressive reward; human-written reasoning chains are not token targets.Source: §B.3 · pp. 16–17
• The progressive reward multiplies format reward by weighted coverage (0.20), progress (0.50), overall gain (0.20), and consistency (0.10); terminal box reward uses answer-box IoU and GIoU.Figure 5 shows training dynamics over the first 6K optimization steps, a window that does not indicate total training duration.Source: §B.3 · p. 17, Figure 5 · p. 18

B.4 BIDIRECTIONAL SAME-POSITION REFINEMENT AND TARGET CONSTRUCTION

• BDR removes causal masking so each coordinate prediction can use bidirectional context from the complete observed response, and decodes aligned coordinate candidates for editing.The observed coordinate tokens remain visible as references, while the reasoning text and response structure are retained.Source: §B.4 · p. 18
• For selected high-overlap RER answers, training adds synchronized geometric perturbations that create nonzero anchors with IoU in [0.2, 0.5] to train correction behavior.The answer, last refinement box, and sufficiently matching earlier steps are perturbed together; the annotation is never an inference input.Source: §B.4 · pp. 18–19
• Geometry supervision combines coordinate cross-entropy with continuous coordinate regression, and L1-derived weighting shifts gradient toward inaccurate coordinates.Cross-entropy retains discrete coordinate representation while regression supplies a continuous numerical direction.Source: §B.4 · p. 19
• Behavior-level preference supervision compares changed input and output coordinates against annotations, while confidence terms sharpen historical digits and only already-correct final digits.The phase-specific objective adds reconstruction, format, geometry, entropy, and preference terms after warm-up; Figure 6 plots these optimization dynamics.Source: §B.4 · p. 20, Figure 6 · p. 21

B.5 INFERENCE PROCEDURE

• Inference generates an RER state and applies two deterministic BDR passes, updating editable coordinate positions while retaining the surrounding reasoning text and structure.Unparseable final answer boxes remain in the evaluation denominator and receive zero IoU.Source: §B.5 · p. 21

B.6 PROMPT AND EVALUATION PROTOCOLS

• Figure 7 presents the standard-grounding, BDR-refinement, and controlled-correction prompts; RER and the structured base condition share a prompt and parsing rule but use different adapter settings.Source: §B.6 · p. 21, Figure 7 · p. 22
• Reproduced standard-grounding baselines use each model family’s native interface and produce one response per example, with one final normalized xyxy box evaluated.Invalid or unparseable outputs remain in the denominator with zero IoU.Source: §B.6 · p. 21, Table 6 · p. 21
• The Native-Output BDR condition turns native visible analysis and its terminal box into a state, then trains a separate editor to change Answer digits while preserving the analysis.The annotation is used only for target and loss construction, never as model input.Source: §B.6 · pp. 21–22
• Controlled-correction methods receive the same image, expression, and deterministic corrupted anchor; BDR edits the final-refinement and Answer coordinates while the sentence and structure stay fixed.No method receives the annotated box as input.Source: §B.6 · pp. 22–23
• Reported evaluation uses normalized coordinates clipped to [0, 1000], deterministic decoding for RER and the native baseline, and digit-vocabulary argmax for BDR.The implementation uses separate adapters and one RER generation followed by two deterministic BDR passes; invalid final boxes receive zero IoU.Source: §B.6 · p. 23, §B.7 · p. 23

C FORMAL PROPERTIES

• The progressive signal is bounded between 0 and 1, which keeps Rprog between zero and Rfmt.Source: C FORMAL PROPERTIES · p. 23
• Weighted box cross-entropy is invariant to common positive rescaling of coordinate weights, and preference-gradient updates increase beneficial-edit margins while decreasing detrimental-edit margins.Source: C FORMAL PROPERTIES · p. 23
• The final-coordinate entropy term cannot directly sharpen a digit whose argmax differs from the selected target digit.The proof states that incorrect positions are excluded from this entropy term and receive zero gradient from it.Source: C FORMAL PROPERTIES · p. 24

D.1 DATASET ACCOUNTING

• The archived localization evaluation uses eight standard splits; high-precision, controlled-correction, and ablation studies use the five test splits.Table 7 summarizes dataset composition used by the archived evaluations.Source: §D.1 · p. 24, Table 7 · p. 24
• 57,457 localization examples contribute to the RER aggregate, while the stricter-IoU, controlled-correction, meaningful-state ablation, and state-source aggregates use 30,969 examples.The latter aggregates are computed jointly over the five test splits.Source: §D.1 · p. 24
• The segmentation experiment uses 57,457 expression instances and retains that denominator separately from the localization tables.Source: §D.1 · p. 24

D.2 RER LOCALIZATION STATE QUALITY

• Table 8 reports terminal answers extracted from RER before BDR refinement, assigning unparseable predictions zero IoU while retaining them in split denominators.Source: §D.2 · p. 24, Table 8 · p. 25
• RER + BDR gains 5.02 points to reach 85.72% mIoU, versus a 2.41-point gain to 84.57% mIoU for the native-output pipeline; their Acc@0.9 values are 68.64% and 66.32%, respectively.Each state source is paired with a separately trained BDR; the RER pipeline starts from lower terminal mIoU but achieves higher final localization quality.Source: §D.2 · pp. 24–25, Table 9 · p. 25
• The zero-anchor ablation provides complementary evidence that meaningful states and anchor-aware objectives matter to the complete system; Table 10 reports split-level metrics.Its upper block reports mIoU / Acc@0.5 and its lower block reports Acc@0.7 / Acc@0.9.Source: §D.2 · p. 25, Table 10 · p. 25

D.3 QUALITATIVE GROUNDING RESULTS

• Qualitative natural-image cases show same-position BDR edits can revise historical steps, the last step, and the Answer while retaining reasoning text, with boundary, instance, relational, and anchor-preservation examples.Source: §D.3 · pp. 25, 27, Figure 8 · p. 26, Figure 9 · p. 27

D.4 CONTROLLED CORRECTION DETAILS

• 87.99% Repair@0.7 is reported for inputs below 0.7 IoU; 30,969 examples start at aggregate input mIoU 58.80, with deterministic assignments across five target-IoU bands and four perturbation types.Each example receives one deterministic nonzero anchor, and CoEvolve uses one BDR pass.Source: §D.4 · p. 28, Figure 10 · p. 28

D.5 REMOTE-SENSING GROUNDING

• On 7,500 DIOR-RSVG expressions, domain-specific retraining retains the natural-image reward, loss, and two-pass editing definitions; qualitative panels include successful edits and a nonmonotonic failure.Source: §D.5 · p. 29, Figure 11 · p. 29

D.6 DOWNSTREAM SEGMENTATION

• The downstream segmentation evaluation feeds each final box to frozen SAM3 without mask fine-tuning and reports split-level cIoU plus the eight-split arithmetic mean.Source: §D.6 · p. 30, Table 12 · p. 30

Imperfect preserve-or-edit decisions

• 15.50% of anchors are degraded by BDR in controlled correction, versus 11.42% for Qwen3.5-9B; quality-gated targets still leave preserve-or-edit decisions imperfect.The authors suggest calibrated edit acceptance, uncertainty-aware refinement, or adaptive stopping as possible directions.Source: E LIMITATIONS · p. 30

State–editor pairing and broader transfer

• The source-matched comparison supports RER→BDR over a separately trained native-output pipeline, while transfer across upstream model families and response formats remains for further study.Source: E LIMITATIONS · p. 30

Coordinate-centric editing

• BDR edits coordinate fields with reasoning text fixed, leaving semantic errors unresolved when coordinate correction cannot address them and not explicitly enforcing consistency between earlier steps and edited boxes.Source: E LIMITATIONS · p. 30