PaperlayerView exact revision on arXiv

Source-linked AI summary

What to Edit Next: Visually Aligned Image-Editing Follow-Up Suggestions in Conversational Systems

Zhijing Zhang, Jinpeng Yu, Xin Song, Bingnan Li, Chuyue Li, Changhui Du, Xiaolin Fang, Jiaming Liu, Ruihua Huang

arXiv:2608.07565v1cs.CVcs.AI

TL;DR

Image-creation assistants lack follow-up edit recommendations grounded in the current image, even though most such edits depend on visual context. This paper introduces a three-stage multimodal framework that combines human-reviewed supervision, click-based reinforcement learning, and visual verification, reducing visual inconsistency from 3.7% to 0.9% while improving key online metrics.

  • Problem

    Existing follow-up edit recommendation systems primarily target text conversations, despite 80.1% of image-creation follow-up queries depending on visual context.

  • Method

    The framework combines human-reviewed intent supervision, click-based multi-objective reinforcement learning, and image-first source–target verification for multimodal follow-up edit recommendation.

  • Results

    The full framework reduces visual inconsistency from 3.7% to 0.9% and lifts CTR 32.70%, image take-away rate 16.32%, and average conversation turns per user 39.90%.

  • Takeaways & Limitations

    In Qwen App, multimodal follow-up edit recommendation improves visual consistency and user engagement metrics in a large randomized online study.

  • Takeaways & Limitations

    When a target is subjective or cannot be judged from the image, the verifier leaves the target check uncertain rather than flagging an error.

Abstract

from arXiv · show

Conversational assistants increasingly recommend follow-up edits to help users continue a task. Existing systems primarily target text-only interactions, leaving image-creation conversations underexplored. In image-creation tasks, useful follow-up edit suggestions must reflect user preferences, offer diverse directions, and remain executable on the current image. We collected 100,000 real multi-turn image-creation conversation samples from Qwen App and found that 80.1% are image-dependent, underscoring the need for multimodal recommendation. We address this setting with a three-stage framework. In Stage 1, we use real online data to build a human-reviewed table of appropriate follow-up editing intents, then create SFT targets and fine-tune a multimodal policy. In Stage 2, to align rule-guided SFT suggestions with actual user choices, we use user click feedback to optimize the policy through multi-objective reinforcement learning. In Stage 3, to reduce visual inconsistencies between suggested edits and the current image, we introduce a visual verifier as additional training supervision. Extensive experiments demonstrate that our framework significantly outperforms baselines on both automatic and human evaluations. In a live user-randomized A/B test with millions of users, our final framework reduces visual inconsistency from 3.7% to 0.9%. Furthermore, it significantly improves recommendation CTR by 32.70%, image take-away rate by 16.32%, and average conversation turns per user by 39.90% (all p<0.05).

1. Introduction

The paper frames multimodal follow-up edit recommendation as necessary for visual-creation conversations, where suggestions must reflect user preferences and remain executable on the current image. It proposes a three-stage framework combining human-reviewed intents, user-click preferences, and visual-consistency supervision, achieving strong offline and live improvements.

  • Motivation: 80.1% of 100,000 audited follow-up editing queries were image-dependent, requiring grounding in the latest image rather than the preceding query alone.The remaining 19.9% were text-dependent or expressible through generic edit templates.
  • Three-stage framework: The framework progressively adds human-reviewed editing intents, position-aware user-click preferences, and visual consistency with the current image.Stage 1 constructs task-specific SFT targets; Stage 2 optimizes behavioral preferences; Stage 3 adds explicit visual-consistency supervision.
  • Visual consistency: 3.7% visual inconsistency after click-based optimization fell to 0.9% with the complete framework while preserving expert-rated suggestion quality.The verifier checks whether suggestions rely on absent sources or request already-satisfied target states.
  • Deployment results: 32.70% recommendation CTR, 16.32% image take-away rate, and 39.90% average conversation turns per user improved in a 14-day randomized A/B test involving millions of users.The comparison was against the previously deployed prompt-engineered policy in Qwen App, with all reported improvements statistically significant at p<0.05.

2. Related Work

Prior work covers query and edit suggestion, click-based preference learning, visual grounding, and multi-objective reward optimization. This paper distinguishes visually valid follow-up edits from caption-level factuality and separately optimizes visual consistency alongside preference and list-quality objectives.

  • Query and edit suggestion: Query suggestion methods use co-occurrence, session graphs, neural sequence models, proactive conversational generation, diversity, intent memory, retrieval context, personalization, and image-search multimodality.These lines of work motivate conversational and multimodal follow-up suggestion.
  • Click-based preference learning and position bias: Preference alignment combines supervised fine-tuning, reward modeling, and policy optimization, while click-based methods use clicks as preference signals for reward modeling or reinforcement learning.Clicks can reflect both user preference and suggestion visibility, motivating position-aware construction.
  • Visual grounding and hallucination alignment: Existing visual-grounding remedies address language-prior conflicts through decoding-time contrast or corrective and hallucination-aware preference feedback.Their evaluation primarily tests whether statements about images are true.
  • Visual grounding and hallucination alignment: Follow-up edit recommendation instead requires visual edit validity: introduced targets may be new, but every consumed source must be visible in the current image.This source–target asymmetry is not captured by caption-level factuality, motivating image-first verification.
  • Multi-objective reward optimization: Composite rewards address multiple product requirements using per-dimension normalization and adaptive weighting, while this framework separately optimizes visual consistency rather than click preference alone.The mechanisms balance behavioral preference and list-quality constraints, with grounding handled by a distinct Stage 3 reward.

3. Method

The method builds follow-up edit slates from multimodal conversational context, then progressively optimizes them for validity, user preference, and visual consistency. Its three-stage pipeline combines human-reviewed intent supervision, click-based reinforcement learning, and verifier-based grounding.

  • Method overview: Each editing round uses the latest image, rewritten query, and editing intent to generate 5–7 ordered candidates, with three randomly displayed.The system targets six suggestions while accepting 5–7 as product-valid.
  • Stage 1: SFT data construction: Stage 1 constructs six-suggestion SFT targets by combining real inputs, 61 human-reviewed follow-up intents, teacher-generated candidates, and validation.The pipeline uses general, priority, and continuation generation tracks, then filters candidates for intent validity, structure, wording, repetition, image relevance, and product rules.
  • Stage 2: Preference optimization: Stage 2 learns a vision-language preference model from position-aware click pairs and optimizes click preference, validity, perplexity, length, and diversity with GRPO.The dataset contains 173,071 pairs, and comparisons use only unclicked suggestions displayed above the clicked suggestion to reduce display-position bias.
  • Stage 2: Preference optimization: The output gate requires valid JSON, 5–7 parsed suggestions, and only allowed intent labels, while the perplexity reward keeps rollouts close to the fixed SFT distribution.The preference score is complemented by these constraints because the policy can otherwise exploit the reward model with malformed, long, or repetitive text.
  • Stage 3: Visual grounding: Stage 2 raises visual inconsistency from 3.0% for SFT to 3.7%, motivating Stage 3’s structured verifier and grounding reward for checking required sources and target states.The verifier keeps object-finding and state-comparison flags separate and fails open when subjective targets cannot be judged from the image.

4. Experiments

Experiments show that the cumulative framework improves offline quality, visual consistency, and online engagement, with the visual verifier addressing errors that click supervision alone cannot reliably resolve. Structured source–target verification further improves recall while preserving valid creative edits through a low false-rejection rate.

  • End-to-End Results: SFT improves all three online metrics over PE, Stage 2 delivers the largest CTR lift, and the full framework achieves the best take-away rate and longest conversations.The full framework retains a similar CTR lift to Stage 2 while improving downstream editing-path outcomes beyond clicks alone.
  • End-to-End Results: The full framework achieves the lowest observed visual inconsistency at 0.9%, while improving expert quality and reducing within-list redundancy to 8.8%.Expert GSB rises from +332 for SFT to +405 for Stage 2 and +446 for the full framework.
  • RL Ablation: Adding the grounding reward yields the lowest observed inconsistency rate (0.9%) and highest GSB after diversity and normalization reduce repetition and recover visual regression.Preference RL lengthens and repeats suggestions; max-pair diversity repairs most list repetition, while per-dimension normalization and dynamic text-side weighting further lower redundancy.
  • Click-Supervision Analysis: Position-aware pair construction improves held-out click accuracy from 0.619 to 0.690 at a matched 51.9K budget, but click supervision does not reliably determine edit feasibility.The tier-0 versus tier-1 inversion halves with position-aware pairs and then stalls at 8B, indicating limits to behavioral preference modeling.
  • Verifier Evaluation: Source–target verification recalls 78.7% of visual inconsistencies and falsely rejects 0.6%, versus 47.5% and 22.2% for the single-pass baseline.It recalls 92.9% of missing sources and 74.5% of already-satisfied targets, while Stage 3 reduces union visual inconsistency by 75.7% from 3.7% to 0.9%.

5. Conclusion … A.2. Generation Tracks

The paper concludes that follow-up image-edit recommendations must jointly reflect user preferences and remain executable on the current image. Its reproducibility protocol specifies request-level leakage controls and three validated generation tracks.

  • 5. Conclusion: The framework combines real-context SFT targets, position-aware multi-objective RL, and image-first source–target verification.These stages align recommendations with human-reviewed intents, click preferences, and visual consistency.
  • 5. Conclusion: Follow-up edit recommendations require both clicks and pixels: they must reflect user preferences while remaining executable on the current image.
  • A.1. Splitting and Leakage Controls: 128,750 paired impressions resulted from retaining at most one eligible impression per non-empty session ID before constructing position-aware pairs.The pipeline splits paired data by request ID using seed 42.
  • A.1. Splitting and Leakage Controls: 22 paired impressions had empty session IDs and were separable only by request ID.No usable user ID was present, so the protocol makes no user-level split claim.
  • A.2. Generation Tracks: The general track covers all retained contexts, while the priority track provides more specific directions for common high-traffic cases.
  • A.2. Generation Tracks: The continuation track adds candidates only when validation leaves fewer than six suggestions.All three tracks use the same intent-label, validator, and product-rule checks.

A.3. Preference-Data Funnel … A.6. External Grounding Audit

The appendix documents how preference data and evaluation denominators are constructed, then audits image dependence and externally validates visual-grounding judgments. The audits show that follow-up edits are predominantly image-dependent and that expert-corrected inconsistency rates closely track the reported judge-based differences.

  • A.3. Preference-Data Funnel: The preference funnel uses clicked, request-grouped impressions with three displayed suggestions and no unclicked-impression denominator.Production sampling selected suggestions without replacement from the generator slate.
  • A.3. Preference-Data Funnel: 128,750 eligible impressions yield 84,429 one-pair and 44,321 two-pair examples, split into 122,313 training and 6,437 validation requests.
  • A.4. Denominators and Statistics: The external grounding audit retains 970 image–sample units with valid judgments, containing 5,811 SFT, 5,725 Stage 2, and 5,716 full-framework suggestions.Each evaluation session contributes four policy samples, while the audit scores two fixed sample indices.
  • A.4. Denominators and Statistics: 172 SFT suggestions are flagged across 143 slates, indicating sparse, local visual inconsistencies and supporting suggestion-level aggregation.
  • A.5. Multi-Turn Query Audit of Image Dependence: 80.1% of follow-up queries are image-dependent and 19.9% text-dependent; at least 23.9% act on a specific source object, versus 3.8% generic edits.The source-object figure is a conservative lower bound because ambiguous references are omitted.
  • A.5. Multi-Turn Query Audit of Image Dependence: Source-referential, target-novel, and generic suggestions win 56.7%, 55.1%, and 55.2% of position-aware click pairs against the fixed template pool.These win rates are separate from the 100K image-dependence classification.
  • A.6. External Grounding Audit: The external judge achieves 90.7% recall, 89.0% precision, and 96.1% agreement across 1,276 reviewed suggestions using inverse-probability weighting for cleared cases.Flagged suggestions are reviewed exhaustively, while cleared suggestions are sampled.
  • A.6. External Grounding Audit: Expert-corrected visual-inconsistency rates are 3.0%, 3.7%, and 0.8% across the three arms, making the Stage 2–full-framework gap 2.9 points versus 2.8 judge-reported points.Balanced agreement is 94.3%, 94.4%, and 94.1%, while judge-call failures are 2.0%, 1.7%, and 1.8%.

A.7. Training Configuration … B.2. Constructed Visual Inconsistency Cases

The paper specifies separate training configurations for supervised fine-tuning, reward modeling, and reinforcement learning, while standardizing evaluation across arms. Its verifier uses structured source–target checks, and constructed cases isolate already-satisfied and missing-source inconsistencies under controlled visual conditions.

  • A.7. Training Configuration: SFT trains Qwen3-VL-8B for five epochs at learning rate 10^-5 with LoRA rank 4, α=16, a frozen visual encoder, sequence length 8192, and 5% warmup.The per-device batch size is 1 with gradient accumulation 2.
  • A.7. Training Configuration: The click reward model trains for one epoch at learning rate 5 × 10^-5 with LoRA rank 16 and α=64, targeting all linear modules while freezing the visual encoder.It trains the multimodal aligner, uses sequence length 4096, and has effective batch size 32.
  • A.7. Training Configuration: RL uses G=8 rollout slates per context, global batch 128, GRPO minibatches of 32, one epoch at learning rate 10^-6, and 5% warmup.The KL and entropy coefficients are 0.15 and 0.001, the clip half-width is ϵ_clip=0.2, and the reward-normalization floor is ϵ_σ=10^-6.
  • A.7. Training Configuration: All experimental arms share initialization, reward model, data order, optimization settings, rollout count, verifier, and evaluation requests, with identical decoding settings during evaluation.Evaluation uses temperature 0.7, nucleus sampling 0.9, at most 512 new tokens, a visual token budget of 1,254,400 pixels, and seed 42; comparisons are regenerated and scored together because serving seeds are not reproducible.
  • B. Source–Target Representation and Verifier: The VLM verifier records the current visual state, extracts required sources and target states from each candidate, and separately checks source existence and target satisfaction.The resulting flags determine the grounding reward, following an image-first decision procedure rather than a single end-to-end validity prompt.
  • B.1. Deployed Output Schema: One verifier call scores an entire slate by emitting a candidate-independent image inventory followed by one structured record per candidate.The inventory captures subjects, text, watermark, background, objects, scene state, style, layout, and person attributes, while candidate records include sources, targets, evidence, states, plausibility, effectiveness, and reasoning fields.
  • B.1. Deployed Output Schema: Sources must be present and receive existence bits plus pointing evidence, whereas visually decidable targets are compared with the current state to detect already-satisfied edits.Missing-source status comes from existence bits, already-satisfied status comes from state comparison, free-form summaries are ignored, and global edits may lack localized sources.
  • B.2. Constructed Visual Inconsistency Cases: Constructed inconsistency cases retain production editing chains only when the output visibly contains the requested change, then manually confirm them before evaluation.They include already-satisfied restatements and missing-source edits, but only discrete target states qualify; missing-source cases are smaller because they require the scarcer input-image side of the logs and complement natural policy-output evaluation.

B.3. Transfer of the Calibration Rates … C.1. Length Preference of the Click Reward Model

The paper specifies a conservative, source–target visual-verification pipeline with one-directional safeguards, then details reward optimization and shows why length must be explicitly controlled. Calibration rates are not directly transferable to production prevalence, while click-reward optimization favors unnecessarily longer suggestions.

  • B.3. Transfer of the Calibration Rates: Calibration-set inconsistency ratios do not estimate online prevalence because the set is constructed from inconsistent cases and expert-selected excellent items.Recall and false rejection remain meaningful under the distribution shift, whereas precision and other prevalence-dependent quantities do not transfer.
  • B.3. Transfer of the Calibration Rates: The source–target verifier compares identical items after retries, but the end-to-end gain cannot be attributed separately to inventory ordering, decomposition, or post-parsing guards.All three components change together between the compared prompts.
  • B.4. Verifier Decision Procedure: The verifier first records conservative image evidence, then classifies candidate noun phrases as existing sources or introduced targets before validating source presence and target-state changes.Targets need not exist beforehand; sources require visual grounding, and only verifier-recorded visual evidence counts.
  • B.6. One-Directional Guard Design: Final visual-inconsistency flags are derived from source-existence and target-state fields, while post-processing may clear parser-induced false alarms but never create new ones.The single-pass baseline lacks the inventory and source–target split, using rubric scores for executability, reasonableness, and contradiction avoidance.
  • B.5. Judged Dimensions and Their Roles Table 7 | Per-candidate dimensions emitted by one verifier call.: The verifier’s per-candidate dimensions distinguish source existence, target satisfaction, grounding plausibility, effective change, and content density, but only inconsistency flags enter the grounding reward.Content density selects the length budget, with global appearance edits assigned the shorter budget.
  • B.7. Operation Semantics: Operation semantics are organized by edit type in a dedicated source-and-target semantics table.The table is identified as Table 8 in the supplied passage.
  • C. Optimization Details: The optimization accepts 5–7 parsed suggestions, masks grounding for gate-failed rollouts, and updates weighted rewards with separate normalization and adaptive length and diversity constraints.Gate-failed rollouts receive zero gate, length, and diversity scores, retain the reward-model score, and contribute no grounding advantage.
  • C.1. Length Preference of the Click Reward Model: The click reward model scores padded rewrites higher in all ten meaning-preserving pairs, and preference-only optimization increases mean suggestion length from 11.9 to 17.0 characters.The length reward is introduced to counter this measurable preference for non-essential modifiers.

C.2. Diversity Signal and Aggregation · C.3. Genericization Stress Test

C.2 shows that max-pair embedding-based diversity, strengthened by adaptive weighting, sharply reduces semantic and lexical redundancy. C.3 shows click-only optimization collapses into generic edits, whereas the full reward set avoids that shortcut.

  • C.2. Diversity Signal and Aggregation: 20.8% redundancy follows from max-pair aggregation, improving on the 28–30% left by lexical Jaccard or mean embedding similarity.The diversity reward uses embedding similarity and max-pair aggregation; the comparison uses one matched protocol across five variants.
  • C.2. Diversity Signal and Aggregation: 9.6% redundancy and 0.4% character-Jaccard near-duplicates result when adaptive weighting is added to max-pair aggregation.Adaptive weighting further reduces redundancy beyond the max-pair formulation alone.
  • C.2. Diversity Signal and Aggregation: 28.1% to 19.1% confirms the same max-pair benefit in the matched-step results from the main training chain.These results show that the aggregation effect extends beyond the separate semantic-redundancy probe.
  • C.3. Genericization Stress Test: The genericization stress test evaluates whether reward composition permits source-free, generic appearance edits that repeat across images.The test targets a possible shortcut that lowers checkable visual inconsistencies by ignoring image-specific content.
  • C.3. Genericization Stress Test: +0.31, +1.25, and +2.46 click-RM score gains after 30, 80, and 160 steps coincide with policy collapse under click-only optimization.The stress policy uses click RM as its sole learned signal and removes the PPL and diversity rewards retained by the preference row.
  • C.3. Genericization Stress Test: 99.9% cross-image reuse at step 160, with 0.1% within-list redundancy, indicates that nearly every image receives one generic edit.An unconstrained per-suggestion attractiveness model therefore rewards this degenerate solution.
  • C.3. Genericization Stress Test: 54.3% reuse versus 57.3% for SFT shows that the full reward set avoids the generic route while maintaining similar length variation, 2.9 versus 3.0 characters.The full reward set includes PPL, max-pair diversity, and a cross-image reuse audit.

D. Human Evaluation Protocol

Eight experts blindly evaluate randomized policy outputs using the latest image, current query, and intent, applying a 0–3 suggestion-quality rubric across 800 judgments per arm. Policy performance is summarized by GSB relative to PE, while verifier quality is assessed separately because score 0 can include multiple severe failures.

  • Evaluation setup: Eight experts evaluate randomized policy outputs with the latest image, current query, and current intent visible, while every arm receives 800 suggestion judgments.Evaluators do not see reward scores, and all arms are assessed under the same protocol.
  • Quality rubric: Each suggestion receives one integer quality score from 0 to 3, spanning excellent continuation to severe failure.Scores reflect intent continuation, reference to real image content, executability, editing value, relevance, redundancy, and constraint violations.
  • Aggregation: GSB is the difference between summed expert scores for a policy and PE, with PE fixed at zero as the reporting reference.With 800 judgments, GSB ranges from [−2400, 2400], and positive values represent net expert score gained over the launch policy.
  • Verifier assessment: Verifier quality is measured on the Section 4.5 calibration benchmark rather than inferred from GSB, because score 0 may combine visual hallucination with other severe failures.This separates verifier assessment from the human suggestion-quality aggregate.

E. Qualitative Analysis

Qualitative cases show that the full framework preserves visual and semantic alignment better than SFT across diverse image-editing requests. Remaining failures include paraphrase-heavy suggestions, missed visual states, and blocked rare intent transitions.

  • Qualitative comparisons: Held-out cases show SFT making visually or semantically mismatched edits, while the full framework suggests office changes, references a visible red seal, and avoids contradictory body edits.On a mathematics grid, it also proposes related area and perimeter exercises.
  • Remaining failures: The model still emits polished but low-value paraphrases, misses subtle visual states when verification fails open, and rejects rare but valid intent transitions.These failures identify remaining weaknesses in suggestion value, visual verification, and intent coverage.
  • Mitigations: Max-pair diversity and the PPL reward partly reduce paraphrase collapse, while validated general backfills reduce empty slates when rare transitions are blocked.The mitigations target both recommendation diversity and coverage under constrained intent transitions.

F. Online Experiment Details

The 14-day online experiment randomized users to concurrent policy arms on Qwen App’s image-generation surface, with each arm receiving equal traffic and a shared three-suggestion display process. It evaluated engagement metrics while enforcing latency, generation-failure, and negative-feedback constraints.

  • Experiment Design: The experiment ran for 14 days on Qwen App’s image-generation surface, with concurrent arms assigned randomly and fixed at the user level.Each user saw the same policy across sessions, and every arm received the same 5% traffic allocation, including millions of users.
  • Experiment Design: Each policy generated a candidate slate, after which the shared display layer randomly selected three suggestions without replacement.
  • Evaluation Metrics: Recommendation CTR, image take-away rate, and average conversation turns per user were the core online metrics, while operational regressions blocked launch.Latency, generation failure, and negative feedback were monitored as constraints; reported PE-relative lifts were all significant (𝑝< 0.05).
Loading 2608.07565v1…