Source-linked AI summary
Aphanta: Diagnosing Task-Aligned Image-Edited Intermediates for Multimodal Reasoning
Hengyuan Xu, Wei Cheng, Yumeng Ji, Xuanyang Zhang, Xianfang Zeng, Gang Yu, Xingjun Ma
TL;DR
Image-edited intermediates may help multimodal reasoning, but their value depends on whether editors realize task-specific transformations and whether downstream models use the returned evidence. Aphanta diagnoses this across 20 tasks by comparing direct, actual-edit, and idealized-reference conditions. The resulting map finds strongest utility in grounding, cue injection, and counterfactual state realization, with lower reliability for symbol-sensitive construction and structural extrapolation.
Problem
A visually plausible image edit may fail the count, relation, symbol, or counterfactual state required by a reasoning task, so visual headroom, editor realization, and evidence use must be distinguished.
Method
Aphanta is an automated framework that compares direct, editor-generated-intermediate, and idealized-reference reasoning across candidate tasks in a closed-loop task-discovery and evaluation process.
Results
Across 20 candidate tasks, current instruction editors are most reliable for grounding, perceptual cue injection, and counterfactual state realization, and less reliable for symbol-sensitive construction or abstract structural extrapolation.
Takeaways & Limitations
Image editing is a task-selected visual workspace requiring verification rather than a universal reasoning mechanism.
Takeaways & Limitations
Aphanta’s exploratory selection can increase gains on retained tasks, overlapping operation labels limit population interpretation, and A/B/C does not establish causal use of returned pixels.
Abstract
from arXiv · showhide
Explicit visual intermediates can help multimodal large language models (MLLMs) externalize spatial evidence and updated visual states, but their utility depends on whether an image editor can faithfully realize the required transformation. We introduce \textbf{Aphanta}, an automated task-discovery and closed-loop diagnostic framework for the MLLM -> image editor -> MLLM pipeline. Aphanta evaluates three conditions---direct reasoning, reasoning with an editor-generated intermediate, and reasoning with an idealized reference intermediate---to separate potential visual headroom from the practical utility of current editors. Across 20 candidate tasks and multiple editor--MLLM combinations, we find that utility is strongly task-conditioned. Gains concentrate in visual cue injection, grounding, and counterfactual state realization, whereas intermediates requiring symbol-sensitive construction or structural extrapolation are substantially less reliable. On the selected positive-task subset, our consolidated Qwen pipeline improves the mean task score from 0.343 to 0.445 ($+10.2$ points; $+29.7\%$ relative), while the full study also retains filtered and unsuccessful tasks to expose the boundary. These results position image editing as a specialized visual workspace rather than a universal reasoning mechanism, and establish Aphanta as a reusable protocol for measuring task--representation alignment, editor realization, and downstream pipeline utility.
1 Fudan University 2 StepFun 3 Shanghai Jiaotong Univeristy
The supplied passage identifies project leadership and corresponding authorship, alongside a task-conditioned account of image-edited intermediate utility.
- Current instruction editors reliably assist selected cue-injection, grounding, and counterfactual state-realization tasks.
- Editors are less reliable when requested intermediates require exact structural extrapolation.
- Project leadership and corresponding authorship are explicitly marked in the paper.
1. Introduction
The introduction frames image-edited intermediates as potentially useful visual workspaces whose value depends on task, faithful editor realization, and downstream evidence use. Aphanta addresses this by comparing direct, actual-edit, and idealized-reference reasoning across 20 candidate tasks.
- A visually plausible edit may still fail the specific count, relation, symbol, or counterfactual state required by a reasoning task.
- Aphanta compares direct reasoning, editor-generated intermediates, and idealized reference intermediates across 20 candidate tasks.
- The framework measures alignment among a task, an intermediate representation, and an editor rather than asking whether diffusion models can reason.
- Current general instruction editors are characterized under the studied protocol and task suite, without generalizing to all diffusion or generative modeling systems.
- The study reports a task-conditioned reliability boundary and guidance for selective, verifiable visual assistance while limiting causal conclusions from the A/B/C protocol.
2. Related Work
Related work develops explicit visual intermediates, conditional visual assistance, reliability diagnostics, and increasingly controllable image editing. Aphanta builds on these lines by studying when edited visual states improve multimodal reasoning and whether models use the returned evidence.
- Earlier systems exposed compositional structure through synthetic scenes, visual programs, imagined rollouts, and predicted future observations.
- Modern methods crop, zoom, mark, sketch, or generate new visual states to support inference-time visual assistance.
- Recent work treats visual assistance as conditional, controlling when, how much, or which RGB intermediate to imagine.
- Reliability studies filter noisy cues and diagnose evidence grounding because final-answer accuracy alone cannot reveal whether the correct evidence was used.
- Causal audits show that tool-call text or surrounding structured text can account for observed gains without establishing use of returned visual evidence.
- Image editing has progressed from paired and unpaired translation to diffusion-based and instruction-guided editing.
3. Aphanta: Diagnosing Visual-Intermediate Utility
Aphanta combines a three-condition utility diagnosis with a human-supervised, four-phase loop for discovering, constructing, evaluating, and iteratively refining task-specific visual intermediates. It distinguishes visual headroom from practical editor utility while treating the editor as a candidate workspace rather than an autonomous solver.
- Framework: Aphanta combines a standardized inference chain, a three-condition diagnostic protocol, and a four-phase task-development loop with human quality control.
- Research Setting: The research setting asks which tasks and to what extent editor-generated RGB intermediates improve a complete MLLM reasoning pipeline.
- Research Setting: The same MLLM generally produces the edit instruction and answers after editing, avoiding model-family changes between reasoning stages within a pipeline.
- Research Setting: The editor is treated as a candidate visual workspace because correct instructions can render incorrectly and plausible renderings can remain unhelpful.
- Three-Condition Utility Diagnosis: A compares direct reasoning, B uses an actual editor-generated intermediate, and C replaces it with an idealized programmatically constructed reference.
- Three-Condition Utility Diagnosis: ∆ref measures diagnostic visual headroom, whereas ∆edit measures practical utility of the complete editor-in-the-loop pipeline.
- Three-Condition Utility Diagnosis: A large positive ∆ref with small or negative ∆edit indicates a realization gap under the tested pipeline.
- Automated Task-Development Loop: The four-phase loop screens tasks, performs preliminary diagnosis, constructs and curates data, then trains, evaluates, and feeds results back into proposal.
4. Task-Conditioned Utility of Edited Intermediates
Aphanta finds that edited-intermediate utility varies sharply by task: state realization, cue injection, and grounding are the clearest positive regions, while exact structural extrapolation remains unreliable. The consolidated Qwen pipeline improves the selected positive-task subset, but editor execution and task selection limit how broadly that gain should be interpreted.
- 13 tasks yielded positive retained pipelines, while Plane Geometry Auxiliary Line and Flowchart Decision lacked reliable final-stage assistance.
- Task-conditioned utility: State-realization tasks produce the clearest gains, with editing clocks, deleting objects, or completing masked states increasing downstream scores by 0.21–0.37.The mechanism is to render the queried visual state so the downstream model can reason from an image aligned with the question.
- Task-conditioned utility: Cue-injection and grounding tasks generally benefit, although an actual edit can sometimes outperform the constructed reference intermediate.The reference condition is diagnostic rather than a strict numerical ceiling.
- Task-conditioned utility: Exact structural extrapolation is less consistent: circuit symbols, geometry constructions, gear relations, and flowchart routing show recurring realization errors despite positive reference headroom.The tested editors do not realize these transformations consistently, although the result is an empirical boundary of the evaluated setup rather than an architectural impossibility claim.
- Consolidation and transfer: The consolidated Qwen pipeline raises mean task score from 0.343 to 0.445 on the selected positive-task subset, a +10.2-point absolute and +29.7% relative gain.The corresponding reference score is 0.558, indicating remaining realization headroom; this is not an all-task editor leaderboard.
- Consolidation and transfer: Editor utility depends on execution and varies across combinations: with Qwen3-VL fixed, editor deltas range from −4.7 to +10.2 points, and no editor dominates every operation.Relative percentages can be inflated by small direct baselines, so absolute deltas are the primary aggregate comparison.
5. Conclusion
Aphanta diagnoses when image-edited intermediates help multimodal reasoning by separating task-level visual headroom from current editor utility. Its results support task-selected, verified visual assistance rather than universal reliance on editing.
- Aphanta compares direct, actual-edit, and idealized-reference conditions to measure visual headroom and editor-in-the-loop utility.
- Current instruction editors are most useful for grounding, perceptual cue injection, and counterfactual state realization.
- Exact symbols, relations, or topology form a less reliable region for current instruction editors.
- Aphanta motivates selecting visual assistance by task, verifying generated evidence, and rejecting or replacing unreliable intermediates.
A.1. Experimental Setup
The experimental setup fixes most Qwen-Image-Edit LoRA hyperparameters while varying training length with data volume. Inference permits iterative editing but averages fewer than two editor calls per evaluated sample.
- Most Qwen-Image-Edit LoRA hyperparameters are fixed across tasks, including AdamW and a learning rate of 1 × 10−4.
- LoRA rank is 32 across tasks.
- Training lasts 20k to 50k steps according to data volume.
- Inference may use iterative editing and averages approximately 1.7 editor calls per evaluated sample.
A.2. Agent Implementation Details
The implementation uses a gated task-packet workflow, standardized data interfaces, constructed reference intermediates, and a unified evaluator for direct, actual-edit, and reference conditions. It records artifacts, failures, and review decisions throughout execution.
- Agent role and execution discipline: Each candidate task moves through five ordered phases with explicit approval gates for data collection and training handoff.The phases are protocol design, preliminary validation, data collection, training handoff, and evaluation.
- Milestone reports and negative records: Stopped tasks remain recorded alongside successful tasks, with reasons including saturated baselines, low headroom, unstable data, and unavailable editor outputs.
- Task packet abstraction: Each task packet contains a source or generator, question template, answer normalizer, edit prompt or planner, reference constructor when feasible, and metric.
- Reference construction: Reference intermediates render intended visual states such as boxes, indices, removals, masked completions, rotations, and completed structures without textual answers.
- Actual-edit inference chain: The evaluator uses a common entry point, feeds edited images back with the original image, and bypasses the editor for the reference condition.
- Planner execution: Malformed planner JSON triggers a stricter retry and then a logged task-dependent fallback instruction.
- Prediction traces and metrics: The shared trace records planner outputs, edit prompts, editor status, per-step answers, and final metrics including direct, actual-edit, and reference accuracy.
- Human review and leakage checks: Human review checks task validity, automatic measurability, reference leakage, and train–evaluation separation before construction and training handoff.
A.3. Data Sources of Subtasks
The evaluated subtasks combine open-source datasets, in-house rule-based edits, and fully procedural generation. Open-source datasets support eight listed tasks, while four tasks use in-house edited data.
- Data source categories: Tasks 8, 9, 17, and 19 use rule-based editing pipelines applied to an in-house editing dataset.
- Data source categories: All remaining tasks are synthesized entirely through rule-based programmatic generation.
- Validation resources: Zebra-CoT is used partly as agent-context examples and for phase-2 validation.
A.4. Detailed Information of Tasks
Aphanta’s detailed audit reports task-level metrics, diagnostic outcomes, and transfer results across the full 20-task pool. The records preserve stopped and unsuccessful cases while making the inference pipeline and its artifacts auditable.
- Task audit: Table 5 reports metrics, training volume, preliminary diagnoses, and final-stage results for all 20 tasks, with reference intermediates used only diagnostically.Reference targets are programmatically constructed rather than editor-produced outputs.
- Transfer to existing benchmarks: Actual edits reduce accuracy for all three tested pipelines on BabyVision, while MIRA shows a small positive actual-edit delta only for Gemini-3-Flash + NB2.All three MIRA reference conditions exceed direct reasoning, indicating reference headroom but weak practical realization under the tested pipelines.
- Inference and artifacts: The evaluator records metric summaries and per-sample traces, enabling separate auditing of planner failures, editor failures, and answer errors.Each sample retains copied inputs, edited images, sidecar step JSON, and final trace JSON.
- Task audit: Structured tasks contain the largest stopped or unsuccessful mass in the complete task audit.The audit organizes outcomes by dominant operation, including structured visual construction.
- Instruction templates: Table 4 documents planner instruction-template families distilled from successful task packets and filled with sample-specific objects, directions, angles, and target categories.The templates are injected as in-context planning examples.