MonetGPT: Solving Puzzles Enhances MLLMs' Image Retouching Skills

Niladri Shekhar Dutt; Duygu Ceylan; Niloy J. Mitra

Source-linked AI summary

MonetGPT: Solving Puzzles Enhances MLLMs' Image Retouching Skills

Niladri Shekhar Dutt, Duygu Ceylan, Niloy J. Mitra

arXiv:2505.06176v1May 9, 2025cs.GRcs.CVcs.LG

TL;DR

MonetGPT tackles the difficulty of planning identity-preserving procedural retouching for raw photographs. It fine-tunes an MLLM with visual puzzles and grounded reasoning data to critique images and generate explainable edit sequences; evaluations report advantages over generative and procedural alternatives, with scope limited by its current operation set.

Problem
Professional procedural retouching preserves identity but requires novices to plan many operations and parameter values, while generative editing can cause identity loss.
Method
MonetGPT makes an MLLM operation-aware through visual puzzles and grounded reasoning data, then uses it to plan parameterized edits from a predefined procedural library.
Results
Evaluations on diverse images report advantages over generative and other procedural alternatives in explainability and identity preservation.
Takeaways & Limitations
The workflow provides executable, explainable procedural retouching suggestions that users can override without inference-time optimization.
Takeaways & Limitations
MonetGPT currently supports only a limited set of global operations and excludes crops and regional edits.

Abstract

from arXiv · show

Retouching is an essential task in post-manipulation of raw photographs. Generative editing, guided by text or strokes, provides a new tool accessible to users but can easily change the identity of the original objects in unacceptable and unpredictable ways. In contrast, although traditional procedural edits, as commonly supported by photoediting tools (e.g., Gimp, Lightroom), are conservative, they are still preferred by professionals. Unfortunately, professional quality retouching involves many individual procedural editing operations that is challenging to plan for most novices. In this paper, we ask if a multimodal large language model (MLLM) can be taught to critique raw photographs, suggest suitable remedies, and finally realize them with a given set of pre-authored procedural image operations. We demonstrate that MLLMs can be first made aware of the underlying image processing operations, by training them to solve specially designed visual puzzles. Subsequently, such an operation-aware MLLM can both plan and propose edit sequences. To facilitate training, given a set of expert-edited photos, we synthesize a reasoning dataset by procedurally manipulating the expert edits and then grounding a pretrained LLM on the visual adjustments, to synthesize reasoning for finetuning. The proposed retouching operations are, by construction, understandable by the users, preserve object details and resolution, and can be optionally overridden. We evaluate our setup on a variety of test examples and show advantages, in terms of explainability and identity preservation, over existing generative and other procedural alternatives. Code, data, models, and supplementary results can be found via our project website at https://monetgpt.github.io.

1 Introduction

MonetGPT addresses the difficulty of planning interpretable procedural retouching by making an MLLM operation-aware through visual puzzles. It evaluates this workflow against generative, procedural, MLLM-based, and commercial alternatives.

1 Introduction: Professional procedural retouching preserves identity and supports non-destructive, resolution-flexible, interpretable edits, but planning operations and parameters remains difficult for novices.The challenge includes both command knowledge and strategic knowledge, with open-ended planning being especially difficult.
1 Introduction: MonetGPT adapts an MLLM to critique photographs, plan parameterized procedural edits, explain each adjustment, and let users override proposed changes.The workflow targets non-destructive, high-resolution procedural editing while preserving user control.
1 Introduction: Visual-puzzle training makes the MLLM operation-aware, enabling it to learn what operations do and produce higher-quality retouching sequences from limited artist data.Direct fine-tuning only partially improves results because MLLMs lack an internal understanding of image operations.
1 Introduction: The evaluation compares MonetGPT with generative editors, Exposure, chain-of-thought and fine-tuned MLLM variants, and GooglePhoto AutoEnhance using expert and novice assessments.The supplied passage identifies a diverse comparison set and qualitative evaluation by both expert reviewers and novice users.
1 Introduction: The paper reports feasibility and benefits of operation-aware MLLM-guided procedural editing over chain-of-thought reasoning MLLMs and generative alternatives.The comparison emphasizes explainability and identity-preserving procedural workflows.

2 Related Work

Prior work spans automated procedural retouching, generative image editing, and MLLM-assisted graphics workflows. MonetGPT instead fine-tunes an MLLM to select predefined procedural filters and their parameters directly.

2.1 Image Retouching: Earlier retouching methods predicted individual filter parameters, residual layers, per-pixel transforms, or regional curves, but their outputs were not linked to editable image-processing operations.This limits subsequent control over the generated result.
2.1 Image Retouching: Exposure [Hu et al. 2018], text-conditioned sequence planning, and differentiable-filter optimization selected operations or parameters from predefined sets using learned planning or assessment.MonetGPT explores pretrained MLLMs instead of training a planning algorithm from scratch.
2.2 Generative Edits: Generative editing methods use text, spatial guidance, or user interaction to regenerate images, but often struggle with identity preservation because they regenerate every pixel.The related methods include diffusion- and language-model-based editing approaches.
2.3 Reusing LLMs for Graphics Tasks: MLLM-based graphics systems have supported layout planning, 3D editing, embodied interaction, and diffusion conditioning, whereas MonetGPT represents editing as applying predefined filters with predicted sequences and parameters.This distinguishes the method from closed-box image generation.
2.3 Reusing LLMs for Graphics Tasks: Recent MLLM agents use iterative feedback for editing plans, while MonetGPT fine-tunes an MLLM to generate procedural operations directly at test time.The paper claims this is the first such fine-tuning approach for procedural image operations and parameters.

3 Design Considerations

MonetGPT uses a staged library of procedural filters and trains an MLLM with visual puzzles that proxy operation, aesthetic, and planning knowledge. Reasoning data is generated by grounding explanations in actual visual adjustments.

3 Design Considerations: The procedural library stages lighting, color-temperature, and color-specific adjustments, including precise ranges for eight colors.The operations are executed in the listed three-stage order and accept a source image plus function parameters.
3 Design Considerations: Three visual puzzles teach what operations do, how strongly to apply them, how to judge an optimal image, and how to plan operation sequences.Puzzle A targets operation effects; Puzzle B targets application and aesthetic assessment; Puzzle C targets planning.
3 Design Considerations: The puzzles act as proxy loss functions for image-adjustment operations during MLLM fine-tuning.
3 Design Considerations: A pretrained MLLM generates reasoning for each puzzle by explaining which operation was used and what problem it fixes, grounded in the actual visual adjustment.These explanations form data for fine-tuning the MLLM to reason when adjustments are not supplied.

4 Method

MonetGPT adapts MLLMs for procedural retouching by teaching operation effects, aesthetic judgment, and multi-operation planning through visual puzzles. It then uses reasoning-driven inference and a reduced-parameter library to generate interpretable, non-destructive editing plans.

4 Method: MonetGPT trains MLLMs with visual puzzles so they learn procedural operation effects, aesthetic quality, and retouching-plan generation.The framework targets expert-level domain knowledge for procedural image retouching.
4.1 Puzzle A: Gaining Understanding of Individual Operations: Puzzle A teaches an MLLM to identify a single operation and adjustment value from source and edited image pairs.Grounded reasoning links observed visual changes to the true operation and value, supporting supervised learning of operation effects.
4.2 Puzzle B: Understanding Image Aesthetics: Puzzle B trains aesthetic understanding by ordering perturbed images, identifying the expert-edited optimum, and estimating corrections back to it.The design uses perceptually linear adjustment values in [-100,+100] and assumes sufficiently large perturbations degrade the expert edit.
4.3 Puzzle C: Generating a Plan for Image Retouching: Puzzle C generates staged retouching plans with operations, parameter values, and <Adjustment, Issue, Solution> reasoning from poorer variants toward expert edits.Lighting, color-temperature, and color-specific adjustments are separated to improve invertibility, reduce complexity, and clarify reasoning.
4.4 Inference: Reasoning as a Pathway for Regression: At inference, the MLLM first predicts a plan from the source image, then regresses precise adjustment values and applies stages sequentially through the procedural pipeline.Reasoning provides a pathway from high-level edit intent to numerical parameters, while intermediate edited images support subsequent stages.
4.5 Authoring a Library with a Reduced Parameter Space: The authors build a reduced-parameter library whose modular operations use one master parameter, support structured JSON execution, and operate non-destructively on high-resolution 16-bit images.The library simplifies coding while providing an approximately comparable subset of tools to Google Photos and Lightroom; its parameters mostly use a perceptually linear [-100,+100] scale.

5 Evaluation

MonetGPT is evaluated against procedural, generative, regression, and commercial baselines using quantitative generalization tests, visual comparisons, user preferences, and operation-awareness analyses. Across these evaluations, it produces competitive expert-like edits, adapts plans to image conditions and styles, and supports editable staged workflows.

Qualitative Comparison: MonetGPT generates balanced aesthetic enhancements, whereas generative methods can alter identity, Exposure or Gemini can over- or under-expose images, and direct regression makes insufficient improvements.These qualitative differences are shown across baseline comparisons in Figure 6 and discussed for the broader evaluation.
Quantitative Comparison: Our method outperforms all open-source baselines on three of four Adobe5k metrics and performs comparably to closed-source Google Photos.The evaluation trains on one PPR10k expert and tests generalization on 400 Adobe5k images; Table 1 summarizes the comparison.
Quantitative Comparison: MonetGPT uses the full range of operation values, while direct regression overfits by predicting the same values across images.Figure 10 compares the distributions of proposed adjustment values on 100 PPR10k images.
Perceptual User Study: Expert and novice users both prefer our retouching results over Exposure and MGIE in the perceptual study.The study compares edits on 50 Adobe5k and Reddit images using 15 novices and 10 experts.
Effect of Image Operation-Awareness: MonetGPT produces distinct retouching plans for subtle lighting changes, unlike the regression baseline, which proposes nearly identical plans.The model adjusts outputs for balanced, overly bright, and underexposed inputs, indicating sensitivity to image conditions.
Personalized retouching: Style tags such as retro, balanced, and vibrant produce different edit plans and retouched results, while users can revise plans during staged autoregressive editing.Edits to an intermediate plan guide subsequent parameter values and later-stage plans.

6 Conclusion

MonetGPT shows that MLLMs can learn procedural retouching operators, generate executable and explainable edit sequences, and outperform alternatives on benchmark datasets. Its current scope is limited by global-only operations, artist-specific bias, subjective errors, and exclusion of generative filters.

MonetGPT demonstrates that MLLMs can learn procedural retouching operators and translate detailed reasoning into executable function-library calls.The method requires no inference-time optimization and remains compatible with existing MLLMs.
The approach shows advantages over various alternatives on benchmark datasets while preserving explainability by design.
MonetGPT currently supports only a limited set of global operations, excluding crops and regional edits.Regional operations would require semantic masking and sufficient artist-edited images linked to regional masks.
Training on 8k expert-edited images may encode artist-specific aesthetic priors or biases.Larger, more diverse datasets are proposed to mitigate bias and improve parameter-prior learning.
Because retouching is subjective, MonetGPT can produce errors such as saturated regions, and the system excludes generative filters whose integration could compromise procedural interpretability.The paper identifies improved data, perturbation modeling, and human-in-the-loop use as possible responses.