MoI-Bench: A Benchmark for Image Generation and Editing on Mobile Devices

1 INTRODUCTION

• MoI-Bench organizes 64 subtasks and 1,150 samples around 16 operations and eight target categories, with applicable Operation × Target combinations manually reviewed.This structure supports comparisons across operations for the same target and across targets for the same operation.Source: §1 · p. 2
• MoI-Eval applies task-specific instruction-following, visual-quality and detail-preservation criteria, aggregating sample questions into overall quality and capability profiles.Operation × Target profiles describe specific requests; criterion profiles identify which output requirements need improvement.Source: §1 · p. 2
• The Quality-Deployability map keeps output quality and deployment costs as separate dimensions for assessing mobile suitability.Quality includes overall score, task coverage and criterion-group scores; deployability includes end-to-end latency, peak memory and deployed model size.Source: §1 · pp. 2–3
• Ovis-U1 matches FLUX.2 [klein] on text removal but falls far behind on text replacement, while broader task coverage deserves more attention than further deployment-efficiency gains.The results also identify visual integrity and coherence weaknesses in DreamLite and a lower identity-preservation score for Ovis-U1.Source: §1 · p. 3
• The benchmark evaluates 19 generative models and measures text-to-image deployment costs on iOS and Android devices.The contributions also include release of license-audited annotations, inference code and an evaluation toolkit.Source: §1 · p. 3

2 RELATED WORK

• Mobile diffusion work addresses storage, memory, computation and operator-support constraints through optimization, compression, pruning, distillation and quantization.Recent systems also improve generation efficiency and extend capabilities to understanding and editing.Source: §2 · p. 3
• Existing benchmarks assess generation, editing or unified task suites, while MoI-Bench organizes tasks by operation and target and jointly evaluates quality and on-device deployability.The passage characterizes MoI-Bench as the first benchmark, to the authors’ knowledge, to jointly evaluate those dimensions.Source: §2 · p. 3
• Table 1 presents the Operation × Target Task Matrix, with cells describing tasks, merged cells marking shared tasks and N/A marking excluded combinations.Source: Table 1 · p. 4

3.1 OPERATION × TARGET TASK MATRIX

• The Operation × Target Task Matrix pairs what a model should do with the visual content to generate or modify, defining applicable combinations as subtasks.The benchmark manually reviews applicability, adapts operations to target-specific needs, and groups some categories under shared subtasks.Source: §3 · p. 4, §3.1 · p. 5

3 MOI-BENCH

• The benchmark scope is guided by a survey of 1,000 participants, whose reported preferences emphasize portrait retouching, object removal, photo restoration, and text editing.Those preferences motivate an emphasis on editing while retaining text-to-image generation to assess basic image-generation capabilities.Source: §3 · p. 4

3.1 OPERATION × TARGET TASK MATRIX

• The taxonomy groups 16 operations into five groups and covers eight target categories, including portraits, animals, general objects, and text and documents.Targets are assigned by the primary visual content to generate or modify, rather than every entity in an image.Source: §3.1 · pp. 4–5, Figure 2 · p. 6

3.2 DATASET CONSTRUCTION AND QUALITY CONTROL

• Samples are allocated by subtask and transformation, collected or processed to support each task, and given manually adapted instructions that specify requested changes and what should remain unchanged.Two annotators review clarity, feasibility, image-instruction consistency, duplicates, and diversity; all evaluated models receive the same processed inputs.Source: §3.2 · pp. 5–6

3.3 BENCHMARK STATISTICS AND COMPARISON

• MoI-Bench contains 1,150 samples across 64 subtasks, with local editing the largest operation group at 560 samples.Portraits are the largest target group; the benchmark also includes 150 text-to-image, 898 single-image, and 102 multi-image samples.Source: §3.3 · p. 6
• Compared with existing benchmarks, MoI-Bench spans generation and editing tasks, supports up to three reference images, and jointly assesses quality and mobile deployability.It excludes multi-turn editing because the user survey indicated lower demand for that interaction pattern in mobile applications.Source: §3.3 · p. 7, Table 2 · p. 7

4.1 QUALITY RUBRIC CUBE

• The rubric cube extends operation–target tasks with instruction-following, visual-quality, and detail-preservation criteria, marking entries as always applicable, sample-dependent, or inapplicable.Identity preservation is included when relevant; for example, removing one person should preserve the identities of the remaining people.Source: §4.1 · pp. 7–8
• For each applicable criterion, MoI-Eval constructs sample-specific questions from the rubric, instruction, and input images, then manually reviews them before evaluation.Two annotators cross-review applicability and question fit; scoring conditions are set without access to model outputs and shared across models and judges.Source: §4.1 · p. 8

4.2 QUALITY SCORING AND CAPABILITY PROFILES

• Question judgments are aggregated into overall quality and profiles by operation, target, criterion group, and individual criterion.Sample quality uses a weighted geometric mean of applicable criterion-group scores, limiting compensation for poor performance in one group; text-to-image uses only instruction-following and visual-quality groups.Source: §4.2 · pp. 8–9
• Quality tiers combine overall quality, equal-weighted task coverage at a 0.6 subtask-mean threshold, and minimum performance requirements for each criterion group.This distinguishes a high average score from broad coverage and consistent criterion-group performance.Source: §4.2 · p. 10

4.3 QUALITY-DEPLOYABILITY MAP

• Deployability is measured separately through end-to-end latency, peak memory, and deployed model size; only DreamLite was successfully deployed for editing, so text-to-image is the common task across evaluated on-device models.The protocol accommodates model-specific deployment optimizations and measures the resulting costs from a mobile user's perspective.Source: §4.3 · pp. 10–11
• The map places quality tiers and device-specific deployability tiers on separate axes, while a complementary scatter plot shows continuous quality, latency, task coverage, and parameter count.Tier thresholds are calibrated and then fixed for reported comparisons; the map helps indicate whether development should prioritize quality and coverage, deployment efficiency, or both.Source: §4.3 · p. 11

5.1 EXPERIMENTAL SETUP

• The evaluation covers 19 representative models, uses GPT-4o and Gemini 3.1 Pro as judges, and measures deployment on two phones for models under 4B parameters.DreamLite is the only evaluated model successfully deployed for on-device editing, so text-to-image generation is the common deployment task.Source: §5.1 · pp. 11–12

5.2 OVERALL QUALITY ANALYSIS

• The three closed-source models reach Q4 and meet the task-quality threshold on all 64 subtasks; among models under 4B parameters, DreamLite covers more tasks than Ovis-U1.GPT-4o task coverage is 0.64 for DreamLite versus 0.59 for Ovis-U1; the judges assign the same quality tiers.Source: §5.2 · p. 12, Table 4 · p. 12

5.3 CAPABILITY ANALYSIS

• DreamLite and Ovis-U1 score 0.29 and 0.33 on text and documents, versus 0.61–0.76 across portrait categories; their operation strengths and weaknesses also differ.Both are strong on stylization, while replacement and restoration remain weaker operations.Source: §5.3 · pp. 13–15, Table 5 · p. 14, Table 6 · p. 14
• Task-level analysis finds DreamLite and Ovis-U1 score 0.66 and 0.71 on text removal, but 0.03 and 0.27 on text replacement.Their reference-editing performance also reverses by target: Ovis-U1 leads on facial features and hairstyles, while DreamLite leads on multiple subjects; neither supports the evaluated base-plus-reference configurations.Source: §5.3 · p. 15, Figure 5 · p. 15
• Criterion profiles show different weaknesses: Ovis-U1 trails in Selection and Identity, while DreamLite trails in Integrity and Coherence against LongCat-Image-Edit.The analysis also notes DeepGen scores 0.65 on identity preservation, below DreamLite’s 0.73 despite higher Fidelity and visual-quality scores.Source: §5.3 · p. 15, Table 7 · p. 16

5.4 MOBILE DEPLOYMENT

• Mobile-O and DreamLite reach D3, but no evaluated configuration reaches D4; latency and memory costs vary substantially by device.SANA-Sprint remains D2 because storage exceeds the 2 GiB limit for D3, despite inference under 30 seconds on both devices; its two-step generation takes 0.78 seconds while decoding takes 3.15 seconds.Source: §5.4 · pp. 15–16, Table 8 · p. 16

5.5 Q-D ANALYSIS.

• Similar iOS latency does not imply similar coverage: Mobile-O and DreamLite take 2.81 and 3.22 seconds but cover 0.11 and 0.64 of tasks, respectively.The proposed priorities differ by model: add editing to generation-only models, broaden input support and improve weak editing combinations for DreamLite, and reduce Ovis-U1’s latency and memory demand.Source: §5.5 · pp. 17–18, Figure 6 · p. 18

• The framework organizes 64 subtasks by operation and target, then applies task-specific criteria to assess quality and build capability profiles.Source: §6 · p. 18
• For users, the profiles support choosing on-device or cloud models for specific requests such as makeup removal or text replacement.Source: §6 · p. 18
• For researchers, the profiles identify unsupported tasks and unmet quality requirements as directions for expanding capabilities and improving quality.Source: §6 · p. 18
• The Quality × Deployability map adds mobile latency, storage, and memory costs to help assess which capabilities are practical to offer locally.Source: §6 · p. 18

• The survey collected 1,000 anonymous responses on mobile image AI, and its reported selection rates describe respondents rather than the wider population.Source: §A.1 · p. 24
• Interest in on-device image AI was 68% Yes, 24% Maybe, and 8% No.Source: §A.1 · p. 24
• Respondents answered 25 scenarios, each listing 10–15 requirements, and could select up to eight requirements per scenario.Q1 covered current use and interest; Q2 asked about desired use cases.Source: §A.1 · p. 24
• Shared-requirement selection rates count each respondent once if they selected that requirement in any scenario.Source: §A.1 · p. 24

A.2 TASK DEFINITIONS AND COVERAGE

• Operations specify the requested change, while targets identify the content involved; applicable combinations form tasks such as an operation on a target.The 16 operation units include replacement, adjustment, outpainting, and inpainting; reference-editing units specify base and reference images.Source: §A.2 · p. 24, Table 12 · p. 24
• The eight target categories identify the primary content being generated or modified, not every visible entity.For example, changing a jacket targets Clothing & Accessories, while removing one person from a group targets Multi-Subjects.Source: §A.2 · pp. 24–25, Table 13 · p. 24
• The benchmark contains 64 subtasks: 8 generation, 28 local-editing, 7 global-editing, 5 understanding-editing, and 16 reference-editing subtasks.Table 14 marks out-of-scope combinations as N/A; included tasks unsupported by a model receive zero scores.Source: §A.2 · p. 25, Table 14 · pp. 27–28

Q3. Usability requirements

• Each of the survey’s 25 Q3 scenarios presented 10–15 requirements and allowed respondents to select up to eight; Q1 asked about current AI-tool usage frequency.Source: Table 11 · p. 26, §A.2 · p. 26
• A sample portrait-retouching question lists requirements including removing the blemish, preserving identity and other content, and keeping the result natural.Source: Q3. Usability requirements · p. 26

B.1 TASK-SPECIFIC RUBRICS AND CRITERION APPLICABILITY

• MoI-Eval defines eight criteria across instruction following, visual quality, and detail preservation, with preservation applying only to untargeted content or attributes.Rubric entries pair a criterion with an operation and target; applicability may be always required, sample-dependent, or excluded.Source: §B.1 · p. 29, Table 15 · p. 29
• Non-applicable criteria are excluded rather than scored as zero, and sample-dependent criteria are resolved for each sample to determine its questions and criterion groups.Source: §B.1 · p. 29

B.2 SAMPLE-GROUNDED EVALUATION AND SCORING

• A sample-grounded example evaluates Removal × Multi-Subjects using 14 questions and eight applicable criteria for removing a specified person while preserving the remaining scene and people.The example reports a sample quality score of 0.873 for the Ovis-U1-3B output evaluated by GPT-4o.Source: §B.2 · pp. 29–30, Table 16 · p. 30
• Sample-level geometric means exclude inapplicable criterion groups and normalize the remaining operation-specific weights; criteria within a group receive equal weight.Source: §B.2 · p. 30, Table 17 · p. 31
• Each judge call scores one question using a JSON response, with the instruction, question, score conditions, and ordered editing inputs or generated image supplied as appropriate.The examples do not enumerate all input configurations.Source: §B.2 · pp. 30–32

B.3 ADDITIONAL EXPERIMENTAL SETTINGS

• Deployment tests hold prompt, seed, resolution, and sampling settings fixed within each model, but some precision comparisons also change session partitioning or resource management.Android Janus-Pro latency excludes precomputed prompt preparation; failed configurations are excluded from successful-run averages, and precision-only attribution is not possible for affected comparisons.Source: §B.3 · p. 32

B.4 TEXT-ENCODER QUANTIZATION

• The quantization comparison changes only the text encoder’s precision for each model and evaluates both configurations under the same MoI-Eval protocol.Source: §B.4 · p. 32
• For all three tested models, 4-bit NF4 text-encoder quantization changed overall quality by less than 0.0035 absolute and less than 1% relative, with quality tiers unchanged.The number of subtasks meeting the 0.6 quality threshold changed by at most two.Source: §B.4 · p. 32, Table 18 · p. 33

C.1 RESULTS EVALUATED BY GEMINI

• Gemini evaluates the same outputs using identical questions and scoring rules.Table 19 provides capability profiles beyond the overall results in Table 4.Source: §C.1 · p. 33

C.2 DETAILED CAPABILITY PROFILES

• Table 20 reports scores for all 19 models across eight evaluation criteria.The criteria cover instruction following, visual quality, and detail preservation; scores range from 0 to 1, with higher values indicating better performance.Source: §C.2 · p. 33, Table 20 · p. 34
• The detailed profiles complement the selected models reported in the main text.Table 20 gives scores to three decimal places.Source: §C.2 · p. 33

C.3 ADDITIONAL QUALITATIVE RESULTS

• DreamLite captures main content in several generation examples, but precise editing is less consistent.The comparisons span generation and editing tasks.Source: §C.3 · p. 34

D.1 CROSS-JUDGE CONSISTENCY

• ρ = 0.968 for overall quality rankings and ρ = 0.790 for paired sample scores; Gemini scores average 0.051 lower.The ranking correlation includes unsupported-task zeros in Q.Source: §D.1 · p. 34
• Both judges assign the same quality tiers, but calibration using cross-judge agreement means this is not independent validation of judge reliability.Source: §D.1 · p. 34

D.2 AGREEMENT WITH HUMAN JUDGMENTS

• Two annotators independently score sampled outputs, with 500 outputs per editing-capable model and all 150 generation outputs per generation-only model.Unsupported tasks and execution failures are excluded, and annotators do not see model names or VLM scores.Source: §D.2 · pp. 34–35
• ρ = 0.904 between annotators’ sample scores; GPT-4o and Gemini correlate with the human reference at ρ = 0.779 and ρ = 0.852.The judge comparison concerns rubric-based scores rather than user preference.Source: §D.2 · p. 35

D.3 SENSITIVITY TO CRITERION-GROUP WEIGHTS

• The weight analysis changes criterion-group aggregation while keeping question scores, applicability, and sample inclusion fixed.Equal weights and one-at-a-time 25% increases or decreases are tested, with weights renormalized over applicable groups.Source: §D.3 · p. 35
• All tested criterion-weight settings preserve the quality ranking and tiers of all 19 models; the largest absolute Q change is 0.0052.The reported sensitivity comparison is relative to default weights.Source: §D.3 · p. 36, Table 21 · p. 36

D.4 SENSITIVITY TO TIER THRESHOLDS

• A 0.05 threshold reduction moves four models from Q2 to Q3, while a 0.05 increase moves Nano Banana 2 and Seedream from Q4 to Q3.A 0.02 reduction moves only DeepGen; all reported changes are limited to one tier.Source: §D.4 · p. 37, Table 22 · p. 38
• All six iPhone 17 Pro baseline configurations retain their tiers under a 10% budget reduction or a 10–20% increase; a 20% reduction moves Mobile-O and DreamLite from D3 to D2.No evaluated configuration reaches D4 under any tested setting.Source: §D.4 · p. 37, Table 22 · p. 38
• Table 22 summarizes unchanged tier counts as quality requirements shift by δ and deployment limits scale by α.Source: Table 22 · p. 38