PaperlayerView exact revision on arXiv

Source-linked AI summary

A Design Space for Visual Interfaces for Generative Image Models

Susie S. Y. Li, Mingwei S. G. Li, Remco Chang

arXiv:2609.18065v1cs.HC

TL;DR

Interactive generative-image systems use varied interfaces, but their techniques are often presented independently, making comparison and analysis of how interactions translate into model behavior difficult. The paper proposes a design space that separates interfaces, model objects, and mappings, and applies it to a 51-system corpus. Its corpus analysis identifies recurring patterns in interactions, controlled objects, and mappings, while the framework supports comparison and exploration of design opportunities.

  • Problem

    Interactive generative-image techniques are often presented as independent systems, making it difficult to compare them or understand how user interactions translate into model behavior.

    Source: §1 · p. 1

  • Method

    The paper analyzes 51 systems and represents each through its user interface, controllable model objects, and the mapping between them.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    The corpus analysis found that systems primarily use text prompts and image-space conditioning, most often expose feature maps and LoRA, and mostly use static one-to-one mappings.

    Source: §4 · p. 2

  • Takeaways & Limitations

    The framework offers a common representation for comparing interaction techniques and examining design opportunities across model families and interface paradigms.

    Source: §3 · p. 2, §5.2 · p. 2

Abstract

from arXiv · show

Interactive visual interfaces have become an important means of controlling generative image models, enabling users to manipulate generation through prompts, direct manipulation, and a range of interactions. However, existing techniques are typically presented as independent systems, making it difficult to understand how they relate, compare their interaction mechanisms, or identify opportunities for new interface designs. We introduce a design space for interactive visual interfaces for generative image models derived from 51 research systems and practitioner tools. The framework decomposes each system into three complementary components: the user interface (U), the controllable model objects (Z), and the mapping function ($\phi$) that translates user interaction into model operations. This decomposition provides a common representation for analyzing heterogeneous interaction techniques across model families, revealing recurring design patterns and underexplored regions of the design space. We further present an interactive corpus explorer that support comparative analysis, and discuss usage scenarios for both educational settings and HCI/AI practitioners identifying research and design opportunities.

1 INTRODUCTION

The paper proposes a three-part design space for comparing interactive generative-image systems and understanding how user actions translate into model behavior. It derives the framework from a corpus of 51 research systems and practitioner tools.

  • Existing techniques include text prompting, semantic sliders, sketching, dragging, brushing, inpainting, and example-based editing.Source: §1 · p. 1
  • The design space represents each system through its user interface, controllable model objects, and the mapping between them.The authors analyzed a corpus of 51 research systems and practitioner tools to derive this framework.Source: §1 · p. 1
  • The framework and corpus explorer support systematic comparison and help identify opportunities for new interface designs.Source: §1 · p. 1

2 CORPUS COLLECTION AND ANALYSIS

The authors iteratively assembled and coded a corpus of interactive generative-control systems, using surveys to expand an initial set and explicit criteria to screen entries. The final corpus contains 51 systems, primarily from 2021–2025, while retaining earlier work for historical context.

  • The final corpus contains 51 systems after three entries were removed during screening.It began with 21 author-known systems and added 33 through four recent surveys.Source: §2 · p. 1
  • Corpus development and framework refinement followed Solen et al.’s three-stage Initialize–Expand–Refine methodology.Source: §2 · p. 1
  • Included systems introduced novel generative-control interactions, supported iterative modification under user intent, and altered or navigated learned generative distributions.The corpus spans academic and practitioner-facing systems, primarily from 2021–2025, with earlier work retained for historical context.Source: §2 · pp. 1–2
  • The authors excluded trivial controls, open-ended ideation or browsing without explicit steering, and analytic tools that did not modify generation.Source: §2 · p. 2

3 DESIGN SPACE

The design space separates interactive generative interfaces into the user-facing interaction, the internal model representations being controlled, and the mapping that connects them. These components provide a common representation for comparing techniques across model families and interface paradigms.

  • Separating interface, model object, and mapping enables techniques from different model families and interface paradigms to be compared using a common representation.Source: §3 · p. 2

UI Space

  • The UI Space describes visible interaction primitives, including direct input conditions, parameterized controls, and direct manipulation.Examples include prompts and masks, sliders and control pads, and dragging, brushing, inpainting, and outpainting.Source: §3 · p. 2

Model Object Space

  • The Model Object Space identifies internal representations that interfaces manipulate, from embeddings and condition maps to latent vectors, feature maps, activations, and weights.These objects differ in locality, semantic interpretability, persistence, and scale of influence.Source: §3 · p. 2

Mapping Space

  • The Mapping Space characterizes how interface actions become model operations, including traversal, activation editing, inversion, feature modulation, and weight adaptation.Mappings are described by properties such as cardinality, linearity, locality preservation, constraints, adaptivity, and sensitivity.Source: §3 · p. 2

4 INTERACTIVE SURVEY EXPLORER

The interactive survey explorer lets users filter and compare the coded corpus through coordinated visualizations. Inspecting the corpus reveals recurring patterns in interaction mechanisms, exposed model objects, and mappings.

  • The explorer filters systems by interaction technique, model family, controlled objects, or mapping mechanism and supports comparison through coordinated visualizations.Source: §4 · p. 2
  • Across the corpus, interaction primarily uses text prompts and image-space conditioning, exposed objects most often include feature maps and LoRA, and mappings are mostly static and one-to-one.Source: §4 · p. 2

5 USAGE SCENARIOS / DESIGN OPPORTUNITIES

The framework helps students distinguish whether systems differ in their interfaces, controlled representations, or mappings. For practitioners, it points to opportunities to align interfaces with model objects and mappings with human perception.

5.1 Educational

  • The decomposition helps students separate differences in model representations and mappings from differences in interface widgets.This supports comparison across model families and reasoning about how design choices influence controllability and user experience.Source: §5.1 · p. 2

5.2 Design Implications for HCI/AI Practitioners

  • Changing the UI while preserving the model object, or redesigning the mapping, offers two directions for new interfaces.The examples are painting directly on diffusion feature maps and using perceptually or task-aware mappings instead of linear or uniform ones.Source: §5.2 · p. 2
  • Feature map painting would let users strengthen or suppress semantic activations in selected regions while prompts retain global semantics.The passage describes this as a way to impose explicit spatial constraints during denoising and provide precise, continuous, localized control beyond prompt-based interaction alone.Source: §5.2 · p. 2
  • GANSlider [3] and SliderSpace use linear slider mappings, motivating mappings that make equal slider increments correspond to approximately equal perceptual changes.The proposed approach compensates for nonlinear geometry in latent and weight spaces using an embedding space such as CLIP [13] or LPIPS [18].Source: §5.2 · p. 2

6 CONCLUSION

The paper presents a design space based on 51 systems, decomposing interactive generative interfaces into user interfaces, model objects, and mapping functions. It proposes this framework as a common foundation for comparing systems and discovering interaction techniques.

6 CONCLUSION

  • The design space decomposes interfaces into user interfaces, model objects, and mapping functions, based on a corpus of 51 systems.The framework is presented as supporting systematic comparison and discovery of new interaction techniques.Source: §6 · p. 2
  • The framework is intended as a common foundation for future research on controllable generative AI interfaces.Source: §6 · p. 2
  • Table 1 presents sample works from the survey to highlight aspects of the design framework.Source: Table 1 · p. 3

Related papers

Loading 2609.18065v1…