PaperlayerView exact revision on arXiv

Source-linked AI summary

PolyGandr: Steering Generative Models via Spatial Navigation through Latent Space

Mingwei S. G. Li, Susie S. Y. Li, Isabel Moreira de Oliveira, Ryan P. Adams, Remco Chang

arXiv:2610.37446v1cs.HC

TL;DR

Generative-model interfaces rely primarily on language, which can be inefficient or imprecise for fine-grained control and exploration. PolyGandr offers example-based spatial steering through linked global and local views. Demonstrations span generative and parametric design spaces, while also identifying limits and needs for feasibility guidance.

  • Problem

    Natural-language interfaces can be inefficient and imprecise for fine-grained control, iterative refinement, and open-ended exploration, and users may recognize variations they cannot express as prompts or parameters.

    Source: §1 · p. 2

  • Method

    PolyGandr connects global exploration and local refinement, letting users define a starting point, steering direction, and step size from examples and revise the operation.

    Source: §1 · p. 2

  • Results

    Demonstrations across StyleGAN2, Stable Diffusion XL, and rotational kirigami show generative and domain-specific uses, while identifying needs for domain-specific refinement and feasibility guidance.

    Source: §1 · pp. 2–3, §7 · p. 27

  • Takeaways & Limitations

    The paper argues that generative interfaces can treat steering as iterative navigation, inspection, and revision of transformations across a structured generative landscape.

    Source: §7 · pp. 27–28

    Straight-line steering assumes a locally Euclidean latent space, but may leave a curved valid-output manifold and produce poor-quality, meaningless, or unpredictable outputs.

    Source: §6.5 · p. 27

Abstract

from arXiv · show

Generative models support content creation across text, images, videos, and design disciplines, yet their interfaces rely primarily on natural language. While effective for specifying high-level intent, language can be inefficient and imprecise for fine-grained control, iterative refinement, and open-ended exploration. We present PolyGandr, a visual interface for exploring relationships between latent representations and generated outputs, complementing natural-language model steering. PolyGandr formulates generative control as high-dimensional navigation, enabling users to define starting points, steering directions, and step sizes through a unified map-and-mixing-palette interface. Combining global exploration with local refinement supports both broad design-space navigation and fine-grained output manipulation without repeatedly translating visual intent into prompts. We evaluate PolyGandr across three use cases: StyleGAN2 face generation, Stable Diffusion chair design, and rotational kirigami engineering. A formative study indicates that PolyGandr complements existing workflows by supporting real-time editing while simultaneously providing an overview and means to explore the broader design space.

1 Introduction

Generative interfaces rely primarily on language, which can make it difficult to express a visual variation once recognized. PolyGandr connects example-based steering with global exploration and local refinement, and is demonstrated across generative and parametric design spaces.

1 Introduction

  • Language-centered interfaces can be inefficient or imprecise for fine-grained control, while users may recognize a desirable variation without knowing how to express it as a prompt or parameter.Source: §1 · p. 2
  • PolyGandr connects example-based steering with global exploration and local refinement through a linked overview and mixing interface.Users can reposition neighboring examples to adjust their influence on the base point or either endpoint of a steering direction.Source: §1 · pp. 2–3
  • Users construct steering operations from a starting point, contrasting examples that define a direction, and a step size, then inspect and revise those examples.Source: §1 · p. 2
  • Across StyleGAN2, Stable Diffusion XL, and rotational kirigami, the examples illustrate generative and domain-specific uses while identifying needs for domain-specific refinement and feasibility guidance.The engineering case study involved a domain expert.Source: §1 · pp. 2–3
  • The interaction framework links global exploration with local refinement to support steering generative models.Source: §1 · p. 2

2 Related Work

Related work covers model steering and interfaces for human-in-the-loop exploration. PolyGandr builds on these lines of work while focusing on spatial construction and refinement of example-defined steering operations.

2 Related Work

  • The related work encompasses steering mechanisms and computational foundations, as well as interfaces for human-in-the-loop steering and exploration.Source: §2 · p. 3

2.1 Model Steering and Conditioning Methods

  • Latent and feature-space methods identify steering directions, while diffusion approaches use attention, image guidance, instructions, spatial conditions, or learned controls; PolyGandr instead focuses on spatially constructing and refining operations through examples.The cited methods include Deep Feature Interpolation [50], GANSpace [24], and Prompt-to-Prompt [21].Source: §2.1 · p. 3

2.1.1 Geometry of Generative Latent Spaces.

  • Linear latent steering is a fast, simple approximation whose validity can vary by location and model; a locally meaningful direction may not remain geometrically or semantically consistent elsewhere.Source: §2.1.1 · pp. 3–4

2.2 Interactive Exploration and Steering Interfaces

  • Interactive systems support discovering and refining controls, while broader exploration tools help users browse latent spaces; PolyGandr uses its global map to construct, visualize, and refine steering operations.Its interaction unit is example-defined vector arithmetic, including a base point, contrasting examples, and step size.Source: §2.2 · p. 4

3 Interface Design Goals and Solution Guidelines

PolyGandr’s design goals connect orientation in a sampled generative space, local refinement, and consistent reasoning across those scales. The corresponding guidelines use a map, a local mixer, and a shared spatial interaction metaphor.

3 Interface Design Goals and Solution Guidelines

  • The design goals progress from understanding the model’s output range, to precise local refinement, to maintaining a consistent interaction metaphor across scales.Source: §3 · p. 5

3.1 DG1: Support Orientation within the Sampled Design Space

  • The global overview maps sampled outputs so users can recognize patterns, clusters, transitions, and gaps, then zoom into regions and generate additional nearby samples on demand.This supports moving from a broad mental model to more detailed exploration of relevant regions.Source: §3.1 · p. 5

3.2 DG2: Support Precise Refinement of Examples and Steering Directions

  • A local mixer exposes neighboring latent representations for weighted interpolation, helping users inspect nearby variations and incrementally refine an output.For rotational kirigami, refinement also uses a polygonal hub editor and controls for target height and twist.Source: §3.2 · p. 5

3.3 DG3: Promote Consistent Reasoning across Global and Local Navigation

  • Both views use an “I want to go there” interaction: users navigate toward examples globally and rearrange local neighbors to reshape their influence on interpolation.With sufficient layout adjustment, users can in principle access any convex combination within the simplex spanned by the selected neighbors.Source: §3.3 · p. 6

4 Method

The section introduces the core techniques underlying PolyGandr’s model-output steering.

4 Method

  • The section describes the core techniques underlying model-output steering in PolyGandr.Source: §4 · p. 6

4.1 Three Components of Model Output Steering

PolyGandr represents steering through a base point, a direction, and a step size. It derives directions from contrasting examples and applies them to the base point, with a simplified global formulation supporting real-time interaction.

4.1.1 The Base

  • A base point starts from a sample landmark in high-dimensional space and can be formed as a weighted combination of nearby samples.Varying the weights continuously enables navigation between known examples; smoothness and differentiability depend on the weighting function.Source: §4.1 · p. 6

4.1.2 The Direction

  • A steering direction can be defined by subtracting a negative example from a positive example, or by averaging differences across matched positive-negative pairs.The text also describes a direction formed as the difference between the averages of two groups.Source: §4.1 · p. 6
  • PolyGandr defines a steering direction as the difference between the mean latent representations of groups representing opposite concepts.The resulting contrast can be applied to a new base point to steer it toward either concept.Source: §4.1 · p. 7
  • PolyGandr simplifies steering by treating the direction as a global vector applied to any base point, enabling real-time interactive specification of the base, direction, and step size.This formulation treats the latent space as Euclidean.Source: §4.1 · p. 7

4.1.3 The Step Size α

  • The step size α controls the magnitude of change: smaller steps yield subtler changes, while larger steps move farther along the chosen direction.In PolyGandr, a slider controls how far the base point moves along that direction.Source: §4.1.3 · p. 7

4.2 Global Overview and Exploration via Zoom-in Lenses

PolyGandr maps generated samples to support global exploration, while distance-weighted interpolation lets users navigate by mixing nearby examples. Base-only, pair, and steer modes offer different ways to inspect regions and trajectories in that space.

4.2 Global Overview and Exploration via Zoom-in Lenses

  • The global overview maps generated samples to expose degrees of freedom and major variation modes; after known variations are fixed, filtered, or binned, remaining representations are projected into 2D or 3D using dimensionality reduction such as UMAP.Source: §4.2 · p. 7
  • Distance-weighted interpolation combines neighboring anchor samples’ latent codes at a control-point location; the weighting function determines how local and discriminating that mixture is.Inverse-distance weighting gives every sample influence everywhere, while smoothstep and logarithmic functions use a cutoff radius for sparser, more local mixtures.Source: §4.2 · p. 8
  • A 2D movement through the map continuously varies the relative influence of multiple examples, changing contributions from high-dimensional latent vectors rather than following one predefined direction.The overview can focus on residual, correlated, or previously unknown variation after known dimensions are fixed, filtered, or binned.Source: §4.2 · pp. 7–8

4.2.1 Three modes of exploration.

  • Base-only mode uses draggable lenses to inspect local regions, and users can add multiple lenses to examine the broader latent-space landscape.Source: §4.2.1 · p. 9
  • Pair mode uniformly interpolates between endpoints of selected lens pairs over a specified number of steps.The figure depicts two independently specified trajectories and their generated outputs.Source: §4.2.1 · p. 9, Figure 3 · p. 9
  • Steer mode specifies a base point, contrastive direction, and step size, then maps the resulting trajectory and its intermediate points back into the global overview.The interface sweeps the step size over a range to visualize the trajectory relative to surrounding latent-space structure.Source: §4.2.1 · p. 9, Figure 3 · p. 9

4.3 Local Refinement via Free-Form Sample Interpolation

PolyGandr’s local mixer lets users reshape neighborhoods of examples to refine interpolated latent codes. Its 2D interface exposes only a slice of the high-dimensional space, so coverage depends on the chosen layout.

4.3 Local Refinement via Free-Form Sample Interpolation

  • Users can reposition neighboring examples to change their distances from the control point and thereby alter the convex combination.The range circle sets which samples contribute; their distances and the selected weighting function determine their influence.Source: §4.3 · p. 9, Figure 4 · p. 10
  • A fixed anchor layout yields only a 2D parameterization and cannot generally cover the full latent space.The interface therefore requires consideration of how the sampled surface relates to the anchor hull, local data manifold, and semantic variation.Source: §4.3 · p. 10
  • Grid mode samples interpolated latent codes across the canvas, making the local 2D surface visible and explorable.The mode can be computationally expensive because each grid location may require a model evaluation; StyleGAN2 ran interactively on an NVIDIA RTX 4090.Source: §4.3 · p. 10, Figure 5 · p. 10
  • The technique is positioned primarily for free exploration; when desired semantics are already specified, existing techniques can provide more direct, complementary control.Source: §4.3 · p. 10

4.4 Linked Views and Interactions

PolyGandr links a global map for navigating and comparing steering trajectories with a local mixer for refining their component examples. Shared latent representations keep the views connected while preserving a stable global reference during local rearrangement.

4.4 Linked Views and Interactions

  • The linked-view prototype combines global exploration, local refinement, configuration controls, and output export in one workflow.Source: §4.4 · p. 11

4.4.1 Global View Interactions.

  • The global view supports map browsing, zooming, panning, and draggable lenses that represent interpolated latent codes; users can inspect steering trajectories and compare multiple operations.Optional grid binning reduces clutter, while marks or thumbnails let users trade overview density against visual detail.Source: §4.4.1 · pp. 11–12

4.4.2 Local View Interactions.

  • The local view lets users move the central code and reposition neighboring anchors, changing interpolation weights and the active neighborhood.Grid mode assigns interpolated codes across the canvas to visualize the 2D surface, but its model-evaluation cost depends on architecture, hardware, and resolution.Source: §4.4.2 · p. 14

4.4.3 Global-local Consistency.

  • Both views use shared latent representations, and generated or refined points can be projected back into the global overview.Local rearrangement changes local coordinates but preserves fixed global coordinates; peripheral ghost anchors retain access to displaced points and their global correspondence.Source: §4.4.3 · p. 15

4.4.4 Other Features.

  • A minimap supports orientation, while progressive and minibatched decoding provide earlier feedback during high-latency interactions.For diffusion models, intermediate denoising results can be shown before final outputs are computed; editable lens labels can be retained in exported views.Source: §4.4.4 · pp. 15–16

5 Evaluation

The evaluation demonstrates PolyGandr with StyleGAN2 and Stable Diffusion XL, then examines its use in rotational kirigami engineering. The examples show latent-space interpolation and steering, while the diffusion case also exposes higher inference costs.

5 Evaluation

  • The StyleGAN2 and SDXL use cases, followed by a kirigami expert study, progress from proof of concept to more complex generation and an engineering design setting.Source: §5 · p. 16

5.1.1 The generative model and data representation.

  • For StyleGAN2 models trained separately on human faces and cat pictures, overview maps use sampled b8 activations (512×8×8) projected by UMAP to 2D or 3D, while interpolation uses 512-dimensional style space.They sampled 20,000 points from StyleGAN2’s input space.Source: §5.1.1 · p. 16

5.1.2 Use case walkthrough.

  • Interpolating between cat points produces smooth transitions toward orange cats, although straight latent paths can appear curved or distorted in the nonlinear map.Source: §5.1.2 · p. 16, Figure 13 · p. 17
  • Interactive steering can probe how a nonlinear projection distorts a straight latent direction, revealing local curvature across positions and directions.Source: §5.1.2 · p. 17
  • The same age direction produces meaningful edits from male and female base points.Source: §5.1.2 · p. 17, Figure 14 · p. 18
  • Local mixing reduces unwanted gender entanglement in a global age direction while preserving the intended age transformation, without rebuilding the steering operation.Source: §5.1.2 · p. 17, Figure 15 · p. 19

5.2.1 Generative Model and Dataset generation.

  • The SDXL chair dataset contains 4,000 samples from six styles using a shared prompt template; generating the samples took approximately three hours.Runs fixed the scheduler and randomization settings, using 20 denoising steps and a guidance scale of 7.5.Source: §5.2.1 · pp. 18, 20

5.2.2 Interpolation.

  • For SDXL interpolation, the system re-noises the weighted average of fully denoised latents and denoises again to produce a coherent sample; the resulting map clusters samples approximately by style.Near-real-time interaction was limited by hardware: pair and steer operations could require several seconds for trajectory denoising, and the use case demonstrates framework generalization beyond GANs.Source: §5.2.2 · p. 20, §5.2.3 · p. 20, §5.2.4 · p. 20

5.3 Case 3: Formative case study on Rotational Kirigami Systems

The rotational-kirigami case applies PolyGandr to a quadrilateral hub-design space, pairing broad map-based exploration with direct geometry edits linked to a solver. A formative expert walkthrough found that this supports a two-stage workflow while exposing the need for stronger feasibility guidance.

5.3 Case 3: Formative case study on Rotational Kirigami Systems

  • The case study explores quadrilateral hubs at the unit level, leaving the later task of arranging compatible units into a complete space frame out of scope.For a prescribed deployment height, changing the hub polygon changes leg geometry, cut pattern, and deployed geometry.Source: §5.3 · p. 22, §5.3.1 · p. 22

5.3.1 Quadrilateral Dataset

  • The dataset contains 20,000 convex quadrilaterals represented by eight vertex coordinates and canonicalized for translation and cyclic vertex re-indexing.UMAP projects this representation into two or three dimensions for interactive exploration.Source: §5.3.1 · p. 22

5.3.2 Geometric Model and Leg Placement Solver

  • The geometry-based overview supports broad navigation across hub shapes, while a linked polygon editor and solver recompute each edited unit’s construction and feasibility.The solver checks geometric and fabrication constraints, including intersecting cuts, severed material regions, and insufficient hinge widths.Source: §5.3.2 · pp. 23–24, Figure 19 · p. 23

5.3.3 Interview Process

  • A recorded 40-minute walkthrough let one rotational-kirigami expert explore 2D and 3D projections, navigate the design space, edit geometries, and discuss difficulties and needed controls.The expert had worked on rotational kirigami systems for five years; the case was treated as formative.Source: §5.3 · p. 20, §5.3.3 · p. 24

5.3.4 Findings

  • The expert used the global map to explore substantially different geometries, then shifted to direct vertex editing to refine a promising configuration.The 2D projection helped orient the expert; the 3D view became more useful for spatial context and finer control once the 2D organization was understood.Source: §5.3.4 · p. 24
  • The expert preferred direct vertex editing to interpolation because refinement required changing a candidate while preserving a feasible kirigami construction.Edits change the cut pattern and deployed geometry and can make the unit infeasible.Source: §5.3.4 · p. 24
  • When the solver reported infeasibility, the expert adjusted height, twist, or vertices largely by trial and error; small vertex changes could often restore feasibility.The expert proposed indicating nearby feasible designs, suggesting geometry changes toward feasibility, and recommending constraint relaxations.Source: §5.3.4 · pp. 24–25
  • The expert suggested organizing the overview by deployed geometry and engineering behavior, including height, shape, stiffness, buckling, and hub rotation under load.The proposed additions also include showing physical consequences in the map and exposing feasible alternative leg placements.Source: §5.3.4 · p. 25

6 Discussion and Future Work

The discussion characterizes the interpolation scheme’s reach and trade-offs, then considers automated mappings, alternative views, constraints, and steering beyond latent spaces. It cautions that linear steering may be locally valid and model dependent.

6.1 Characteristics of interpolation

  • With fixed anchors, the interface traces only a 2D subset of their convex hull, so it cannot generally navigate the full latent space.The induced surface depends on the weighting function, anchor choices, and their 2D layout; finite-support weights restrict mixtures to nearby anchors.Source: §6.1 · pp. 25–26

6.2 Inverse Design of Local Refinement Layout

  • Manual interpolation design can entail trade-offs or miss task-specific properties, while a single 2D or 3D local layout cannot capture all meaningful variation as intrinsic dimensionality rises.The paper proposes inverse-designing the mapping from explicit desiderata to make trade-offs explicit and tailor mappings to models and use cases.Source: §6.2 · p. 26

6.3 Decoupling Latent Space Structure from Navigation

  • The interface could separate understanding latent-space structure from navigating it, using dimensionality-reduction views for exploration and a consistent mixing palette for steering.The formative study suggests users benefit from both 2D and 3D representations.Source: §6.3 · p. 26

6.4 Boundary Conditions and Constraints

  • Unrestricted navigation can enter regions with invalid or undesirable outputs; the paper proposes constraint-aware movement near valid-data manifolds or model-density boundaries.Possible approaches include projecting movements onto valid regions, adjusting directions near boundaries, or indicating trajectories approaching infeasible areas.Source: §6.4 · p. 27

6.5 Non-Euclidean Curved Spaces

  • Straight-line steering can leave a curved valid-output manifold and pass through low-density regions, making its validity local and model dependent.The paper notes that locally meaningful directions may not remain semantically consistent elsewhere, motivating locally varying directions or manifold-following paths.Source: §6.5 · p. 27

6.6 Beyond Latent Space

  • The base-plus-direction-plus-step-size framework can also describe steering through activations, parametric inputs, or model parameters, not only latent codes.A direction may be a semantic vector, activation difference, or low-rank parameter update.Source: §6.6 · p. 27

7 Conclusion

PolyGandr links global exploration and local refinement to help users discover and revise steering operations. Across StyleGAN2, Stable Diffusion XL, and rotational kirigami, the paper demonstrates generative-model exploration and domain-specific use while identifying limits and opportunities for guidance.

7 Conclusion

  • PolyGandr combines a global overview and local mixing view so users can discover contrasts, construct steering operations, inspect trajectories, and revise their defining examples.The operation uses a base point, direction, and step size.Source: §7 · p. 27
  • Across StyleGAN2, Stable Diffusion XL, and rotational-kirigami design, the framework supports generative-model exploration and domain-specific uses, while exposing local-validity, latency, and feasibility-guidance challenges.The paper presents linear vector arithmetic as an approximation users can explore and refine to expose where it succeeds or breaks down.Source: §7 · p. 27
  • Generative-model interfaces can treat steering as iteratively navigating, inspecting, and revising transformations across a structured generative landscape.This conclusion frames steering as more than parameter adjustment.Source: §7 · p. 28

Related papers

Loading 2610.37446v1…