Source-linked AI summary
Integrating Multi-Source Feedback in Computational Design
Francisco Erivaldo Fernandes Junior, Thomas Langerak, Mira Keränen, Danqing Shi, Ardak Alipova, Antti Oulasvirta
TL;DR
Designers combine heterogeneous feedback, but accessible ways to integrate it into HCI optimization are limited. The paper uses independent source models and a no-code interface; across six studies, the approach improved four cases and matched the baseline in two, while designers reported confidence and flexibility when using simulator feedback.
Problem
HCI practitioners lack accessible ways to incorporate multiple heterogeneous evaluation sources into optimization, despite designers’ reliance on varied feedback.
Source: §1 · p. 2Method
The paper models each evaluation source independently, aggregates its predictions to guide Bayesian optimization, and implements the approach in a no-code design tool.
Source: §1 · p. 2, §9 · p. 21Results
The multi-surrogate approach had clear advantages in four of six studies and matched the baseline in two, while visualization-task participants reported confidence and flexibility using simulator feedback.
Source: §4.7 · p. 14, §7 · p. 20Takeaways & Limitations
Independent modeling supports flexible integration of diverse signals, and the no-code interface offers a practical environment for designers to interact with sources during optimization.
Source: §1 · p. 3, §9 · p. 21Both approaches struggled when optima shifted over time, and slow simulator response and opaque metrics limited how effectively designers could use that feedback.
Source: §8 · p. 21
Abstract
from arXiv · showhide
In real-world design practice, evaluations rarely rely on a single source of judgment. Designers routinely combine expert opinions, empirical studies, and computational models, each with distinct strengths and limitations. While machine learning offers methods to integrate multiple feedback sources, these approaches remain largely inaccessible to designers without technical expertise. In this paper, we explore how to integrate multiple feedback sources, primarily through: (1) a practical approach for multi-source integration, and (2) its implementation in MUSE, a no-code tool that allows designers to combine and balance diverse sources. Our technical findings show that independent modeling of multiple evaluation sources enables exploration across heterogeneous feedback, accommodates different evaluation speeds, surfaces disagreements between sources, and supports an adaptable evaluation setup that designers can reconfigure during their process. In a visualization design study, participants navigated their own judgments alongside simulator feedback, reporting a perception of enhanced confidence and flexibility. Our results highlight the viability of multi-source integration to support computational design, offering a step toward bridging the gap between advanced optimization methods and design practice.
1 Introduction
Designers combine expert, empirical, and computational feedback, but HCI practitioners lack accessible ways to integrate heterogeneous signals into optimization. The paper develops independent source models and a no-code tool to support exploration and reflective design decisions.
1 Introduction
- Existing HCI optimization tools generally use single evaluation signals, leaving practitioners without an accessible way to combine subjective judgments, simulator outputs, and expert opinions.The paper frames this as a practical gap in supporting exploration and decision-making across heterogeneous feedback.Source: §1 · p. 2
- The approach models each evaluation source independently and linearly aggregates predictions into a fused surrogate that guides optimization.The authors emphasize flexibility for heterogeneous sources rather than predictive superiority under idealized conditions.Source: §1 · p. 2
- Muse is a no-code tool that displays designs, predictions, and source conflicts, letting designers adjust source weights or investigate problematic sources.It is intended to bring multi-source optimization into a familiar workflow without requiring knowledge of Gaussian processes or Bayesian optimization.Source: §1 · p. 2
- Independent modeling accommodates noisy and unreliable sources, different evaluation speeds, visible disagreements, and sources added, removed, or adjusted during design.The approach is presented as a technical foundation for reflective exploration across evaluations from subjective judgments to simulator outputs.Source: §1 · pp. 2–3
- Unlike multi-objective Bayesian optimization, the goal is to combine heterogeneous feedback sources, not optimize competing objectives along a Pareto front.The paper’s contributions include multi-surrogate modeling, structural flexibility with comparable outcomes to single-surrogate approaches, and the Muse interface.Source: §1 · p. 3
2 Related Work
Prior design-optimization work uses diverse evaluation sources, but often in isolation or through rigid multi-objective frameworks. This paper focuses on combining heterogeneous feedback as independent channels for interactive design.
2.1 Evaluation Sources in Design Optimization
- Design optimization uses hand-crafted objectives, human feedback, simulations, and LLM-based proxies, but these sources are typically explored in isolation.Scalar objectives can miss subjective or contextual factors; human feedback, simulations, and LLM proxies each have limits that motivate combining sources.Source: §2 · p. 3, §2.1 · pp. 3–4
2.2 MOBO in Design Optimization
- MOBO balances competing objectives, while this work treats feedback as independent channels that can have heterogeneous sampling rates and incomplete observations.The distinction is between trading off objectives and building an architecture for combining diverse evaluation sources.Source: §2.2 · p. 4
2.3 Multi-Source Surrogate-based Design Optimization
- Earlier multi-source and multi-fidelity methods generally assume numerical, aligned, directly comparable sources that can be integrated into a unified model.Those assumptions differ from HCI feedback that may be heterogeneous, asynchronous, subjective, or incomplete.Source: §2.3 · p. 4
- The proposed architecture maintains independent surrogates and combines predictions during evaluation, revealing disagreements and respecting heterogeneous sampling schedules.The setup could combine designer feedback, simulations, and LLM proxies, which the authors say has not been fully explored in computational design or HCI.Source: §2.3 · p. 5
3 Multi-Surrogate Bayesian Optimization
The method models each feedback source separately within Bayesian optimization, aggregates source predictions to guide search, and also exposes source disagreement. Its illustrative case shows how relative preferences and absolute metrics can contribute to one optimization process.
3 Multi-Surrogate Bayesian Optimization
- Each evaluation source is modeled as an independent surrogate, and posterior means are aggregated into an evaluation function that guides candidate selection and updates.The section presents this as a multi-source extension of Bayesian optimization for sample-efficient search.Source: §3 · p. 5, §3.2 · p. 6
3.1 Problem Formulation: Design Optimization with Multiple Evaluation Sources
- Each source’s GP is fit to source-specific data, and its posterior mean contributes to an aggregate that approximates idealized design quality.The formulation assumes overall quality can be represented as weighted criteria, while treating that quality as unobservable.Source: §3.1 · p. 5
3.2 Optimizing with Multiple Surrogates
- The acquisition function selects candidate batches; available sources evaluate them and update their surrogates independently, with asynchronous updates for sources operating at different speeds.The loop can stop at a fixed budget or when a designer accepts a solution.Source: §3.2 · pp. 6–7
3.3 Computing Disagreement Scores
- Disagreement is measured as variance among surrogate posterior means; designers can use visible conflicts to consider collecting more feedback or down-weighting a divergent source.The disagreement score uses equal weighting and is not itself used in optimization, unlike the weighted aggregate that drives search.Source: §3.3 · p. 7
3.4 Illustrative Case: Preference and Metric-based Sources
- In the illustrative case, preference judgments are modeled with a Plackett–Luce model and rescaled to [0, 1], while metric outputs are modeled with GP regression.This lets relative preferences and absolute scores be aggregated without first reducing them to the same representation.Source: §3.4 · pp. 7–8
4 Technical Evaluation
Six controlled studies use ground-truth functions to test multi-source optimization under varied sampling rates, source disagreement, changing sources and weights, larger source sets, and shifting objectives.
4.1 Study 1: Multi-Pace Sampling
- Across six noise regimes, multi-surrogate modeling had lower MSE in 54–72% of runs, outperforming the single-surrogate baseline in a majority of runs in each regime.Each regime used 100 independent seeds; the baseline was restricted to the slowest source’s sampling rate.Source: Figure 2 · p. 9, §4.1 · p. 9
4.2 Study 2: (Dis)Agreement Between Sources
- The disagreement score identified where source predictions diverged, while average predictive variance helped distinguish source conflict from model uncertainty.The authors suggest using these regions to focus data collection on resolving ambiguity and aligning surrogate estimates with the true objective.Source: Figure 3 · p. 10, §4.2 · p. 10
4.3 Study 3: Runtime Adjustment of Sources
- Multi-surrogate modeling achieved lower average objective values across all phases, with its advantage most apparent after source S2 was removed in Phase 4.The method reused the previously trained S2 surrogate; the baseline retaining past evaluations performed moderately well, while the baseline discarding data performed worst.Source: §4.3 · pp. 10–11, Figure 4 · p. 11
4.4 Study 4: Mid-Optimization Reweighting
- After the moderate weight shift at t=21, the multi-surrogate strategy started closer to the new optimum and maintained lower error than all three single-surrogate baselines.All methods behaved identically before reweighting; supplementary small- and large-shift results showed the same trend with lower and higher variability, respectively.Source: §4.4 · pp. 11–12, Figure 5 · p. 12
4.5 Study 5: Increasing the Number of Sources
- Across 1, 2, 5, 8, and 10 sources, both approaches produced identical optimization trajectories, and increasing source count improved convergence for both.Multi-surrogate modeling natively supports heterogeneous feedback, whereas the single-surrogate approach requires sources to be merged into one scalar function.Source: Figure 6 · p. 12, §4.5 · p. 13
4.6 Study 6: Non-Stationary Optimization
- Across all five blocks, single- and multi-surrogate methods performed the same and both struggled to track the optimum shifting every 10 steps.Neither surrogate approach addresses non-stationary objectives.Source: Figure 7 · p. 13, §4.6 · pp. 13–14
4.7 Summary of Results
- Multi-surrogate modeling outperformed the baseline in four studies and matched it in two, improving optimization with heterogeneous or dynamic sources while remaining comparable under aligned conditions.Robustness checks across seeds and noise regimes found Study 1 win rates of 54–72%; gains were strongest under moderate weight shifts in Study 4.Source: Table 1 · p. 14, §4.7 · p. 14
5 Muse: Multi-Source Evaluation
MUSE brings multi-source optimization into a no-code design workflow, where designers can configure evaluations, inspect candidates, and adjust source weights. Its toggle demonstration shows how different feedback combinations explore the design space and produce distinct outcomes.
5 Muse: Multi-Source Evaluation
- MUSE is a no-code proof of concept that combines simulator performance with designer feedback and lets designers reweight sources, tune optimization, and inspect disagreement scores.Candidate evaluations are aggregated in the backend; the system uses a ReactJS frontend and Python backend with FastAPI.Source: §5 · p. 14
5.1 Walkthrough
- The interface lets designers configure the design space and optimization, inspect visual candidates and simulator metrics, and select preferred designs as feedback.Disagreement scores indicate uncertainty or conflicts, and optimization parameters can be updated without restarting.Source: Figure 8 · p. 15, §5.1 · p. 16
5.2 Demonstration: Designing a Toggle with Three Sources
- The toggle demonstration combined a design-experienced student’s judgments, LLM evaluations, and WAVE, an automated aesthetic metric based on color, symmetry, and balance.The seven runs tested individual sources and their combinations over 48 optimization steps, with four candidate designs per batch.Source: §5.2 · p. 16
- The toggle’s seven adjustable dimensions covered color, visual style, thumb shape, track roundness, knob size, and animation duration and easing.The design space also included optional glow or shadow effects.Source: §5.2 · p. 16
- Only the Human Only run matched all of the designer’s target toggle characteristics; mixed-source runs produced alternatives shaped by their different feedback profiles.Human + WAVE balanced preferred geometry with WAVE’s color bias, while the three-source run broadened exploration rather than converging to one match.Source: Figure 9 · p. 15, §5.2 · p. 16
6 Empirical Evaluation with Designers
The formative study examined how designers use MUSE with their own judgments and simulator feedback in realistic chart-design tasks. Participants described confidence and flexibility benefits, while the exploratory ratings showed no significant effects and interviews surfaced practical constraints.
6.1 Method
- The study compared designer-only feedback with designer-plus-Chartist feedback in a within-subjects design, with task-condition pairings and order counterbalanced.Chartist predicts visual attention and scores the proportion of simulated fixations on task-relevant chart areas.Source: §6.1 · p. 17
6.1.1 Participants. / 6.1.2 Tasks.
- Fifteen experienced designers completed theme-park attendance and Paralympic-competitor chart tasks requiring iterative refinement of visual parameters.Participants had HCI or interaction-design training; programming experience was not required.Source: §6.1.1 · p. 17, §6.1.2 · p. 17
6.1.3 Procedure.
- After a tutorial and warm-up, participants completed both tasks until satisfied, then answered post-task questionnaires and a semi-structured interview.The interview covered overall impressions, satisfaction, and perceived tool value.Source: §6.1.3 · pp. 17–18
6.1.4 Measures.
- The study collected subjective user-experience feedback and 5-point Likert ratings of satisfaction with the tool and resulting designs.Source: §6.1.4 · p. 18
6.2 Results
- Simulator-only examples often ignored brief-specific colors and placed bars close together because Chartist prioritized bar-chart readability over adherence to the design briefs.Fig. 10 shows outcomes from different participants alongside the Simulator-only baseline.Source: §6.2 · p. 18, Figure 10 · p. 18
- Ratings showed no significant effects of feedback type, task, or their interaction; the authors characterize these quantitative findings as exploratory, not definitive statistical evidence.The post-hoc Wilcoxon tests found no significance or effect sizes for any questions (all p > .4 and r < .20).Source: §6.2.1 · p. 18
6.2.2 Tool preference.
- Participants said adding the simulator provided confidence and flexibility; perceived efficiency was also mentioned, but was not supported by quantitative evidence.One participant reported needing fewer rounds, while the study found no quantitative evidence for that perceived efficiency.Source: §6.2.2 · p. 19
6.2.3 System performance.
- The simulator’s slow responses curtailed some participants’ iterations, and unclear metric explanations made its feedback harder to use effectively.The authors also note that the 15-participant formative study makes the quantitative findings exploratory rather than statistically significant.Source: §6.2.3 · p. 19, §6.2.4 · p. 19
7 Discussion
The discussion frames independent surrogates and linear aggregation as a practical way to handle heterogeneous feedback while retaining designer control. The authors connect this flexibility to technical evaluations and the formative study, while noting needs for transparency and further evaluation.
7 Discussion
- Independent surrogates handle sources with different sampling rates, noise levels, domain coverage, and changing inputs, while supporting adaptive weights, visible disagreements, and source changes without restarting.The authors report these capabilities across Studies 1–4 and state that they do not sacrifice algorithmic optimization performance.Source: §7 · p. 19
- Linear aggregation with independent surrogates avoids assumptions about source interactions, keeps contributions transparent, and proved sufficient across Studies 1–4.The authors say nonlinear alternatives would add opacity without demonstrated benefit in the studied design scenarios.Source: §7 · p. 19
- Optimization gains from adding sources diminish after about five as information becomes redundant, while strategic use of partial or reweighted sources can matter more than source count.Independent surrogates can be fitted in parallel, and the authors describe their computational cost as modest relative to obtaining feedback.Source: §7 · p. 20
- Designers in the empirical study completed visualization tasks with simulator feedback and described confidence and flexibility, while emphasizing the need to understand how evaluations are produced.The study was formative, and the authors present these as observations about participants’ engagement rather than objective design outcomes.Source: §7 · p. 20
- MUSE leaves decisions about unreliable feedback to designers, who can adjust source weights or investigate sources, but this approach was not formally evaluated in the user studies.The authors identify evaluation of how designers manage unreliable feedback as future work.Source: §7 · p. 20
- MUSE presents designs, predictions, and source conflicts through the interface so designers can adjust weights without needing to know Gaussian processes.The interface prioritizes transparency and usability over exposing optimization complexity.Source: §7 · p. 20
- The authors frame multi-source optimization as structuring exploration to expose tensions and support informed choices, rather than primarily producing one best outcome.They describe lightweight integration into established workflows as a way for designers without optimization expertise to navigate alternatives and reconcile inputs.Source: §7 · pp. 20–21
8 Limitations and Future Work
The authors identify non-stationary objectives and constraints in simulator feedback as open challenges. They propose adaptation and more extensive evaluation as directions for future work.
8 Limitations and Future Work
- Both single- and multi-surrogate approaches struggled when the optimum shifted over time, while slow simulator responses and opaque metrics limited how effectively feedback informed design decisions.Source: §8 · p. 21
- Future work includes adapting surrogates to changing objectives, testing disagreement-aware acquisition, and evaluating source reweighting in larger controlled studies and real workflows.The authors also call for objective measures such as task efficiency or design quality, and for improving source transparency and responsiveness.Source: §8 · p. 21
9 Conclusion
The paper presents independent source modeling and a no-code interface as a practical approach to multi-source Bayesian optimization. Technical evaluations preserved optimization performance across heterogeneous settings, while a formative study found participants reported greater perceived flexibility and confidence.
9 Conclusion
- Independent source models combine predictions into a unified decision process, while Muse provides a no-code interface for interacting with multiple sources.The approach is intended to support workflows where feedback can be adjusted, added, or removed without disrupting optimization.Source: §9 · p. 21
- Technical evaluations found that independent modeling preserved optimization performance across varied heterogeneous source settings, including dynamic conditions.The formative study examined how designers navigated multi-source feedback while exploring visual alternatives.Source: §9 · p. 21
- Participants reported a perception of enhanced flexibility and confidence when working with combined feedback.The authors call for systematic evaluation of designer engagement and effects on final design outcomes in unconstrained workflows.Source: §9 · p. 21
- The authors frame multi-source Bayesian optimization as a viable approach for human-in-the-loop design tools that accommodate multiple evaluative signals without disrupting human intent.They position Bayesian optimization as an interactive tool for navigating complex multi-source environments, not only as a mechanism for automated convergence.Source: §9 · p. 21
A Technical Evaluation Details
The technical evaluations use synthetic functions and classical optimization benchmarks to examine heterogeneous sources, source disagreement, runtime source changes, and reweighting. Studies use repeated randomized instances and fixed budgets; known optima support objective-based evaluation.
A Technical Evaluation Details
- The technical evaluation uses synthetic functions and classical 2D benchmarks to represent design challenges such as changing priorities and evolving definitions of quality.Studies use 100 repetitions and fixed modeling budgets; known optima allow objective evaluation.Source: A Technical Evaluation Details · p. 26
A.1 Study 1: Multi-Pace Sampling
- The multi-surrogate model maintains independent GPs for S1, S2, and S3 at their native sampling rates, and combines predictions by averaging the GP means and uncertainties.The sources use distinct sampling densities: S1 has 100 samples, S2 has 10, and S3 has 20.Source: §A.1 · p. 26
- 72% win rate under Noisy S1 was the clearest multi-surrogate advantage over the single-surrogate baseline.Across All low, Default, and Noisy S3 the win rate was 65%; it was 54% for Noisy S2 and 57% for All high.Source: §A.1 · p. 27
A.2 Study 2: (Dis)Agreement Between Sources
- Disagreement is measured by variance across predicted source means, while average predictive variance distinguishes source conflict from uncertainty due to limited data.The study considers sources that observe only part of the parameter domain.Source: §A.2 · p. 28
A.3 Study 3: Runtime Adjustment of Sources
- The runtime-adjustment study compares independent source GPs retained across phases with single-GP baselines that either retain or discard past evaluations.Its four phases add S2, then S3, and finally remove S2; performance is tracked using average and best true objective values.Source: §A.3 · pp. 28–29
A.4 Study 4: Mid-Optimization Reweighting
- The reweighting study changes two source weights once at step 21 and compares restart strategies, including a multi-surrogate method with reweighted history and EI warm-start.The main comparison uses K=20 initialization points, with additional sweeps over K values from 20 to 300.Source: §A.4 · pp. 29–30
A.5 Study 5: Increasing the Number of Sources
The appendix describes evaluations of increasing source counts and changing objectives, alongside a walkthrough of Muse’s optimization interface. The studies compare single- and multi-surrogate modeling under their specified synthetic conditions.
A.5 Study 5: Increasing the Number of Sources
- The increasing-source study compares a single GP trained on averaged source outputs with one GP per source whose predictions are fused by averaging means.It varies the source count across 1, 2, 5, 8, and 10, using weighted combinations of shifted Rosenbrock functions and 100 repetitions.Source: §A.5 · pp. 30–31
A.6 Study 6: Non-Stationary Optimization
- The study compares single-surrogate and multi-surrogate methods on a shifting objective: its global minimum is s(t)+(1,1), shifting every 10 steps across five stationary blocks of 10 steps.Runs use 50 trials and 10 random initial evaluations, repeated 100 times with shared source biases across methods.Source: §A.6 · p. 32
B Muse Walkthrough
- Muse’s walkthrough shows configuration of the optimizer, simulator training, evaluation scripts, and design space, then presents candidates with evaluation metrics and source-disagreement estimates.The interface supports visual candidate selection.Source: B Muse Walkthrough · p. 32
C.1 Design Task
The design demonstration uses a Duolingo daily-reminder toggle task to explore a parameterized visual design space. The appendix describes its rendering, visualizations of search outcomes under source combinations, and an LLM-based evaluator.
C.1 Design Task
- Participants designed a settings-screen toggle for daily reminders, with a cheerful, approachable, accessible tone and hover-triggered previews.The toggle was intended to fit naturally within a settings-card layout.Source: §C.1 · p. 32
C.2 Design Parameters
- The toggle design space has seven parameters, including animation duration, border radius, thumb size and shape, easing, visual style, and color scheme.Categorical parameters were mapped to zero-indexed integers for optimization and decoded into visual styles during rendering.Source: §C.2 · pp. 32–34
C.3 Rendering Details
- A React frontend rendered parameterized toggles on a fixed 200-by-100-pixel canvas, using CSS shapes, transforms, and transitions.Polygonal clip paths implement shapes such as diamonds, hexagons, stars, and triangles.Source: §C.3 · p. 34
C.4 t-SNE and UMAP Visualizations of Toggle Designs
- Search visualizations show that Human, LLM, WAVE, and combined sources guide optimization into distinct design-space regions and produce different visual and interactive outcomes.The visualizations include t-SNE projections at perplexities 10 and 50 and a UMAP projection; dark stars mark each condition’s best solutions.Source: §C.4 · p. 35, Figure 14 · p. 35
C.5 LLM Evaluator Prompt for Toggle Demonstration
- The toggle demonstration’s LLM evaluator scores designs from 0 to 1 for brand alignment, clarity, and usability, using a prompt that specifies design and UX criteria.The evaluator is GPT-4o via Azure, prompted to act as a senior UI designer assessing a Duolingo daily-reminder toggle.Source: §C.5 · pp. 35–36