PaperlayerView exact revision on arXiv

Source-linked AI summary

Structural and Representational Agency in LLM-Driven Timbre Generation and Interpolation

Joseph Cameron, Alan Blackwell

arXiv:2610.12532v1cs.SDcs.HCcs.LGcs.MMeess.AS

TL;DR

Creative interfaces can present language-model predictions as measured user-position displays, while most creative domains lack an agreed reference for checking them. This study measures a working LLM-driven sound-design tool against audio-derived references and analyzes its reachable space. The displayed position is substantially inaccurate, while the graph’s structural constraints preserve the user’s control over the sounds available.

  • Problem

    Creative interfaces may present language-model predictions of an artefact as measured user-position displays, and most creative domains lack an agreed reference for checking those reports.

    Source: §1 · pp. 1–2

  • Method

    The study characterizes a sound-design tool’s reachable set and compares its reported positions and refinement outcomes with measurements of rendered audio.

    Source: §1 · p. 2

  • Results

    The displayed position is wrong by roughly three times the width of the user’s reachable region, and most refinement instructions produced no audible change while reporting success.

    Source: §10 · p. 6

  • Takeaways & Limitations

    The user-authored graph structurally constrains what the system can produce, but the accuracy of the displayed position depends on the model’s report.

    Source: §8 · p. 6

    The study has no human participants and examines one system, synthesiser, and feature space, with small sample sizes throughout.

    Source: §9 · p. 6

Abstract

from arXiv · show

Generative creative interfaces routinely report the user's position within a space of possibilities. Such displays imply that the quantities shown were measured from the artefact; in a growing class of systems they are instead predicted by a language model describing its own output, and the two are indistinguishable on screen. Most creative domains lack an agreed measurement against which such a self-report can be checked; audio admits one. In a measurement study of a working LLM-driven timbre interface, what a user can cause proves separable from what the interface reports. The first is exactly characterisable: the reachable set is a zonotope whose dimension increases by one per sound placed, so the space the model may act within is demonstrably the user's own. The second is inaccurate. The displayed position differs from measurement by approximately three times the width of that region, and the error has two independent sources: model-predicted coordinates, whose accuracy varies twofold between the two models evaluated, and the interpolation step, which arises from combining synthesiser parameters and is unaffected by model quality. End to end, 26 of 36 refinement instructions produced no perceptible change, and under the weaker model the facility performed no action while reporting convergence. Effective agency (what a user can cause) and epistemic agency (whether they can know it) are therefore separable and independently engineered: structurally enforced constraints survive model failure, whereas guarantees resting on the model's own report do not. The failure mode that remains is silence presented as success. We close with a short protocol for building and auditing such displays.

1 Introduction

Some creative interfaces present language-model predictions of an artefact as if they were measured, while most creative domains lack an agreed reference for checking those reports. This study uses audio measurement to examine how such displays relate to user agency.

1 Introduction

  • Language-model descriptions of generated artefacts can be displayed as measurements, although the quantities were predicted rather than measured from the artefact.The interface presents predicted and measured coordinates identically, while prompting is computationally negligible compared with measurement infrastructure.Source: §1 · pp. 1–2
  • When instrumentation derives from the model rather than the artefact, users’ control and knowledge of that control can come apart.If the display also serves as an optimiser’s objective, the system pursues a target that does not correspond to the artefact, without indicating error.Source: §1 · p. 2
  • The study treats a working sound-design tool as an instrument for testing model-derived displays because audio admits objective measurement.It asks what output region users can reach, whether their position is reported correctly, and whether refinement instructions change the audio.Source: §1 · p. 2
  • The paper characterises the reachable set as a zonotope, measures display fidelity against rendered audio, and evaluates refinement against a measured noise floor.It decomposes display error into model-dependent and model-independent terms and identifies a silent failure mode.Source: §1 · p. 2

2 Background and related work

Prior work addresses user control, trust in automation, model evaluation, and perceptual timbre measurement. The paper distinguishes the user’s ability to cause an outcome from their ability to know what they are causing.

Control and agency in creative systems

  • Creative-systems research largely treats agency as what users can cause, whereas this paper distinguishes it from whether users can know what they cause.Related work covers mixed-initiative interaction, human-AI guidelines, creativity support, music steering, ownership [5], and subjective authorship.Source: §2 · p. 2

Trust, reliance and unfaithful displays

  • Trust research links appropriate reliance to accurate signals about system capability, while over-reliance can persist despite explanations [8].Here, the potentially miscalibrating signal is the coordinate system itself, whose reliability changes with the selected language model.Source: §2 · p. 2

Models reporting on their own output

  • This study measures the consequences of embedding a model’s perceptual judgements of synthesiser-generated sound in a live interface as instrumentation.Prior work examined language-model evaluator bias [9] and alignment of model judgements with human judgements [10].Source: §2 · p. 2

Timbre semantics and measurement

  • The reference measurements use AudioCommons models trained on human listening data to predict seven timbre ratings from audio.The ratings approximate rather than replace perceptual judgement; stimuli follow the THINGS procedure, and the dimensions have an empirical basis.Source: §2 · p. 2

3 System description

Timbre Canvas turns text prompts into synthesiser patches and lets users interpolate among sounds they have elicited. Refinement targets are model-proposed, but the model can act only within the user-built graph.

3 System description

  • Timbre Canvas makes each prompted synthesiser patch a canvas vertex and continuously blends connected sounds by interpolating their synthesiser parameters.Bead positions set the relative contributions of connected vertices; mutually connected triples form patches.Source: §3 · p. 3
  • For natural-language refinement, the model proposes a target in seven AudioCommons dimensions, and coordinate descent adjusts beads along existing edges.The model cannot introduce material outside the graph of sounds elicited by the user; the paper calls this arrangement enclosure.Source: §3 · p. 3
  • The interface obtains each vertex’s AudioCommons vector in closed form from the model’s prediction, although it renders and measures audio for a separate feature family.The paper examines the consequences of this asymmetry in Section 6.Source: §3 · p. 3

4 Measurement setup

The measurement setup uses normalized seven-dimensional AudioCommons coordinates, a preset corpus and a frozen concept set as references, and two models across three naturally authored canvases. Repeated renders establish a per-dimension noise floor for interpreting changes.

4 Measurement setup

  • All positions use seven AudioCommons dimensions normalized to [0, 1], with Euclidean distance unless otherwise stated.Two random presets differ by 0.68 on average, and a random preset differs from the corpus mean by 0.50.Source: §4 · p. 3
  • The reference corpus contains 10,292 measured synthesiser presets, with a median nearest-neighbour distance of 0.0765.That distance provides a natural resolution for interpreting feature-space distances.Source: §4 · p. 3
  • A frozen set of 156 audio exemplars covers 52 timbral concepts, with three clips per concept and all 12 raters identifying each retained clip correctly.Concept values are means of the three exemplars’ measured vectors; the set was frozen before this study.Source: §4 · p. 3
  • The study evaluates gpt-oss:20b and gemma4:31b-cloud across three canvases containing two, three, and four sounds.The canvases were built by the first author during ordinary tool use months before the study, using hosted models.Source: §4 · pp. 3–4
  • Repeated renders varied by up to 0.074 per dimension, with a mean variation of 0.024, and reported effects are thresholded against this noise floor.Without the threshold, changes smaller than re-render variance cannot be distinguished from signal.Source: §4 · p. 4

5 Reachability of the authored space

The interface’s reachable perceptual region is exactly a zonotope, with one added dimension per placed sound; connections alter control, not reach.

5 Reachability of the authored space

  • Each edge contributes 1.0 to the weight normaliser, so it equals |E| regardless of bead positions and yields a closed-form audible perceptual vector.Source: §5 · p. 4
  • Adjusting beads translates the audible point along edge segments, whose summed translations form a zonotope and make distance to a target computable without rendering audio.The closed form agrees with the application’s mixing implementation to 1.1 × 10−16, or machine precision.Source: §5 · p. 4
  • For N placed sounds, the reachable region has affine dimension min(N −1, 7); the 2-, 3-, and 4-sound canvases span 1, 2, and 3 of 7 dimensions.Among 3,000 presets, none, 4, and 6 fell within the corpus’s own resolution of those canvases; a fully connected 12-sound canvas reaches 2.3% of sampled presets.Source: §5 · p. 4
  • With six sounds fixed, increasing connections from a 5-edge spanning tree to a 15-edge complete graph leaves dimension at 5.0 and coverage within 11.9%–13.0%.Further sound placements extend reach; additional connections change how beads address the region.Source: §5 · p. 4

6 Fidelity of the displayed position

Against rendered audio, the displayed position is substantially inaccurate: its error exceeds the region’s width, with model-dependent and interpolation-related sources.

6 Fidelity of the displayed position

  • Across 51 bead configurations at the three canvases’ vertices, displayed-position error is approximately three times the navigated region’s width.Table 1 reports display error relative to rendered audio and separates the vertex and linearity distances rather than treating them as additive parts of total error.Source: Table 1 · p. 4, §6 · p. 4
  • The error has independent sources: predicted vertex coordinates and a mismatch between synthesiser-parameter mixing and perceptual-vector interpolation.The audible mix interpolates synthesiser parameters, while the display interpolates perceptual vectors; the mean parameter sets need not correspond to the mean sounds.Source: §6 · p. 5
  • Across 52 concepts, gpt-oss:20b has mean correlation r = 0.357 and error 0.515, versus gemma4:31b-cloud at r = 0.729 and error 0.375.The always-predicting-corpus-mean error is 0.476; the weaker model exceeds that baseline, while the larger model is below it.Source: §6 · p. 5
  • Replacing predicted vertices with measured coordinates leaves mean error 0.275, approximately 1.7 times the reachable extent; endpoint measurement halves error but does not remove it.The remaining linearity term is model-independent and also applies when interfaces compose model outputs and display the composed position.Source: §6 · p. 5

7 End-to-end outcomes and a silent failure mode

Across 36 refinement trials, most instructions produced no perceptible change; under the weaker model, complete failure was reported as successful convergence.

7 End-to-end outcomes and a silent failure mode

  • 26 of 36 instructions produced no change above the noise floor; 9 of the 10 perceptible changes were in the requested direction.Table 2 reports end-to-end outcomes thresholded against re-render noise, comparing gemma4:31b-cloud with gpt-oss:20b.Source: Table 2 · p. 5, §7 · p. 5
  • Every descent under the capable model ended at a local minimum because each requested target lay outside the reachable set.The trials used six known-direction instructions on each canvas under two models.Source: §7 · p. 5
  • Under the weaker model, parsing failed and fallback paths returned the current position unchanged, so the system reported convergence after zero steps without surfacing an error.No parameter was altered, presenting complete facility failure to the user as success.Source: §7 · p. 5

8 Discussion

The discussion separates users’ ability to constrain what the system can produce from the reliability of the display they use to navigate. Structural constraints hold despite model error, but silent failure and limited study scope qualify the implications.

8 Discussion

  • Users have effective agency over a precisely bounded territory, while epistemic agency is limited by a display whose error is roughly three times that territory’s width.Whether the display is merely imprecise or wholly uninformative depends on the model configuration.Source: §8 · pp. 5–6
  • Graph geometry enforces enclosure regardless of model accuracy, converting representational error into inaction rather than incorrect action.This structural guarantee does not improve performance; it keeps outputs within the user-authored space when the system acts.Source: §8 · p. 6
  • A parse failure that returns the input can make unchanged audio appear converged, so failing safely does not remove the need to report failure.The authors identify input-echo defaults as a design pattern that can generalize beyond this implementation.Source: §8 · p. 6
  • The practice recommendations include measuring the artefact, establishing a noise floor, normalizing display error by reachable-region width, and reporting parse failures instead of echoing input.The paper reports that measuring composition rather than only its parts halved display error, from roughly three times the reachable width to 1.7 times.Source: §8 · p. 6
  • The authors claim the mechanism generally but not the measured magnitudes; audio is unusual because it admits objective checking.The study does not claim its numerical results transfer to other domains.Source: §8 · p. 6

9 Limitations

The study establishes that the display is unfaithful, but does not test whether users are misled by it.

9 Limitations

  • The study has no human participants and covers one system, synthesiser, and feature space, with small samples; its reference values are regression-model outputs rather than perceptual judgements.The authors describe the reference values as appropriate internally because the system’s optimiser targets them.Source: §9 · p. 6

10 Conclusion

The conclusion distinguishes what users can cause from what they can know about it: structural constraints survive model failure, while model-reported positions and refinements can be misleading.

10 Conclusion

  • The displayed position is wrong by roughly three times the reachable-region width, and most refinement instructions produced no audible change while reporting success.The authors conclude that effective and epistemic agency are separable and identify whether users detect the difference as a key next question.Source: §10 · p. 6

A Reproducibility details

The appendix documents the models, prompts, deterministic refinement, and audio-measurement setup used in the study, alongside the corpus-derived calibration and analysis inputs.

A Reproducibility details

A.1 Models and sampling parameters

  • gpt-oss:20b and gemma4:31b-cloud were evaluated as coordinate predictors; the three canvases were authored using gemma4:31b-cloud (two) and gpt-oss:120b-cloud (one).Calls used temperature 0, a 30 s timeout, and a local Ollama server; stochastic analyses used seed 20260809.Source: §A.1 · p. 8, §A.9 · p. 10

A.2 Prompts

  • Prediction prompts were constructed programmatically and requested Timbre Toolbox and AudioCommons features, using a fixed guide and calibration examples from extreme-valued reference presets.The fixed feature guide covers nine Timbre Toolbox features.Source: §A.2 · p. 8

A.3 Coordinate descent

  • Refinement prompts map feedback to feature adjustments and return JSON; bead descent uses ε = 0.04, with loss-based stopping, while parse failures silently preserve the current state.Descent stops below 5% of starting loss, at a local minimum with no improvement greater than 10−5, or after 200 steps.Source: §A.2 · p. 8, §A.3 · p. 9

A.4 Instructions used in the end-to-end trials

  • The end-to-end trials comprised 36 instructions: six predefined timbre changes on each of three canvases under two models.Each instruction specified a target dimension and direction in advance.Source: §A.4 · p. 9

A.5 Rendering, measurement and normalisation

  • Audio was rendered as 2.0 s, 44.1 kHz mono at MIDI note 69, loudness-normalised to −23 LUFS, and measured using seven AudioCommons features mapped to [0, 1].Repeated identical renders established a mean noise floor of 0.024, with variation up to 0.074 for roughness; effects were thresholded against it.Source: §A.5 · p. 9, Table 3 · p. 9

A.6 Reachability sweep

  • min(N −1, 7) reachable dimensions are attained exactly, saturating at eight placed sounds, while the region remains a small fraction of synthesiser output.Figure 2 plots region growth for complete-graph canvases and marks the three analysed canvases.Source: Figure 2 · p. 9, §A.6 · p. 9

A.8 The 52 concepts

  • The 52-concept set was assembled independently from published vocabulary sources, with Japanese mimetics glossed before audio selection and retained clips validated by all 12 raters.Each concept has three audio exemplars; mimetics were presented using their validated English glosses.Source: §A.8 · p. 10
Loading 2610.12532v1…