PaperlayerView exact revision on arXiv

Source-linked AI summary

GenCtrl -- A Formal Controllability Toolkit for Generative Models

Emily Cheng, Carmen Amo Alonso, Federico Danieli, Arno Blaas, Luca Zappella, Pau Rodriguez, Xavier Suau

arXiv:2601.05637v1cs.AIcs.LGcs.SYeess.SY

TL;DR

Generative-model control methods often assume controllability, but formal ways to verify that assumption are lacking. GenCtrl models dialogue as a control process and estimates controllable sets with statistical guarantees. Experiments find controllability fragile and dependent on the model, task, and initial state.

  • Problem

    Controlled-generation methods often assume reachability, universal controllability, and calibration, while formal tools to verify these assumptions are lacking.

    Source: §1 · p. 1

  • Method

    GenCtrl formalizes dialogue as a control process and estimates reachable and controllable sets for opaque models using Monte Carlo algorithms with PAC bounds.

    Source: §1 · pp. 1–2

  • Results

    Controllability is not guaranteed and varies across modalities, models, and tasks; experiments also find it sensitive to task semantics and initial state.

    Source: §1 · p. 2, §6.3 · p. 10, §7 · p. 10

  • Takeaways & Limitations

    The findings support treating controllability as an explicit object of analysis rather than an implicit assumption.

    Source: §7 · p. 10

    Guarantees apply only to the practitioner-defined input distribution, readout map, and initial-state distribution, and do not transfer to new choices.

    Source: §7 · p. 10

Abstract

from arXiv · show

As generative models become ubiquitous, there is a critical need for fine-grained control over the generation process. Yet, while controlled generation methods from prompting to fine-tuning proliferate, a fundamental question remains unanswered: are these models truly controllable in the first place? In this work, we provide a theoretical framework to formally answer this question. Framing human-model interaction as a control process, we propose a novel algorithm to estimate the controllable sets of models in a dialogue setting. Notably, we provide formal guarantees on the estimation error as a function of sample complexity: we derive probably-approximately correct bounds for controllable set estimates that are distribution-free, employ no assumptions except for output boundedness, and work for any black-box nonlinear control system (i.e., any generative model). We empirically demonstrate the theoretical framework on different tasks in controlling dialogue processes, for both language models and text-to-image generation. Our results show that model controllability is surprisingly fragile and highly dependent on the experimental setting. This highlights the need for rigorous controllability analysis, shifting the focus from simply attempting control to first understanding its fundamental limits.

1 Introduction

GenCtrl addresses the unexamined assumption that generative models are controllable by formalizing control and estimating controllable sets with probabilistic guarantees. Its experiments find controllability is not guaranteed and varies across modalities, models, and tasks.

Motivation

  • Existing control methods often assume reachability, universal controllability, and calibration, but tools to formally verify these assumptions are lacking.Calibration means the output is a direct function of the control variable.Source: §1 · p. 1

Approach

  • The paper frames human-model interaction as a control process and develops a framework to quantify reachability and controllability with probabilistic guarantees.It treats generative models as opaque nonlinear systems and supports discrete or continuous input and output spaces.Source: §1 · p. 1

Contributions

  • The framework formalizes reachability and controllability for opaque systems, including generative models, to characterize their operational boundaries.The authors describe it as the first formal language for this purpose, to their knowledge.Source: §1 · p. 2
  • Monte Carlo algorithms estimate reachable and controllable sets from opaque system interactions, with PAC confidence bounds.They are distribution-free, assume only boundedness of the target attribute, and account for discrete bottlenecks.Source: §1 · p. 2
  • Controllability is not guaranteed, and results vary substantially across modalities, models, and tasks, motivating case-specific analyses.The empirical analysis covers LLM text generation and text-to-image generation.Source: §1 · p. 2
  • The authors release a PyTorch toolkit intended to facilitate controllability analysis by the broader research community.The passage gives the project repository as https://github.com/apple/ml-genctrl.Source: §1 · p. 2

Conclusion

  • The paper argues that controllability should become an explicit subject of analysis rather than an implicit assumption.It presents formal tools as a foundation for future work in controllable AI.Source: §1 · p. 2

2 Related Work

Related work spans prompting, finetuning, and representation engineering, while existing reachability methods face limitations for opaque generative models. GenCtrl adapts sampling-based estimation to address these gaps.

Controlling generative models

  • Generative-model control uses prompt engineering, finetuning, and representation engineering; this work focuses mainly on prompting.The framework can assess any of these control mechanisms.Source: §2 · p. 2

Control theory and reachability analysis for generative models

  • Existing data-driven reachability methods may target states rather than outputs, require unverifiable dynamics assumptions, or yield vacuously large continuous estimates for countable outputs.The authors identify these as barriers to applying prior methods to LLMs and text-to-image models.Source: §2 · p. 2
  • GenCtrl develops sampling-based estimates of non-vacuous reachable and controllable sets for generative models with probabilistic guarantees.The cited passage positions this approach as bridging limitations in prior methods.Source: §2 · p. 2

3 Problem Formulation

The paper models dialogue with generative models as a discrete-time control process, then defines which measured outputs are reachable or controllable. Its formulation accounts for the countable reachable sets induced by discrete prompts.

3 Problem Formulation

  • A dialogue process models LLM or text-to-image interaction as a control process; Figure 1 depicts its schema and an example.The figure caption says Theorems 1 and 2 specify sample counts for inputs and initial states to answer Q1 and Q2 with confidence δ.Source: §3 · p. 2, Figure 1 · p. 3

3.1 Dialogue with a Generative Model as a Control Process

  • Time advances in discrete turns, with initial states drawn from contexts and later states representing model generations.For LLMs the state space is strings; for text-to-image models it includes strings and images.Source: §3.1 · p. 3
  • Control inputs are user or designer interventions, principally prompts here, that affect generation and define the model's dynamics.The intervened model maps the current and prior states and inputs to the next generation.Source: §3.1 · p. 3
  • A task-specific measurement and deterministic readout map evaluate model behavior, such as output length or the number of cars in an image.Controlled generation aims to restrict measurements to a desired subset.Source: §3.1 · p. 3

3.2 Reachability & Controllability in Dialogue Processes

  • Reachability asks what outputs some control sequence can achieve from a fixed initial state over t steps.The output set is defined over the measurement space for a specified initial state.Source: §3.2 · p. 4
  • Controllability asks whether every initial state can reach the same desired output set at some time using an input sequence.The paper distinguishes this question from designing an input that elicits a particular result.Source: §3.2 · p. 4
  • Discrete prompts make reachable sets countable, a bottleneck affecting both reachability and controllability; the paper proposes coarse-grained reachability and PAC estimates.The estimation plan first addresses reachability from a given initial state, then controllability of the system.Source: §3.2 · p. 4

4 Monte Carlo Reachability for Dialogue Processes

The section develops sampling-based reachable-set estimation for dialogue processes, adapting reachability to discrete or quantized measurements and deriving PAC guarantees. Its algorithms use sampled trajectories without requiring knowledge of the model’s dynamics.

4 Monte Carlo Reachability for Dialogue Processes

  • Because generative models have unknown nonlinear dynamics, the framework estimates reachable sets from empirical trajectories; the ∞-norm also supports multiple measurement attributes.The stated examples include formality and sentence length as separate readout dimensions.Source: §4 · p. 4, §4.2 · p. 5

4.1 Coarse-Grained Reachability Overcomes the Discrete Bottleneck

  • The discrete bottleneck makes continuous-set estimators vacuously large, so continuous measurements are quantized with error margin γ while categorical measurements need no abstraction.The quantized estimate targets values within γ of a reached measurement.Source: §4.1 · pp. 4–5

4.2 Sample complexity bound for coarse-grained reachable set estimation

  • The sample complexity bound requires a finite measurement discretization and retains bins whose probability mass is at least p.For continuous measurements, Yq is a minimal cover using ∞-balls of radius γ/2; categorical measurements use Yq = Y.Source: §4.2 · p. 5
  • After m samples, the estimate covers all reachable bins with confidence 1 − δ; omitted target sets are unreachable with probability at least 1 − δ.For γ-quantized measurements, every estimated point is within γ of a reached point.Source: §4.2 · p. 5

4.3 Monte Carlo Sampling Algorithm for Reachability in Dialogue Processes

  • With the theorem’s sample-size requirements, reachability estimation has confidence 1 − δ without model or dynamics information, and m scales as O(N log N).The trajectory count does not depend on whether the model is stochastic or on the timestep.Source: §4.3 · p. 6

5 Monte Carlo controllability for dialogue processes

The section estimates controllability by intersecting reachable sets sampled from initial states, defining partial controllability over measurement bins. It provides PAC guarantees and selects sampling confidences to control total sample use.

5 Monte Carlo controllability for dialogue processes

  • The estimated controllable set is formed by intersecting reachable sets from k sampled initial states, approximating targets reached by a large proportion of initial states.Partial controllability is defined over measurement bins using the proportion of initial states that p-approximately reach each bin.Source: §5 · p. 6
  • Successive intersections shrink the running estimated set, while the estimate remains a strict overapproximation with no false negatives.The supplied passage states that the estimated set contains the target controllable set by construction.Source: §5 · p. 6
  • Theorem 2 is distribution-free, requires Yq to be finite and consistent across initial states, and applies to stochastic control systems.The theorem supplies a PAC bound for the partially controllable set.Source: §5 · pp. 6–7

5.1 Monte Carlo sampling algorithm for controllable sets

  • The algorithm allocates δ between initial-state sampling and reachable-set estimates to meet the overall confidence while minimizing total samples n = m · k.The overall confidence is 1 − δ := (1 − δC)(1 − δR)^k.Source: §5.1 · p. 7

6 Experiments

Across these experiments, controllability varies with the task, model, and prompting setup, and no single model or strategy guarantees control across all tasks. The section evaluates dialogue control using estimated controllable-set coverage and calibration metrics, with examples spanning language and text-to-image models.

6 Experiments

  • Experiments test greedy-decoded LLMs and text-to-image models; controllable-set coverage measures target reachability, while ρ, R, and MAE proxy monotonicity, linearity, and identity.The formalities setup tests 0-shot and 5-shot prompts, while subsequent feedback incorporates the previous output.Source: §6 · p. 7, §6.1 · p. 7

6.1 Controlling Formality in LLMs

  • With formality control, no model is fully controllable after five 0-shot turns; Qwen3-4B and Gemma3-4B reach full coverage after five 5-shot turns, unlike SmolLM3-3B.Qwen3-4B is the most faithful to user requests among the fully controllable models in this setting.Source: §6.1 · p. 8
  • Prompting effects vary by model: five-shot examples help Qwen3-4B and Gemma3-4B more than five-turn feedback, while SmolLM3-3B shows the reverse; dialogue trajectories also overshoot targets.The overshoot is more visible with five-shot prompting for Qwen3-4B and Gemma3-4B.Source: §6.1 · p. 8
  • In five-turn Qwen tests, controllability and calibration improve with size but calibration saturates at 4B; in one-turn tests, R plateaus around 8B.In the five-turn setting, even the 14B model has MAE ≈0.25 against an error tolerance γ = 0.1.Source: §6.1 · p. 8, Figure 3 · p. 9

6.2 Controlling Number of Objects in T2Is

  • For text-to-image object counting across 80 COCO classes, FLUX-s has MAE 3.52 and ρ, R > 0.9, while performance varies substantially by requested object.The task prompts for 0–20 objects on a white background; the authors describe counting as harder than expected.Source: §6.2 · p. 9, Figure 4 · p. 9

6.3 Additional Experimental Results

  • Additional tasks confirm strong task dependence: Gemma3-4B is well calibrated on integer and string tasks but poorly calibrated on formality, while image saturation is not controllable.Text-to-image object-location control is poor, and controllability and calibration also improve with model size on string-length and average-word-length tasks.Source: §6.3 · p. 9
  • No single model or prompting strategy guarantees control across tasks: even strong formality results coexist with object-count errors and failed object-location control in text-to-image models.The authors frame these tasks as a lower bound on real-world application complexity and use the findings to validate the framework's utility for identifying failures.Source: §6.3 · pp. 9–10

7 Limitations and Conclusion

The paper presents a formal toolkit for estimating controllability, while its experiments show that controllability depends on the model, task, and initial state. Its guarantees and practical conclusions are bounded by the chosen system definition, black-box access, and the complexity of the reachable set.

Limitations

  • Guarantees apply only to the control system defined by the practitioner’s input distribution, readout map, and initial-state distribution; they do not transfer to new choices.Accordingly, experimental takeaways apply only to the setting evaluated.Source: §7 · p. 10
  • Because the framework treats models as black boxes, it cannot causally diagnose controllability failures on model internals.The sampled trajectories can reveal which inputs elicit which measurements and regions that are systematically uncontrollable.Source: §7 · p. 10
  • Reachable-set sample complexity scales with covering number N, and high-precision estimation remains difficult for intrinsically complex sets.The authors suggest quantizing the attribute space to a tractable number of bins, such as two safe-versus-unsafe regions.Source: §7 · p. 10

Conclusion

  • Controllability is fragile and highly dependent on the model, task, and initial state, motivating explicit analysis rather than treating it as a given.The proposed toolkit is intended to support safer and more reliable controllable AI.Source: §7 · p. 10
  • The framework estimates controllable sets for opaque models with statistical guarantees, and its experiments show that controllability is not guaranteed.The authors also describe possible uses in safety and compliance, including comparing control mechanisms and estimating reachable sets under adversarial inputs.Source: §7 · p. 10

Reproducibility Statement

The reproducibility materials include open-source libraries and public datasets, with code and implementation materials planned for public release. The paper also points readers to its algorithms, experimental setup, task input distributions, and package example.

Reproducibility Statement

  • The work is based on open-source libraries and public datasets.The authors state that code will be made publicly available on GitHub and as a Python package.Source: Reproducibility Statement · p. 11
  • The main Monte Carlo reachability and controllability algorithms are provided in Algorithm 1.Experimental setup details are in Section J, with task input distributions in Section J.1.Source: Reproducibility Statement · p. 11
  • Section K includes a sample of the Python package.The package is also identified as a planned public code-release format.Source: Reproducibility Statement · p. 11

Ethics Statement

The ethics statement argues that controllability should be analyzed rather than assumed, given the reported fragility of model control. The appendix supplies extended results and technical details, including experimental settings for prompting and decoding.

Ethics Statement

  • The authors argue that fragile controllability calls for shifting attention from simply controlling models to understanding their fundamental limits.They regard this shift as necessary for more robust, transparent, and safe generative models.Source: Ethics Statement · p. 11

Appendix

  • The appendix collects supplementary experiments, formal definitions, guarantee interpretations, proofs, sample-complexity analysis, and package information.It also covers discrete bottlenecks, expected outputs, hypothesis testing, and experimental setup.Source: Appendix · p. 15

A.1 Setup

  • The experiments span 0-shot and 5-shot dialogue prompting and stochastic or deterministic decoding; the appendix shows 5-shot controllability summaries and deterministic trajectories for both prompting settings.The authors report choosing 5-shot summaries because that setting gave the best results.Source: §A.1 · p. 15

A.2 Results: Deterministic systems

This section presents results on controllability in deterministic and stochastic generative models and formalizes how reachable-set structure and estimation guarantees depend on output type and system stochasticity. It gives a probability guarantee for continuous-valued targets and a sample-complexity bound for finite discrete reachability.

A.2 Results: Deterministic systems

  • Deterministic results show task-dependent LLM controllability, while T2IMs struggle with requested object placement despite FLUX-s spanning the measured output space; coverage does not ensure perfect control.Gemma3-4B excels on formality and average word length but does not cover the full measurement-value space for even/odd outputs; FLUX-s reaches the whole measured space with maximal coverage, yet placement accuracy remains imperfect.Source: §A.2 · pp. 15, 17, Figure A.7 · p. 16, Figure A.10 · p. 17

A.3 Results: Stochastic systems

  • Stochastic sampling makes output distributions more diffuse and faithfulness metrics worse than in deterministic systems, with the same sample complexities.The reported stochastic summaries cover formality, average word length, string length, and even/odd outputs.Source: §A.3 · p. 18, Figure A.13 · p. 19

A.4 Example qualitative analysis: Qwen, formality

  • For formality, Qwen3-4B adapts the closest-shot example’s theme and wording, but its response to a 96.31% request is too formal.The qualitative examples are from three sampled trajectories in the 5-shot, 5-turn setting.Source: §A.4 · p. 20, Table A.1 · p. 21, Table A.2 · p. 21

C Reachability and Controllability for deterministic vs. stochastic systems

  • Under deterministic dynamics and control, α,p-controllability guarantees an exact or within-γ target for at least 1−α of initial states; under stochastic dynamics it instead concerns the output distribution, while E[y] admits reachable-set bounds.The expected-output bound is given as Cor. G.1, and the controllability procedure can use that reachability-estimation method.Source: C Reachability and Controllability for… · p. 23

D.1 Comparison to our Thm. 1

  • Unlike DA19’s overapproximating interval, this method returns a set guaranteed within γ to contain a probabilistically approximate reachable set.DA19’s approach is adapted from state to output reachability and from continuous-time to discrete-time systems.Source: D Theorem 1 of Devonport and Arcak… · p. 23, §D.1 · p. 24

E Discrete bottleneck in dialogue processes

  • Deterministic LLM controllable sets are bounded by min(|Σ|^τ, |U|^τ), deterministic T2IM sets by |Σ*|×|U| under prompting, stochastic LLM sets by |Σ|^τ, and stochastic T2IM sets may be uncountable.The deterministic T2IM bound reflects finite-length discrete prompting; continuous inputs such as activation steering remove that discrete bottleneck.Source: §E.1 · p. 25, §E.2 · p. 25

E.3 Expected outputs controllability

  • For stochastic models, output attributes remain bottlenecked, but their expectations are limited by input cardinality; stochastic LLM expected measurement values have controllable sets of size at most |U|^τ.Here τ is the number of generated tokens for an LLM and equals 1 for a T2IM.Source: §E.3 · p. 25

F.1 Theorem 1: continuous-valued version

  • The γ-quantized reachability theorem bounds sample complexity for continuous bounded outputs using their covering number N, requiring samples to hit all N bins with probability at least 1−δ when each bin has probability above p.The supplied theorem passages give the setup and bin-hitting condition but do not show the explicit sample-complexity expression.Source: §F.1 · p. 26
  • With probability at least 1 − δ, all N bins of radius γ/2 are hit under the theorem’s sample condition.The proof bounds the probability that any bin remains unhit using a union bound.Source: §F.1 · p. 26

F.2 Corollary of Theorem 1: categorical version

  • For discrete Y with finite cardinality N, Corollary F.1 states a sample-complexity bound for categorical reachability.Source: §F.2 · p. 27

F.3 Sample complexity of Theorem 1

The section examines sample-complexity behavior, validates Theorem 1 empirically, and extends reachability and controllability estimation to expected outputs and sampled initial states. It also describes how to select confidence parameters to minimize total sampling cost.

F.3 Sample complexity of Theorem 1

  • Sample complexity rises most sharply as p decreases, also rises as δ approaches zero, and is relatively robust to increasing N above about 50; at (δ, p, N) = (0.05, 0.05, 100), m = 149.The discussion sketches an O(N log N) dependence on N.Source: Figure F.14 · p. 27, §F.3 · p. 27

F.5 Setup

  • The validation tests stochastic dynamics with γ-quantized reachability on Gemma3-4B for text formality, using 0-shot prompting, T = 1, and BOS as the initial state.The true output distribution is proxied with 10,000 sampled and scored responses.Source: §F.5 · p. 28

F.4 Empirical validation of Theorem 1

  • Theorem 1 recommends m=104 samples for 95% confidence; the observed minimum was around 40, making the bound roughly 2.5× inefficient in this setting.The validation uses 200 runs and checks empirical confidence against the requested guarantee.Source: §F.5 · p. 28

G Expected output reachable set

  • For stochastic dynamics, expected-output reachability avoids the discrete bottleneck; the estimate uses sampled means bounded with Hoeffding’s inequality.The paper presents this result as a corollary estimating the reachable set of expected outputs.Source: G Expected output reachable set · pp. 28–29

H.1 Output controllable set

  • The output controllable set at time t is the intersection of the reachable sets across initial states.It is defined as a subset of the output space.Source: §H.1 · p. 29

H.2 Proof of Theorem 2

  • With enough sampled initial states, the empirical intersection bounds the error from bins that survive sampling despite having mass below 1 − α, while combining reachable-set and initial-state confidence guarantees.The proof conditions on all sampled reachable sets containing the true target.Source: §H.2 · p. 30

H.3 Auto-parameter selection for Theorem 2

  • Given a target overall confidence, the procedure selects δR and δC to minimize n = m · k, trading reachable-set samples against initial-state samples.The parameter search finds a minimum satisfying Theorems 1 and 2 and the overall confidence constraint.Source: §H.3 · pp. 30–31

H.4 Sample complexity of Theorem 2

The section analyzes how Theorem 2’s sample requirements vary with hyperparameters and notes that confidence-set estimates can support hypothesis testing.

H.4 Sample complexity of Theorem 2

  • For α and p near 0.001, total sample complexity approaches or stays below 10^7; the number of initial-state samples is most sensitive to α.The number of initial-state samples is comparatively robust to δC and ϵ.Source: §H.4 · p. 31, Figure H.16 · p. 31, Figure H.17 · p. 32
  • Figure H.17 varies total sample count n across δ, ϵ, α, p, and N, one fixed hyperparameter at a time.The other parameters are held at 0.05, except N, which varies.Source: Figure H.17 · p. 32

I Hypothesis testing

  • Theorem 1 and Theorem 2 confidence sets can be used for hypothesis testing.The section introduces this use through an example for reachable sets.Source: I Hypothesis testing · p. 33

J Extended experimental setup

The extended setup lists experimental hyperparameters and task input distributions, then provides a Python toolkit for controllability and reachable-set estimation. The authors also describe their use of AI writing assistance.

J Extended experimental setup

  • Tables J.4 and J.5 provide hyperparameters for controllability experiments and model sampling, including stochastic LLM sampling and Mistral initial-state generation.The table captions identify the scope of the settings but do not provide specific values here.Source: Table J.4 · p. 33, Table J.5 · p. 33

J.1 Input distributions

  • The input distributions cover dialogue tasks such as integer generation, string length, formality, and word length, as well as text-to-image object and position prompts.Examples include instructions for generating a strictly positive integer and placing an object at a specified image position.Source: §J.1 · pp. 33, 34, 36

K Controllability package

  • The open-source PyTorch package supports local or HuggingFace models for initial-state and input distributions and readout maps, with APIs for reachability and controllability experiments.The example defines a control system and then constructs reachability and controllability verifiers.Source: K Controllability package · p. 36

L Use of AI Writing Assistance

  • The authors report using an LLM to refine language, grammar, and clarity, with human authors reviewing and approving suggestions and retaining responsibility for the manuscript.They state that the content was originally drafted by the human authors.Source: L Use of AI Writing Assistance · p. 37

Related papers

Loading 2601.05637v1…