PaperlayerView exact revision on arXiv

Source-linked AI summary

FGSS-BO: Feedback-Guided Hypothesis Management for LLM-Based Bayesian Optimization

Hao Chen, Ziyang Ling, Ronald X. Xu, Mingzhai Sun

arXiv:2610.21246v1cs.CL

TL;DR

LLAMBO uses past observations to guide search but does not separately maintain identified, evidence-linked hypotheses and their test outcomes. FGSS adds a curated hypothesis state whose interventions and eligible comparisons can guide and revise search. FGSS-BO improves sample-mean outcomes on two retrospective protocol/media tasks, with mixed component effects and task-dependent HPO transfer.

  • Problem

    LLAMBO conditions its LLM calls on prior observations but lacks separately identified search hypotheses linked to evidence and later test outcomes.

    Source: §1 · p. 1

  • Method

    FGSS uses an LLM reflector and deterministic curator to maintain evidence-linked hypotheses and test records that guide candidate construction and priorities.

    Source: §1 · pp. 1–2

  • Results

    FGSS-BO achieves higher sample means on two retrospective protocol/media tasks and gains on some direct-training HPO tasks, with mixed component effects across settings.

    Source: §1 · p. 2, §7 · p. 7, §8 · p. 8

  • Takeaways & Limitations

    FGSS exposes hypotheses, their execution, and cross-round revisions for inspection while supporting search gains in selected settings.

    Source: §8 · p. 8

    Primary protocol/media results use historical-table nearest-neighbor responses and do not establish prospective laboratory performance.

    Source: §9 · p. 8

Abstract

from arXiv · show

Large language model (LLM)-guided Bayesian optimization (BO) uses past observations to guide search, but proposed directions and their revision can remain implicit. We propose Feedback-Guided Search State (FGSS), a structured hypothesis-management interface connecting language-based guidance to observed evidence, executable parameter interventions, and comparative support updates. An LLM reflector proposes and revises hypotheses; a deterministic curator validates them and maintains identities and test records. The evolving state guides candidate construction and priorities. Instantiated on LLAMBO, FGSS-BO retains its surrogate and default expected improvement selection, with a bounded testing policy. On two retrospective protocol/media tasks with nearest-neighbor response oracles, it attains higher sample mean final best and search AUC than the matched LLAMBO baseline and evaluated non-LLM baselines. Prose-reflection and EI-only controls show mixed component effects. Six direct-training hyperparameter-optimization tasks show task-dependent transfer, strongest on diabetes SVM; exact-grid and reaction controls examine other settings. FGSS exposes hypotheses, their execution, and cross-round revisions for inspection while supporting search gains in selected settings.

1 Introduction

FGSS gives LLM-guided Bayesian optimization an explicit, revisable hypothesis state linked to evidence and search actions. FGSS-BO improves sample-mean outcomes on two retrospective protocol/media tasks, while component effects and HPO transfer vary by setting.

1 Introduction

  • LLAMBO conditions its LLM calls on prior observations but does not maintain identified hypotheses with evidence references and later test outcomes.A candidate that changes multiple coordinates may not directly test a suggested parameter direction, a concern under limited evaluation budgets.Source: §1 · p. 1
  • FGSS links each hypothesis to evidence, an anchor, a parameter intervention, and an expected outcome direction, with reflector revisions and deterministic curator validation.The curator maintains identities and lifecycle records; accepted hypotheses guide candidate construction and priorities, while eligible matched tests update comparative support.Source: §1 · p. 1
  • FGSS-BO adds hypothesis management to LLAMBO while retaining its surrogate and default expected improvement selection, with a bounded exception for hypothesis-test selection.The matched baseline shares the feasibility-preserving candidate interface.Source: §1 · p. 1
  • FGSS-BO attains higher sample mean final best and search AUC than its matched baseline on two retrospective protocol/media tasks.Prose-reflection and testing ablations show mixed component effects; direct-training HPO gains are clearest on diabetes SVM.Source: §1 · p. 2
  • The contributions include a persistent, evidence-linked hypothesis representation and deterministic curation that converts proposed hypotheses into feasible candidate priorities and matched tests.Comparative support updates link eligible tests to their primary hypothesis.Source: §1 · p. 2
  • The evaluation pairs offline protocol/media comparisons and component ablations with direct-training HPO transfer checks and other setting-specific controls.The two retrospective tasks are iPSC-RPE protocol search and C2C12 culture-media search; additional controls include exact-grid, reaction, and chromatography settings.Source: §1 · p. 2

2 Related Work

Related work places FGSS at the intersection of Bayesian optimization, LLM-guided search, and agent reflection and memory. Its distinction is explicit, executable hypotheses whose tests can be linked to later revisions.

Bayesian optimization for scientific discovery

  • BO combines surrogate updating with acquisition-based query selection, while LLM-guided methods add language interfaces to parameter semantics and experimental knowledge.LLAMBO uses LLMs for warm-starting, candidate sampling, and surrogate prediction; related systems address other scientific and molecular optimization settings.Source: §2 · p. 2

LLM-guided optimization

  • Unlike feedback controls that diagnose outcome sensitivity, FGSS focuses on representing and revising search hypotheses and translating them into search actions.Its records specify an observed anchor, parameter intervention, and expected direction; candidate lineage links eligible numerical comparisons to the corresponding hypothesis.Source: §2 · p. 2

Feedback, memory, and reflection in LLM agents

  • Prior reflection and memory systems revise outputs or accumulate reusable guidance, and FGSS adapts these principles to numerical BO.Its prose-summary control tests a simpler continuously updated memory interface.Source: §2 · p. 2

3 Background and Problem Setup

The paper frames expensive black-box optimization as sequential selection under a limited evaluation budget. LLAMBO adds LLM-based candidate sampling and surrogate prediction, while FGSS adds a separate state for revisable search hypotheses.

3 Background and Problem Setup

  • The setup considers maximizing an expensive black-box objective over a search space, beginning with initial observations and proceeding under a limited evaluation budget.Bayesian optimization selects unevaluated configurations by maximizing an acquisition criterion.Source: §3 · p. 2
  • LLAMBO augments the BO loop with LLM-based candidate sampling and surrogate prediction conditioned on task descriptions and previous evaluations.The LLM calls are integrated into candidate generation and outcome prediction.Source: §3 · p. 2
  • FGSS augments the observation history with hypothesis, evidence-reference, revision, and eligible-test records that specify proposed actions and can change across rounds.The method supports numerical, ordinal, and categorical variables; primary retrospective tasks use a historical-table oracle rather than new laboratory measurements.Source: §3 · pp. 2–3

4 Method

FGSS extracts evidence from observations, maintains curated hypotheses and test records, and uses that state to guide candidate generation and selection. Matched comparisons can revise hypothesis support, while a bounded policy preserves LLAMBO’s surrogate and EI-based selection.

4.1 Framework Overview

  • The framework extracts evidence from the observation history, revises the hypothesis state, and uses that state to guide candidate construction across rounds.An exact intervention can change one parameter at an observed anchor, so comparison with that anchor can update the parent hypothesis.Source: §4.1 · p. 3

4.2 Evidence and Feedback State

  • The extractor summarizes ranked configurations, recent improvement, stagnation, parameter coverage, duplicate regions, and local outcome contrasts as evidence for proposing or revising hypotheses.Each hypothesis records a persistent identifier, scope, anchor, intervention, expected direction, evidence references, uncertainty, heuristic support, and lifecycle status; direct tests target one or two variables.Source: §4.2 · p. 3

4.3 Comparative Support and State Evolution

  • Only eligible direct tests update their primary hypothesis, using a weighted comparative outcome update and a clipped heuristic support score that is not a calibrated probability.Invalid matches receive no outcome update, and global-incumbent progress is tracked separately; table-backed tests allow bounded nuisance differences and receive discounted weight.Source: §4.3 · p. 3

4.4 Regime Control and Candidate Generation

  • A deterministic regime controller constrains exploitation, preserves diversity, and triggers recovery after repeated low-yield search; candidate allocation spans global diversity, local refinement, hypothesis testing, and recovery.The generator proposes feasible, unevaluated configurations and uses hypothesis scopes, directions, and anchor contexts to guide construction and priority.Source: §4.4 · pp. 3–4, Figure 1 · p. 4

4.5 Integration with LLAMBO

  • FGSS-BO retains LLAMBO’s surrogate and EI, while a bounded exception can select an eligible matched intervention when test-budget and spacing conditions permit.For T = 20, the policy allows four base test slots and up to two additional challenger slots during stagnation, with a minimum interval of three rounds; EI-selected tests also consume base-test usage.Source: §4.5 · p. 4

5 Experimental Setup

The experiments compare protocol/media search under retrospective nearest-neighbor oracles with direct-training HPO and other cross-domain checks. Methods share budgets and initial observations within each setting, with the primary comparisons using a low-score start.

5.1 Datasets and Tasks

  • The primary tasks use retrospective nearest-neighbor scores, ten shared lowest-scoring observations, and twenty search evaluations; this is a stress test, not label-free prospective initialization.iPSC-RPE uses a discrete seven-setting protocol space; C2C12 uses fourteen continuous media coordinates and a source label for response lookup.Source: §5.1 · p. 5

5.2 Baselines

  • The matched LLAMBO baseline retains its surrogate and EI with the shared feasibility-preserving interface, while FGSS ablations isolate reflection and test-selection components.Comparators also include Random Search, HEBO, and GP-EI; six HPO tasks use direct model training and label-free public initial configurations.Source: §5.2 · p. 5, §5.1 · p. 5

5.3 Metrics

  • Final best and search-stage best-so-far AUC summarize performance; reported means and sample standard deviations cover six paired runs.Search AUC averages performance after initialization, not ROC AUC; paired bootstrap intervals are exploratory and unadjusted for multiple comparisons.Source: §5.3 · pp. 5–6

5.4 Implementation Details

  • Primary runs use batch size one, ten candidates, two templates, and ten generations; FGSS adds a reflection call each round and therefore increases total LLM compute.The setup uses GPT-4o mini, disables optional domain-prior cards and injections, and retains task descriptions and pretrained LLM knowledge.Source: §5.4 · p. 6

6 Results

FGSS-BO leads the matched LLAMBO baseline in sample mean final best and search AUC on both primary retrospective tasks, while component, transfer, and cross-domain results vary by setting.

6.2 Component Ablations

  • Reflective Summary raises both sample means over LLAMBO, but full FGSS exceeds it by 0.0285 in final best and trails by 0.0061 in AUC; component controls show mixed effects.State-only feedback permutation also has mixed effects, with both paired difference intervals including zero.Source: §6.2 · p. 6, Table 2 · p. 7

6.3 Direct-Training HPO Transfer

  • Diabetes SVM shows the clearest HPO gain: mean final CV MSE falls from 0.57768 to 0.51711 (10.48%), and search AUC from 0.61046 to 0.55636 (8.86%).Transfer is task-dependent: breast SVM improves modestly, breast RF and digits SVM are nearly tied, and digits RF and diabetes RF have mixed results.Source: §6.3 · p. 6

6.4 Candidate-Sampling Diagnostics

  • Only 18.4% of iPSC-RPE free-form proposals are unique and 3.4% survive validity and novelty checks, motivating legal-space construction.Freeform remains viable on C2C12, with sample means close to Feasible; both main LLM methods use the same feasible interface.Source: §6.4 · p. 6

6.5 Oracle and Cross-Domain Controls

  • On exact-grid iPSC, final-best means are close (0.840 LLAMBO, 0.848 FGSS-BO), but mean search AUC favors LLAMBO (0.750 versus 0.710); HEBO and GP-EI outperform both LLM methods on all five Summit tasks.On four Summit tasks FGSS-BO exceeds LLAMBO in final-best means and on three in AUC; HPLC has higher FGSS-BO means on both metrics.Source: §6.5 · pp. 6–7

6.1 Main Results on Protocol and Media Search

  • +0.123/+0.147 for iPSC final best/AUC and +0.445/+0.141 for C2C12 are FGSS-BO’s mean differences over LLAMBO; it has the highest sample means on both metrics in both primary tasks.The iPSC final-best intervals versus GP-EI include zero, while its AUC interval lies above zero; C2C12 intervals versus GP-EI cross zero on both metrics.Source: §6.1 · p. 6, Table 1 · p. 7, Figure 2 · p. 7

7 Discussion

The discussion frames FGSS as an inspectable record-and-action interface, while distinguishing targeted comparative support from global objective progress. Its observed benefits vary across search settings.

Discussion

  • A hypothesis record links an observed anchor and parameter intervention to a later comparison that can revise the same record; the LLM proposes changes while fixed rules govern admissibility and support updates.Prose reflection and EI-only selection also achieve higher sample means than LLAMBO, so the results do not establish a uniform optimization advantage for full FGSS.Source: §7 · p. 7
  • A candidate can outperform its matched control without improving the incumbent, so local comparative support and global progress answer different questions tracked separately by FGSS.Higher AUC for No-Override and FGSS w/o Tests suggests a trade-off between early objective progress and targeted comparisons under this protocol.Source: §7 · p. 7
  • HPO gains are strongest on diabetes SVM, while exact-grid iPSC has lower mean FGSS AUC; the discussion therefore motivates adapting guidance and testing to the interface and response landscape.Future refinements could preserve more global exploration when local hypotheses dominate and allocate testing using observed test utility.Source: §7 · pp. 7–8
  • Free-form iPSC proposals show severe duplication, supporting the shared feasibility-preserving interface used in the primary LLM comparison.The earlier diagnostics compare free-form proposals with feasibility-preserving construction.Source: Figure 3 · p. 8

8 Conclusion

The conclusion presents FGSS as structured hypothesis management for LLM-guided BO, with search gains in selected settings and mixed results across controls.

Conclusion

  • FGSS turns language-based proposed directions into executable parameter interventions and links eligible comparisons to later hypothesis revisions.Its records expose the path from hypotheses to search actions and updates.Source: §8 · p. 8
  • FGSS-BO retains LLAMBO’s surrogate and default EI selection, with a bounded testing policy.Source: §8 · p. 8
  • The complete system has higher sample means on two retrospective protocol/media tasks and gains on some direct-training HPO tasks, while component and oracle controls show mixed effects.Source: §8 · p. 8

9 Limitations

The evaluation is bounded to one LLM backend and base optimizer, with FGSS adding reflection cost; the primary tasks are retrospective, and broader prospective validation remains necessary.

9 Limitations

  • The primary tasks use historical-table nearest-neighbor responses and outcome-conditioned initialization, so their results do not establish performance in prospective laboratory search.Prospective validation is needed to assess measurement noise, batch effects, experimental failures, and temporal drift.Source: §9 · p. 8
  • Evaluation uses one LLM backend and base optimizer, while FGSS adds reflection cost.Future evaluation could extend paired runs to ten seeds and examine additional backends, initialization policies, budgets, and optimizers.Source: §9 · p. 8
  • Noise-aware support updates and test prioritization are proposed as refinements to balance local hypotheses and exploration.Source: §9 · p. 8

A Additional Experimental Details

The appendix places algorithm settings and prompt/state examples before detailed experimental controls.

A Additional Experimental Details

A.1 Algorithm Details

The implementation combines state-guided candidate generation and bounded direct testing with deterministic curation, support updates, and an EI-based selection policy.

A.1 Algorithm Details

  • Each round extracts evidence, asks the reflector for hypothesis operations, validates and deduplicates them, applies lifecycle updates, and constructs candidates from global, guided, local, and recovery branches.The loop evaluates one selected candidate, updates eligible parent-hypothesis support against a control, records state transitions, and repeats until the budget is used.Source: §A.1 · p. 10
  • Local observational contrasts use target and nuisance-distance thresholds, retain up to four pairs per parameter and twelve overall, and summarize directional differences and agreement.These contrasts are observational, not identified causal effects.Source: §A.1 · p. 10
  • Direct tests link a parent hypothesis, target variables, and an observed anchor; only the primary hypothesis receives an outcome-based support update.Open-space tests hold non-target coordinates fixed, while table-backed matches allow bounded nuisance differences and are not isolated biological causal-effect estimates.Source: §A.1 · p. 10
  • Candidate generation combines global sampling, local exploitation, state-guided anchor interventions, recovery from repeated low-gain centers, and evidence-linked challenger proposals.Unusable guided quotas return to global search or recovery rather than generating ungrounded tests.Source: §A.1 · pp. 10–11
  • The proposed support score is a validated heuristic, not a calibrated posterior probability or test-success frequency; evidence weight discounts uncertainty, match quality, and control reuse.Support increments and decrements are fixed, staleness is applied separately, and empirical tolerance is not a noise-calibrated significance threshold.Source: §A.1 · p. 11
  • Eligible tests are prioritized deterministically using test counts, uncertainty, challenger status, recency, and context eligibility; the priority is not a learned acquisition function.The policy caps base and challenger slots, checks support and interval conditions for overrides, and retains EI-selected tests even when override budgets are exhausted.Source: §A.1 · p. 11
  • The allocation presets preserve global-search floors and cap guided candidates; Table 4 presents nominal presets rather than fixed realized ratios.For T = 20, the base and additional challenger caps are four and two slots, respectively.Source: §A.1 · p. 11, Table 4 · p. 12
  • The deterministic gate uses fixed task-scale response cutoffs and reliability conditions; Table 5 records cutoff sources, while Table 3 lists inherited settings without a hyperparameter sweep.The selection policy uses EI unless a valid test-policy slot is available; Table 6 reports actual selection changes and EI-selected matched tests.Source: §A.1 · pp. 10, 12, Table 5 · p. 12, Table 3 · p. 12

A.2 Realized Test Policy and Matching

This section examines how candidate restrictions, state feedback, and test-selection policies relate to observed results, alongside audits of test execution and evaluation procedures. Outcomes vary by setting: Freeform uses recovery fallback in every iPSC search round and has means close to Feasible on C2C12, while direct-training HPO transfer is strongest on diabetes SVM and small, tied, or mixed on other tasks.

A.2 Realized Test Policy and Matching

  • Across six full-FGSS primary runs, iPSC had 18 overrides in 120 search evaluations and 19 EI-selected tests with support updates, while C2C12 had neither overrides nor direct updates.All 37 iPSC updates used approximate covariate matches and one primary hypothesis; match quality had median 0.748 and range [0.664, 0.918]. Logged validation rejections were 155/482 (32.2%) for iPSC and 329/476 (69.1%) for C2C12; these are check-event rates, not unique-test rejection rates.Source: §A.2 · pp. 12–13

A.4 Reflector Prompt and State Schema

  • The reflector proposes evidence-linked hypothesis operations, while deterministic validation requires evidence references and executable ranges and prevents silently changing a falsifiable test in place.Search tasks use shared paired initializations; primary iPSC-RPE and C2C12 use low-score initialization and nearest-neighbor response lookup, whereas exact-grid iPSC restricts candidates to observed rows.Source: §A.4 · p. 13

A.8 Cross-Domain Summit Evaluation

  • Across five Summit tasks, HEBO and GP-EI exceed FGSS-BO and LLAMBO on both metrics; FGSS-BO beats matched LLAMBO on four final-best means and three AUC means, but loses on Case 4.All five tasks use six paired seeds, eight shared initial observations, and twenty search evaluations.Source: §A.8 · p. 15, Table 10 · p. 17

A.9 Exact-Grid iPSC-RPE Control

  • On exact-grid iPSC, FGSS-BO’s final best is 0.848 ± 0.064 versus LLAMBO’s 0.840±0.026, while its search AUC is 0.710 ± 0.096 versus 0.750 ± 0.009.This control restricts candidates to observed configurations and removes nearest-neighbor assignment, so it does not isolate oracle effects from candidate-space restriction.Source: §A.9 · p. 15

A.10 Olympus HPLC Control

  • FGSS-BO’s Olympus HPLC means exceed LLAMBO’s for final best (1962.651 ± 237.131 vs. 1803.720 ± 521.827) and search AUC (1765.846 ± 381.965 vs. 1677.365 ± 465.783).The comparison covers the two LLM pipelines across six paired runs.Source: §A.10 · p. 15

A.11 Component-Ablation Protocol and Paired Differences

  • Full FGSS has a higher observed final-best mean than the prose-summary control but no AUC advantage; removing the test bundle has mixed effects.The prose-summary comparison changes state representation, candidate control, and test management together; FGSS w/o Tests disables test generation, EI overrides, and direct-test support updates while retaining structured reflection and state-guided candidates.Source: §A.11 · pp. 16–17, Table 11 · p. 18

A.12 State-Feedback Permutation on iPSC-RPE

  • Aligned feedback has a higher mean final objective and lower final-outcome variability than permuted feedback, while permutation has faster early progress and higher mean search AUC; both paired intervals include zero.True-minus-permuted mean differences are 0.0275 for final best and −0.0349 for AUC, with exploratory 95% intervals [−0.0232, 0.0850] and [−0.1236, 0.0631].Source: §A.12 · p. 17, Table 12 · p. 18, Figure 5 · p. 18

A.13 Test-Selection Ablation and Immediate Opportunity Cost

  • Full FGSS minus No-Override has mean paired differences +0.0117 in final best and −0.0495 in search AUC, with both exploratory intervals including zero; No-Override’s sample means exceed LLAMBO’s.Across 18 selection changes, the mean response gap is 0.0563 and mean one-step incumbent gap is 0.0854 in favor of EI; the six first-round changes favor EI immediately, while the remaining twelve favor tests on both mean gaps.Source: §A.13 · pp. 18–19, Table 13 · p. 19

A.14 Freeform and Feasible Candidate Interfaces

  • On iPSC, Freeform scores 0.346 for both final best and search AUC, versus Feasible’s 0.691 ± 0.117 and 0.494 ± 0.057.All 120 Freeform search rounds use recovery fallback, yielding six identical trajectories; on C2C12, no round uses fallback, and Freeform’s final best and search AUC are 1.287 ± 0.673 and 1.058 ± 0.583 versus Feasible’s 1.281 ± 0.220 and 1.025 ± 0.264.Source: §A.14 · p. 19
  • Feasible includes candidate refinement, whereas Freeform retains validity checks, deduplication, retries, and recovery and is not a strict reproduction of original LLAMBO.Both main LLM methods use the same feasibility-preserving interface across tasks; Freeform is reported separately as a candidate-interface control.Source: §A.14 · p. 20

A.15 Direct-Training Bayesmark HPO Transfer

  • The fixed HPO transfer suite retained six breast cancer, handwritten-digits, and diabetes tasks paired with RandomForest and SVM, comparing complete LLM pipelines over six paired seeds.Configurations were evaluated by five-fold cross-validation; the held-out test set was used only for reporting, not candidate generation, surrogate prediction, or state updates.Source: §A.15 · p. 20, Table 14 · p. 21
  • Diabetes SVM improves mean final MSE and search AUC by 10.48% and 8.86%, respectively, while other HPO tasks show small, tied, or mixed effects.The paired differences use unrounded per-seed metrics; intervals are task-specific, exploratory, and unadjusted for multiple comparisons.Source: §A.15 · p. 21, Table 16 · p. 21

Related papers

Loading 2610.21246v1…