Source-linked AI summary
Why Sample What You Can Enumerate? Exact Policy Optimization for Genomic Tool Selection
Haoyue Liu, Xiaoyu Ma, Ye Chen, Zhichao Wang, Xiaoying Tang
TL;DR
Sampled policy optimization can become mismatched with enumerable specialist tool spaces because policy concentration causes reward collisions and vanishing advantages. The paper introduces FGPO, which exactly evaluates all tool subsets and precomputes question–subset rewards. Across five frozen reasoners and three genomic benchmarks, FGPO outperforms GRPO in all 15 settings by 6.75 points on average, while reducing GenomeQA tool use from 2.36 to 1.40 per question.
Problem
In specialist genomic settings with compact enumerable tool spaces, GRPO estimates action expectations from sampled rollouts, and concentrated policies can produce coincident rewards with zero advantages.
Method
FGPO optimizes the exact action expectation over every tool subset and precomputes each question–subset reward in an exhaustive table.
Results
FGPO outperforms GRPO in all 15 benchmark–reasoner settings by 6.75 points on average, while invoking 1.40 versus 2.36 tools per GenomeQA question.
Takeaways & Limitations
Exact optimization consistently improves genomic tool selection across three benchmarks and five frozen reasoners while invoking fewer tools.
Takeaways & Limitations
The approach is studied in a specialist regime where compact tool libraries suffice; larger libraries may require adaptively constructed compact candidate sets.
Abstract
from arXiv · showhide
Reinforcement learning over a frozen reasoner has become a common recipe for teaching a policy which external tools to invoke. We show that this recipe becomes structurally mismatched in specialist scientific settings where the complete tool-subset space is enumerable. There, a small set of recurring computational capabilities covers the domain, so the space of tool subsets is combinatorial yet small enough to enumerate, and GRPO still estimates an action expectation from a handful of sampled rollouts. Worse, the approximation degrades as training succeeds: as the policy concentrates on preferred subsets it resamples them, sampled rewards collide, and the group-normalized advantage vanishes. On genomic reasoning the fraction of questions yielding no reward signal rises from 0.2% under a uniform reference policy to 20.8% after GRPO training. As a remedy, we introduce FGPO (Full-Group Policy Optimization), which (1) scores every tool subset and optimizes the exact action expectation, so each update sees the complete action space, and (2) precomputes the reward of each question--subset pair into an exhaustive table, removing frozen-reasoner calls from the training loop entirely. Across five frozen reasoners and three genomic benchmarks, FGPO outperforms GRPO in all 15 settings by 6.75 points on average and up to 14.20, while a standard on-demand GRPO schedule would require 2.4 times as many frozen-reasoner reward evaluations and, on GenomeQA, FGPO cuts invoked tools per question from 2.36 to 1.40.
1 INTRODUCTION
Genomic reasoning exposes a mismatch between sampled policy optimization and small enumerable tool-subset spaces: GRPO can lose reward signal as sampled subsets collide, whereas FGPO evaluates the complete action space exactly. The paper introduces FGPO to address this mismatch with exhaustive subset scoring and precomputed rewards.
- Motivation: Genomic questions require explicit computation over nucleotide sequences, including motif scanning, splice-site scoring, and composition analysis.External tools provide these specialist capabilities to a frozen reasoner.
- Problem: The four genomic tools form only 16 possible subsets, making the complete action expectation exactly computable rather than requiring sampled rollouts.This compact action space creates the central setting for comparing GRPO with exact optimization.
- Problem: 20.8% of questions form dead groups after GRPO training, up from 0.2% under a uniform reference policy, because coincident sampled rewards make all normalized advantages zero.The dead-group rate reaches 79.6% under the differential reward used by GRPO.
- Approach: FGPO scores every tool subset, optimizes the complete action expectation, and precomputes each question–subset reward to remove frozen-reasoner calls from training.Its objective exposes the full action space on every update.
- Results: Across five reasoners and three genomic benchmarks, FGPO beats GRPO in all 15 settings by 6.75 points on average and up to 14.20 points.On GenomeQA, it reduces average invoked tools from 2.36 to 1.40.
2 FGPO: EXACT POLICY OPTIMIZATION OVER ENUMERABLE TOOL SUBSETS
FGPO treats genomic tool selection as an enumerable action space: it scores every subset, uses exact reward expectations, and precomputes rewards for training. Its LLM implementation addresses sequence-length bias, entropy, evaluation control, and sampled-estimator failure modes.
- 2.2 THE EXACT OBJECTIVE: FGPO evaluates all 2^n tool subsets and optimizes the complete action expectation instead of estimating it from sampled rollouts.The reward table supplies every question–subset outcome to the exact objective.
- 2.1 SETTING: The action space contains 16 subsets for the four-tool main setting, with each subset fully specifying independent tool execution.The modular library is frozen, and tools analyze the same sequence without consuming one another’s outputs.
- 2.2 THE EXACT OBJECTIVE: A dead GRPO group occurs when sampled rewards coincide, making every normalized advantage zero; concentration increases this risk as sampling coverage shrinks.The converged GRPO policy places 98% of its mass on one subset, while the mean dead probability reaches 20.8%.
- 2.3 INSTANTIATING THE EXACT ESTIMATOR ON AN LLM POLICY: FGPO uses per-token mean candidate scores because raw sequence probabilities favor the shortest, empty-subset action among strings of unequal length.Its candidate-normalized distribution differs from the generation-induced distribution used by the GRPO sampler.
- 2.3 INSTANTIATING THE EXACT ESTIMATOR ON AN LLM POLICY: Entropy regularization broadens candidate probabilities, while controlled candidate scoring removes format failures and applies the same decoding rule across policies.Under controlled scoring, FGPO remains ahead by 6.71 points, although the reported main results use free generation.
- 2.4 THE EXHAUSTIVE REWARD TABLE: Precomputing all question–subset rewards requires 32,032 reasoner calls versus 76,800 for the reported on-demand GRPO schedule, a 2.4× difference.After table construction, downstream training uses lookups without further frozen-reasoner calls.
3 EXPERIMENTS
Across three genomic benchmarks and five frozen reasoners, FGPO consistently outperforms sampled and offline baselines while learning selective, query-dependent tool use. Experiments link this advantage to complete action-space coverage, which avoids sampling starvation and improves both accuracy and tool economy.
- A1: FGPO outperforms sampled and offline baselines: FGPO outperforms GRPO in all 15 benchmark–reasoner cells, exceeding it by 6.75 points on average.It is the best or tied-best learned or deployable method in 14 of 15 cells.
- A1: FGPO outperforms sampled and offline baselines: On GenomeQA, FGPO invokes 1.40 tools per question versus 2.36 for GRPO.FGPO also uses 3.5–4.1× fewer input tokens than exhaustive All-Tools.
- A2: FGPO learns query-dependent genomic evidence routing: FGPO routes tools by task family, favoring T2 for transcription-factor motifs and T4 elsewhere, while never invoking T1 across benchmarks.GRPO invokes T1 on 21–64% of questions, whereas FGPO uses it on 0.0%.
- A3: Sampling starves because training succeeds: At GRPO’s group size G=6, dead groups rise from 0.2% under a uniform reference policy to 20.8% after training.Under the coarser differential reward used by GRPO, the converged dead fraction reaches 79.6%.
- A3: Sampling starves because training succeeds: Dead groups affect unresolved cases: 6.7% of all questions are simultaneously dead, wrong, and repairable by some available subset.The exact estimator retains signal on these questions because it does not rely on sampled groups.
4 RELATED WORK
Prior work spans tool-use training, prompted acting, API-scale instruction tuning, and RL-based selection policies for frozen reasoners. This paper focuses on the combinatorial regime where the complete tool-subset space is small enough to evaluate but sampled optimization remains in use.
- RL-based tool selection trains policies to choose external tools for a frozen reasoner, while related work also studies self-supervised call insertion, prompted acting, API-scale instruction tuning, and call-correctness benchmarks.
- The paper studies genomic subset selection as a distinct regime in which the full tool-subset space is small enough to evaluate exhaustively.
- Sampled optimization differs from exact action integration because policy concentration progressively increases the probability of zero-advantage groups.
- FGPO applies action-space integration to tool selection, extending variance-reduction ideas to enumerable genomic tool subsets.
5 CONCLUSION AND FUTURE WORK
FGPO addresses information starvation in sampled optimization by evaluating the exact enumerable action space and reusing exhaustive rewards. The paper concludes that this approach improves reasoning across genomic benchmarks while invoking fewer tools, but its current scope is compact specialist tool libraries.
- FGPO optimizes the exact action expectation over every tool subset and precomputes each question–subset reward, eliminating frozen-reasoner calls during policy training.
- FGPO consistently improves reasoning across three genomic benchmarks and five frozen reasoners while invoking fewer tools.
- Future work could adapt FGPO to larger libraries by constructing compact, coverage-preserving candidate sets and retaining exact optimization within each set.
- The study uses public reference-organism corpora and excludes human subjects and clinical or individually identifiable genomic data.
- The method learns which standard sequence-analysis tools to invoke, without designing sequences or adding capabilities beyond those tools provide.
B.2 SENSITIVITY CHECKS
Sensitivity checks examine objective choices, selector size, library expansion, reproducibility, and task-family behavior. They show that FGPO’s advantage is not confined to a particular selector scale, while its routing adapts tool usage to task difficulty.
- Sensitivity checks: 0.25 points: increasing the entropy coefficient from 0.03 to 0.08 changes the diagnostic peak by only 0.25 points.
- Sensitivity checks: 50.42 versus 44.25: a 1.5B selector still outperforms six-rollout GRPO, indicating the advantage is not specific to the 7B selector.
- Five-tool library: 45.38 accuracy: the five-tool policy exceeds exhaustive invocation at 44.51 and the best post-hoc fixed subset at 45.18.
- Five-tool library: FGPO’s five-tool policy improves across four epochs, from 43.04 to 45.38, without selecting the final checkpoint using test performance.
- Task-family analysis: 20.6 points on taxonomy and 23.0 on TF motif: FGPO’s gains are larger there than on splice sites and histone marks, where the per-question oracle is lower.
- Task-family analysis: 1.18 to 1.53 tools: FGPO calls more tools on harder task families, spending its budget where one tool does not settle the question.
C.3 THE TRANSFER GRID, READ AS SHAPES
The transfer grid shows that FGPO’s advantage persists across frozen reasoners and benchmarks, although its size varies with benchmark difficulty and reasoner strength. The comparison also shows that fixed subsets cannot reliably capture query-dependent optionality.
- All 15 benchmark–reasoner cells favor FGPO over GRPO, with margins from +0.92 to +14.20 and a median of +4.48.
- The best fixed subset differs by benchmark, and omitted tools can remain uniquely necessary for particular questions.
- +11.20 to +14.20: the largest FGPO–GRPO margins occur on GenBench-X, while the smallest is +0.92 with the 1.5B reasoner on GenomeQA.
- 0.05 points or less: live transfer evaluation, exhaustive-cache evaluation, and the per-epoch harness agree closely on GenomeQA measurements.
C.5 CEILINGS: THE BEST FIXED SUBSET AND THE PER-QUESTION ORACLE
The exhaustive tables provide both a best fixed subset and a per-question oracle for benchmarking FGPO. FGPO leads the best fixed reference in most cells, but a substantial oracle gap remains and winning subsets vary by benchmark.
- FGPO’s advantage over GRPO is nonnegative in all 15 reasoner×benchmark cells.
- FGPO leads BestFixed∗ in 10 of 15 cells, with remaining margins ranging from 0.10 to 2.10 points.
- The winning fixed subset varies by benchmark: T4 alone on GenBench-X, T3+T4 on BM4, and T1+T2+T4 on GenomeQA.
- The per-question oracle reaches 77.41 on GenomeQA, 90.60 on GenBench-X, and 86.90 on BM4, compared with FGPO’s 52.62, 65.20, and 54.80.
- The coverage intervention evaluates k ∈ {2, 4, 8, 16} actions per visit, with k=16 corresponding to FGPO.
D.3 DATA-SCALE ABLATION
FGPO retains strong GenomeQA accuracy with reduced training data, although the data-scale relationship is not monotone. Even 500 training questions nearly match the full-data peak and exceed GRPO.
- 52.23 accuracy with 500 training questions is within 0.4 points of the full-data peak of 52.59.
- 50.92 is the peak for the 1,000-question arm, so accuracy does not increase monotonically with training-set size.
- All reduced-data arms remain above GRPO’s 45.46 accuracy on GenomeQA.
- Figure 11 analyzes where FGPO and GRPO rescues fall across task families and compares FGPO’s choices with the oracle.
D.4 TOOL-LIBRARY SINGLE-REMOVAL ABLATIONS
Single-tool removal exposes uneven tool value across benchmarks and reference ceilings. No tool is universally dispensable, while T4 has especially large effects on GenBench-X and best-fixed performance.
- Removing T4 costs the GenBench-X oracle 15.5 points and the best fixed subset 20.5 points.
- Removing T3 costs the best fixed subset at most 0.30 points anywhere.
- No single tool is dispensable across all three benchmarks.
- The ablation reports oracle and best-fixed ceiling losses computed from exhaustive tables without model runs.
E.1 GRPO CHECKPOINT AND CONFIGURATION SWEEPS
Checkpoint and configuration sweeps support the reported GRPO baseline while showing that DPO remains near the no-tool floor. The theoretical analysis links dead groups to reward collisions and clarifies limits of the exact objective.
- GRPO checkpoint and configuration sweeps: GRPO accuracy is flat from step 50 onward while mean tool count is also flat, making 45.46 a converged plateau.
- GRPO checkpoint and configuration sweeps: Every DPO β checkpoint remains within 0.4 points of the no-tool floor, because the policy stops selecting tools.
- Dead groups and exact objective: Reward collisions produce dead groups when all sampled actions fall in one reward class, and coarsening reward classes cannot reduce that probability.
- Dead groups and exact objective: Under the fine reward partition, the closed-form dead-group rate is 20.8%, versus 79.6% under the coarser differential reward.
- Dead groups and exact objective: The exact objective computes the per-question expectation and gradient from all candidate scores, eliminating variance from action sampling while retaining minibatch variance across questions.
- Dead groups and exact objective: Exhaustive evaluation removes sampled-collision zeros, but small action probabilities, coupled LoRA updates, and entropy regularization do not guarantee rapid recovery or a probability floor.
F.2 DEAD-GROUP GRIDS FOR ALL THREE BENCHMARKS
The dead-group grids show that sampled policy optimization becomes unreliable as reward classes collide, while FGPO’s high dead-group rate is expected because it does not sample. Comparisons across group sizes and policies expose why sampling behavior is not a valid proxy for FGPO’s effectiveness.
- Measurement: The closed-form rates are computed from generation-induced distributions over the 16 valid parsed subset strings, with reward classes defined by Eq. 1.Table 15 reports these rates across group sizes and benchmarks.
- GRPO and reference behavior: 17–21% dead-group probability occurs for converged GRPO at the G=6 group size used in training under the fine-grained Eq. 1 partition.A uniform reference policy is almost never dead beyond G=4.
- FGPO behavior: 94% dead-group probability for converged FGPO at G=6 is expected because FGPO evaluates the complete action space instead of sampling groups.The FGPO row is therefore not evidence that its optimization is ineffective.
- SFT behavior: 92.8% of SFT argmax decisions select the empty action, while its accuracy of 42.17 trails GRPO’s.SFT’s diffuse sampling distribution does not imply useful routing.
F.3 EMPTY-SUBSET PREFERENCE: HOW OFFLINE OBJECTIVES COLLAPSE
The empty subset is often reward-optimal, but policies that imitate this modal action can collapse into poor routing. The benchmark results show that semantic tool descriptions help prompted baselines, yet FGPO can outperform them without semantic information.
- Empty-subset preference: 61.9% of GenomeQA questions have the empty subset as the reward-optimal action.This creates a strong modal answer that can attract offline objectives toward no-tool selection.
- Offline collapse: 92.8% of SFT argmax decisions select the empty subset, producing a strong imitation of the modal answer but a poor policy.The resulting policy ignores the remaining 38% of questions where the empty subset is not optimal.
- Action and evaluation setup: The no-tool action is represented explicitly by an empty answer, while selected tools contribute rendered evidence blocks to the frozen reasoner.The reasoner receives the question, options, and selected-tool evidence, then returns one option letter.
- Tool usage: 2.8% of questions invoke T5 under the exact objective, because the tool is rarely decisive on this benchmark despite being well implemented.The exhaustive reward table makes this usage pattern visible without a hand-written routing rule.
- Semantic information: 47.77 GenomeQA accuracy for the prompted baseline exceeds GRPO’s 45.43, while FGPO reaches 52.65 without semantic tool information.The prompted baseline receives real tool names and functional descriptions, whereas trained policies do not.