PaperlayerView exact revision on arXiv

Source-linked AI summary

Formalizing Learning from Language Feedback with Provable Guarantees

Wanqiao Xu, Allen Nie, Ruijie Zheng, Aditya Modi, Adith Swaminathan, Ching-An Cheng

arXiv:2506.10341v2cs.LGcs.CL

TL;DR

The paper addresses the lack of a principled framework for learning from language feedback when rewards are latent. It formalizes LLF, introduces transfer eluder dimension, and develops HELiX; empirically, HELiX consistently outperforms CoT baselines on Battleship and Minesweeper.

  • Problem

    LLF lacks a rigorous account of when natural-language feedback enables efficient learning and how its difficulty compares with reward-observed bandits.

    Source: §1 · p. 1

  • Method

    The paper formalizes LLF, introduces transfer eluder dimension, and develops HELiX to explore and exploit through hypothesis verification.

    Source: §1 · pp. 1–2, §2 · p. 2, §4 · p. 6

  • Results

    HELiX consistently outperforms the CoT baseline and a no-exploitation variant on Battleship and Minesweeper, especially when information gathering is essential.

    Source: §5.1 · p. 7

  • Takeaways & Limitations

    Informative language feedback can support principled learning and, in certain environments, yield exponential efficiency gains over reward-only learning.

    Source: §1 · p. 2, §7 · p. 8

    The practical approach assumes LLMs can select optimal actions, score hypotheses fairly, and generate diverse faithful hypotheses, which may not hold for all LLMs.

    Source: §5.2 · pp. 7–8

Abstract

from arXiv · show

Interactively learning from observation and language feedback is an increasingly studied area driven by the emergence of large language model (LLM) agents. Despite impressive empirical demonstrations, so far a principled framing of these decision problems remains lacking. We formalize the Learning from Language Feedback (LLF) problem, assert sufficient assumptions to enable learning despite latent rewards, and introduce $\textit{transfer eluder dimension}$ as a measure to characterize the hardness of LLF. We formalize the intuition that information in the language feedback governs the learning complexity, and demonstrate cases where learning from rich language feedback can be exponentially faster than learning from reward. We develop a no-regret algorithm, called $\texttt{HELiX}$, that provably solves LLF problems through sequential interactions, with performance guarantees that scale with the transfer eluder dimension. Across several empirical domains, we show that $\texttt{HELiX}$ performs well even when repeatedly prompting LLMs does not work reliably. Our contributions mark an important step towards designing principled interactive learning algorithms using generic language feedback.

1. Introduction

The paper formalizes Learning from Language Feedback (LLF) to explain when language feedback enables efficient learning despite latent rewards. It introduces transfer eluder dimension and HELiX, with theory and experiments showing that informative feedback can substantially improve learning.

1. Introduction

  • Language feedback can convey action-relevant information, such as what is wrong and where to improve, beyond coarse scalar evaluation.The paper contrasts detailed feedback with rewards that provide only coarse evaluation.Source: §1 · p. 1
  • LLF addresses the lack of rigorous understanding of when and how language feedback enables efficient learning.The framework studies language feedback as an alternative to scalar rewards in interactive decision problems.Source: §1 · p. 1
  • The paper formalizes LLF under assumptions enabling learning with latent rewards and proves its complexity is no larger than that of pure reward learning.The framework models language feedback generated by an underlying hypothesis in a bandit-like interaction protocol.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Transfer eluder dimension dimT E captures information in language feedback and satisfies dimT E ≤ dimE under suitable assumptions.Here dimE is the eluder dimension for learning from reward.Source: §1 · p. 2
  • HELiX is a provably no-regret LLF algorithm whose guarantees scale with learning complexity and can be exponentially more efficient than reward-only learning in some environments.The paper also presents HELiX as an inference-time strategy that samples hypotheses and self-verifies rather than relying on one chain-of-thought context.Source: §1 · p. 2

2. Learning from Language Feedback

This section defines LLF as learning to maximize latent reward through observable language feedback, then models environments with text hypotheses and formalizes how verifiers extract learning-relevant information.

2. Learning from Language Feedback

  • LLF provides a formal mathematical model for studying learning from language feedback under natural assumptions.The model introduces text hypotheses and a hypothesis verifier as foundations for learnability and regret analysis.Source: §2 · p. 2

2.1. Formal Setup of LLF

  • In the LLF setup, the agent chooses actions, observes feedback sampled from a feedback function, and incurs an unobserved reward used to define regret.The objective is to maximize reward despite observing only feedback.Source: §2.1 · p. 2

2.2. Environment Model and Text Hypothesis

  • A hypothesis is a text description that determines the environment’s reward and feedback functions, while the true hypothesis remains unknown to the agent.The hypothesis space may be exponentially large and can represent natural language, code, or numerical arrays.Source: §2.2 · p. 3

2.3. Measuring Information in Feedback

  • Learning from feedback is impossible without a connection between feedback and reward, so LLF requires feedback containing action-relevant information.The framework exposes this information through the hypotheses that remain consistent with observed feedback.Source: §2.3 · p. 3
  • The hypothesis verifier assigns loss according to how consistently a candidate hypothesis explains an action-feedback pair, without certifying that the hypothesis is true.It functions like a unit test that can eliminate hypotheses contradicted by observed feedback.Source: §2.3 · p. 3
  • The verifier can exploit richer language feedback to eliminate hypotheses more efficiently than binary reward-like feedback.In the coin-toss example, descriptive feedback can convey more information than “match/miss” and support more efficient hypothesis elimination.Source: §2.3 · p. 3
  • Unbiased feedback permits noise by requiring each generating hypothesis to minimize expected verifier loss under its induced feedback distribution.This weakens the stronger requirement that feedback always be perfectly self-consistent.Source: §2.3 · pp. 3–4
  • Hypothesis equivalence identifies descriptions that induce identical verifier losses across all actions and feedback outcomes.The resulting distance defines the geometry of the hypothesis space while treating semantically indistinguishable descriptions alike.Source: §2.3 · p. 4

3. Learnability

LLF learnability depends on how feedback reduces uncertainty about hidden rewards, which transfer eluder dimension formalizes across feedback types. The section shows that informative feedback can yield exponential gains over reward-only learning, while discriminative feedback is no harder than reward learning.

3.1. Transfer Eluder Dimension

  • Transfer eluder dimension measures feedback-dependent reward uncertainty by testing whether hypotheses consistent with prior feedback still disagree on a new action’s reward.It differs from traditional eluder dimension by combining the hypothesis verifier loss with the reward function.Source: §3.1 · p. 4

3. Learnability

  • Feedback information governs LLF learnability: uninformative feedback can make transfer eluder dimension infinite, whereas informative feedback can accelerate learning.The measure captures how effectively feedback reduces uncertainty about unknown rewards.Source: §3.1 · p. 4, §3 · p. 4, §3.2 · p. 5

3.2. Informative Feedback Reduces Learning Complexity Exponentially

  • Reward-only learning over L-step reasoning requires exponential complexity, while feedback identifying the first mistake enables stage-wise decomposition and an exponential improvement in L.The comparison uses feedback types corresponding to reward and hindsight-negative feedback.Source: §3.2 · p. 5
  • More constructive feedback can remove dependence on |S| by showing how to correct mistakes, while feedback revealing the answer makes the learning complexity constant.These cases illustrate progressively richer feedback improving over reward and first-mistake feedback.Source: §3.2 · p. 5
  • The transfer eluder dimension is an analysis measure rather than an algorithm input, and cannot be estimated directly when LLMs provide the feedback.The paper presents it for characterizing complexity, not as a quantity available to the proposed algorithms.Source: §3.2 · p. 5

3.3. Learning from Informative Feedback Is No Harder Than Learning from Reward

  • Discriminative feedback means the hypothesis verifier distinguishes hypotheses in proportion to differences between their corresponding rewards.Feedback that encodes reward and can be decoded by the verifier, including classical reinforcement learning, is a special case.Source: §3.3 · p. 5
  • For discriminative feedback, dimT E(H, CF ℓ, ϵ) ≤ dimE(RH, ϵ), so LLF is no harder than learning from the induced reward class.Here RH = {rη : η ∈H} is the effective reward class.Source: §3.3 · p. 5
  • The LLF framework also covers non-discriminative feedback that reveals the optimal action, preserving information that reward-only formulations discard.Thus LLF can represent settings where language feedback contains useful information beyond reward.Source: §3.3 · pp. 5–6

4. HELiX Algorithm

HELiX combines hypothesis-consistent feedback filtering, consensus exploitation, and optimistic exploration to solve LLF with regret guarantees tied to transfer eluder dimension. Its LLM implementation approximates these steps by sampling hypotheses and scoring candidate actions.

4. HELiX Algorithm

  • HELiX maintains a confidence set of feedback-consistent hypotheses, chooses an optimistic hypothesis and policy, and uses optimism to guide exploration.The algorithm updates hypotheses from observed actions and feedback before selecting the policy with maximal optimistic reward.Source: §4 · p. 6
  • A consensus stopping criterion exploits an action that is optimal across all surviving hypotheses, preventing unnecessary exploration once the optimal action is identified.The criterion applies when the minimax regret over the confidence set is zero.Source: §4 · p. 6

4.1. Analysis

  • Under Assumptions 1–3, HELiX achieves a regret bound that depends sublinearly on transfer eluder dimension, with slower minimal-assumption rates tightened under structural loss assumptions.The analysis notes an order eO(T^3/4) rate under bounded verifier loss and an order eO(T) rate when the loss has additional structure such as squared loss.Source: §4.1 · p. 6
  • Directly prompting an LLM for actions can fail to explore reliably, whereas HELiX reliably outperforms this greedy baseline in the experiments.The comparison motivates explicit exploration rather than relying on a single hypothesis sampled from history-conditioned prompting.Source: §4.1 · p. 6

4.2. Implementing HELiX with Large Language Models

  • The LLM implementation approximates hypothesis-set maintenance, minimax exploitation, and optimistic exploration using sampled hypotheses, candidate-action scoring, and a shared score matrix.These approximations implement the theoretical algorithm in a compute-efficient manner.Source: §4.2 · pp. 6–7

5. Experiments

Experiments evaluate HELiX on Battleship and Minesweeper using cumulative reward over environment interactions. HELiX consistently outperforms CoT and its no-exploitation ablation, while the practical method depends on assumptions about LLM action selection, scoring, and hypothesis quality.

5. Experiments

  • The no-exploitation ablation shows that optimistic exploration alone is insufficient in the experimental setup.The ablation removes the consensus-based exploitation step from HELiX.Source: §5 · p. 7, §5.1 · p. 7

5.1. Discussion of Results

  • HELiX consistently outperforms both the CoT baseline and the no-exploitation variant across Battleship and Minesweeper.The experiments plot cumulative reward against interaction steps, with shaded areas showing standard error across scenarios.Source: §5.1 · p. 7, Figure 2 · p. 7
  • In information-gathering environments such as Minesweeper, strategic exploration and exploitation outperform the CoT baseline by a large margin.The comparison uses cumulative reward over environment interaction steps.Source: §5.1 · p. 7

5.2. Limitations and Disclaimers

  • The practical implementation assumes that LLMs can select optimal actions under hypotheses and assign fair cross-hypothesis action scores.The paper states that these assumptions may not hold for all LLMs.Source: §5.2 · p. 7
  • HELiX also requires sampled hypotheses to be diverse and faithful to interaction history, but whether current LLMs reliably produce such hypotheses remains uncertain.The paper reports evidence pointing in both directions and calls for further investigation.Source: §5.2 · pp. 7–8

6. Related Work

Related work spans empirical LLM agents, fine-tuning, exploration methods, language-guided learning, and bandit-style feedback models. This paper distinguishes itself by providing a formal LLF framework and guarantees for learning from language feedback.

6. Related Work

  • Prior LLM methods either incorporate feedback into prompts or memory, or update model weights through fine-tuning, while newer work targets exploration empirically.These approaches include supervised fine-tuning, preference-based learning, reinforcement learning, and prompting agents to mimic Bayesian learners.Source: §6 · p. 8
  • Unlike reward-observing UCB-like methods, this framework decodes information from language feedback through a hypothesis-verifier loss without observing rewards.The algorithm follows optimism under uncertainty while constructing confidence sets from feedback.Source: §6 · p. 8
  • Language has previously been used for manuals, behavior shaping, exploration guidance, feedback learning, and language-annotated pretraining followed by reinforcement learning.These approaches improve learning empirically but do not provide a unifying framework or theoretical guarantees.Source: §6 · p. 8
  • In contrast to feedback models that treat language as useful only for decoding rewards, LLF allows language to encode additional learning signals and can be strictly easier than reward-based learning.This distinction separates LLF from rich-feedback settings focused on reward estimation.Source: §6 · p. 8
  • The paper extends the eluder-dimension perspective from reward-aware reinforcement learning to characterize learning complexity in LLF.This extension is presented as a route toward understanding agent learning with generative AI.Source: §6 · p. 8

7. Conclusion

The paper positions LLF as a formal framework that subsumes several feedback-learning paradigms and introduces transfer eluder dimension and HELiX to characterize and solve it. It also identifies gaps between worst-case guarantees and practical LLF complexity, including the role of hypothesis design.

7. Conclusion

  • Transfer eluder dimension captures feedback-based learning complexity, while HELiX provides no-regret guarantees in terms of this measure.The framework treats language feedback as an alternative to scalar rewards and uses a hypothesis verifier to assess consistency.Source: §7 · p. 8
  • LLF can be easy even when transfer eluder dimension is infinite, so the dimension is not a lower bound on learning difficulty.If feedback directly reveals an optimal action, HELiX can solve the instance in one step despite arbitrary rewards.Source: §7 · p. 8
  • Worst-case verifier guarantees may overstate practical difficulty because LLM inductive biases can favor explicitly stated optimal actions.HELiX can exploit this behavior through early stopping, whereas naive reward-driven UCB fails in the described counterexample.Source: §7 · pp. 8–9
  • The hypothesis class is a design choice that can change both the learning target and the resulting behavior and complexity.The experiments use LLM-generated thoughts from reasoning traces, but other hypothesis designs remain an open direction.Source: §7 · p. 9

A. LLF and its relationship to existing paradigms

  • LLF provides a common framework encompassing reinforcement learning, partial monitoring, interaction-grounded learning, discriminative LLF, multi-objective RL, preference-based RL, and imitation learning.These paradigms are represented through different hypothesis reward, feedback, or verifier structures.Source: A. LLF and its relationship to… · pp. 13–14

B.1. Proof Sketch

The proof sketch constructs confidence sets from empirical verifier losses, decomposes regret into version-space widths, and bounds large widths using transfer eluder dimension. Concentration of the verifier loss then yields the HELiX regret guarantee.

B.1. Proof Sketch

  • Confidence sets retain hypotheses whose cumulative empirical verifier loss is within βt of the best hypothesis.The proof aims to show that the true hypothesis remains in these sets with high probability.Source: §B.1 · p. 14
  • Regret is decomposed into version-space widths, measuring the largest reward disagreement among hypotheses still considered plausible at the chosen action.An additional term accounts for the event that the true hypothesis has been excluded.Source: §B.1 · p. 14
  • If an action has width greater than ϵ, transfer eluder dimension limits how often such large-width actions can occur across disjoint historical sequences.This bounds the number of large-width version spaces by O(βT d/ϵ^2).Source: §B.1 · p. 14
  • Under bounded verifier loss, βT has a decay rate, and HELiX regret is bounded above by the no-exploitation-step regret because triggered exploitation makes subsequent per-step regret zero.Source: §B.1 · pp. 14–15

B.2. Complete Analysis

The complete analysis derives HELiX’s regret bound by combining confidence-set validity, regret decomposition, and transfer-eluder control of version-space widths. Stronger verifier-loss structure improves the confidence rate and can tighten the final guarantee.

B.2. Complete Analysis

  • The analysis reduces regret to the expected sum of widths of confidence-set version spaces, provided the true hypothesis remains included with high probability.Upper-confidence action selection converts reward uncertainty within each version space into a regret bound.Source: §B.2 · p. 15
  • Martingale exponential inequalities establish high-probability concentration for verifier-loss differences and support simultaneous confidence-set guarantees.The proof uses zero-mean martingale differences adapted to the history of observed actions and feedback.Source: §B.2 · pp. 15–16
  • With probability at least 1 − δ, actions with version-space width above ε are ε-dependent on fewer than 3βT/ε² disjoint prior subsequences, while some is dependent on at least τ/d − 1.Here d = dimT E(H, ℓ, ε).Source: §B.2 · pp. 16–18
  • The exploitation step cannot increase regret because once triggered, all subsequent per-step regret is zero.Therefore, the full HELiX regret is bounded by the version-space analysis without exploitation.Source: §B.2 · p. 19
  • The regret rate depends on verifier-loss concentration: bounded losses give a slower guarantee, while stronger structure such as α-strong convexity permits tighter bounds.Under suitable structural assumptions, βT can be taken of constant order.Source: §B.2 · p. 19

B.3. Version Space Construction for General Bounded Loss

For general bounded verifier loss, the analysis constructs confidence sets using empirical losses, pseudometric covering, and concentration, producing a slower regret rate under minimal assumptions. For squared loss with sub-Gaussian noise, it derives a rate-optimal bound.

B.3. Version Space Construction for General Bounded Loss

  • Under only bounded verifier loss, confidence-set validity is established through martingale concentration and union bounds over a finite or discretized hypothesis class.The true hypothesis is retained in the version space with high probability.Source: §B.3 · pp. 19–20
  • The hypothesis-space pseudometric measures the supremum verifier-loss discrepancy across actions and feedback, enabling α-covering-number control for infinite hypothesis classes.The analysis verifies that this distance is a pseudometric before applying covering arguments.Source: §B.3 · pp. 20–21
  • Under minimal assumptions, regret has horizon dependence of order eO(T^3/4), and this slower rate is a principled consequence of assuming only bounded verifier loss.Source: §B.3 · p. 22

B.4. Rate-Optimal Bound for Squared Loss

  • For squared verifier loss with sub-Gaussian noise, LLF-UCB achieves a rate-optimal regret guarantee under the stated structural assumptions.The refined analysis permits logarithmic-order confidence parameters.Source: §B.4 · p. 22, §B.3 · p. 22

C.1. Proof for Proposition 1

The proof establishes that transfer eluder dimension is bounded by ordinary eluder dimension under a reward-discriminative verifier, while the auxiliary results characterize optimal consensus actions and discriminative feedback.

C.1. Proof for Proposition 1

  • dimT E(H, CF ℓ, ϵ) ≤ dimE(R, ϵ) because every transfer-independent action sequence is also eluder-independent.The proof maps each transfer-independent sequence to an ordinary eluder-independent sequence of the same length.Source: §C.1 · p. 22

C.2. Proof for Lemma 5

  • A minimax policy supported outside the hypotheses’ common optimal-action set can be improved, so every action in its support must be jointly optimal.The contradiction replaces suboptimal support with a uniform distribution over common optimal actions.Source: §C.2 · p. 23

C.3. Proof of the Discriminative Feedback Example

  • The squared verifier loss is minimized by the generating hypothesis, making the feedback discriminative under the stated reward and feedback construction.The construction defines reward as the expected value of a known feedback statistic and verifies that the true hypothesis minimizes expected loss.Source: §C.3 · p. 23

C.4. Proof of Reasoning Example

The reasoning example shows that structured feedback can sharply reduce transfer eluder dimension, while the extensions clarify scope, contextual applicability, feedback-generation assumptions, verifier noise, and empirical settings.

C.4. Proof of Reasoning Example

  • The reasoning example bounds independent action sequences by |S|L + 1, despite an action space of size |S|L.Feedback identifying the first incorrect reasoning step yields a monotonic sequence of indices, while correction feedback can reveal the correct action.Source: §C.4 · pp. 23–24
  • In the demonstration-feedback variant, the correct reasoning sequence is revealed independently of the action, giving transfer Eluder dimension 1.Any hypothesis pair consistent with the feedback has identical rewards for all actions.Source: §C.4 · p. 24

E.1. Special Case of Reward-Agnostic Feedback

  • Reward-agnostic feedback can identify the optimal action without identifying rewards, so transfer eluder dimension may be large even when efficient action selection is possible.This special case marks a boundary where the proposed complexity measure does not fully capture the usefulness of feedback.Source: §E.1 · p. 24

E.2. Extension to Contextual Bandits

  • The framework extends to contextual bandits by treating context-action pairs as actions, with the context-free analysis carrying over directly.Time-varying available actions and time-indexed policies encode the observed context.Source: §E.2 · pp. 24–25

E.3. Alternative Formulation of Feedback Generation

  • The model-based formulation assumes feedback follows a fixed hypothesis-indexed mapping, whereas a streaming oracle can support stochastic or adversarial feedback but requires new complexity measures.The broader oracle formulation is identified as a future direction for guarantees beyond the hypothesis-testing paradigm.Source: §E.3 · p. 25

E.4. Relaxations of Assumption 3

  • Verifier errors and approximate reward mappings degrade regret gracefully through additional bias terms, including an additive O(δT) term for reward approximation.A Δ-approximately correct verifier adds a linear bias, while uniformly δ-approximate rewards create approximation error over T steps.Source: §E.4 · pp. 25–26

F.1. Environments

  • In Battleship and Minesweeper, agents receive environment feedback while hidden numerical rewards evaluate performance over at most 20 turns.These settings support sequential exploration and exploitation without exposing the numerical reward to the agent.Source: §F.1 · p. 26

F.2. Baselines

  • The CoT baseline prompts an LLM for reasoning and actions, with an epsilon-greedy variant using ε0 = 0.5 and step-based annealing.The exploration probability is reduced according to εt = εt−1 ∗ max(0, 1 − t/20).Source: §F.2 · p. 26

F.3. HELiX Implementation Details

The practical HELiX implementation samples multiple LLM hypotheses and actions, scores their pairings, and exploits an action that is optimal across the remaining hypotheses when one exists.

F.3. HELiX Implementation Details

  • The hypothesis-conditioned policy produces reasoning-based actions, while a separate reward mapping scores each action under each sampled hypothesis.πLLM generates hypothesis-action pairs, and RLLM evaluates their quality in [0, 1].Source: §F.3 · p. 27
  • HELiX maintains feedback-consistent hypotheses and approximates its policy searches with finite candidate hypothesis and action sets.At iteration t, the implementation uses approximate spaces ˆHt and ˆAt rather than enumerating all hypotheses and policies.Source: §F.3 · p. 27
  • The implementation forms an N × N score matrix whose rows are hypotheses and columns are candidate actions, with diagonal entries expected to be high.Each diagonal pair combines a sampled hypothesis with its conditionally generated action.Source: §F.3 · p. 27
  • HELiX exploits immediately when the candidate action sets share an action that is highest-scoring for every remaining hypothesis.This consensus-action test implements the theoretical exploitation step through set intersection.Source: §F.3 · p. 27

F.4. Additional Results

This section presents additional results on HELiX variants, assumption checks, computational costs, and implementation details. It also illustrates the algorithm through Battleship traces, prompt templates, and the selection and sampling procedures.

F.4. Additional Results

  • HELiX with πref performs significantly better than the CoT baseline and other HELiX variants on BATTLESHIP and MINESWEEPER.The variant uses reference-policy re-scoring to break exploration ties.Source: §F.4 · pp. 28–29

F.5. Additional Studies on Assumption Satisfaction

  • Both trajectory-averaged and step-averaged Assumption 3 satisfaction exceed 95% in manually annotated Battleship trajectories.The check tests whether HELiX’s shrinking candidate-hypothesis set retains the true hypothesis throughout each trajectory.Source: §F.5 · p. 29, Table 3 · p. 29

F.6. Discussion on Computational Cost

  • HELiX’s token count is 3.73x to 4.02x the baseline rather than the expected 9x, despite sampling costs that can scale quadratically with top hypotheses.Early termination, parallel sampling, and prefix caching can further reduce practical computational cost.Source: §F.6 · pp. 29–30, Table 4 · p. 30

F.7. Reasoning Traces for Battleship

  • A Battleship reasoning trace shows hypothesis-conditioned scoring assigning 0.3 to an unrelated action and 1.0 to an action aligned with the hypothesis.The trace also demonstrates prompting the LLM to propose multiple valid, diverse actions with associated reasoning.Source: §F.7 · p. 31

F.8. Prompt Templates

  • Prompt templates request hypotheses and actions, including unexpected creative actions, then evaluate each action’s alignment with a hypothesis and assign scores between 0 and 1.Source: §F.8 · p. 32

F.9. Code Implementation

  • The implementation builds a hypothesis-action score matrix, returns consensus actions when row-wise argmax sets intersect, and otherwise applies elimination and tie-breaking.With random actions, tie-breaking subtracts their average scores before further hypothesis elimination and action selection.Source: §F.9 · pp. 33–34
  • The implementation eliminates hypotheses by retaining rows with the highest score while preserving all actions.It constructs a reduced score matrix using the selected hypothesis rows and the existing action columns.Source: §F.9 · pp. 34–35
  • Two-tiered argmax sampling first selects the hypothesis with the highest overall score, then chooses its highest-scoring action.The implementation identifies the best hypothesis via the maximum row score and returns that hypothesis, action, score, and action index.Source: §F.9 · p. 35
Loading 2506.10341v2…