PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Jev be Your Q or Policy in Reinforcement Learning?

Yi Ma, Tianpei Yang, Yaodong Yang, Weixun Wang, Hongyao Tang

arXiv:2610.11692v1cs.LGcs.AI

TL;DR

Foundation models offer priors for RL, but token-by-token generation makes queries sequential and costly, and Jev’s role in RL had remained untested. The paper maps Jev’s answer forms to RL functions and builds three training uses; across MiniGrid and Atari, Jev-assisted training improves learning, including where learners cannot progress alone.

  • Problem

    Jev had not been studied as a sequential decision-maker in RL or as a component inside an RL learning loop, leaving its RL roles and training potential untested.

    Source: §1 · pp. 1–2

  • Method

    The paper maps Jev’s typed answers to RL functions and builds algorithms using it as a reference policy, exploration judge, and replay rater.

    Source: §1 · p. 2

  • Results

    Across nine MiniGrid tasks and three Atari games, training with Jev improves early learning and can let learners outperform direct Jev control while acting independently at evaluation.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    A decision model need not be a reliable standalone controller to improve RL training, and guidance can be adapted to the learner’s needs.

    Source: §6 · p. 10

    Experiments are confined to MiniGrid and Atari, and improved benchmark performance does not establish the safety of learned policies in real-world settings.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Foundation models supply reinforcement learning (RL) with priors that mitigate its longstanding weaknesses in sample efficiency and transfer, but their token-by-token generation makes queries sequential and costly. Jev, a recently released decision model, generates nothing and returns calibrated, typed answers in a single forward pass. Existing work studies foundation models in RL either as models to be trained or as generators to be prompted, and Jev belongs to neither category, having so far served only as a black box in single domains. How well such a model decides on its own in RL environments, and how it can improve RL as a component of training, therefore remain unaddressed. To this end, in this paper we first examine the requirements that the objects of an RL system place on the answers they consume, and establish that Jev can fulfill all of them except the cardinal use of a value function. The remaining objects form positions that admit several roles each. We then construct algorithms that employ Jev at three of these positions, as a reference policy, an exploration judge, and a replay rater, to improve sample efficiency, exploration, and learning performance. Across nine MiniGrid tasks and three Atari games, training with Jev outperforms a standard RL learner, including where the learner makes no progress alone, while the model itself remains untrained. To our knowledge, we present the first use of Jev within the RL learning process and establish a frozen decision model as a usable component of RL training, inviting further exploration of how Jev and other advanced decision models can improve RL.

1 INTRODUCTION

Jev’s single-pass typed answers offer a different interface from generative foundation models, but its role in reinforcement learning had not been established. The paper maps Jev to RL functions and tests three training roles across MiniGrid and Atari.

1 INTRODUCTION

  • Unlike token-generating models, Jev returns calibrated, typed answers in one forward pass, while existing RL work uses foundation models as models to train or generators to prompt.Jev accepts a situation description and typed questions, returning a choice, rubric score, or probability.Source: §1 · p. 1
  • The paper evaluates whether Jev decides directly and whether it can improve RL as a training component.The experiments use nine MiniGrid tasks and three Atari games, with matched information interfaces for decision comparisons and fixed interaction budgets for learner comparisons.Source: §1 · p. 2
  • Jev can serve as a reference policy, exploration judge, or replay rater to guide an RL learner’s policy, exploration, or replay.These roles target sample efficiency, exploration, and learning performance.Source: §1 · p. 2
  • The paper examines Jev as a function approximator for policy, value, reward, and transition objects, finding that only cardinal use of a value function falls outside its answer forms.The excluded use requires arithmetic on the model’s answer.Source: §1 · p. 2
  • The paper reports the first use of Jev in the RL learning process, demonstrating efficacy across nine MiniGrid tasks and three Atari games, including tasks where learners cannot progress alone.The experiments also compare direct decision-making under matched information interfaces and test training methods under fixed interaction budgets.Source: §1 · p. 2

2 BACKGROUND

Prior work uses foundation models in several RL roles, but Jev had not been evaluated in sequential decision-making or a learning loop. The background introduces Jev’s typed-question interface and the function notation used to describe its roles.

2.1 RELATED WORK

  • Foundation models have been used as policies, reward designers or providers, exploration guides, teachers, and direct value or policy priors in RL.Prior value or policy priors were demonstrated in text-based decision-making rather than continuous control or pixel-based benchmarks.Source: §2.1 · p. 2
  • Jev answers typed questions in one forward pass, whereas prior Jev applications were mostly single-domain black-box uses and zero-shot demonstrations.No cited work measures Jev on a sequential decision or places it inside a learning loop, leaving its RL capability boundary unmapped.Source: §2.1 · pp. 2–3

2.2 PRELIMINARIES AND NOTATION

  • The paper represents Jev as a question-selected function, fu(x) = Jev(u, x), from a situation to the answer space declared by question u.A choice question yields a distribution over options, a score question gives a point on an ordered rubric, and a proposition question returns a probability.Source: §2.2 · p. 3
  • A request may include several independently answered questions, though the paper formalizes one question per call so each role corresponds to one function.Interface limits, context budget, latency, and cost are deferred to Appendix A.Source: §2.2 · p. 3

3 JEV AS A DECISION MAKER: ITS ROLES IN RL

The paper assesses whether Jev’s typed answers can meet the requirements of RL functions, then groups compatible functions into positions Jev can fill. Only cardinal use of a value function is excluded; the remaining positions support distinct roles.

3.1 WHAT IT TAKES TO FULFILL A ROLE IN RL

  • RL function approximators must choose actions, value state-action pairs, assign rewards, or predict next states, and Jev’s choice, score, and noul answers can fit these forms.A role is supported when the operations RL applies to its answers are defined for that answer form.Source: §3.1 · p. 3
  • Only cardinal value-function use is excluded; policy, transition, and reward uses require, respectively, finite action and state sets and a shared reward rubric.Jev can rank values, but its ordered score rubric does not supply the cardinal scale needed for arithmetic updates.Source: §3.1 · p. 4

3.2 THE ROLES JEV CAN PLAY

  • Jev’s four positions are Policy, Score Rater, Meta Regulator, and Transition, each defined by its question’s input and answer space.Policy can act online or label states; Score Rater can provide ordinal rankings or a shared rubric for rewards, shaping, bonuses, or credit assignment; Meta Regulator decides whether an event occurs; Transition predicts among finite states.Source: §3.2 · p. 4
  • The Transition position is set aside because it constrains the state space rather than supplying the learner’s signal.The paper builds on the first three positions in its subsequent algorithms.Source: §3.2 · p. 4

4 JEV FOR RL: ALGORITHMS

The paper develops Jev-based RL algorithms that use it as a reference policy, exploration judge, or replay rater. Each channel supplies a different training signal while Jev remains frozen.

4 JEV FOR RL: ALGORITHMS

  • Three Jev channels guide policy learning, exploration, and replay prioritization, as organized in the framework figure.The channels use Jev as a policy reference, exploration gate or novelty rater, and experience-importance judge.Source: §4 · p. 4, Figure 1 · p. 5

4.1 RL WITH JEV REFERENCE

  • Jev-BC imitates Jev’s preferred legal actions, Jev-PEX lets the learner assess Jev’s choice against a student proposal, and Jev-Mixed Sampling uses Jev actions for exploratory draws.The reference stays unupdated, requires no offline dataset or pretrained student, and influences learning only at states the learner visits.Source: §4.1 · p. 5

4.2 RL WITH JEV EXPLORATION

  • Jev-Gate replaces ϵ-greedy’s exploration decision, while Jev-Bonus adds action-, transition-, or state-novelty bonuses to rewards and stores them for replay.The bonuses are auxiliary objectives without a policy-invariance guarantee, and their empirical value depends on their content and scale.Source: §4.2 · p. 6

4.3 RL WITH JEV-REWEIGHTED REPLAY

  • Jev-Reweighted Replay uses Jev’s importance ratings as sampling priorities, with a floor that preserves support for every stored transition.Intuitive-importance and success-importance variants differ by instruction; new transitions are rated on insertion, up to eight are re-rated every 100 environment steps, and no importance-sampling correction is applied.Source: §4.3 · p. 6

5 EXPERIMENTS

The experiments evaluate frozen Jev as a direct decision maker and as a learning component across nine MiniGrid tasks and three Atari games. Results show useful but interface-dependent standalone decisions and selective learning gains through policy reference, exploration, and replay.

5.1 RQ1: HOW CAPABLE IS JEV AS A ZERO-SHOT POLICY?

  • With matched structured inputs, Jev matches or exceeds Qwen3-8B on all nine MiniGrid tasks, but Atari results are mixed and direct control is not uniformly reliable.FourRooms and MultiRoom-N4 use a full-navigation input; the other tasks use the direct-Jev interface.Source: Figure 2 · p. 7, §5.1 · p. 7

5.2 RQ2: HOW MUCH DOES THE INFORMATION INTERFACE MATTER?

  • Removing engineered navigation sharply reduces success on FourRooms and MultiRoom-N4, while DoorKey-8×8 and MultiRoom-N2 remain largely solvable.The comparison retains identical observational memory and task rules, isolating the navigation features.Source: §5.2 · p. 8

5.3 RQ3: CAN JEV GUIDE POLICY LEARNING?

  • Jev-BC, Jev-PEX, and Jev-Mixed Sampling improve learning on multiple tasks, including where DQN remains unsuccessful within the budget.Their ranking varies; Seaquest benefits more from reference-guided collection than from imitation.Source: Figure 3 · p. 8, §5.3 · p. 8

5.4 RQ4: CAN JEV IMPROVE EXPLORATION?

  • Exploration gains are selective: state-novelty data improves LavaGap despite matched mean spatial coverage, and original Jev scores beat group means on two of three tasks.The fixed-experience comparison indicates reward distinctions beyond coarse visitation summaries.Source: §5.4 · p. 9

5.5 RQ5: CAN JEV IMPROVE REPLAY?

  • Jev improves selected replay tasks but not consistently over PER; on MultiRoom-N2, intuitive weights beat both controls, while success-oriented weights beat the reward-class control but trail uniform replay.The controls hold experience fixed to isolate replay effects.Source: §5.5 · p. 9

6 CONCLUSION

The conclusion presents frozen Jev as a potentially useful component of RL training, with benefits that depend on the task, information interface, and how its judgments enter learning.

6 CONCLUSION

  • Across the experiments, Jev can improve early learning, sometimes enabling students to outperform direct Jev control while acting independently at evaluation.The conclusion frames Jev as guidance rather than a required standalone controller.Source: §6 · pp. 9–10

REPRODUCIBILITY STATEMENT

  • Jev results are tied to the recorded API version and input construction; Appendices C and F document protocols and query details.The source files include figure inputs and plotting scripts.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

  • The experiments are confined to MiniGrid and Atari, and improved benchmark performance does not establish safety in real-world settings.The ethics statement calls for independent evaluation, safeguards, and human oversight for safety-critical applications.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

  • AI coding and language tools assisted implementation and manuscript polishing; Jev and Qwen3-8B were experimental subjects with roles and interfaces specified separately.Source: AI USE STATEMENT · p. 10

A Jev: Experimental Interface

The listed appendix materials cover an early literature snapshot, experimental details and learning curves, analysis experiments, and subgoal guidance.

A Jev: Experimental Interface

  • The appendix contents list an early literature snapshot, experimental details and learning curves, analysis experiments, and Jev for subgoal guidance.Source: A Jev: Experimental Interface · p. 14

A JEV: EXPERIMENTAL INTERFACE

The appendix records experimental interface settings and surveys early Jev-related work by application. These applications motivate studying fixed-model judgments as reusable components in systems, complementing the paper’s focus on their effects on a separately trained RL learner.

A JEV: EXPERIMENTAL INTERFACE

  • The Jev settings are experimental configurations, not measurements of API capacity or vendor-reported performance.Action preference and numerical rating use the same output primitive, but distinct questions; numerical rating spacing comes from an assigned auxiliary scale rather than calibration to task returns.Source: A JEV: EXPERIMENTAL INTERFACE · p. 15

B JEV: AN EARLY LITERATURE SNAPSHOT

  • The literature snapshot covers document tasks, agent infrastructure, simulation and annotation, and evaluation, including studies of typed-decision interfaces that do not evaluate Jev itself.Examples include clinical factuality (Huang et al., 2026), agent memory (Jiang et al., 2026), population simulation (Li, 2026), and model comparisons (Ibrahim & Zaki, 2026).Source: B JEV: AN EARLY LITERATURE SNAPSHOT · p. 15
  • These applications motivate studying reusable decisions in agent systems; this paper instead examines how a fixed model’s judgments affect a separately trained RL learner through defined channels.The appendix provides context for the decision-model interface, not a quantitative comparison with the control experiments.Source: B JEV: AN EARLY LITERATURE SNAPSHOT · p. 15

C EXPERIMENTAL DETAILS AND LEARNING CURVES

The appendix specifies evaluation, input construction, learning and scoring configurations, and diagnostic procedures. Its learning curves supplement endpoint comparisons and use the same DQN source across categories.

C EXPERIMENTAL DETAILS AND LEARNING CURVES

  • MiniGrid evaluations use seeds 700000–700049 and a greedy student, while Atari uses seeds 700000–700004 and the student’s ϵ = .01 policy.Evaluation uses native task rewards without auxiliary training bonuses; plotted uncertainty is SEM over training seeds, not evaluation episodes.Source: C EXPERIMENTAL DETAILS AND LEARNING… · pp. 15–16
  • MiniGrid and Atari inputs differ: MiniGrid teachers use local observations and task-specific state descriptions, while Atari inputs describe RGB-detected objects and estimated motion.FourRooms and MultiRoom-N4 action teachers also retain observed-map memory and engineered route advice; the state-novelty query omits action proposals.Source: C EXPERIMENTAL DETAILS AND LEARNING… · p. 16
  • The Qwen3-8B comparison uses the same structured situations, questions, and legal-action descriptions as Jev, with temperature-zero JSON action output and thinking disabled.The navigation ablation uses a shared raw-memory prompt in both arms; its four tasks use 50 paired seeds each.Source: C EXPERIMENTAL DETAILS AND LEARNING… · p. 16
  • MiniGrid DQN variants use an MLP and one update per ten environment steps, while Atari DQN variants use a convolutional learner and one update per four decisions.Jev-PEX and Jev-BC use actor learners, and DQN is their common task baseline; diagnostic fixed-experience fits retain the learner while changing one supervision source at a time.Source: C EXPERIMENTAL DETAILS AND LEARNING… · pp. 16–17
  • Figures 6–8 supplement endpoint comparisons with learning curves; all categories share the same DQN source, and axes stop at 10k MiniGrid and 50k Atari steps.The reference-method plot’s dashed horizontal line is independently evaluated pure-Jev policy performance, with no training-step dependence.Source: C EXPERIMENTAL DETAILS AND LEARNING… · p. 17, Figure 6 · p. 17

D ANALYSIS EXPERIMENTS

The analyses test how Jev affects learning through collected data, fine-grained state scores, replay sampling, and milestone-based subgoal guidance. Results show benefits from some data and rating choices, while subgoal guidance did not reliably produce learning gains under the tested protocol.

D ANALYSIS EXPERIMENTS

  • Jev-collected data improves mean success on FourRooms and LavaGap, but results are mixed on MultiRoom-N2, even after auxiliary rewards are removed.State-novelty data improves LavaGap despite matching DQN data in mean spatial coverage, suggesting the benefit involves useful experience rather than simply more locations visited.Source: D ANALYSIS EXPERIMENTS · p. 17, Figure 10 · p. 19
  • After 10k steps, Jev-PEX agrees with Jev more often than Jev-BC on both measured tasks, yet both students remain below direct Jev control on FourRooms.Action agreement does not guarantee episode success.Source: D ANALYSIS EXPERIMENTS · p. 17, Figure 9 · p. 18
  • Original Jev state scores outperform scores averaged within visitation and collection bins on LavaGap and DoorKey-5×5, while FourRooms performs similarly.The averaging control preserves group means but removes distinctions among observations within each group.Source: D ANALYSIS EXPERIMENTS · pp. 17, 19, Figure 11 · p. 19

E JEV FOR SUBGOAL GUIDANCE

  • The subgoal channel tests whether Jev can select milestones that guide behavior toward events requiring several actions to complete.For MiniGrid, candidate milestones are constructed from observed objects and connectivity, such as getting a key, opening a door, traversing a passage, or reaching the goal.Source: E JEV FOR SUBGOAL GUIDANCE · p. 19, §E.1 · pp. 19–20

D ANALYSIS EXPERIMENTS

  • On MultiRoom-N2, intuitive replay weights outperform uniform and reward-class controls; success-oriented weights beat the reward-class control but underperform uniform replay.On LavaGap, success-oriented weights yield the highest mean success, with large variability across seeds; following Jev ratings is not consistently better than uniform sampling.Source: D ANALYSIS EXPERIMENTS · p. 19, Figure 12 · p. 20

E.1 CHANNEL

  • A selected milestone earns an auxiliary completion reward of α = .025, capped at .1 per episode, with completion checked by the environment.The selector is queried again after completion, timeout, or prolonged lack of progress; repeating a completed milestone earns no further reward.Source: §E.1 · p. 20
  • In the primary MiniGrid comparison, students choose primitive actions from ordinary observations and do not receive the selected milestone as an extra input.Uniform and rule-based selectors share Jev’s candidate set and reward mechanism, separating milestone-based reward guidance from learned goal-conditioned options.Source: §E.1 · p. 20

E.2 RESULTS

Subgoal guidance did not reliably produce learning gains in the tested settings. The diagnostics point to a mismatch between milestone selection and what the current controller can achieve within its execution horizon.

E.2 RESULTS

  • DQN and all tested goal selectors finish at zero success after 20k steps on DoorKey-8×8 and FourRooms in the single-seed diagnostic.FourRooms briefly reaches 10% with Jev at 8k, but the gain does not persist; rule-based variants with progress rewards or explicit goal inputs also finish at zero.Source: §E.2 · p. 20
  • With 50k steps of shared low-level pretraining and 50k student steps, Jev-weighted goals yield 382.7 on Seaquest versus 345.3 for uniform goals, but 17.2 on Freeway versus 19.7.The 100k-interaction diagnostic therefore gives mixed evidence for semantic goal selection even with a pretrained controller.Source: §E.2 · p. 20
  • In the common 12M-decision Montezuma’s Revenge prefix, 77.1% of Jev goal selections target the key, which is completed twice among 481,153 selections.Overall selected-goal completion is 0.36%, compared with 1.98% for uniform selection at the same reward coefficient; the query lacks measured success rates for the current controller.Source: §E.2 · pp. 20–21
  • The missing property is effective temporal guidance: connecting task prerequisites to milestones the current controller can achieve within its execution horizon.The paper concludes that local action and experience judgments are easier to turn into learning gains than milestone preferences requiring an executable sequence.Source: §E.2 · p. 21

F STRUCTURED INPUT EXAMPLES

This section presents structured-input examples by pairing environment scenes with logged requests and distinguishing shared conventions from method-specific questions, evidence, outputs, and memory inputs across Atari action and exploration judgments, Montezuma’s Revenge subgoals, and a matched Jev–Qwen action interface.

Shared request structure and answer scale

  • Each card shows actual queried scene fields and a method’s logged request; Tables 4–6 separate shared conventions from method-specific questions, evidence, outputs, and memory inputs.Different rows can represent different moments, and matching fields do not imply matching questions.Source: F STRUCTURED INPUT EXAMPLES · p. 22, Table 4 · p. 22, Table 5 · p. 23
  • Each question returns its own distribution over ordered rating levels mapped from zero to 1, and its expected value is a score, not an action-selection probability.The shared choices are zero, low, medium, high, very_high, and unknown, mapped to 0, .25, .5, .75, 1, and 0.Source: F STRUCTURED INPUT EXAMPLES · p. 22, Table 4 · p. 22, Table 5 · p. 23

Shared action and coordinate conventions

  • MiniGrid queries use named primitive actions and agent-relative object coordinates, while Atari uses screen pixels; non-Jev controls make no Jev query and variants reuse one.Atari coordinates are RGB pixels with x increasing right and y increasing down.Source: F STRUCTURED INPUT EXAMPLES · p. 23

Navigation memory is a different input

  • FourRooms and MultiRoom-N4 action queries add within-episode observed-map memory, agent pose, visit counts, and recent movements; this clears on reset and differs from exploration memory.The navigation ablation removes candidate routes and their progress/detour evidence while retaining observational memory.Source: F STRUCTURED INPUT EXAMPLES · p. 24, Table 6 · p. 24

Atari method-specific state examples

  • Atari scene examples list detected object categories, boxes, and estimated motion, but are non-exhaustive, so missing detections do not establish absence.This caveat is stated for both Seaquest and Freeway scene excerpts.Source: F STRUCTURED INPUT EXAMPLES · pp. 34–35

F STRUCTURED INPUT EXAMPLES

  • Freeway and SpaceInvaders examples encode action, gate, novelty, transition-diversity, state-novelty, and replay queries as typed JSON fields.The SpaceInvaders scene excerpt contains RGB-derived object categories, bounding boxes, and estimated motion; it is not exhaustive, and a missing detection does not establish absence.Source: F STRUCTURED INPUT EXAMPLES · pp. 35–36

Montezuma’s Revenge: image PPO+RND subgoals

  • Montezuma’s Revenge subgoal queries provide RGB-derived regions, key and inventory evidence, room geometry, and death rules to select milestones for an image-only PPO+RND controller.A visible floor key does not confirm carried inventory, and locked doors require the key; lethal creatures, pits, and unsupported falls can cost a life.Source: F STRUCTURED INPUT EXAMPLES · p. 37

Pure Jev and the matched Qwen interface

  • Pure Jev executes its most likely action, while matched Qwen receives the same structured state, task question, and legal-action descriptions and must return one legal action as JSON.Qwen uses qwen3-8b at temperature zero, with a 48-token limit and thinking disabled; the specified MiniGrid tasks use either full-navigation or direct-Jev local-observation settings.Source: F STRUCTURED INPUT EXAMPLES · p. 37
Loading 2610.11692v1…