Source-linked AI summary
Up and Down the Abstraction Ladder: Code-Based Skills for Language Agents
Bartłomiej Cupiał, Jens Tuyls, Maciej Wołczyk, Davide Paglieri, Martin Klissarov, Benjamin Eysenbach, Piotr Miłoś, Karthik R. Narasimhan
TL;DR
Long-horizon language agents must chain low-level actions, while abstraction can reduce repeated decisions but may leave gaps requiring primitive control. We compare primitive, skill, and mixed agents using CodeHack in NetHack; skills nearly triple progression and reduce episode inference cost, and skill-based controllers gain more in RL.
Problem
Long-horizon environments require language agents to chain many low-level actions, creating a need to understand how temporal abstraction affects performance, inference cost, and learning.
Source: §1 · pp. 1–2Method
Results
Skills nearly triple NetHack progression and reduce per-episode inference cost by 86%; in RL, skill-only and mixed controllers gain 7.2x and 8.6x primitive-only gains in dungeon depth.
Source: §1 · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
Language agents struggle to act and learn in environments that require long sequences of low-level actions. Code-based abstractions can make these agents more productive by letting them invoke reusable skills instead of repeatedly selecting individual actions. The code handles recurring local decisions, while the language model decides which skills to use and how to combine them. Yet abstractions are leaky, and situations beyond a skill's capabilities may require a return to primitive actions. Motivated by this tradeoff between productivity and flexibility, we systematically study how code-based action abstraction affects the performance, inference cost, and learning of language agents. We study this in NetHack, a challenging, long-horizon game environment, using CodeHack, our library of code-based skills with natural-language descriptions. We use this library to compare agents restricted to primitives with those using semantic skills alone or in combination with primitives. We evaluate these agents in three settings: zero-shot prompting, supervised fine-tuning, and reinforcement learning. Across a broad zero-shot evaluation on NetHack, we find that compared with primitives, skills nearly triple game progression, while reducing inference cost per episode by 86%. Combining skills with primitives retains much of this benefit while preserving a path back down to low-level actions. Finally, in RL, we find that skill-based agents learn significantly faster than agents acting on primitives, achieving a 7.2x larger average gain in dungeon level over the same training budget. These results show that a supplied skill library can improve performance, efficiency, and learning, while retaining primitives provides flexibility when the library is insufficient. We release CodeHack together with training and evaluation code.
1 Introduction
Long-horizon agents must chain low-level actions, while abstraction can reduce this burden but may leave gaps that require primitive control. This paper studies primitive, skill, and mixed control with CodeHack, reporting performance, cost, and learning benefits from skills.
Motivation
- NetHack’s long games, high cost of mistakes, and varied situations make complete fixed skill coverage impractical, motivating mixed control that can return to primitives.The study first isolates navigation, exploration, combat, and item use in MiniHack, then evaluates long-horizon behavior in NetHack.Source: §1 · p. 2
Contributions
- The study systematically tests action abstraction for long-horizon language agents and includes ablations relevant to skill coverage.The comparison spans agents using primitives, skills, or both.Source: §1 · p. 2
- CodeHack supplies reusable Python skills and shared training and evaluation code for comparing primitive-only, skill-only, and mixed control in zero-shot, SFT, and RL settings.The shared control layer supports both language-model agents and learned RL policies.Source: §1 · p. 2, Figure 2 · p. 3
Zero-shot control and RL
- Skills gain 55 percentage points on MiniHack and nearly triple NetHack progression while cutting per-episode inference cost 86%; RL gains are 7.2x for skill-only and 8.6x for mixed versus primitive-only.Mixed control generally suffers less from missing skills than skill-only control, and trajectories show primitive fallback before agents return to skills.Source: §1 · p. 2
2 Preliminaries
The paper frames its skills through the options formalism, adapting it to partially observable interaction histories. It distinguishes these environment-operating policies from other LLM-centric uses of “skills.”
Options
- In NetHack’s partially observable setting, each skill is defined over interaction history, with an initiation set, intra-skill policy, and termination function; the high-level policy can select skills, primitive actions, or both.The high-level policy is a language model using call-and-return control; a skill terminates on task completion or an unexpected situation such as lost health.Source: §2 · p. 3
Options and skills
- Here, skills are policies operating in the same environment as the controller, unlike agentic markdown files, reasoning procedures, or tools.The paper presents options as the formal framework underlying its work.Source: §2 · p. 3
Zero-shot results
- Figure 3 compares GPT-5 skill and primitive control in MiniHack success rates and NetHack milestone reach, with skill-based control reaching deeper NetHack milestones.Each plot averages 32 episodes per task.Source: Figure 3 · p. 4
3 CodeHack: a library of code skills for NetHack
CodeHack combines reusable code-based skills with a runtime shared across primitive-only, skill-only, and mixed agents. Its inspectable, broad repertoire targets recurrent behaviors but is intentionally incomplete.
CodeHack architecture
- CodeHack provides one shared runtime for primitive-only, skill-only, and mixed agents across NetHack and MiniHack, with a controller choosing actions and the runtime executing them.The runtime tracks symbolic state, inventory, and map memory; skills can access this structured state and adapt to observations while issuing primitive actions.Source: §3 · p. 4
Code-based skills
- Explicit names, readable implementations, and testable behavioral specifications make code-based skills easier to inspect, modify, and extend than latent skills.They give language models semantically grounded actions such as exploration, combat, and item management.Source: §3 · p. 4
Skill repertoire
- The NetHack experiments use 78 curated skills; MiniHack uses subsets of that repertoire plus 7 skills for lava crossing and levitation.Examples include explore, goto_room, fight_melee, pickup_armor, open_doors, leave_shop, and emergency_escape.Source: §3 · p. 4
Coverage
- The skill action space prioritizes frequent, reusable behaviors rather than complete NetHack coverage, so agents must still handle missing skills, edge cases, and local failures.The library is strong but incomplete by design.Source: §3 · p. 4
4 Experimental setup
The experiments use controlled MiniHack tasks and long-horizon NetHack evaluation, holding game settings and prompts fixed across action interfaces. They compare zero-shot, missing-skill, PPO, and SFT-initialized policies using performance and cost measures.
Environments
- MiniHack’s seven tasks isolate navigation, exploration, combat, and item use, while NetHack tests coordinating these behaviors over thousands of steps across more than 50 dungeon levels.NetHack agents seek the Amulet of Yendor and must return to the surface.Source: §4 · p. 5
Environment setup and action interfaces
- Across action interfaces, the runtime, game parameters, observation format, and prompt template remain fixed; only available actions and their descriptions change.Agents receive a bounded interaction history and current observation through a chat template.Source: §4 · p. 5
Evaluation metrics
- MiniHack is evaluated by task success, while NetHack uses score, progression, and maximum dungeon level; the study also reports token use and estimated episode inference cost.Progression estimates progress from highest dungeon and experience levels using a mapping derived from human trajectories.Source: §4 · p. 5
Zero-shot evaluation
- Zero-shot evaluation compares all three interfaces on NetHack across 14 models, while GPT-5 compares primitive-only and skill-only control on MiniHack.The model sweep covers GPT, Llama, Gemma3, Gemma-4, and Qwen-3.5 families.Source: §4 · p. 5
Missing-skill evaluation
- The missing-skill experiment removes one skill family at a time and compares skill-only with mixed control using the same reduced library for Gemma-4-31B.Mixed control retains primitive actions under the reduced-library condition.Source: §4 · p. 5
Learning experiments
- PPO trains Llama-3.1-8B-Instruct and Qwen-3.5-4B under all three interfaces, comparing policies after 264 gradient updates and plotting learning against updates and wall-clock time.Equal rollout lengths can contain different numbers of primitive environment transitions because skills may execute multiple actions per decision.Source: §4 · p. 5
SFT initialization
- For skill-only RL, both models are fine-tuned for one epoch on 1,024 zero-shot teacher trajectories, then evaluated directly and used to initialize additional PPO runs.The PPO comparison uses the same number of gradient updates, while SFT adds dataset-collection and supervised-training cost.Source: §4 · pp. 5–6
5 Language agents with skills: Experimental results
Across the evaluated tasks, code-based skills improve performance and reduce inference cost; mixed control retains most skill benefits, and skill-based agents learn faster under RL.
Zero-shot performance
- Skill-only agents achieve 2.9x the progression, 3.8x the score, and 2.4x the dungeon depth of primitive-only agents across the 14-model zero-shot NetHack sweep.The values are averages across models.Source: Table 1 · p. 6, §5 · p. 6
- Skills improve success rates across all seven evaluated MiniHack tasks, averaging a 55-percentage-point gain over primitive control.On WoD-Hard-Full, success rises from 28% to 97%; on Quest-Hard, skills reach 25% success while primitives solve no episodes.Source: §5 · p. 6
Inference cost
- 86% lower inference cost per episode and 74% lower token usage versus primitives; skills also move most models deeper at lower cost.Skills reduce language-model calls per episode by an average factor of 5.1 because code policies handle many environment steps on CPU.Source: §5 · pp. 6–7, Figure 4 · p. 7
Mixed control
- Mixed control retains 95% of skill-only progression, 91% of score, and 97% of dungeon depth, at 2.3x the inference cost and 2x token usage.It occasionally exceeds skill-only performance and allows primitive actions when skills do not cover a behavior.Source: §5 · p. 7
Skill coverage
- Mixed control is generally less affected by removed skill families; in selected trajectories, primitives handle behaviors unavailable to skills before skill-based control resumes.Examples include dropping gold to satisfy a guard and throwing a javelin when ranged-combat skill returns without acting.Source: §5 · p. 7
Reinforcement learning
- 7.2x and 8.6x larger average dungeon-level gains for skill-only and mixed control, respectively, than primitive-only control over the same RL budget.Learning curves show the faster improvement across training steps and wall-clock time; the final performance gap is larger than at the start.Source: §5 · pp. 7–8, Table 2 · p. 8
SFT initialization
- SFT initialization raises mean progression by 29% for Llama-3.1-8B-Instruct and 20% for Qwen-3.5-4B; mean scores rise approximately 51% for both.The SFT-initialized policies gain less during RL, but their stronger starting points yield better final policies within the evaluated budget.Source: §5 · p. 8
6 Related work
The paper situates its work among language-agent control and hierarchical RL research, emphasizing its focus on sequencing supplied code skills and comparing them with primitives.
Language agents for control
- Rather than generate task-specific low-level policies as Liang et al. [2023] or update skill sets as Wang et al. [2024], this work sequences existing code policies.The language model acts as the high-level policy over code-based skills.Source: §6 · pp. 8–9
- Unlike Jeurissen et al. [2024], this work systematically compares primitive and skill-based NetHack play using a runtime that supports both interfaces.It also studies SFT and RL effects on language-model skill control.Source: §6 · p. 9
Hierarchical RL
- The work assumes skills are supplied as prior knowledge, structurally resembling Parr and Russell [1997], and leaves skill learning as an open problem outside its scope.This assumption lets the study quantify how a strong skill repertoire affects learning.Source: §6 · p. 9
7 Discussion
CodeHack’s mixed-control design pairs reusable skills with primitive fallback, but the library remains incomplete and does not solve NetHack. Its runtime maintains structured state and interrupts skills when local conditions change, giving controllers a route to lower-level action.
7 Discussion
- The hand-designed skill library remains incomplete and does not solve NetHack; mixed control retains most skill-only benefits while preserving primitive fallback.The authors identify fine-grained tactics, rare game knowledge, resource management, and strategic planning as situations current skills may not encode.Source: §7 · p. 9
A Technical Details of CodeHack
- CodeHack is a control layer of reusable Python skills, released with training and evaluation code to support study and extension.The appendix presents implementation details to make the artifact easier to understand, reproduce, and extend.Source: §A.1 · p. 13, A Technical Details of CodeHack · p. 13
A.1 Library overview
- Skills were developed iteratively from human NetHack knowledge, tested in end-to-end tests and playthroughs, and refined after observed failures.The library prioritizes routines intended to be reliable and broadly useful across evaluated tasks.Source: §A.1 · p. 13
A.2 Wrapper and control interface
- The wrapper exposes named skills through call-and-return execution, while allowing primitive commands to be registered for primitive-only, skill-only, or mixed control.Skills can return after reaching a goal, failing to proceed, an interrupt, or episode termination; the runtime tracks primitive steps and reward.Source: §A.2 · p. 14
A.3 Internal state representation
- A persistent state manager turns raw observations into structured information about the current game, including map, inventory, navigation, character, combat, and traps.Persistent level memory lets skills use previously observed rooms, corridors, stairs, doors, and hazards, not only what is visible now.Source: §A.3 · p. 14
A.4 Panic handlers and interruption
- After each primitive step, panic handlers can interrupt a skill when the agent loses HP or a newly reachable hostile appears, returning control to the outer controller.The controller can then select another skill or, in mixed control, use lower-level actions suited to the changed state.Source: §A.4 · pp. 14–15
A.5 Experimental skill set
The experimental skill repertoire covers recurring NetHack behaviors, while the appendix describes the environments, observations, and progress measures used to evaluate agents. NetHack score is an imperfect proxy for meaningful progress, so the study also reports BALROG NLE-progress.
A.5 Experimental skill set
- CodeHack’s experimental skills cover exploration, navigation, combat, terrain interaction, item and equipment management, recovery, and MiniHack-specific lava crossing.The listed repertoire includes skills such as explore, fight_melee, pickup_armor, emergency_escape, and levitate_over_lava_river.Source: §A.5 · p. 15
- Additional skills handle food, prompts, recovery, and cleanup, while MiniHack configurations add specialized lava-crossing and levitation skills.Examples include eat_food_inventory, heal, drop_cursed_items, and freeze_lava_wand.Source: §A.5 · p. 15
B Additional experimental details
- The appendix documents NetHack and MiniHack, the NetHack progress metric, the MiniHack suite, prompt templates, and SFT/RL hyperparameters.NetHack runs use a 5,000-step episode limit and a 150-step no-progress timeout unless otherwise noted.Source: B Additional experimental details · p. 15, §B.1 · p. 16
B.1 Environment details
- NetHack combines procedurally generated dungeon exploration, combat, inventory management, and long-horizon planning, with more than 50 levels and an eventual return to the surface.NLE provides a synchronous RL interface while preserving the underlying game dynamics.Source: §B.1 · p. 15
- Language agents receive natural-language descriptions and an ASCII map rather than NLE’s raw multimodal observations, while primitive control requires chaining interface actions for many commands.Throwing, zapping, or applying an item can require command selection followed by menu, inventory, prompt, or direction inputs.Source: §B.1 · p. 16
- NetHackScore-v0 rewards changes in game score, but score is not perfectly aligned with solving the game, so the study also reports BALROG NLE-progress.BALROG maps highest dungeon and experience levels to empirical probabilities of eventual human ascension, with level-one progress at 0% and ascension at 100%.Source: §B.1 · p. 16
- Fresh random seeds and possible backtracking or branching mean NetHack agents must reuse knowledge across changing contexts rather than memorize layouts or follow a fixed progression.SFT and RL experiments focus on this long-horizon domain.Source: §B.1 · p. 16
- MiniHack supplies controlled NetHack-style tasks with custom layouts and mechanics, complementing NetHack by making failures easier to interpret.Both environments use the same NLE language wrapper for a shared text-based observation and action interface.Source: §B.1 · p. 16
B.2 MiniHack task suite
The seven MiniHack tasks range from navigation and exploration to combat, item use, and sequenced subgoals. Across primitive, skill, and mixed settings, the study keeps the observation format and prompt template consistent while changing the available actions and descriptions.
B.2 MiniHack task suite
- The seven-task MiniHack suite spans exploration, combat, inventory use, and long-horizon subgoal sequencing.Corridor-R10 is a custom extension, while the Corridor and Quest task families otherwise come from MiniHack.Source: §B.2 · p. 17
- Corridor-R3 requires reaching a staircase through three procedurally arranged rooms, including opening or kicking doors and searching for hidden passages.It probes short-horizon navigation and brittle action repetition.Source: §B.2 · p. 17
- Corridor-R5 increases the room count to five, raising the exploration burden and the cost of wasted movement or local oscillation.It tests directional progress, map memory, and recovery from detours.Source: §B.2 · p. 17
- Custom Corridor-R10 tests steady exploration over a longer navigation horizon while separating exploration from survival and difficult combat.It extends the standard Corridor family by increasing the number of rooms.Source: §B.2 · p. 17
- Quest-Easy requires a short sequence of exploration, item use to cross lava, limited combat, and reaching the staircase.Its layout is relatively simple and mostly fixed.Source: §B.2 · pp. 17–18
- Quest-Medium adds a swarm fight in a narrow corridor and requires saving an item for later lava crossing before reaching the staircase.Success calls for corridor-aware combat and delayed item use.Source: §B.2 · p. 18
- Quest-Hard combines an efficient maze escape under hunger pressure with item collection, lava crossing, and using a wand of death against a minotaur.The maze must be solved before the rest of the quest begins.Source: §B.2 · p. 18
- WoD-Hard-Full tests finding or acquiring a wand of death and using it to kill the minotaur blocking the staircase.The task combines item pickup, item use, and combat.Source: §B.2 · p. 18
B.4 Training details
SFT and RL use a shared 5,000-step episode limit and a maximum skill horizon of 100,000 primitive steps unless noted otherwise. Training uses LoRA adapters and PPO rollouts, with collection and updates overlapped through double buffering.
B.4 Training details
- SFT and RL runs use a 5,000-step episode limit and a 100,000-primitive-step maximum skill horizon unless otherwise noted.Table 3 summarizes the main training hyperparameters.Source: §B.4 · p. 20, Table 3 · p. 21
- SFT trains a rank-128, scaling-64 rank-stabilized LoRA adapter, merges it into the base model, and uses the merged model to initialize SFT-primed RL.RL-from-base instead starts directly from the base model.Source: §B.4 · p. 20
- The default RL setup uses 32 rollouts per trainer GPU, 16 controller decisions per rollout, 350 gradient steps, λ = 0.95, clip range 0.2, and 2 × 10^-6 policy and value learning rates.The RL configuration also uses rank-stabilized LoRA with rank 128 and scaling 64.Source: §B.4 · p. 20
- Workers collect fixed-length rollouts for PPO updates, and double buffering overlaps data collection with training at the cost of policy lag.Because vLLM and PyTorch probabilities can differ, the PPO importance ratio uses the vLLM log-probabilities recorded during collection.Source: §B.4 · p. 20
- Each SFT or RL run took under one day on 8 H100 GPUs, while each zero-shot or evaluation run took under one day on one H100 GPU.Source: §B.4 · p. 20
C Full zero-shot NetHack results
Additional zero-shot tradeoff plots favor skill-based control over primitive-only control, while mixed control is typically intermediate. Skill-only control depends on coverage of essential behaviors, and primitive fallback mitigates—but does not eliminate—the effects of missing skills.
C Full zero-shot NetHack results
- Table 4 reports aggregate zero-shot results for 14 models, and Table 5 compares average performance and resource use across those models.Table 4 covers 64 episodes per setting except GPT-5, which uses 32; Table 5 reports paired model-level means.Source: C Full zero-shot NetHack results · p. 22, Table 4 · p. 22, Table 5 · p. 23
- Skill-based control defines a stronger progression-versus-cost frontier than primitive-only control, while mixed control is typically intermediate.The additional plots also show skills generally achieving higher progression at lower or comparable cost, with the advantage persisting when efficiency is measured in tokens.Source: Figure 12 · p. 23, Figure 13 · p. 24, Figure 14 · p. 24
C.1 Sensitivity to incomplete skill coverage
- Removing exploration, navigation, and search skills also removes staircase skills, severely restricting exploration and dungeon progress under skill-only control.The reported progression drop demonstrates dependence on coverage of these behaviors.Source: §C.1 · p. 26
- With primitive fallback, removing skill families is generally less damaging, but exploration/navigation/search removals still hurt most and fallback does not guarantee recovery.The next largest effects follow from removing doors/terrain/environment and combat/tactical skills.Source: §C.1 · p. 26, Figure 17 · p. 26
C.2 Qualitative analysis of mixed control
Mixed-control trajectories illustrate primitives handling problems that skills fail to solve or do not cover, followed by a return to higher-level skills. The examples also expose a routing inefficiency: the controller may switch late or use primitives aimlessly.
C.2 Qualitative analysis of mixed control
- In the two GPT-5 examples, primitives complete menus, recover from a failed skill, or supply a missing behavior; the agent then returns to skills.The ranged-combat case shows recovery from a skill failure, while the vault case uses primitives for behavior absent from the library.Source: §C.2 · p. 27, Table 6 · p. 27
- The vault example includes unrelated menu actions and backtracking, illustrating that mixed control can route inefficiently before resolving a problem.The examples are selected trajectories; their frequency and how RL changes these decisions remain open questions.Source: §C.2 · p. 27
D Full RL results
Across both models and both training views, skill-based and mixed controllers improve faster than primitive control and retain a large end-of-training advantage. RL also changes action distributions and transfers unevenly to MiniHack, where skills retain higher average success overall.
D Full RL results
- Skill-based and mixed controllers improve faster than primitive control across gradient-step and wall-clock views, retaining a large performance advantage by training’s end.The full curves cover Llama-3.1-8B-Instruct and Qwen-3.5-4B under all three action interfaces.Source: D Full RL results · p. 28, Figure 20 · p. 29, Figure 21 · p. 29
D.1 Action-distribution analysis for Llama-3.1-8B-Instruct
- In mixed control, aggregate primitive share stays nearly unchanged (33.3% to 32.0%), while median skill fraction rises from 41.1% to 77.7% and invalid calls fall sharply.RL concentrates calls on a small set of useful routines without eliminating primitive fallback.Source: §D.1 · p. 30, Figure 22 · p. 30, Figure 23 · p. 31
- After RL, skill-only calls concentrate on a few reusable routines, while primitive-only control remains dominated by local movement and search.In mixed control, common skills include explore, descend_stairs, and fight_melee, alongside retained movement and staircase primitives.Source: Figure 24 · p. 31, Figure 26 · p. 32, Figure 27 · p. 33
D.2 OOD MiniHack transfer for Llama-3.1-8B-Instruct
- Average MiniHack success rises from 56.5% to 61.8% for skills and from 12.2% to 24.8% for primitives; skills remain higher overall, while WoD-Hard worsens and Quest-Hard stays unsolved.Primitive gains are larger in absolute terms, while skill-based RL also improves Quest-Easy and Quest-Medium and brings Corridor success close to saturation.Source: §D.2 · p. 30, Figure 28 · p. 33