PaperlayerView exact revision on arXiv

Source-linked AI summary

FARM: Fundamental Agentic Reward Model For Multi-task Wireless Network Optimization

Feiran You, Changxu Ni, Haozhe Ma, Jun Li, Hongyang Du

arXiv:2610.02947v1eess.SYcs.SY

TL;DR

Policy-based transfer couples reusable knowledge to task-specific action mappings, limiting knowledge reuse across heterogeneous wireless tasks. FARM learns a task-conditioned reward prior from source trajectories and reuses it to guide target-specific policies; on the combined FAR-M, u15, and u23 group, its mean gain is +29.8%, compared with +16.1% for CRA Transfer.

  • Problem

    Policy sharing entangles reusable optimization knowledge with task-dependent action generation, while reward evaluation must also account for differing task contexts and unseen configurations.

    Source: §I · pp. 1–2

  • Method

    FARM learns a task-conditioned reward prior from heterogeneous source-task trajectories and freezes it for auxiliary guidance of target-specific policy adaptation without transferring source policies.

    Source: §I · p. 2

  • Results

    On the combined FAR-M, u15, and u23 group, FARM achieves a mean gain of +29.8%, compared with +16.1% for CRA Transfer.

    Source: D. Stage-II Transfer to Seen and… · p. 10

  • Takeaways & Limitations

    Both Mamba and Transformer support reward-prior transfer, while Mamba exhibits robustness under longer trajectory histories.

    Source: §VII · p. 13

Abstract

from arXiv · show

Future wireless networks require learning agents to adapt across heterogeneous channel conditions, traffic patterns, quality-of-service (QoS) requirements, objectives, and operational constraints. Reusing decision knowledge across such tasks is challenging because conventional multi-task and transfer reinforcement learning methods primarily share or transfer policies, coupling transferable knowledge with task-dependent action mappings. This paper proposes FARM (Fundamental Agentic Reward Model for Multi-task Wireless Network Optimization), a reward-space transfer framework that shifts cross-task knowledge reuse from policy space to trajectory-level decision evaluation. FARM introduces an Agentic Reward Model (ARM) that learns a task-conditioned reward prior from heterogeneous source-task trajectories and provides auxiliary guidance for task-specific policy optimization. In Stage I, ARM jointly models task conditions, temporal trajectory dependencies, and objective-dependent reward structures while each source task retains its own controller. In Stage II, the learned reward prior is frozen and reused to guide the adaptation of a target-specific controller for previously unseen tasks, without transferring source-task policies. Experiments on heterogeneous multi-access edge computing (MEC) tasks show that FARM achieves a mean late-stage gain of 29.8% over Single-task SAC on unseen Rate-Latency targets, compared with 16.1% for CRA Transfer, and reaches a 46.2% gain on the moderate-OOD FAR-M case. Further analysis shows that both Mamba and Transformer trajectory encoders support Reward-Space Transfer, while Mamba provides improved robustness as longer history dependencies are introduced.

I. INTRODUCTION

Heterogeneous wireless tasks vary in channels, traffic, QoS, constraints, and objectives, making cross-task knowledge reuse difficult when policies are tied to task-specific action mappings. FARM instead learns trajectory-level reward guidance from source tasks and reuses it for target-specific policy optimization.

I. INTRODUCTION

  • Future wireless control must accommodate diverse users, services, and operational requirements across changing channels, traffic, QoS preferences, and optimization objectives.The introduction identifies MEC as a representative setting for joint communication, computation, and energy-resource optimization.Source: §I · p. 1
  • Policy-based transfer entangles reusable optimization knowledge with task-dependent action generation, motivating reward representations as an alternative across heterogeneous tasks.The same trajectory can have different evaluation value under different task contexts, while task-specific action mappings may differ substantially.Source: §I · p. 1
  • Reward evaluation must reflect task context, balance shared knowledge with objective-specific preferences, and remain useful under unseen configurations and distribution shifts.The same trajectory may have different utility across contexts, while fully shared or independent reward models each miss a stated need.Source: §I · pp. 1–2
  • FARM learns a task-conditioned reward prior from heterogeneous trajectories and reuses it as auxiliary guidance, while source and target controllers remain task-specific.Stage I models task conditions, trajectory dependencies, and objective-dependent reward structures; Stage II freezes the prior and transfers no source policy.Source: §I · p. 2
  • ARM combines task-condition encoding, temporal trajectory modeling, condition-aware feature modulation, objective-specific experts, and adaptive weighting of auxiliary rewards.These components retain task-specific objective and constraint information while learning reusable evaluation knowledge.Source: §I · p. 2

A. Multi-Objective Optimization in Wireless Networks

  • Wireless multi-objective methods include scalarization and Pareto-oriented optimization, but typically rely on explicit objectives and system assumptions.The cited approaches include weighted-sum and ϵ-constraint formulations, as well as evolutionary optimization and successive convex approximation.Source: A. Multi-Objective Optimization in… · p. 2

B. DRL for Wireless Resource Management

The related work spans task-specific DRL, multi-task policy transfer, and auxiliary reward modeling, while the system model grounds FARM in heterogeneous MEC tasks. The channel model varies fading conditions and temporal dynamics across tasks.

B. DRL for Wireless Resource Management

  • DRL approaches often learn task-specific policies, tying decision rules to particular objectives, QoS requirements, and network conditions.SAC variants and constrained formulations are among the approaches discussed for wireless resource management.Source: B. DRL for Wireless Resource Management · p. 2

C. Multi-Task RL for Wireless Networks

  • Multi-task and meta-RL methods share policies, value representations, or decision representations, which remain coupled to action generation.The review also notes trajectory-history modeling, including WiSDoM's sparse Decision Transformer with Mixture-of-Experts.Source: C. Multi-Task RL for Wireless Networks · p. 3

D. Reward Shaping and Auxiliary Reward Modeling

  • Reward shaping and auxiliary reward models provide learning signals, but CRA does not explicitly model task-conditioned trajectory semantics across simultaneously varying task factors.The compared variation includes objectives, QoS requirements, channel dynamics, and constraints.Source: D. Reward Shaping and Auxiliary Reward… · p. 3

III. SYSTEM MODEL

  • The system model uses an uplink MEC domain with users served by a base station and edge server, with tasks varying in channel, traffic, QoS, constraints, and objectives.The per-base-station decision domain can be instantiated at each base station in a multi-base-station network.Source: §III · p. 3

A. Network and Channel Model

  • Tasks vary across Rayleigh, Rician, and Nakagami-m fading, with channel evolution modeled by a task-specific first-order autoregressive process.Achievable data rate is measured in Mbps.Source: A. Network and Channel Model · pp. 3–4

B. Computation Offloading and Resource Consumption Model

The model represents computation offloading as a fraction of each user's input sent to the edge, then accounts for local and offloaded execution in latency and resource consumption.

B. Computation Offloading and Resource Consumption Model

  • Each user's computation-intensive input can be processed locally or offloaded, with βk,t specifying the fraction sent to the edge server.The model defines local CPU and edge-server computing frequencies for execution.Source: B. Computation Offloading and Resource… · p. 4
  • Local execution latency and offloaded-task completion latency are modeled separately, with overall completion latency accounting for parallel local and edge execution.Offloaded completion latency includes transmission and edge-execution latency.Source: B. Computation Offloading and Resource… · p. 4
  • Energy consumption includes local computation and transmission, while effective processing throughput and energy efficiency characterize delay-energy performance.The throughput calculation uses a small positive constant for numerical stability.Source: B. Computation Offloading and Resource… · p. 4

C. Task Definition and User Objective

The formulation treats each wireless task as a distinct constrained optimization problem, with task-specific preferences determining a unified objective and QoS and constraint violations incorporated into its reward. FARM adds transferable auxiliary guidance while preserving the task's original optimization semantics.

IV. PROBLEM FORMULATION

  • The problem extends single-task resource allocation by seeking reward representations that provide reusable guidance across tasks with different channels, traffic, QoS, and preferences.A task is characterized by its channel configuration, traffic intensity, QoS requirements, active constraints, and objective preference weights.Source: C. Task Definition and User Objective · p. 4, §IV · p. 5

A. Constrained MDP Formulation

  • Each task is a constrained Markov decision process with task-dependent transitions, environment reward, and constraint-cost function.Its state includes channel, workload, queue, previous-action, and constraint information; actions specify transmission power and offloading ratio.Source: A. Constrained MDP Formulation · p. 5

B. Task-Dependent Utility and Constraint Modeling

  • Task-specific preference weights combine throughput, latency, energy consumption, and energy efficiency into a unified objective utility.The utility is represented by Uτ, while QoS requirements and active constraints are aggregated into a normalized violation cost.Source: B. Task-Dependent Utility and… · p. 5
  • The SAC environment reward preserves each task's objective and penalizes normalized QoS and constraint violations.The task-specific reference value is used for normalization, and FARM supplies auxiliary guidance without replacing that reward.Source: B. Task-Dependent Utility and… · p. 5

C. Reward-Space Transfer Objective

FARM learns a transferable reward prior alongside task-specific source controllers, then freezes the prior to guide a target-specific controller on an unseen task. This separates reusable trajectory-level evaluation knowledge from policy transfer.

Stage I

  • During Stage I, ARM and task-specific SAC controllers are jointly developed through alternating policy and reward-prior updates on source-task trajectories.ARM-generated rewards guide policy optimization, and the resulting interaction trajectories update ARM.Source: C. Reward-Space Transfer Objective · pp. 5–6
  • Each source controller retains its own task objective, while ARM-generated trajectory-conditioned reward supplies auxiliary guidance.The Bellman target remains anchored to the environment reward rather than replacing or imitating it.Source: C. Reward-Space Transfer Objective · pp. 5–6

Stage II

  • Stage II freezes the pretrained ARM backbone and adapts a target-specific SAC controller, transferring reward knowledge rather than source-task policies.The target controller is optimized under its admissible discounted constraint-violation budget.Source: C. Reward-Space Transfer Objective · p. 6

V. ALGORITHM DESIGN OF FARM

FARM's two-stage Reward-Space Transfer learns trajectory-level reward knowledge from heterogeneous source tasks, then reuses it to guide an unseen-task controller without transferring a source policy.

V. ALGORITHM DESIGN OF FARM

  • Stage I learns a transferable reward prior from heterogeneous tasks while keeping each SAC controller task-specific; Stage II freezes the prior and adapts the target controller.The target-specific lightweight adapter is optimized alongside the target SAC controller.Source: §V · p. 6, Figure 3 · p. 6

A. Design Principle: Agentic Reward Modeling

FARM adds a condition-aware auxiliary reward to the environment reward, preserving each task's objective. ARM conditions trajectory evaluation on task features and models objective preferences with specialized reward experts.

A. Design Principle: Agentic Reward Modeling

  • The training reward adds a normalized, condition-weighted ARM reward to the environment reward to guide temporal credit assignment and policy optimization.This keeps the original task objective intact.Source: A. Design Principle: Agentic Reward… · p. 6

B. Task Condition Interface

  • The task-condition vector encodes channel configuration, objective preferences, QoS requirements, active constraints, and a severity descriptor.It remains constant for a task and is supplied to ARM's reward encoder, FiLM module, objective experts, and reward-weighting module.Source: B. Task Condition Interface · pp. 6–7

C. Agentic Reward Model for Transferable Reward Reasoning

  • ARM represents each trajectory with a window of observations, actions, and previous rewards, encoded by Mamba or a causal Transformer.The encoder produces a temporal reward-related representation.Source: C. Agentic Reward Model for… · p. 7
  • Condition-aware FiLM modulation and objective-specific experts generate auxiliary rewards, while CondLam adapts their contribution to task conditions.The realized auxiliary-reward weight is clipped to [0.05, 1.0] for stable policy optimization.Source: C. Agentic Reward Model for… · p. 7

D. Stage I: Multi-Task Reward Prior Learning

Stage I learns reusable trajectory-evaluation knowledge across heterogeneous tasks while each source task keeps its own controller. Stage II freezes that reward prior and adapts a target-specific controller using task-conditioned guidance, without transferring source policies.

D. Stage I: Multi-Task Reward Prior Learning

  • Stage I defines the transferable prior using the encoder, FiLM, objective experts, and CondLam parameters.The task bank supplies the setting for learning shared trajectory-evaluation knowledge while preserving task-specific optimization.Source: D. Stage I: Multi-Task Reward Prior… · p. 7
  • ARM learns a transferable reward prior across heterogeneous tasks while each source task retains an independent SAC policy.The policies optimize their corresponding environments; the shared component is trajectory-level reward knowledge, not a common controller.Source: D. Stage I: Multi-Task Reward Prior… · p. 7

E. Stage II: Reward-Space Transfer

  • Stage II freezes the pretrained ARM backbone and optimizes a target-specific adapter and SAC policy for the new environment.The adapter is the only reward-prior pathway component optimized during target adaptation.Source: E. Stage II: Reward-Space Transfer · p. 7
  • Objective-aware routing selects a source prototype by task-condition cosine similarity, activates its objective expert, and generates target-conditioned auxiliary guidance.The frozen prior's outputs and CondLam auxiliary weight remain conditioned on the target task.Source: E. Stage II: Reward-Space Transfer · pp. 7–8

VI. EXPERIMENTAL RESULTS

The evaluation tests reward-prior learning and transfer across heterogeneous MEC tasks, using matched task-specific objectives and relative gains for comparisons across objective scales. The protocol includes held-out targets and a separate longer-history evaluation of Mamba and Transformer encoders.

VI. EXPERIMENTAL RESULTS

  • Experiments address multi-task reward learning, transfer to unseen tasks, and the effect of trajectory-modeling backbones.The research questions compare independent SAC, policy sharing, shared-reward learning, and reward-prior transfer.Source: §VI · p. 8

B. Evaluation Metrics

  • Performance uses task-specific objective Uτ, while relative gain compares each method with scratch SAC on the same target task.Because objective families differ in physical meaning and scale, absolute values are reported within families; the history-length comparison evaluates frozen priors pretrained at W = 32 with W ∈ {32, 64, 128}.Source: §VI · p. 8, B. Evaluation Metrics · p. 9

A. Stage-I and Stage-II Task Protocol

  • The Stage-I bank contains 10 heterogeneous MEC tasks, each with its own SAC controller, while ARM learns shared trajectory-level reward knowledge.Tasks vary in objectives, fading, temporal correlations, traffic, QoS, and constraints.Source: A. Stage-I and Stage-II Task Protocol · p. 8
  • Stage II tests held-out Energy, Latency, Rate–Energy, and Rate–Latency targets with a frozen prior and no source-policy transfer.The target-specific SAC controller uses environment reward and auxiliary guidance; adaptation normally runs for 40×10^3 environment steps.Source: A. Stage-I and Stage-II Task Protocol · p. 8

C. Stage-I Multi-Task Reward Prior Learning

FARM shares a task-conditioned trajectory-level reward prior while retaining task-specific controllers, and its Stage-II transfer improves on scratch SAC across seen and unseen targets. The largest reported gains occur on moderate-OOD targets; stronger shifts reduce, but do not eliminate, the reported benefit.

C. Stage-I Multi-Task Reward Prior Learning

  • FARM's Stage-I late-stage gaps are +1.207 on Nakagami tasks and +0.812 on harsh/fast-fading tasks, versus +0.041 on Energy tasks.The Energy improvement is modest but positive; the supplied figure describes mean late-stage objective gaps relative to Single-task SAC for challenging groups.Source: C. Stage-I Multi-Task Reward Prior… · p. 10, Figure 4 · p. 10
  • Stage I shares reward knowledge through a task-conditioned trajectory-level prior while keeping action generation task-specific.The reported comparisons describe this design as more consistent across challenging conditions than CRA and as reducing interference relative to Shared MTSAC.Source: C. Stage-I Multi-Task Reward Prior… · p. 10

D. Stage-II Transfer to Seen and Unseen Tasks

  • For seen tasks, FARM improves over scratch SAC by +10.0%, compared with +1.2% for CRA Transfer; Transformer-L=2 reaches +10.2%.The prose reports these Stage-II gains against scratch SAC under matched target-task evaluation.Source: D. Stage-II Transfer to Seen and… · p. 10
  • On the moderate-OOD FAR-M case, FARM gains +46.2%, while the Transformer variant gains +42.2% and CRA Transfer +35.6%.For hard open-set users u15 and u23, FARM's mean improvement is +21.6%, compared with +18.1% for Transformer and +6.4% for CRA Transfer.Source: D. Stage-II Transfer to Seen and… · p. 10, Figure 5 · p. 11
  • For the combined FAR-M, u15, and u23 unseen-task group, FARM (Mamba-L=2) gains a mean 29.8% over scratch SAC, versus 26.1% for Transformer and 16.1% for CRA Transfer.Source: D. Stage-II Transfer to Seen and… · p. 10, Figure 6 · p. 11
  • Both Mamba and Transformer support Stage-II transfer at W = 32 without transferring source-task policies.The frozen prior guides adaptation of a target-specific controller; longer-history behavior is evaluated separately.Source: D. Stage-II Transfer to Seen and… · p. 11

1) Main Transfer Gains:

FARM generally yields positive transfer on measured open-set users and larger gains than CRA Transfer, with results varying across targets. User-level comparisons and task-condition geometry provide complementary views of transfer and target similarity.

1) Main Transfer Gains:

  • The task-condition distance measures correlate with one another: Pearson r = 0.80 and Spearman ρ = 0.94.Fig. 8 pairs a PCA projection of source and target task conditions with a comparison of handcrafted and Constraint-FiLM OOD distances.Source: Figure 8 · p. 11
  • 29.8% is FARM's mean gain on the unseen Rate–Latency group, versus 26.1% for Transformer-L=2 and 16.1% for CRA Transfer.Fig. 6 aggregates by user family; Fig. 5(b) relates gain to distance from the nearest Stage-I task and target difficulty.Source: 1) Main Transfer Gains · p. 12

2) Per-User Transfer Analysis:

  • FARM has the largest gain in most complete three-way user comparisons, including +46.2% on FAR-M.FARM and Transformer remain close on seen cases; T07 is omitted from the geometry scatter because its percentage gain exceeds the plotting range.Source: 2) Per-User Transfer Analysis · p. 12

3) Stage-II Evaluation-Set Geometry:

Stage-II evaluation users occupy structured neighborhoods around source tasks, representing controlled distribution shifts. Both temporal encoders support transfer at the matched history length, while Mamba shows greater robustness with longer histories.

3) Stage-II Evaluation-Set Geometry:

  • Stage-II users represent controlled distribution shifts: held-out tasks remain closest to related source configurations, and condition-space relationships are largely preserved by FARM’s condition pathway.The mapped and original condition-space relationships agree strongly (Pearson r = 0.80, Spearman ρ = 0.94).Source: 3) Stage-II Evaluation-Set Geometry · p. 12

E. Temporal Representation for Reward Prior Transfer

VII. CONCLUSION

  • FARM reuses trajectory-level evaluation knowledge to guide task-specific policies without transferring source-task policies, improving adaptation to seen and unseen configurations over independent SAC and conventional shared-reward transfer.The conclusion also reports reduced interference associated with direct policy sharing in Stage I.Source: §VII · p. 13
Loading 2610.02947v1…