PaperlayerView exact revision on arXiv

Source-linked AI summary

Multitask Reinforcement Learning for Assisting Choice Model Specification

Gabriel Nova, Stephane Hess, Sander Van Cranenburgh

arXiv:2609.18441v1econ.GNcs.AIq-fin.EC

TL;DR

Discrete choice model specification is an iterative, time-consuming process, and existing methods struggle to reuse modelling experience across datasets. Delphos learns a shared specification policy across datasets using reinforcement learning and a DeepSet-Q representation. The multitask agent outperforms independently trained single-task agents during training and finds competitive specifications on unseen datasets in less than 20 minutes.

  • Problem

    Model specification is time-consuming, and existing assisted methods have limited ability to reuse modelling experience across datasets, despite transferable behavioural principles.

    Source: §1 · pp. 2–3

  • Method

    Delphos trains a shared multitask reinforcement-learning policy, representing utility specifications as sets of modelling terms to transfer decisions across datasets with different variables.

    Source: §1 · p. 3, §6 · p. 16

  • Results

    During training, Delphos consistently outperforms independently trained single-task agents in rewards, sample efficiency, successful-estimation share, and candidates explored; on unseen datasets it finds competitive specifications in less than 20 minutes.

    Source: §6 · p. 16

  • Takeaways & Limitations

    Delphos can automate part of specification search while allowing modellers to compare, refine, and validate candidates and retain control over final selection.

    Source: §1 · p. 3

    The evaluation trains on nine datasets and tests on two unseen datasets; the policy remains fixed at inference, and rewards focus mainly on fit and convergence.

    Source: §6 · pp. 16–17

Abstract

from arXiv · show

Discrete choice model specification is a time-consuming task in which modellers often specify and estimate multiple models while balancing goodness-of-fit, parsimony, and behavioural plausibility. We present Delphos, a multitask reinforcement learning framework that learns transferable specification strategies across transport choice datasets. Delphos frames model specification as a sequential decision-making problem in which it applies a sequence of modelling actions and receives feedback from an estimation environment based on model performance and convergence. To transfer modelling decisions across datasets with different sets of variables, Delphos represents utility specifications as sets of modelling terms using a DeepSet-Q architecture, allowing a shared specification policy to learn across multiple datasets. Trained on nine transport choice datasets, Delphos consistently outperforms independently trained single-task agents, indicating that sharing modelling experience improves learning efficiency and helps identify promising sequences of modelling decisions with fewer unsuccessful estimation attempts. When applied without further training to the unseen Swissmetro and Decisions datasets, the same agent identifies competitive specifications in less than 20 minutes on a standard CPU. It achieves a higher log-likelihood per observation than the VNS metaheuristic on Swissmetro and performance comparable to a published MNL specification developed by expert modellers on Decisions. These findings show that accumulating and reusing modelling experience enables Delphos to function as an intelligent assistant for discrete choice model specification. It reduces manual trial-and-error while allowing modellers to retain control over model diagnosis, refinement, and final selection.

1 Introduction

Discrete choice model specification is an iterative, time-consuming task, and existing assisted methods have limited capacity to reuse modelling experience across datasets. Delphos addresses this gap with a shared reinforcement-learning policy and a modeller-controlled workflow for proposing and reviewing specifications.

1 Introduction

  • DCM specification requires modellers to balance fit, parsimony, and behavioural plausibility through iterative decisions, making the process cognitively demanding and time-consuming.Specifications involve choices about explanatory variables, functional forms, and preference heterogeneity.Source: §1 · p. 2
  • Existing assisted methods include metaheuristic search and machine-learning systems, but their approaches do not fully support cumulative transfer of modelling experience across datasets.Metaheuristics use predefined search operators, while machine-learning methods learn from prior modelling exercises but generally rely on dataset-specific representations.Source: §1 · pp. 2–3
  • Delphos uses a shared multitask policy and DeepSet-Q representation of utility specifications as modelling-term sets to transfer decisions across datasets with different variables.The shared latent representation is conditioned on the current modelling context to select decisions.Source: §1 · p. 3
  • Modellers define a catalogue of candidate attributes and structures, then review successfully estimated models on a Pareto front while retaining control over refinement and final selection.The workflow is intended to reduce manual trial-and-error while supporting utility functions from which behavioural measures can be derived.Source: §1 · p. 3

2 Background

The background introduces sequential decision-making through Markov decision processes, then describes reinforcement learning as learning a policy through repeated interaction with an environment.

2.1 Markov decision process

  • An MDP models an agent that observes a state, chooses an action, receives feedback, and transitions to a new state; its tuple includes states, actions, transitions, rewards, and a discount factor.The discount factor determines the relative importance of future and immediate rewards.Source: §2.1 · pp. 3–4
  • The agent seeks to maximise cumulative reward by following a policy, with the value function representing expected cumulative reward from a state under that policy.The optimal value function is the maximum achievable return from a state.Source: §2.1 · p. 4

2.2 Reinforcement learning

  • Reinforcement learning learns a policy through repeated environment interaction, balancing exploration for information with exploitation of actions expected to yield higher returns.Transitions record the current state, action, reward, and next state and are used to refine the policy.Source: §2.2 · p. 4
  • Q-learning estimates action values to select feasible actions with the highest expected return, while DQNs approximate those values with a neural network when the state space is large.A DQN target network helps stabilise training targets.Source: §2.2 · p. 4

3 Methodological framework

The framework formulates choice-model specification as sequential decisions evaluated by an estimation environment, then shares modelling experience across datasets through a common representation of utility specifications. A jointly trained policy learns from multiple tasks and can be applied to unseen datasets for proposing candidates that modellers inspect and compare.

3.1 Problem formulation

  • The agent builds a utility specification through modelling actions, then receives reward based on fit, convergence, and behavioural expectations after estimation.Episodes begin with a linear additive specification and end when the agent terminates the sequence.Source: §3.1 · p. 5
  • A shared representation separates behavioural modelling decisions from dataset-specific variable labels and availability, enabling transfer across related datasets.Variables representing the same construct, such as total and in-vehicle travel time, can be encoded consistently.Source: §3.1 · p. 6
  • The reward uses baseline-relative per-observation log-likelihood improvement to provide a stable signal across heterogeneous datasets.The tanh transformation bounds reward, and non-convergent specifications receive a fixed penalty of −1.Source: §3.1 · pp. 7–8

3.2 Learning transferable modelling decisions

  • DeepSet-Q represents utility specifications as sets of modelling terms and combines their embedding with dataset-specific context to estimate feasible decisions.The shared encoder maps terms to a fixed-length latent representation; the context vector records which modelling components are available for the task.Source: §3.2 · pp. 8–9, Figure 2 · p. 8

3.3 Multitask training and evaluation

  • Joint training shares network parameters and a replay buffer across tasks, while balanced mini-batches limit domination by datasets with longer trajectories.The agent samples a dataset each episode and learns from transitions generated by the resulting estimated specification.Source: §3.3 · p. 9
  • At inference, the trained agent proposes and estimates candidates on unseen datasets without further training for modellers to inspect and compare.Candidate review can consider fit, behavioural plausibility, and parameter significance.Source: §3.3 · pp. 9–10

4 Experiments

The experiments evaluate learning and transfer across transport choice datasets, using separate metrics for training and inference and studies of cross-dataset learning and unseen-dataset specifications.

4.1 Datasets

  • The first nine datasets train Delphos, while Decisions and Swissmetro are held out for inference; the datasets differ in choice context and available variables.They share behavioural concepts such as travel time and travel cost.Source: §4.1 · p. 10, Table 1 · p. 11

4.2 Evaluation metrics

  • Training evaluation measures average reward, maximum reward, learning-curve area, convergence rate, and the proportion of novel specifications explored.The metrics respectively characterize trajectory quality, best-specification quality, learning efficiency, successful convergence, and exploration.Source: §4.2 · p. 10
  • Inference applies the trained agent to unseen datasets without further training and uses the same metrics to assess generated specifications and policy transferability.The Pareto front examines the trade-off between model fit and complexity.Source: §4.2 · p. 10

4.3 Evaluation studies

  • The multitask DeepSet-Q agent is compared with independently trained single-task DQN agents, with all agents trained for 10,000 episodes.The multitask agent trains jointly across the training datasets; each single-task agent trains on its corresponding dataset.Source: §4.3 · pp. 10–11
  • The unseen-dataset study compares Delphos’s Pareto-optimal specifications with original-study benchmarks and other assisted approaches on performance, parsimony, and behavioural plausibility.A separate evaluation asks whether modelling experience improves sample efficiency and whether generated specifications balance those three criteria.Source: §4.3 · pp. 10–11

5 Results

Across training datasets, multitask Delphos generally learns faster and reaches higher rewards than single-task agents; on unseen Swissmetro and Decisions, it identifies competitive specifications within 20 minutes. Its Swissmetro result exceeds the VNS benchmark, while its Decisions result is comparable to an expert-developed MNL specification.

5.1 Learning transferable specification strategies

  • The multitask agent generally learns faster and reaches higher rewards, with higher mean rewards, maximum rewards, and learning-curve areas than single-task agents.Learning curves and Table 2 compare the agents across training datasets.Source: §5.1 · p. 12, Figure 3 · p. 12, Table 2 · p. 13
  • Transfer gains are larger on LondonModeChoice, SpainParkingChoice, and ApolloModeChoice, but smaller on two-attribute NorwayVTT and VanCranenburghVOT.The authors suggest larger specification spaces may benefit more from guidance transferred from related datasets.Source: §5.1 · p. 12
  • Higher learning-curve areas indicate earlier high-performing specifications, while fewer novel specifications and more successful estimates suggest a more sample-efficient search.The multitask agent also produces a larger proportion of successfully estimated models across almost all datasets.Source: §5.1 · pp. 12–13

Swissmetro

  • On unseen Swissmetro, Delphos proposed 283 models in 20 minutes and found a best log-likelihood of −0.681 per observation, exceeding VNS at −0.72 and the linear baseline at −1.029.Thirteen converged specifications were non-dominated.Source: Swissmetro · p. 13, Figure 4 · p. 14

Decisions

  • On unseen Decisions, the best-AIC specification achieved −0.37 log-likelihood per observation, improving on the linear model at −0.46 and nearing the expert MNL value of −0.36.The same agent was applied without further training using the same 20-minute budget.Source: Decisions · p. 15, Figure 5 · p. 15

Swissmetro; Decisions

  • Delphos exceeds the VNS benchmark on Swissmetro and matches the published expert-developed MNL specification on Decisions.The Pareto-front figures show converged candidates and non-dominated specifications for both datasets.Source: Figure 4 · p. 14, Decisions · p. 15, Figure 5 · p. 15

6 Conclusions

Delphos applies multitask reinforcement learning to choice model specification, learning transferable strategies across datasets with different variables. It outperforms single-task agents during training and identifies competitive specifications on two unseen datasets, while leaving diagnosis and final selection to modellers.

6 Conclusions

  • A DeepSet-Q architecture represents candidate specifications as sets of modelling terms, allowing a shared decision rule to apply across datasets with different variables.The framework formulates specification as sequential modelling actions evaluated by an estimation environment.Source: §6 · p. 16
  • On unseen Swissmetro and Decisions, Delphos identifies competitive specifications in less than 20 minutes on a standard CPU, beating VNS on Swissmetro and matching an expert-developed MNL specification on Decisions.During training, it also achieves higher rewards, greater sample efficiency, and more successfully estimated specifications while exploring fewer candidates than independently trained single-task agents.Source: §6 · p. 16
  • Delphos may reduce the cognitive and computational burden of manual trial and error, while modellers retain control over diagnosis, refinement, and final selection.The framework may also help reduce misspecification risk and support utility functions used to derive marginal effects and willingness-to-pay measures.Source: §6 · pp. 16–17
  • The policy was trained on nine datasets and evaluated on two unseen datasets; broader validation is needed, and inference currently uses a fixed policy without adaptation.The reward is based mainly on goodness-of-fit and convergence and could be extended with explicit behavioural constraints.Source: §6 · pp. 16–17
  • Future work proposes testing more datasets, using context representations based on dataset statistics and behavioural features, and evaluating few-shot adaptation to new datasets.The proposed context representations may enable more informed decisions across heterogeneous datasets.Source: §6 · p. 17

Appendix

The appendix includes a table of the hyperparameter configuration used for DeepSet-Q.

Appendix

  • Table 5 lists the hyperparameter configuration used for DeepSet-Q.Source: Table 5 · p. 18
Loading 2609.18441v1…