Shaking the foundations: delusions in sequence models for interaction and control

1. Introduction

• Passive sequence prediction is optimal in expectation under log-loss and corresponds to Bayesian prediction when latent variables govern the sequence [7].Exchangeable sequences are one example of sequences governed by latent variables.Source: §1 · p. 2
• The predictions made by the model converge quickly even when the task the expert is performing is initially uncertain.This is the passive-prediction result that motivates asking whether the same model can support imitation.Source: §1 · p. 2
• The paper asks whether an autoregressive model of expert-environment interactions can serve as an adaptive imitation policy by sampling its next-action prediction from past actions and observations.The proposed appeal is to build such agents from demonstrations across tasks and domains without expensive reward maximization.Source: §1 · p. 2
• Unlike passive prediction, adaptive action selection does not work in general under the same broad conditions without assumptions such as a causally sufficient MDP and a stationary expert policy.Actions must be treated as causal interventions while observations remain conditions; otherwise the model can mistake its own actions for evidence about the world, producing self-delusions [14].Source: §1 · p. 2

• In passive prediction, an expert's action identifies the box configuration, making the subsequent prize observation certain.Source: §2 · pp. 3–4
• With hidden Θ, conditioning on the model's sampled box falsely predicts the prize, whereas intervening leaves the outcome uncertain.Source: §2 · pp. 4–5
• Θ confounds action and outcome; intervention breaks the link between Θ and the generated action, preserving evidence about the future but not the past.Source: §2 · pp. 4–5
• Intervention-based evaluation requires the causal structure: if A precedes Θ, choosing A can fix the box configuration.Source: §2 · p. 5
• When Θ is observed, conditioning and intervening on the action yield the same prediction because conditioning blocks upward information flow.Source: §2 · pp. 4, 6
• The solution also holds for stochastic expert choices and outcomes; control and reinforcement learning treat actions as indices over distributions, not random variables.Source: §2 · p. 6
• In the general case, conditioning on a chosen goal can falsely reveal the latent task parameter in goal-conditioned planning.Source: §2 · p. 6
• Observations that implicitly contain the task identifier bring the setting closer to full observability, limiting the delusion's impact.Source: §2 · p. 6

• The sequential-game formulation makes hidden moves by Nature explicit as information sets, which constrain which state-specific actions an imitator can choose.The paper uses a triplet structure for convenience and notes that other structures, such as hierarchical topic, sentence, and word variables, may suit other domains.Source: §3 · p. 7
• For partially observed interactions, the next action is sampled using past actions as interventions and past observations as conditions, yielding an adaptive policy that converges to the expert policy under broad conditions.The resulting policy is described as essentially Thompson sampling [12] [14].Source: §3 · p. 7
• The interaction process can be represented as a sequential game with hidden task parameters, but its marginal form is causally insufficient for evaluating intervention effects.The model therefore needs to retain the underlying causal structure rather than only the interactions visible to the agent.Source: §3 · p. 7, Figure 3 · p. 8
• Treating actions as conditions can cause repeated choices and convergence to the wrong bandit arm, while interventions support exploration that identifies the best arm.The comparison uses a five-arm bandit over twenty rounds, with three trajectories per treatment.Source: §3 · p. 9
• In the language-model example, a generated token is an intervention, so its likelihood is omitted when updating beliefs about the latent information that shaped expert text.The authors warn that relearning from a corpus containing the model’s own generated text can confirm its biases.Source: §3 · pp. 9–10
• The difference between conditioning and intervention is small when observations reveal the latent information, but can become large with ambiguous observations and no online data collection.Source: §3 · p. 10

4. Sequential models built from data

• A learned Markov transition function f cannot by itself compute the interventional distribution P(X_t+1 | do(x_t)).The section frames this as the challenge of learning causal-probabilistic sequential models from data.Source: §4 · p. 10

4.1. Meta-learning and “counterfactual teaching”

• The agent’s action and observation predictions and their memory transitions are learned from memory states that summarize the interaction history.The setup includes a task-parameter prior, tasks, an expert, and a learnable agent.Source: §4.1 · pp. 10–11
• Factual teaching minimizes observation log-loss, while counterfactual teaching scores the expert’s action after the agent samples its own action as an intervention.The sampled action must not propagate gradients; at deployment, this setup is intended to make the agent imitate the expert while treating past actions as interventions.Source: §4.1 · p. 11, Figure 4 · p. 11
• Treating actions and observations differently ensures that past histories mixing conditions and interventions receive the appropriate teaching signals.For interactive language models, tokens may need both standard and intervened forms because either the user or the model can produce them.Source: §4.1 · p. 12
• For binary variables, three past symbols yield 64 distinct factual/intervened histories, rather than 8 factual-only histories.Each variable has two standard and two intervened possible values, so these histories must be meta-learned separately.Source: §4.1 · p. 12

4.2. Offline Adaptation and Control

• Offline demonstrations generally cannot support counterfactual teaching when the task parameter is unknown and the agent’s sampled action differs from the recorded action.Switching to another trajectory with that action may not help, because it may come from a different latent task parameter.Source: §4.2 · p. 13
• The offline difficulty persists even with every possible interaction sequence; addressing it requires assumptions about confounders, whose integration into meta-learning remains unclear.The report describes this as an open problem for training an adaptive policy from demonstrations alone.Source: §4.2 · p. 13

5. Conclusion

• Naively conditioning action choice on the past can create delusions when confounders obscure causal relationships between actions and observations.The agent then mistakes its own actions for evidence about the task.Source: §5 · p. 14
• Treat generated actions as causal interventions so the agent learns about the task through their effects; this requires access to the causal model.The report proposes counterfactual teaching for action probabilities and factual teaching for observation probabilities.Source: §5 · p. 14
• The delusion effect may be negligible without latent confounders or when observations are sufficiently informative about the latent task and expert intentions.The report notes that some domains may not be significantly affected by self-delusions.Source: §5 · p. 14

• The analysis assumes finite alphabets and a function approximator capable of representing the optimal solution; the latent parameter is unobserved by the agent.Expert actions are used to compute losses and then discarded, so they do not affect the remaining trajectory.Source: A. Why does factual/counterfactual… · p. 14
• At each time step, the agent receives a factual observation loss and a counterfactual loss for predicting the expert’s action.The losses are defined as negative log probabilities of the observed outcome and expert action, respectively.Source: A. Why does factual/counterfactual… · p. 14
• Direct prediction links propagate gradients, whereas stochastic action-sampling links do not, separating the action and observation loss paths.Each loss depends only on its corresponding conditional probabilities for the fixed history.Source: A. Why does factual/counterfactual… · p. 15
• The action log-loss averages cross-entropies between agent predictions and causal action probabilities over histories, with past actions intervened in the target.The argument applies across histories of length t−1.Source: A. Why does factual/counterfactual… · p. 15
• Factual and counterfactual loss signals yield conditional probabilities that correctly treat past actions as interventions.Source: A. Why does factual/counterfactual… · p. 16