PaperlayerView exact revision on arXiv

Source-linked AI summary

Counterfactual Quotient Models: Learning What Actions Change, Not What the World Does

Junlin Chen, Ruijie Wang, Jianxin Li

arXiv:2608.22092v1cs.LG

TL;DR

Reinforcement-learning predictors often model complete futures even though action selection depends on differences between candidate consequences, leaving action-independent variation as unnecessary modeling burden. The paper introduces CQM, which quotients out shared future content and learns action effects directly from synchronized counterfactual rollouts. Across physics domains, it reports large gains on Cartpole and Reacher, a smaller advantage on Cheetah, and only a narrow gain on Walker over absolute and successor-feature models.

  • Problem

    Existing reinforcement-learning models predict complete future states, observations, or occupancies even though choosing among actions requires only differences in their consequences.

  • Method

    CQM represents action-conditioned futures modulo action-independent components and trains centered action effects directly from synchronized counterfactual rollouts.

  • Results

    Across four physics domains, CQM shows large gains on Cartpole and Reacher, a smaller consistent advantage on Cheetah, and only a narrow gain on Walker.

  • Takeaways & Limitations

    The results demonstrate that modeling action-dependent information can support choice without modeling everything that happens in the environment.

  • Takeaways & Limitations

    The implemented training procedure assumes synchronized counterfactual branches, which are natural in simulators but generally unavailable in physical systems or fixed observational datasets.

Abstract

from arXiv · show

Reinforcement-learning models commonly predict complete future states, observations, or feature occupancies, even though action selection depends only on differences between the consequences of candidate actions. As a result, these models may devote substantial statistical and representational capacity to high-dimensional phenomena that evolve independently of the agent's current choice. We introduce the Counterfactual Quotient Model, which treats action-conditioned futures as equivalent when they differ only by a component shared across actions. Its canonical centered representation removes this common component while preserving every pairwise action comparison expressible by the modeled reward family. The implemented model learns these action-dependent effects directly from synchronized counterfactual rollouts, so shared stochastic dynamics cancel before function approximation rather than after complete futures have been predicted. We establish the decision sufficiency, identifiability, common-mode invariance, approximation behavior, and regret properties of the resulting representation. Controlled experiments in physics-based environments provide initial evidence for these properties: direct effect learning suppresses action-independent variation, supports previously unseen reward queries, and improves action ranking relative to models trained to predict absolute futures.

1 INTRODUCTION

CQM focuses prediction on how candidate actions differ rather than reconstructing complete futures. Its quotient representation removes action-independent content while preserving action comparisons, and paired-rollout training estimates these effects directly.

  • 1 INTRODUCTION: Absolute future prediction can spend capacity on lighting, weather, background motion, and other dynamics unaffected by the current action.These phenomena may dominate observation variance without changing the action ranking.
  • 1 INTRODUCTION: CQM treats action-conditioned futures as equivalent when their difference is shared across actions.The quotient discards only an action-independent baseline.
  • 1 INTRODUCTION: The canonical signed representative preserves every pairwise action contrast while removing shared future content.Integrating a reward against it recovers the corresponding centered action value.
  • 1 INTRODUCTION: The paper establishes decision sufficiency, identifiability, common-mode invariance, approximation and rank behavior, and decision-regret guarantees.These properties characterize both the quotient and its practical approximations.
  • 1 INTRODUCTION: CQM uses synchronized counterfactual branches with shared noise and continuation actions, alongside a low-rank architecture enforcing the quotient constraint.The constraint is incorporated into the practical training objective and model design.
  • 1 INTRODUCTION: The experiments compare CQM with equally sized absolute predictors and reward-aware baselines in DeepMind Control Suite environments with high-dimensional common dynamics.The common protocol is designed to compare representations under matched architectural size.

2 RELATED WORK

Related approaches preserve predictive or control-relevant information through world models, successor features, value equivalence, state abstraction, and counterfactual credit assignment. CQM differs by quotienting action-indexed successor measures rather than states, specified rewards, or reward functions.

  • World models and predictive control: World models learn transition, observation, latent, or predictive-state models for simulation, planning, or policy optimization.Their learned object remains an absolute future model.
  • Successor representations and transfer: Successor representations and successor features encode expected future occupancy and support transfer across linear reward weights.CQM centers the complete successor measure, whereas successor features arise after projection through selected features.
  • Decision-aware and value-equivalent models: Value-equivalent and decision-aware models preserve information needed for selected Bellman updates, values, rewards, or policies.Their equivalence relations depend on the specified function or policy classes.
  • State abstraction and nuisance removal: State-abstraction methods remove distinctions while preserving control-relevant reward and transition structure, including nuisance variation.These abstractions generally operate on states rather than action-indexed futures.
  • Counterfactual credit assignment: Counterfactual credit-assignment methods estimate an earlier action’s contribution to later outcomes using future-conditioned critics or contribution coefficients.CQM instead targets a complete reward-independent signed future measure.
  • Invariances and reward equivalence: Potential-based shaping quotients reward functions, whereas CQM quotients action-indexed successor measures by action-independent signed-measure additions.Both separate decision content from a behaviorally irrelevant gauge, but they act on different mathematical objects.

3 COUNTERFACTUAL QUOTIENT MODEL

CQM represents action-dependent changes in discounted successor measures, removes action-independent components through centering, and projects the result into a practical feature model. Paired counterfactual rollouts provide direct effect targets, while an alternative density-ratio form extends the concept to ordinary trajectories.

  • 3.1 SIGNED SUCCESSOR MEASURES: The successor measure describes the discounted future following first action a while later actions follow a fixed continuation policy π.The construction is reward-independent but policy-dependent.
  • 3.1 SIGNED SUCCESSOR MEASURES: The counterfactual effect is a finite signed measure whose positive mass marks events made more likely and negative mass marks events suppressed by an action.It is not generally a probability measure.
  • 3.2 MEASURE-LEVEL QUOTIENT: The quotient identifies models differing only by an action-independent function or measure addition, with a reference distribution selecting a centered representative.Different full-support references yield different representatives of the same pairwise contrasts.
  • 3.3 PROJECTED QUOTIENT AND GAUGE: The measure-level quotient remains queryable by arbitrary bounded rewards, while the feature projection is exact for rewards linear in the selected outcome features.General rewards incur the approximation error characterized by proposition 4.
  • 3.3 PROJECTED QUOTIENT AND GAUGE: The projected quotient retains every action contrast but no action-independent baseline.Its canonical representative is obtained by centering action-indexed feature predictions.
  • 3.4 CQM ARCHITECTURE: Centering is applied as the final layer, so the quotient constraint holds for every parameter value; a rank-m decoder allocates capacity to action effects.The architecture centers raw action heads after state and action encoding.
  • 3.5 PAIRED-ROLLOUT TRAINING: CQM-B shares exogenous noise and continuation actions across candidate first-action branches, then trains on centered discounted feature outcomes.For uniform ρ, centered regression has the same minimizers as complete pairwise-difference regression.
  • 3.6 ORDINARY-TRAJECTORY DENSITY-RATIO FORM: The ordinary-trajectory density-ratio form uses outcome-conditioned action classification to estimate the quotient when synchronized branches are unavailable.The experiments evaluate only the branched feature-projection estimator CQM-B.

4 EXPERIMENTS

The experiments test CQM on physics-based environments with mixed action-independent dynamics, finite intervention sets, synchronized counterfactual rollouts, and held-out linear reward queries. CQM generally improves action selection and effect estimation, while results expose limits from finite action sets and insufficient representation capacity.

  • Experimental setup: The benchmark mixes a 64-dimensional autonomous process into every observed coordinate, preventing nuisance removal by dropping a known observation block.The process is combined with native task observations through a fixed random orthogonal transformation.
  • Experimental setup: Evaluation uses five fixed action prototypes in most domains and measures the best first action for unseen linear rewards under a fixed uniform continuation policy.Reported optimality is restricted to the finite intervention set, and held-out accuracy is not native-task return or long-horizon closed-loop success.
  • Experimental setup: CQM-B, the world model, and successor features use identical architectures, so their differences primarily reflect prediction targets rather than parameter budgets.Each vector predictor uses two width-128 hidden layers, a 16-dimensional action embedding, an eight-dimensional bottleneck, and a linear decoder.
  • Aggregate results: 95.53% Reacher accuracy compares with 25.98% for the world model and 34.56% for successor features, while CQM reduces Cartpole normalized regret by roughly six orders of magnitude versus the world model.Direct quotient supervision estimates action effects despite absolute predictors achieving absolute-future NMSE near 0.92–0.93 and effect NMSE near or above one.
  • Aggregate results: CQM remains strongest on Cheetah at 64.11% accuracy and 0.0542 regret, but Walker reaches only 31.34% accuracy with effect NMSE above one.Walker’s confidence intervals overlap and its result is treated as a stress test showing quotient targeting does not overcome an underspecified rank-eight decoder.
  • Aggregate results: CQM matches or improves reward-aware methods across several domains while learning a reusable action-effect representation without reward directions in its representation loss.The experiments therefore support reusability of the primal action-effect representation rather than superiority over reward-aware critics in representational ability.

5 THEORETICAL PROPERTIES

The theoretical results characterize the quotient as the coarsest exact decision representation, establish identification and invariance to action-independent baselines, and bound regret under approximation. They also explain why paired counterfactual learning can remove shared stochastic variation before function approximation, while finite-feature projections introduce a separate reward-approximation error.

  • Decision sufficiency: The quotient preserves exactly all bounded-reward pairwise action comparisons while discarding only action-independent future content.Its signed effect measure is the coarsest exact representation for the modeled reward-query family.
  • Identification: Pairwise branched-rollout learning identifies the centered effect up to an action-independent vector, and minimum-norm centering selects a unique representative.The canonical representative is determined by the chosen reference distribution and has zero weighted mean across actions.
  • Invariance: Changing the full-support reference distribution changes the representative only by a baseline constant across actions, leaving pairwise differences and action rankings unchanged.Thus, the quotient remains invariant to action-independent future additions and different gauge choices encode the same decision contrasts.
  • Common-mode cancellation: Shared nuisance variation contributes no variance to paired quotient targets when counterfactual branches use common random numbers.With independent branch noise, the shared nuisance instead contributes 2 Var(C) to the difference variance; approximate sharing leaves the branch-to-branch nuisance mismatch.
  • Approximation: If action effects lie in an r-dimensional subspace, a low-rank decoder is justified pointwise in state, but the relevant subspace may rotate across states.The pointwise rank result does not imply one globally shared output subspace.
  • Regret bounds: Feature-level quotient error δ yields regret at most 2Wδ for rewards rw = w⊤ϕ, while reward projection error adds a distinct representation error for rewards outside the feature span.Increasing feature dimension may reduce projection error ϵr but can increase statistical error δ and weaken the low-rank advantage.

6 LIMITATIONS

The paper’s empirical claims are bounded by controlled simulator settings, synchronized counterfactual data, finite reward-feature projections, discrete actions, limited seeds, and task-family-specific objectives. These constraints leave open how the approach performs with ordinary data, richer rewards, continuous actions, larger complexity, and non-decision uses.

  • Empirical scope: The experiments use state observations, simulated action-independent dynamics, and random linear rewards in four physics domains, so they do not establish broad real-world control performance.The evidence does not cover native objectives, pixel-level robustness, physical robotics, multi-agent systems, or real-world data.
  • Data and training assumptions: Training assumes synchronized counterfactual branches with shared random numbers and continuation actions, a setting generally unavailable in physical systems or fixed observational datasets.Approximate branches from a learned generative model could introduce model bias into the quotient.
  • Approximation: The finite-dimensional projection is exact only for rewards linear in selected outcome features; general rewards incur approximation error.Expanding the feature map may reduce error while increasing action-effect dimension and weakening the capacity advantage.
  • Action space: The current architecture handles finite discrete action sets, while continuous-action extensions require reference-distribution integration or learned action sampling and introduce approximation error.Paired-data cost scales linearly with the number of discrete actions, and the experiments use only three or five action prototypes.
  • Evaluation and scaling: Only five seeds are reported, and Walker shows that the fixed rank-eight bottleneck can fail as native state dimension and dynamical complexity increase.Comparisons involving reward-aware critics often have overlapping confidence intervals, while 32 training reward directions may favor held-out interpolation.
  • Scope of representation: CQM omits action-independent predictive information, limiting its role for anomaly detection, perception reuse, safety monitoring, or planning under changed action sets.The authors characterize it as a task-family-specific decision model rather than a universal replacement for generative world models.

7 CONCLUSION

The conclusion presents Counterfactual Quotient Models as representations that retain action-dependent effects while removing futures shared across choices. Across four controlled physics domains, CQM outperforms absolute and successor-feature models most strongly on Cartpole and Reacher, with smaller gains on Cheetah and Walker.

  • Conclusion: CQM represents action-conditioned futures as zero-total-mass signed successor measures that retain choice-dependent effects and remove shared content.The measure-level quotient is described as the coarsest exact object for all bounded-reward action comparisons.
  • Conclusion: A finite feature projection provides a practical low-rank model with guarantees for reward approximation, effect rank, and decision regret.The paired-rollout estimator cancels common random futures before approximation, while a density-ratio identity offers a route to ordinary trajectories.
  • Conclusion: Across four DM Control domains, CQM shows a large gap over equally sized absolute world and successor-feature models on Cartpole and Reacher, a smaller consistent advantage on Cheetah, and only a narrow gain on Walker.CQM also competes with reward-aware critics despite receiving no reward direction in its representation loss.
  • Conclusion: The results demonstrate the central point for the specified query family: modeling action-dependent information can support choice without modeling everything that happens.The conclusion identifies ordinary replay data, richer feature maps, continuous actions, and native closed-loop objectives as next steps.
Loading 2608.22092v1…