PaperlayerView exact revision on arXiv

Source-linked AI summary

What Is Worth Representing? Representational Empowerment for Continual Model Construction

Fei Dai, Hanqi Zhou, Alison Gopnik, Charley Wu

arXiv:2609.02322v1cs.LGcs.AI

TL;DR

The paper asks which representational elements agents should build, retain, and reuse when they cannot reconstruct the full world. It proposes RepEmp and a Curator-Actor architecture, finding better transfer and more compact libraries across closed- and open-vocabulary experiments, while identifying probe coverage and task scope as limitations.

  • Problem

    Agents must choose representational elements under bounded resources because simplified task models and reusable libraries cannot include a complete reconstruction of the world.

  • Method

    The paper defines RepEmp as the value of representations that expand future modeling and planning capacity, and implements it through hierarchical Curator-Actor agents across three experiments.

  • Results

    RepEmp-guided construction aligned with human choices, improved simulated structure recovery and transfer, and produced more transferable, compact open-vocabulary libraries than baselines.

  • Takeaways & Limitations

    The results support representing what expands achievable goal space and compressing until only minimum sufficient structure remains for continual model construction.

  • Takeaways & Limitations

    Evidence is limited by fixed interventions and granularities, two open-vocabulary grid-world types, and probe quality that can miss relevant goal structure.

Abstract

from arXiv · show

The first problem of modeling the world is not just estimating the right parameters or causal structure, but deciding what should be represented at all. We frame this problem as continual model construction: an agent maintains an environment-specific model M of an inaccessible world W and curates a persistent library L of reusable representational elements across environments. We propose Representational Empowerment (RepEmp) to score candidate elements by how much they expand the agent's future capacity to model and plan, complementing the classic definition of empowerment, but redefined as control over internal representations instead of external states. We realize the framework as a hierarchical Curator-Actor architecture and test it across three experiments. In a closed-vocabulary causal-learning task, human participants construct causal models at varying abstraction granularities to maximize goal reachability rather than fidelity to the world, a signature better predicted by RepEmp than by information-gain alternatives. Matched simulations reveal that RepEmp-guided construction contributes more than exploration to sufficient structure recovery and cross-task transfer. Finally, in an open-vocabulary planning domain, an LLM-augmented Curator builds more compact symbolic libraries, which also generalize better than baselines. Ablating RepEmp eliminates these benefits. Together, these results identify RepEmp as a key principle for continual model construction: deciding what to build, retain, and reuse under bounded resources.

1 Introduction

The paper frames modeling as deciding which representations to build, retain, and reuse rather than merely estimating parameters or reconstructing the full world. It proposes continual model construction and RepEmp as a criterion for those decisions.

  • Motivation: Human problem-solving uses simplified, task-specific models that retain representations transferring successfully to future goals.The cooking example illustrates constructing only enough structure for the task and discarding what was specific to a prior dinner.
  • Motivation: Existing continual, world-model, causal-learning, and LLM-agent approaches do not provide a general retention criterion under resource limitations.They preserve knowledge, estimate structure within supplied spaces, or reuse symbolic knowledge without making representational selection the main optimization target.
  • Framework: The framework separates the inaccessible world W, environment-specific task model M, and persistent reusable library L in a W →M →L pipeline.Construction changes M for current goals, while curation determines what remains available in L across environments.
  • Framework: RepEmp measures how new representations expand future capacity to model and plan, guiding both task-model construction and library curation.The paper implements this criterion through a Curator-Actor architecture that evaluates and retains elements through planning and execution.
  • Evidence: Across the paper’s experiments, RepEmp is evaluated through human choices, matched simulations, and open-vocabulary symbolic-library construction.The contribution statement specifies fixed-granularity closed-vocabulary tests and LLM-generated open-vocabulary PDDL libraries.

2 Representational Empowerment for Model Construction

RepEmp evaluates representational changes by their capacity to expand future internal representations, with empirical measurement grounded in goal-achievement outcomes. This reframes empowerment from control over external states to control over the agent’s representations.

  • Three modeling objects: Continual model construction distinguishes inaccessible world process W, task model M, and reusable library L because bounded agents need not represent all of W.The task model is assembled from candidate elements across environments, while the library carries selected elements forward.
  • Why RepEmp: Information gain rewards uncertainty reduction about W, whereas RepEmp evaluates whether representational edits expand usable future modeling and planning capacity.Environmental empowerment measures controllable diversity of external states; RepEmp makes internal representational change the channel.
  • Formal criterion: RepEmp treats edits to representational state Z_t as inputs and future representational states as outputs over a finite edit alphabet and horizon T.The channel’s capacity is high when edits reliably reach diverse future representations.
  • Empirical grounding: For empirical evaluation, the framework projects future representations onto goal-achievement signatures, preserving distinctions visible through success or failure on goals.The projection is a measurement of RepEmp rather than a replacement definition, and can be evaluated directly when enumerable.

3 The Curator-Actor Architecture

The hierarchical Curator-Actor architecture separates slow representational decisions from fast planning and execution. It adapts task models within environments and compresses reusable libraries across environments using execution-grounded evidence.

  • Architecture: The Curator edits the current task model and library, while the Actor plans and acts under the supplied model without depending on edit history.This division reflects the paper’s two timescales: persistent representational commitments and within-task environmental interaction.
  • Propose and probe: The Curator assembles a model from library and candidate elements, then proposes positive and regression probes to test gained and preserved capabilities.Positive probes target newly enabled goals, while regression probes test goals already supported by the previous model.
  • Execute: The Actor grounds abstract plans into primitive environmental actions, executes them, and returns goal outcomes and failed predicted effects.These outcomes form the goal-achievement signature used to distinguish reliable candidates.
  • Score and promote: In enumerable settings the Curator scores candidates exactly; in open-vocabulary settings it uses finite probes, estimated validity, witness goals, achievement rates, and representational cost.Candidates are accepted when their estimated value is positive and probe successes are reliable.
  • Curation across tasks: Before each new environment, the Curator removes unreliable, redundant, or narrowly task-specific elements so the compressed library seeds subsequent model construction.The experiments span fixed candidate vocabularies and LLM-proposed open vocabularies, testing both enumerable and approximated selection.

4 Closed-Vocabulary Causal Learning

In closed-vocabulary causal learning, participants and simulated agents favored representations that supported goal achievement, with RepEmp explaining and improving representation selection across conditions.

  • Experiment 1: Human choices: Participants chose medium granularity most often, at 43% of decisions versus 38% fine and 19% coarse.RepEmp likewise preferred medium because it provided sufficient goal-relevant resolution without excessive noise.
  • Experiment 1: Human choices: Medium use increased from 35% to 46% across tasks, while coarse use fell from 25% to 17%.This movement away from coarse contradicted a cost-only account that would consistently favor the cheapest representation.
  • Experiment 1: Performance: Greater medium use was associated with more goals achieved and fewer steps, whereas fine use showed the opposite pattern.Medium use positively predicted goal achievement, with β=1.29 in a logistic mixed model over 560 binary outcomes.
  • Experiment 1: Criterion comparison: RepEmp best predicted granularity choices, EnvEmp best predicted intervention source, and their combination produced the largest joint fit improvement.The comparison used leave-one-task-out predictions while controlling for population and participant effects, choice stickiness, and intervention familiarity.
  • Experiment 2: Mechanistic evaluation: In simulations, the factored RepEmp-granularity and EnvEmp-intervention condition achieved the best train and held-out graph-recovery performance.Ablations replacing RepEmp as the granularity criterion reduced held-out performance close to the random floor, assigning roughly three times the weight to granularity selection as intervention selection.
  • Experiment 2: Mechanistic evaluation: RepEmp adapted granularity to reliability regimes, selecting medium 63% of the time when medium was most reliable and fine 97% when fine was most reliable.With a lower budget of 20 interventions in the coarse-optimal regime, RepEmp selected coarse on 59% of decisions.

5 Open-Vocabulary Symbolic Construction

The open-vocabulary regime tests whether an LLM-augmented Curator can invent, evaluate, and retain reusable symbolic elements across grid-world environments. RepEmp-guided curation improves transfer while reducing library size and LLM cost, though performance depends on proposal and probe coverage.

  • Setup: Open-vocabulary construction generates typed predicates and operator schemas online, which the Curator evaluates for validity and retention cost.The task model is a PDDL domain, while the persistent library carries promoted elements across environments.
  • Overall result: RepEmp curation requires substantially fewer LLM tokens than WorldCoder while producing compact libraries that transfer better.Removing the empowerment criterion collapses the transfer gains, isolating curation rather than proposal quality as the relevant difference.
  • Transfer: RepEmp-scored curation achieves the best backward and held-out transfer in both BabyAI and Zelda, whereas accumulated code and environment-tied policies transfer less reliably.WorldCoder is competitive on Zelda forward transfer but fails on the backward and held-out columns; PPO and Motif transfer poorly.
  • Curation: General schemas such as navigate_to_object_to_grab are retained, while position- and context-specific operators are discarded despite succeeding on individual probes.The discarded candidates have narrow witness sets and therefore do not earn persistence.
  • Efficiency: The Curator’s final library is 2.3× smaller in BabyAI and 1.6× smaller in Zelda than WorldCoder’s, remaining approximately constant across the curriculum.Carrying over typed predicates and operator schemas lets new environments rebind proven symbols instead of accumulating separate models.
  • Limitation: One BabyAI held-out environment remained below 15% success for every method because training probes omitted the required conjunction of object-object relations.RepEmp behaved correctly on the available probes, but those probes did not cover the relevant goal structure.

6 Discussion

The discussion positions continual model construction as selecting both task-specific models and persistent libraries under bounded resources. It develops RepEmp as an internal, goal-grounded criterion and clarifies its approximation and limitations.

  • Conclusion: The paper concludes that agents should represent what expands achievable goal space, then compress until only minimum sufficient structure remains.The broader generality of RepEmp remains an open question.
  • Positioning: The framework connects state abstraction, symbolic vocabulary construction, and empowerment by treating both M and L as objects of selection.These related approaches otherwise vary in which modeling component changes and which criterion governs that change.
  • Framework: The framework separates the inaccessible world W, environment-specific model M, and persistent library L into a W →M →L construction-and-curation pipeline.Construction edits M within an environment, while curation edits L across environments.
  • Grounding: Goal projection removes syntactic representation differences that do not change goal-achievement signatures, yielding grounded empowerment no greater than exact empowerment.The projection lower bound follows from the Markov chain U →ZT →Γe(ZT) and data processing.
  • Proxy: The open-vocabulary proxy scores the magnitude and reliability of candidate-induced achievement changes rather than directly estimating channel capacity.Restricting evaluation to currently blocked goals removes goals whose achievement bits cannot change.
  • Limitation: Singleton marginal-gain approximation can overestimate redundant bundles and underestimate complementary ones, so proposed bundles are jointly verified before acceptance.Higher-order representational synergy remains a limitation of the approximation.

B.5 Minimum-cost sufficiency for the coverage proxy

The coverage proxy formally favors models that achieve task-relevant goals while minimizing representational cost. Its stage-dependent extension predicts refinement during discovery and compression after sufficient structure is learned.

  • A model is minimum-cost goal-sufficient when it achieves the required goal signature and no cheaper goal-sufficient model exists.
  • Under finite candidate sets, additive costs, exact signatures, and sufficiently small λ, maximizing the proxy excludes coverage-suboptimal models and minimizes cost among goal-sufficient models.
  • Goal-irrelevant elements can be pruned because removing them preserves coverage while strictly reducing cost.
  • Fine representations can have greater proxy value during discovery, whereas coarser representations can become preferable after the relevant relation is learned.
  • Information gain rewards uncertainty reduction about the world even when the resolved distinction does not affect achievable goals, unlike grounded RepEmp.
  • The causal experiments vary representation granularity while preserving fundamental-value propagation in the underlying structural equation model.

C.2 Experiment 1 instantiation: human experiment

Experiment 1 uses a fixed causal graph and a four-task curriculum to study human choices of observation granularity and interventions. Participants explore task-specific causal structures while transferring across overlapping task windows.

  • The human experiment uses a fixed eight-variable, nine-edge causal graph and a four-task curriculum with visible edges and goals.
  • Participants never see fundamental values and specify interventions using bins determined by the selected granularity.
  • The matched simulation replaces the hand-designed graph with ten-variable graphs randomly sampled per seed using edge probability pe=.25.
  • Each simulation seed contains 5 training tasks followed by 5 held-out test tasks with overlapping windows of 4–7 variables, enabling cross-task transfer analyses.
  • The human study recruited n=70 adults, who completed four tasks with up to 20 exploration steps, graph reporting, and goal-directed tests.
  • Analyses use 3,299 valid exploration decisions and retain only steps where all candidate rules were admissible and finite.

D.2 Behavioral analyses

Human granularity choices are persistent, compress within tasks, and converge across tasks, with overall medium use associated with goal achievement. Response-time analyses do not support effort as the source of the pattern, while criterion comparisons remain sensitive to specification.

  • Participants repeat the preceding granularity on 67.2% of consecutive within-task steps versus 46.4% under participant-level shuffling (p<.001).
  • Compression within and across tasks: Within tasks, revisiting a probed source favors coarser over finer granularity (557 vs. 430, p<.001), while coarse use rises from 17.7% to 22.3%.
  • Compression within and across tasks: Across tasks, medium use rises from 35.0% to 45.9% and coarse use falls from 24.7% to 16.6% (both p=.006), while fine use does not change reliably.
  • Granularity use and goal performance: Overall medium use is positively associated with goal achievement (β=1.29, SE=.62, z=2.07, p=.039), whereas task-specific deviations are unreliable.
  • Effort and response time: Median response times are nearly identical across coarse, medium, and fine choices (7.6/7.6/7.3 s), and effort adds no held-out fit.
  • Criterion comparison: Across 12 belief-update specifications, fixed-goal RepEmp is strongest in a plurality of granularity cells, but its median increment is near zero and no positive interval persists.

E.2 Granularity selection across noise regimes

Across noise regimes, RepEmp adapts its selected granularity to reliability, unlike EnvEmp and InfoGain, but can diverge from graph-recovery optima in a coarse regime.

  • RepEmp shifts from 97% fine in the flat regime to 63% medium in R-medium, then returns to 97% fine in R-fine.
  • In R-coarse, RepEmp selects fine on 91% of decisions and obtains .139 held-out F1 despite coarse having the highest fixed-policy graph recovery.
  • Exploratory sweeps raise the full-budget coarse share only to 23%; with budget 20, it reaches 59% but F1 falls to .176 versus .326 at full budget.

F.3 Training and evaluation

The open-vocabulary evaluation trains Curator-Actor agents on BabyAI and Zelda curricula, then measures library transfer to later and held-out environments under controlled budgets.

  • Curator and WorldCoder receive 200 LLM calls per BabyAI environment and 400 per Zelda environment, while Motif uses approximately 5,000 calls.
  • Each environment’s loop stops after three consecutive cycles without new promotions or when its per-environment LLM-call budget is exhausted.
  • Forward transfer freezes the previous library and evaluates it on the next environment without training trajectories or new candidate proposals.
  • Held-out transfer evaluates the frozen final library zero-shot on disjoint environments whose goal templates and configurations were not seen during training.

F.5 Baseline

The baselines include model-free PPO, preference-based Motif, and WorldCoder, with comparisons designed to separate symbolic transfer from exploration and to expose task-context failures.

  • PPO uses extrinsic reward plus Random Network Distillation intrinsic novelty, with no intrinsic, count-based, and RND-based conditions compared.
  • Motif collects episodes, labels 5,000 trajectory pairs with an LLM lacking the mission string, and trains a reward model from those preferences.
  • In one identical-observation pair, the LLM ranked doing nothing above picking up an object because it hallucinated a goto task.
  • Task-specific language appeared in 78.8% of LLM responses despite the labeler lacking access to the mission.
  • WorldCoder carries synthesized transition and reward code across environments and edits inconsistent transitions, but has no library object to prune or retire prior code paths.

G.1 Cross-stage transfer

Across BabyAI and Zelda, the Curator’s curated library supports forward transfer and preserves or improves earlier-environment performance, whereas WorldCoder and Hoarder show weaker transfer or forgetting.

  • On BabyAI env3, Curator success reaches ∼.30 before training and ∼.45 after its own training stage.
  • Earlier BabyAI environments retain performance after later training, with env2 showing retroactive improvement from later-promoted predicates tightening earlier operators.
  • WorldCoder’s env1 success peaks at ∼.69 after training and drops to ∼.10 by +env3 as later code edits break earlier transitions.
  • The Hoarder ablation shows neither forward nor backward gains on BabyAI or Zelda.
  • On Zelda env1, Curator success is ∼.20 before training, ∼.49 at +env1, and ∼.43 after +env2, while WorldCoder peaks at ∼.36 and decays.

G.2 Held-out transfer

The Curator transfers a compact, empowerment-selected library to held-out environments without training there, outperforming baselines on most configurations but failing when training probes omit the required relational structure.

  • BabyAI held-out transfer: ∼.45-.75 success on three held-out BabyAI configurations, 15-30 points ahead of WorldCoder and an order of magnitude ahead of policy baselines.The frozen-library protocol uses no held-out training trajectories, candidate proposals, or re-synthesis; predicates and operators re-bind to new objects and layouts.
  • Failure boundary: Every method falls below .15 on MoveTwoAcrossS5N2, which requires simultaneous conjunctions of binary object-object spatial relations absent from training goals.The limitation is attributed to insufficient proposer goal diversity: the needed predicate is not promoted because training witnesses do not test that structure.
  • Zelda held-out transfer: ∼.48 success on both held-out Zelda environments, 8-12 points ahead of WorldCoder, while PPO and Motif remain near floor and Hoarder reaches ∼.06.The smaller margin reflects Zelda’s narrower training-and-held-out gap and better transfer of WorldCoder’s mission-keyed reward functions.
  • Selection versus accumulation: ∼.03-.06 held-out success for Hoarder across both environment families indicates that retaining every proposed element does not substitute for empowerment-based selection.The Curator’s gains are associated with keeping the library compact and broadly applicable rather than merely generating useful candidates.
  • Representation footprint: 2-6 KB per environment for the Curator versus WorldCoder’s 11.2-24.7 KB total carried representation shows a smaller, roughly flat library footprint.The Curator retains 5-6 predicates and 0-3 operator schemas per environment, whereas WorldCoder’s reward-side count grows from 6.2 KB to 20.4 KB.
Loading 2609.02322v1…