PaperlayerView exact revision on arXiv

Source-linked AI summary

OMNI: Open-endedness via Models of human Notions of Interestingness

Jenny Zhang, Joel Lehman, Kenneth Stanley, Jeff Clune

arXiv:2306.01711v3cs.AIcs.LG

TL;DR

Open-ended learning must choose among infinitely many tasks while avoiding learnable but uninteresting repetitions. OMNI uses foundation models as models of human interestingness alongside learning progress, and it improves performance over uniform sampling and learning-progress-only baselines across three domains.

  • Problem

    Open-ended learning lacks a reliable way to prioritize tasks that are both learnable and interesting among infinitely many possibilities.

    Source: §1 · p. 1

  • Method

    OMNI combines learning-progress-based task selection with a foundation-model model of interestingness derived from human-generated data.

    Source: §1 · p. 2

  • Results

    OMNI outperforms uniform task sampling and learning-progress-only baselines across Crafter, BabyAI, and AI2-THOR.

    Source: §1 · p. 2

  • Takeaways & Limitations

    OMNI supports intelligently selecting which tasks to focus on next for endless learning and may advance self-improving AI and AI-Generating Algorithms.

    Source: §1 · p. 2, §6 · p. 13

    Optimizing against an interestingness model can expose Goodhart-style pathologies, motivating refinement with additional human feedback.

    Source: §6 · p. 13

Abstract

from arXiv · show

Open-ended algorithms aim to learn new, interesting behaviors forever. That requires a vast environment search space, but there are thus infinitely many possible tasks. Even after filtering for tasks the current agent can learn (i.e., learning progress), countless learnable yet uninteresting tasks remain (e.g., minor variations of previously learned tasks). An Achilles Heel of open-endedness research is the inability to quantify (and thus prioritize) tasks that are not just learnable, but also $\textit{interesting}$ (e.g., worthwhile and novel). We propose solving this problem by $\textit{Open-endedness via Models of human Notions of Interestingness}$ (OMNI). The insight is that we can utilize foundation models (FMs) as a model of interestingness (MoI), because they $\textit{already}$ internalize human concepts of interestingness from training on vast amounts of human-generated data, where humans naturally write about what they find interesting or boring. We show that FM-based MoIs improve open-ended learning by focusing on tasks that are both learnable $\textit{and interesting}$, outperforming baselines based on uniform task sampling or learning progress alone. This approach has the potential to dramatically advance the ability to intelligently select which tasks to focus on next (i.e., auto-curricula), and could be seen as AI selecting its own next task to learn, facilitating self-improving AI and AI-Generating Algorithms. Project website at https://www.jennyzhangzt.com/omni/

1 INTRODUCTION

Open-ended learning needs to prioritize tasks that are not only learnable but also worthwhile and novel. OMNI uses foundation models trained on human-generated data to model interestingness and improve task selection.

  • Human intuition for interestingness helps identify potential breakthroughs, but conventional reinforcement-learning tools capture only limited aspects of it.Learning progress and intrinsic motivation can guide exploration without representing whether a task is worthwhile or sufficiently novel.Source: §1 · pp. 1–2
  • Open-ended agents face infinitely many possible tasks, including countless learnable but uninteresting variations that can distract learning.Learning-progress methods identify suitable difficulty but cannot reliably distinguish valuable tasks from minor repetitions.Source: §1 · p. 2
  • OMNI uses foundation models as models of interestingness to select tasks that are both learnable and interesting.The approach relies on foundation models’ exposure to extensive human-generated data and their inferred understanding of human notions such as worthwhile novelty.Source: §1 · p. 2
  • Across Crafter, BabyAI, and AI2-THOR, OMNI outperforms uniform task sampling and learning-progress-only baselines.The evaluation spans 2D crafting, grounded language learning, and 3D embodied robotics environments.Source: §1 · p. 2

2 RELATED WORK

Prior auto-curricula select tasks using difficulty, regret, or learning progress, while approaches that quantify novelty or interestingness can produce pathological outcomes. OMNI instead uses foundation models to model human notions of interestingness for open-ended task selection.

2.1 AUTO-CURRICULUM LEARNING

  • Learning-progress curricula adapt task selection to the agent’s changing capabilities, but existing auto-curricula cannot distinguish interesting tasks from infinitely many uninteresting learnable ones.Handcrafted curricula are difficult to design in vast task spaces, and regret-based methods may favor stochastic, less learnable tasks.Source: §2.1 · p. 3

2.2 ATTEMPTS TO QUANTIFY INTERESTINGNESS

  • Optimizing hand-designed novelty, diversity, exploration, or open-endedness metrics can produce outputs that satisfy the metric while missing the intended goal.Examples include superficially different trivial solutions and narrow exploration caused by biased intrinsic-motivation measures.Source: §2.2 · p. 3
  • OMNI proposes using foundation models trained on human-generated data to model otherwise difficult-to-quantify human notions of interestingness.This approach replaces manually specified criteria with an FM-based model of interestingness.Source: §2.2 · p. 4

2.3 PRE-TRAINED FOUNDATION MODELS IN OPEN-ENDEDNESS

  • Foundation models have also been used for task decomposition, commonsense priors, success estimation, and task generation, but OMNI selects tasks adaptively using agent history and direct environment interaction.Compared with related approaches, OMNI addresses stochastic settings and changing agent performance rather than relying only on deterministic success or domain-specific code APIs.Source: §2.3 · p. 4

3 METHODS

OMNI trains task-conditioned agents and combines learning-progress-based sampling with a foundation-model judgment of task interestingness. The resulting curriculum emphasizes tasks that are both learnable and interesting.

3.1 PROBLEM FORMULATION

  • OMNI formulates task-conditioned reinforcement learning as a partially observed Markov decision process and generates learnable, interesting conditioning tasks.The formulation includes states, actions, dynamics, rewards, observations, task conditioning, and discounting.Source: §3.1 · p. 4

3.2 LEARNING PROGRESS CURRICULUM

  • The learning-progress curriculum samples tasks near the agent’s capability frontier using recent changes in success probability.Exponential moving averages estimate fast and slow success probabilities; bidirectional progress also emphasizes performance degradation from forgetting.Source: §3.2 · pp. 4–5
  • Success-rate normalization reduces the learning-progress curriculum’s early bias against tasks that initially appear successful by chance.The normalization uses success rates from a random-action policy while retaining reweighting for later training.Source: §3.2 · p. 5

3.3 MODELING WHAT HUMANS FIND INTERESTING

  • An FM-based model of interestingness filters high-learning-progress tasks to avoid endless variations of uninteresting tasks.The FM receives few-shot examples, the environment description, and the agent’s existing proficiency when predicting what humans would find interesting to learn next.Source: §3.3 · p. 5
  • OMNI assigns learning-progress sampling rates first, sharply downweights tasks predicted boring, and normalizes the remaining rates into sampling probabilities.Boring-task weights are multiplied by 0.001.Source: §3.3 · p. 5

4 EXPERIMENTS IN A FINITE TASK SPACE

Experiments evaluate OMNI in Crafter and BabyAI, comparing uniform sampling, learning progress, and learning progress filtered by a foundation-model notion of interestingness. Across both domains, OMNI learns more tasks and achieves higher average success, while remaining nearly indistinguishable from an oracle interestingness model.

4.1 CRAFTER ENVIRONMENT

  • Crafter contains procedurally generated grid worlds with technology-tree prerequisites, whereas BabyAI uses partially observable rooms and 1364 instruction-sequence tasks.BabyAI tasks use randomized layouts and objects, with sequences of up to five instructions.Source: §4.1 · p. 5, §4.2 · pp. 6–7, Figure 2 · p. 6

4.3 RESULTS

  • Learning progress substantially outperforms uniform sampling because it targets suitable-difficulty tasks, but it still allocates attention to boring task variants.Uniform sampling learns only 4 Crafter tasks and 1 BabyAI task, whereas LP learns 55 and 4, respectively.Source: §4.3 · p. 8
  • OMNI learns 82 Crafter tasks and 8 BabyAI tasks, exceeding learning progress alone’s 55 and 4 tasks, respectively, while achieving higher average success rates.In Crafter, OMNI reaches 0.56 average success across interesting and boring tasks versus LP’s 0.42; in BabyAI, it reaches 7.5e-3 versus 5.9e-3.Source: §4.3 · p. 8
  • OMNI combines learning progress with a few-shot foundation-model predictor of interestingness to focus on tasks that are both learnable and interesting.This filtering avoids distracting learnable but uninteresting tasks while retaining learning-progress-based curriculum selection.Source: §4.3 · pp. 7–8

4.1 CRAFTER ENVIRONMENT

  • In Crafter, OMNI learns 14 interesting tasks (CI: 14–14) and achieves 0.78 average task success on interesting tasks (CI: 0.76–0.80), while 15 non-survival tasks are considered interesting.These 15 tasks were diluted with 90 boring numerical-repeat tasks and 1023 extremely challenging tasks as potential distractors.Source: §4.1 · p. 6, §4.3 · p. 8

4.3 RESULTS

  • FM-based interestingness is nearly as effective as the hand-designed oracle: OMNI’s performance is almost indistinguishable from LP plus the oracle across both domains.Most OMNI–oracle differences are not statistically significant, with all but one comparison having p > 0.05.Source: §4.3 · pp. 8–9
  • OMNI remains effective when Crafter’s boring tasks are compounds or synonyms rather than repetitive numerical variants.In both additional settings, OMNI significantly outperforms LP-only and uniform curricula.Source: §4.3 · p. 9

5 EXPERIMENTS IN AN INFINITE TASK SPACE

OMNI addresses infinite-task open-ended learning by using foundation models to propose learnable, interesting tasks and generate code-based reward functions that verify completion.

5 EXPERIMENTS IN AN INFINITE TASK SPACE

  • OMNI targets infinite task spaces by having foundation models propose new learnable and interesting tasks instead of relying on a predefined task set.Source: §5 · p. 9, §5.1 · p. 9

5.1 METHODS

  • The same foundation model generates environment-state code that determines whether each proposed task has been completed during training and evaluation.This supplies a universal reward-function mechanism for tasks in the open-ended setting.Source: §5 · p. 9, §5.1 · p. 9
  • The method combines task generation with completion-code generation so every proposed task includes an executable definition of success.Source: §5 · p. 9, §5.1 · p. 9
  • GPT-4 receives tasks the agent performs well and tasks it cannot yet do, then suggests subsequent tasks in a zero-shot manner.Tasks performed well are identified using a predefined success threshold, set to 0.6 in the AI2-THOR experiments.Source: §5.1 · p. 9

3. Tasks the agent currently does well and tasks the agent tried but cannot do yet

In AI2-THOR, OMNI uses foundation-model suggestions within a learning-progress curriculum to generate and learn tasks in an infinite embodied kitchen task space. It outperforms uniform sampling and learning-progress baselines.

5.2 AI2-THOR ENVIRONMENT

  • AI2-THOR provides an embodied 3D kitchen with many objects, 13 discrete actions, visual observations, and sequential environment-state task specifications.The task space is infinite because tasks can vary in the number and length of sequential state conditions.Source: §5.2 · p. 10

5.3 AI2-THOR PROMPT EXAMPLE

  • GPT-4 uses the agent’s successful and unsuccessful tasks to reason about current abilities and suggest three next tasks, such as slicing a potato or picking up a knife.Source: §5.3 · pp. 10–11
  • GPT-4 translates suggested natural-language tasks into sequences of environment states that can be used to verify completion.Source: §5.3 · p. 11

5.4 UNIFORM SAMPLING BASELINE

  • Random-task generation filters unachievable environment states and limits each task to at most 10 sequential states before GPT-4 produces natural-language descriptions.Source: §5.4 · p. 11
  • The Uniform and LP baselines add randomly generated tasks, whereas OMNI adds foundation-model-generated tasks selected for learnability and interestingness within the learning-progress curriculum.The AI2-THOR experiments ran for 1 million time steps with 10 random seeds.Source: §3 · p. 9, §5.5 · p. 12, §5.6 · p. 12

5.6 RESULTS

  • 13 (CI: 11 – 17) tasks were learned by OMNI, compared with 2 (CI: 0 – 3) for LP and 0 (CI: 0 – 2) for Uniform sampling.Tasks count when their conditional success probability reaches at least 0.6; OMNI significantly outperformed both baselines throughout training (all p < 1e-3).Source: §5.6 · p. 12

6 DISCUSSION, FUTURE WORK, AND CONCLUSION

OMNI uses foundation models to estimate human notions of interestingness alongside learning progress, steering open-ended learning toward meaningful tasks. The discussion describes extensions involving richer modalities, FM-based progress assessment, and human feedback while noting Goodhart-related risks.

6 DISCUSSION, FUTURE WORK, AND CONCLUSION

  • OMNI combines statistical learning-progress estimates with a foundation-model-based Model of Interestingness to prioritize learnable and interesting tasks.The proposed framework is presented as a general recipe for accelerated learning in environments with potentially infinite task spaces.Source: §6 · p. 13
  • Future versions could use multimodal models to assess task diversity and progress, including progress toward tasks receiving little or no reward.The paper gives object manipulation and egg peeling as an example of progress that statistical measures might misinterpret.Source: §6 · p. 13
  • FM-based interestingness judgments may require additional human feedback because optimizing against a model can expose Goodhart-related pathologies.The proposed OEHF direction would fine-tune a pretrained MoI using human evaluations to reduce suboptimal judgments or adapt it to specific domains.Source: §6 · p. 13

A LEARNING PROGRESS CURRICULUM DETAILS

  • The learning-progress curriculum normalizes task success rates against a random-action baseline before smoothing and reweighting progress.The curriculum then transforms reweighted progress into sampling probabilities using z-scoring, a sigmoid, and normalization.Source: A LEARNING PROGRESS CURRICULUM DETAILS · p. 20

A.1 LEARNING PROGRESS CURRICULUM ABLATION

The appendix reports a learning-progress normalization ablation and documents the prompts and algorithmic workflow used across Crafter, BabyAI, and AI2-THOR. OMNI evaluates tasks using progress estimates and FM-based interestingness to update sampling distributions.

A.1 LEARNING PROGRESS CURRICULUM ABLATION

  • LP with random-action normalization achieves higher average task success and learns more tasks than LP-no-norm in Crafter.Differences between LP and LP-no-norm are statistically significant across reported metrics and checkpoints, with all p < 0.05.Source: §A.1 · p. 20, Figure 7 · p. 21

B CRAFTER PROMPT

  • Crafter prompts GPT-3 to classify candidate tasks as interesting or boring based on tasks the agent already performs well.Few-shot examples treat novel skills as interesting and minor repetitions of mastered tasks as uninteresting.Source: B CRAFTER PROMPT · pp. 21–22

C BABYAI PROMPT

  • BabyAI prompts GPT-4 to identify tasks that introduce new actions beyond the agent’s existing task combinations.The prompt defines interesting tasks as sufficiently different, novel, diverse, and worth learning.Source: C BABYAI PROMPT · pp. 22–24

D AI2-THOR PROMPT - GENERATE TASKS

F OMNI ALGORITHM

  • OMNI periodically evaluates all tasks, normalizes success rates, smooths them twice with EMA, computes learning progress, and combines it with FM-derived interestingness weights.Interesting tasks receive weight 1 while boring tasks receive weight 0.001 before the task-sampling distribution is updated.Source: F OMNI ALGORITHM · p. 27

G PARTITIONING ALL TASKS INTO INTERESTING OR BORING

OMNI partitions tasks into interesting and boring sets by combining evaluated success rates with FM judgments about novelty relative to already interesting tasks. The procedure iteratively removes redundant tasks from consideration.

G PARTITIONING ALL TASKS INTO INTERESTING OR BORING

  • The partitioning algorithm repeatedly selects the highest-success uncategorized task as interesting, asks the FM to identify boring remaining tasks, and updates both sets.It continues until every task belongs to either the interesting or boring set.Source: G PARTITIONING ALL TASKS INTO… · pp. 27–28
  • The highest-success task is chosen because it is both sufficiently distinct from already interesting tasks and closer to the agent’s current skill level.The FM’s prior boring classifications establish the distinctness condition during each iteration.Source: G PARTITIONING ALL TASKS INTO… · p. 27
  • In Crafter, after selecting collect wood, the FM marks collect 2..10 wood as boring so a distinct task such as place table can be selected next.Without the FM filter, a minor variation such as collect 2 wood might be selected instead.Source: G PARTITIONING ALL TASKS INTO… · p. 27

H CRAFTER BORING TASKS

The appendix lists the boring tasks used in the main Crafter experiments and summarizes the policy architectures and optimization procedures for Crafter, BabyAI, and AI2-THOR.

H CRAFTER BORING TASKS

I CRAFTER POLICY AND OPTIMIZATION DETAILS

  • Crafter uses RGB convolutional features, task encodings, an LSTM policy, and PPO with General Advantage Estimation.The policy and value functions use separate two-layer linear heads with Tanh activations.Source: I CRAFTER POLICY AND OPTIMIZATION… · p. 28

J BABYAI POLICY AND OPTIMIZATION DETAILS

K AI2-THOR POLICY AND OPTIMIZATION DETAILS

  • AI2-THOR uses RGB observations, GRU task encoding, FiLM-conditioned processing, an LSTM, and PPO with General Advantage Estimation.Its architecture is described as similar to the BabyAI policy architecture.Source: K AI2-THOR POLICY AND OPTIMIZATION… · p. 29

L SUPPLEMENTARY METRICS AND PLOTS

Supplementary plots show OMNI concentrating sampling on interesting tasks while improving success and task acquisition over uniform sampling and learning progress alone. Oracle comparisons indicate that FM-based interestingness achieves comparable performance in Crafter and BabyAI.

L SUPPLEMENTARY METRICS AND PLOTS

  • OMNI focuses sampling on high-learning-progress tasks that are also interesting, unlike uniform sampling or learning progress alone.In Crafter and BabyAI, the sampling plots show OMNI narrowing learning-progress selection to interesting tasks.Source: Figure 8 · p. 30, Figure 14 · p. 32
  • OMNI achieves higher success rates on interesting Crafter tasks than uniform sampling or learning progress alone.The interesting-task performance and conditional-success plots report higher average success and more learned interesting tasks.Source: Figure 9 · p. 30, Figure 10 · p. 31, Figure 11 · p. 31
  • OMNI achieves close to 16% average success on BabyAI tasks with instruction lengths two or less and learns more tasks than LP or Uniform.This subset contains tasks more likely to be learned within the available compute budget.Source: L SUPPLEMENTARY METRICS AND PLOTS · pp. 29–30, Figure 12 · p. 31
  • Across all BabyAI tasks, OMNI achieves higher task success rates than uniform sampling or learning progress alone despite many tasks being difficult to learn.The sampling plot shows OMNI selecting interesting tasks within the high-learning-progress set.Source: Figure 13 · p. 32, Figure 14 · p. 32

M ORACLE MODEL OF INTERESTINGNESS

N USING COMPOUNDS AS BORING TASKS IN CRAFTER

When compound tasks are treated as boring, OMNI substantially outperforms uniform sampling and learning progress alone while approaching the oracle’s performance. Its advantage comes from filtering high-learning-progress tasks for interestingness.

N USING COMPOUNDS AS BORING TASKS IN CRAFTER

  • Uniform sampling learns 6 tasks and 3 interesting tasks, achieving 0.019 overall success and 0.097 success on interesting tasks.These results are reported for the 30-million-step compound-task setting.Source: N USING COMPOUNDS AS BORING TASKS IN… · p. 35
  • LP learns 67 tasks and 9 interesting tasks, with 0.19 average success across all tasks and 0.40 on interesting tasks, but is distracted by boring tasks.LP significantly outperforms uniform sampling while sampling tasks with high learning progress.Source: N USING COMPOUNDS AS BORING TASKS IN… · p. 35, Figure 21 · p. 37
  • OMNI learns 96 tasks and 11 interesting tasks, achieving 0.29 average success across all tasks and 0.59 on interesting tasks, significantly outperforming LP.Differences between OMNI and LP are significant across training checkpoints, and OMNI learns challenging tasks faster.Source: N USING COMPOUNDS AS BORING TASKS IN… · pp. 35–36, Figure 19 · p. 35
  • The oracle learns 101 tasks and 11 interesting tasks, reaching 0.30 overall success and 0.62 on interesting tasks, sometimes significantly exceeding OMNI.OMNI nevertheless produces similar task success and sampling patterns and approaches the oracle benchmark.Source: N USING COMPOUNDS AS BORING TASKS IN… · p. 36, Figure 20 · p. 36

O USING SYNONYMS AS BORING TASKS IN CRAFTER

With synonymous tasks treated as boring, the original OMNI model underperforms the oracle on all-task metrics because it assumes a language prior the agent lacks. Updating the prompt to reflect this limitation restores oracle-comparable performance.

O USING SYNONYMS AS BORING TASKS IN CRAFTER

  • The original OMNI learns 307 tasks and 66 interesting tasks, achieving 0.31 overall success and 0.56 on interesting tasks, but does not consistently outperform LP on all-task metrics.It significantly outperforms LP on interesting-task metrics, but not always on the combined task set.Source: O USING SYNONYMS AS BORING TASKS IN… · pp. 38–39
  • The oracle learns 389 tasks and 66 interesting tasks, reaching 0.42 overall success and 0.62 on interesting tasks, significantly outperforming the original OMNI on all-task metrics.The gap is attributed to the FM-based MoI treating synonymous tasks as boring because it assumes the agent already has a language prior.Source: O USING SYNONYMS AS BORING TASKS IN… · p. 39
  • OMNI with an updated MoI learns 393 tasks and 66 interesting tasks, achieving 0.43 overall success and 0.62 on interesting tasks, comparable to the oracle.Differences from the oracle are not statistically significant, with similar task success and sampling patterns.Source: Figure 22 · p. 38, O USING SYNONYMS AS BORING TASKS IN… · p. 39
  • GPT-3 and GPT-4 revise synonymous-task interestingness predictions after receiving agent-performance information and analysis.The models generated plausible analyses and automatically updated their predictions.Source: O USING SYNONYMS AS BORING TASKS IN… · p. 39, Figure 23 · p. 40, Figure 24 · p. 40

P EXPERIMENTS IN CRAFTER WITH SURVIVAL COMPONENTS

In the unmodified, more challenging Crafter environment with survival components retained, OMNI outperforms both learning progress and uniform sampling. These results are based on shorter runs and fewer repetitions than the main experiments.

P EXPERIMENTS IN CRAFTER WITH SURVIVAL COMPONENTS

  • OMNI achieves much higher average task success rates and learns more tasks than learning progress or uniform sampling with survival components retained.Differences between OMNI and LP, and between LP and uniform sampling, are statistically significant at all reported training checkpoints.Source: Figure 28 · p. 43, P EXPERIMENTS IN CRAFTER WITH SURVIVAL… · p. 44
  • The survival-component experiments use 30 million time steps and three random-seed repetitions, compared with 100 million steps and ten repetitions in the main experiments.Survival requirements add off-task skills such as fighting monsters and gathering food, increasing environmental difficulty and noise.Source: P EXPERIMENTS IN CRAFTER WITH SURVIVAL… · pp. 43–44

Q SENTENCE EMBEDDINGS AS INTERESTINGNESS METRIC

Sentence embeddings can match autoregressive foundation-model interestingness judgments in simple cases, but they underperform when judging compound tasks requires contextual reasoning. The paper also identifies integrating multiple interestingness aspects and addressing open-endedness safety as important future directions.

Q SENTENCE EMBEDDINGS AS INTERESTINGNESS METRIC

  • Embedding-based interestingness can distinguish unique task descriptions but misses contextual reasoning about novelty and learning progress.It may overestimate compound tasks such as collecting wood and stone relative to practicing a genuinely new skill such as collecting coal.Source: Q SENTENCE EMBEDDINGS AS… · p. 44, Figure 32 · p. 46
  • OMNI-embed performs comparably to OMNI when only repetitive tasks are considered boring.Differences across metrics are not statistically significant, and both methods produce similar task success and sampling patterns.Source: Q SENTENCE EMBEDDINGS AS… · p. 44, Figure 29 · p. 45, Figure 30 · p. 45
  • OMNI significantly outperforms OMNI-embed when repetitive and compound tasks are considered boring.Across 25%, 50%, 75%, and 100% of training, all metric differences are statistically significant with p < 0.05.Source: Q SENTENCE EMBEDDINGS AS… · p. 44, Figure 31 · p. 46

R INTEGRATING MORE ASPECTS OF INTERESTINGNESS INTO FMS

  • Few-shot prompting suggests GPT-4 can synthesize learnability and interestingness concepts when selecting tasks.The evidence is preliminary and motivates further investigation into models integrating multiple aspects of interestingness, including learning progress.Source: R INTEGRATING MORE ASPECTS OF… · p. 47, Figure 33 · p. 47

S TENSIONS BETWEEN OPEN-ENDEDNESS AND SAFETY

  • Open-ended algorithms require further research to balance their creative potential with safety and value alignment.The paper suggests human or AI feedback could update interestingness models, while noting that their effectiveness remains to be established.Source: S TENSIONS BETWEEN OPEN-ENDEDNESS AND… · p. 47
Loading 2306.01711v3…