PaperlayerView exact revision on arXiv

Source-linked AI summary

When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data

Peter Hase, Mohit Bansal

arXiv:2102.02201v2cs.CLcs.AIcs.LG

TL;DR

The paper asks when explanations can improve modeling, as models may already know or infer task information. It formalizes explanation use and studies retrieval-based modeling with a synthetic task and three existing datasets. Retrieval yields no performance improvements on those three datasets, and the authors identify conditions that may explain when retrieval can be useful.

  • Problem

    The paper asks when explanations improve modeling, since models may already store relevant knowledge or infer tasks from their inputs.

    Source: §1 · p. 1

  • Method

    The paper develops a formal framework and studies retrieval of past explanations as model inputs using a synthetic task and existing datasets.

    Source: §1 · p. 2

  • Results

    Explanation retrieval does not improve task performance on e-SNLI, TACRED, or SemEval, suggesting these settings fail at least one success criterion.

    Source: §8 · p. 17

  • Takeaways & Limitations

    Retrieval is promising when explanations help a model infer relevant latent information better than the input alone and the retrieval conditions are met.

    Source: §8 · pp. 16–17

    Collecting explanations at prediction time may be infeasible in fully automated settings and can leak labels in tasks such as NLI and QA.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

Many methods now exist for conditioning model outputs on task instructions, retrieved documents, and user-provided explanations and feedback. Rather than relying solely on examples of task inputs and outputs, these approaches use valuable additional data for improving model correctness and aligning learned models with human priors. Meanwhile, a growing body of evidence suggests that some language models can (1) store a large amount of knowledge in their parameters, and (2) perform inference over tasks in textual inputs at test time. These results raise the possibility that, for some tasks, humans cannot explain to a model any more about the task than it already knows or could infer on its own. In this paper, we study the circumstances under which explanations of individual data points can (or cannot) improve modeling performance. In order to carefully control important properties of the data and explanations, we introduce a synthetic dataset for experiments, and we also make use of three existing datasets with explanations: e-SNLI, TACRED, and SemEval. We first give a formal framework for the available modeling approaches, in which explanation data can be used as model inputs, as targets, or as a prior. After arguing that the most promising role for explanation data is as model inputs, we propose to use a retrieval-based method and show that it solves our synthetic task with accuracies upwards of 95%, while baselines without explanation data achieve below 65% accuracy. We then identify properties of datasets for which retrieval-based modeling fails. With the three existing datasets, we find no improvements from explanation retrieval. Drawing on findings from our synthetic task, we suggest that at least one of six preconditions for successful modeling fails to hold with these datasets. Our code is publicly available at https://github.com/peterbhase/ExplanationRoles

1. Introduction

The paper asks when explanations can improve modeling, given that models may already know or infer task information. It develops a framework for explanation use and studies conditions for their usefulness experimentally.

1. Introduction

  • The paper contrasts sparse label supervision with verbal explanations and feedback, which can accelerate human learning and align behavior with a teacher’s priors.Source: §1 · p. 1
  • Prior approaches use explanations as constraints, augmentation, supervision, structured variables, or model inputs.Examples include regularization (Ba et al., 2015; Zhang et al., 2016; Andreas et al., 2018; Liang et al., 2020), data augmentation (Hancock et al., 2018; Wang et al., 2019a; Awasthi et al., 2020), supervision (Pruthi et al., 2020), intermediate variables (Camburu et al., 2018; Wiegreffe et al., 2020), and inputs (Rupprecht et al., 2018; Co-Reyes et al., 2019; Zhou et al., 2020).Source: §1 · p. 1
  • These approaches seek either information beyond model inputs and parametric knowledge or a further specification of which models are desirable.The paper frames explanations as serving many roles, including targets, inputs, regularizers, structured variables, and rules for automatic labeling.Source: §1 · pp. 1–2
  • Explanations may add little when models already store relevant knowledge or infer tasks from inputs, leaving unclear when explanations help or cannot help in principle.Prior studies also report elusive improvements despite collecting thousands of explanations.Source: §1 · p. 2
  • The paper presents a framework for explanation use and experimentally studies their usefulness with a controlled synthetic task and three human-curated datasets.The proposed approach retrieves past explanations and supplies them as model inputs at prediction time.Source: §1 · p. 2
  • RQ1 asks when models can infer each sequence’s task and when they must receive task information.Source: §1 · p. 2
  • RQ2 asks whether retrieving past explanations enables a model to solve the synthetic task.Source: §1 · p. 2

3. RQ3. Useful information might be distributed over sev-

The section poses research questions about whether explanations can be aggregated, represented, retrieved, and used to improve predictions across settings.

RQ3

  • Figure 1 contrasts fine-tuning input s with additional task instructions τ or explanations e, which may help some models.A car-travel instruction and an Addis Ababa–Dessie travel-time query illustrate these inputs.Source: Figure 1 · p. 2, §3 · p. 2
  • RQ3 asks whether models can aggregate information across explanations to improve prediction.Source: §3 · p. 2

RQ4

  • RQ4 compares textual explanation inputs with pooled extracted representations as ways to represent explanations for prediction.Source: §3 · p. 2

RQ5

  • RQ5 asks what makes an explanation relevant across data points and enables retrieval for a new query.Source: §3 · p. 2

RQ6

  • RQ6 asks whether explanations can encourage reliance on strong causal features rather than weak spurious features.Source: §3 · p. 2

RQ7

  • RQ7 asks how classifier–retriever co-dependence affects joint training, since the retriever’s training signal depends on how the classifier uses retrieved explanations.Source: §3 · p. 2

RQ8

  • RQ8 asks whether explanation retrieval improves performance on existing datasets after identifying conditions for success on the synthetic task.Source: §3 · p. 2

2. Formalizing the Role of Explanations in Modeling Data

The framework groups explanation use as targets, inputs, or priors, then examines when explanations add useful information to modeling. It motivates retrieval-based inputs while identifying assumptions and practical constraints.

2.1. What Is an Explanation?

  • “Explanation” refers here to data elicited by asking why an input has its label, without imposing a formal boundary on the data’s form.The paper uses illustrative examples and describes the human explanations used in its experiments later.Source: §2.1 · p. 3

2.2. Formal Framework and Relevant Work

  • The framework treats labels as targets, observed data as inputs, and parameter distributions as priors, with latent variables available to represent additional task information.It characterizes supervised modeling through MAP inference over model parameters.Source: §2.2 · p. 3
  • Prior methods use explanations as targets, inputs, or priors, but cited target-based work found no task-performance gains; input-at-prediction-time methods raise collection and label-leakage concerns.A global explanation set can handle about a hundred explanations but does not scale to many thousands; a prior retrieval approach also allowed a data point’s own explanation during training.Source: §2.2 · pp. 3–4

2.3. How Explanations Achieve One Goal As Targets, Inputs, or Priors

  • Simple regression examples show that additional targets or priors can induce the same MAP estimate, while task information can also enter as an input or a conditional prior.The paper presents these as illustrative cases and says a more formal analysis is needed for neural language models and structured outputs.Source: §2.3 · p. 5

2.4. How Explanations Complement the Input in NLP

  • Pretrained models’ ability to infer tasks from text leaves open when task information helps beyond the input alone, a question the experiments address.The paper notes that question answering can make it difficult to identify what counts as a sufficient input without extra information.Source: §2.4 · pp. 5–6

2.5. Assumed Structure of Data

  • The assumed data consist of supervised examples paired with token-sequence explanations, and the approach permits some or most training explanations to be missing.Explanations without corresponding input-label pairs are also allowed, though not in the described datasets.Source: §2.5 · p. 6

2.6. Our Model

  • The proposed retrieval approach uses past explanations as latent inputs, avoiding collection at test time and the risk of conditioning on a point’s own label-revealing explanation.The paper motivates retrieval as more scalable than a global explanation set and focuses on explanations as inputs rather than targets or structured variables.Source: §2.6 · p. 6
  • For a query, TEXTCAT retrieves explanations, allocates them across k classifier inputs, and marginalizes the resulting predictions over the latent explanation sets.The method retrieves the top Ck explanations, arranged in sets of C; the value of k is kept relatively small for computational efficiency.Source: Figure 3 · p. 6, §2.6 · p. 7

3. Synthetic Task

The synthetic task gives controlled tests of when explanations help by making task-relevant information explicit while preserving ways for models to infer it from inputs. Its design varies explanation relevance and completeness to probe conditions that may also occur in human-curated data.

3. Synthetic Task

  • The synthetic dataset controls data and explanation properties, enabling experimental answers to six of the paper’s eight research questions that available datasets could not provide.Its task classifies integer sequences according to counts of selected integers.Source: §3 · p. 7

3.1. Generated Data

  • Each sequence’s label depends on whether one selected integer occurs more often than another, with its index and indicator identifying which pair to count.The explanation provides candidate integer pairs; the indicator selects the causal pair rather than a distractor.Source: §3.1 · pp. 7–8

3.2. Important Data Properties

  • The task models a human-curated-data setting where instructions or explanations may reveal information missing from the input, although a model might already infer or know it.The index is analogized to a question topic or statement referent, and the explanation identifies the feature determining the label.Source: §3.2 · p. 8
  • Retrieval can be unnecessary when a model learns to infer the hidden task information directly from the input, as it can in principle learn the index-to-counting-rule map.The authors also report that models infer the task even when the index is removed, apparently by counting recurring sequence integers.Source: §3.2 · pp. 8–9
  • Explanation relevance depends on whether it informs another sequence’s task; the design varies task frequency and uses either matching indices or smoothly related indices to control relevance.With unrelated sampled indices, a nonmatching explanation provides no information about the sequence’s task.Source: §3.2 · p. 9
  • Although explanations can encourage selection of a causal feature over a correlated distractor, models do so only in a narrow set of situations..Source: §3.2 · p. 9

3.3. Kinds of Explanations

  • The authors distinguish full-information explanations from evidential explanations, whose independent noise averages toward the true task parameters.Existing datasets may provide only partial task information; evidential explanations add noise to the underlying integer values.Source: §3.3 · p. 9
  • Recomposable explanations split task parameters across examples so that the full set of relevant explanations can reconstruct the original values.The experiments include two-piece decompositions and a similar four-piece variant, with results reported only for the two-piece setting.Source: §3.3 · p. 9

4. Computational Methods

The computational methods condition classifiers on retrieved explanations, either by combining explanations with the input as text or by averaging their representations. Retrieval uses Sentence-BERT representations, while the setup excludes each training point’s own explanation to avoid train–test mismatch and label leakage.

4. Computational Methods

  • The methods compute predictions from an input and retrieved explanation set, using either a classifier over conditioned inputs or a model that retrieves explanations.Source: §4 · p. 9

4.1. Conditioning Mechanisms

  • TEXTCAT concatenates the input with explanations as text, allowing the model to process them together; marginalizing over sets can condition predictions on more explanations than fit in one context.The concatenated sequence is passed to RoBERTa, pooled, and classified with a one-layer MLP.Source: §4.1 · pp. 9–10
  • H-MEAN separately represents the input and each explanation, then averages those representations before classification, avoiding TEXTCAT’s sequence-length constraint.Separate processing may impede integration across explanations, and marginalization requires Ck forward passes per prediction.Source: §4.1 · p. 10

4.2. Retrieval

  • The retrieval model uses Sentence-BERT sentence representations and cosine similarity to find top-ranked explanations, with FAISS providing approximate sub-linear search.Source: §4.2 · p. 10
  • Training excludes each data point’s own explanation, and test-time retrieval is limited to past training explanations to avoid distribution mismatch and potential label leakage.Source: §4.2 · p. 10

5. Experimental Setup

The experiments combine the controlled synthetic task with three English datasets carrying explanations. The setup specifies model training, embedding refresh, and different uncertainty and hypothesis-testing procedures for synthetic and existing-dataset experiments.

5. Experimental Setup

  • Experiments use the synthetic task plus e-SNLI, TACRED, and SemEval, with human explanations available for the latter three datasets.TACRED and SemEval are relation-extraction tasks with a subset of points annotated; Table 2 gives example data points.Source: §5 · p. 10, Table 2 · p. 11
  • Models are trained end-to-end with AdamW and cross-entropy, and training explanation embeddings are rebuilt every 20% of an epoch unless noted otherwise.The authors report that embeddings need updating at least once per epoch for predictions and gradients to reflect current parameters.Source: §5 · p. 10
  • Larger context size C and retrieval parameter k usually yield higher accuracy with more stable training, according to the reported hyperparameter analysis.Source: §5 · p. 11
  • Synthetic results use confidence intervals for seed variance from five to ten seeds, while existing-dataset conditions generally use three seeds and select by development accuracy.Some large-data conditions use one seed; selected models are tested for binomial-mean differences in test accuracy.Source: §5 · p. 11

6. Experiment Design and Results

The experiments test when models can infer task information and when retrieved explanations help, first on a controlled synthetic task and then on e-SNLI, TACRED, and SemEval. Retrieval succeeds under some synthetic conditions, but yields no statistically significant improvements on the three existing datasets.

6.1. RQ1: When can models solve our synthetic problem by inferring each sequence’s task, and when must they be given the task information?

  • Task inference reaches as low as 62.2% accuracy at 500 tasks, while 50,000 training points raise accuracy to 87.11%; providing the index slightly eases inference but does not enable memorizing its task map.RoBERTa-large helps at fewer than 250 tasks, but RoBERTa-base performs better beyond that point.Source: §6.1 · pp. 11–12

6.2. RQ2: Can retrieval of past explanations enable a model to solve our task?

  • With 500 tasks, learned retrieval reaches above 98% accuracy and improves on the no-retrieval baseline by around 37 percentage points; it also improves on fixed retrieval by about 7 points.Each conditioning mechanism sees roughly the same improvement, and learned retrieval performs as well as optimal retrieval.Source: Figure 6 · p. 12, §6.2 · p. 12

6.3. RQ3: Can models aggregate information across explanations for better prediction?

  • Evidential explanations yield above 90% accuracy but around 92% versus full-info’s 98%; optimal retrieval with recomposable explanations lets TEXTCAT exceed 98%, while learned retrieval trails optimal by 6–8 points.With evidential explanations, learned retrieval approaches optimal and the conditioning mechanisms perform similarly.Source: Figure 7 · p. 12, §6.3 · p. 12

6.5. RQ5: What makes an explanation relevant across data points? What enables a retrieval model to find relevant explanations for a new data point?

  • Learned retrieval beats the baseline by 7.6 points in the smooth condition versus 4.3 in the non-smooth condition; similar indices map to similar task integers only in the smooth condition.Without the index feature linking data points and explanations, learned retrieval accuracy falls from 98.6% to 54.7%.Source: Figure 8 · p. 13, §6.5 · p. 13

6.4. RQ4: What is the best way to compute explanation representations for prediction?

  • TEXTCAT exceeds H-MEAN by 9.3 points at n = 1000 and 9.2 points at n = 1500, the methods perform equally well after n = 5000, and H-MEAN takes 61% longer to train at C=k=4.TEXTCAT also outperforms H-MEAN by 4.58 points with learned retrieval and recomposable explanations.Source: §6.4 · p. 13

6.6. RQ6: Can explanations help models learn to use strong (causal, generalizable) features rather than weak ones?

  • Above 99% test accuracy occurs only with full-info explanations and zero correlation between strong and weak features; causal-feature explanations alone do not help the model select the causal feature.The authors suggest that interpreting explanations can be difficult when they blend into the input sequence.Source: Figure 9 · p. 14, §6.6 · p. 14

6.7. RQ7: How does the co-dependence between classifier and retrieval model influence the viability of joint training?

  • Joint training from epoch 0 fails, and retrieval is learnable only with Sentence-RoBERTa; noisy initial retrieval can quickly make retrieval unlearnable.The classifier must be warmed up, or the retrieval model fixed, for some epochs before training the other model.Source: Figure 10 · p. 14, §6.7 · p. 15

6.8. RQ8: Does retrieval of explanations improve model performance on existing datasets?

  • Explanation retrieval produces no statistically significant gains on e-SNLI, TACRED, or SemEval at any tested training size; doubling TACRED training data from 5,000 to 10,000 raises baseline accuracy by 1.28 (p=.0017).Increasing k at test time also does not improve accuracy.Source: Table 3 · p. 15, §6.8 · p. 15

7. When Can Explanations Help?

The section frames explanation usefulness as conditional: explanations must add relevant information and retrieval must be learnable. The experiments also show that explanation interpretation and benchmark setting matter to whether retrieval helps.

Conditions for usefulness

  • Explanations can help when the model infers relevant latent information better from the input plus explanation than from the input alone.Relevant latent information can include facts or task specifications that assist prediction.Source: §7 · p. 16
  • Retrieval needs an easily computable feature linking explanations to query points and explanations relevant across data points for predicting future labels.Source: §7 · p. 16
  • The classifier must have a known or identifiable interpretation of explanations that yields a useful representation.Source: §7 · p. 16
  • Both classifier and retrieval model need sufficient quality before the other is trained.Source: §7 · p. 16

Interpretation and feature selection

  • Explanations break ties between strong and weak features only when the model can interpret them; synthetic causal-feature explanations did not reliably induce causal feature selection.The authors report learning to interpret explanations only when the features are not perfectly correlated.Source: §7 · p. 16

Scope of the findings

  • On e-SNLI, TACRED, and SemEval, retrieval brings no statistically significant accuracy improvement, while more unexplained data can improve performance.The authors suggest that at least one retrieval precondition may fail in these settings.Source: §7 · p. 16
  • The authors describe a tension: stored model knowledge may reduce the need for retrieved information, while better interpretation can make task descriptions or explanations useful for some tasks.Source: §7 · p. 16

8. Conclusion

The conclusion recommends retrieval-based modeling as a suitable role for explanations and summarizes conditions for learnability, while reporting no improvement on three existing datasets. The appendices provide preprocessing, hyperparameter-analysis, and runtime details.

8. Conclusion

  • The paper presents a formal framework and favors retrieving past explanations as model inputs; retrieval learnability depends on linkable, relevant explanations, interpretable representations, and sufficient initial model quality.The authors report no task-performance improvement on e-SNLI, TACRED, or SemEval and suggest those settings fail one or more criteria.Source: §8 · pp. 16–17

A.1. Data Preprocessing

  • The appendix reports sequence-length preprocessing for e-SNLI, TACRED, and SemEval, notes removal of a TACRED explanation linked to a dev-set point, and presents accuracy analyses by conditioning, k, and explanation type.Source: §A.1 · p. 20, Figure 11 · p. 20, Figure 12 · p. 20

A.2. Runtimes.

  • Reported runtimes include 8.6 hours for 20 epochs of learned retrieval on synthetic data and 32.9 hours for 10 epochs on TACRED; full-data e-SNLI retrieval took 7 days for 5 epochs.Source: §A.2 · p. 20

A.3. Training Hyperparameters and Analysis

The appendix sets optimization and representation-refresh choices, then examines how context size and k affect performance, stability, and compute.

A.3. Training Hyperparameters and Analysis

  • The authors use AdamW with a 1e−5 learning rate, gradient clipping at norm 1, a linear warmup-and-decay schedule, and batch size 10.Warmup peaks at 10% of training steps for synthetic experiments and 1% for existing datasets.Source: §A.3 · p. 20
  • Re-encoding explanation representations at least once per epoch avoids the noticeable performance drop seen when they are never rebuilt.They use re-encoding every 20% of an epoch in most experiments and every 30% for full-data e-SNLI.Source: §A.3 · p. 21
  • Stop-gradient updates only query data-point embeddings, not explanation embeddings; the authors report improved stability and efficiency without accuracy differences.The text says this choice contributed to stability in early experiments and confirmed no accuracy change.Source: §A.3 · p. 21
  • A larger context size C helps each conditioning method until around 8, after which performance plateaus.The comparison uses evidential explanations and the optimal retrieval model.Source: §A.3 · p. 21
  • Training performance can be sensitive to k; higher k is preferable when possible, but multiplies training forward passes by k.The authors describe this as a trade-off between compute budget and k.Source: §A.3 · p. 21

A.4. Model Selection in Experiments

Model selection uses development performance, with special handling for unstable synthetic runs and tuning of retrieval inputs and hyperparameters.

A.4. Model Selection in Experiments

  • Within each training run, the selected model is the one with the best dev accuracy at an epoch's end..Source: §A.4 · p. 21
  • When synthetic training fails after a few epochs and trends toward random 50% accuracy, the authors rerun with another seed to obtain a stable run.They report stable dynamics typically emerged with one additional seed.Source: §A.4 · p. 21
  • No retrieved-input ablation significantly changes dev accuracy; the authors use e for NLI and (y, e) for relation extraction.They test adding x to e for e-SNLI and adding x to (y, e) for relation extraction, roughly controlling sequence length; all use k = 4.Source: §A.4 · p. 21, Table 5 · p. 21
  • For existing datasets, C and k are tuned separately by training-set size using dev accuracy; the full-data e-SNLI result uses one (C=2, k=4) run.Three seeds are used for each tuning condition, except for full-data e-SNLI, where one run is reported.Source: §A.4 · p. 21

B. Experimental Details And Additional Results

Appendix experiments extend the synthetic-task analyses across model and data choices, explanation representations, feature correlations, and retrieval-model choices.

B. Experimental Details And Additional Results

B.2. RQ2: Can retrieval of past explanations enable a model to solve our task?

B.4. RQ4: What is the best way to compute explanation representations for prediction?

  • 9.3 points at n = 1000 and 9.2 points at n = 1500 separate TEXTCAT from H-MEAN; the gap narrows to 1.3 points at n = 2500, and they tie after n = 5000.These comparisons use the optimal retrieval model.Source: §B.4 · p. 22, Figure 15 · p. 22

B.5. RQ5: What makes an explanation relevant across data points? What enables a retrieval model to find relevant explanations for a new data point?

  • Ordering domain and codomain values before pairing makes nearby explanation indices correspond to nearby m and n task parameters, enabling more precise inference from similar indices.The non-smooth condition omits that ordering.Source: §B.4 · pp. 22–23

B.6. RQ6: Can explanations help models learn to use strong features rather than weak ones?

  • 75% is the upper bound for optimal retrieval with perfectly correlated features; explanations help the features be used, but do not resolve which feature to use.As correlation rises, no-retrieval performance also rises, but never reaches retrieval performance.Source: §B.6 · p. 23, Figure 16 · p. 23
  • Adding 5 to every integer in full-info explanations makes them harder to interpret with little data, but the correct interpretation emerges with more data.This is shown for zero correlation using optimal retrieval.Source: §B.6 · p. 23, Figure 17 · p. 23

B.7. RQ7: How does the co-dependence between classifier and retrieval model influence the viability of joint training?

  • Learning retrieval on the synthetic task requires pretrained Sentence-RoBERTa; pretrained RoBERTa-base does not suffice and can underperform an identically structured random model.The authors suggest mean-token pooling and cosine similarity may explain the result.Source: §B.7 · p. 23, Figure 18 · p. 23

B.8. RQ8: Does retrieval of explanations improve model performance on existing datasets?

On existing datasets, the appendix reports no benefit from increasing test-time k and compares one prior retrieval condition with baselines; it also describes uncertainty estimation.

B.8. RQ8: Does retrieval of explanations improve model performance on existing datasets?

  • Existing-dataset experiments use 5 epochs for full-data e-SNLI, 20 epochs for n ≤ 10000, and 10 epochs for other larger training sets..Source: §B.8 · p. 23
  • For e-SNLI with n = 10000, dev accuracy shows no meaningful change across k values from 1 to 20, so larger test-time k is not a reliable way to improve accuracy.The figure plots dev set accuracy across k.Source: §B.8 · pp. 23–24, Figure 19 · p. 24
  • 87.99% on TACRED and 76.46% on SemEval fall below their respective baselines of 88.29% and 76.94% for ELV-M.ELV-M is H-MEAN with fixed retrieval and (C=10, k=1).Source: §B.8 · p. 24

B.9. Confidence Intervals and Hypothesis Testing

  • Synthetic-task confidence intervals estimate seed variance from representative conditions and assume it remains invariant across other experimental factors; existing-dataset intervals represent sample variance.The synthetic estimates prioritize explanation kind, then conditioning mechanism; more runs would increase the experiment count from 172 to 1035.Source: §B.9 · p. 24
  • For existing datasets, the reported accuracy intervals are 95% intervals on the underlying binomial probability, and RQ8 uses two-sided difference-in-binomial-means tests..Source: §B.9 · p. 24

C. Synthetic Task Generative Process

The synthetic task pairs sampled indices with integer tuples, then generates sequences and labels for each index. The process controls task balance and excludes training sequences from development and test sets.

C. Synthetic Task Generative Process

  • Development and test sequences are constrained not to appear in training; the general process can be altered for particular experiments, including RQ5.The general setup uses maximum integer and index values of 100 and 10,000, respectively.Source: C. Synthetic Task Generative Process · p. 24

1. Sample {indext}num-tasks

  • Indices are sampled uniformly without replacement from integers 1–10,000, and each is paired without replacement with a tuple of four distinct integers sampled from 1–100.The paired index–tuple set is put in arbitrary order.Source: §1 · p. 24, §2 · p. 24

2. Sample {(m, n, r, d)t}num-tasks

  • Each index receives sample-size // num-tasks points, with balanced vectors controlling indicators, feature correlation, and which integer pair is more numerous.The correlation vector’s balance changes when the intended strong–weak correlation changes.Source: §2 · pp. 24–25
  • For each point, the index and indicator occupy the first two sequence elements, tuple integers are allocated to the remaining 18 slots, and any unfilled slots receive independent uniform samples from 1–100.The text specifies allocation based on the corresponding vectors’ elements.Source: §2 · p. 25
Loading 2102.02201v2…