PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Class Marginals: Bounding Rehearsal Gaps without Freezing Class Co-occurrence

Congren Dai, Nat Roongjirarat, Fei Ye

arXiv:2609.25735v1cs.LGcs.AIstat.ML

TL;DR

Class-balanced replay controls class frequency but does not specify when a stored class returns. The paper studies rehearsal gaps separately from class marginals and co-occurrence, introducing randomized-pass replay. RPR improves final accuracy in the primary ER-ACE comparisons, while the effects vary across learners and regimes.

  • Problem

    Storage-side balancing and class-balanced retrieval control memory occupancy and expected replay share, but neither specifies when a stored class returns.

    Source: §1 · p. 1

  • Method

    The paper separates class marginals, rehearsal gaps, and class co-occurrence, then evaluates replay schedules that bound resident-class gaps while preserving the fixed-set time-averaged marginal.

    Source: §1 · p. 2

  • Results

    RPR gains 0.72–1.67 percentage points over independent class-balanced retrieval under reservoir storage, with positive mean effects under both storage rules in every tested stream.

    Source: §5.2 · p. 7

  • Takeaways & Limitations

    RPR improves final accuracy across the primary ER-ACE storage settings, while pretrained gains concentrate in the tested LT10 small-batch settings.

    Source: §6 · p. 9

    The temporal controls change gaps and co-occurrence jointly, so the experiments do not isolate rehearsal-gap length from other forms of temporal dependence.

    Source: §6 · p. 9

Abstract

from arXiv · show

Class-balanced replay controls class frequency but does not determine the interval between successive replay appearances of a class. We study this interval, the rehearsal gap, separately from the class marginal and class co-occurrence, and introduce randomised-pass replay (RPR), which visits each resident class once per shuffled pass. For a fixed set of C resident classes and replay batch size b less than or equal to C, RPR preserves the balanced time-averaged class marginal and bounds every gap by 2*ceil(C/b)-1; a churn-conditional bound applies while the resident set changes. The scheduler uses no future class information and adds no replay examples or forward passes. In a linear-head ER-ACE diagnostic, joint absence from the incoming and replay batches produces a one-sided classifier-bias gradient. Longer absence episodes are associated with larger negative bias displacement, and removing the incoming-loss mask attenuates the scheduling effect. In the primary ER-ACE experiments, RPR improves final average accuracy by 0.72-1.67 percentage points relative to independent class-balanced retrieval under reservoir storage, with positive effects also observed under balanced storage. Pretrained ViTs show positive effects on the tested LT10 streams with small replay batches, while matched larger-batch controls show no material effect. Fixed-cycle and reused-pass controls change more than one temporal statistic, so the experiments do not isolate rehearsal-gap length from all other forms of temporal dependence. The accuracy effects depend on the learner and operating regime.

1 INTRODUCTION

Class-balanced replay controls expected class exposure but not the timing of replay appearances or the classes paired in each batch. The paper separates these schedule properties and introduces randomized-pass replay (RPR), which bounds rehearsal gaps while preserving the fixed-set time-averaged class marginal.

1 INTRODUCTION

  • 24.82 and 25.00 steps are the mean gaps for independent retrieval and shuffled-pass scheduling, but their maximum gaps are 263 and 49, respectively.The example uses a fixed balanced memory of 200 classes, eight replay examples per step, and a 3,000-step run.Source: §1 · p. 1
  • Replay schedules can share the same class marginal yet differ in rehearsal-gap tails and class co-occurrence.The paper distinguishes class marginal, rehearsal gap, and class co-occurrence as separate properties of a replay sequence.Source: §1 · p. 1, Figure 1 · p. 2
  • RPR visits resident classes using a persistent shuffled deck, reshuffling after each pass and updating the deck as classes enter or leave memory.On a fixed resident set, each class is visited once per pass.Source: §1 · p. 2
  • For a fixed resident set, RPR preserves the balanced time-averaged marginal and bounds every gap by 2⌈C/b⌉−1.The scheduler requires no future class identities, total class count, or task boundaries; a churn-conditional gap bound applies as memory changes.Source: §1 · p. 2
  • The ER-ACE diagnostic examines how joint absence from incoming and replay batches relates to classifier-bias updates, while accuracy effects are evaluated separately.The incoming-loss mask removes a seen but absent class from incoming competition, while replay examples of other classes continue to contribute positive gradients to its classifier bias.Source: §1 · p. 2
  • The evaluation compares storage policies and replay budgets, but its temporal controls do not isolate rehearsal-gap length from other forms of temporal dependence.The shuffled-pass primitive is standard; the paper's contributions concern its use in replay.Source: §1 · p. 2

2 RELATED WORK

Related work addresses memory contents, retrieval criteria, and replay ordering through distinct approaches. This paper positions its contribution as class-level scheduling at fixed storage, loss, and replay budget.

2 RELATED WORK

  • Storage methods such as CBRS (Chrysakis & Moens, 2020) and BRS (Buzzega et al., 2021) alter memory contents, whereas MIR and ASER (Shim et al., 2021) optimize per-example retrieval criteria.The paper compares schedules while holding storage, loss, and replay budget fixed.Source: §2 · p. 3
  • The Balanced primitive uses fixed class-id order and bounded gaps, with repeated class partitions when b | C; RPR instead orders resident classes without task boundaries.Task-level scheduling learns which previous tasks to replay (Klasson et al., 2023).Source: §2 · p. 3
  • Replay losses differ in whether each example's gradient contribution is independent of other examples in the batch, a property formalized for replay terms in Definition 1.ER-ACE masks seen-but-absent classes from the incoming loss, while DER regresses stored logits and DER++ adds supervised replay cross-entropy (Buzzega et al., 2020).Source: §2 · p. 3

3 SETTING AND NOTATION

The setting is task-free online learning: a learner processes a stream, retrieves from memory, and updates its parameters at each step. Storage determines which examples remain resident, while retrieval determines which are replayed.

3 SETTING AND NOTATION

  • A task-free learner processes the stream once without task-boundary signals, receiving incoming minibatches and retrieving examples from memory.The memory contains at most m previously observed examples.Source: §3 · p. 3
  • At each step, the learner takes one gradient step on the sum of incoming and replay losses.The requested replay budget is min(b, |M_t|), and the realized replay batch can be smaller; an empty memory contributes no replay loss.Source: §3 · p. 3
  • The resident set K_t contains classes represented in memory, and the scheduler needs only this current set rather than future class identities or counts.Storage chooses which examples remain in memory; retrieval chooses which return in the replay batch.Source: §3 · p. 3

4 RANDOMISED PASS REPLAY

The section separates replay class marginals, rehearsal gaps, and class co-occurrence, then develops randomised-pass replay (RPR) and analyzes its schedule properties and interaction with ER-ACE.

4.1 REPLAY MARGINALS, GAPS, AND CO-OCCURRENCE

  • The analysis distinguishes class marginal, rehearsal gap, and class co-occurrence; under churn, finite-horizon empirical frequencies need not match across retrieval policies.Equal replay size does not ensure per-class frequency equality, and similar aggregate mean gaps do not establish it either.Source: §4.1 · pp. 3–4
  • On a fixed set, i.i.d. balanced retrieval has geometric gaps with mean C/b and unbounded support when b<C; fixed cycling bounds gaps but can repeat pairings.Fixed cycling repeats class pairings within a fixed partition when b divides C.Source: §4.1 · p. 4

4.2 RANDOMISED PASS SCHEDULER

  • On a fixed resident set, RPR preserves the balanced time-averaged marginal and mean gap C/b while bounding every gap by 2⌈C/b⌉−1.The churn-conditional bound applies only while a class remains resident; retrieval cannot bound latency after eviction.Source: §4.2 · pp. 4–5
  • With churn, the conditional guarantee concerns intervals when the class remains resident, not wall gaps that include eviction periods.The bound uses the maximum resident count and insertions during the interval, and may be more conservative than the fixed-set bound.Source: §4.2 · p. 5

4.3 GAP BOUNDS AND CO-OCCURRENCE

  • Any recurrent schedule has maximum gap at least ⌈C/b⌉; when b divides C, attaining that gap exactly forces each class to retain the same batch partners.This fixed-partition conclusion requires exact equality, not merely gaps approaching the lower bound.Source: §4.3 · p. 5
  • When b divides C, RPR redraws the class partition each pass; its gap-variance ratio to i.i.d. retrieval approaches 6, while its gap bound is independent of n.The i.i.d. maximum-gap scale grows as O(log(nC)/log(1/(1−p))) for p=b/C<1.Source: §4.3 · p. 5

4.4 INTERACTION WITH ER-ACE

  • For composition-blind replay, matched per-example inclusion probabilities give the same one-step expected replay gradient at fixed parameters, but temporal dependence can alter later trajectories.Dark Experience Replay satisfies the composition-blind definition for a batch-independent forward map.Source: §4.4 · pp. 5–6
  • In ER-ACE, joint absence yields one-sided bias pressure across the g−1 interior steps, but changing probabilities prevent gap length alone fixing magnitude; this diagnostic does not explain primary cosine-classifier gains.The equation characterizes a bias gradient, not a decrease in every class-c logit.Source: §4.4 · p. 6

5 EXPERIMENTS

The experiments test replay scheduling across learners, storage policies, stream types, and replay budgets. RPR improves accuracy in several tested settings, but effects vary by regime, and the controls do not isolate rehearsal-gap length from other temporal dependencies.

5.1 EXPERIMENTAL SETUP

  • The primary study uses one-pass class-incremental streams, chiefly ER-ACE with a randomly initialized ResNet-18, ten paired seeds, and final task-mean accuracy.It also tests DER, DER++, OCM, and pretrained ViT-B/16; the from-scratch comparisons characterize a small-replay-budget regime, not other ER-ACE protocols.Source: §5 · p. 6, §5.1 · pp. 6–7

5.2 RESULTS

  • RPR improves final accuracy by 0.72–1.67 percentage points over independent class-balanced retrieval under reservoir storage, with positive mean effects under both storage rules in every tested stream.Within BRS on CIFAR-100 LT10, RPR gains 1.38 points and all ten paired differences are positive.Source: Table 1 · p. 7, §5.2 · p. 7
  • On LT10 streams, pretrained ViTs show positive RPR effects with b = 8, while matched b = 32 controls show no material effect.At b = 8, gains are 2.08 and 1.53 points; at b = 32, effects are −0.14 and +0.17 points, with both intervals including zero. On balanced streams at b = 32, effects are −1.09 on CIFAR-100 and +0.22 on ImageNet-R.Source: Figure 2 · p. 8, §5.2 · p. 9, Table 3 · p. 9
  • RPR has higher mean accuracy than fixed cycling in all eight storage–stream combinations, but only six intervals exclude zero; both CIFAR-100 LT100 contrasts remain inconclusive.All eight RPR–i.i.d. and six RPR–fixed contrasts remain significant after within-family Holm adjustment.Source: §5.2 · p. 8
  • Under changing memory, RPR’s mean per-run resident maxima are 23.9, 48.1, and 46.3 steps across the three tested streams, with no churn-conditional bound violations.Independent balanced draws have similar aggregate mean gaps but mean per-run maxima of 81–176 steps.Source: §5.2 · p. 8
  • In linear-head ER-ACE, longer joint-absence episodes accompany larger negative bias displacement, and removing the incoming mask attenuates RPR’s scheduling effect.For CIFAR-100, mean displacement changes from −0.00191 at 1–4 steps to −0.00926 at ≥33 steps; RPR reduces the resident-gap 99th percentile by about 47% on both datasets, while mask–schedule interactions are 0.93 and 1.06 points with intervals above zero.Source: §5.2 · p. 8, Table 2 · p. 8
  • Results depend on the replay loss and temporal controls: DER, DER++/Refresh, and OCM are inconclusive or dataset-dependent, and reused-pass and fixed-cycle controls do not isolate gap tails from other temporal dependencies.Fixed cycling has the shortest gaps among those controls without the largest accuracy gain; RPR’s accuracy is non-monotone in pair diversity.Source: §5.2 · pp. 8–9
  • In the CIFAR-100 LT10 buffer sweep, class deactivations fall from 16.1 to zero as capacity rises from 200 to 5,120, while RPR shortens gaps throughout.Mean accuracy differences vary non-monotonically, with endpoint estimates of 0.30 and 1.23 points.Source: §5.2 · p. 9

6 CONCLUSION AND LIMITATIONS

RPR bounds rehearsal gaps while preserving the fixed-set time-averaged class marginal, and improves final accuracy in primary ER-ACE experiments. The evidence is scoped by learner and regime, with marginal preservation under churn and isolation of gap effects remaining unresolved.

6 CONCLUSION AND LIMITATIONS

  • RPR improves final accuracy across primary ER-ACE storage settings, while pretrained gains concentrate in tested LT10 streams with small replay batches.Effects depend on learner and operating regime.Source: §6 · p. 9
  • Marginal preservation under a changing resident set remains unestablished.Source: §6 · p. 9
  • The fixed-set gap bound is vacuous when b ≥ C, and its application under churn is conditional on storage.Evidence covers single-pass CIFAR-100, Tiny-ImageNet, and ImageNet-R, not multi-epoch, offline, or ImageNet-scale continual learning.Source: §6 · p. 9
  • Fixed-cycle and reused-pass controls change gaps and co-occurrence jointly, so they do not isolate rehearsal-gap effects from other temporal dependence.The linear-head bias diagnostic does not fully explain cosine-head or cross-host results.Source: §6 · p. 9

7 AI USE STATEMENT

The appendices document tool use, reproducibility and ethics, then report measured gap statistics, implementation details, and further scheduling and gradient analyses. The fixed-set gap diagnostics match the RPR bound, while churn bounds require the class to remain resident.

7 AI USE STATEMENT

  • AI assisted linguistic refinement, restructuring, formatting, experiment orchestration, analysis, and verification, while authors retained research and interpretation decisions and verified reported quantities.Source: §7 · p. 10

A REPRODUCIBILITY STATEMENT

  • The release includes the scheduler, method integrations, configurations, and analysis scripts; test comparisons use paired seeds, with validation runs separate from test evaluation.The 256 validation runs do not contribute to a test contrast.Source: A REPRODUCIBILITY STATEMENT · p. 12

B ETHICS STATEMENT

  • The study uses public image benchmarks without new data or human subjects, but notes possible source-data biases and recommends per-class evaluation when disparities matter.Rare classes can benefit in from-scratch long-tailed settings and lose accuracy in a pretrained control.Source: B ETHICS STATEMENT · p. 12

C MEASURED REHEARSAL GAPS

  • At C = 200 and b = 8, CB and RPR mean gaps are 24.82 and 25.00 steps, while their maxima are 263 and 49, respectively.The fixed-memory comparison holds the long-run class-visit marginal fixed; upper quantiles and maxima expose long absences that the mean does not.Source: C MEASURED REHEARSAL GAPS · pp. 12–13, Table 4 · p. 12

D THE SCHEDULE IN PSEUDOCODE

  • The scheduler maintains a persistent shuffled class deck across calls, updates it from current memory, and freshly shuffles each class's example list at replay steps.It receives neither future class identities nor total class count.Source: D THE SCHEDULE IN PSEUDOCODE · p. 13

E.1 STATIC SCHEDULE DIAGNOSTICS

  • Across four configurations, RPR's observed maximum equals 2⌈C/b⌉−1; its 99th-percentile gaps are 2.0–2.5 times smaller than i.i.d. balanced retrieval.RPR's mean-gap deviation from C/b is at most 0.005, versus 0.18 for the i.i.d. draw.Source: §E.1 · p. 14

E.2 COMPOSITION DEPENDENCE AND THE ER-ACE MASK; E.3 FROM LOGIT GRADIENTS TO PARAMETER UPDATES

  • The ER-ACE incoming-loss mask excludes seen-but-absent classes from incoming competition, while other-class replay contributes positive classifier-bias gradients during joint absence.The accumulated pressure spans g−1 interior steps, but its magnitude is not determined by gap length alone; shared-trunk effects and optimizer details limit sign conclusions beyond the stated linear-head setting.Source: §E.2 · p. 14, §E.3 · p. 14

E.4 PROOF OF THE CHURN-CONDITIONAL BOUND

  • The churn-conditional gap bound requires the class to remain resident; no retrieval policy can include a class while it is absent from memory.Source: §E.4 · p. 15

E.5 ADDITIONAL SCHEDULING RESULTS

The additional results compare RPR's worst-case gap to a universal lower bound, characterize exact attainment, and examine stream-length scaling and convex gap costs. These scheduling results do not establish an accuracy relationship.

E.5 ADDITIONAL SCHEDULING RESULTS

  • For any recurrent schedule, the worst-case gap is at least ⌈C/b⌉; RPR's worst-case ratio to this lower bound is at most 2 − 1/⌈C/b⌉.A window of ⌈C/b⌉−1 steps cannot visit all C classes.Source: §E.5 · p. 15
  • When b divides C, attaining the lower-bound gap for every class and visit forces a fixed partition into blocks of size b, with within-block classes always co-occurring.Classes in different blocks never co-occur; RPR instead resamples a class permutation each pass.Source: §E.5 · p. 15
  • The i.i.d. maximum gap grows logarithmically with stream length, whereas RPR's bound 2⌈C/b⌉−1 is independent of stream length.The result concerns scheduling statistics and does not imply a monotone relationship with accuracy.Source: §E.5 · p. 16
  • The finite-m variance expression is exact when b divides C; otherwise a partial pass changes the finite-sample distribution, making √6 an asymptotic reference rather than an exact prediction.Source: §E.5 · p. 16
  • For a specified non-decreasing convex gap cost at fixed mean gap C/b, the expected cost is bounded by the cost at that mean.Strict convexity makes equality require constant gaps; under mere convexity, nonconstant gaps can also attain equality when the cost is affine on their support.Source: §E.5 · p. 16

F ROBUSTNESS TO THE EVALUATION-TIME WEIGHT AVERAGE

Without the averaged readout, RPR retains material gains over uniform retrieval in all four primary operating cells, while balanced-draw effects are material in only one. Across storage policies, RPR also exceeds independent draws in every adjusted test.

F ROBUSTNESS TO THE EVALUATION-TIME WEIGHT AVERAGE

  • Removing the EMA increases the balanced-draw effect in the evaluated readout comparison, showing that its relative effect depends on readout.The comparison used reservoir storage, replay batch 32, and four dataset–stream cells.Source: F ROBUSTNESS TO THE EVALUATION-TIME… · p. 16
  • RPR gains over uniform retrieval are +2.41, +2.22, +1.39, and +1.92 points in the four primary cells without the averaged readout, with material effects in all four.Its margins over the balanced draw are +2.63, +1.14, +1.03, and +1.28 points; balanced-draw effects are material in only one cell.Source: F ROBUSTNESS TO THE EVALUATION-TIME… · pp. 16–17

G ACCURACIES FOR CROSSED STORAGE AND RETRIEVAL POLICIES

  • RPR has the highest mean accuracy within every storage–stream row, and BRS with RPR gives the best mean in all three long-tailed streams.On balanced CIFAR-100, reservoir storage with RPR is slightly higher than BRS with RPR, 20.61 versus 20.55.Source: G ACCURACIES FOR CROSSED STORAGE AND… · p. 17, Table 5 · p. 17

H PAIRED STATISTICAL TESTS

  • Every RPR–i.i.d. comparison remains significant at the 5% level after adjustment under both reservoir and BRS storage.RPR exceeds fixed cycling in six of eight adjusted tests, with both exceptions on CIFAR-100 LT100.Source: H PAIRED STATISTICAL TESTS · p. 17

I MEASURED GAPS INSIDE THE ONLINE RUNS

Online gap measurements test the churn-conditional bound and distinguish resident gaps from wall-clock gaps when classes leave memory. Accuracy analyses then compare frequency groups and macro forgetting across from-scratch and pretrained settings.

I MEASURED GAPS INSIDE THE ONLINE RUNS

  • Table 6 reports paired tests of RPR against independent balanced retrieval and fixed cycling.The paired-test variation is in within-seed differences, and each Holm adjustment covers four streams for one storage policy and comparator.Source: Table 6 · p. 18
  • RPR shows no observed churn-conditional bound violations in any of three settings, while independent retrieval exceeds the threshold hundreds of times per run.On Tiny-ImageNet LT100, RPR's wall maximum is 47.8 steps versus a resident maximum of 46.3, alongside 7.4 class deactivations per run.Source: I MEASURED GAPS INSIDE THE ONLINE RUNS · p. 18, Table 7 · p. 18

J HEAD, MID AND TAIL ACCURACY AND PER-CLASS FORGETTING

  • Frequency groups are formed by sorting classes by observed online frequency, and macro accuracy weights classes equally.The table compares RPR with i.i.d. class-balanced retrieval; negative forgetting differences indicate improved retention.Source: J HEAD, MID AND TAIL ACCURACY AND… · pp. 18–19
  • From-scratch runs show positive mean accuracy differences in all frequency groups, with tail gains at least as large as head gains and macro forgetting reduced in all three settings.The intervals exclude zero for every group except the Tiny-ImageNet LT100 head; pretrained b = 32 controls show a 2.25-point CIFAR-100 tail loss and greater forgetting, while all ImageNet-R intervals include zero.Source: J HEAD, MID AND TAIL ACCURACY AND… · p. 19, Table 8 · p. 19

K.1 BIAS DISPLACEMENT DURING JOINT ABSENCE

The section examines bias displacement during joint absence, how renewal frequency changes co-occurrence and gaps, and how buffer capacity relates to scheduling effects. Across these diagnostics, the comparisons have limits on causal attribution and class availability.

K.1 BIAS DISPLACEMENT DURING JOINT ABSENCE

  • The bias-displacement diagnostic groups completed episodes by replay-to-replay gap and accumulates displacement only over jointly absent updates.An incoming-only return neither closes nor censors an episode, and eviction discards an open episode.Source: §K.1 · p. 19
  • All populated gap bins have a negative-displacement fraction of 1.00, and mean displacement magnitude increases with episode length under both retrieval policies.The bins condition on different episodes and classes, so policy differences within a bin are not isolated causal estimates; displacement also does not determine every logit's movement through the shared representation.Source: §K.1 · pp. 19–20, Table 9 · p. 19

K.2 RENEWAL OF CLASS CO-OCCURRENCE

  • On CIFAR-100, mean accuracy gain falls from 1.23 points for RPR to 0.47 for fixed cycling, while intermediate schedules have slightly larger point estimates than RPR on Tiny-ImageNet.Less frequent renewal reduces pair coverage and entropy on both datasets, but all bounded schedules have positive mean gains; renewal changes gap distributions too, so pair diversity is not isolated.Source: Table 10 · p. 20, §K.2 · p. 20

K.3 BUFFER CAPACITY AND CLASS RESIDENCY

  • RPR gains increase to 0.68, 1.16, and 1.23 points at the three larger buffer capacities, while class deactivations fall from 1.9 to zero.RPR shortens the gap tail at every capacity, but its accuracy effect is small at buffers 200 and 500, and the 500 interval includes zero; retrieval cannot revisit a class absent from memory.Source: Table 11 · p. 20, §K.3 · p. 21

L LOSS INTERVENTIONS AND CROSS-HOST COMPARISONS

Loss interventions and cross-host comparisons test how retrieval effects vary with the learner, while trajectory and additional retrieval controls examine their scope. The results support setting-dependent effects rather than a universal advantage across learners and operating regimes.

L LOSS INTERVENTIONS AND CROSS-HOST COMPARISONS

  • Removing ER-ACE's incoming-loss mask attenuates the balanced-draw retrieval effect, but the controls do not establish mask dependence as a necessary or sufficient rule across hosts.The mask itself improves ER-ACE by 5.39 points on CIFAR-100 and 4.16 on Tiny-ImageNet, a separate contrast from the retrieval effect.Source: Table 12 · p. 21, L LOSS INTERVENTIONS AND CROSS-HOST… · p. 21
  • Without the ER-ACE mask, balanced-draw point estimates are near zero and both intervals include zero; with the mask, RPR gains 1.02 points on CIFAR-100 and 0.87 on Tiny-ImageNet in DER++.Adding the mask to DER++ moves both point estimates positive, but neither interval excludes zero; OCM's 0.30-point effect is inconclusive, and DER++/Refresh on CIFAR-100 has a +0.90-point effect.Source: L LOSS INTERVENTIONS AND CROSS-HOST… · p. 21

M ACCURACY ACROSS THE LEARNING TRAJECTORY

  • At b = 8, RPR improves mean task-boundary accuracy over independent balanced retrieval by 1.30 points on CIFAR-100 and 1.13 points on ImageNet-R.At b = 32 both intervals include zero, and every fixed-cycle contrast includes zero; these measurements do not establish an additional effect over fixed cycling.Source: Table 13 · p. 22, M ACCURACY ACROSS THE LEARNING… · p. 22

N ADDITIONAL RETRIEVAL CONTROLS

  • On CIFAR-100, relative test-accuracy changes are −50.1% for MIR, 0% for uniform retrieval, +5.8% for independent balanced retrieval, and +7.4% for RPR.MIR retrieves 7.4 distinct classes per batch on average versus 21.7 for uniform retrieval and has lower accuracy on both datasets; the comparison changes retrieval objective as well as batch composition.Source: Table 14 · p. 22, N ADDITIONAL RETRIEVAL CONTROLS · p. 22

Related papers

Loading 2609.25735v1…