PaperlayerView exact revision on arXiv

Source-linked AI summary

Verification Learning Theory: A Process Model of Checking and Retention in AI-Assisted Judgment

Christoph Trattner

arXiv:2610.37875v1cs.AI

TL;DR

Accurate AI assistance can improve immediate judgments while reducing practice for independent performance, but evidence does not yet establish how perceived reliability changes checking and delayed rule-specific transfer. VLT models checking as the link between reliability and retention, then uses simulations to examine its predictions and parameter dependence.

  • Problem

    Research has not established how perceived assistant reliability changes checking during practice and thereby supports delayed, rule-specific transfer.

    Source: §2 · p. 4

  • Method

    VLT models how task demands and perceived reliability prompt evidence checks, which add rule-specific traces when followed by correct, independent explanations.

    Source: §2 · p. 4, §3 · p. 7

  • Results

    5.8, 3.6, and 2.4 points were the simulated median balanced-accuracy gains for task, fallibility, and belief, respectively.

    Source: §5 · p. 13

  • Takeaways & Limitations

    The model makes checking histories a basis for forecasting which kinds of claims a person may misjudge, while treating recognition of the applicable rule as beyond its current account.

    Source: §4 · p. 9, §7 · p. 17

    VLT addresses controlled rule use with supplied evidence and correct, rule-naming explanations, but does not yet describe recognizing the applicable rule when unfamiliar claims offer ambiguous or competing cues.

    Source: §7 · p. 17

Abstract

from arXiv · show

Accurate AI assistance can improve immediate judgments while reducing opportunities to practise the operations needed for independent performance. Verification Learning Theory (VLT) formalises this possibility for claim-evaluation practice followed by correct, independent, rule-specific feedback. The model links perceived assistant reliability to evidence checking and links the history of those checks to retained rule use. Its strict form assumes that task demands and perceived reliability influence learning only through checking, with equal learning per genuine check. It therefore predicts that perceived fallibility can improve delayed unaided judgment even without observed assistant errors, and that this advantage disappears when checking is already required and performed. Rule-specific gains and forecasts from checking histories provide further tests. Simulations examine the consequences and parameter dependence of these commitments; they do not constitute empirical validation. Comparisons with depth-of-processing and feedback-encoding accounts identify where predictions diverge and where the accounts remain observationally equivalent. The framework specifies experiments that can test the proposed mechanism and separates retention of practised rules from the broader problem of recognising which rule applies to a new claim.

1 The learning problem

The section frames AI-assisted judgment as a learning problem: correct assisted answers can leave users with less practice for later unaided performance. It defines the target as learning evidence-evaluation rules through checking, and proposes that task demands and perceived reliability shape retention by prompting those checks.

1 The learning problem

  • The account addresses a controlled instructional setting in which people judge claims against supplied evidence and receive correct, independent explanations of the relevant rule.Transfer to unfamiliar claims with ambiguous or competing rule cues is proposed as a test of the account’s reach, not an established simulation result.Source: §1 · p. 2
  • Learning a rule means learning to test whether evidence warrants a claim, rather than merely remembering an answer such as “the drink works.”For example, a trial in trained athletes does not establish an effect for adults who do not exercise regularly.Source: §1 · p. 2
  • A check is an attempted application of the relevant evidential rule before reading the explanation; a wrong attempt counts, but copying a correct verdict does not.The section distinguishes a check-requiring task, which cannot be answered correctly without the check, from a copyable task, which can be answered from a correct verdict.Source: §1 · pp. 2–3
  • Delayed unaided judgment concerns applying evidence-evaluation rules to new claims without the assistant, including recognizing which rule applies and withholding judgment when evidence fails its check.The model formalizes retention and use of a specified rule, not how people recognize the relevant rule amid ambiguous or competing cues.Source: §1 · pp. 2–3
  • VLT predicts that task demands and perceived assistant reliability affect retained rule use only through the checks they prompt, with equal learning from each genuine check.This proposed mechanism is the accuracy paradox: an always-right assistant may teach less than one perceived as fallible, even when both are followed by correct explanations.Source: §1 · p. 3

2 From reliance to retained learning

VLT combines established ideas about reliance, learning through practice, and memory to propose that evidence checks link perceived assistant reliability to retained rule use. It addresses a gap in understanding how reliability changes checking during practice and thereby supports delayed, rule-specific transfer.

2 From reliance to retained learning

  • VLT combines reliance, learning-through-processing, and memory principles: people monitor reliable aids less, retained learning depends on practice processing, and knowledge strength changes with use and spacing (Parasuraman & Manzey, 2010; Chi & Wylie, 2014; Cepeda et al., 2008).The account also treats transfer as distinct from retention: people may not notice unprompted that a new case calls for a known rule.Source: §2 · pp. 3–4
  • Prior work on claim evaluation reports that some interventions’ effects faded within weeks or months, while rules taught about treatments still helped a year later.These literatures do not establish how perceived reliability changes checking during practice and thereby supports delayed, rule-specific transfer.Source: §2 · p. 4
  • With correct rule-specific feedback held constant, task demands and perceived fallibility can substitute for one another by prompting the same learning-relevant act: checking the evidence.The strict model assumes that a genuine check has the same learning value regardless of what prompted it; this is most plausible for short, well-defined checks learners can already attempt.Source: §2 · p. 4
  • The paper studies four ways claims can exceed their evidence: inferring causation from association, extending findings to unstudied people, treating a proxy as the outcome, or extending effects beyond tested conditions.Rules test whether evidence warrants the claim and can be applied across topics; responses are acceptance, rejection, or withholding when no evidence passes the applicable checks.Source: §2 · pp. 4–5
  • VLT’s contribution is its formal restriction and resulting experimental predictions, not the general distinction between assisted performance and independent capability.Within a broader cognitive-work-allocation framework, it links perceived reliability to checking and the timing and distribution of checks to retained rule use.Source: §2 · pp. 5–6

3 The process model

VLT links belief updating, evidence checking, learning and retention, and response selection in a model of delayed unaided judgment. It specifies how assistant reliability and task demands affect checking, while retained rule strength influences later responses.

Belief updating

  • Perceived reliability blends the stated reliability with the share of observed verdicts that were correct, with the statement weighted as κ verdicts.Errors lower perceived reliability and correct verdicts raise it; the component requires ratings to move toward observed accuracy.Source: §3 · pp. 6–7

Checking

  • With copyable tasks, checking becomes more likely as perceived reliability falls; when a task requires checking, earnest responders check regardless of assistant reliability.The model treats compliance with check-required tasks as an assumption; differing rates of guessing or using another rule across reliability conditions would violate it.Source: §3 · p. 7

Learning and retention

  • A successful or failed check adds a full-strength trace to the relevant rule, while an unchecked explanation adds a small, condition-invariant fraction ε.The strict model assumes equal learning per check, regardless of what prompted it.Source: §3 · pp. 7–8
  • Rule strength combines pre-practice strength with practice traces that fade over time; the model assumes spaced checks last longer than massed ones.The retention component is challenged if check histories forecast later errors no better than simpler models using check counts and time since the last check.Source: §3 · p. 8

Response selection

  • Retained strength increases recognition of the rule a new claim calls for, but the model does not explain how people identify applicable rules amid ambiguous cues.Response selection also separates learning effects from general response tendencies under assumptions about how those tendencies and rules affect claims.Source: §3 · pp. 8–9

Outcome and measurement

  • Delayed unaided judgment is measured as balanced accuracy across warranted responses, while checks are unobserved and must be inferred differently across task types.Inferred checks require a measurement model because the available indicators differ by task and are noisy; the proposed model has not been validated.Source: §3 · p. 9

4 Predictions and closest alternatives

The section turns VLT into predictions that vary task demands and perceived assistant reliability, then compares them with alternative learning accounts. Its central discriminator is whether doubt changes learning when checking is already required.

Predictions

  • VLT predicts better delayed judgment from required checks (E-X over E-R), belief-driven checks (B-R over E-R), and observed errors (F-R over both), but no reliability gain when checking is required.The predictions concern new claims months later, with the same claims, evidence, and correct explanations across conditions.Source: §4 · p. 9
  • A person’s check history should forecast later rule-specific misjudgments, with practice condition adding no more than a prespecified margin once checks are known.This prospective test requires a validated measurement model for checks.Source: §4 · pp. 9–10

Closest alternatives

  • Where checking is optional, VLT and the depth account predict similar patterns, but the depth account predicts larger gains and effects of doubt when checking is required.If doubt changes only whether a check occurs, requiring the check removes its effect; if doubt also deepens processing, B-X and F-X should exceed E-X.Source: §4 · p. 10
  • B-X exceeded E-X by median 0.5 balanced-accuracy points at half-strength doubt, 0.9 when doubled, and 1.4 when tripled in simulation.With three practice claims per rule instead of eight, doubling the learning value left a 2.2-point difference.Source: §4 · p. 10
  • The F-X versus E-X comparison is less diagnostic than B-X versus E-X because wrong verdicts add error-finding work and may change check success.Asking for an initial judgment before the flip response reduces, but does not remove, the asymmetry.Source: §4 · pp. 10–11
  • Fallibility-driven caution can mimic a balanced-accuracy gain under prior acceptance bias, so tests should report responses by warranted response and rule practice status.In simulation, caution alone lowered accuracy on claims warranting acceptance by 9.8 points and raised it on the other two response types by 4.3 points each.Source: §4 · p. 11
  • Six transparent benchmarks isolate answer retrieval, explanation learning, task-only engagement, error-driven learning, two-route effects, and general engagement.Some alternatives share VLT’s smaller fallibility gain under the flip task, so that comparison alone cannot distinguish them.Source: §4 · p. 11

5 Computational consequences

The simulations derive the strict model’s predicted patterns and show how gain sizes depend on open parameters. These are consequences of assumptions, not empirical evidence about how people learn.

What follows by construction

  • With a check-requiring task, the three assistants produce exactly equal delayed judgment; an effort-matched task equals the copyable task, and neither unpractised rules nor known checks add a condition effect to forecasts.These are identities of the strict model, not results that test whether people learn as VLT proposes.Source: §5 · pp. 12–13

What depends on parameter choices

  • 5.8 points for the task gain, 3.6 for the fallibility gain, and 2.4 for the belief gain were the median simulated 90-day gains in balanced accuracy.Across parameter sets, the share of claims checked averaged 100% with the flip task, 57% with an erring assistant, 40% with a stated error rate, and 16% with an assistant said to be always right.Source: §5 · p. 13
  • As ε rises, simulated gains shrink without reversing and vanish only as ε approaches 1; the belief gain ranges from about half to four-fifths of the fallibility gain depending on statement weight.The belief-gain share is higher when people give the statement the weight of 20 to 30 verdicts rather than 2 to 10.Source: §5 · p. 13, Figure 3 · p. 14
  • Retention assumptions and timing also matter: gains grew from a 7-day to a 90-day test and changed little up to a year, while weekly spacing produced 1.6 more median points than one-session practice with required checks.Replacing logistic checking with linear or threshold functions changed gain sizes but did not reverse their signs.Source: §5 · p. 13
  • Within copyable conditions, checking share correlated positively with 90-day accuracy (.22 to .42), while practice accuracy with an always-right assistant correlated negatively (−.31) because it came from copying.The simulations used 1,000 parameter sets and model expected differences, not rates at which a study would detect effects.Source: §5 · p. 13

What remains untested

  • The simulations cannot establish that people check, learn, or forget as VLT proposes; they also omit check-measurement simulation, rule identification, and comparison with simpler memory models.The latter comparison discriminates only when practice spacing varies between people.Source: §5 · p. 14

6 How to test the account

The proposed test holds practice materials and explanations constant while varying assistant reliability and whether the task requires checking. It specifies validity checks, decision margins, and a separate forecast test for whether checking histories predict retention.

One design

  • The design uses identical claims, evidence, explanations, and schedules across six conditions plus an effort-matched task; flip choices require identifying the rule-relevant change.An unpractised rule is counterbalanced, and the stated error rate requires ethical approval and debriefing.Source: §6 · p. 14
  • The primary 2×2 comparison of always-right and stated-error-rate assistants across copyable and check-requiring tasks tests the belief gain, its equivalence under required checking, and the task-by-belief interaction.The two assistant conditions have identical verdicts and explanations, isolating what participants are told and what the task asks.Source: §6 · p. 14
  • Error-condition comparisons estimate the fallibility gain but require caution because wrong verdicts can add work; having participants judge before flipping reduces, but cannot eliminate, that asymmetry.The design also proposes comparing evidence time across error and correct trials and adding a no-verdict condition.Source: §6 · p. 15

What must hold

  • A test counts against VLT only if reliability differences, genuine checking, matched effort, and delayed unaided judgment are verified independently of the outcome.The proposed delayed test uses new topics and includes claims requiring an unpractised rule.Source: §6 · p. 15

Margins and sizes

  • Near-zero predictions require equivalence margins fixed in advance; detecting a 2-point difference with the stated assumptions requires about 530 people per condition, versus about 2,100 for a 1-point margin.The design can rule out large effects of doubt under required checking, but not small ones.Source: §6 · p. 15, Table 3 · p. 16

Forecasting from checks

  • Forecasting from checks requires a validated measurement model and out-of-sample scoring against prior-knowledge, practice-accuracy, caution, and simpler-retention alternatives.Varying practice spacing is needed to distinguish the memory model from simpler models of the same checks; even a successful forecast would not establish that task and assistant effects operate only through checking.Source: §6 · pp. 15, 17

7 Scope and implications

VLT is limited to controlled rule use with evidence supplied and correct, rule-specific explanations; it does not yet model broader transfer, learning without correct feedback, or everyday oversight. Its implications concern when assistance supports learning and what delayed unaided judgment can reveal.

Scope

  • VLT does not yet describe how people identify the applicable rule when unfamiliar claims have ambiguous or competing cues, or how rules transfer across topics.The account also excludes practice without correct feedback and everyday skill erosion under AI use.Source: §7 · p. 17

Boundary conditions

  • The account assumes people can make a check and that explanations after errors provide the knowledge needed; continued copying despite perceived errors would undermine its checking mechanism.People may also discount explanations that follow errors, making a fallible assistant teach less even when a task requires checking.Source: §7 · pp. 17–18

Implications

  • If the accuracy paradox holds, assistance that raises performance while present may not be the better teacher, and accuracy during use cannot stand in for what users learn.With feedback held constant, the paper asks whether retention depends on why people checked or only on whether they did.Source: §7 · p. 18

Declarations

The author reports using Claude to develop and prepare the manuscript and simulation materials, while retaining responsibility for the article. No data were collected or analysed, and simulation code and outputs are available.

Use of generative AI

  • Claude helped draft, critique, and revise the theory and manuscript, and drafted figures and simulation code under the author's direction.The author decided the content, revised and approved the text, checked references, and takes full responsibility.Source: Declarations · p. 18

Data and code

  • The simulation code and its outputs are available on the Open Science Framework.The author reports checking every number from the simulations against the code output.Source: Declarations · p. 18

Open practices statement

  • No data were collected or analysed for the article, which presents a theory and simulations of its formal model.No study was preregistered.Source: Declarations · p. 18

Supplementary Material S1. Simulation details

The supplement tests VLT’s parameter dependence and alternatives through simulations, not empirical data. Its results show how gain sizes depend on assumptions while strict-model predictions remain distinct from alternatives in specified conditions.

Simulation design

  • Across 1,000 parameter sets, simulations used 300 people per condition, 24 practice claims across three rules, and a test 90 days later on new claims.The test included practised and unpractised rules; parameter ranges were assumptions chosen to keep delayed judgment between chance and ceiling.Source: Supplementary Material S1. Simulation… · p. 23

Sensitivity analysis

  • At ε = .3, median task, fallibility, and belief gains were 2.9, 1.7, and 1.1 points; at ε = .9, each was 0.2 points or less.Larger ε means explanations read without checking contribute more; discounts after errors can reduce or reverse the fallibility gain.Source: Supplementary Material S1. Simulation… · p. 23

Failed checks

  • With failed checks weighted .5 rather than 1, task, fallibility, and belief gains were 5.8, 3.5, and 2.3 points; B-X equalled E-X, while F-X was slightly lower.F-X fell below E-X by a median 0.1 points because wrong verdicts made checks fail more often.Source: Supplementary Material S1. Simulation… · p. 23

Alternative accounts

  • Under the depth or feedback-encoding account, B-X exceeded E-X as θ increased, whereas strict VLT predicts exactly zero for both B-X − E-X and F-X − E-X.A response-tendency alternative changed response-specific accuracy but slightly lowered balanced accuracy in the simulations.Source: Supplementary Material S1. Simulation… · p. 24

Extra checks

  • Within the sampled parameter ranges, an extra early check never lowered a rule’s strength at 90 days in the 1,000 simulated parameter sets.In a grid exceeding 120,000 combinations, reductions occurred only after at least 15 checks one second apart and were less than 10^-7.Source: Supplementary Material S1. Simulation… · p. 24

Supplementary Material S2. A proposed measurement model for checks

The proposed measurement model estimates whether each practice claim was checked from shared and task-specific indicators, then carries uncertainty in those checks into memory-model forecasts. It remains a proposed, unvalidated model.

Latent checks

  • The proposed model treats each practice-claim check as a latent binary variable and estimates its condition-specific prior probability.The supplement states that the measurement model has not been validated.Source: Supplementary Material S2. A proposed… · p. 24

Shared indicators

  • Shared indicators include opening or dwelling on rule-relevant evidence before the explanation and time spent on evidence after the verdict.The same indicator measurement parameters must hold across compared conditions, and the test should check this assumption.Source: Supplementary Material S2. A proposed… · p. 25

Task-specific indicators

  • For fixed-option flip tasks, answers can indicate an attempt at the claim’s rule, an attempt at another rule, or a guess; wrong-answer attempts still count as checks.With copyable tasks, overrides of wrong verdicts are noisy evidence and should be compared with the same person’s overrides of right verdicts.Source: Supplementary Material S2. A proposed… · pp. 25–26

Forecasting

  • The memory and measurement models are estimated together, and forecasts average across posterior draws of latent checks.For forecast-scored participants, checks are inferred from practice data without using test responses; indicator fit is checked by condition.Source: Supplementary Material S2. A proposed… · p. 26
Loading 2610.37875v1…