PaperlayerView exact revision on arXiv

Source-linked AI summary

Comparing GenAI to Machine Learning and Manual Review for eDiscovery

Gordon V. Cormack, Maura R. Grossman, Jeremy Pickens

arXiv:2610.16925v1cs.LGcs.AIcs.IR

TL;DR

The paper examines why Redgrave's reported aiR and RAL recall results are not directly comparable, given differing review objectives, assessor judgments, and trade-offs. It compares them under matched filtered and unfiltered tasks and evaluates CAL and S-CAL; the analysis finds aiR does not outperform RAL at common cutoffs, while S-CAL performs strongly with a 2,000-document review budget.

  • Problem

    The reported aiR and RAL results are not directly comparable because review objectives, assessor conceptions of responsiveness, and recall–precision trade-offs differ.

    Source: §1 · pp. 1–2

  • Method

    The analysis compares filtered and unfiltered TAR gain curves and recall–precision estimates for aiR, RAL, CAL, and S-CAL using SME coding and a weighted sample for S-CAL.

    Source: §3 · p. 2

  • Results

    At common cutoffs, filtered-task RAL achieves 64% recall and aiR 59.5%, while unfiltered-task aiR achieves 88% and RAL 97.2%; S-CAL substantially matches aiR points with a fixed budget of 2,000 documents.

    Source: §7 · pp. 5–6

  • Takeaways & Limitations

    S-CAL dominates RAL, substantially matches aiR points, exceeds 90% recall, and eventually reaches 100% recall, which aiR never reaches.

    Source: §7 · p. 6

    The Redgrave Case Study covers one matter, one L1 review, and one prompt-engineering effort, using a small non-random subset that may not represent the larger or other collections.

    Source: §6 · p. 5

Abstract

from arXiv · show

Keeling et al. [Keeling2026Generative] report on a case study recently conducted by Redgrave LLP in which they assert that Relativity's Generative AI Review tool (aiR) achieved 88% recall while Relativity Assisted Learning (RAL), its machine-learning tool, achieved 64% recall. These results are not comparable because the methods employed have different objectives, because of inter-assessor disagreement, and because they represent different recall-precision and cost-benefit trade-offs. Our analysis of the Redgrave data indicates that, when these differences are taken into account, aiR and manual review are comparable. Furthermore, established machine-learning models employing Continuous Active Learning (CAL) show state-of-the-art effectiveness, and superior efficiency, when trained only on a sample of 2,000 first-pass-reviewer responsiveness determinations.

1 Introduction

The reported aiR–RAL recall comparison is not like-for-like because the methods use different review objectives, assessors, and recall–precision trade-offs. The analysis finds their effectiveness differences essentially vanish under matched filtered or unfiltered tasks, while CAL and S-CAL yield equal or better efficiency and effectiveness.

1 Introduction

  • The reported results are incomparable: RAL production required junior-lawyer review, while aiR production did not; assessors differed in responsiveness judgments, and the methods represented different recall–precision trade-offs.Choosing a trade-off involves balancing completeness, cost, burden, risk, and timeliness.Source: §1 · pp. 1–2
  • When the L1 team decides responsiveness or all system-identified documents count as responsive, aiR and RAL effectiveness differences essentially vanish.These are the paper's filtered and unfiltered TAR settings, respectively.Source: §1 · p. 2
  • Established CAL for filtered TAR and S-CAL for unfiltered TAR yield equal or better efficiency and effectiveness than the compared methods.The paper identifies CAL and S-CAL as decade-old supervised machine-learning protocols.Source: §1 · p. 2

2 Dataset

The Redgrave Case Study used 45,004 documents from the UCSF Opioid Industry Documents Archive and compared RAL and aiR review against an SME sample.

2 Dataset

  • 45,004 documents from the UCSF Opioid Industry Documents Archive formed the Redgrave Dataset.The SME independently reviewed a simple random sample of 1,000 documents and identified 73 as responsive.Source: §2 · p. 2
  • RAL reviewers coded 34,409 documents identified by RAL, then the remaining 10,595, producing L1 coding for the full dataset.The aiR workflow instead used GenAI prompts to label each document.Source: §2 · p. 2
  • Against SME coding, RAL achieved 64.4% recall and 39.2% precision, while aiR achieved 87.7% recall and 28.6% precision.The reported estimates include 95% confidence intervals for both measures.Source: §2 · p. 2

3 Filtered and Unfiltered TAR Comparisons

The comparisons evaluate filtered and unfiltered TAR against SME coding, using gain curves for aiR, RAL, and machine-learning methods. S-CAL uses a weighted sample of 2,000 L1 determinations.

3 Filtered and Unfiltered TAR Comparisons

  • CAL and S-CAL gain curves were compared with aiR and RAL against SME coding in filtered and unfiltered TAR, respectively.The implementations derive from the open-source BMI used for the TREC 2015 and 2016 Total Recall Tracks.Source: §3 · p. 2
  • Figure 1 compares responsive documents identified by aiR, RAL, and CAL after filtering by L1 responsiveness coding.The gain curves use SME coding as ground truth.Source: §3 · p. 2, Figure 1 · p. 3
  • Figure 2 compares aiR, RAL, CAL, and S-CAL without L1 responsiveness filtering; thin S-CAL curves show 20 trials and the thick curve their mean.S-CAL requires L1 coding for a weighted statistical sample of 2,000 documents, excluding the 1,000-document SME sample.Source: §3 · p. 2, Figure 2 · p. 3

4 Assessor Agreement and Disagreement

The analysis treats assessor disagreement as a material qualification on recall and precision comparisons, rather than assuming any assessor provides infallible ground truth. Within each task, continuous gain curves fall within the sampling margin of error of aiR results.

4 Assessor Agreement and Disagreement

  • L1 reviewers marked 34% of SME-responsive documents non-responsive, while the SME marked 61% of L1-responsive documents non-responsive.Recall and precision therefore reflect a particular assessor's judgments, not an infallible ground truth.Source: §4 · p. 4
  • Overlap, or Jaccard index, is the shared responsive-document count divided by the count either assessor labels responsive.Table 1 compares SME coding with L1, S-CAL, and liberal and strict aiR interpretations; Table 2 reports literature values for intra- and inter-assessor overlap.Source: §4 · p. 4, Table 1 · p. 4, Table 2 · p. 4
  • Reported reviewer disagreement ranges from 0.158 to 0.350 between professionals and from 0.602 to 0.806 across the same SME's assessments at different times.Such disagreement does not imply either review is unreasonable or establish an infallible ground truth.Source: §4 · p. 4
  • Within each task, continuous gain curves come within the sampling margin of error of discrete aiR results, and remaining differences have little practical significance.The remaining differences may reflect chance, training, or method.Source: §4 · pp. 4–5

5 Proportionality and the Recall-Precision Trade-off

Proportionality weighs the burden of reviewing or producing more documents against their value. Gain curves can inform whether continued review is worthwhile, but aiR offers no principled way to choose between its two viable cut points.

  • Proportionality balances the burden of reviewing or producing more documents against their value.Source: §5 · p. 5
  • For RAL and CAL, the incremental frequency of finding responsive documents estimates the gain-curve slope and helps assess whether to proceed.Source: §5 · p. 5
  • S-CAL’s estimated full gain curve can identify diminishing returns, while aiR offers only two viable cut points with no principled selection method.Source: §5 · p. 5

6 Limitations

The case study covers one matter, one L1 review, and one prompt-engineering effort. Its dataset and setting also constrain how broadly its findings may apply, and aiR pricing prevents evaluating cost in the proportionality analysis.

  • The study considers only a single matter, a single L1 review, and a single prompt-engineering effort.Source: §6 · p. 5
  • The Redgrave Dataset is a non-random subset of a 1.4M-document collection, with 7.3% estimated richness that may not represent the larger collection or others.Source: §6 · p. 5
  • Results from the heavily publicized opioids litigation may not apply to novel matters that are not equally publicized.Source: §6 · p. 5
  • Redgrave’s aiR prompts and the methodology used to develop them remain confidential.Source: §6 · p. 5
  • Opaque aiR subscription allowances and overage charges preclude evaluating its cost in the proportionality analysis.Source: §6 · p. 5

7 Discussion and Conclusions

For this task, aiR and RAL are comparable under either final-decision rule, while CAL and S-CAL offer competitive effectiveness with different review-effort and calibration trade-offs. Scaling analysis also contrasts review effort with S-CAL’s low computational cost and aiR’s unknown large-scale run cost.

  • At filtered cutoffs, RAL reaches 64% recall versus aiR’s 59.5%; unfiltered, aiR reaches 88% versus RAL’s 97.2%.Source: §7 · p. 5
  • Unfiltered RAL or CAL would most likely be disproportionate because of review cost; with 2,000 reviewed documents, S-CAL dominates RAL and substantially matches both aiR points.Source: §7 · pp. 5–6
  • Like RAL, S-CAL exceeds 90% recall and eventually reaches 100%, which aiR never reaches.Source: §7 · p. 6
  • At its self-calibrated recall-precision crossover, S-CAL agrees more closely with the SME than its L1 teacher, and self-calibration can select other gain-curve points.Source: §7 · p. 6
  • If richness is similar, review effort for CAL and RAL could increase by an order of magnitude on the full collection; adding mostly non-responsive documents would have little effect.Source: §7 · p. 6
  • S-CAL runs on commodity hardware in 8 seconds elapsed time and 32 seconds CPU time after vectorization, which takes less than one hour; aiR’s 1.4M-document run cost is unknown.Source: §7 · p. 6
  • For this task, aiR is comparable to RAL under either final-decision rule; CAL matches RAL’s recall with less review, while S-CAL matches aiR using 2,000 L1 determinations.Source: §7 · p. 6

Related papers

Loading 2610.16925v1…