PaperlayerView exact revision on arXiv

Source-linked AI summary

LitUbik: open-source software for human-led systematic reviews with traceable AI assistance

Adolphe J. Bequet, Sebastien Laurent, Baptiste Bouvier

arXiv:2610.37181v1cs.DL

TL;DR

Systematic reviews face fragmented workflows, rising reporting requirements and concerns about the reproducibility and oversight of AI assistance. LitUbik brings the review lifecycle into a self-hosted application with human-validated AI proposals, and the authors use a review of 31 tool papers to position the software and illustrate its use.

  • Problem

    Review teams often stitch separate tools together, while reporting expectations rise and LLM screening results can vary across reviews, prompts and decision rules.

    Source: §1 · p. 2

  • Method

    LitUbik is a self-hosted review application in which named reviewers validate optional AI proposals and an audit journal records calls for disclosure.

    Source: §1 · p. 3, §4 · p. 12

  • Results

    The authors report that, to their knowledge, LitUbik is the only open-source application covering the full review lifecycle in one self-hosted system, and they position it through a review of 31 tool papers.

    Source: §5 · p. 13

  • Takeaways & Limitations

    LitUbik supports collaboration on a shared server or across disconnected instances while retaining the same guarantees, according to the authors.

    Source: §5 · p. 13

    LitUbik lacks local active-learning prioritisation and recall estimation; SQLite suits small teams, while PostgreSQL remains future work.

    Source: §4 · p. 13

Abstract

from arXiv · show

LitUbik is an open-source, self-hosted tool for conducting systematic reviews, from the registered protocol to the PRISMA 2020 diagram. LitUbik covers calibrated search equations, automated searching, deduplication, blinded multi-reviewer screening, full-text retrieval, snowballing, appraisal, extraction and reporting, on one server or across disconnected instances. At each stage, optional AI assistance from commercial or local language models offers proposals that a named reviewer accepts or rejects. Every AI call is logged, so the AI-use disclosure is written from a record. A systematic review of 31 papers on review tools, conducted in LitUbik, compares it with existing tools and illustrates its use. LitUbik is distributed under AGPLv3.

1. Motivation and significance

Systematic reviews face rising reporting and reproducibility demands, while teams often stitch separate tools together and AI-assisted review raises oversight and reproducibility concerns. LitUbik addresses these needs in one self-hosted application with human-validated AI, and the paper positions it through a review of review-support tools.

Motivation and significance

  • Review teams often stitch separate tools together, and a changed decision can silently invalidate manuscript PRISMA counts.Reporting also suffers when search dates are omitted or counts of papers and studies are mixed.Source: §1 · p. 2
  • LLM screening performance varies across reviews, prompts and decision rules, and nominally identical runs disagreed on 94 of 1,131 records, including 29 eligible records.The cited discussion supports supervised assistance rather than autonomous review, alongside human oversight and disclosure.Source: §1 · p. 2
  • The paper's review covers 31 tool papers from 2013–2026 and compares 14 multi-stage workbenches, 14 single-stage or generative tools, and commercial discovery assistants.Table 1 condenses the comparison for the ten most comparable workbenches.Source: §1 · p. 2
  • The reviewed tools show gaps in search calibration, post-screening automation, blinded dual screening with live agreement statistics, logged AI use, and scalable collaboration.None of the compared tools calibrates equations against must-find references; only one of ten automatically retrieves full texts, and none logs its AI calls for disclosure.Source: §1 · pp. 2–3
  • LitUbik lacks hosted production service, mobile app, machine-learning prioritisation, an RCT classifier, and a published user evaluation.The review also found no accessibility-conformance documentation among the ten tools, although library guidance asks institutions to check it.Source: §1 · p. 3
  • LitUbik combines review stages in one self-hosted application, with named reviewers validating AI proposals and an audit log recording billed calls.It supports multiple question frameworks and commercial or local models, while enforcing double blinding and computing Cohen's and Fleiss' κ.Source: §1 · p. 3

2. Software description

LitUbik runs as a client–server application that can serve a team from one machine, with methodological rules enforced in its API. Its twelve-stage review workflow combines configurable review functions with optional, reviewer-validated AI assistance and supports collaboration across shared or disconnected instances.

2.1. Software architecture

  • A client–server application lets one researcher's machine serve a team through a local-network port or Docker deployment.Deployment options include a packaged Windows executable, a one-click source launcher and Docker Compose.Source: §2.1 · p. 5, Figure 1 · p. 5
  • The API enforces methodological rules, while an in-process job runner handles long operations and streams progress to the browser.The LLM layer supports commercial providers and OpenAI-compatible local servers, with configurable tasks, editable prompts and a call journal.Source: §2.1 · p. 5, Figure 1 · p. 5

2.2. Software functionalities

  • The twelve review stages map onto PRISMA 2020 phases, with optional LLM proposals validated by named reviewers and logged with model, author and cost.Snowballing and living-review rounds can re-enter at deduplication and repeat stages 5 to 7.Source: §2.2 · p. 5, Figure 2 · p. 6
  • A registered protocol is structured with reviewer-validated AI proposals, then its stable criterion codes frame later prompts and exclusion reasons.Search variants are translated for 19 databases, checked against must-find references for recall, and frozen with a timestamp when chosen.Source: §2.2 · p. 6
  • Searches run through open APIs or assisted-manual imports, while deduplication uses exact identifiers, normalized titles and guarded fuzzy matching with arbitration for borderline pairs.Living-review passes can rerun frozen equations on a schedule where databases allow.Source: §2.2 · pp. 6–7
  • Screening supports blinded reviewer allocation, consensus or arbitration, continuous κ statistics, criterion-based AI assessments and human-checked title-triage proposals.Full-text retrieval uses a cascade of open-access and institutional routes; exclusions require coded reasons, and documents over 45 pages are not sent to a model.Source: §2.2 · pp. 7–8
  • Snowballing combines backward and forward citation harvesting, deduplicates each iteration against the corpus, and traces retained records to their seeds.Appraisal supports JBI, MMAT or declared instruments; extraction includes typed grids and validation of AI-prefilled cells, while synthesis records SWiM and GRADE items.Source: §2.2 · p. 9, Figure 4 · p. 9
  • Reporting derives PRISMA counts from live records and drafts Methods and AI-use disclosures from the audit trail and call journal.Disconnected instances exchange snapshots and contribution packages; imports preserve row-level provenance and recompute conflicts, κ and PRISMA.Source: §2.2 · pp. 9–10, Figure 5 · p. 10

3. Illustrative example

The authors used LitUbik to conduct a two-round review of review-support tools, yielding 31 included studies, 13 found through snowballing, and a recorded account of AI use.

3. Illustrative example

  • 5,148 records were retrieved and deduplicated to 2,156 unique records after reviewers arbitrated 43 borderline pairs.The frozen search equation ran on four APIs and was also run manually on three databases.Source: §3 · pp. 10–11
  • Both reviewers screened 393 records blind after an administrator validated 1,763 AI-proposed exclusions; title-and-abstract agreement was 89.8% (κ = 0.63).They resolved 57 conflicts, and 38 records went to full-text screening.Source: §3 · p. 11
  • Of 62 reports sought, 38 were retrieved automatically and 20 uploaded manually; ten were excluded as inaccessible full texts.Four could not be obtained, and six retrieved files were wrong or corrupt.Source: §3 · p. 11
  • Snowballing added 13 studies to the 18 included from round one, bringing the review to 31 studies.The first iteration returned 2,104 candidates; 2,001 new records entered round two, and a second iteration to test saturation is planned.Source: §3 · p. 11
  • An LLM prefilled a 36-column extraction grid that reviewers checked and used to support Table 1.The counts reported in the review come from the v5 project archive, which can be imported to reproduce them.Source: §3 · pp. 10–11
  • The AI-use journal recorded 302 calls across nine tasks, and all 67 title-and-abstract “exclude” suggestions matched the final decision.Of 45 full-text suggestions, 38 (84%) matched; because suggestions were visible before voting, these matches do not establish independent agreement.Source: §3 · p. 11
  • 31 studies were included, 13 found only through snowballing, as shown in the PRISMA flow diagram.The review comprised two identification streams and coded exclusion reasons.Source: §3 · p. 11, Figure 6 · p. 12

4. Impact

LitUbik links human-controlled AI assistance, review records, collaboration, and AI-use data in one review workflow. Its current boundaries include missing prioritisation and recall estimation, and a SQLite backend tested only for small teams.

AI assistance under human control

  • At each stage, a named reviewer accepts, edits, or rejects AI proposals, while logged calls support disclosure and local providers keep embargoed material on the team’s machines.Rounds can hide suggestions until after the reviewer votes.Source: §4 · p. 12

One consistent record

  • A changed decision updates every statistic, figure, and draft because LitUbik records equations, database runs, imports, and deduplication alongside screening decisions.Its golden-set mechanism makes search calibration documented.Source: §4 · p. 12

Collaboration without shared infrastructure

  • Teams can collaborate on one server with server-enforced blinding and live agreement statistics, or exchange complete review state and contributions across disconnected instances.Merging recomputes conflicts, κ, and PRISMA under project rules.Source: §4 · pp. 12–13

A dataset on AI assistance

  • Pairing human decisions with model assessments on the same records under the same protocol creates data for evaluating LLM screening in real conditions.Section 3 provides a first such dataset and allows comparison of declared AI use with recorded activity.Source: §4 · p. 13

Evaluation and scope

  • Four beta testers outside the development team have used LitUbik and reported on its usability.The authors identify a larger-team review and a usability study using the System Usability Scale as future work.Source: §4 · p. 13

Limitations

  • LitUbik lacks active-learning prioritisation and recall estimation; SQLite is the only tested database, and PostgreSQL remains future work.The authors state SQLite suits small teams; the packaged Windows executable also omits browser-based and institutional retrieval routes.Source: §4 · p. 13

5. Conclusions

LitUbik brings human-led review work and AI assistance together in a self-hosted system, with collaboration supported on shared servers or disconnected instances. The authors position it through a 31-paper review conducted in the tool and identify further evaluation as future work.

5. Conclusions

  • LitUbik is open-source software for human-conducted reviews with AI assistance under human validation and full provenance.The authors state that, to their knowledge, it is the only open-source application covering the full review lifecycle in one self-hosted system.Source: §5 · p. 13
  • Collaboration is supported with the same guarantees on a shared server or across disconnected instances.The paper illustrates LitUbik through a systematic review of 31 tool papers conducted in the application.Source: §5 · p. 13
  • Future work includes recall estimation, a larger-team real-world review, and a usability study using the System Usability Scale.Source: §5 · p. 13

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

The authors disclose using OpenAI models through LitUbik for review tasks and Anthropic Claude through Claude Code for software and manuscript work, with human review of outputs. The declaration also reports logged AI use and its scope and cost.

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

  • OpenAI GPT models supported title triage, screening-decision suggestions, and search-concept drafting in the review; Claude supported software writing, extraction-grid prefilling, manuscript and diagram drafting, and citation checks.The authors reviewed and edited outputs as needed and take responsibility for the publication.Source: Declaration of generative AI and… · p. 13
  • The disclosure table records AI-use levels by activity, tools and versions, material shared with providers, and human validation.Its review-use entries came from LitUbik’s AI-use journal, except the extraction grid prefilled outside LitUbik; other activities were author-declared.Source: Declaration of generative AI and… · p. 14, Table A1 · p. 14
  • All 67 title/abstract “exclude” suggestions matched final decisions, 20 of 137 “maybe” suggestions ended included, and 38 of 45 full-text suggestions matched final decisions.The review had no control sample; that option was added later.Source: Declaration of generative AI and… · p. 14
  • 302 calls used 1.45 million input and 0.50 million output tokens, with estimated list-price cost of about USD 29.60.The journal showed USD 6.78 because two models were missing from its price table and were priced at gpt-5 rates. The review’s version did not count refusals or attempts failing before provider failover; version 1.0.0 does.Source: Declaration of generative AI and… · pp. 14–15

Data availability

The paper provides source code, a versioned review archive, and supplementary materials, while specifying what the archive excludes and what a replicator must retrieve independently.

Data availability

  • LitUbik 1.0.0 source code is available under AGPL-3.0-or-later, with versioned Zenodo archives and a public replication archive for the Section 3 review.The public demonstration opens an anonymous guest session on an example review.Source: Data availability · pp. 15–16
  • The replication archive retains metadata, decisions, reasons, extraction, and the AI-use journal, reproducing Section 3 counts, but excludes full texts, abstracts, raw database exports, API keys, and reviewer work sessions.Replicators must retrieve full texts using their own access.Source: Data availability · p. 15
  • Supplementary S2 provides the extraction grid behind Table 1, including the page and quote supporting each value.S1 and S2 are released under CC BY 4.0, except bibliographic metadata, which remains subject to database terms.Source: Data availability · p. 15

CRediT authorship contribution statement

The contribution statement assigns software and review responsibilities across three authors, including a second reviewer for the Section 3 review.

CRediT authorship contribution statement

Funding

LitUbik was developed to support the RESCOUSSE project’s WP1 systematic review.

Funding

  • Development of LitUbik supported the RESCOUSSE project (WP1), funded by the French Road Safety Delegation under grant no. 2201600314.Source: Funding · p. 16
Loading 2610.37181v1…