The LLENS Platform for On-Premises Large Language Model Agents in a Japanese University Hospital: Development and Evaluation Study

Introduction

• About 37% of physicians’ working time is spent on the electronic health record, while writing documents and entering orders take about one-fifth of clinical time in Japanese hospitals.Routine tasks such as writing summaries and registering data are identified as potential AI-assist targets.Source: Introduction · p. 2
• Because hospital information systems run inside closed networks and patient-derived data stay there, the model and its tools must operate in-network rather than rely on external APIs.The passage notes that Hokkaido University Hospital has the same constraint.Source: Introduction · p. 3
• Hospital work includes reading scanned forms, aggregating laboratory values, preparing templates, comparing materials, and saving reusable procedures, beyond searching and summarizing documents.A chat screen and search function alone cannot perform this range of work; the system must govern model inputs, tools, and outputs.Source: Introduction · p. 3
• The paper identifies no prior report of a general-purpose agent platform that reads materials, computes, and returns artifacts inside a closed hospital network.The cited prior reports describe small, purpose-specific models or agents fixed to one use, alongside a general-purpose in-hospital LLM report.Source: Introduction · p. 3
• The authors designed, implemented, and deployed LLENS inside Hokkaido University Hospital’s closed network, reporting platform design, model serving, document question answering, work-scenario tests, and demonstrations.The electronic health record connection was still being built, with user and work effects reserved for later evaluation.Source: Introduction · p. 3

Model and Serving

• Three open-weight models with native function calling were compared on Japanese medical benchmarks and system characteristics, then served on an 8-GPU host using SGLang, with EAGLE-3 and up to 16 concurrent requests.Benchmark runs used provider prompts, temperature 0, and one response per question; Kimi K2.6 received images while the other candidates received text. Serving tests used synthetic requests and measured decode speed, input-length effects, and GPU power.Source: Model and Serving · p. 4, Table 1 · p. 4

Architecture

• The platform ran inside the hospital’s closed network on one host, with Nucllei, a modified Open WebUI fork, the model, document service, and isolated sandbox forming the agent harness.Nucllei removed external-facing and embedding-based features and added skills, tools, subagents, and asynchronous execution; tools were assigned server-side by model preset.Source: Architecture · p. 5, Figure 1 · p. 6, Harness Components · p. 6

Reading Materials

• Attachments were passed in full as text when they fit, while images and scanned PDFs were read as images; hospital documents used a path-addressed tree with string search and no vector index.The collection contained 7,954 files across 6 domains; the same tree was mounted read-only in the sandbox for conditional extraction and counting. Skills supplied on-demand domain guidance and tools, with 21 skills available at submission.Source: Reading Materials · p. 8, Hospital Documents · p. 9, Skills · p. 9

Sandbox

• The sandbox isolated Python and shell execution, while the model could check its own generated figures and PDFs, delegate material reading to subagents, and use workspace-based memory and saved recipes.The sandbox could not access the outside or hospital network except through prepared APIs, and bundled packages supported administrative and analytical work. Microphone input used on-host transcription and responses used on-host speech synthesis; cloud speech recognition was not used.Source: Sandbox · p. 9, Self-Checking of Generated Output · p. 10, Subagents · p. 10

Access to Hospital Documents

• Interactive access was compared with no documents and fixed retrieval on 180 questions across 6 domains, with each question run 3 times; fixed retrieval supplied the top 10 BM25 chunks, while interactive access supplied document tools, sandbox, orientation, and 7 skills.Accuracy was automatically matched to reference answers; configuration differences used per-question means, bootstrap 95% confidence intervals, and sign-flip tests, with the interactive-versus-fixed comparison designated as primary.Source: Access to Hospital Documents · pp. 11–12

Automated Test of Component Triggering

• Thirteen scenarios were each tested four times, counting runs that triggered expected components rather than answer correctness; the platform moved to the hospital medical information network September 14, 2026, and has operated there since.Source: Automated Test of Component Triggering · p. 12, Operation · p. 13

Model and Serving Performance

• 908 tokens/s was aggregate output throughput at 16 concurrent requests with EAGLE-3; single-user generation was 119 tokens/s with thinking and 129 tokens/s without thinking.Time to first token was 0.26 s at 1,130 input tokens and 41.6 s at 245,891 input tokens.Source: Model and Serving Performance · p. 13
• 480/500 on the 2026 Japanese Medical Licensing Examination was Kimi K2.6’s score with images and thinking; it was selected for comparable or better candidate scores, image input, long context, and concurrent-request capacity.On matched text-only questions, the candidates scored 369 for Kimi K2.6 and 370 for GLM-5.1 and DeepSeek V3.2.Source: Model and Serving Performance · p. 13, Table 3
• Serving performance was assessed under the deployment configuration using aggregate throughput and time to first token for concurrent requests, plus single-user decode speed with EAGLE-3 on and off.The supplied passages identify these measures but do not provide their values.Source: Table 4 · p. 14, Model and Serving Performance · pp. 14–15

Interactive Access Compared With Fixed Retrieval

• Mean accuracy was 97.2% with interactive access, versus 81.3% with fixed retrieval and 10.7% with no documents; interactive access exceeded fixed retrieval by 15.9 percentage points (95% CI 10.9-21.3; P<.001).Source: Interactive Access Compared With Fixed… · p. 15
• 71.1 percentage points separated interactive access (87.8%) from fixed retrieval (16.7%) on filtering/counting questions, while lookup accuracy was 100% versus 96.3%.Fixed-retrieval chunks included at least one evidence path in 65.4% of filtering/counting responses, and none of the 21 responses with only some evidence paths was correct.Source: Interactive Access Compared With Fixed… · p. 15, Figure 2 · p. 16
• 97.0% accuracy with sandbox only and 94.6% with document tools only compared with 97.2% with both; on lookup questions, both took 13.3 seconds versus 21.6 seconds with sandbox only.On filtering/counting questions, sandbox only scored 91.1%, document tools only 74.4%, and both 87.8%; document-tools-only runs included six execution failures.Source: Interactive Access Compared With Fixed… · pp. 15–16

Scenarios and Components

• 12 of 13 scenarios triggered all expected components in all four runs; for falls-data aggregation and plotting, self-checking triggered in 3/4 runs and return in 2/4.The scenarios demonstrate combinations such as reading attached laboratory values, searching the document tree for criteria, processing falls data in the sandbox, and returning a figure.Source: Scenarios and Components · pp. 16–17, Table 5 · p. 17

Principal Findings

• On filtering/counting questions, interactive access in which the model chose what to read with document tools and sandbox scored 87.8%, versus 16.7% for fixed retrieval passing the top 10 chunks from one question-text search.Source: Principal Findings · p. 21

Comparison With Prior Work

• Unlike earlier single-use hospital systems, LLENS combines existing components into scenarios and lets users save and share recipes to broaden platform uses.The authors distinguish developer-written skills, which explain how to use the system, from user-saved recipes for routine work.Source: Comparison With Prior Work · pp. 21–22

Next Steps

• Direct EHR and departmental-system connections are planned; workplace evaluation will begin with routine-work input time and later examine behavior, care quality, and patient outcomes.Currently, users must export EHR data and attach it.Source: Next Steps · p. 22

Limitations

• The study did not evaluate actual clinical use, work effects, or patient outcomes, and document-access results used team-authored questions and untuned fixed retrieval.The authors also note that results may depend on the question-type mix and that comparisons were not made at an equal inference budget against retrievers with vector search or query rewriting.Source: Limitations · pp. 22–23

Conclusions

• LLENS is an operating, single-host platform in the hospital’s closed network that combines a trillion-parameter-class open-weight model with an agent harness.The authors report multistep work involving document reading, computation, artifact return, and filtering or counting across documents; direct patient-record access is planned.Source: Conclusions · p. 23

• One author founded and directs Technoplasm Inc., co-developed Nucllei, and holds company equity; the other authors report no conflicts.Source: Conflicts of Interest · p. 23

• The LLENS snapshot is published for reading but cannot run without hospital services and documents; Nucllei is published under terms based on the PolyForm Noncommercial License 1.0.0.Source: Data Availability · p. 24
• Evaluation questions and instructions are available in appendices and the snapshot, but the internal hospital document collection is unavailable outside the hospital, preventing end-to-end reproduction without access to it.Source: Data Availability · p. 24

• LLENS means Large Language Enhanced Nexus System, and LLM means large language model.Source: Abbreviations · p. 24
• EHR means electronic health record, GPU means graphics processing unit, and KV means key-value.Source: Abbreviations · p. 24
• BM25 means Best Matching 25, the Okapi BM25 lexical ranking function; SQL means Structured Query Language.Source: Abbreviations · pp. 24–25

Multimedia Appendix 1. Supplementary tables

• Appendix 1 contains supplementary tables on candidate-model benchmark scores, configuration-level question-answering results, fixed-retrieval evidence coverage, grading sensitivity, and third-party model evaluations.The tables cover benchmark scores, accuracy, wall time, tool calls, retrieval coverage, grading correction, and external tool-use and long-context evaluations.Source: Multimedia Appendix · p. 25

Multimedia Appendix 2

• Appendix 2 provides the 180 evaluation questions, reference answers, accepted variants, and evidence paths in Japanese.The spreadsheet uses English column headers and labels.Source: Multimedia Appendix · p. 25

Multimedia Appendix 3

• Appendix 3 supplies the prompts and evaluation skills used in the evaluation.Source: Multimedia Appendix · p. 25

Multimedia Appendix 4

• Appendix 4 documents the 13 automated component-triggering scenarios, including request texts, preconditions, detection rules, and run counts by scenario and component.Source: Multimedia Appendix · p. 25

Table S1

• Table S1 reports Japanese medical-examination and IgakuQA benchmarks, with all-question inputs unmatched because only Kimi K2.6 received images; text-only columns provide the matched comparison.Its cells report score and number correct; the 2025 values were rescored.Source: Table S1 · p. 30

Table S2

• Table S2 defines accuracy, wall-time, and tool-call measures across five configurations and notes that all question-type analyses are exploratory.Execution failures count as incorrect; question-type contrasts use pointwise bootstrap intervals and paired sign-flip tests, with Holm adjustment within each question-type and metric family.Source: Table S2 · p. 31

Table S3

• Table S3 groups fixed-retrieval responses by question type and defines evidence-path retrieval by a matching or ancestor chunk path.The retrieval baseline is the top 10 chunks from one BM25 search; unavailable-source and some whole-domain filtering/counting questions have no evidence paths.Source: Table S3 · p. 32

Table S4

• Table S4 rechecks all 2,700 responses, identifies 18 grading errors, and cautions that the recheck is not exhaustive.Corrected verdicts flip the identified errors while leaving other verdicts unchanged; the supplied analysis notes eight false negatives and ten false positives.Source: Table S4 · p. 33, Multimedia Appendix 1. Supplementary… · p. 33

Table S5

• Table S5 compiles third-party tool-use and long-context evaluations, whose values were measured through common-protocol APIs rather than the study’s deployment configuration.The table was accessed October 23, 2026; operator-specific protocols and some evaluation-specific conditions are noted.Source: Table S5 · p. 35

Answer instructions

• The prompt requires concise, accurate Japanese answers about hospital documents and checking the specified edition, date, and scope.Source: Answer instructions (common system… · p. 37
• It limits evidence to materials actually consulted and says to report when information cannot be confirmed or is absent from the collection.Source: Answer instructions (common system… · p. 37
• It treats instructions found inside documents as untrusted and requires a short answer and consulted document paths on separate labeled lines.Source: Answer instructions (common system… · p. 37
• If no documents are provided, the prompt forbids guessing hospital-specific values and directs the assistant to say they cannot be confirmed.Source: Answer instructions (common system… · p. 37

Orientation text for the document collection

• Folders use index files for contents and scope, while child documents carry titles and domain-specific attributes defined by parent-node schemas.Source: Orientation text for the document… · pp. 37–38
• Logical document paths map to filesystem paths, and the prompt instructs users to verify extensions, index placement, edition, revision, and scope.Source: Orientation text for the document… · pp. 37–38
• The evaluation’s reference source is limited to the document collection; linked PDFs, spreadsheets, images, and external networks are not retrieval targets.The same source boundary applies whether documents are returned by tools or read in code.Source: Orientation text for the document… · pp. 37–38

Medical fee schedule (診療報酬)

• Check the target and edition of the fee schedule, facility standards, and official interpretations, and distinguish categories from individual procedure codes.Source: Medical fee schedule (診療報酬) · p. 38
• The collection uses hierarchical Markdown/YAML, with folder indexes for contents and scope and child-document front matter for titles and domain-specific attributes.Parent fields define child attribute schemas; logical paths map to filesystem paths, so verify actual extensions and index placement.Source: Medical fee schedule (診療報酬) · p. 38
• For numerical conditions and aggregation, verify the target set, exclusions, and duplicates while exploring the relevant scope.Source: Medical fee schedule (診療報酬) · p. 38
• Exploration is limited to the document collection; linked PDFs, CSVs, spreadsheets, images, and external networks are outside the evaluation’s retrieval scope.The same source boundary applies whether tools return documents or code reads them.Source: Medical fee schedule (診療報酬) · p. 38

DPC (DPC)

• Distinguish DPC classifications, target diseases, defined secondary diseases, and branching conditions.Source: DPC (DPC) · p. 39
• The collection uses hierarchical Markdown/YAML, with folder indexes for contents and scope and child-document front matter for titles and domain-specific attributes.Parent fields define child attribute schemas; logical paths map to filesystem paths, so verify actual extensions and index placement.Source: DPC (DPC) · p. 39
• For numerical conditions and aggregation, verify the target set, exclusions, and duplicates while exploring the relevant scope.Source: DPC (DPC) · p. 39
• Exploration is limited to the document collection; linked PDFs, CSVs, spreadsheets, images, and external networks are outside the evaluation’s retrieval scope.The same source boundary applies whether tools return documents or code reads them.Source: DPC (DPC) · p. 39

Neurological disease guidelines (神経疾患ガイドライン)

• Neurological guideline review should check the relationship between main texts and supplements or revisions, and each clinical question’s target and recommendation conditions.The documents are hierarchical Markdown/YAML, with folder indexes describing contents and scope and child documents carrying titles and domain-specific attributes.Source: Neurological disease guidelines… · pp. 39–40

Thoracic oncology guidelines and CTCAE (肺癌ガイドラインとCTCAE)

• For lung cancer guidelines, check histological type, stage, and applicability; for CTCAE, check the edition, adverse event, and grade.Parent-node fields define child-attribute schemas, and document versions, revisions, and scope should be checked.Source: Thoracic oncology guidelines and CTCAE… · p. 40

Patient safety (医療安全マニュアル)

• Patient-safety manual review should establish the target occupations and procedures, plus reporting conditions and exceptions.The stated exploration starting point is /院内マニュアル/医療安全管理マニュアル.Source: Patient safety (医療安全マニュアル) · p. 41

Infection control (感染対策)

• Infection-control review should identify relevant materials from the contents and check their targets and procedures.Numerical conditions or aggregations require checking the target set, exclusions, and duplicates.Source: Infection control (感染対策) · pp. 41–42

Information system FAQ (医療情報システムFAQ)

• The information-system FAQ covers hospital system operation and administration, which should be confirmed from the stored text.The exploration starting point is /医療情報システム.Source: Information system FAQ (医療情報システムFAQ) · p. 42

Procedure

• The automated test recorded tool calls to determine whether harness components were triggered, without judging answer correctness.Scenarios were run through the Nucllei API with preconditions restored before and after each test; trigger rules mapped specific tool calls to components such as document-tree access, sandbox use, artifact return, self-check, subagents, skills, recipes, and memory.Source: Procedure · p. 43

Request texts and preconditions

• The component-trigger table reports counts across four runs per scenario, marks expected components, and notes some unanticipated triggers.The test covered 13 work scenarios; the table also describes unexpected triggers in lab report transcription and procedure sharing and rerunning.Source: Request texts and preconditions · p. 47
• Both Self-check and Return were triggered in all 4 runs for research data analysis and lab report transcription and trend chart.These were the 2 of 3 scenarios expecting both components.Source: Request texts and preconditions · p. 49
• Falls data aggregation and plotting showed inconsistent figure handling: runs 1 and 2 inspected but did not return the figure, run 3 returned it without inspection, and run 4 did both.Writing a workspace path or image markdown in the answer did not deliver the figure to the user.Source: Request texts and preconditions · p. 49
• The falls request did not ask for figure inspection or handoff, whereas the other two requests asked for figure inspection and both actions occurred in all 4 runs.Source: Request texts and preconditions · p. 49