Source-linked AI summary
OASIS: A Rubric-Based Multimodal Assessment Platform Using Large Language Models
Ameer H. Shakur, Shinyoung Kang, David Hein, Michael J. Holcomb, Huong-Tra Ngo, Aarash Zakeri, Minhan Park, Licheng Yi, Dhanush Jain, Hunter Schuler, Andrew R. Jamieson
TL;DR
Large-scale rubric-based LLM assessment requires more than successful single-prompt scoring: it needs managed encounters, multimodal execution, provenance, and human review. OASIS provides an integrated architecture for these needs, and its production deployment at UT Southwestern has processed more than 7,000 encounters while preserving review and traceability.
Problem
Deploying rubric-based LLM assessment at scale requires encounter management, modality-aware execution, provenance, and human review beyond a single model call.
Method
OASIS compiles rubrics into typed multimodal grading workflows with content-addressed identities, progressive execution, explicit review states, and auditable artifacts.
Results
More than 7,000 encounters have been processed in production at UT Southwestern since Fall 2023.
Takeaways & Limitations
OASIS provides a shared, provider-neutral assessment workflow in which scores can be traced, reviewed, compared, and replayed across operating modes.
Takeaways & Limitations
OASIS is limited to post-hoc grading of recorded encounters, and outcomes remain dependent on provider capabilities and expert rubric design.
Abstract
from arXiv · showhide
OASIS (Open Assessment and Scoring Infrastructure Stack) is a systems platform for rubric-based grading of video, audio, and text with large language models. Scoring one artifact with an LLM is straightforward; deploying assessment at scale requires encounter management, rubric versioning, modality-aware execution, provenance capture, and human review. OASIS pairs a standalone command-line interface with a canonical integrated Elephant + MAPLES stack for encounter management and multimodal grading orchestration. Both paths can target hosted APIs or self-hosted open-weight models through Ollama and OpenAI-compatible endpoints such as vLLM. SimRubrics rubric authoring and the Wayfinder conversational agent gateway are optional extensions that use the same authenticated interfaces as human operators. Given a rubric and recorded encounters, OASIS produces per-criterion scores, evidence, and rationales, preserving execution artifacts for audit. Distinctive features include rubric-as-program compilation, progressive execution plans, content-addressable grading identity, transcript-augmented multimodal grading, explicit review state, and a shared command surface for humans and autonomous agents. Though developed in medical education, the architecture is domain-agnostic, applying wherever structured performance can be evaluated from recorded or written artifacts. In production at UT Southwestern Medical Center since Fall 2023, the platform has processed more than 7,000 encounters. This publication includes the report and project information, not application source, binaries, installation materials, sample data, or a tagged software release.
Motivation
Structured performance assessment is widespread but labor-intensive, costly at scale, and imperfectly reliable. OASIS frames deployment as a systems problem requiring managed encounters, rubric and model state, modality-aware execution, provenance, and human review.
- Motivation: Performance assessment workflows require evaluators to inspect recordings or written artifacts, consult rubrics, and assign scores.The abstraction spans medical exams, practicums, language tests, and simulation-based training.
- Motivation: Assessment cost grows linearly with encounters, while human inter-rater reliability is often modest.The passage identifies both scaling cost and imperfect reliability as central workflow concerns.
- Motivation: Single LLM scoring successes do not constitute a deployable assessment system.Deployment also requires encounter management, preserved rubric and model state, modality-aware routing, execution artifacts, and human review.
- Motivation: OASIS treats trustworthy LLM assessment as inspectable software with compiled rubrics, stable grading identities, and recorded review transitions.The system is designed for academic deployment without dependence on a proprietary end-to-end vendor.
- Motivation: The standalone workflow progresses from environment checks to a no-spend plan and one-item sample before larger runs.The public publication does not include the binary, installable source, or internal synthetic starter pack.
Related Work
Prior work addresses text-only scoring, proprietary interview-video assessment, and LLM judging, but leaves a gap for an integrated, rubric-native multimodal assessment system. OASIS positions itself as that architecture, combining data management, execution, provenance, and review.
- Related Work: Existing work spans text-only essay and short-answer scoring, proprietary interview-video products, and medical-education studies of LLM scoring.The cited categories differ in modality, openness, and workflow generality.
- Related Work: LLM-as-judge research documents position, verbosity, and self-preference biases, reinforcing the need for structured review and provenance.OASIS scores human performance artifacts against evaluator-defined rubrics rather than benchmarking models on fixed task suites.
- Related Work: General-purpose APIs and workflow orchestrators provide prompting or scheduling but lack assessment-specific rubric compilation, modality routing, provenance, and evaluator override.The missing layer connects rubric-aware execution with encounter and artifact management.
- Related Work: The identified gap is a rubric-native multimodal system integrating data management, execution, provenance, and review in one architecture.OASIS is intended to fill this gap.
Contributions
OASIS contributes an end-to-end, provider-neutral assessment architecture spanning standalone, institutional, and agent-facing operation. Its core design compiles rubrics into typed multimodal workflows and supports progressive execution, content-addressed identity, review, and comparison.
- Contributions: OASIS provides an end-to-end architecture spanning standalone CLI use, institutional deployment, and agent-facing operation.The architecture uses contract-enforced component boundaries across an umbrella workspace and service submodules.
- Contributions: Spreadsheet rubrics compile into modality-aware prompts and typed result schemas rather than remaining informal prompt text.This makes rubric structure part of runtime execution and persisted result validation.
- Contributions: Staged multimodal execution combines progressive, cost-visible plans and transcript-augmented grading with content-addressable grading identity.The design supports cache reuse, cross-model comparison, and distributed execution.
- Contributions: A shared command-service layer exposes CLI, terminal UI, and MCP tools, while the institutional tier adds a typed platform gateway and Wayfinder assistant.These surfaces are designed for agent-native operation through the same workflow.
- Contributions: More than 7,000 production encounters since Fall 2023 provide supporting operational evidence for the architecture’s design choices.The report presents this experience as evidence rather than as a separate contribution.
Evolution and Design Philosophy
OASIS evolved from a clinical grading pipeline into a reusable platform, agent-native operating surface, and accessible evaluation tool. Across these stages, shared contracts preserve the same workflow for human and autonomous users while lowering adoption barriers.
- Evolution and Design Philosophy: The original MAPLES pipeline established rubric-based LLM grading of clinical performance artifacts at operational scale.The platform layer then added Elephant for encounter and file management, SimRubrics for rubric quality assurance, and shared service contracts.
- Evolution and Design Philosophy: The agent-native layer exposed the workflow through the CLI, terminal UI, and MCP surfaces so assistants could plan, inspect, and recover grading runs.A platform gateway and Wayfinder later extended the same property to the institutional web interface.
- Evolution and Design Philosophy: Accessibility features include a standalone binary, guided workflow, reproducibility bundle, and synthetic starter pack for evaluation without institutional infrastructure.These features form the fourth evolutionary layer described in the passage.
Software Design
OASIS combines assessment-specific contracts, staged execution, and reviewable evidence into an inspectable software system spanning standalone and institutional workflows. Its architecture supports multimodal grading, provider flexibility, provenance, and shared human-agent operation.
- OASIS combines assessment-specific contracts, staged execution, and reviewable evidence rather than relying on a single prompt or service boundary.
- Rubric-as-program compilation converts spreadsheet criteria into modality-aware prompts and typed result schemas for downstream grading.
- The platform pairs a standalone CLI with the integrated Elephant + MAPLES stack, while SimRubrics and the platform agent gateway remain optional extensions.
- Elephant manages encounter metadata and files, MAPLES orchestrates parallel multimodal grading, and the CLI supports local-file execution without server infrastructure.
- The CLI, terminal UI, and MCP tools share one command-service layer, while the institutional gateway exposes authenticated MAPLES workflows to platform agents.
- Deployment locality applies to the complete processing chain, so a self-hosted primary model with hosted downstream processing is reported as hybrid.
Progressive Execution and Execution Plans
OASIS stages grading through validation, planning, sampling, and scaling, externalizing execution state so humans and agents can inspect, resume, or abandon workflows. The shared command layer extends these controls across CLI, terminal UI, and MCP surfaces.
- OASIS moves grading through initialize, setup, dry-run, single, and scale checkpoints before full execution.
- The workflow validates credentials, data, rubrics, mappings, and modalities, then estimates work and surfaces warnings before expensive calls.
- A sample run lets operators inspect quality before scaling to a full cohort or distributed batch execution.
- Execution-plan.v2 artifacts store immutable step IDs, fingerprinted inputs, waiver metadata, and an explicit lifecycle for inspection, annotation, resumption, forking, verification, or abandonment.
- The canonical plan graph runs from doctor and validation through manifest, rubric, mapping, context, storage, planning, confirmation, sampling, grading, verification, review, and export.
- CLI, terminal UI, and MCP server share the same CommandService layer, inheriting common dispatch, envelopes, compound operations, and error semantics.
- MCP tools carry planning metadata and compound operations, making the agent surface operationally aware of read-only discovery and irreversible actions.
- SimRubrics uses multi-model critique, evaluator-controlled revisions, immutable rubric snapshots, and sample grading to identify brittle criteria before institutional execution.
MAPLES: Pipeline Lifecycle
MAPLES turns encounter groups and rubrics into scored, typed, reviewable outputs through modality-aware compilation and staged execution. Its default grading path separates free-form media analysis from typed parsing, caching both stages independently.
- MAPLES turns a rubric and encounter collection into scored outputs through a six-stage execution lifecycle.
- Rubric ingestion and compilation: Rubrics enter as Excel workbooks or compiled YAML, then rubric_zipper validates structure, groups items, and generates modality-specific prompt batches.V2 YAML separates item data from prompt text and composes prompts at runtime.
- Modality routing: Provider capabilities determine whether grading uses native video, native audio, or transcript/text paths, with early validation of structurally invalid workflows.Audio can route through a cached transcript so text-only endpoints can participate in audio-derived grading.
- Typed results: Rubric compilation dynamically creates typed response schemas whose fields correspond to modality-specific item models and are reused for validation and postprocessing.
- Two-stage grading: The default two-stage path first obtains a free-form LLM response, then parses evidence, rationale, and scores into typed results linked to the encounter, run, and rubric version.A provisional result remains inspectable until an evaluator accepts or overrides it.
- Two-stage grading: Both grading stages are independently cached, allowing parsing retries from recorded analysis without repeating expensive media analysis.
Multimodal Depth: Transcript-Augmented Audio and Segment Decomposition
OASIS separates perception from assessment by caching transcripts and using focused or inferred media segments before grading. This supports lower-context multimodal execution while preserving content-addressed identity, review, and provenance across runs.
- Transcript augmentation: Timestamped transcripts for audio and video are cached by file hash so later grading can reuse perception artifacts across models or machines.Text extraction is automatic and deterministic, while audio and video use transcript extraction before grading.
- Focused media access: Focused time_hint windows can reduce transcript-grading token use to roughly 1/45th of full-recording context for phase-specific rubric items.
- Segment decomposition: Deep execution infers likely time windows, grades segments independently, and synthesizes cross-segment evidence; one internal validation used roughly 1/40th of full-context tokens.
- Grading identity: Content-addressed input and output keys distinguish the graded content from the exact model-specific result.The input key uses rubric, media, prompt, and optional time-hint hashes; the output key additionally includes model and provider.
- Grading identity: Stable input identity enables deduplication, exact rerun cache hits, cross-model comparison, and distributed work reuse across machines.A second model receives a new output key while preserving the same input key.
- Review and provenance: Review converts provisional machine output into reportable data through evaluator acceptance or override, with review history retained in the provenance chain.
- Review and provenance: The recorded lineage spans rubric version, prompt batch, model output, evidence, and review decision, enabling post-hoc inspection and troubleshooting.
Availability
The report describes evaluated standalone, integrated, and optional agent-native configurations, but the public publication does not include installable software or release artifacts. Deployment boundaries include provider-dependent data residency, institutional governance, and mandatory human review.
- Publication availability: The public publication includes project information and a technical report, but excludes application source code, binaries, installation materials, sample data, and a tagged release.
- Evaluated configurations: OASIS was evaluated in standalone CLI, integrated Elephant–MAPLES, and optional agent-native configurations serving local evaluation, institutional workflows, and assistant operation.
- Evaluated configurations: The standalone profile ran on a laptop, the integrated profile used one Docker host, and shared or hosted operation was not the reference configuration.
- Operational boundaries: Production throughput is primarily bounded by provider rate limits, while reported cost and elapsed time are illustrative and provider-dependent.
- Data residency: Local transcription, grading, schema conversion, and postprocessing can keep encounter content local, whereas hosted downstream stages receive their configured inputs.Native video and hosted-audio grading transmit media to the configured provider.
- Responsible use: OASIS is an assessment-support system: machine scores remain provisional until a qualified evaluator accepts or overrides them.Institutions retain responsibility for privacy, consent, retention, and assessment governance.
Operational Experience
OASIS has operated at UT Southwestern since Fall 2023, grading more than 7,000 encounters with high observed agreement and substantial reductions in manual grading effort. These figures come from production observations and a companion multimodal deployment rather than controlled universal benchmarks.
- Since Fall 2023, OASIS has graded clinical skills exams across multiple programs at UT Southwestern Medical Center.
- More than 7,000 encounters and over 3,000 learners have been covered in production.
- 93–96% item-level agreement with human graders and 0.830 AI–human Cohen’s κ were observed in production and a recent multimodal deployment, respectively.Human–human inter-rater κ in the same setting was 0.732.
- Estimated manual grading effort decreased by 95–97% under review-first adjudication using bulk acceptance and targeted overrides.The estimate reflects equivalent cohorts rather than a controlled measurement.
- The platform has also supported end-to-end reproduction studies and model, station, modality, and encounter-subset comparisons through declared specifications and run artifacts.
Limitations
OASIS remains constrained by provider and modality differences, assessment-design quality, and runtime factors that limit exact reproducibility. Its outputs require carefully designed rubrics and bound execution environments for meaningful comparison or replay.
- Provider dependence and scope: Multimodal capability remains dependent on the model, runtime, and media presentation, with native video currently best served by Google’s Gemini.Extracted frames are not equivalent to native video, and native-audio support varies with architecture, runtime, context limits, and hardware.
- Assessment-design dependence: OASIS faithfully executes its rubric, so ambiguous criteria, overlapping levels, weak constructs, and prompt wording can produce unstable or shifted scores.Expert rubric design remains necessary even when SimRubrics makes refinement explicit.
- Reproducibility boundaries: OASIS records rubric, prompt, provider, output, and review details but cannot prevent hosted providers from changing behavior behind stable API names.Open-weight operation improves artifact retention, yet tokenizer, processor, runtime, quantization, generation configuration, and media presentation also affect results.
Conclusion
OASIS treats scalable rubric-based LLM assessment as an auditable systems problem rather than a sequence of isolated model calls. Its architecture supports local and institutional deployment, shared human and agent workflows, and traceable production use.
- OASIS compiles rubrics into specifications, executes grading progressively with visible costs, assigns content-addressed identities, and records human review and provenance.
- The same workflow spans a single binary and a containerized Elephant + MAPLES stack, while hosted and self-hosted models share rubric, review, comparison, and provenance structures.
- Agents use the same authenticated interfaces and review gates as human operators, preserving shared identity, permissions, and accountability.
- Nearly three years of production use at UT Southwestern and more than 7,000 encounters ground the platform’s contracts, gates, and provenance mechanisms in operational experience.