Ontology-Based Contextual AI Evaluations (OB-CAIE) Methodology

• The work was contributed to by Dr. Diego Garcia-Olano (Meta), Annie Brown (Reliabl), and Theodora Skeadas (Humane Intelligence).Source: §1 · p. 1

• The methodology addresses unclear testing coverage, the balance between human expertise and automation, and reproducibility of evaluation environments.Source: §3 · p. 2
• OB-CAIE represents an evaluation’s problem space with the Domain-Specific Ontology (DSO), the “what,” and the Evaluation Process Ontology (EPO), the “how.”The DSO gathers domain context and subject-matter expertise, while the EPO specifies the evaluation process.Source: §3 · p. 2
• Its problem space can support one or multiple evaluations and allows human judgment where feedback is irreducible or machines cannot replace it.Source: §3 · p. 2
• OB-CAIE defines what will be tested and lets failure points be traced, visualized, analyzed, and compared across evaluations, teams, and products.This supports auditing, mitigation, and retesting.Source: §3 · p. 3
• The methodology can mathematically measure part of the declared problem space, accommodate engineering and budget constraints, and tailor testing to salient failures or annotator expertise.It also aims to incorporate human expertise transparently, fairly, and non-extractively.Source: §3 · p. 3
• The authors present OB-CAIE as applicable beyond its initial development context, including research labs, evaluation organizations, auditing firms, and in-house teams.Source: §3 · p. 3

4.1 A Lack of Scientific Rigor and Transparency

• Unclear declarations of what is tested can undermine rigor; benchmarks may withhold prompts, while human red teaming may rely on undocumented or arbitrary definitions.Model cards may describe what was tested but lack detail on how, and their dense information can impede understanding limitations and guardrails.Source: §4.1 · p. 3
• Limited transparency about coverage can make performance claims exaggerated or inaccurate, such as claims about clinical care based mainly on electronic medical records.Limited, unrepresentative patient samples can also constrain such claims.Source: §4.1 · p. 3

4.2 Human Expertise and Automations

• Human-centered evaluation can be extractive and costly, while LLM-as-a-judge can miss context; both fully human and fully automated approaches have drawbacks.The paper identifies where and how human expertise should inform evaluation as a difficult question.Source: §4.2 · p. 4

4.3 Reproducibility of the Testing Environment

• Comparable, standardized coverage is difficult without rerunning the same prompts or tasks on the same system, and training-data exposure can affect results.Repeatedly designing evaluations with human labor can increase time and cost, while fully automated design risks missing context.Source: §4.3 · p. 4

5 OB-CAIE Methodology

• Experts scope the domain and deployment conditions, build the DSO, and declare users, operating context, and out-of-scope areas.Domain, evaluation, and organizational experts contribute; stakeholder-input ownership matters, and formal audits require separation between the testing and evaluation organizations.Source: §5 · pp. 4–5
• The evaluation operator writes requirements and acceptance criteria for an operational envelope; domain experts judge relevance and organizational experts sign off.Source: §5 · p. 5
• Experts derive realistic or target situations from the DSO and operational envelope and approve the test plan before execution.Source: §5 · p. 5
• The evaluation runs probes against the system as conversations and collects responses as evidence.The paper names Humane Intelligence’s red-teaming software or another testing platform as options.Source: §5 · p. 5
• For each probe response, evaluators record a supported pass, fail, or cannot-be-determined judgment against acceptance criteria, then aggregate judgments into an outcome.The domain expert attests to the final outcome and notes evidence sufficiency and test appropriateness.Source: §5 · p. 5
• A machine check determines coverage and performance within the declared envelope; domain experts note gaps, operators recommend actions, and the domain expert signs off on the report.The testing organization operationalizes recommendations, and organizational experts accept the report.Source: §5 · p. 5

5.1 Steps 1 and 2: Scope the domain and its deployment conditions, and declare the operational envelope

• The DSO supplies domain concepts while the EPO provides structure for experts to declare the operating environment and the acceptable range of system behavior.The operating environment records deployment conditions and assumptions; the operational envelope states what is and is not evaluated.Source: §5.1 · p. 5

5.1.1 The DSO

• The DSO encodes domain concepts and relationships as a graph; a context-specific knowledge graph supplies the reference against which probe responses are graded.Knowledge graphs conform to DSO rules while adding context-specific information, and can be machine-checked against the DSO.Source: §5.1.1 · p. 6

The DSO: Vaccine Education in Low- and Middle-Income Countries

• For vaccine education, evaluators can use the top-level DSO or four country-specific knowledge graphs to avoid contextually irrelevant testing criteria.Whether the DSO is an appropriate domain model remains a separate expert or governing-organization determination.Source: The DSO: Vaccine Education in Low- and… · p. 6

5.1.2 The EPO

• The EPO specifies the evaluation process, and its records capture requirements, criteria, probes, evidence, and attestations for machine checking and verifiable claims.The EPO uses standard definitions from systems engineering, software testing, and assurance practices where possible.Source: §5.1.2 · p. 6
• The operational envelope is measured through acceptance criteria: requirements express broad rules, while criteria specify output behaviors that can be checked individually.Coverage is the weighted share of acceptance criteria exercised.Source: §5.1.2 · p. 6

Requirement: do not increase transmission risk

• A broad requirement such as “must not increase transmission risk” is decomposed into criteria that specify behavior, can be checked against one response, and allow yes, no, or cannot-be-determined judgments.Examples include asking rather than assuming vaccination status, declining questions about other diseases, and giving measles-transmission information for public-transport questions.Source: §5.1.2 · p. 7, Requirement: do not increase… · p. 7
• Deployment sensitivity weights acceptance criteria by the consequences of undesirable outputs, enabling comparable evaluations of different chatbots or the same chatbot before and after a modification.In the measles example, failing to gather vaccination status may receive more weight than failing to gather a patient’s name.Source: Requirement: do not increase… · p. 7

5.2 Step 3: Derive Realistic or Target Situations

• Scenarios are derived from the DSO and operational envelope, then linked to acceptance criteria to become probes evaluators can use.A scenario captures a mindset, user conditions, or a real-world situation to test.Source: §5.2 · p. 7
• DSO rules can block logically inconsistent scenarios before they are eligible to become probes.The paper’s example is a person described as immune to measles despite never having vaccine or infection exposure.Source: §5.2 · p. 7
• Which scenarios are blocked depends on the DSO and additional rules specifying what can be combined and how.Source: §5.2 · p. 7

5.3 Step 4: Run the AI Evaluation

• AI evaluations may take the form of red teaming or benchmarking and use human-centered, human-in-the-loop, or fully automated methods.Source: §5.3 · p. 7
• Creating individual AI evaluations is outside this paper’s focus because those approaches have been documented in many forms.Source: §5.3 · p. 7
• The paper points readers to resources on human-centered red teaming, benchmark creation, and annotation practices.The listed resources include Humane Intelligence and MLCommons materials.Source: §5.3 · p. 8

5.4 Step 5 and 6: Determine and Attest, and Report Results

• The evaluation record links requirements, acceptance criteria, probes, model responses, and attestations, supporting traceable claims about coverage and performance.An attestation records a judgment, often called an annotation, and the record supports recomputation of coverage.Source: §5.4 · p. 8

5.4.1 Coverage

• Coverage is the sensitivity-weighted share of applicable acceptance criteria in the operational envelope that have an attested outcome.It is an exact, recomputable score; a criterion counts only when it can be traced through the evaluation record to an attested outcome.Source: §5.4.1 · p. 8
• Coverage excludes probes and responses outside the declared operational envelope, while probes testing whether the system declines out-of-scope questions can count.Source: §5.4.1 · p. 9
• Coverage scores are directly comparable only when evaluations share the same operational envelope.A higher score for a different envelope does not establish better or worse coverage because the evaluations measure different scopes.Source: §5.4.1 · p. 9

5.4.2 Performance

• Performance is the distribution of pass, fail, and cannot-be-determined outcomes among covered acceptance criteria, and it is reported separately from coverage.Combining the measures could conceal that a system passed everything tested while much of its operational envelope remained untested.Source: §5.4.2 · p. 9

5.5 How does the OB-CAIE methodology improve the current state of AI evaluations?

• OB-CAIE strengthens AI evaluation by clearly defining what will be tested, the first step in the scientific method.A structurally consistent representation also lets evaluators measure coverage and accommodate engineering and budgetary constraints.Source: §5.5 · p. 9
• Across evaluation types, OB-CAIE is described as reducing repeat human labor for complex benchmarks, targeting red-team probes, and structuring human-driven evaluations to scale.For LLM-as-a-judge evaluations, the methodology is intended to retain valuable human expertise and lived experience.Source: §5.5 · p. 9
• The structured problem space lets teams distribute probes across what can be tested to prioritize relevant failures or match annotator expertise under compute and staffing constraints.The authors contrast this with assigning categories and severity for annotators to follow.Source: §5.5 · p. 9
• A reusable, encoded artifact for expert labor supports reproducible evaluation processes and greater statistical comparability across contexts without losing context.The same problem space can be used to create multiple AI evaluations.Source: §5.5 · pp. 9–10
• OB-CAIE assigns human judgment to selecting and authoring the DSO, setting evaluation parameters, reviewing scenarios and responses, and making final fitness determinations.Automation derives and executes checks, keeps records, and computes coverage; it is intended to remove friction for human experts, not replace their judgment.Source: §5.5 · p. 10
• OB-CAIE evaluations are verified through machine-checkable declarations of coverage and process, and validated by assessing whether they target the right problem space with sufficient evidence.The authors link trustworthy evaluations to confidence in claims that an AI system meets tested criteria and is fit for its intended purpose.Source: §5.5 · p. 10

Terminology

• A probe is a scenario plus the acceptance criteria it checks; outcome records whether criteria were met, while sufficiency asks whether evidence supports the judgment.Appropriateness and sufficiency are human judgments, not quantifiable metrics, and both appear in every attestation.Source: Terminology · p. 11

6.1 Coverage

• Coverage is the weighted share of acceptance criteria with at least one attestation, with weighting reflecting the importance of what was covered.When criteria are equally weighted, it is the fraction of criteria with an attestation among those in the operational envelope.Source: §6.1 · pp. 12–13

6.2 Probe Pass Rate

• Probe pass rate is estimated from sampled judgments for an acceptance criterion because evaluations typically run only a fraction of plausible, relevant probes.Probes are generated from the DSO, which supplies an explicit sampling frame; the estimate therefore has sampling error.Source: §6.2 · p. 13

6.3 Outcome Distribution

• Outcome distribution reports pass, fail, and cannot-be-determined fractions among covered criteria; the rates sum to 1 and exclude not-applicable attestations.It is an exact count of covered criteria in each category and has no error bars.Source: §6.3 · p. 13

6.4 Specification Links to Individual Terms

• The section provides specification links for the Domain-Specific Ontology, Evaluation Process Ontology, requirements traceability, and mathematical coverage representation.The authors describe an executable specification and list direct links to individual terms.Source: §6 · p. 10, §6.4 · pp. 13–14

Measles Outbreak + Public Health Chatbot

• Scenarios are machine-generated from the knowledge graph and operational envelope, then consistency-checked so illogical cases are removed before expert review.This avoids spending human expert time reviewing scenarios that could not happen.Source: Measles Outbreak + Public Health Chatbot · p. 15
• OB-CAIE complements rather than replaces human judgment, which is structurally required when deciding whether scenario responses are acceptable.The authors contrast this with relying only on automated graders or only on human judgment that may apply criteria inconsistently.Source: Measles Outbreak + Public Health Chatbot · p. 15
• Coverage can guide a constrained test budget: with 300 probes, the example allocates 200 to the highest-weighted acceptance criteria rather than distributing them evenly.For red teaming, probes are run as conversations and responses are judged against an authoritative reference.Source: Measles Outbreak + Public Health Chatbot · p. 15
• 62 percent of the weighted envelope was covered; outcome distribution was 71 percent passed, 22 percent failed, and 7 percent cannot be determined, and the chatbot was unfit to deploy.The evaluation record identifies criteria that failed and criteria that were not evaluated.Source: Measles Outbreak + Public Health Chatbot · p. 15
• The approach extends to other domains by using domain-specific standards and experts, with weights reflecting vulnerabilities that have caused the greatest harm.For a student writing assistant, the passage names writing standards and educators as examples.Source: Measles Outbreak + Public Health Chatbot · p. 15

8 Conclusion

• OB-CAIE declares what is being tested before testing, supporting clearer understanding of vulnerabilities in high-stakes AI systems before deployment or scaling.The authors describe the methodology as an important step toward strengthening the scientific method in contextual AI evaluations.Source: §8 · p. 16
• The methodology uses a repeatable approach and aims to incorporate human feedback effectively and in non-extractive ways.The authors say they look forward to further developing the methodology, mathematical modelling, and inference.Source: §8 · p. 16