Source-linked AI summary
Evaluating Artificial Wisdom (AW): A Trajectory-Based Framework for Context-Sensitive Decision Regulation
Md. Jalil Piran
TL;DR
Capability benchmarks measure what models can do, but say little about whether their recommendations remain appropriate as decision conditions change. This paper proposes a trajectory-based framework and prospective protocol for measuring context-sensitive regulation of judgment. Its simplified family-level simulation detected a moderate Brier improvement more often with twelve than eight families, but does not establish power for the operational analysis.
Problem
Capability benchmarks assess knowledge, reasoning, and task completion, but provide limited evidence about whether models use capability well as decision conditions change.
Source: §1 · p. 1Method
The paper proposes an auditable prospective protocol integrating specification-indexed adequacy, registered transition materiality, and noncompensatory reporting across decision trajectories.
Source: §1 · p. 1Results
For a moderate improvement (∆= .027), the simplified simulation’s detection probability was .508 with eight families and .742 with twelve, without establishing operational-protocol power.
Source: §6.6 · p. 10, §8 · p. 12Takeaways & Limitations
The framework’s next empirical test is whether its profile predicts decision quality beyond capability and existing metacognitive measures.
Source: §9 · p. 12The twelve-family target remains provisional pending feasibility assessment and calibration, and external validity is limited by the advisory design and available families.
Source: §6.3 · p. 9, §8 · p. 12
Abstract
from arXiv · showhide
Capability benchmarks measure what large language models can do, but not whether their recommendations remain appropriate as decision conditions change. This paper proposes a trajectory-based measurement framework and prospective validation protocol for artificial wisdom, defined behaviorally as context-sensitive regulation of judgment. Versioned reference specifications determine admissible responses and transition materiality before evaluation. Seven stage-level criteria assess judgment under current conditions, while two transition components assess material-update adequacy and commitment-preserving stability. Stability requires prior adequacy; switching within registered commitment classes is permitted, and persistent error earns no stability credit. A nine-component profile is primary, with a secondary bottleneck score and separate critical-failure reporting. Three simplified simulations examine precision, bottleneck estimation, and sensitivity to model-family count. A single-criterion simulation met its precision criteria at 24 scenarios per domain. Near a bottleneck tie, the estimated minimum had a downward bias of 0.55 points at that size. A simplified incremental-prediction simulation detected a Brier improvement of 0.027 with probabilities of .508 and .742 using eight and twelve families, respectively; it does not establish power for the operational protocol. An exploratory predecessor-rubric study comprising 71 evaluation records illustrates aggregation sensitivity but does not validate the proposed instrument. The twelve-family target remains provisional pending feasibility assessment and calibration of the complete analysis. Whether the profile predicts independent decision outcomes beyond capability remains an open empirical question.
1. Introduction
Capability benchmarks assess what models can do, but not whether recommendations remain appropriate as decision conditions change. The paper proposes an auditable, trajectory-based framework for evaluating context-sensitive regulation of judgment.
1. Introduction
- Sound evaluation must test both warranted revision after relevant change and resistance to irrelevant pressure, rather than treating agreement, frequent revision, or consistency alone as evidence of good judgment.Responses are assessed for adequacy given the information and constraints available at that stage.Source: §1 · p. 1
- AW is defined behaviorally as reliable, context-sensitive decision regulation, combining stage-level judgment quality with transition-level adequacy and commitment preservation.The paper distinguishes instrumental capability, judgment, and decision regulation without estimating a separate cognitive-maturity score.Source: §1 · p. 1
- Psychological accounts motivate attention to uncertainty, context, competing interests, and perspective taking, without assuming that artificial and human systems share mechanisms.The framework borrows these requirements as behavioral considerations.Source: §1 · p. 1
- The framework integrates specification-indexed adequacy, registered transition materiality, and noncompensatory reporting in a prospective measurement protocol.It specifies response records and inquiry rules, plus stage-level criteria and transition indicators for material updates, commitment-preserving stability, persistent error, and recovery.Source: §1 · pp. 1–2
- The paper specifies estimation, a bottleneck summary, and a family-level validation protocol, supported by simulations of bottleneck estimation and family-level power.It also reports a predecessor-rubric clinical study to illustrate how compensatory and failure-discounted averages discard component information.Source: §1 · p. 2
2. Related work
Prior work motivates evaluating judgment beyond task competence, including regulation across changing situations and responsiveness to morally relevant considerations. This paper integrates those concerns through explicit specifications, transition criteria, and prospective validation.
2. Related work
- The framework distinguishes warranted revision from acquiescence and appropriate consistency from persistent error through specification-indexed admissibility, registered transition materiality, and commitment classes.Pluralistic alignment recognizes that several responses may be defensible under different reasonable commitments.Source: §2 · p. 2
- Stateful benchmarks examine actions as conversational and environmental states change, whereas this framework evaluates non-executed advisory trajectories and responses to new information or irrelevant pressure.Its conclusions concern advisory decision regulation rather than execution reliability.Source: §2 · p. 2
- Calibration and plausible public rationales do not establish that recommendations respect constraints or change appropriately; evidence interventions and provenance ledgers offer complementary behavioral checks.The framework requires invariance only for changes registered as immaterial and conditions stability on prior adequacy.Source: §2 · p. 2
- Because the profile is formative, internal consistency and factor fit cannot establish validity; evidence must address content coverage, reproducibility, independent outcomes, and intended score uses.The prospective protocol tests incremental validity beyond capability and existing metacognitive measures.Source: §2 · pp. 2–3
- Table 1 summarizes complementary research emphases and the framework’s integration, while noting that validation remains prospective.The entries are not an exhaustive comparison of differences.Source: Table 1 · p. 3
3. Construct and decision setting
The paper defines artificial wisdom as regulation of decisions relative to evidence, uncertainty, alternatives, interests, oversight, and consequences. It scores responses against stage-specific conditions and preregistered rules for inquiry and material change.
3. Construct and decision setting
- Each stage’s judgment is assessed against the conditions then in force, and ignoring a newly material stakeholder counts as both poor judgment and failed regulation.Timeliness matters when delay causes harm; less irreversible options receive no automatic preference.Source: §3 · p. 3
- A response record combines an action, a public rationale, and an uncertainty representation; the action includes disposition, decision content, and safeguards.Permitted dispositions include acting, inquiring, deferring, escalating, or rejecting, and no private reasoning is required.Source: §3 · p. 3
- Scoring uses actual history and post-answer predicates after inquiry; a consequential directive without required inquiry is a procedural failure and is not scored as though the answer were known.Hidden facts or later outcomes cannot make an earlier justified decision inadequate.Source: §3 · p. 3
- Materiality is fixed before model evaluation and concerns whether a transformation warrants reconsidering a decision-relevant component, even when the principal action remains defensible.Inquiry exchanges occur within a scripted stage and do not create additional registered transitions.Source: §3 · pp. 3–4
- The framework uses explicit scenario assumptions and may retain several internally consistent specifications when reasonable assumptions yield different verdicts.It estimates no universal harm coefficients and measures performance relative to reference conditions, not a culture-free ordering of wisdom.Source: §3 · p. 4
4. Measurement
The measurement framework fixes reference specifications before evaluation, scores stage-level judgment and trajectory transitions separately, and reports a nine-component profile with a secondary bottleneck score. Its transitions distinguish adequate updating from commitment-preserving stability, which is credited only after prior adequacy.
4.1. Stage-level criteria
- Raters score each applicable criterion from 0 to 4, with high ratings requiring integration of applicable requirements and registered boundary conditions into the decision.Length, caution words, and stakeholder lists earn no credit unless they affect the decision; applicability is fixed in advance.Source: §4.1 · p. 4
4.2. Response adequacy
- Specifications are proposed by a mixed panel, challenged by a second team, and frozen with applicability masks, commitment classes, and an inquiry, action, and escalation decision tree.Clarifications are versioned and applied to all systems; substantive changes invalidate the affected confirmatory analysis.Source: §4.2 · p. 4
4.3. Transitions and selective corrigibility
- Commitment classes may contain distinct admissible alternatives when switching preserves every registered obligation, so equivalent implementations need not count as commitment changes.Switching within a class gives zero commitment distance; tolerance never permits removing a mandatory safeguard or violating a registered obligation.Source: §4.3 · pp. 4–5
- For required-change transitions, destination adequacy alone does not establish observed revision; stability excludes transitions preceded by an inadequate response, so persistent error earns no stability credit.Stability comparisons report conditional stability alongside eligibility and the joint rate.Source: §4.3 · p. 6
4.4. Estimation, profile, and bottleneck score
- Stage-level probabilities estimate the chance an independent qualified rater assigns a rating of at least 3, using a cumulative-link mixed model marginalized over the registered design.If convergence requirements are unmet, the framework reports design-weighted empirical adequacy rates with scenario-cluster uncertainty.Source: §4.4 · p. 6
- Seven criterion probabilities and two transition rates form the primary profile; the minimum component is a secondary score, with critical failures reported separately.The bottleneck score is released only when every component has eligible opportunities in at least 20 independently authored scenario clusters.Source: §4.4 · pp. 6–7
5. Worked example and scoring properties
The worked examples illustrate how trajectory-level adequacy, revision, and stability can diverge, and how a noncompensatory bottleneck score responds to the weakest component. These are authored illustrations rather than empirical observations.
5. Worked example and scoring properties
- An always-withholding trajectory has Rcor = 1/3 while H = 1/2, illustrating how the harmonic mean can exceed the weaker requirement.The trajectory is adequate at stages 0, 3, and 4; its rates are Rmat = 1/3 and Rstab = 1/1.Source: §5 · p. 7
- Four difficult cases are coded as variants of the worked trajectory to illustrate cases that simpler schemes mishandle.The table is an authored illustration, not an observed-response comparison.Source: §5 · p. 7, Table 4 · p. 7
- At stage 3, unchanged withholding becomes adequate after reference conditions change, but does not count as observed principal-decision revision; subsequent preservation at stage 4 earns stability credit.Stability requires an adequate preceding response and preservation of registered commitments on an immaterial transition.Source: §5 · p. 7
6. Validation protocol and design simulations
The protocol preregisters content-valid scenarios, independent outcome labels, and family-held-out prediction, then uses simulations to assess precision and sensitivity. Simulations met single-criterion precision targets at 24 scenarios per domain, but the simplified bottleneck and power results do not establish performance of the complete operational analysis.
6.1. Content validity and sampling
- Scenarios are planned across four domains with expert content review, six-stage trajectories, registered material and immaterial changes, and matched demographic variants.The provisional design targets 12 lineage clusters and two configurations per cluster; Figure 2 illustrates distinct adequacy, revision, and stability cases.Source: §6.1 · pp. 7–8, Figure 2 · p. 8
6.2. Outcomes and hypotheses
- The validation separates predictor and outcome scenarios, freezes decision-quality rules before system outputs are examined, and tests incremental prediction against prespecified baselines.Confirmatory testing is conditional on calibration of the complete nested pipeline; the .01 Brier-improvement margin is provisional.Source: §6.2 · pp. 8–9
6.4. Single-criterion precision
- At 24 scenarios per domain, all simulated single-criterion precision criteria were met, although known nuisance parameters make this check optimistic.The transition-summary 95th-percentile absolute errors were .091 and .092 at that size.Source: Table 6 · p. 9, §6.4 · p. 9
6.3. Feasibility and data handling
- Feasibility checks include rating completion, agreement, annotation burden, and lineage support; confirmation will not proceed under unsupported workload or lineage allocation.The planned allocation is approximately 29,491 trajectory-rating assignments and up to 1.24 million criterion ratings before adjudication.Source: §6.3 · p. 9
6.5. Bottleneck estimator
- In the clear configuration, percentile-interval coverage fell from .942 to .922, so reliable nominal coverage was not established.The 200-draw bootstrap budget also requires assessment with a larger resampling budget.Source: Table 7 · p. 10, §6.5 · p. 10
6.6. Family-level power
- With 12 families, detection probability for a Brier improvement of ∆= .027 was .742, versus .508 with eight; the simplified simulation does not establish operational-protocol power.For ∆= .017, detection remained .356 with twelve families; the provisional target is not justified by these results.Source: Figure 3 · p. 10, §6.6 · p. 10
7. Exploratory evaluation of clinical advisory scenarios with a predecessor rubric
The exploratory predecessor-rubric study illustrates how aggregation changes summaries of clinical advisory ratings, but it does not validate the proposed instrument.
7. Exploratory evaluation of clinical advisory scenarios with a predecessor rubric
- The study rated six-stage advisory sequences across five clinical scenarios and six interface configurations, using seven predecessor items rated from 0 to 4.Nine registered evaluators, including seven physicians and two AI researchers, supplied the masked ratings.Source: §7 · p. 11
- The predecessor items map only approximately to final criteria, and absent reference specifications, adequacy predicates, and commitment classes prevent computing transition indicators.The study is reported only to illustrate information discarded by compensatory and failure-discounted averages, not to establish feasibility, reliability, or validity.Source: §7 · p. 11
- Figure 4 summarizes 71 completed records, showing uneven domain profiles and how weakest-domain emphasis or critical-error discounting changes summaries from the same ratings.These differences illustrate aggregation sensitivity, not comparative validity or safety.Source: §7 · p. 11
- 2.70 was the strategic item's lowest pooled mean and it was lowest or tied-lowest for five of six configurations; domain-ordering concordance was Kendall’s W = .586, and summary correlation was ρ = .771.Four evaluation records carried unadjudicated critical-error flags; these observations do not establish comparative performance, failure prevalence, or safety.Source: §7 · p. 11
8. Discussion
The discussion emphasizes that trajectory scores depend on warranted change and prior adequacy, while simulations qualify bottleneck interpretation and planning. The framework also faces limits in construct coverage, validity, and generalizability.
8. Discussion
- Consistency and responsiveness count only when conditioned on warranted change, prior-response adequacy, and preserved registered commitments; commitment classes accommodate plural admissibility.Stability and revision therefore cannot be judged by frequency alone.Source: §8 · p. 11
- Near ties, the plug-in bottleneck was downward-biased even at 96 scenarios per domain, making the profile, conservative interval, and identity frequencies part of score interpretation.At the simulated scenario count, moderate improvement detection probabilities were .508 with eight families and .742 with twelve.Source: §8 · p. 12
- Construct coverage is tested but not guaranteed, and external validity is limited by the advisory design and available families, while simulated power relies on assumed effect structures.The discussion also identifies rhetorical style and predictor–outcome overlap as internal-validity threats and notes that training on indicators may invite Goodhart effects.Source: §8 · p. 12
9. Conclusion
The conclusion presents a trajectory-based framework for evaluating artificial wisdom and outlines empirical validation as the next step.
9. Conclusion
- The proposed framework evaluates artificial wisdom as context-sensitive regulation of judgment across decision trajectories.It combines specification-indexed references, structured action records with inquiry rules, transition indicators, and a bottleneck summary.Source: §9 · p. 12
- The exploratory clinical study illustrates information lost through averaging but does not test the final instrument.Source: §9 · p. 12
- Next steps are a feasibility study and, if progression rules are met, a twelve-family confirmatory test of prediction beyond capability and existing metacognitive measures.Source: §9 · p. 12