Source-linked AI summary
Governed Human-AI Prioritization Under Uncertainty: Adaptive Estimation and Dependency-Constrained Portfolio Selection
Azzeddine Ihsine, Sara Ihsine
TL;DR
The paper addresses how software teams can govern heterogeneous human–AI estimates, strategic parameters, calibration, dependencies, and capacity in prioritization. It evaluates explicit decision operators with controlled synthetic experiments, finding evidence for reliability-weighted and adaptive estimation, divergence-based calibration, and objective-matched portfolio optimization.
Problem
AI-native prioritization must govern heterogeneous estimates, uncertainty, strategic parameters, dependencies, and limited capacity while keeping decisions inspectable and recalibratable.
Method
The study evaluates D-POAF operators for parameterized ranking, reliability-weighted estimation, calibration signals, and dependency-constrained portfolio selection using controlled experiments.
Results
The experiments support reliability-weighted and adaptive estimation, model–collective calibration diagnostics, and competitive but not generally optimal ODP distance for constrained portfolio selection.
Takeaways & Limitations
Evidence-weighted authority makes parameters, estimator influence, calibration signals, and optimization objectives explicit enough to measure, challenge, version, and improve.
Takeaways & Limitations
The findings are computational and controlled; industrial deployments are needed to determine field effect sizes and organizational boundary conditions.
Abstract
from arXiv · showhide
AI-native software engineering increasingly combines human judgment, historical analogy, parametric estimation, and AI-generated forecasts inside the same prioritization decision. The resulting problem is not merely how to rank candidate work, but how to govern heterogeneous estimates, uncertainty, strategic parameters, dependencies, and limited capacity in a way that remains inspectable and recalibratable. We study five quantitative operators used in the D-POAF decision practice: Business Value Score (BVS), Effort and Risk Score (ERS), Prioritization Value Score (PVS), Collective Calibration Score (CCS), and Optimal Development Path (ODP). Controlled synthetic experiments characterize their behavior under known latent variables and explicit error processes. Moderate BVS-weight perturbations preserved global rankings (median Spearman 0.986), while broader strategic changes reduced top-10% overlap to 0.788. Reliability-weighted effort aggregation achieved MAE 0.616, a 42.6% reduction relative to the best individual estimator (MAE 1.073), and outperformed a simple mean (MAE 0.638). Under estimator drift, adaptive reliability weighting reduced average RMSE by 4.5% relative to equal weighting. Model-collective divergence detected the highest-error priority estimates with ROC-AUC 0.906. In 800 dependency-constrained portfolio instances, value-to-effort achieved mean objective ratio 0.962 and ODP distance 0.953 against the exact optimum; bootstrap intervals confirm a small but systematic advantage for value-to-effort under the declared objective. These results establish a quantitative basis for evidence-weighted human-AI prioritization and define the optimization boundary between shortest-path ODP formulations and general release-portfolio selection.
1 Introduction
The paper frames AI-native prioritization as a governed decision problem involving heterogeneous estimates, strategic parameters, uncertainty, dependencies, and constrained capacity. It evaluates parameter sensitivity, reliability-based estimation, calibration signals, and constrained portfolio selection as explicit computational mechanisms.
- AI-native engineering combines expert, analogy, parametric, and AI estimators whose accuracy depends strongly on context.
- The study asks how parameter changes affect rankings, whether predictive reliability should determine estimator influence, and how transparent heuristics compare with exact portfolio optimization.
- It tests these questions through D-POAF operators for parameterized ranking, reliability-weighted estimation, calibration, and dependency-constrained portfolio selection.
- The contribution separates expected value, feasibility, calibration, and post-delivery value rather than allowing one score to represent all four.
- The work provides sensitivity evidence, adaptive estimator aggregation, calibration diagnostics, and exact portfolio benchmarking within one governed decision model.
2 Related Work
Prior work treats software prioritization as a value–cost and constrained release-selection problem, while estimation research emphasizes context dependence and ensemble methods. Human–AI decision research adds the need to calibrate reliance, motivating explicit decomposition and reliability-updated estimator influence.
- Software prioritization research frames decisions as explicit value–cost trade-offs and shows that realistic release selection under budgets is not generally reducible to sorting.
- The paper keeps value, feasibility, calibration, dependencies, budget, and measured outcomes separately traceable instead of collapsing them into an opaque ranking.
- Software estimation studies report strong context dependence, while ensemble reviews find frequent gains and comparatively limited dynamic ensemble selection.
- Human–AI collaboration requires calibration beyond predictive accuracy because standalone accuracy does not necessarily identify the best teammate and assistance can induce overreliance.
3 Decision Model
The decision model represents value, effort, risk, feasibility, calibration, and dependency structure as explicit, inspectable components. It treats parameter changes and optimization choice as governed decisions rather than hidden tuning details.
- BVS aggregates business impact, urgency or criticality, and strategic opportunity through explicit weights that can be versioned and replayed.
- ERS combines estimated effort and technical risk, with coefficients controlling sensitivity and multiple estimation sources permitted.
- PVS uses an exponent α to control how strongly business value dominates feasibility, making its ranking effect an explicit sensitivity target.
- CCS uses divergence between model priority estimates and independently generated collective signals to identify possible miscalibration, while keeping calibration separate from realized value.
- ODP uses ERS/BVS distance and shortest-path reasoning for path objectives, but general budgeted prerequisite selection requires benchmarking against an objective-matched optimizer.
4 Research Questions and Experimental Method
The experiments use controlled synthetic data with known latent variables and explicit error processes to evaluate ranking stability, estimation accuracy, calibration discrimination, and constrained portfolio quality. Metrics are matched to each research question and include sensitivity to the PVS exponent.
- The study evaluates four questions covering parameter sensitivity, heterogeneous estimator error, adaptive calibration under nonstationarity, and constrained portfolio optimization.
- Rank stability is measured with Spearman correlation and top-10% overlap, while effort prediction uses MAE, RMSE, signed bias, and standardized accuracy.
- CCS diagnostic quality uses ROC-AUC and precision/recall at a fixed 20% flagging rate, and portfolio quality uses objective ratio, normalized regret, and exact-optimum rate.
- PVS sensitivity is evaluated by comparing ranking behavior across exponent values relative to α = 1.5.
5 Results
The experiments show that governed parameter changes can reshape top-portfolio membership, while reliability-aware aggregation and objective-matched optimization improve transparent prioritization under uncertainty.
- 5.1 RQ1: governed parameters materially shape the top portfolio: Median Spearman correlation fell from 0.986 under narrow BVS perturbations to 0.945 under broader shifts, while top-10% overlap fell to 0.788.PVS also changed materially: α = 1 versus α = 2 retained 0.955 correlation but only 0.764 top-10% overlap.
- 5.2 RQ2: reliability weighting improves heterogeneous effort aggregation: MAE 0.616 from reliability weighting improved on the best individual estimator's MAE 1.073 and the simple mean's MAE 0.638.This corresponds to a 42.6% reduction relative to the best individual estimator.
- 5.3 RQ3: adaptive calibration tracks drift and exposes miscalibration: A 4.5% RMSE reduction showed that adaptive reliability weighting outperformed equal weighting under estimator drift.Average RMSE was 0.757 with adaptive weighting versus 0.793 with equal weighting.
- 5.3 RQ3: adaptive calibration tracks drift and exposes miscalibration: ROC-AUC 0.906 showed that model–collective divergence detected the highest-error priority estimates in the calibration experiment.The experiment injected stale-context bias into 15% of model estimates and used an independently generated collective signal.
- 5.4 RQ4: portfolio selection requires an objective-matched optimizer: 0.962 mean objective ratio made value-to-effort the strongest heuristic, ahead of ODP distance at 0.953, under the declared objective.The paired advantage over ODP distance was 0.0094 objective-ratio points, with bootstrap 95% CI [0.0038, 0.0150].
- 5.4 RQ4: portfolio selection requires an objective-matched optimizer: ODP distance is exact only for its declared additive path objective; general budgeted prerequisite selection requires an optimizer matched to the portfolio decision class.For general release selection, ERS/BVS is a heuristic preference signal rather than a proof of global optimality.
6 Discussion
The discussion frames prioritization as an evidence-governed system in which parameters, estimator influence, calibration signals, and optimization objectives remain explicit and reviewable. It also distinguishes shortest-path optimality from heuristic portfolio selection under budgets and dependencies.
- Governed parameters: Parameter ownership, rationale, version history, and replayable rankings are part of decision evidence because strategic parameters can change top-portfolio membership.BVS weights, ERS coefficients, and the PVS exponent can preserve broad ordering while materially changing cutoff decisions.
- Adaptive estimation: Estimator influence can be updated from observed predictive performance, replacing fixed role-based authority with measurable reliability while retaining human governance.Contextual constraints and accountability remain governance decisions, but forecasting influence need not remain arbitrary.
- Calibration: Model–collective divergence identified priority miscalibration with AUC 0.906 and can route high-divergence cases to review, context refresh, or rescoring.The collective signal is used as an independent calibration channel rather than as a replacement for model scoring.
- Objective-matched optimization: “Optimal” portfolio selection depends on the explicit objective and constraint set, so general release selection requires an algorithm matched to its problem class.Shortest paths can be exact for declared additive path objectives, whereas budget- and prerequisite-constrained release selection may require integer, dynamic, evolutionary, or multi-objective methods.
- Objective-matched optimization: Figure 5 reports mean normalized regret for portfolio heuristics, with lower values indicating better performance.The figure’s comparison is interpreted through the declared portfolio objective and constraints.
7 Scope and Validity
The study’s validity is bounded by declared constructs, controlled error processes, synthetic computational conditions, and objective-specific optimality. Industrial deployments and alternative objectives require further validation.
- Internal validity: The experiments support mechanism-level claims under known latent variables and explicit error processes rather than uncontrolled organizational variation.This design permits direct measurement of ranking sensitivity, aggregation error, drift response, diagnostic discrimination, and optimization regret.
- Construct scope: BVS, ERS, and CCS operationalize declared decision variables and are not asserted to be universal measurement scales for business value or organizational risk.The sensitivity results characterize these operators once their variables are instantiated.
- Internal validity: Estimator bias, noise, drift, and stale-context error were controlled by design, so the reported differences characterize the stated model class.Alternative error structures can be tested as additional experimental conditions without changing the evaluation method.
- External validity: Industrial deployments will determine field effect sizes and organizational boundary conditions, making field validation the next external-validity layer.The current computational findings establish mechanism behavior under explicitly defined conditions, not deployment effects.
- Optimization validity: All portfolio optimality statements are relative to maximizing declared BVS under an effort budget and prerequisite constraints.Risk-adjusted value, diversity, reliability, or multi-period utility would define different optimization problems.
8 Conclusion
The study turns D-POAF decision logic into a testable quantitative layer for governed human–AI prioritization. Its findings support explicit governance of parameters, adaptive estimator influence, calibration review, and optimization matched to the declared problem class.
- Conclusion: Reliability-weighted aggregation reduced effort-estimation error, adaptive weighting tracked estimator drift, and model–collective divergence provided a strong miscalibration signal.These findings support evidence-weighted authority across heterogeneous forecasts and calibration decisions.
- Conclusion: Strategic parameters materially affect top-portfolio membership, requiring explicit governance of their ownership, rationale, version history, and replayable rankings.The conclusion treats parameter governance as part of the decision evidence rather than documentation overhead.
- Conclusion: ODP distance is competitive but not generally optimal for budget- and dependency-constrained release selection, so optimization must match the declared problem class.The paper distinguishes transparent sequencing from exact or alternative methods for general portfolio problems.
- Conclusion: The findings move D-POAF decision logic from descriptive practice to a testable quantitative model whose evidence can be measured, challenged, versioned, and improved.Subsequent field deployments can measure external effects while preserving the established experimental and governance structure.
Availability
The paper provides a verification pack with data summaries, figure assets, the raw portfolio benchmark, manuscript source, and an audit script for recomputing reported statistics and intervals.
- Availability: The verification pack includes data summaries, figure assets, the 800-instance raw portfolio benchmark, revised-manuscript LaTeX source, and an audit script.The script independently recomputes portfolio summary statistics and bootstrap confidence intervals.
- Availability: The D-POAF Canonical Specification, Official Terminology, and Operating Guide are available through the cited Zenodo records and practitioner documentation.These resources document the framework and its reference mechanisms.