Source-linked AI summary
SENTRY: Deterministic, Intelligent Risk Assessment for IT Change Management
Daniel Arulpragasam, Christer Henrysson, Ella Ly, Deepika Anbalagan, Leo Feng
TL;DR
Questionnaire-based change risk assessment is subjective, gameable, and inconsistent, while manual investigation is labor-intensive. SENTRY combines XGBoost with hybrid retrieval-augmented generation to produce deterministic, explainable risk assessments; on enterprise-scale data, it achieves ROC AUC 0.87 and detects high-risk changes at 3.25 times the existing process’s rate.
Problem
Questionnaire-based assessments are subjective, inconsistently interpreted, easy to game, and often lack dependency, incident-history, and historical-similarity information.
Method
SENTRY combines an XGBoost classifier using structured operational features with hybrid semantic and lexical retrieval over historical changes, compressing retrieved risk into a single feature with SHAP explanations.
Results
0.87 ROC AUC and 3.25 times the existing questionnaire-based process’s rate of detecting high-risk changes were achieved on enterprise-scale change data.
Takeaways & Limitations
The architecture uses unstructured change-request information while preserving determinism, auditability, and per-prediction explainability for regulated change management.
Takeaways & Limitations
The model uses a fixed historical-data snapshot, so predictive accuracy will degrade as the technology landscape evolves unless the model is periodically retrained.
Abstract
from arXiv · showhide
Technology change management in large financial institutions depends on risk assessments that are accurate, consistent, and auditable. In practice, many institutions still rely on self-reported questionnaires. Those questionnaires are subjective, easy to game, and poor at separating routine changes from the ones that later trigger major incidents. This paper presents SENTRY, a risk assessment platform that replaces questionnaire-based scoring with a deterministic machine learning pipeline built from gradient-boosted decision trees (XGBoost) and hybrid retrieval-augmented generation (RAG). The system combines structured operational metadata, application dependency graphs, and historical incident records with a hybrid semantic and lexical search over historical change requests. The retrieval step captures the risk signal in unstructured change request text, then compresses that signal into a single scalar feature before model inference. That design keeps the model deterministic and preserves per-prediction explainability via SHAP values. Evaluated on enterprise-scale change data, SENTRY achieves a ROC AUC of 0.87 and 85% overall accuracy, and it detects high-risk changes at roughly 3.25 times the rate of the existing process. We close by examining the architectural trade-offs behind this design and what they imply for the use of machine learning in regulated change management.
1 Introduction
SENTRY addresses subjective, inconsistent change-risk assessment in financial institutions by combining operational data with deterministic machine learning and retrieval from historical changes. The proposed architecture is intended to improve risk evaluation while preserving auditability and explainability.
- Major incidents can originate from changes assessed as low risk, creating regulatory, operational, financial, and reputational consequences.
- Self-reported questionnaires are subjective, can be manipulated to reduce apparent risk, and often provide little explanation for their ratings.
- Manual review adds dependency checks and incident-history investigation, but these checks are informal and make the process labor-intensive and less consistent as volume increases.Each change request is reviewed in about 10 minutes by approximately 4 approvers.
- SENTRY replaces questionnaire-based assessment with an objective pipeline that evaluates proposed-change risk independently of the requestor’s self-assessment.
- The architecture combines 27 structured features with a hybrid RAG-derived scalar score from historical change requests, retaining deterministic inference, auditability, and SHAP-style explanations.The retrieved text includes descriptions, implementation plans, test plans, and backout plans.
2 Related Work and Background
The paper situates SENTRY against questionnaire-based assessment, general AIOps research, and candidate machine-learning approaches. It selects deterministic, explainable tree models and adapts RAG by converting retrieval results into a scalar feature rather than generating with an LLM.
- Traditional Change Risk Assessment: Traditional change-risk assessment commonly uses requestor-completed questionnaires whose responses are aggregated into categorical ratings.
- Traditional Change Risk Assessment: Questionnaire assessment remains subjective and can be shaped by informed requestors, producing many low-risk classifications even for changes that later cause incidents.
- Machine Learning Approaches: ML applications in IT operations address areas such as incident prediction, anomaly detection, and root-cause analysis, while change-risk assessment remains underexplored.
- Machine Learning Approaches: LLM scoring can process unstructured change text but is unsuitable for regulated risk assessment because identical inputs may produce different outputs and reasoning is not recoverable.
- Machine Learning Approaches: Deep neural networks are deterministic and expressive, but their predictions lack a natural per-feature decomposition and post-hoc attributions introduce approximation error.
- Machine Learning Approaches: XGBoost was selected because fixed inputs produce identical outputs and its native pred_contribs API provides SHAP-style per-feature contributions.
- Retrieval-Augmented Generation: SENTRY adapts RAG by using hybrid semantic and lexical retrieval to identify similar historical changes without feeding retrieved records into an LLM.
3 System Design
SENTRY is organized as a frontend, orchestration API, machine-learning model, and separate structured and vector data stores. Its backend coordinates enterprise data sources that supply change, application, dependency, incident, and similarity-search information.
- Architecture Overview: SENTRY comprises a web frontend, backend API layer, trained risk-classification model, and separate structured-record and vector-similarity data stores.
- Architecture Overview: The paper limits its observation to the backend, specifically the machine-learning model.
- Architecture Overview: For each change request, the backend queries ITSM records, application metadata and dependency mappings, and vector-based similarity search in parallel.
- Data Sources: The ITSM database supplies change records, configuration items, application codes, and incident data across software and infrastructure assets.
- Data Sources: Application portfolio data includes criticality, regulatory and data-classification attributes, recovery objectives, user counts, and downstream dependencies for blast-radius analysis.
- Data Sources: The vector database stores dense embeddings and a precomputed lexical index for historical change-request text used by the hybrid RAG pipeline.
4 Methodology: XGBoost with Hybrid RAG
SENTRY converts structured operational context and unstructured change-request text into a deterministic, explainable risk assessment. Hybrid retrieval compresses historical similarity into one scalar feature for XGBoost rather than exposing raw embeddings directly.
- Architecture: SENTRY passes a single deterministic risk-weighted score from hybrid retrieval into XGBoost alongside structured features.The model uses 28 normalized features spanning application context, blast radius, incident history, similarity, and change type.
- Hybrid retrieval: The hybrid RAG pipeline embeds change-request text, performs semantic and lexical retrieval, and combines the ranked lists with Reciprocal Rank Fusion.Semantic search captures conceptual similarity, while lexical search captures technical terms, application names, and procedural language.
- Training safeguards: Temporal filtering excludes the current request and later requests, preventing historical retrieval from leaking future information into training.The query filters to requests opened before the target change request’s creation date.
- Risk-score computation: The retrieval score is computed over incident-bearing neighbors, with higher-priority incidents contributing more points and non-incident changes contributing zero.The default retrieval depth is k=100, and the score is a weighted sum over the retrieved neighbors associated with incidents.
- Explainability: Direct PCA embedding features reduced accuracy and produced opaque feature rankings, motivating their replacement with one interpretable retrieval feature.The resulting explanation can expose contributions such as hybrid_search_score and crown_jewel through SHAP-style values.
- Evaluation: On the held-out test set, the ROC curve has an AUC of 0.87.The figure reports the evaluation result for the held-out test set.
5 Evaluation
On a held-out test set, SENTRY detects substantially more incident-causing changes than the questionnaire baseline, while balancing recall against false alarms. Its hybrid search score also ranks among the strongest predictive features.
- Evaluation setup: The XGBoost evaluation uses a held-out test set of 203 change requests and standard classification metrics.
- High-risk detection: 63% recall on the positive class represents a 3.25× improvement over the questionnaire-based process’s approximately 20%.The positive class comprises changes that later cause major incidents and are identified as medium or high risk.
- High-risk detection: Precision on the positive class is 0.55, so some changes flagged for scrutiny do not later cause major incidents.The paper frames this as a trade-off between missing high-risk changes and generating excessive false positives.
- Feature importance: The hybrid search score ranks fourth among the top eight gain-based features, alongside three structured metadata features in the top five.This indicates that the retrieval-derived feature contributes materially alongside structured inputs.
6 Discussion and Limitations
SENTRY’s limitations center on scarce positive examples, static training and retrieval data, and institution-specific calibration. These constraints bound fine-grained high-risk learning, freshness, and portability across organizations.
- Limited positive examples: Approximately 183 label-1 examples constrain the model’s ability to learn fine-grained distinctions among high-risk changes.The examples correspond to change requests that caused P1/P2 incidents.
- Static training data: The model uses a fixed historical-data snapshot, so predictive accuracy may degrade as applications, dependencies, and change patterns evolve.An automated retraining pipeline is planned but not yet implemented.
- Temporal scope of embeddings: The vector database requires manual updates, leaving the hybrid-search corpus on a static snapshot that may omit recent change activity.This limits the temporal coverage of the RAG pipeline until update automation is implemented.
- Generalizability: Deployment in another organization requires local feature re-extraction, normalization recalibration, and model retraining.Although the methodology is domain-general, feature definitions, normalization ranges, and thresholds were calibrated to one large financial institution.
7 Conclusion
SENTRY combines hybrid retrieval with XGBoost to incorporate unstructured change-request risk while retaining deterministic, auditable, and explainable assessment. On enterprise-scale data, it achieved ROC AUC 0.87 and detected high-risk changes at 3.25 times the existing process’s rate, supporting decision assistance for human reviewers.
- Conclusion: SENTRY combines XGBoost with hybrid retrieval-augmented generation to use unstructured change-request risk without sacrificing determinism, auditability, or explainability.The architecture retrieves risk-relevant information from change text while preserving regulated-environment requirements.
- Conclusion: Hybrid semantic and lexical search retrieves similar historical changes, compresses their risk into one score, and supplies it to XGBoost as a feature.The resulting model uses 28 features and produces three-tier classifications with per-prediction SHAP explanations.
- Conclusion: ROC AUC 0.87 was achieved on enterprise-scale change data from a major financial institution.The evaluation used change data from a large financial-institution setting.
- Conclusion: 3.25 times the rate of the existing questionnaire-based process was achieved for detecting high-risk changes.This is the reported high-risk detection comparison for the evaluated system.
- Conclusion: SENTRY is a decision-support system that gives human reviewers better information, faster, and with more transparency rather than automating approval decisions.The system is positioned as assistance for reviewers as change volumes and technology complexity increase.