Deep Learning for Mortgage Risk

Justin Sirignano; Apaar Sadhwani; Kay Giesecke

Source-linked AI summary

Deep Learning for Mortgage Risk

Justin Sirignano, Apaar Sadhwani, Kay Giesecke

arXiv:1607.02470v2Jul 8, 2016q-fin.ST

TL;DR

Mortgage-risk research often relies on restricted functional forms despite evidence of nonlinear borrower behavior. This paper applies a deep learning model to a nationwide dataset of over 120 million mortgages, finding especially strong nonlinearities in prepayment and a dominant, state-dependent role for unemployment.

Problem
Prior mortgage models often impose pre-specified, commonly linear relationships, limiting analysis of nonlinear borrower responses to risk factors.
Method
The paper estimates multi-period mortgage-state transition probabilities with deep learning using loan-level, borrower-specific, and local economic variables.
Results
Prepayment exhibits the strongest nonlinear effects, while state unemployment has the greatest explanatory power and interacts with borrower characteristics.
Takeaways & Limitations
The findings provide implications for mortgage-backed security investors, rating agencies, and housing finance policymakers.
Takeaways & Limitations
Earlier studies use smaller samples and shorter periods, whereas the dataset covers roughly 70% of US mortgages and several economic cycles.

Abstract

from arXiv · show

We develop a deep learning model of multi-period mortgage risk and use it to analyze an unprecedented dataset of origination and monthly performance records for over 120 million mortgages originated across the US between 1995 and 2014. Our estimators of term structures of conditional probabilities of prepayment, foreclosure and various states of delinquency incorporate the dynamics of a large number of loan-specific as well as macroeconomic variables down to the zip-code level. The estimators uncover the highly nonlinear nature of the relationship between the variables and borrower behavior, especially prepayment. They also highlight the effects of local economic conditions on borrower behavior. State unemployment has the greatest explanatory power among all variables, offering strong evidence of the tight connection between housing finance markets and the macroeconomy. The sensitivity of a borrower to changes in unemployment strongly depends upon current unemployment. It also significantly varies across the entire borrower population, which highlights the interaction of unemployment and many other variables. These findings have important implications for mortgage-backed security investors, rating agencies, and housing finance policymakers.

1 Introduction

The paper addresses nonlinear mortgage-risk relationships using deep learning and an unprecedented nationwide dataset. It finds pervasive interactions, especially involving prepayment and unemployment, with implications for mortgage-risk analysis.

Data and approach: Over 120 million mortgages and more than 3.5 billion loan-month observations support a nonlinear analysis spanning 1995–2014 and over 30,000 US zip codes.The data combine origination and monthly performance records with loan-, borrower-, macroeconomic, and demographic variables.
Data and approach: The deep learning model avoids a pre-specified relationship between risk factors and loan performance, allowing nonlinearities and interactions among variables.It estimates multi-period probabilities across current, delinquency, foreclosure, REO, and prepaid states.
Nonlinear borrower behavior: Prepayment exhibits especially strong nonlinear effects, with major drivers including original and current loan balances, interest rates, and interest-rate spreads.These variables interact, and jumbo borrowers with relatively small current balances are estimated to prepay most often.
Macroeconomic interactions: State unemployment has the greatest explanatory power among the studied variables, and its effect varies with prevailing unemployment and borrower characteristics.High- and low-FICO borrowers’ prepayment sensitivities converge as unemployment rises above 11 percent.
Macroeconomic interactions: Unemployment interacts with loan-to-value ratios, mortgage rates, and house-price appreciation, affecting both prepayment and delinquency behavior.Delinquency responses to house-price appreciation depend on prior appreciation and the unemployment scenario.
Implications: The findings indicate a tighter connection between housing finance markets and the macroeconomy than prior work suggested, informing investors, rating agencies, and policymakers.Common exposure to economic cycles can generate substantial loan-to-loan correlation distinct from foreclosure contagion.

2 The Data

The study assembles a comprehensive mortgage dataset combining detailed origination characteristics, monthly performance, and local and national economic factors. Mortgages are represented through multiple transition states, with geographic information supporting analysis of local conditions and contagion.

Data coverage: Over 120 million mortgages cover roughly 70% of US originations from 1995–2014 across more than 30,000 zip codes.The sample includes 25 million subprime and 93 million prime mortgages.
Loan features: Each loan includes detailed origination features such as FICO, LTV, DTI, balance, interest rate, product type, property type, penalties, and location.Many features are categorical and can contain numerous categories.
Loan performance: Monthly records track delinquency, rates, balances, REO, foreclosure, payoff, and recent borrower behavior.The performance data are observed month by month between 1995 and 2014.
Economic factors: Loan records are matched to local and national economic factors, including housing prices, unemployment, income, mortgage rates, and geographic default and prepayment rates.Zip-code information supports local matching and construction of lagged neighborhood performance measures.
Mortgage states: The data contain seven mortgage states: current, 30-, 60-, and 90+-day delinquent, foreclosed, REO, and paid off.REO and paid off are treated as absorbing states in the transition analysis.

3 Deep Learning Model

The paper models mortgage-state transitions dynamically with a neural network that maps loan, mortgage-state, and economic variables to conditional probabilities. This architecture is designed to capture nonlinearities, interactions, multi-period dynamics, and correlation from shared geographic and economic exposures.

Dynamic formulation: The model represents mortgage performance as transitions among states over discrete periods such as months.Possible states include current, delinquent, foreclosed, REO, and paid off.
Dynamic formulation: The transition function hθ maps the current mortgage state and explanatory variables to conditional probabilities for the next state.The explanatory variables include loan-level and economic factors.
Portfolio implications: The dynamic formulation captures loan-to-loan correlation from geographic proximity and common economic factors.Pool-level outcomes can be computed by simulating individual loans and aggregating their cashflows.
Neural-network architecture: A neural network replaces a pre-specified linear relationship with learned nonlinear transformations of the explanatory variables.With no hidden layers, the specification reduces to logistic regression; additional layers fit more complex patterns.
Neural-network architecture: The softmax output is a probability distribution over the mortgage states.The output dimension equals the number of possible states.
Multi-period forecasting: Multi-period transition probabilities are obtained by combining transition matrices across periods, with expectations taken over evolving economic covariates or using a frozen-covariate approximation.The frozen approach can reduce computation and may be accurate over shorter horizons.

4 Likelihood Estimation

The paper estimates the dynamic neural-network transition model by maximum likelihood using observed mortgage states and explanatory variables. It addresses exogeneity, overfitting, computational scale, hyperparameter selection, and out-of-sample testing.

Likelihood estimation: The dynamic model avoids fitting separate static models for every forecast horizon.The alternative would require many models for horizons such as one month, six months, and one year.
Likelihood inputs: The explanatory variables include loan features, contemporary economic factors, and lagged zip-code default and prepayment rates.The lagged geographic rates are included among the variables describing aggregate local performance.
Likelihood estimation: The model parameters are estimated by maximizing the likelihood of observed mortgage-state trajectories conditional on explanatory variables.The likelihood has an analytical form because the economic variables are treated as exogenous.
Model fitting: Regularization, dropout, and ensemble modeling are used to address neural-network overfitting.The objective includes an ℓ2 penalty, and dropout is applied in each layer during fitting.
Model fitting: Hyperparameters are selected by cross-validation, producing an architecture with five hidden layers and 200 units in the first layer.Each subsequent hidden layer has 140 units, with rectified linear unit activation.
Out-of-sample testing: Training uses data before May 2012, validation covers May–October 2012, and testing covers November 2012–May 2014.The final model is refitted on the combined training and validation sets before out-of-sample testing.

5 Empirical Results

The fitted deep learning model reveals highly nonlinear relationships between explanatory factors and borrower behavior. Borrower behavior also depends on nonlinear interactions among multiple factors.

Many highly nonlinear relationships exist between explanatory factors and borrower behavior.
Borrower behavior has nontrivial dependencies on nonlinear interactions between multiple factors.
The fitted model is used to understand how explanatory factors relate to borrower behavior.

5.1 Explanatory Power of Variables

State unemployment has by far the greatest explanatory power for borrower behavior, exceeding standard loan-level predictors. This result indicates a tighter connection between housing finance markets and the macroeconomy, while separating economic-cycle correlation from foreclosure contagion.

State unemployment: State unemployment has by far the highest explanatory power among all variables.The analysis measures explanatory power by the increase in out-of-sample negative average log-likelihood when a variable is removed.
State unemployment: Standard loan-level variables such as credit score and loan-to-value ratio have less explanatory power than state unemployment.
Macroeconomic connection: The result suggests a much tighter connection between housing finance markets and the macroeconomy than previously thought.Earlier studies used shorter periods, smaller samples, selected loan products, and narrower borrower profiles.
Correlation and contagion: Economic-cycle exposure can create substantial loan-to-loan correlation, distinct from foreclosure contagion.The model controls for contagion using lagged zip-code-level default rates, which retain some explanatory power.

5.2 Economic Significance of Variables

Economic significance measures how fitted transition probabilities respond to small changes in individual variables, complementing explanatory power measured jointly across transitions. Prepayment and delinquency exhibit different influential variables, and linear models can misstate these sensitivities when relationships are nonlinear.

Measurement: Economic significance is measured by the magnitude of a fitted transition probability’s derivative with respect to a variable, averaged over a representative sample.A sensitivity of z means a small variable change of ∆ approximately changes the transition probability by z∆.
Prepayment: Original and current outstanding loan balance have strong economic significance for prepayment.Other influential variables include original interest rate, interest-rate differentials, house-price appreciation, loan age, FICO score, lagged prepayment rates, and state unemployment.
Delinquency: Recent borrower behavior dominates economic significance for transitions from current to 30 days delinquent.The key variables include the number of recent delinquencies and the number of times the borrower was current during the past year.
Nonlinearity and sensitivity: Linear logistic regression understates the sensitivity of loan-balance variables and overstates the sensitivity of interest-rate variables for prepayment.The paper attributes these inaccurate sensitivities to nonlinear relationships.
Measurement: Explanatory power evaluates variables jointly across all transitions, whereas economic significance concerns a particular transition and small changes in one variable.Thus, unemployment can dominate explanatory power without being the most economically significant variable for a specific transition.

5.3 Nonlinear Effects

Fitted prepayment and delinquency probabilities vary nonlinearly with influential variables, often changing sharply near economically meaningful thresholds. Prepayment is especially sensitive to current balance and incentives, while delinquency is strongly path-dependent and related to loan size.

Prepayment behavior: Most fitted prepayment relationships are highly nonlinear when other covariates are held at their dataset averages.These plots represent an average borrower or loan while examining one variable at a time.
Prepayment behavior: Prepayment probability exceeds 60% for the smallest current balances, falls to about 10% near $8,000, and is relatively flat above $15,000.Current outstanding balance is the most influential prepayment variable.
Prepayment behavior: Prepayment probability is non-monotonic in original loan balance.The paper suggests large balances may be harder to refinance or may reduce the perceived benefit relative to transaction effort.
Prepayment behavior: Prepayment probability is nearly zero for negative refinancing incentives, rises across 2.5−5%, and accelerates above 5%.The incentive is current interest rate minus the national mortgage rate.
Prepayment behavior: After home prices double, a borrower is 50% more likely to prepay, while the effect levels off beyond roughly 250% appreciation.
Delinquency behavior: The likelihood of 30-day delinquency rises from under 0.5% with no recent delinquencies to about 4%, 7%, and 12% after one, two, and three 30-day delinquencies.This pattern indicates path-dependent mortgage credit risk.
Delinquency behavior: Delinquency probability falls from around 3.5% for a $100,000 loan to 1.5% for a $200,000 loan and about 0.1% for loans of $300,000 or more.

5.4 Interactions between Variables

Mortgage risk depends on interactions among variables, not only their standalone effects. These interactions produce borrower-specific regimes for prepayment and delinquency, including sharp differences by balance, credit quality, interest rates, and recent payment behavior.

Interaction measurement: Cross partial derivatives measure how the effect of one variable depends on the size of another variable’s change.Higher-order interactions can also be estimated using finite differences.
Prepayment interactions: Original interest rate, state unemployment, loan-balance variables, and FICO score strongly interact in explaining prepayment.Contour plots display joint effects of influential variable pairs and triplets.
Prepayment interactions: For any original balance, prepayment becomes more likely as current balance decreases; at a given current balance, it is more likely for larger original loans.Borrowers with large original loans and small current balances are most likely to prepay.
Prepayment interactions: Prepayment becomes more likely with both lower current balance and greater refinancing incentive, but very large balances can suppress prepayment despite strong incentives.
Prepayment interactions: For 30-year loans, prepayment is highest when borrowers were current throughout the prior 12 months and lowest when they were never current.
Delinquency interactions: Original interest rate, interest-rate differentials, term, FICO score, balances, and past delinquency behavior strongly interact for 30-day delinquency.
Delinquency interactions: When FICO exceeds 800, delinquency is insensitive to interest rates; below 650, delinquency rises with interest rates and is more sensitive at higher rates.
Delinquency interactions: Monthly delinquency likelihood exceeds 25% when borrowers are rarely current, have low FICO scores, and face high interest rates.More frequent current payments reduce delinquency probability across FICO scores and interest rates.

6 Out-of-Sample Analysis

Out-of-sample analyses show that nonlinear deep networks improve mortgage-state prediction, especially for prepayment, and improve pool-level forecasting and portfolio selection. Model complexity helps in-sample fit but requires regularization and careful ensemble design to control over-fitting.

Goodness-of-Fit: Deeper networks improve in-sample fit but do not always improve out-of-sample fit because higher capacity increases over-fitting and estimation difficulty.The out-of-sample loss follows a U-shaped pattern as network depth increases.
Goodness-of-Fit: Dropout improves out-of-sample fit for deeper networks by providing regularization against over-fitting.The benefit is observed for networks with greater numbers of layers.
Goodness-of-Fit: An ensemble of eight 5-layer networks outperforms the other formulations in fit, while larger ensembles provide only marginal additional gains at linearly increasing computational cost.The result indicates a trade-off between incremental fit improvements and ensemble size.
Goodness-of-Fit: The largest likelihood-ratio improvement comes from moving from a linear model to one permitting nonlinear relationships between borrower behavior and explanatory variables.All reported depth comparisons are highly significant, with p-values below 0.01.
Loan-Level Predictive Accuracy: Prepayment transitions show the strongest nonlinear effects and the largest out-of-sample AUC improvements across mortgage-state transitions.Greater network depth generally increases AUC, with the most striking gains for transitions to paid off.
Investment Management: Nonlinear loan ranking is evaluated by selecting mortgages with the highest predicted probability of remaining current to construct portfolios seeking uninterrupted cashflow.The comparison uses a 5-layer nonlinear network and a 0-layer linear network.
Pool-Level Accuracy: The 5-layer network produces fairly accurate pool-level prepayment predictions and distributions, with lower variance and means closer to observed prepayments than the linear model.These results indicate that nonlinear effects extend from individual borrower behavior to correlated prepayment events.

7 Conclusion

The paper finds that mortgage borrower behavior is highly nonlinear and that local economic conditions, especially unemployment, are central to explaining risk. These findings imply that mortgage risk assessment and MBS diversification should account for interactions and geographic exposure to economic cycles.

Contribution: Over 120 million US mortgage records from 1995–2014 are analyzed with a nonlinear deep learning model of multi-period borrower-state transitions.The model incorporates loan-, borrower-, economic-, and demographic variables measured from national to zip-code levels.
Findings: Borrower behavior exhibits ubiquitous interaction effects, with prepayments showing the strongest nonlinear effects among mortgage events.The findings challenge the adequacy of many linear models used in prior work.
Findings: State unemployment has the greatest explanatory power among the factors examined, and borrower sensitivity to unemployment depends on current unemployment and borrower characteristics.This supports a tighter connection between housing finance markets and the macroeconomy than previously thought.
Implications: Substantial loan-to-loan correlation can arise from borrowers’ shared exposure to economic cycles, motivating geographic diversification of mortgage-backed security risk.The paper distinguishes this common-cycle channel from foreclosure contagion.

A Data Cleaning

The data-cleaning procedure removes records lacking essential mortgage characteristics or containing impossible state transitions, while retaining other observations with missing features when possible.

Missing Data: Samples missing FICO, LTV ratio, original interest rate, or original balance are removed from the analysis.These variables are treated as required core mortgage characteristics.
Missing Data: Other missing features are retained because some may reflect information not supplied by borrowers rather than reporting errors.Debt-to-income ratio is given as an example of a feature often unavailable for subprime borrowers.
Data Errors: Records containing impossible monthly mortgage-state transitions are removed from the CoreLogic dataset.Examples include current to 60-days delinquent and 30-days delinquent to 90+ days delinquent transitions.

B Implementation of MLE

Maximum-likelihood training is designed for an extremely large, dynamic dataset and relies on minibatches, parallel hardware, and software optimizations. The implementation addresses memory, computation, and data-ordering constraints during neural-network estimation.

Computational Scale: The training set contains roughly 3.5 billion monthly samples, nearly 2 terabytes of data, and 272 features per mortgage.Only a fraction of the data can fit in RAM at one time.
Hardware Acceleration: GPU acceleration provides more than a 10x speedup over CPU for training the deep learning models.The speedup comes from parallelizing many simple operations such as matrix multiplication.
Software Optimization: Torch, optimized C routines, and single-precision arithmetic are used to reduce training time and computational demands.These choices target frequently used neural-network operations and numerical storage costs.
Optimization: Minibatch gradient descent with momentum fits the models by repeatedly loading data subsets and updating parameters using minibatch gradients.The learning rate determines the step size for each update.
Data Ordering: Minibatches must be sampled randomly to keep gradients unbiased, but scrambling mortgage data is computationally challenging because records are organized by region, time, and loan type.Biased gradients may prevent convergence and reduce accuracy.

C Hyperparameter Selection

The authors select neural-network hyperparameters by comparing validation-set log-likelihoods across a sparse grid, including network depth and training choices. Cross-validation favors a five-hidden-layer architecture and a specified learning-rate schedule.

Selection procedure: Hyperparameters are selected by training each grid configuration and choosing the one with the lowest validation error.The search includes layers, ℓ2 penalty, learning-rate schedule, batch size, and nonlinearity.
Learning-rate schedule: The learning-rate schedule must balance oscillation risk from excessive rates against slow learning from insufficient rates.The chosen schedule is reported after testing several alternatives.
Selected architecture: Cross-validation selects five hidden layers, with 200 units in the first layer and 140 units in each subsequent layer.The same passage reports an initial learning rate of 0.1 and a half-life of 800 epochs.
Selected nonlinearity: The rectified linear unit outperforms the sigmoid nonlinearity in performance and convergence speed.The selected ReLU is defined as σ(x) = max(0, x).

D Finite-Difference Approximation of Sensitivities

The paper estimates mortgage-risk sensitivities from fitted transition probabilities using dataset-level finite differences. These calculations quantify pairwise and higher-order interactions, while the accompanying data summaries describe broad loan coverage, mortgage states, products, and predictive evaluations.

Sensitivity estimation: Finite-difference estimators compute sensitivities of fitted transition probabilities using relevant mortgages and times for each transition.The resulting sensitivity is aggregated across the entire dataset rather than evaluated at one representative point.
Pairwise interactions: Pairwise interaction measures isolate changes in fitted transition probabilities that independent shifts in two covariates cannot explain.For smooth functions and small equal perturbations, the interaction formula approximates a second-order sensitivity.
Higher-order interactions: Third-order interaction measures detect three-covariate effects not explained by the sum of pairwise interactions.A nonzero value indicates nonlinearity beyond a quadratic model, and the framework can generalize to higher orders.
Empirical analyses: The paper evaluates transition probabilities, variable importance, sensitivities, interactions, network depth, AUC, and portfolio-level state and prepayment predictions.The reported tables include transition matrices, leave-one-out explanatory power, gradient-based sensitivity analyses, interaction analyses, and out-of-sample comparisons.
Nonlinear patterns: Empirical prepayment rates vary nonlinearly with FICO score and loan age, including a plateau near 500 FICO points and spikes at one-, two-, and three-year ages.These patterns motivate models capable of learning nonlinear functions of borrower and mortgage variables.