Deep Learning for Mortgage Risk

1 Introduction

• The deep learning model captures nonlinearities and interactions without pre-specified functional forms, using over 120 million mortgages and more than 3.5 billion loan-month observations from 1995 to 2014.The data cover prime and subprime loans, more than 30,000 zip-codes, diverse mortgage products, and loan-specific and zip-code-level macroeconomic variables.Source: §1 · p. 2
• State unemployment has the greatest explanatory power, with its nonlinear effect varying by unemployment levels, borrower characteristics, and other risk factors.Prepayment sensitivities differ across FICO groups and unemployment ranges, while high- and low-FICO sensitivities converge above 11 percent unemployment.Source: §1 · pp. 3, 4, 6, §1.1 · p. 10
• House-price appreciation reduces delinquency less at higher appreciation levels, but its protective effect is stronger under high unemployment and can eliminate unemployment differences after prices double.These interactions suggest that accumulated home equity can insulate borrowers from labor-market shocks.Source: §1 · p. 4

1.1 Related Literature

• Unemployment dominates loan-level and other variables in explanatory power, providing evidence of a tighter connection between housing finance markets and the macroeconomy than prior studies found.The result is based on a longer sample period and richer mortgage performance and covariate data than earlier work.Source: §1 · p. 5, §1.1 · pp. 8–9

1 Introduction

• The model yields superior out-of-sample multi-period risk predictions at individual-loan and mortgage-pool levels and supports selecting performing mortgage investment portfolios.Its predictive accuracy is presented as useful for applications including mortgage-backed-security valuation.Source: §1 · p. 6

1.1 Related Literature

• The model estimates full conditional transition probabilities across current, delinquent, foreclosure, REO, and prepaid states, capturing frequent transitions that binary default-prepayment models omit.Observed examples include loans returning from foreclosure to current and loans remaining delinquent without entering foreclosure.Source: §1 · p. 6, §1.1 · p. 10

2.1 Loan Performance and Feature Data

• Over 120 million mortgages cover roughly 70% of US originations from January 1995 through June 2014 across more than 30,000 zip codes.The data contain 25 million subprime and 93 million prime mortgages.Source: §2.1 · p. 11, §2 · p. 11
• Each mortgage combines detailed origination features with monthly performance records, yielding roughly 3.5 billion observations and 272 explanatory variables per monthly sample.Origination variables include FICO, LTV, DTI, balances, rates, product and property types, while performance variables track delinquency, balances, foreclosure, REO, payoff, and recent delinquency history.Source: §2.1 · pp. 11–12

2.2 Local and National Economic Factors

• Local and national economic factors are matched to mortgages by geography, including housing prices, unemployment, income, mortgage rates, vintage, and lagged zip-code default and prepayment rates.The lagged geographic rates are intended to capture potential contagion in nearby mortgage defaults.Source: §2.2 · pp. 12–13

2.3 Mortgage States and Transitions

• Mortgages transition among seven states, while REO and paid off are treated as absorbing states; empirical matrices show frequent movement between current and delinquency states.Even extremely delinquent loans can return to current, and foreclosure-to-current transitions are relatively frequent.Source: §2.3 · p. 13

2.4 Nonlinear Effects

• Prepayment rates exhibit significant nonlinear relationships with interest-rate incentives, FICO score, time since origination, and LTV ratio, motivating a nonlinear loan-performance model.Prepayment propensity plateaus above a FICO score of about 500, while spikes occur at one, two, and three years since origination.Source: §2.4 · p. 14

• The model represents each mortgage’s evolution across discrete-time states with conditional transition probabilities driven by loan-specific and macroeconomic variables.The transition function defines the probability of moving from the current state to each possible state, forming a conditional transition matrix.Source: §3 · p. 15
• Common economic and geographic variables induce correlated transition probabilities across loans, allowing the model to capture loan-to-loan dependence.Source: §3 · p. 15
• A neural network replaces fixed basis functions with learned nonlinear feature transformations, enabling flexible transition probabilities beyond logistic regression.Additional layers and units extract increasingly complex features, including interactions among covariates.Source: §3 · pp. 16–17
• The dynamic formulation produces multi-period transition matrices by composing one-period matrices, with expectations computed using simulated future covariates or frozen economic covariates.Freezing economic covariates reduces computation and can be accurate over shorter horizons, while deterministic mortgage elements continue evolving.Source: §3 · pp. 17–18
• Pool-level prepayment and cash-flow distributions are obtained by simulating individual loans and aggregating their outcomes.With economic covariates frozen, Poisson or central-limit approximations can provide closed-form pool-level distributions.Source: §3 · p. 18
• One shared neural network models all transitions and products, improving parsimony while pooling data to estimate nonlinear features common across transitions.This architecture avoids fitting separate networks for each initial state, product, or borrower class.Source: §3 · p. 18

Likelihood formulation

• Maximum likelihood estimation uses an analytical likelihood for observed mortgage states that depends on observed covariates but not their exact parameterization.The covariates are treated as exogenous, so their likelihood can be handled separately from that of the mortgage states.Source: §4 · p. 19

Overfitting control

• The model addresses neural-network overfitting with ℓ2 regularization, dropout, and ensemble averaging across bootstrapped, randomly initialized networks.Ensemble averaging reduces idiosyncratic prediction variance across individual networks.Source: §4 · pp. 19–20

Implementation

• Efficient maximum-likelihood fitting handles many observations and explanatory variables using GPU parallel computing on an Amazon Web Services cluster.The implementation is described in Appendix B.Source: §4 · p. 20

Hyperparameter selection

• Cross-validation selects network depth, layer widths, activation, regularization, and fitting parameters; the chosen architecture has five hidden layers with 200 then 140 units per layer.The rectified linear unit activation yielded better performance and faster convergence than the sigmoid.Source: §4 · p. 20

Evaluation design

• The model is trained before May 1, 2012, validated through October 31, 2012, refit on both sets, and tested out-of-sample from November 1, 2012 through May 31, 2014.Variables are normalized using means and standard deviations computed only from the training set.Source: §4 · p. 20

• The fitted deep learning model is used to analyze how explanatory factors relate to borrower behavior.Source: §5 · p. 21
• Many highly nonlinear relationships exist between explanatory factors and borrower behavior.Source: §5 · p. 21
• Borrower behavior depends nontrivially on nonlinear interactions among multiple factors.Source: §5 · p. 21

Explanatory power

• Explanatory power is measured by the increase in out-of-sample negative average log-likelihood when a variable is removed.A variable is considered highly explanatory when its removal substantially worsens fit and its information is not largely contained in remaining covariates.Source: §5.1 · p. 21
• State unemployment has by far the highest explanatory power among all variables, exceeding standard loan-level predictors such as credit score and loan-to-value ratio.The result is based on an unprecedented sample spanning several economic cycles over two decades.Source: §5.1 · p. 21
• Unemployment remains critical after controlling for zip-code lagged default rates, indicating additional loan-to-loan correlation through borrowers’ exposure to the local economy beyond foreclosure contagion.The findings emphasize geographically diversifying mortgage-backed security risk beyond conventional borrower characteristics.Source: §5.1 · pp. 21–22

Sensitivity Measurement

• Economic significance is measured as the average magnitude of a fitted transition probability’s derivative with respect to each variable over a representative sample.A sensitivity of z implies an approximately z∆ change in transition probability for a small variable change ∆.Source: §5.2 · p. 22

Prepayment

• Loan balance variables have the strongest economic significance for prepayment, overshadowing interest rates, loan age, LTV-related factors, and other previously emphasized variables.Other significant prepayment variables include original interest rate, interest-rate differentials, house price appreciation, FICO score, lagged prepayment rates, and state unemployment.Source: §5.2 · p. 23

Delinquency

• Recent borrower behavior dominates transitions from current to 30-day delinquency, with 30-day delinquencies during the prior year the most economically significant predictor.The influence of recent delinquency and current-status counts indicates strongly path-dependent borrower behavior and makes standard loan and borrower characteristics less influential than previously thought.Source: §5.2 · p. 23

Nonlinearity

• Nonlinear relationships make single linear-model coefficients unreliable measures of economic significance, with logistic regression understating loan-balance sensitivity and overstating interest-rate sensitivity for prepayment.The model’s sensitivities depend on multiple parameters representing nonlinear connections between variables and transition probabilities.Source: §5.2 · pp. 22–23

Interpretation

• Unemployment is economically significant for prepayment but is not the most sensitive variable for that transition, despite being dominant in explaining outcomes jointly across transitions.The distinction reflects different questions: local sensitivity for one transition versus joint explanatory power across all transitions.Source: §5.2 · p. 23

5.3.1 Prepayment Behavior

• Prepayment probability is highly nonlinear across influential variables, including current and original balances, interest-rate incentives, and zip-code house-price appreciation.Figures 2 and 7 examine one-dimensional relationships for an average borrower/loan with other covariates fixed at dataset averages.Source: §5.3.1 · p. 24
• Prepayment exceeds 60% at the smallest current balances, falls to about 10% near $8,000, and becomes relatively flat above $15,000.Borrower behavior changes most significantly around $8,000 and is relatively insensitive to sufficiently high balances.Source: §5.3.1 · p. 24
• Original-balance effects are non-monotonic: prepayment rises through about $350,000, peaks near 5% around $600,000, then levels off and declines.Large balances may be harder to refinance or may reduce borrowers’ willingness to pursue refinancing.Source: §5.3.1 · pp. 24–25
• Prepayment jumps near a zero interest-rate differential, rises linearly from 2.5−5%, accelerates above 5%, and levels off when house-price appreciation exceeds 250%.After home prices double, borrowers are 50% more likely to prepay, consistent with selling to realize gains and move into more expensive homes.Source: §5.3.1 · p. 25

5.3.2 Delinquency Behavior

• Delinquency probability is highly nonlinear and path-dependent: it rises from under 0.5% with no recent delinquencies to about 4%, 7%, and 12% after one, two, and three.For original balances, delinquency is about 3.5% at $100,000, 1.5% at $200,000, and flat near 0.1% for loans of $300,000 or more.Source: §5.3.2 · pp. 25–26

5.4 Interactions between Variables

• Cross-partial derivatives quantify how one covariate’s effect on a transition changes with another covariate, with finite-difference and higher-order extensions provided.The method measures pairwise interactions and generalizes to third-order interactions.Source: §5.4 · p. 26

5.4.1 Prepayment

• Prepayment strongly reflects interactions among original interest rate, state unemployment, loan balances, and FICO score, revealing complex joint effects beyond standalone importance.Tables 14–15 identify influential pairs and triplets, while Figures 9–10 use contour plots to describe their joint effects on prepayment.Source: §5.4.1 · p. 26
• Prepayment is highest for borrowers with large original loans but small current balances, especially when incentives are high, whereas large balances or negative incentives suppress it.For any original balance, prepayment falls with current balance; borrowers with small balances and high incentives are most likely to prepay.Source: §5.4.1 · p. 27
• Prepayment depends on payment history and macroeconomic conditions: frequently current 30-year borrowers prepay most, while high unemployment makes prepayment very low and insensitive to interest-rate variables.The three-way interaction among original interest rate, interest-rate incentive, and unemployment is strong in favorable conditions but largely disappears when unemployment is high.Source: §5.4.1 · pp. 27–28

5.4.2 Delinquency

• Delinquency rises with past delinquency, current balance, and lower FICO scores, reaching almost 20% or 10% monthly for especially high-risk borrower groups.Borrowers delinquent more than eight times in twelve months have nearly 20% monthly transition rates, while low-FICO borrowers with large balances reach 10%.Source: §5.4.2 · p. 28
• Delinquency interactions are strongly nonlinear: interest-rate sensitivity is negligible for FICO scores above 800 but rises with rates below 650, especially when borrowers are rarely current.In the low-FICO regime, sensitivity is greater at high interest rates; frequent current payments reduce delinquency across FICO scores and rates, with risk exceeding 25% monthly in the riskiest combination.Source: §5.4.2 · pp. 28–29

6.1 Goodness-of-Fit

• Out-of-sample loss is U-shaped in network depth: dropout selects five layers, no dropout selects three, and an eight-network ensemble outperforms the other formulations.Dropout improves fit for deeper networks by reducing over-fitting, while larger ensembles offer only marginal gains at linearly increasing computational cost.Source: §6.1 · pp. 29–30
• Likelihood-ratio tests are all significant at p-values below 0.01, with the largest fit improvement arising when moving from a linear to a one-layer nonlinear model.This provides additional evidence that mortgage performance data contain nonlinear relationships between borrower behavior and explanatory variables.Source: §6.1 · p. 30

6.2 Loan-Level Predictive Accuracy

• Out-of-sample AUCs generally increase with network depth, and one-layer models improve prepayment AUCs over linear models by 3% to 30%, with the largest effects from severe delinquency.Prepayment transitions exhibit stronger nonlinear effects than other transition events, particularly transitions from 90+ days delinquent to paid off.Source: §6.2 · p. 31

6.3 Mortgage Investment Portfolios

• The nonlinear 5-layer portfolio produces a 46% smaller loss than the linear 0-layer portfolio after one year, demonstrating investment benefits from modeling borrower nonlinearities.The portfolios are selected from 100,000 test mortgages by predicted probability of remaining current; the five-layer portfolio also has a significantly lower one-year prepayment rate.Source: §6.3 · p. 32

6.4 Pool-Level Accuracy

• Five-layer networks accurately predict pool-level prepayments, with lower variance and means closer to observed prepayments than the 0-layer model.Observed prepayments fall near the five-layer distribution’s center but in the linear model’s tail, indicating nonlinearities also capture correlated borrower behavior.Source: §6.4 · p. 33

Model and data

• The nonlinear deep learning model analyzes origination and monthly performance records for over 120 million US mortgages originated between 1995 and 2014, incorporating loan-, borrower-, economic-, and demographic variables across geographic levels.The model captures multi-period borrower state transitions using variables measured nationally, by state, county, and zip code.Source: §7 · p. 34

Borrower behavior

• Borrower behavior is highly nonlinear, with ubiquitous interaction effects; prepayments show the strongest nonlinear effects, driven especially by interacting original and current loan balances.Jumbo borrowers with relatively small current outstanding balances are most likely to prepay.Source: §7 · p. 34

Macroeconomic effects

• State unemployment has the greatest explanatory power, while borrower sensitivity to unemployment depends strongly on current unemployment and varies significantly across borrowers.These patterns reveal interactions between unemployment and standard borrower characteristics, rather than constant or isolated effects.Source: §7 · p. 34

MBS implications

• The dominant role and nonlinear influence of unemployment expose MBS portfolios to economic-cycle correlation, requiring geographic mortgage-risk diversification and nonlinear hedging and rating practices.Rating agencies must account for nonlinear borrower behavior when assessing MBS exposure to adverse macroeconomic conditions.Source: §7 · p. 34

Missing Data

• Samples missing FICO, LTV ratio, original interest rate, or original balance are removed, while other missing features receive indicator variables rather than triggering sample deletion.This distinguishes likely reporting errors in key contract or borrower variables from incomplete borrower-provided information.Source: A Data Cleaning · p. 35

Erroneous Events

• Samples with impossible delinquency transitions are removed, and subsequent monthly observations are excluded after servicing release or unknown, discontinued, or invalid status reports.These events are treated as infrequent data errors or as points after which the mortgage’s reporting becomes unreliable.Source: A Data Cleaning · p. 35

Computational scale

• Roughly 3.5 billion samples and nearly 2 terabytes of data make conventional desktop training impractically slow, whereas up to 10 models can be trained simultaneously within a few days.Only a fraction of the dataset fits in RAM, and individual networks contain tens of thousands of free parameters.Source: B Implementation of MLE · pp. 35–36

Parallel computing

• More than 10x GPU speedup and distributed AWS training enable parallel model estimation without replicating data across nodes.The implementation uses up to 10 single-GPU nodes connected to a central data server.Source: B Implementation of MLE · p. 36

Software optimization

• Torch’s optimized C routines and single-precision arithmetic accelerate training, halve memory requirements, and leave parameter estimates practically unchanged.The specialized library speeds commonly used neural-network operations by an order of magnitude.Source: B Implementation of MLE · p. 36

Optimization algorithm

• Minibatch gradient descent with momentum fits models by loading data blocks sequentially, with batch size and learning rate optimized for fast, stable convergence.Each update uses the gradient computed from the currently loaded data block.Source: B Implementation of MLE · pp. 36–37

Data organization

• Randomly distributed training samples are required for unbiased minibatch gradients, so hash tables reorganize geographically and temporally partitioned static and dynamic loan data.Loan IDs map to randomly chosen destination folders, while a second hash table matches origination records with monthly performance updates without exhaustive search.Source: B Implementation of MLE · p. 37

Cross-validation

• Hyperparameters were chosen through sparse-grid cross-validation, selecting the grid point with the lowest validation error.The search varied layers, neurons per layer, the ℓ2 penalty, learning-rate schedule, batch size, and nonlinearity.Source: C Hyperparameter Selection · pp. 37–38

Learning-rate schedule

• The learning-rate schedule was treated as critical because excessive rates cause oscillations, whereas insufficient rates slow learning.The chosen schedule had initial learning rate LR0 = 0.1 and an 800-epoch half-life; each epoch contained approximately 1.5 million training samples.Source: C Hyperparameter Selection · p. 38

Network architecture and training

• 5 hidden layers, with 200 units in the first and 140 in each subsequent layer, were selected; ReLU outperformed sigmoid and converged faster.The selected batch size was 4,000, with approximately 375 gradient steps per epoch.Source: C Hyperparameter Selection · p. 38

Dataset-wide sensitivities

• The estimator for sensitivity (7) aggregates over only relevant mortgages and times, yielding transition sensitivities across the entire dataset rather than at a representative point.It uses explanatory variables and mortgage states over time to estimate the probability of reaching state v at time t + 1 from state u.Source: D Finite-Difference Approximation of… · p. 38

Pairwise interactions

• The second-order interaction estimator measures the change in fitted transition probability unexplained by independent shifts in covariates i and j.For smooth fitted functions with ∆i = ∆j = ∆ ≪ 1, 2∆^2 Interaction(u, v, i, j) estimates the corresponding second-order sensitivity.Source: D Finite-Difference Approximation of… · pp. 38–39

Scope and interpretation

• The finite-difference formulas apply to nonlinear piecewise-linear functions such as ReLU neural networks and can be generalized to fourth- and higher-order interactions.The paper empirically investigates second- and third-order interactions; third-order interactions equal zero when only pairwise interactions are present.Source: D Finite-Difference Approximation of… · pp. 38–40

Higher-order interactions

• The third-order interaction estimator detects effects among covariates i, j, and k that cannot be explained by summing their pairwise interactions.A nonzero third-order interaction indicates nonlinearity beyond a quadratic model and identifies strongly interacting covariate triplets.Source: D Finite-Difference Approximation of… · p. 39