Quantitative vs. Qualitative Approach and Variable Selection
As a crucial part of capital planning, Pre-Provision Net Revenue (PPNR) modeling has been under the spotlight for careful review and scrutiny from the Federal Reserve Board (FRB) since 2012 for the annual Comprehensive Capital Assessment and Review (CCAR) submission. Although bank holding companies (BHCs) have become better equipped at utilizing PPNR models for their capital planning purposes in recent years, large and small BHCs continue to face challenges in meeting the FRB’s increasingly stringent requirements, as well as the banks’ own business needs.
A key task is to develop conceptually sound and robust PPNR models that perform well in stressed economic environments and capture both the BHC’s own business characteristics and the underlying macroeconomic drivers. In its simplest form, PPNR is the sum of net interest income and noninterest income, minus noninterest expense. Subcomponents may include deposit balance, loan balance, noninterest-related fees and expenses, and others. As such, PPNR projections are typically built from subcomponents. Challenges arise for BHCs when trying to model these subcomponents using macroeconomic factors and/or BHC-specific business drivers utilizing different types of estimation techniques. Although the expectation is to use a quantitative-based, data-driven approach for modeling PPNR components, in situations where certain PPNR components may be less sensitive to macroeconomic factors, judgmental and qualitative approaches are often applied.
Challenges and Opportunities
Protiviti has worked with many clients to develop and validate PPNR models and has observed various challenges consistent across BHCs. Our capabilities for addressing these challenges include assumption testing, back-testing and scenario analysis results interpretation, data limitations, modeling approach, model performance metrics and assessment.
This paper focuses on two of commonly encountered challenges: the choice between qualitative and quantitative modeling approaches and the variable selection process.
- Choice between quantitative and qualitative approaches
At the early stages of the modeling process, it is important to determine whether the modeled PPNR components for any specific portfolio are sensitive to intuitive macroeconomic factors. Building quantitative models without first examining their nature and historical trends could lead to costly model development cycles that fail the model validation process or receive scrutiny from regulators.
- Explanatory variable selection process
The main challenges during the variable selection process are finding the balance between model specifications that are built solely based on statistical property by automated algorithms compared to incorporating key business and macroeconomic drivers that are intuitive and robust in the short term as well as in the long-term future.
Our Point of View
Choosing Between Quantitative and Qualitative Approaches
Protiviti observed several cases where PPNR models were built solely based on a quantitative approach, and, as a result, generated poor model fit and model performance.
From our experience, one of the reasons for poor modeling results is that the given component has a low correlation with macroeconomic variables. In theory, many PPNR subcomponents, such as asset balance, deposit balance and interest income, should be sensitive to changes in the macroeconomic environment and therefore, can be projected using quantitative models based on macroeconomic variables.
In practice, however, whether a BHC’s PPNR component is correlated with the macroeconomic environment, the results depend on other factors, such as the bank’s business strategy, customer profile, footprint, product nature and segmentation granularity of the PPNR component.
For example, it is generally believed that asset growth will increase during an economic recovery, and decrease during a recession. However, it is possible that asset growth may not be sensitive to the national-level macroeconomic environment for BHCs that focus on regional markets with product offerings that are impacted significantly by the local economy and population profiles, such as life cycle, size and income.
As another example, deposit balance growth would be expected to increase when interest rates rise and decrease when the interest rates fall. However, because of product design and target customers, each deposit product could have different sensitivities to interest rate changes. Without appropriate segmentation, certain types of deposit products can make overall deposit balance movement less sensitive to changes in the macroeconomic environment. It can also cause counterintuitive movement between the overall deposit balance and changes in the value of macroeconomic factors.
Protiviti suggests that BHCs analyze their portfolios carefully and apply appropriate segmentation structure prior to determining whether to use quantitative or qualitative approaches to project specific PPNR components. With appropriate segmentation, BHCs can select the proper modeling approach for each component.
Furthermore, once the segmentation is determined, BHCs should assess whether the projected components demonstrate meaningful trends throughout the business cycle. Some projected components could show stable trends without dramatic changes through the years. Following the above example, after the segmentation process, BHCs may find that some types of deposit products, such as checking account balance, could be quite stable for years, and only demonstrate occasional growth rate changes due to business strategy or acquisition activity. However, other deposit accounts, such as certificates of deposit, or deposits designed to support working capital, more likely could be by interest rates and the overall economic environment.
In the above example, a bank may decide to use a qualitative approach to project the checking account balance, but use a quantitative modeling approach to project the balance of the certificate of deposit. A similar choice of approach could occur with other components as well.
When a bank decides to apply a qualitative approach to project a PPNR component, significant evidence should be provided to prove that the component has very limited correlation with macroeconomic factors.
Explanatory Variable Selection Process
After determining the modeling approach for each segment, the next step of model development is the variable selection, which should take into account both qualitative and quantitative perspectives to select the final set of variables for the model.
A general variable selection process includes applying automated variable selection algorithms to identify several combinations of statistically significant variables, sharing those combinations of variables with business stakeholders, and finalizing the selection based on business intuition and model performance. The figure below demonstrates a high-level variable selection and finalization process.
It is suggested that modelers include business stakeholders in every stage of the development process — including early stages. Through early-stage discussions, the modelers can learn the nature of the portfolio and the potential macroeconomic drivers of the portfolio based on the opinions from business stakeholders.
Then, modelers can take into account business knowledge, along with the results of statistical analyses, to identify several groups of business-intuitive and statistically significant explanatory variables to further discuss with the business stake-holders. When assisting clients developing models, Protiviti applies its significant industry experience to both conduct effective communication with business owners and accelerate the model development process. For model validation engagements, Protiviti can also effectively challenge the variable selection process based on our extensive experience.
Furthermore, the second discussion with the business stakeholders could result in more than one preferred group of variables. Modelers should perform the appropriate outcome analysis, such as back-testing, stability analysis and scenario analysis, and share the results with the business stakeholders to finalize the model.
How We Help Companies Succeed
Protiviti has extensive experience in assisting CCAR banks with PPNR model development, validation and audit. Our dedicated professionals advise clients on a range of PPNR model development processes, including portfolio segmentation analysis, methodology selection, variable transformation and selection, outcome analysis, sensitivity analysis, and scenario analysis.
We also guide clients in making reasonable business judgements for different model development steps, such as determining the appropriate segmentation, selecting intuitive explanatory variables and interpreting scenario analysis outputs. In addition, we leverage our expertise to raise effective challenges in validating and auditing PPNR models.