Challenges to Model Validation Under the New CECL Standard

Challenges to Model Validation Under the New CECL Standard

Challenges to Model Validation Under the New CECL Standard


On June 16, 2016, the Federal Accounting Standards Board (FASB) finalized the Current Expected Credit Loss (CECL) standard, which incorporates significant differences from previous standards and is slated to go into effect at the end of 2019 for SEC filers and the end of 2020 for others. The differences in the new standard require banks and other lenders to make major changes to their loss reserve models, and in some cases develop new models entirely. The validation requirements of the new models entail a substantial level of effort as well.


Most organizations will validate their new models prior to their parallel run periods. Banking organizations that are subject to Supervisory Guidance on Model Risk Management (SR 11-7/OCC 2011-12) are required to validate all models, but many other institutions are likely to follow this guidance as leading industry practice. Because CECL models are different from previously developed models for Allowance for Loan and Lease Losses (ALLL) and stress testing in several respects, the validation activities must be specifically tailored for CECL models. We discuss some of these considerations below.

Less Prescriptive Requirements and a Wide Validation Scope

According to SR 11-7/OCC 2011-12, model governance must include model development, model implementation and model usage. In addition, each validation activity must be adapted to the FASB CECL standard. Furthermore, the CECL requirements are not highly prescriptive, and institutions are encouraged to make their own modeling choices according to their size, resources and risk profile. Therefore, a variety of validation approaches are under development. In the absence of cut-and-dried requirements, each institution must make a number of choices when developing its CECL framework, processes and models. Each institution is required to support its choices with reasonable and sufficient evidence for each step in that process.

Determining Scope

For CECL model validation, the scope should include portfolio sampling and applicability, portfolio segmentation, model design, variable selections, model performance, macro-environment scenarios and a loss projection period. It is important that model validators have not only knowledge and insights pertaining to credit data, credit risk modeling, financial assets and products, financial and business functions related to ALLL decisions but that they also understand the CECL guidance. Finally, as CECL requires loss estimation over the lifetime of an asset and takes into account the impact from macroeconomic environment change, longer history of portfolio data will be required to support credit loss modeling at different stages of the asset age and economic cycle. Model validators should verify and determine whether the data are sufficient to support the modeling methodology that the institution has chosen to estimate its loss reserve under CECL.

Validating Segmentation and Methodology Selection

Validators should review the segmentation process first to evaluate whether the portfolio segmentation supports CECL financial disclosure requirements and takes into account data availability, product features, risk factor feasibility and risk differentiation. Once the segmentation is determined as reasonable, validation can then assess the reasonableness of the modeling methodology for each segment.

In general, in order to determine whether the modeling methodology is reasonable, a validator should consider all relevant and available information and challenge the methodology from various perspectives. For example, from the perspective of the business, a validator should understand the portfolio features, the types of products and the potential risk factors of the portfolio. With this information, the validator can challenge the decision to use segment-level or loan-level models. In general, segment-level models are more applicable to portfolios with highly concentrated customer types, products and collateral profiles while loan-level models are often used for more complex and diversified portfolios.

The validator should also be able to challenge the model selection decision based on data quality and sufficiency. In some cases, the available data may not support loan-level modeling, and a portfolio-level model or qualitative adjustment approach may be needed.

Credit Risk Model Methodology Selection Process for CECL

Validating the Model Design and Target Variable

Besides the model methodology selection process, the model design also requires attention from validators. One of the main elements of model design is the definition of target variables for CECL modeling purpose for each specific portfolio. These definitions will have a direct impact on the model output and result in different loss estimates, and therefore it is important that the model design incorporates the appropriate definitions. For example, the default for a portfolio may be defined as “when a debtor files for bankruptcy, and loans are more than 179 days past due.” To determine the reasonableness of the definition of default, it will be necessary for the validator to review the internal accounting policy of the institution and conduct discussions with business-line subject-matter experts to understand the charge-off identification process from a business perspective.

In practice, Protiviti has observed that charge-off definitions are not always aligned with internal accounting policies or the proper business perspective. This may result in miscounting charge-offs from a CECL perspective, which can lead to overestimates or underestimates of the loss reserve. Experienced validators can challenge model owners regarding separation of business judgment effects from accounting policy definitions. Such effects can be reflected or adjusted through a qualitative adjustment process.

For portfolios with models that include prepayment effects, the definition of prepayment should also be considered. A general example of a prepayment definition could be “when all the unpaid principal balance in the loan is paid off prior to the loan maturity date.” However, in practice, a model may utilize data which incorporates business perspective into the prepayment definition. The validator should also assess the reasonableness of prepayment definitions and determine their impact on model outputs and resulting CECL loss estimates.


While CECL model validation requires activities similar to those used for incurred loss and stress-testing models, a new scope and plan must be developed that combines elements from both of these and includes new aspects specific to the CECL requirements.

Todd Pleune, Managing Director, Model Risk, Protiviti


Validating the Overall CECL Framework

For CECL models, validators should not focus exclusively on the credit risk modeling, but should also include the overall CECL loss estimation framework, which is different from the incurred loss approach for many aspects of the validation scope. In general, the elements of the framework include qualitative adjustments, reasonable and supportable period, mean reversion approach, asset lifetime and macroeconomic scenario design.

Institutions may use qualitative or quantitative approaches to support the above decisions. However, no matter how institutions make these decisions, the processes and evidence should be documented in detail, and validators should be able to challenge and replicate the decision results accordingly.

To determine a reasonable and supportable period, multiple practices exist. For example, the determination may be based on the accuracy of the macroeconomic factors projection or on the accuracy of the projection from the credit loss model. No matter which method an institution applies, the rationale, the supportive numbers and the results should be documented in detail to defend the decision of reasonable and supportable period.

Another element to be considered is the qualitative adjustment process. This process is also part of the loss reserve decision framework for the existing incurred-loss model. Protiviti suggests that the validator review the qualitative adjustment process based on the CECL accounting guidance and regulatory feedback. In addition, the review should determine whether:

  • Each of the qualitative adjustment factors in the CECL guidance has been considered.
  • The rationale for each factor adjustment has been documented and supported by evidence.
  • The impact from each factor adjustment on the final loss reserve has been presented and the reasonableness has been discussed.
  • The adjustment rationale and trend align with management’s perspective and are consistent through time. 

Compared to CCAR and Basel II implementations, CECL applies to more institutions, and those organizations are required to integrate more models into the process. CECL also offers opportunities for finance and risk professionals to work closely together. Individuals with the background and experience from both sides will be in high demand as the model validation work gets underway.

Shaheen Dil, Managing Director, Model Risk, Protiviti


How We Help Companies Succeed

To validate CECL models, a validator needs broad experience and knowledge from different areas, including credit risk modeling, financial products, accounting guidance, and credit and financial function processes. Protiviti’s Model Risk professionals bring deep knowledge and hands-on expertise to assist financial institutions with developing, validating and assessing their ALLL and CECL models and processes.

Many institutions must have their new CECL models implemented and capable of generating loss reserves by the end of 2019. Due to the tight time frame, model validation is likely to be done under significant time pressure. Protiviti has developed accelerators to test the significance of predictive variables and to generate benchmark model(s) for model performance comparison, which can significantly speed up the validation process and help institutions meet the stringent deadlines.

About Protiviti

Protiviti is a global consulting firm that delivers deep expertise, objective insights, a tailored approach and unparalleled collaboration to help leaders confidently face the future. Protiviti and our independently owned Member Firms provide consulting solutions in finance, technology, operations, data, analytics, governance, risk and internal audit to our clients through our network of more than 75 offices in over 20 countries.
We have served more than 60 percent of Fortune 1000® and 35 percent of Fortune Global 500® companies. We also work with smaller, growing companies, including those looking to go public, as well as with government agencies. Protiviti is a wholly owned subsidiary of Robert Half (NYSE: RHI). Founded in 1948, Robert Half is a member of the S&P 500 index.

About Protiviti's Data Management And Advanced Analytics Practice

Our Data Management and Advanced Analytics Practice helps companies harness data to make intelligent business decisions that drive performance and growth while managing risks. We use leading practices to deliver customized solutions in the areas of data strategy, advanced modeling and analytics while building meaningful relationships and delivering excellence to our clients.

Contacts

Shaheen Dil
Managing Director
+1.212.603.8378
[email protected]
Charlie Anderson
Managing Director
+1.312.364.4922
[email protected]
Suresh Baral
Managing Director
+1.212.471.9674
[email protected]
Bill Byrnes
Managing Director
+1.312.476.6387
[email protected]
Todd Pleune
Managing Director
+1.312.476.6455
[email protected]
Yimin Yang
Senior Director
+1.212.603.8315
[email protected]
Benjamin Shiu
Director
+1.212.603.8372
[email protected]
 

Ready to work with us?

Shaheen Dil, Protiviti
Shaheen Dil
Managing Director
+1.212.603.8378
Linked
Todd Pleune
Todd Pleune
Managing Director
+1.312.476.6455
Linked