Complying with New Supervisory Guidance on Model Risk

Complying with New Supervisory Guidance on Model Risk

The new guidance is far more comprehensive than the regulators’ previous communications on model risk and appears to impose a large burden on small and medium-sized banks. But if broken down into manageable components and focused on areas that banks haven’t paid a great deal of attention to, compliance can be both manageable and highly beneficial to a bank’s overall risk management.


The new supervisory guidance on model risk, issued in April 2011, was developed jointly by the Office of the Comptroller of the Currency (OCC) and the Board of Governors of the Federal Reserve System. The guidelines were adopted by the OCC as Bulletin 2011-12 and by the Federal Reserve as FRB SR 11-7. The new OCC bulletin supersedes OCC Circular 2000-16, which focused primarily on an independent and comprehensive model-validation function.

OCC 2011-12 and FRB SR 11-7 retain the requirements on model validation, but broaden the scope of model-risk management to include model development, implementation, use, governance, and control structure. The new guidance is philosophically consistent with Circular 200016, but it also incorporates many of the regulations put forward by the Basel Committee on Banking Supervision with respect to the new calculations for regulatory capital.

Notably, while the Basel regulatory framework applies only to the largest globally active banks, OCC 2011-12 and FRB SR 11-7 apply to all banks. And while the guidance does document some new standards, examiners have been speaking with bank managements on many of these issues for a number of years, particularly if the organizations have large, complex exposures.

The new supervisory guidance acknowledges the increased use of models in business decisions, but also notes that while they may improve business practices, they may also have the effect of increasing risk. Consider, for example, a bank having to face the costs of poor decisions that were based on incorrectly specified or misused models. These heightened risks require a firm-wide approach to model-risk management and suggest that banks should treat model risk in the same manner as they treat credit, market, or operational risk.

Banks are now required to identify the sources of model risk, assess its magnitude, and establish a framework for managing it. The new framework identifies three elements of a strong process for managing model risk:

  1. Robust model development, implementation, and use.
  2. Sound model-validation practices.
  3. A solid governance framework.

Robust Model Development, Implementation, and Use

Emphasis on the early stage of model development is a key difference in the new guidance.

  1. Model developers must ensure that technical experts work with business users so that the design, theory, and logic of the model rest on sound conceptual principles as well as established industry practice.
  2. The model development process must include rigorous assessment of data quality and relevance, particularly if the bank is using external as well as internal data.
  3. Model testing should include checking the model’s accuracy, assessing potential limitations, defining compensating controls (if available), and evaluating the model’s behavior over a range of input values.
  4. Model uncertainty, or model risk, exists with every model, and banks may need to hold additional capital against this potential risk.
  5. Every step of the process should be thoroughly documented.

Sound Model-Validation Practices

Model validation was the primary focus of OCC Circular 2000-16, and the new guidance reinforces the required practices. One key difference is the heightened emphasis on vendor models.

  1. Validation must be conducted with a degree of independence from model development and use.
  2. Banks can ensure appropriate independence through their compensation practices.
  3. Staff performing the validation should have the requisite knowledge, skills, and expertise.
  4. Validation staff should have the authority to conduct credible challenges.
  5. Models should be reviewed at least annually, but also whenever major changes are made to the model, if the economic environment changes, or if a model is put to a use different from its original intention.
  6. Validations should identify model errors, track corrective actions, and ensure appropriate use.
  7. The validation framework should include three elements:
    • Evaluation of conceptual soundness.
    • Ongoing monitoring and process verification.
    • Outcomes analysis, including back testing.
  8. Vendor models are now required to follow the same principles as in-house models, although the Bulletin acknowledges that the process may need to be modified as vendors seek to protect proprietary knowledge.

A Solid Governance Framework

Governance, policies, and reporting were required in OCC Circular 2000-16, so banks may believe they are already there with respect to these areas. However, the new guidance places heightened emphasis on the role of senior management and the board of directors in evaluating the effectiveness of a comprehensive framework for managing model risk.

  1. The bank’s board of directors is responsible for ensuring that management establishes a bank-wide approach to managing model risk.
  2. The board may delegate the responsibility for executing and maintaining an effective model-risk management framework to senior management, via appropriate committees.
  3. Board members should ensure that the level of model risk taken by a bank is within their tolerance, and they should direct changes when the risk goes beyond certain acceptable limits.
  4. Banks must have a set of effective policies and procedures in place around model-risk management.
  5. Roles and responsibilities should be clearly designated among business units, control functions, and compliance personnel.
  6. Internal audit should play a key role in evaluating whether model-risk management is comprehensive, rigorous, and effective. 

How to Comply

At first glance, the new guidance seems to call for significant investments in model validation and audit, the back-end functions of the modeling process. But by focusing on strengthening the front-end function, banks not only achieve compliance, but also greatly improve their overall decision-making and risk management. Bearing in mind the regulatory focus on building a risk management infrastructure that provides for multiple “levels of defense,” banks should create a model-risk process with distinct first, second, and third lines of defense.

First Line of Defense (Model Developers)

  1. Strengthen the model development unit by building a quality assurance (QA) function within the model development group. The QA process for model development and implementation should include the following activities:
  • Rigorous verification of data acquisition and flow through to the models. The standards used here should be consistent with Basel data-quality standards, particularly for mandatory and opt-in banks. These standards are also a good business practice for non-mandatory banks.
  • Robust change-control processes—that is, limiting access to change models once approved. Again, this would be important for Basel advanced systems qualification, but it is also a good practice for non-mandatory banks.
  • Given the importance of accounting standards for all financial processes, model developers should take into account not only the financial and mathematical conceptual framework for model development, but also the accounting framework that governs that particular model or function. Modeling groups should have either accounting experts embedded in the group, or ready access to an accounting policy group to discuss model development efforts. The signatories to model approval are front office, independent risk management, and finance. 
  • Develop a checklist of procedures that modelers and documentation specialists agree upon. Both modelers and documentation specialists should follow this checklist routinely in building, testing, and documenting models.
  • Remember: Document, document, document! The documentation should 1) encompass a “soup to nuts” approach to modeling; 2) describe the model, data, and risk factors; and 3) support the selection of the methodology that was chosen over others in the marketplace. Model limitations should be identified with compensating controls identified and accountabilities made clear so that the model is used with full understanding by management. Regulators have raised the bar on documentation, and modeling groups should be mindful of the new standards.
  1. These activities don’t necessarily mean doubling the size of the model-building—or “quant”—staff. Given the heavy demand by all banks for serious quantitative talent, especially by Basel mandatory and opt-in banks, even finding the right kind of quantitative talent can be challenging (and very expensive). Instead, this QA function should include some individuals with audit or Sarbanes-Oxley backgrounds who would work side-byside with quants to document and test models as they are being built. This approach will enable banks to comply with the heavy documentation requirements of the new guidance and greatly enhance the quality of models being built up-front, without doubling the size of the quant staff. Ultimately, it will also facilitate the work that will need to be done by model validation and audit. Large banks will need more documentation specialists than community and mid-size banks will.
  2. Create a peer-review group consisting of representatives from all modeling groups so that quants have a nonthreatening forum where they can discuss modeling assumptions, issues, and problems. This approach will greatly enhance the model-verification process by imposing a level of consistency in the way that models are conceived of, developed, tested, and implemented throughout the organization. The peer-review group can play a key role in strengthening up-front model verification. Model validation and audit could have a nonvoting presence in this group, but it is important that the quants themselves manage this group and agree on the parameters of the process.
  3. There are multiple advantages to this approach:
  • Models would have detailed technical documentation, so that even if the initial model developer leaves, others would be able to step in and duplicate the model. Moreover, validators would be able to review documentation, test-cases, and evidentiary matter in support of the test results. Meanwhile, audit and regulators would find a clear and comprehensive paper trail.
  • Management could have more confidence that models, when implemented, would be likelier to yield reliable results. No model is ever fail-safe, but if you focus on a rigorous model-implementation process with a robust QA function up-front, potential errors would be caught early, mitigating the risk of model errors affecting management decisions. 
  • By strengthening resources (both in terms of skills sets and actual boots on the ground) within the model development areas, banks give modelers themselves the ability to build, document, and test robust models. You won’t necessarily need to double the number of Ph.D. quants. It may be more useful to hire model documentation specialists (perhaps those with analytic and audit backgrounds) to work directly with quants to document models up-front. Depending on the complexity of the model, there might be a ratio of one or two quants to a documentation specialist.

Second Line of Defense (Risk)

Banks that already have a fully functioning model-validation process should be aware that this process will also need enhancement in order to comply with the new guidance.

  1. Strengthen the Model Validation Group (MVG) with either internal or external validators, or possibly both.
    • Almost all banks will need more internal staff devoted to model validation than prior to the issuance of the new guidance.
    • Banks may also consider having external validators on call (perhaps consultants or academics who would be willing to come in during periods of heavy demand on model-validation work).
  2. The MVG should compile a comprehensive inventory of all models used in the bank; the inventory should include the name and purpose of the model, any restrictions on its usage, and remediation actions if required. The model inventory should also be risk-ranked to prioritize review.
  3. The MVG should evaluate the documentation about the model, testing, outcomes, limitations, and added controls provided by the modeling groups through their QA function.
  4. The MVG should also become involved in the validation of models before they are fully completed and implemented; this will reduce the possibility of banks spending time and resources on building models where even the conceptual framework or intended use is not appropriate.
  5. The MVG should focus on enhancing the process for validating vendor models to make sure it is up to standards similar to those for in-house models. Note that this is a key difference from OCC Circular 2000-16.
  6. The model-validation process must demonstrate that models are scrutinized with adequate “credible challenges” by an independent party.
  7. It would be good (although not critical) to designate a full-time chief model risk officer (CMRO) for mediumsized and larger banks, or for banks that rely heavily on modeling (such as retail and mortgage portfolios). 
  8. The CMRO, or other senior business leader, should chair a senior-level Model Validation Committee (MVC), which reviews the effectiveness of the bank’s overall compliance with the new model-risk guidance. This committee should include senior leaders from the businesses, risk, and finance.
  9. Recall that the CMRO, or other representative of the MVG, will need to make an annual presentation to the board of directors, the group that is ultimately accountable for ensuring an effective model-validation function.

Third Line of Defense (Audit)

Internal audit will probably also need to add staff, but if the first and second lines of defense do their jobs properly, audit’s work will be made much easier.

Some audit groups may find it sufficient to develop a solid network of external quantitative experts who may be called upon to assist with technical aspects of the newly expanded model-risk review functions.

  1. Internal audit should not have to duplicate the work performed by the Model Validation Group.
  2. The function of internal audit is to evaluate the effectiveness of model-risk management practices.
  3. Internal audit must verify that the policies are appropriate, that implementing procedures are in place, and that the model-validation process is subject to suitable governance, control, and reporting.
  4. Internal audit should track any remediation plans in place.
  5. As the third line of defense, internal audit must inform senior management and the board of directors of any existing or potential problems with the model-risk management process. 

Getting Started

Here is a five-step approach that will put you on the road to compliance:

  1. Conduct an internal gap assessment to see where you are with respect to complying with the new model-risk guidance. The gap assessment can be conducted by the peer-review group, so long as all major groups in the process are involved—including model validation and internal audit. Alternatively, your bank can choose to engage an outside consultant to perform this activity. The advantage of engaging a consultant is that it lends a level of objectivity and independence to the gap assessment, though clearly the cost will be higher. Your bank can also conduct the initial gap assessment in-house and then hire a consultant to review the findings. This approach may be less expensive, and it will still provide the independent view. Internal audit could also perform the oversight of the in-house gap assessment.
  2. Set up a team to develop an action plan to close all identified gaps in order to bring the bank into compliance with the new requirements. Assign all gaps to an individual or group, with a clear timeline and accountabilities so that all gaps will be closed by a set date. Establish a preparer/reviewer structure to ensure that remediation actions meet the spirit and letter of the supervisory guidance.
  3. Have a conversation with your regulators to get them comfortable with your action plan and timeline. Remember, you won’t be compliant until the regulators say you are!
  4. Develop a training module on model-risk management so that all parties involved (modelers, business users, model validators, risk staff, finance staff, and auditors) are trained in the requirements of the new model-risk guidance. This approach ensures a consistent understanding of the requirements across all lines of business.
  5. Internal audit should track and report on progress against the action plan, as well as all remediation activities.

Conclusion

Don’t leave everything up to model validation and internal audit. Adding staff and work efforts only at the back-end of the modeling process may get the job done, but it will cost more, require more people, and add less value to frontline business decision-making than adding more resources up-front. Adding a QA process and a peer-review group at the front-end of modeling will get it done right the first time, enhance business decision-making (managers can have more confidence in models as they are rolled out), and facilitate the work that will need to be done by model validation and audit at the back end. 

About the author

Shaheen Dil is a Managing Director with Protiviti, a global consulting firm, responsible for their model risk practice as well as capital management activities. Previously, she was an executive vice president and basel implementation leader for PNC Financial Services Group. She can be reached at [email protected].

References

  1. Office of the Comptroller of the Currency. “Sound Practices for Model Risk Management: Supervisory Guidance on Model Risk Management.” OCC Bulletin 2011-12, April 4, 2011.
  2. Board of Governors of the Federal Reserve System. “Supervisory Guidance on Model Risk Management.” Supervision and Regulation Letter SR 11-7, April 4, 2011.
  3. Office of the Comptroller of the Currency. “Risk Modeling: Model Validation.” OCC Bulletin 2000-16, May 30, 2000.

Ready to work with us?

Shaheen Dil, Protiviti
Shaheen Dil
Managing Director
+1.212.603.8378
Linked