As intense scrutiny of financial models continues, many organizations struggle to balance the need for model validation with the investments (time and resources) necessary for full validations. Model verification is an alternative technique that provides valuable insight into a model’s processing in a time and cost-effective manner by focusing on the implementation, where costly and subtle errors can occur.
Quantitative models are once again under a microscope. Models are often blamed for inaccurate valuations, incorrect risk management strategies and significant investment losses. Though models alone did not cause the global financial crisis, many assign a large portion of the blame to them.
Model risk is unavoidable; no organization is immune. However, it can be managed and controlled. As such, model validations and model verifications are a growing part of financial institutions’ model risk mitigation strategies as well as regulator expectations for overall risk management of the firms.
Challenges and Opportunities
Modeling is a complex task that can lead to large financial losses if model risk is not assessed and mitigated appropriately. Modeling presents many challenges:
- Models are typically implemented into computer platforms – which limits transparency.
- A model error can have significant adverse impact on a firm’s business.
- Regulators require model validation as part of a risk mitigation strategy.
- Model validations can be a very expensive and time-consuming endeavor.
Model verification is an alternative approach to model validation that may be appropriate to mitigate model risk in some cases. Examples of model risk include corrupted input data, wrong assumptions, incorrectly applied methodology, incorrect implementation, and inaccurate and/or misinterpreted model outputs.
Any of these risks can lead to significant financial, reputational, regulatory and legal risks if models provide inaccurate or erroneous output. For these reasons, model risk needs ongoing governance and mitigation efforts, including effective model validation processes.
Regulatory requirements such as OCC 2000-16 and Basel II require models to be validated on a periodic basis as part of a bank’s model risk mitigation program. Financial institutions also recognize the benefits of model validations and often have internal initiatives to reduce model risk and comply with the aforementioned regulatory requirements.
Our Point of View
A full model validation is an essential activity for critical models and is a primary defense against model risk. Model verification is an effective tool that should be part of any financial institution’s model risk management strategy. Model verifications complement a firm’s model validation program and help to reduce a firm’s overall model risk.
A model validation typically includes reviewing the model’s governance and oversight, input data, assumptions, analytics, implementation, and output and reporting. On the other hand, model verification specifically focuses on one step in the model validation process, the model’s implementation, where subtle but critical errors can occur.
Model verification typically includes a review of the model documentation, assessing the model’s design and risk factors, and customizing the model testing. This is a particularly effective tool when the model is built and implemented by an individual (who may not have the requisite technical skill set) other than the model developer. Model verification can be an appropriate tool to use for targeted purposes, such as model change management. Model verifications can be used to identify potential weaknesses in the model’s implementation, many of which can be remediated quickly. Independent model verifications are effective in determining whether the model has been implemented in accordance with the model documentation, and that the documentation reflects the model’s intended purpose.
How We Help Companies Succeed
Our Model Risk Management practice helps organizations by assessing, designing and implementing model governance programs and by conducting independent model validations and model verifications. We also develop customized quantitative models, refine and calibrate existing models, and design stress testing and scenario analysis programs to supplement existing analytics. A comprehensive approach to model risk management provides greater coverage and yields greater benefits to model developers and users.
Our model verification efforts focus on the risk of incorrect model implementation. They generally include assessing the model’s risk and customizing model testing based on the model’s design and risk factors. This is intended to reduce implementation errors and ensure the model operates as intended.
Protiviti was engaged by a major global financial services firm to perform model validation procedures on several of the firm’s proprietary models. The firm’s primary concerns were that the models were not built and implemented properly or that these models had been modified but were not tested appropriately prior to continuing use. A verification approach was appropriate to mitigate the firm’s specific model risks and saved the organization significant hours and dollars that would have been associated with full-scope validations.
For each verification, we reviewed the model’s documentation, built a replica model and benchmarked the model against the replica. Our verifications identified a number of weaknesses in the client’s model implementations, such as:
- Insufficient technical documentation to properly replicate the methodology independently
- Specific test cases within the model’s intended operating regime where the model behavior deviated from expectations due to hard-coded numbers
- Table lookups that resulted in incorrect values being assigned to instruments
- Drop-down menu with choices that caused the model to result in a calculation failure
Protiviti’s team provided recommendations to address the limitations identified and ensure the model operated as intended per the client’s expectations. The verification approach delivered valuable insight into the models’ operations in a time and cost-effective manner, which allowed the client to move quickly to address model problems.