A common theme we are seeing across the industry and a key area of focus for regulators is model risk and institution’s abilities to identify, report and monitor the impacts. The PRA have engaged with organisations to gauge the impact on current models, including questions concerning the current behaviour of IRB/IFRS 9 models. This has focused on identifying specific concerns for particular segments; longer term impacts of models; specific considerations around PD, EAD and LGD; information provided to model governance committees; and direction received and the necessity for recalibrations and/or redevelopments.
Model Risk should be, and generally is, included in a financial organisation’s Risk Appetite statement. The reliance on models in decision making, external reporting and balance sheet recognition highlight the potential consequences if risk levels exceed those seen as acceptable. However, whilst the importance of model risk and the magnitude of the consequences of ‘getting it wrong’ is readily accepted, there is generally less understanding throughout the organisation of what model risk means and how to quantify it.
For many years model risk primarily focused simply on the performance from back testing the models in place and indeed, in many organisations this rudimentary approach endures. However, the nature of Covid-19 has highlighted that this approach is not fit-for-purpose when model risk is at its highest i.e. times of change, when the historical performance is likely to have little bearing on future performance. Therefore, there is an urgent need for the function of model risk to evolve significantly in order to consider the wider universe of the models, including data, governance, resource capacity, weaknesses and limitations etc.
A model risk framework that is not-fit-for-purpose in times when risk levels are heightened, is simply not fit for purpose.
— Chris Warhurst, 4most Partner
Whilst this limitation has been highlighted by Covid-19 it is not new and simply reinforces the argument that all risk measures and tolerances need to be scrutinised, well developed and in place to account for all environments. A model risk framework that isn’t fit-for-purpose in times when risk levels are heightened, is simply not fit for purpose.
An evolved view of model risk allows the creation of dashboards presented throughout the governance process, including the Board Risk Committee, to provide a greater understanding of all risks to models and their uses. Therefore, as part of developing an enhanced model risk framework, consideration should be given to wider measures including but not limited to data, resource capacity, process efficiency and governance controls. Through having a wider view of the model risk landscape, it is possible to identify unseen elements of model performance and also detect early warning indicators, for example, long term low resource levels are likely to cause impacts to maintenance and governance controls in the future.
Each company’s model risk reporting will be bespoke to the environment that they operate in, however an illustration of a board reporting deck is shown below with example areas for consideration.
Clearly, to allow triggers to be assigned to each area requires tolerances to be assigned, and these will vary by organisation in terms of the breadth and rigidity desired. However, examples of amber or red tolerances could be:
Performance of Models: One or more backtesting triggers are red
Governance: One or more highly materiality models, implemented outside of Governance / two models outside of periodic review schedule
Data and Process: Three or more high/medium materiality issues with no remediation
Resource: Headcount < 85% for more than three months
Development: One or more material project development statuses are amber or green
A key advantage of this multi component view is that it allows executive level insight and challenge to both the current and potential model risk, as well as insight into some of the root causes and interaction.
So far, we have considered how model risk is considered and reported on an aggregated basis. However, it should be expected that underpinning this is a more detailed part of the framework where individual model risks are identified and where appropriate remediation is considered and proposed.
In some circumstances the identification and remediation may be more simplistic and/or non-technical, for example, low resource levels with the remediation of hiring of staff. However, for complex risk and issues, there should be a desire to apply a similar framework i.e. a component-based approach at a more granular level. This has been particularly apparent in the current Covid-19 environment.
Throughout the recent unprecedented environment, models have not reacted as expected. For example, risk scores have improved due to suppressed arrears rates and bureau scores have increased; macro-economic models have output unrealistic responses. The response to this has been to apply judgemental overlays to impairment models, reduce the decision-making power of automated processes etc. but this has generally been done without a full understanding of the underlying reason, primarily the model risk. This has led to further questions regarding double counting of adjustments, whether all deficiencies have been accounted for and offsetting impacts between deficiencies.
Through a more detailed model risk framework, including identification, reporting and remediation, it is possible to gain a holistic view of the challenges being faced and the optimal way to address them. The below schematic gives a simple overview of the approach to identify and address model performance risk issues in the most efficient manner.
A key element of this process is the aggregation of deficiencies and associated remediation. This allows for efficient remediation of multiple deficiencies whilst maintaining a clear linkage to the original items.
In the below example, individual deficiencies have been identified and labelled in Data (D1 – D5), Calculations (C1 – C6) and Management overlay (M1 – M4). These have then been considered in combination with potential remediation options, of which two are shown below.
This process allows for a clear flow and alignment between the remediation and the deficiencies. This is particularly advantageous where remediation takes the form of overlays and adjustments, as it allows for transparency as to the root cause of the overlays being applied. Therefore, more comfort should be felt at the executive level with regards to the approval of any final numbers relative to the issues that are inherent, as well as greater certainty in removing particular overlays as issues are resolved.
This concise approach through identification and aggregation ensures a full understanding of the individual limitations within the model performance landscape. In ‘normal’ circumstance it would be expected that this level of the framework would not be reported throughout the governance process, for example, to the board and instead contribute to the overall RAG status. However, in times of greater model risk stress, it does facilitate the creation of reporting that allows a clear understanding of the outstanding actions against the identified deficiencies, as shown below.
The adoption of a framework, as outlined, in advance of Covid-19 would clearly have provided a significant advantage, with the responses to the challenges faced internally and the communication to regulators. In lieu of this, we have seen tactical approaches to both the identification of model risk and associated remediation, for example, qualitative assessments of final outputs and post model adjustments. However, these have generally been quite broad and generic without the underlying detail and transparency a more advanced approach offers. As it is likely that there will still be great uncertainty for a significant period of time, it should be seen as a priority to implement a more evolved view of model risk. This can be done in a staged approach with concentration being given to today’s priorities, with the remainder of the framework following. This will allow the organisation to better address the challenges we currently face and be significantly more prepared for the next.
