Measuring the Right Metrics and Leveraging Risk and Performance Indicators to Enhance the End-to-End Transaction Monitoring Programme

Challenges and Opportunities

Financial institutions face multiple challenges with respect to obtaining and leveraging the right metrics from their TM systems. Based on our experience, the following situations are typical:

• Not knowing what to measure. Often, institutions don’t know exactly what metrics they need to measure. This typically is a result of not understanding which metrics are used to indicate an ineffective system. To ensure that the right metrics are being utilised, a well-defined AML governance framework must be established, along with clearly articulated metrics that can be used to support the business and meet regulatory requirements.
• Not understanding the data requirements. More typically, institutions struggle with identifying and sourcing useful, consumable data. Inconsistent, duplicate or out-of-date data will result in poor quality of information from the measurements. Sometimes, the problem is operational – not knowing how to obtain the data on which to perform measurements. To ensure that the right data is used to perform measurements, institutions should a) perform an analysis to determine the correct data parameters to feed into a metric; b) ensure the defined data is suitable, i.e., available, complete and accurate; c) determine how often the data is refreshed; and d) understand the volume of data required.
• Having disparate information management systems. We often see institutions with databases that are fed information from multiple TM systems, delivering inconsistent output from system to system. These inconsistencies typically stem from information systems having different business requirements; from business rules not being applied consistently to all information management systems; or from a lack of understanding of the data structure, resulting in misinterpretation of the sourced data. More often than not, organisations struggle to identify the root cause of disparate reporting on the same metrics when data is sourced from different repositories. These organisations need to ensure that documents supporting the design of existing architecture, particularly business requirement documents, are clear, available and not written at a level that is too high or difficult to comprehend.
• Lack of a feedback loop between TM and case management systems. We also see a number of instances where institutions are unable to get the entire end-to-end picture because the MI reports generated from the TM system are not linked to the end results provided by the case management systems. This also creates issues with feeding vital information from cases resulting in suspicious activity reports and suspicious transaction reports (SARs or STRs) back to the TM system’s alert review and tuning processes. An appropriate governance structure will help identify and extract the correct data for the TM system and link case results back to the system, closing the loop.
• Managing competing requirements. Another typical problem within organisations is the inability to manage multiple stakeholders and deal with a large number of diverse business requirements. Numerous requests for similar reports can clog the system as it attempts to run multiple queries from the same data. Often, the overload stems from a lack of coordination and/or clarity when requesting information. For example, the information requirements that are passed to analytics teams for the same type of report can differ based on who is requesting the report (e.g., middle management vs. executive management). This can lead to analysts spending much of their time producing multiple MI reports instead of leveraging the information from these reports to provide better information for the business.
• Lack of clarity around broader organisational goals. TM initiatives driven by upper management may not always trickle down to middle management. Likewise, initiatives driven by middle management may not always align with the overarching enterprise strategy, resulting in disparate and potentially conflicting strategic initiatives moving forward, complicating operating models and wasting effort.

Once these challenges are overcome, however, institutions can gain a number of opportunities to leverage the right metrics to understand and enhance their TM programmes:

• Detection logic effectiveness and alert volumes. Armed with the right measurements and information, management will be able to identify underperforming detection models and scenarios (e.g., no alerts or too many alerts generated) or changes in alerts stemming from a shift in business requirements, which could highlight the need for tuning of the TM system.
• Data accuracy. MI report results that are well outside of management’s expectations may alert management to data or technical issues and prompt the institutions to address them.
• Identification of emerging risks. Reports that are accurate and based on the right metrics may provide insight into new geographic areas or transaction types that are posing increased risk to the institution.
• Staff performance and competency level. By reviewing MI report results, management will be able to gauge better the efficiency and productivity of employees and may be able to ascertain whether processes need enhancement or whether additional training should be provided to the staff charged with reviewing the alerts.

Our Point of View

Obtaining the right metrics can provide institutions with information about a number of key risk and performance indicators (KRIs and KPIs) used to gain insight into the effectiveness of the deployed TM system. These indicators can also help organisations in a number of other ways. For example, KRIs can help track an organisation’s risk appetite and also can help identify potential emerging risks (i.e., regulatory changes, industry standardisation) and drive appropriate risk mitigation activities. KPIs can help organisations analyse historical data and allow for pattern recognition and forecasting – which can be utilised in the areas of alert management and capacity planning.

KPIs and KRIs can and should be leveraged to uncover and address areas of inefficiencies in the end-to- end TM process. Below are examples of indicators pointing to an ineffective TM system or process:

 

 

 

 

To get to these correct metrics, we recommend institutions take the following steps with regard to data, governance and reporting:

Data Considerations

• Review data sets to verify the accuracy, completeness and availability of appropriate data elements (parameters) feeding into reports.
• Review historical records associated with a repository/data source to determine if any filters are impacting the data quality. For example, for data sources that are not supported by good documentation, perform testing and root cause analysis to identify filters, transformation rules, etc. 
• Establish data lineage to ensure that the appropriate data is being extracted for metric calculations and all data transformation rules are identified and assessed.
• Create a data mart to aggregate data from disparate systems and have one system of record for generating reports. Tightly integrate the TM and case management systems to leverage business intelligence developed at the investigation level.

Metrics and Governance Considerations

• Develop effective metrics using the correct parameters. Ask yourself if what is being measured is in fact what is required by the business to answer questions about risk exposure or performance.
• From a governance perspective, clearly define a process to help record the metrics being produced as part of the alert management process (e.g., false positive alerts, suspected SARs, actual SARs generated, alert volumes, etc.). Manage business user expectations and align/rationalise business requirements where possible through working sessions.
• Establish drivers for business requirements and determine if the same solution should be applied to meet similar sets of requirements.
• Centrally manage initiatives and review against enterprise strategies and goals to ensure alignment. This will minimise duplicated effort, identify opportunities for synergies between projects/initiatives and effectively leverage the right resources across the organisation.
• Establish governance committees to review project progress and identify instances of deviations from initial proposals/objectives in order to reassess effectively the value derived.

Reporting Considerations

Establish a user interface that references the data mart from which users (e.g., the business) can generate pre-established reports.

Allow users to create ad hoc reports from the user interface. Ad hoc reports call to a refreshed data set at the moment the user creates the report. This enables users to get reports with the most up-to-date data, as well as view only the data they want to view, saving users’ time.

Combining the use of metrics, data analytics, AML technology and suspicious activity monitoring can help managers and stakeholders at financial institutions to:

• Provide information on risks affecting the organisation
• Use better information in real time to ensure compliance with current local regulations
• Become aware of whether current business practices meet regulatory requirements and are aligned with organisational risk strategy
• View cross-business and jurisdictional transactions for easy identification of trends and exceptions
• Determine if existing TM systems and processes require enhancement using operational and system indicators
• Re-estimate targets for each metric and assess the operational impact of the alerts on time, cost and resources
• Provide data for applying techniques such as scenario analysis, black-box testing, data quality reviews, etc.
• Identify the root cause(s) of an ineffective programme
• Develop targeted solutions based on root cause analysis completed
• Refine the TM approach, technology, methodology and templates based on key observations, trends and identification of high-risk indicators

How We Help Companies Succeed

Our AML professionals and our team of modeling experts, including Ph.D.-level professionals with deep quantitative skills, help institutions implement and maintain a sound and robust threshold-setting and tuning methodology. We have experience with a number of AML transaction monitoring systems on various platforms, including but not limited to Actimise, Detica NetReveal AML (Norkom), Mantas and SAS AML, Fiserv, as well as a number of homegrown systems.

Our AML transaction monitoring technology services include:

• Developing and executing a sound and efficient scenario-setting and tuning methodology and approach
• Performing any or all of the following tasks by acting as an independent team:
  • AML red flag gap analysis
  • Data validation
  • Scenario logic validation
  • Threshold values validation
• Performing customer segmentation
• Recommending improvements to scenarios/thresholds

Example: Using Key Metrics to Enhance Management Information Reporting

A large global bank sought our assistance to enhance AML MI reports in order to identify improvement opportunities in its end-to-end transaction monitoring systems and supporting processes. Our integrated team of AML and Business Intelligence experts performed a data quality review, identified key metrics and developed dashboards, which successfully helped our client in enhancing its MI reporting process. Our work helped the client achieve the following:

• Enhanced usefulness and reliability of data. We generated reports on data quality and completeness, which allowed the institution to identify areas of increased risk (where data was incomplete) and re-prioritise remediation efforts to fix the data issues and increase its monitoring coverage.
• Operating effectiveness and increasingly mature TM processes. We created customised reports that provided middle management with real-time information on alert clearing productivity. These reports prompted the institution to find a different method for managing the alerts, which resulted in reduced headcount and costs.
• Management information governance framework. The governance framework we implemented allowed the institution to set in place procedures to review and update MI reports on an ongoing basis to ensure accuracy and timeliness, creating a sustainable reporting environment.
• Improved reporting to regulators. The accurate and timely MI reports on the end-to-end TM process enabled senior management to substantiate its discussions with regulators by using the MI reports to support its messages.