OMNIMARKETS

MODEL RISK MANAGEMENT

Science for Finance2017

AGENDA

Model Risk Management at OmniMarkets

Regulatory Guidance & Requirements

Lessons Learnt15

2

7

Model Risk Managementat

OmniMarkets

2

Overview

Focus: Model Risk Management

OmniMarkets provides a range of services to the financial industry on valuation, trading and risk management with effective and

customized solutions covering these topics. By working with global leaders on a broad range of issues, we have developed a

unique global perspective on the needs and the challenges they face and their practical solutions. We combine our industry

insights with quantitative skills and proven methodologies to help our clients transform their processes and systems and meet the

ever increasing regulatory and business requirements.

Models have been at the core of OmniMarkets from the very beginning, and more recently, responding to a growing need in

the industry and leveraging our experience, we have started a Model Risk Management ("MRM") practice. Our model

coverage includes: Valuation, Risks, IM/SIMM, CCAR, DFAST, Liquidity, SVF/WSVF, FRTB, Structured finance, CECL.

The need for MRM is only growing (more automation) and the scrutiny from Regulators (OCC & FRS) is only increasing.

At the same time MRM is still maturing and evolving and more progress is to be expected in the future.

To promote this progress, OmniMarkets has started the Model Risk Management Exchange (non-profit organization) and we

are building an MRM app.

3

Leadership

Mr. Leignadier-Fahlströmlaunched OmniMarkets in 2004. Prior to OmniMarkets, he was the head of Scandinavian interest-rate derivatives trading at SvenskaHandelsbanken in London. He also headed the Securitized Credit Analytics team at CalyonAmericas and was a derivatives trader and quantitative developer for Banque Indosuez in Paris and Stockholm.

Mr. Zhang is a Managing Partner at OmniMarkets specializing in structured finance and quantitative research and development. Prior to joining OmniMarkets, Mr. Zhang was a statistical researcher at the Centre for Economic Development Research in China and an equity analyst at CITIC Securities in Wuhan, China.

Mr. Stoikov is a Quantitative Finance Advisor at OmniMarkets. He is currently the Head of Research at Cornell Financial Engineering Manhattan (CFEM), a satellite campus of Cornell University. Previously, he was a Senior Vice President of High Frequency Trading at Cantor Fitzgerald.

Mr. Tian is a Model Risk Management Advisor at OmniMarkets. He is currently the director of risk modeling at a regional bank in New England, managing a team of quants to develop all credit risk models to support operation, reporting and compliance purposes. Previously, he was a senior manger of MRM at GE Capital, and a senior analyst of MRM at Freddie Mac Internal Audit.

Pierre Leignadier Kevin Zhang Sasha Stoikov Robert Tian

4

Existing Assets

5

• Deep understanding

of what governance means for MRM

• Deep expertise on models in general;

Body of work on some of the trickiest models

• Methodologies for the whole model lifecycle

(which we use ourselves in our work)

• In-house development and replication

capacity

• Established network of seasoned

professionals (we have worked with)

available if needed

• Up-to-date knowledge of the MRM field

Templates and documentation guidance

Checklists and governance

Flexible reporting modules

Statistical libraries for results analysis

Computing power (in-house & on cloud)

Back testing libraries

Models code

Data analysis and reporting

Data and data collection libraries

A strong team

Policies, Procedures & Governance Adapted to You

Model Development and Implementation

Model Validation of Internal and Vendor Models

Inventory of All Your Models and Readiness Assessment

Holistic MRM Procedures and Management Tool

1

2

3

4

5

What We Can Offer

6

Customized, Practical and Interactive Training6

Regulatory Guidance & Requirements

7

Key Idea 1: HOLISTIC viewKey Idea 2: a PROCESS, not an event

DevelopmentValidation

Ongoing Monitoring

Model Risk Management

Internal AuditModel Risk Management (“MRM”) is still a relatively new concept.

The landmark bulletin OCC 2000-16 focused mainly on mitigating MRM by means of Model Validation. OCC 2011-12 (SR 11-7) presented a more comprehensive approach.However, this holistic view is often not yet fully adopted and implemented. This presentation briefly reviews the key elements of MRM and how they can be brought together to help the real management of model risk.

8

1st LoD(Level of Defense)

Development and Implementation

• Specify Model Requirements:

Element #1

Collaboration between stakeholders to specify model requirements

Assess methodology and determine

modeling approach

Data quality check and initial data processing

Formulate model, testing alternatives

Review with stakeholders

• Data Analysis • Modeling & Test • Document Finalization

• Literature Review• Specify Model Requirements

Loop may be necessary, depending on feedback from stakeholders and test results.

Start documentation from the beginning Finalize documentation

• Literature Review:

• Data Analysis:

• Modeling & Test:

• Document Finalization:

Joint effort between Modeler and primary stakeholders, where model requirements are defined in accordance with business needs and intended model uses.

Determining high-level modeling approach, which will guide data collection, model framework selection, implementation and documentation.

Data are collected by Modeler in accordance with the chosen modeling approach, and necessary data cleaning, review, transformation, and sampling are performed.

The model is formulated through model segmentation, variable selection, functional form specification and parameter estimation. Alternative model formulations are tested. Testing includes outcomes analyses, benchmarking analyses, and sensitivity analyses. Model limitations should also be identified.

The model should be reviewed with primary stakeholders, all sub-stages of the model development stage should be sufficiently documented to allow for outside validation processes to be performed, and Modeler should initiate steps to implement the model. 9

Element #2-1: Validation – Internal Models

Evaluation of Conceptual Soundness

The conceptual soundness is

evaluated based on the

documentation completeness,

the model effectiveness and

model stability and robustness.

Evaluation of Technical Soundness

Technical soundness is assessed

by proofreading the original

code, or by comparing to well-

validated benchmark model

with similar theories and inputs,

or by doing model replication.

This process is performed by

comparing model outputs to

corresponding actual outcomes,

and analyzing the

discrepancies found in

between.

Outcome Analysis Document Finalization

Document should articulate

model aspects that were

reviewed, highlighting potential

deficiencies over a range of

conditions, and determining

whether adjustments are

warranted.

STEP 1 STEP 2 STEP 3 STEP 4

DocumentationDocumentationDocumentation

10

Element #2-2: Validation – Vendor Models

Evaluation of Compatibility

STEP 2-2

Vendor models are

subject to the common

validation process as

internal models. Limited

access to computer coding

and implementation

details may lead the

validator to rely more on

sensitivity analysis and

benchmarking.

The input data and assumptions provided by the vendor are

needed to assess whether they are representative of the

bank’s current situation. The customization choices used by

the bank should also be documented and justified as part of

the validation.

11

Element #3: Ongoing Monitoring

BenchmarkingCompare model outputs to estimates from alternative

data or models

Override Analysis

Override performance should be tracked and

analyzed

Keep track of code changesKeep track of system integration changes

Technical Changes Verification

Input Verification

Monitor data inputMonitor risk factor Documentation

from beginning to end

Input Verification: Data input should continue to be

accurate, complete and consistent; risk factors should

consistently fall within their effective boundaries

regarding the model

Technical Changes Verification: All changes in code

should be approved, logged and can be audited;

system may need to be updated to reflect any

changes in the data or its use

Override Analysis: Overrides may be an indication

of model limitations. If the rate of overrides is high, or

if the override process consistently improves model

performance, the underlying model may need revision

or redevelopment

Benchmarking: Discrepancies between the model

output and benchmarks should trigger investigation

into the sources and degree of the differences, and

examination of whether they are within an expected

or appropriate range given the nature of the

comparison

12

Element #4: Internal Audit

Verify General Policies

• Documentation standards• Model inventory

• Model updating procedures• Clear responsibility assignment

Review Model Development

Review Ongoing Monitoring

Review Model Validation

• Methodologies are recorded specifying merits and limitation

• Data is clean and representative• Models have been tested properly

• Evaluate process for monitoring limits on model usage

• Assess supporting operational systems

• Evaluate the reliability of data

• Independency of validators• Effective challenge is being

carried out• Validations are performed in a

timely manner

Internal audit should ensure

that the model development

is conducted in a rigorous

manner and is well-

documented

Internal audit should verify

that acceptable policies are

in place and that model

owners and control groups

comply with those policies

The review should be

focused on ensuring that the

ongoing monitoring process

is comprehensive and

effective.

Internal audit should review

validation activities conducted

by internal and external

parties with the same rigor to

see if those activities are being

conducted in accordance with

the guidance of SR 11-7

4-01 4-02 4-03 4-04

13

Element #5:Bringing Everything Together

The inventory of models, policies and procedures are all mandated by bulletin OCC 2011-12.

We suggest bringing them all together with the core elements of MRM in a workflow/alert based system.Such a system greatly improves the ‘management’ in MRM and it can also be utilized as the basis for a rudimentary quantification of MRM (questionnaire & scorecards).

By doing this, you will:1. Bring your

organization in compliance with the regulations.

2. Have the necessary framework in place for model risk measurement, monitoring and tracking.

3. Be able to have an impact and “see” it.

14

Lessons Learnt

15

Some random lessons & thoughts on MRM

16

Governance:

• Evaluate current MRM practices (gap analyses) and develop remediation plans

• Firm-wide committee and governance of MRM with better integration across LoBs

• Greater attention to MRM by board • Reports appropriately scaled for board vs

senior management • Firm pays attention to model materiality,

scales MRM accordingly

Model Overlays and Vendor Models: Both are in scope for MRM

Policies and Procedures:

• Firm has policies that apply to models everywhere in the organization

• P&P tie to supervisory expectations• Evidence that P&P serve as key guiding

documents• Clear to all in the firm which P&P are the

“law of the land” • P&P regularly updated• Firm has mechanisms to assess conformance

with P&P

Special attention:

• Good policies without compliance do not help • Uneven application of better practices, even within a bank • How to keep it going will be a challenge – ensure sustainability and regular updates • Good to demonstrate within the firm that MRM is worth the substantial costs (beyond simple compliance)

Internal Audit:

• Audit makes a full assessment of the whole MRM framework

• Planning and scopes are clearly defined and communicated ex ante

• Audit focuses on evaluating processes and controls

• Audit has sufficient standing in the organization to feel comfortable pointing out shortcomings and to have them addressed

• Audit specifies clearly what was covered and what was not

• Regular reports provided to senior management and the board (or its delegate)

Parallel review process allows validation to be completed shortly after the development ends

Ongoing feedback from validators to developers, while maintaining independence

“Observation and Request Log” documents interactions

Any “showstopper” issues Identified early

Validation finished shortly after development

PreliminaryAnalysis

Model Build Final ModelDataset Construction

Kick off

Development Documentation

Preliminary Analysis

Preliminary Model Review

Final Model Approval

Data Review

Model Design

Validation Report

This optimized approach requires additional safeguards:

Preserve effective challenge Validation staff not involved in making any decisions about

the model Policy states clearly that validation’s role is to determine if

there are model limitations Model should then be reviewed on an ongoing basis 17

The three lines of defense

1st line of defense

2nd line of defense

3rd line of defense

Model owners, Model developers, Model sponsors, Model users

Independent model reviewModel oversight committee

Internal audit

Ideal state: Appropriate oversight on 1st LODCurrent state: Significant oversight on 1st LOD

Quality of execution by 1st LoD:• Understanding and adhering to regulatory

expectation and industry best practices.

Stature/Influence of 2nd and 3rd LoD:• Ability to influence the culture of the

organization.• Ability to influence senior management

and board of directors.

Qualification and incentives provided to 2nd

and 3rd LoD:• Familiarity across a broad spectrum of

technical and business expertise.• Up to date knowledge of evolving

regulatory landscape.• Ability to communicate and influence all

stakeholders.• Organization’s focus to provide

appropriate incentive to hire right talent.

A strong and competent 1st LOD results in an efficient 2nd and 3rd LOD

Increasing amount of resources dedicated

18

Aggregating risk for interconnected models

• Dependency risk (downstream models)

• Contagion risk (upstream models)

• Measure such as model centrality can be employed

Model risk needs to be captured not only in individual

models but also in the stream of interconnected models:

Critical to manage the risk of interconnected models in

its life cycle:

• Model identification and tiering

• Model development and validation: understanding upstream and downstream risk

• Risk mitigation during model usage

• Monitoring and change control

19

Contact Us

Pierre.Leignadier@OmniMarkets.com

Kevin.Zhang@OmniMarkets.com

(212)784-6487 160 Broadway, Suite 700

New York, NY 10038