Return on Quality

Statistics is baffling enough without being footloose with terminology.  So let’s clear up what we mean by a statistical sample.

The term “Statistical Sample” has a very specific meaning in the Cogent system.  It refers to a sample that is randomly selected from the entire population of loans eligible for a particular sample type (aka “audit shell”).  The suggested sample size is calculated every period by the system and is designed to yield a 95% confidence and 2% precision over 12 months.  This is the standard originally established by FNMA, FHLMC, and HUD for lenders who qualify to substitute ’statistical sampling’ for the traditional 10% random sample.

The generic term ’statistical sample’ is not very meaningful, in and of itself.  It simply refers to a sample in which some statistical principle has been employed, without defining which principle.  For example, it could refer merely to a randomly drawn sample, without specifying what population is being drawn from or how much precision will be achieved across what period.

Image by johnbullas
Rick Astley reference 

To illustrate: most Cogent ProductionQC clients have at minimum a “Production” sample type, for which all loans originated in a particular period (typically a month) are eligible.  When a Statistical Sample (in the Cogent definition) is randomly drawn from this population, all loans have the same probability of being selected.  No distinction is made between loan type, loan source or any other loan characteristic.  It is intended  to establish a baseline of overall loan quality across the organization.

In order to achieve a 95% confidence and 2% precision for a particular category of loan, it is necessary to go beyond the “Statistical Sample”.  For example, in the Cogent system, to achieve this standard for all FHA loans originated, define a Targeted Query (Loan Type = FHA) and run the query.  The resulting screen displays all qualifying loans, including qualifying loans  that were randomly drawn previously in the “Statistical Sample” or any other samples in this period.  These count towards the total required.  Use the embedded Cogent Statistical Calculator to calculate the required (”suggested”) sample size for the period.  From the suggested sample size, subtract the number of qualifying loans that have previously been sampled and enter the result in the Sample Size box. The Cogent system will then randomly select the entered number of loans from the qualifying loans.

The Cogent “Stratified Sample” is in effect a pre-defined Targeted Sample.  Most typically, the Cogent system stratifies originations by Source or Channel and automatically tracks and calculates the sample size required for each stratum (Source or Channel), net of qualifying loans randomly drawn in the “Statistical Sample.”  Over 12 months, the Stratified Sample achieves 95% confidence and 2% precision for each stratum.  In Targeted Samples, this automated operation is performed by the user, using Cogent’s embedded tools.

Thus, in order to leverage the Cogent system’s sampling optimization, clients should begin sampling from the broadest category (all loans eligible) to the most narrow category (e.g., individual underwriters).  In this way, all loans selected in previous broad categories are counted towards ever narrower categories, minimizing the number of loans to be sampled and audited.

Image by Rutty 

Berkery Noyes, “the only middle-market investment bank with research and M&A transaction teams dedicated to the mortgage technology, regulatory and compliance market,” has just published its 2009 Recap and 2010 Predictions For the Mortgage Technology Market. 

Providing a view from 30,000 feet, the report lists a number of high-profile M&A transactions from 2009 and offers up a few trends in the mortgage technology space, including:

• accelerating vendor movement towards becoming a complete end-to-end solution provider
• more stringent lending guidelines and regulations increasing the need for compliance, fraud prevention and risk-mitigating technology solutions
• burden of ensuring proper compliance moves to the point of sale rather than merely at closing, and falls on all mortgage industry participants

In the context of these trends, the report states that market participants no longer see compliance and auditing solutions as “nice to have” - they have become “must have” solutions.

We couldn’t agree more.

One of the ways that servicers or investors can excise nonperforming assets from their portfolios is to try to put them back to originators by claiming fraud and/or breach of representations and warrants.   Potential malfeasance like this is uncovered via forensic audits, which have become increasingly popular since the mortgage crisis hit.  While forensic auditing for this purpose has typically been limited to institutions - investors, servicers, originators and mortgage insurers - the practice has now spread to the retail borrower, as described in today’s MortgageOrb article “When Forensic Loan Audits Are Used Against Lenders.”

This phenomenon has even touched Cogent.  Recently, we’ve been approached by a handful of clients to help them extract loan audit data from their Cogent systems for the purpose of forensic loan auditing.  Typically, this is in support of litigation in which opposing counsel requires every scrap of data that could be relevant.  Without a knowledge of both the mortgage quality control workflow and the application supporting it, this can be difficult for clients to accomplish on their own, especially after layoffs have reduced knowledgeable staff. 

Although projects like this are outside Cogent’s normal scope of work, we have the expertise to help clients.  More than anything, though, this is yet another reminder of why robust quality control is imperative in today’s world. 

Image: Dena M. Roudybush

Here’s a timely and relevant learning experience: Sheshunoff, an established publisher of financial information, is offering a webinar on February 23rd called “Mortgage Servicing from A to Z: What Every Mortgage Servicer Needs to Know“. 

The blurb reads as follows: “With the continuing scrutiny on the mortgage industry and the ever-changing regulatory landscape, it is more important than ever for mortgage servicers to stay on top of the latest regulatory developments and industry trends affecting this volatile area. Join our interactive audio conference, Mortgage Servicing from A to Z, as our expert speaker gives you an insiders view into federal and state regulatory requirements, hot topics and best practices.”

The ‘expert speaker’ in question is Dena M. Roudybush, senior counsel for Compliance Counsel, PC a Virginia based law firm that is dedicated to serving the mortgage banking and financial services industries.

Cogent, for one, has seen tremendous activity in the mortgage servicing arena, with particular interest in servicing quality control.  This session sounds like a good way to get up to speed on the major changes taking shape.

Image by Peregrine Blue 

As a consequence of the financial crisis, we are undergoing wholesale change in the regulatory environment of real estate finance.  The changes are coming hard and fast and continuously, making it difficult to establish and manage new business processes.  How do you keep up?

There’s no single answer to this question.  We have clients who take it on themselves to keep abreast of regulatory changes because they want full control over their regulatory compliance.  Other lenders want to outsource the management of change and turn to third parties to keep them up to date. 

Wherever you fall in the spectrum, you will probably welcome any tools that ease the burden of staying current.  And such tools are cropping up.  For example, Wolters Kluwer has just launched ”The Reg Z Center“, a free online resource which provides an overview of the changes affecting loans covered by Regulation Z, the effective dates for changes and suggested solutions for implementing changes. 

Being a for-profit entity, the company does promote its compliance solutions on the site.  But this is in keeping with the ‘Web 2.0′ business model, whereby useful information is provided for free, interaction among interested parties is encouraged (via a forum, blog, or discussion group), and a long-term relationship is cultivated while simultaneously establishing credibility.

The Reg Z Center joins the company’s other similar resource centers for Fair and Accurate Credit Transactions (FACT) Act “red flags” and the Real Estate Settlement Procedures Act (RESPA).

It’s official.  Version 2.0 of CogentQC.NET has now been released.  You may have seen the news in Housingwire or MortgageOrb but if not, you can find the Cogent news release here. 

2009 was an unusual year for client IT departments (unless it’s actually “the new normal”, God forbid.)  After the mortgage crisis and economic meltdown, it seemed like IT staffing and budgets were reduced and lenders were stuck in neutral, reluctant to take any steps other than cost-cutting.  IT staff had too many projects to handle and consequently Cogent saw only a handful of clients upgrading to CogentQC.NET.

However, the end of 2009 saw an acceleration of activity.  We now have enough upgrade projects to take us through the first quarter, at least.  And as business gets back to normal, we have even begun to talk with clients about enhancements to their systems that will automate and optimize more of their business processes. 

It’s going to be a busy year.

If you’re interested in the world of mortgage servicing, MortgageOrb has just published an interesting overview of the major milestones of 2009 and what to expect in 2010.  With input from several industry veterans, the article surveys the changing role of the servicing function, the performance of various government programs designed to ease the mortgage crisis, the conflicts of interest between servicers and investors, the potential impact of millions more ARM resets due in 2010 and 2011, and more.  Well worth a read.

We came across a well-written article recently titled “The New Rules For Compliance In The Post-Crash Environment” by Louis Pizante, CEO of Mavent, Inc. and a veteran of the industry. 

In addition to a concise synopsis of the events leading up to the current regulatory overhaul of the mortgage industry, the article outlines some of the changes we can expect in the way compliance reviews are performed.  In short: earlier in the origination process, more electronically and with greater automation. 

Mortgage compliance and quality control reviews can and should be performed on many levels, and at different points in the loan life cycle.  Tools such as automated compliance engines, automated fraud engines and automated valuation models can flag loans that merit further review.  The next step is to apply a rigorous methodology to digging deeper in order to:

1) determine whether a complete file review confirms the automated findings;2) fix the individual loans if possible;3) identify the source(s) of the issue(s);4) select additional loans from these source(s) and conduct complete file reviews to see if there is a pattern of issues;5) generate feedback to the field and document corrective actions to fix the flaw(s) in the origination or servicing processes and/or stop doing business with the identified sources.

We are seeing more tools for automating more aspects of the mortgage life cycle.  Don’t forget, though, that it was partially the over-reliance on automated underwriting tools that got us into our current mess.  Maybe tomorrow will be different, but today we still need human beings checking to see if it all makes sense.

One of our favorite books is “How to Lie with Statistics”, a tongue-in-cheek primer on using statistics to make just about any argument you like.  It illustrates how easy it is to mislead people who are unaware of basic statistical principles.

For instance, what do you make of the headline “Median home price in Jefferson County falls by 27%”?  Sounds pretty dire when you read it in bold headline on a newspaper as you’re walking by.  You might think that home prices in Jefferson County have all fallen by 27%.  But it’s worth digging deeper.  What period are we talking about?  What are we comparing to?  What does ‘median’ mean (compared to ‘mean’, for instance)? How many homes were sold and does that make a difference in the statistic?  Are we including detached homes and condos?  None of this is clear from the headline, and there are many other questions to ask.

Among other things, basic statistical understanding gives you a sense of context, scale and precision.  In the knowledge economy, where information is coming at you from a variety of sources, that’s pretty important.  Whenever you see a statement involving statistics - or indeed, any measurement - it’s worth asking whether there is any ambiguity in the statement.  And if there is, dig deeper.

To help you get comfortable with some basics, we’ve compiled a short list of statistical concepts for mortgage quality control.  If there’s anything else you’d curious about, let us know.

Key Statistical Concepts for Mortgage Quality Control

CONTENTS:

• Sample inferences & statistical precision
• Random Selection
• Sample size estimation
• Qualitative analysis & defect rates
• Random variation & statistical control
• Sampling error & non-sampling error
• Correlation vs. Causation
   

Sample inferences & statistical precision

The fundamental purpose of sampling for Quality Control is to render judgments regarding quality of the overall loan portfolio, i.e., to infer general conclusions from the sample’s findings.  The degree to which those conclusions can be reliably inferred is measured by statistical precision. Keep in mind that the goal of Quality Control is to focus on the forest, not the trees. Accordingly, your objective is not to identify and correct errors or defects in specific loan files, but to use the incidence of such errors to infer conclusions about your loan origination process.

Critical issues:

1. Statistical inference must be based on random selection; the most common error is to draw conclusions from a non-random sample. To avoid this error, you should eliminate all non-random selections from any group used to make statistical inferences to the population.
2. Statistical precision (e.g., of two percent) must be demonstrated on the actual sample defect rate (i.e., the number of loans with defects divided by the number of loans reviewed). If you were unable to review some of your randomly sampled loan files, then the precision achieved by your process will be degraded.
    

Continued…

It’s encouraging to see the adoption of statistical methods in the world of mortgage quality control.  Done right, it can lead to enormous returns on your investment in qualiy control - what we call ’return on quality’.  But you have to do it right.  There is plenty of misinformation about statistics on the Internet and a non-expert may have difficulty sifting through what’s right or wrong, especially as applied to mortgage quality control.  

So we’d like to present the principles and methods that Cogent’s statisticians and QC experts have honed over the past 15 years.  We invite your comments.  We begin with an overview of statistical sampling in mortgage quality control, which is available as a white paper (PDF) here.

______

Statistical Sampling in Mortgage Quality Control
By Hakki Etem, CEO, Cogent Economics

Introduction

Statistical methods are well-established tools for efficiently measuring and improving product quality in a variety of industries.  Unfortunately, statistical analysis has been slow to gain acceptance in the mortgage industry, although the ability to originate the best quality product at the lowest possible cost is just as valuable to mortgage originators as it is to automobile manufacturers.  There are many reasons why the mortgage industry has avoided statistical methods, but surely one reason is the subject itself:  few disciplines can be as mind-numbing as statistical theory.

Nevertheless the most effective way for QC managers to measure and improve loan quality — at the lowest possible cost — is to employ statistical methods.  Although this means that QC managers must necessarily become familiar with basic statistical concepts, with the right tools and professional support the process can be greatly simplified. 

Continued…