Return on Quality

The title and central theme of this blog is “return on quality”, which we broadly define as the benefits to be gained from an intelligent and continuous approach to improving mortgage loan quality. 

We said in an earlier post that we would try to formulate “return on quality” and as a step in that direction, we offer a Cogent white paper called “Quality Performance Benchmarking” that was originally developed for an audience of mortgage quality control professionals. 

Image by jacob earl

In this paper, we talk about the prevalence in the mortgage industry of a production maximization mentality, in which metrics and compensation are centered on volume; the potential hazards of this mentality; guidelines for estimating the costs of poor quality, (the inverse of the return on quality); how to reward good quality; and how to craft appropriate performance metrics, or benchmarks.  The second part of the paper talks in depth about one of the most powerful tools for benchmarking performance, control charts.

This white paper was written in 2002.  Nothing has changed in the methodology.  But in the last couple of years, the eyes of most of us in the industry have been opened to the dangers of focusing exclusively on volume, volume, volume.  We welcome your comments.

Statistics is baffling enough without being footloose with terminology.  So let’s clear up what we mean by a statistical sample.

The term “Statistical Sample” has a very specific meaning in the Cogent system.  It refers to a sample that is randomly selected from the entire population of loans eligible for a particular sample type (aka “audit shell”).  The suggested sample size is calculated every period by the system and is designed to yield a 95% confidence and 2% precision over 12 months.  This is the standard originally established by FNMA, FHLMC, and HUD for lenders who qualify to substitute ’statistical sampling’ for the traditional 10% random sample.

The generic term ’statistical sample’ is not very meaningful, in and of itself.  It simply refers to a sample in which some statistical principle has been employed, without defining which principle.  For example, it could refer merely to a randomly drawn sample, without specifying what population is being drawn from or how much precision will be achieved across what period.

Image by johnbullas
Rick Astley reference 

To illustrate: most Cogent ProductionQC clients have at minimum a “Production” sample type, for which all loans originated in a particular period (typically a month) are eligible.  When a Statistical Sample (in the Cogent definition) is randomly drawn from this population, all loans have the same probability of being selected.  No distinction is made between loan type, loan source or any other loan characteristic.  It is intended  to establish a baseline of overall loan quality across the organization.

In order to achieve a 95% confidence and 2% precision for a particular category of loan, it is necessary to go beyond the “Statistical Sample”.  For example, in the Cogent system, to achieve this standard for all FHA loans originated, define a Targeted Query (Loan Type = FHA) and run the query.  The resulting screen displays all qualifying loans, including qualifying loans  that were randomly drawn previously in the “Statistical Sample” or any other samples in this period.  These count towards the total required.  Use the embedded Cogent Statistical Calculator to calculate the required (”suggested”) sample size for the period.  From the suggested sample size, subtract the number of qualifying loans that have previously been sampled and enter the result in the Sample Size box. The Cogent system will then randomly select the entered number of loans from the qualifying loans.

The Cogent “Stratified Sample” is in effect a pre-defined Targeted Sample.  Most typically, the Cogent system stratifies originations by Source or Channel and automatically tracks and calculates the sample size required for each stratum (Source or Channel), net of qualifying loans randomly drawn in the “Statistical Sample.”  Over 12 months, the Stratified Sample achieves 95% confidence and 2% precision for each stratum.  In Targeted Samples, this automated operation is performed by the user, using Cogent’s embedded tools.

Thus, in order to leverage the Cogent system’s sampling optimization, clients should begin sampling from the broadest category (all loans eligible) to the most narrow category (e.g., individual underwriters).  In this way, all loans selected in previous broad categories are counted towards ever narrower categories, minimizing the number of loans to be sampled and audited.

One of our favorite websites is www.informationisbeautiful.net, where David McCandless, an independent London-based “visual & data journalist” (his words) practices the art of data visualization and information design. 

We like the way he conveys information, often multi-layered, with the minimum of words.  As he puts it, “I’m interested in how designed information can help us understand the world, cut through BS and reveal hidden connections, patterns and stories underneath. Or, failing that, it can just look cool!”

It’s that last bit that we want to talk about briefly here.  One of David’s recent creations is a graphic that compares, side by side, point by point, the assertions of “The Global Warming Skeptics” against “The Scientific Consensus”. 

This particular visualization is more wordy that most of his work.  For that reason, one is inclined to think that a lot of research went into it.  And that is the case.  Every familiar argument made by the “skeptics” seems to have been researched and refuted.  If you’re a believer in global warming, you might think that this is close to the final word on the debate. 

Yet one look at the comments section at the bottom of this page suggests otherwise.  It’s worth spending a few minutes examining the visualization and then the comments.  That should be enough to convince you that a beautifully presented argument is not proof.  It has to be backed by solid evidence.  And if there’s any debate that lacks conclusive data, it has to be the global warming issue, where complex meteorological phenomena meet millennial time spans in a cauldron of scant measurement.

Statistics can serve as a good sleep aid, particularly when they’re presented as a simple row or table of numbers.  That’s the way most of us have encountered statistics, with the figures often morphing into zzzzzzzz’s.  But it is possible to make statistics come alive.  The secret is in helping the audience visualize the data in context so that they can quickly derive meaning.  And today’s media tools make this easier than ever.  When we find interesting visualizations of data or statistics, we will share them with you.  Here’s one to start us off: 

If you can’t see the video above, you can find it here:

http://www.ted.com/talks/hans_rosling_shows_the_best_stats_you_ve_ever_seen.html