Common mistakes in the statistical analysis of credit decisions

Abstract

The modern science of probability and statistics provides many tools of considerable value in credit analysis. This is not surprising since statistics and credit analysis have much in common. For example, a primary concern in statistics is deriving techniques for selecting a sample with certain attributes including restrictions on risk. The equivalent issue in credit analysis is developing procedures for constructing a receivables portfolio with limited risk. Although there are potential gains from the use of statistics in credit analysis, there are also possible problems. Many studies have found that even educated and experienced people are prone to particular mistakes in the interpretation of statistical information. The following examples demonstrate these errors in situations that are common in credit analysis.

Predicting the Future

The consistent result of numerous studies is that important indicators of economic success follow a random walk. The most famous example is that of common stock prices: changes in stock prices are independent in the sense that changes in the future do not depend on changes in the past. Since stock prices have no memory, studying their history is of no use in predicting their future.

There is often a reluctance to accept the notion that economic statistics vary in a random fashion. A common assertion is that stock prices are not random variables but reflections of earnings, growth prospects and other important characteristics of a firm or industry. However, this argument misses the important point that the random walk concerns changes in values. A random walk is consistent with a business environment in which chance events generate changes in economic circumstances. The following example applies this concept to credit analysis in an analysis of historical loss rates.

Figure 1 shows the Federal Deposit Insurance Corporation estimate of the percent of loans charged off by U.S. commercial banks from the first quarter of 1985 through the third quarter of 2001. One possible interpretation of these figures is to see an upward trend until 1992, a sharp downward trend until 1995, and a slight upward trend until 2001. The identification of the current trend often becomes a basis for predicting future loss rates. For example, the recent upward trend supports the conclusion that loss rates will continue to increase slightly in the near future. In fact, further analysis suggests that the random walk model offers a good representation of this historical pattern of loss rates.

Figure 2 depicts the quarter-to-quarter changes of the loss rates shown in Figure 1. The average value of these quarterly changes is zero and there is no correlation between the changes in one quarter and the next. In general, loss rates show no affinity for a particular value and change randomly from year to year. With an increase just as likely as a decrease, the best forecast of the future loss rate is the current rate.

Assigning Probabilities

Determining the probability of an uncertain event is a crucial element in any decision involving risk. The usual examples of probability are one chance in six of throwing a particular number on a die or one chance in fifty-two of drawing a particular card from a deck. Although the concept of probability is fairly intuitive, there are common problems in deriving probabilities from historical information. For instance, the following example shows the importance of interpreting probabilities correctly.

This table separates 1,000 companies into four categories according to age and failure. The first column of figures shows that of 990 firms that did not fail, 96 were active for five years or less and 894 were in business for more than five years. The second column indicates that of 10 firms that failed, 4 were new and 6 were old. The final column shows the overall distribution of firms by age: 100 are new and 900 are old. The figures in the first row indicate 4 of 100 new firms failed. The second row shows that 6 of 900 old firms failed. The data in the last row indicate that 10 of the total of 1,000 companies failed, a rate of 1 percent.

The important probability that a new firm will fail is found using the data in the first row. Four of 100 new firms or four percent failed. The inverse of this failure rate is found in the second column. The probability that a failed firm is new is 4 of 10 or 40 percent. The central point is that these inverse probabilities are very different. The failure rate of new firms is not prohibitively high and certainly much less than the 40 percent chance that a failed firm is new.

A second example shows the mistake of confusing inverse probabilities in the interpretation of credit scoring. Suppose that the failure rate of firms is about I percent and that a credit-scoring program has a 90 percent success rate in predicting whether a business will succeed or fail. If the creditscoring program predicts a particular firm will fail, what is the probability that this firm will actually fail?

Assume that 1,000 firms apply for credit. The last column indicates that about 10 (1 percent) of these firms will fail and 990 (99 percent) will succeed. The next-to-last column shows the credit-scoring program will identify 108 firms as failures. The program incorrectly identifies 99 (10 percent) of the 990 successful firms as failures and correctly identifies 9 (90 percent) of the 10 failing firms as failures. In summary, only 9 of the total of 108 firms identified as failures will actually fail, about 8 percent.

The Law of Large Numbers

A fundamental tenet of statistics is that conclusions based on large samples are more reliable than those derived from small samples, This principle accords with common sense, but many studies indicate that people have difficulty recognizing its application in practical situations. As an example, assume that experience with many problem accounts over many years indicates that there is a 50-50 chance of resolving a delinquency with no loss to the firm. Suppose one credit manager handles three of four problem accounts without loss, and another twelve of twenty accounts. Since a 75 percent success rate (3 of 4) is much higher than a 60 percent rate (12 of 20), most people decide that the evidence is stronger in the first case that the credit manager is doing a good job. However, a formal statistical analysis indicates that there is a 30 percent chance of realizing three or more successes with four accounts and only a 25 percent chance of achieving twelve or more successes with 20 accounts. The results of flipping a fair coin are equivalent to this situation. The chances are very good of getting three heads with only four tosses of a coin. With many tosses the chances of getting a result so different from the true odds of 50-50 are very small.

Credit managers should be concerned about how the law of large numbers affects the evaluation of their performance. Suppose that on average 10% of accounts become delinquent. For a credit manager who approves 100 accounts, there is a 30% probability that 12% or more will become delinquent. However, for a credit manager who approves 400 accounts, there is only a 10% probability that 12% or more will become delinquent. The credit manager with the larger number of accounts has a definite advantage.

Another basic principle of theoretical statistics that relates to performance evaluation is the so-called regression to the mean: in any situation that involves uncertainty an extraordinary outcome is likely to be followed by a more ordinary result. As an example, suppose two credit managers have a long history of resolving 50 percent of delinquent accounts without a loss. In the current year one manager handles 12 of 20 accounts successfully while the other resolves only 8 of 20 without loss. The first manager receives a large bonus for outstanding performance while the second receives no bonus for the year. What is their likely performance in the coming year?

In this example, the odds favor a decrease in the performance of the first manager and better results for the second. When this happens there is a natural tendency to infer a cause and effect relationship between the bonus decisions and the changes in performance. The operation of the regression principle prompts the conclusion that rewards do not work but punishments do.

Framing the Question

Over the years credit managers have developed modern techniques for collecting information, analyzing financial statements, and forecasting cash flows. These sophisticated procedures for evaluating risk provide an objective basis for the analysis of credit decisions. The following example illustrates the paradox that decisions often depend not on these complex procedures for analysis but on the simple matter of framing the question.

Suppose a $3 million account is past due and demanding immediate payment from the firm would save $1 million. Granting an extension and allowing the company an opportunity to correct its problems has a one-third chance of saving the entire $3 million and a two-thirds chance of saving nothing. In this situation most people favor insisting on immediate payment in order to realize the certain savings of $1 million. Now consider the following situation.

Assume there are two approaches to dealing with a $3 million account in serious trouble. Taking immediate action to force the firm into liquidation would lose $2 million. Allowing the company to maintain its operations has a two-thirds chance of losing $3 million and a one-third chance of losing nothing. In this case most people decide to allow the firm to continue its operations in the hope of avoiding the sure loss of $2 million. The paradox is that the two situations are the same except for the wording: the first is in terms of dollars saved while the second is in terms of dollars lost. In the first case people choose the certain gain of $1 million and avoid the risk inherent in allowing the company to continue in operation. In the second case people choose the risky alternative in order to avoid the certain loss of $2 million. The basic lesson is to try rewording questions to ensure that decisions do not depend on the framing of the problem.

The general finding of many studies is that people avoid risk when seeking gains but choose uncertainty over certain losses. Credit managers overseeing the activities of others should consider an important implication of this finding: in problem situations people will tend to take risks that they would avoid in other circumstances. The natural focus in problem situations is on the potential financial exposure and the natural tendency is to prefer any opportunity to avoid a loss.

Conclusion

The preceding examples demonstrate common mistakes in the analytical assessment of risky situations. It is important for credit managers to consider how these possible errors might influence their evaluations of uncertainty in practical situations. These common mistakes include:

-Trying to impose trend and structure on random data

-Confusing inverse probabilities

-Forgetting the basic regression principle that extraordinary events will usually be followed by more ordinary results

-Believing that small samples are just as reliable as large ones

-Reaching different decisions according to the wording of gains and losses in the question

By: John K. Ford, D.B.A.

John K. Ford, D.B.A. is the Nicolas Salgo Professor of Business Administration at Maine Business School, University of Maine, Orono, Maine. He also serves as a Research Fellow for the Credit Research Foundation. John graduated from the U.S. Military Academy and served as a field artillery battery commander in Germany and Viet Nam. After the service he earned an MBA at the Wharton School and worked as a credit analyst for two years at U.S. trust in New York City. He then went on to the Harvard Business School where he received his DBA. He has been teaching finance at the University of Maine for the past twenty years.

Copyright Credit Research Foundation Second Quarter 2002
Provided by ProQuest Information and Learning Company. All rights Reserved