How to check if a rating model is valid [Part 1]
One of the key elements in the development of a methodology for rating or credit scoring is understand the accuracy of the results: only an accurate model can be put into production!
The characteristics of a good method of assessing creditworthiness are:
- an adequate distribution of failures between rating categories;
- a high discriminatory power;
- a comparison between the theoretical probability of default and rates actual default.
- adequate matrices migration and consequent triennial cumulative default rates
We see in detail how to address each point; in this part of the blog we will see the first two points, while in the next publication will study the following two.
Distribution of bankruptcies
Check that historically the failures have affected only the companies which received a low rating when they were still operating, is definitely a reliability index of the model. This analysis acquires a capital importance: it is necessary to establish that the guesswork and postulate underlying the model are actually valid.
The easiest way for this type of analysis is to see the evolution and distribution of failed enterprises in relation to those not failed, evolution that must be followed over the years.
Indicating, as usually in the literature, with IG (Investment Grade) aggregation class AAA to BBB, with NIG (Non-Investment Grade) classes from BB to CCC and D (Distressed) the lower classes to CCC, the division of bankrupt is the one shown in the figure. It is interesting to note that the portion of the companies considered risky (Distressed) increases almost exponentially approaching insolvency.
The discriminative power
Like many authors propose the best statistical measure of discriminated ability of a credit scoring model is the Gini index, which often is joined to the graphical representation of the ROC curve, which allows you to conduct an analysis of two types: qualitative and quantitative. The qualitative analysis consists of making the comparison curves of two different ratio and it is observed whether one overhang the other.
Below the graphic representation of the curve and the definition of the accuracy ratio.
As can be seen the three curves identify two areas indicated in the figure as A and B; the relationship between the two areas is obtained by the Gini index (called Also accuracy ratio AR):
where we wanted to indicate with f (x) curve ROC and p the default rate observed (ie the ratio between the number of bankrupt companies and the total number of companies in the population). Obviously the Gini index is a value that varies between 0 and 1.
Following the evolution Gini and ROC for the rating method MORE modeFinance:
Also in this case it is important to study the evolution of the accuracy over the years; should normally studied at 1.2 and 3 years from the date of bankruptcy.
What is useful is to have the reference values on the accuracy of the rating model used here is the table.
In the next post we will see the other two points to study the best possible accuracy of rating models!