Response to consultation Paper on draft RTS and ITS on benchmarking portfolios
Go back
The classes of benchmarks outlined in the RTS appear sufficiently flexible, with the only exception of the comparison with figures computed according to standardized approach.
However, it would be important that the benefits of diversification (in portfolios and geographies) were taken into account in the benchmarking assessment.
Regarding the proportionate aspect of the proposal, as anticipated previously in point 16, the usage of the first and last quartile would systematically require competent authorities to enquire on half of the contributions, therefore it appears disproportionate. When this benchmark is applied to several figures (RWA, PD and LGD), the number of banks outside the benchmark would even grow.
Such a benchmark (first and last quartile) would instead be reasonable when used as a variable in a scorecard built on several benchmarks.
We would appreciate the use of a kind of scorecard to combine different benchmarks.
Market Risk focus:
We deem the benchmarks outlined in the RTS to be fairly adequate, with the only exception of the comparison with figures computed according to standardized approach.
In general, we genuinely believe that standardized figures do not (and cannot) allow for a better understanding of internal risk measures (let alone risk underestimation, in light of their coarse modelling approach).
In addition, calculation of the standardised charges by internal model banks implies considerable implementation and maintenance costs and efforts A parallel system would indeed be required with no other use or benefit than benchmarking. Under such scenario, internal models, benefiting of several years of use-testing, validation and scrutiny, would be benchmarked against one off models not subject to the same daily controls and level of scrutiny.
All in all, we advocate for the exclusion of standardized charges from the benchmark list, at least until a new standardised approach, in the context of the “Fundamental Review of the Trading Book”, will be established.
(i) As previously anticipated, the combination of a first and fourth quartile approach used for screening for a more in-depth assessment of the internal approaches would lead to a minimum of 50% of all banks outside the limit. This approach could be useful if used only to drive initial supervisory interest but not for directly deriving conclusions. For that purpose it would be more appropriate to use a narrower sample, e.g. decile buckets and measures/thresholds based on multiples of standard deviation of benchmarking portfolio distributions.
(ii) As previously anticipated in point 16, we see no meaningful role for benchmarks based upon standardised risk weights. The objective of identifying significant and systematic underestimation of own funds requirements cannot be achieved on the basis of a simplistic indicator but rather on a thorough analysis of model assumptions and performance.
Moreover, IRBA and SA are basically different approaches. For some segments, in the SA some risk sensitivity is given, but depending on external ratings. Following external ratings would be not the desired development in risk assessment. For low risky business, IRBA risk weights in retail and corporate segments are/can be lower than the SA risk weights. Otherwise, higher risky business would be rewarded, as the risk weights in IRBA are/can be higher than the SA ones. Therefore, this benchmark is not able to identify a risk underestimation especially in the case of a high risk portfolio.
Moreover, on the operational side the implementation of such a benchmark measure may require institutions to set up appropriate IT procedures for parallel calculation currently not required for other purposes. From a technical point of view, the use of Standardized measures at counterparty/portfolio level can result even in a higher degree of “not comparability” because of the requirements of the Exposure Classes definition in the Standardised Approach: for example, for default portfolio the RW Standard is not comparable with an IRB approach requirements calculation for defaulted portfolios; an IRB exposure may results in several STD exposure classes (in case of applicability of the rules for Exposure secured by mortgages on immovable property; Exposure Associated with particularly high risk; Exposure in form of Covered Bonds; Exposures to Institutions and Corporates with a short term credit assessment). As a consequence, the “Standardized method” benchmarking would require a set of new rules for comparability issues of STD and IRB measures at counterparty/portfolio level, with even higher impacts for the Institutions
(iii) The comparison of estimates with outturns is broadly used in the internal validation and should provide a basis for supervisory engagement provided that the following points are considered:
a. Generally speaking, statistical tests (i.e. binomial test) for LDPs need to be evaluated with an increased rate of wariness given the low data availability, that can reduce their robustness.
b. The comparison between estimates and outturns is very dependent on the number of rating classes. More clear definition is needed to eliminate effects from different numbers of rating classes, different master scales and used rating methodology (e.g. rating scales with a larger number of classes would be discriminated when only the underestimation in single rating classes is considered and no overestimation effects. In principle, a more granular rating scale provides a more detailed distribution and should be preferred. In the given benchmarking exercise, a bank with a more granular rating scale is disadvantaged, as the probability of outliers in single rating classes is higher). The number of rating classes should be specified, too.
c. In the case, risk underestimation in one rating class goes along with overestimation in other rating classes and using only one-side confidence intervals, the RWA* correction could lead to an overestimation of risk. A suggestion is to apply the PD / DR back testing also on portfolio level.
d. Different model philosophies (TTC vs. PIT) lead necessarily to deviations in the PDs and therefore in the RWA numbers.
e. The calculation of RWA* has to be explained/defined in detail for the case when RWA is calculated on continuous PD values and is not based on the PD of the rating class. In this case the recalculation of RWA using average PDs for each rating class or the master scale PDs lead already to different RWA – lower or larger than the original one depending on the distribution of counterparts and exposure in the rating classes. Just to provide a generic example, one possible definition of PD* could be done the following way: Defining the average PD in the rating class and its difference to the lower confidence level for the default rate. This PD difference could be added to each PD used in the RWA calculation (for each individual counterpart). Thus, a PD* for each counterpart could be defined as input into the RWA* calculation. The same procedure should be used for RWA**.
f. A question arises how HDP is defined. Currently, typical HDP can show for sub-segments very low default rates also for the corporate portfolio and an analysis for single rating classes results in statistically not reasonable results. Only single additional defaults can lead to meaningless results. One example: Overall about 10-15 defaults per year in a module of a corporate rating model for commercial real estate finance. A minimum number of defaults per rating class should be defined to implement the given procedure.
g. Additionally, it has to be noted, that the definition of RWA* on asset class level results in a mixing of different rating methods and models and a mixing of different master scales within one benchmarking portfolio. This leads to a lack of representativeness and homogeneity of benchmarking portfolios.
h. How to calculate RWA** in case an internal model does not exist for the last 5 years?
i. How to proceed, if borrowers change from one internal model to another one (e.g. from Small Business to Corporate due to size increase)?
In any case, EBA should give precise guidance on the computation of outturns (such as a spreadsheet), and these computations should take into account all components of the statistical validity of risk estimates.
Market Risk focus:
(i) Selected quintiles are deemed too aggressive in detecting outliers. Basically, 50% of surveyed banks will trigger this criteria. Thus, we recommend a review of the selected percentiles or the adoption of alternative methods for outlier detection.
(ii) Please refer to Q2. We advocate for the exclusion of standardized charges from the benchmark list, at least until a new standardised approach, in the context of the “Fundamental Review of the Trading Book”, will be established.
(iii) VaR back testing against profit & loss is a well-known model validation tool. However, it is only suitable for VaR. At the same time, it requires a clear definition and standardization of P&L notion (i.e. hypothetical, actual, dirty, clean, etc.) to be compared to the risk measure. Comparing risk forecasts to any notions of P&L can provide with useful information, but the implications on regulatory benchmarking will be different.
Again, the back testing framework for IMA benchmarking purposes needs to be clearly devised and specified.
If the benchmarking portfolios are comparable, the using of statistical values as standard deviation or median/quartiles is acceptable.
As for the time being the national requirements for the development of internal models differ, country specific benchmark portfolios are reasonable.
A valid benchmark to test potential underestimation of own funds requirements is the comparison between models’ estimates and actual long-run credit losses. In this regard, analysts have published studies based on pillar 3 reports with comparisons between model forecasts and actual losses. Reportedly, most of the banks show actual losses way below the estimates across all portfolios even during the years after the wake of the crisis. This evidence clearly indicates that the problem of modelling differences is not the underestimation of capital requirements but the difficulties posed in the comparative analysis of institutions.
Market Risk focus:
Comparison of market risk with the actual (unadjusted) P&L is the ultimate test of internal model’s performance. It is not advisable, however, to perform this test in isolation, since a failure is difficult to interpret.
In the case of low-default-portfolios (LDP) the classes of assets included is somewhat ambiguous. For instance, there is no clarity as to whether and how specialised lending should be included in the LDP benchmarking.
In addition, it should be noted that individual clusters as defined may be immaterial for certain banks. In such cases, banks should also be afforded exemptions from reporting where to do so is overly burdensome.
The exemptions should be based on situations where the institutions are under a model validation process for the portfolios included in the samples or for non-material portfolios. As such, a materiality threshold could be defined as:
(i) An absolute portfolio size;
(ii) A relative portfolio size, in comparison to the total consolidated balance-sheet or to the balance-sheet size of the subsidiary.
Portfolios with partial roll-out should also be exempted (or alternatively, the share of the portfolio under Standard approach should be highlighted and the bias resulting from different capital treatments should be eliminated).
Local entities supervised by a host supervisor should also be exempted from solo reporting as long as their portfolios are included in the consolidated vision submitted to the home supervisor.
Q2. Do you consider that the benchmarks outlined in the RTS are sufficiently proportionate and flexible? Do you have any alternative benchmark proposals? If yes, please provide details.
Credit Risk focus:The classes of benchmarks outlined in the RTS appear sufficiently flexible, with the only exception of the comparison with figures computed according to standardized approach.
However, it would be important that the benefits of diversification (in portfolios and geographies) were taken into account in the benchmarking assessment.
Regarding the proportionate aspect of the proposal, as anticipated previously in point 16, the usage of the first and last quartile would systematically require competent authorities to enquire on half of the contributions, therefore it appears disproportionate. When this benchmark is applied to several figures (RWA, PD and LGD), the number of banks outside the benchmark would even grow.
Such a benchmark (first and last quartile) would instead be reasonable when used as a variable in a scorecard built on several benchmarks.
We would appreciate the use of a kind of scorecard to combine different benchmarks.
Market Risk focus:
We deem the benchmarks outlined in the RTS to be fairly adequate, with the only exception of the comparison with figures computed according to standardized approach.
In general, we genuinely believe that standardized figures do not (and cannot) allow for a better understanding of internal risk measures (let alone risk underestimation, in light of their coarse modelling approach).
In addition, calculation of the standardised charges by internal model banks implies considerable implementation and maintenance costs and efforts A parallel system would indeed be required with no other use or benefit than benchmarking. Under such scenario, internal models, benefiting of several years of use-testing, validation and scrutiny, would be benchmarked against one off models not subject to the same daily controls and level of scrutiny.
All in all, we advocate for the exclusion of standardized charges from the benchmark list, at least until a new standardised approach, in the context of the “Fundamental Review of the Trading Book”, will be established.
Q3. What limitations do you see in relation to the use of the proposed benchmarks, i.e., (i) first and the fourth quartiles; (ii) comparison between own funds under the internal models and the standardised approach; and (iii) comparison between estimates and outturns?
Credit Risk focus:(i) As previously anticipated, the combination of a first and fourth quartile approach used for screening for a more in-depth assessment of the internal approaches would lead to a minimum of 50% of all banks outside the limit. This approach could be useful if used only to drive initial supervisory interest but not for directly deriving conclusions. For that purpose it would be more appropriate to use a narrower sample, e.g. decile buckets and measures/thresholds based on multiples of standard deviation of benchmarking portfolio distributions.
(ii) As previously anticipated in point 16, we see no meaningful role for benchmarks based upon standardised risk weights. The objective of identifying significant and systematic underestimation of own funds requirements cannot be achieved on the basis of a simplistic indicator but rather on a thorough analysis of model assumptions and performance.
Moreover, IRBA and SA are basically different approaches. For some segments, in the SA some risk sensitivity is given, but depending on external ratings. Following external ratings would be not the desired development in risk assessment. For low risky business, IRBA risk weights in retail and corporate segments are/can be lower than the SA risk weights. Otherwise, higher risky business would be rewarded, as the risk weights in IRBA are/can be higher than the SA ones. Therefore, this benchmark is not able to identify a risk underestimation especially in the case of a high risk portfolio.
Moreover, on the operational side the implementation of such a benchmark measure may require institutions to set up appropriate IT procedures for parallel calculation currently not required for other purposes. From a technical point of view, the use of Standardized measures at counterparty/portfolio level can result even in a higher degree of “not comparability” because of the requirements of the Exposure Classes definition in the Standardised Approach: for example, for default portfolio the RW Standard is not comparable with an IRB approach requirements calculation for defaulted portfolios; an IRB exposure may results in several STD exposure classes (in case of applicability of the rules for Exposure secured by mortgages on immovable property; Exposure Associated with particularly high risk; Exposure in form of Covered Bonds; Exposures to Institutions and Corporates with a short term credit assessment). As a consequence, the “Standardized method” benchmarking would require a set of new rules for comparability issues of STD and IRB measures at counterparty/portfolio level, with even higher impacts for the Institutions
(iii) The comparison of estimates with outturns is broadly used in the internal validation and should provide a basis for supervisory engagement provided that the following points are considered:
a. Generally speaking, statistical tests (i.e. binomial test) for LDPs need to be evaluated with an increased rate of wariness given the low data availability, that can reduce their robustness.
b. The comparison between estimates and outturns is very dependent on the number of rating classes. More clear definition is needed to eliminate effects from different numbers of rating classes, different master scales and used rating methodology (e.g. rating scales with a larger number of classes would be discriminated when only the underestimation in single rating classes is considered and no overestimation effects. In principle, a more granular rating scale provides a more detailed distribution and should be preferred. In the given benchmarking exercise, a bank with a more granular rating scale is disadvantaged, as the probability of outliers in single rating classes is higher). The number of rating classes should be specified, too.
c. In the case, risk underestimation in one rating class goes along with overestimation in other rating classes and using only one-side confidence intervals, the RWA* correction could lead to an overestimation of risk. A suggestion is to apply the PD / DR back testing also on portfolio level.
d. Different model philosophies (TTC vs. PIT) lead necessarily to deviations in the PDs and therefore in the RWA numbers.
e. The calculation of RWA* has to be explained/defined in detail for the case when RWA is calculated on continuous PD values and is not based on the PD of the rating class. In this case the recalculation of RWA using average PDs for each rating class or the master scale PDs lead already to different RWA – lower or larger than the original one depending on the distribution of counterparts and exposure in the rating classes. Just to provide a generic example, one possible definition of PD* could be done the following way: Defining the average PD in the rating class and its difference to the lower confidence level for the default rate. This PD difference could be added to each PD used in the RWA calculation (for each individual counterpart). Thus, a PD* for each counterpart could be defined as input into the RWA* calculation. The same procedure should be used for RWA**.
f. A question arises how HDP is defined. Currently, typical HDP can show for sub-segments very low default rates also for the corporate portfolio and an analysis for single rating classes results in statistically not reasonable results. Only single additional defaults can lead to meaningless results. One example: Overall about 10-15 defaults per year in a module of a corporate rating model for commercial real estate finance. A minimum number of defaults per rating class should be defined to implement the given procedure.
g. Additionally, it has to be noted, that the definition of RWA* on asset class level results in a mixing of different rating methods and models and a mixing of different master scales within one benchmarking portfolio. This leads to a lack of representativeness and homogeneity of benchmarking portfolios.
h. How to calculate RWA** in case an internal model does not exist for the last 5 years?
i. How to proceed, if borrowers change from one internal model to another one (e.g. from Small Business to Corporate due to size increase)?
In any case, EBA should give precise guidance on the computation of outturns (such as a spreadsheet), and these computations should take into account all components of the statistical validity of risk estimates.
Market Risk focus:
(i) Selected quintiles are deemed too aggressive in detecting outliers. Basically, 50% of surveyed banks will trigger this criteria. Thus, we recommend a review of the selected percentiles or the adoption of alternative methods for outlier detection.
(ii) Please refer to Q2. We advocate for the exclusion of standardized charges from the benchmark list, at least until a new standardised approach, in the context of the “Fundamental Review of the Trading Book”, will be established.
(iii) VaR back testing against profit & loss is a well-known model validation tool. However, it is only suitable for VaR. At the same time, it requires a clear definition and standardization of P&L notion (i.e. hypothetical, actual, dirty, clean, etc.) to be compared to the risk measure. Comparing risk forecasts to any notions of P&L can provide with useful information, but the implications on regulatory benchmarking will be different.
Again, the back testing framework for IMA benchmarking purposes needs to be clearly devised and specified.
Q4. What in your view is the most appropriate benchmark and/or approach for the assessment of the level of potential underestimation of own funds requirements?
Credit Risk focus:If the benchmarking portfolios are comparable, the using of statistical values as standard deviation or median/quartiles is acceptable.
As for the time being the national requirements for the development of internal models differ, country specific benchmark portfolios are reasonable.
A valid benchmark to test potential underestimation of own funds requirements is the comparison between models’ estimates and actual long-run credit losses. In this regard, analysts have published studies based on pillar 3 reports with comparisons between model forecasts and actual losses. Reportedly, most of the banks show actual losses way below the estimates across all portfolios even during the years after the wake of the crisis. This evidence clearly indicates that the problem of modelling differences is not the underestimation of capital requirements but the difficulties posed in the comparative analysis of institutions.
Market Risk focus:
Comparison of market risk with the actual (unadjusted) P&L is the ultimate test of internal model’s performance. It is not advisable, however, to perform this test in isolation, since a failure is difficult to interpret.
Q5. Which set of market risk portfolios do you consider more appropriate for the initial exercise conducted under Article 78?
In order to minimise the burden to Banks and supervisors and to avoid duplication of efforts, given the parallel running of several data-collection initiatives, same portfolios used in 2013 benchmarking exercise of the Basel Committee on Banking Supervision should be used with only minor adaptations (if any at all), in order to maintain the portfolio validity.Q6. As explained in the background section, do you consider the approach proposed by the EBA appropriate for future annual exercises?
We consider the approach proposed as adequate.Q7. Do you have any alternative proposals? If yes, please provide details.
N.A.Q8. Which of the two options for phasing-in do you consider preferable?
The phase-in option 2 is preferable from our side.Q9. Do you see any potential ambiguities in the credit risk portfolios defined in Annex I? Please identify the relevant portfolio providing details and any suggestions that would eliminate these ambiguities.
Regarding residential mortgages, it would be worth considering specificities that can materially change the risk profile like the different forms of state guarantee programs. Evidence found in the EBA fourth analysis on the consistency of risk weighted assets clearly indicates that member states in which a state guarantee is commonplace have different risk drivers. The credit losses experience is also different and can be explained with long-term observed data. Therefore, it would make sense to split the portfolio and analysed separately with bespoke benchmarks the transactions that are backed with such a state guarantee.In the case of low-default-portfolios (LDP) the classes of assets included is somewhat ambiguous. For instance, there is no clarity as to whether and how specialised lending should be included in the LDP benchmarking.
Q10. Do you have any suggestions for additional credit risk portfolios? Please provide details.
N.A.Q11. Do you see any potential ambiguities in the market risk portfolios defined in Annexes VII.a and VII.b? Please identify the relevant portfolio providing details and any suggestions that would eliminate these.
In general, instruments specifications are not detailed enough to avoid issues in the booking into position keeping systems.Q12. Do you have any suggestions for additional market risk portfolios? Please provide details.
N.A.Q13 Do you agree with the possibility of allowing firms to refrain from reporting portfolios if one of the conditions stated in Article 3 is met?
Yes, banks should be exempted from reporting requirements as outlined in Article 3 if it is well justified and it is also made clear to peers.Q14 Do you have any suggestion about additional exemptions from reporting? If yes, please provide details.
Credit Risk focus:In addition, it should be noted that individual clusters as defined may be immaterial for certain banks. In such cases, banks should also be afforded exemptions from reporting where to do so is overly burdensome.
The exemptions should be based on situations where the institutions are under a model validation process for the portfolios included in the samples or for non-material portfolios. As such, a materiality threshold could be defined as:
(i) An absolute portfolio size;
(ii) A relative portfolio size, in comparison to the total consolidated balance-sheet or to the balance-sheet size of the subsidiary.
Portfolios with partial roll-out should also be exempted (or alternatively, the share of the portfolio under Standard approach should be highlighted and the bias resulting from different capital treatments should be eliminated).
Local entities supervised by a host supervisor should also be exempted from solo reporting as long as their portfolios are included in the consolidated vision submitted to the home supervisor.
Upload files
EBA_Benchmarking_Consultation_final_v2.pdf
(181.57 KB)