Week 6

Journal of Banking & Finance 44 (2014) 72–92

Contents lists available at ScienceDirect

Journal of Banking & Finance

journal homepage: www.elsevier .com/locate / jbf

Risk models-at-risk

http://dx.doi.org/10.1016/j.jbankfin.2014.03.019 0378-4266/� 2014 Elsevier B.V. All rights reserved.

⇑ Corresponding author. Address: Univ. La Reunion, CEMOI, 15 avenue Réné Cassin, CS 92003, 97744 Saint-Denis Cedex 9, France. Tel.: +33 686431914.

E-mail addresses: [email protected] (C.M. Boucher), j.daniels- [email protected] (J. Daníelsson), [email protected] (P.S. Kouontch- ou), [email protected] (B.B. Maillet).

1 VaR is defined as the maximum expected loss on an investment over a specified horizon at a particular confidence level. As is widely known, the a-VaR is the (1� a)- quantile of the portfolio’s return distribution, where generally :5 < a < 1; that is, the minimum potential loss that will be sustained with a probability a. Since the BIS accords, VaR follow-ups play a central role in financial risk measurement and management. For a review of the literature of the VaR calculations, please see Engle and Manganelli (2001) and Jorion (2007).

2 In the finance literature, the term ‘‘model risk’’ frequently applies to un about the risk factor distribution (e.g. Gibson, 2000; Jorion, 2009a,b), alth term is sometimes used in a wider sense (e.g. Derman, 1996; Crouhy et al.,

Christophe M. Boucher a,b, Jón Daníelsson c, Patrick S. Kouontchou b, Bertrand B. Maillet a,d,⇑ a A.A.Advisors-QCG (ABN AMRO), France b Variances and Univ. Lorraine (CEREFIGE), France c Systemic Risk Centre and London School of Economics, United Kingdom d Variances, Univ. La Reunion and Orleans (CEMOI, LEO/CNRS and LBI), France

a r t i c l e i n f o

Article history: Received 24 July 2012 Accepted 10 March 2014 Available online 21 March 2014

Jel classification: C50 G11 G32

Keywords: Model risk Value-at-risk Backtesting

a b s t r a c t

The experience from the global financial crisis has raised serious concerns about the accuracy of standard risk measures as tools for the quantification of extreme downward risks. A key reason for this is that risk measures are subject to a model risk due, e.g. to specification and estimation uncertainty. While regula- tors have proposed that financial institutions assess the model risk, there is no accepted approach for computing such a risk. We propose a remedy for this by a general framework for the computation of risk measures robust to model risk by empirically adjusting the imperfect risk forecasts by outcomes from backtesting frameworks, considering the desirable quality of VaR models such as the frequency, indepen- dence and magnitude of violations. We also provide a fair comparison between the main risk models using the same metric that corresponds to model risk required corrections.

� 2014 Elsevier B.V. All rights reserved.

1. Introduction

Recent crises have laid bare the failures of standard risk models. High levels of model risk caused models to under forecast risk prior to crisis events, to be slow to react as a crisis unfolds, and then slow to reduce risk levels post-crisis. It is as if the risk models got it wrong in all states of the world. Addressing this problem provides the main motivation for our work. In particular, we explicitly adjust risk forecasts for model risk by their historical performance, so that a risk model learns from its past mistakes. While our focus is on Value-at-Risk (VaR),1 the analysis applies equally to other risk measures such as expected shortfall (ES).

While there is no single definition of model risk.2 it generally relates to the uncertainty created by not knowing perfectly the true data generating process (DGP). This inevitably means that any prac- tical definition is linked to such an uncertainty and thus is context dependent. In our case, the end product is a risk forecast, so model risk is the uncertainty in risk forecasting arising from estimation error and the use of an incorrect model. This double uncertainty is responsible both for the range of plausible risk estimates (see, e.g. Beder, 1995), and more generally the inability to forecast risk with acceptable accuracy.

To formalize this, in our view a risk forecast model should meet three desirable criteria: the expected frequency of violations, the absence of violation clustering and a magnitude of violations consistent with the underlying distributional assumptions. These three criteria provide the lens through which to view our empirical results.

We can motivate our contribution by means of an example represented in Fig. 1, where, for each day in a sample of the Dow Jones (DJIA) index over a century, we show the outcomes from applying state of the art VaR forecast methods. We also show periodically which method generated the highest and the lowest

certainty ough the 1998).

1904 1911 1919 1927 1934 1942 1950 1957 1965 1973 1980 1988 1996 2003 2011 −40%

−20%

0%

20%

Va R

L ev

el

Dates

G

CV

G

CV

CV

RM RMRM RM RMRM RMRM

CF

RMRM RMRM

RM GPDCF

RMRM RMRMRM

G G G

RM RM

GCF GG G G GG

RM RMG

RM

G

RM RM

RMCF

1904 1911 1919 1927 1934 1942 1950 1957 1965 1973 1980 1988 1996 2003 2011 0

1

2

3

x 104

Pr ic

e

MinVaR MaxVaR

DJI (right axis)

Fig. 1. DJIA and the range of daily 99% VaR forecasts. Daily DJIA index returns from the 1st January, 1900 to the 20th September, 2011. We use a moving window of four years (1040 daily returns) to dynamically re-estimate parameters for the various methods. The letters ‘‘H’’, ‘‘N’’, ‘‘t’’, ‘‘CF’’, ‘‘RM’’, ‘‘G’’, ‘‘CV’’, ‘‘GEV’’, ‘‘GPD’’ stand for, respectively, historical, normal, Student, Cornish–Fisher, exponential weighted moving average (EWMA or RiskMetrics), GARCH, CAViaR, GEV and GPD methods for VaR calculation.

C.M. Boucher et al. / Journal of Banking & Finance 44 (2014) 72–92 73

forecasts. By highlighting the wide disparity between the most common risk forecast methods, the figure illustrates one of the biggest challenges faced by risk managers. Typically, the VaR does not vary much, but when it does, it reacts sharply and belatedly to extreme returns. The range of plausible VaR forecasts is large, where the models producing the highest and lowest forecasts frequently change position across time. Even right after WWII, during a relatively quiet period for financial markets, the most conservative VaR was four times the most aggressive one.

As in Daníelsson et al. (2014) and Daníelsson (2002) the main conclusion from this brief analysis is that risk managers face a large range of plausible forecast methods and their associated model risk, having to choose between desirable criteria such as performance, degree of conservativeness or forecast volatility. This challenge motivates our main objective where we propose a general method for the correction of imperfect risk estimates, whatever the risk model.

We illustrate our approach by considering events around the Lehman Brothers’ collapse, as presented in Fig. 2 for the period of January 1st, 2007 to January 1st, 2009. The Figure displays peaks-over-VaR for one-year rolling daily historical 99% VaR on the S&P 500 index.

The figure shows that the hits are excessively frequent, highly autocorrelated and, around October 2008, far from the estimated VaR, even if it progressively adjusted after the hits. This suggests that an optimal buffer would make the VaR forecast more robust.

However, it is not trivial to calculate the buffer, after all, the properties of hits are significantly different in terms of frequency, dependence and size, depending both on the underlying VaR model and probability level as well as the magnitude of the buffer. A large (respectively small) buffer correction will lead to a too conserva- tive (too little) protection. The question for the risk manager is then how to ex ante fix the size of this buffer, as illustrated by the three arbitrary correction factors labelled #1, #2 or #3, on the right-hand side y-axis in Fig. 2.

In the financial literature, a number of papers have considered estimation risk for risk models, see for instance Gibson et al., 1999; Talay and Zheng, 2002. The issue of estimation risk for VaR has been considered for the identically and independently distributed return

case by, for example, Pritsker (1997) and Jorion (2007). Estimation risk in dynamic models has also been studied by several authors. Berkowitz and O’Brien (2002) observe that the usual VaR estimates are too conservative. Figlewski (2004) examines the effect of estima- tion errors on the VaR by simulation. The bias of the VaR estimator, resulting from parameter estimation and misspecified distribution, is studied for ARCH(1) models by Bao and Ullah (2004). In the iden- tical and independent setting, Inui and Kijima (2005) show that the nonparametric VaR estimator may have a strong positive bias when the distribution features fat-tails. Christoffersen and Gonçalves (2005) study the loss of accuracy in VaR and ES due to estimation errors and constructed bootstrap predictive confidence intervals for risk measures. Hartz et al. (2006) propose a re-sampling method based on bootstrap to correct the bias in VaR forecasts for the Gauss- ian GARCH model. For GARCH models with heavy-tailed distribu- tions, Chan et al. (2007) derive the asymptotic distributions of extremal quantiles. Escanciano and Olmo (2009, 2010, 2011) study the effects of estimation risk on backtesting procedures. They show how to correct the critical values in standard tests used, when assessing the quality of VaR models. Gouriéroux and Zakoïan (2013) quantify in a GARCH context the effect of estimation risk on measures for estimation of portfolio credit risk and show how to adjust risk measures to account for estimation error. Gagliardini et al. (2012) propose estimation and granularity adjustments for VaR, whilst Lönnbark (2010) derives adjustments of interval fore- casts to account for parameter estimation.

In the context of extreme risk measures, our work also relates to Kerkhof et al. (2010), who first propose an incremental market risk capital charge calibrated on the backtesting framework of the regulators. Our present work documents the proposed methodol- ogy and complements their approach, generalizing the tests used for defining the buffer. Alexander and Sarabia (2012) also explicitly deal with VaR model risk by quantifying VaR model risk and propose an adjustment to regulatory capital based on a maximum relative entropy criterion to some benchmark density. In a similar manner, Breuer and Csiszár (2013, 2014) and Breuer et al. (2012) define model risk as an amplified largest loss based on a distribu- tion which is at a reasonable, Mahalanobis or Kullback–Leibler, distance to a reference density.

02/07 04/07 06/07 08/07 10/07 12/07 02/08 04/08 07/08 09/08 10/08 01/09 −10.0%

−8.0%

−6.0%

−4.0%

−2.0%

0.0% (a) Negative Returns and One−year rolling VaR at 99%

Negative Returns 1−year rolling historical VaR99%

02/07 04/07 06/07 08/07 10/07 12/07 02/08 04/08 07/08 09/08 10/08 01/09 0.0%

1.0%

2.0%

3.0%

4.0%

5.0%

6.0% (b) Exceptions and various Adjusted Estimated VaR

← 1−year rolling historical VaR99%

← Adjusted VaR99% #1

← Adjusted VaR99% #2

← Adjusted VaR99% #3

Fig. 2. S&P500 negative returns and daily 99% VaR forecasts around the 2008 Lehman Brothers’s event. Daily S&P500 index from the 1st January, 2003 to the 1st January, 2009. The figure presents peak-over-VaR based on the four-year rolling daily historical 99% VaR on the S&P 500 index, as well as corrected VaR estimates with various ad hoc incremental buffers (numbered from #1 to #3).

74 C.M. Boucher et al. / Journal of Banking & Finance 44 (2014) 72–92

We start with a controlled experiment, whereby we simulate an artificial long time-series which exhibits the salient features of financial return data. We then estimate a range of VaR forecast models with this data, both identifying model risk and more importantly dynamically adjusting the risk forecasts with respect to such risk.

The conclusions from this exercise lead us to a number of inter- esting conclusions. First, by dynamically adjusting for estimation bias we significantly improve the performance of every method, suggesting that such an approach might be valid in routine appli- cations of risk forecasting. Second, the model bias is large in gen- eral, and sometimes to the same order as the VaR measure itself, and very different across methods. Finally, the bias strongly depends upon the probability confidence level. This suggests that a commonly advocated approach of probability shifting—whereby we estimate a model with one probability to better estimate a VaR with a less extreme probability—is not valid.

The Monte Carlo results motivate our main contribution, the development of a practical method for dealing with model uncer- tainty. Since we do not know the ‘‘true’’ model, we instead learn from history by evaluating the historical errors in order to use them to dynamically adjust future forecasts. We reach a range of empirical conclusions from this exercise.

1. The magnitude of corrections can sometimes be large, espe- cially around the 1929 and 2008 crises, ranging from 0% to 15% for some methods to more than 100% in some circumstances.

2. The EWMA3 and GARCH VaR are among the preferred models, since the minimum correction to pass main backtests are among the smallest.

3 The Exponentially Weighted Moving Average (EWMA) refers to JP Morgan’s RiskMetrics proposed method (RiskMetrics, 1996), which consists of placing more weight on the most recent observations than on the older ones when calculating volatility (using an exponential moving average with an exponential factor tradi- tionally fixed at .94 with daily data).

3. Regardless of the model, a ten year sample period is needed to have a fairly good idea of the magnitude of the required correction.

4. The model risk of the correction buffer can be measured and the buffer fine-tuned according to the link between the confidence level on the required correction. This enables risk managers to explicitly tailor the buffer to major financial stress episodes such as the Great Depression of 1929 or the 2008 crisis, if they choose to do so.

5. By considering multivariate indexes and portfolios, we find that the model risk adjustment buffer is in line with the multiple k imposed by regulators (from 3 to 5).

6. The general methodology can be used to gauge the plausibility of traditional handpicked stress-test scenarios.

The outline of the paper is as follows: Section 2 evaluates the extent to which elementary model risks affect VaR estimates based on realistic simulations. Section 3 proposes a practical method to provide VaR estimates robust to model risk. Section 4 finally con- cludes, whilst the Appendix follows, outlining some description and examples of model risks and the main backtesting methods used in the paper.

2. Analysis of estimation and specification errors

Consider a general setting where we know the ‘‘true’’ VaR (best case scenario), but where the sample size is so small that it entails some estimation problems. In this case, the estimated VaR will inevitably be an imperfect estimate of the theoretical (‘‘true’’) VaR. In particular, there exists a bias function, denoted biasðh0; ĥ;aÞ, that makes the equality between the theoretic and empirical exact4:

ThVaRðh0;aÞ ¼ EVaRðĥ;aÞ þ biasðh0; ĥ;aÞ; ð1Þ

4 The bias function is implicitly defined from Eq. (1). See the Appendices for examples of such bias functions in various contexts of model risks.

5 See Hamilton and Susmel, 1994; Gray, 1996; Klaassen, 2002; Haas et al., 2004, for more details on the process.

6 As a complement (not reported here for space reasons, but available on demand in a web Appendix), we also made use of other alternative frameworks: a Student versus a normal density, as well as Brownian, Lévy and Hawkes processes, with the same qualitative response with a relative model error for VaR ranging from 5% to 15% in the simplest cases (Gaussian estimation risk with 250 observations) to as large as 200% when the process is complex and the sample small (the case of Hawkes processes).

7 The estimated parameters of the MS(2)-GARCH(1,1) model on the DJI Index are x1 ¼ 3:1699e�006; b1 ¼ 0:90801, a1 ¼ 0:0733081; x2 ¼ 2:509e�005, b2 ¼ 0:10453; a2 ¼ 0:0064734; l1 ¼ 0:00, l2 ¼ 0:00; t ¼ 5:56; p11 ¼ 0:99654 and p22 ¼ 0:99328. Bauwens et al. (2010) obtain approximately the same results on the S&P. This estimation is crucial since the transition probabilities between states and auto-regressive parameters both affect the persistence of the simulated processes. Our estimates are here very similar to those exhibited in the literature (e.g. Bauwens et al., 2010; Billio et al., 2012; Frésard et al., 2011). Moreover, when artificially considering different probabilities related to the second state, we find the same qualitative results in what we are interested in: model risk of risk models. Last but not least, when we have adopted other representations of financial returns (either using processes or densities), we again reach the same order of magnitude of the worst errors of forecasting (additional results available upon request).

C.M. Boucher et al. / Journal of Banking & Finance 44 (2014) 72–92 75

where ĥ denotes the estimated parameters, h0 the true parameters and a the probability level of the VaR. The theoretic true VaR is denoted by ThVaRðh0;aÞ and the estimated VaR by EVaRðĥ;aÞ.

This general Eq. (1) can be written more precisely in the context where we know the DGP (up to the parameter h0)’’, e.g. in simula- tion exercises. In this case, we can also know the general bias which depends on some parameters so that this bias is now denoted biasðh0; ĥ;aÞ. In this setting (when the DGP is known), we also know the bias function, and thus the Perfectly Estimated Adjusted VaR is indeed estimated via the bias, which itself depends upon some nuisance parameters (such as, for instance, the window length for the dynamic estimation. . .). We can therefore obtain the perfect estimation adjusted VaR (PEAVaR), with the estimated VaR (EVaRðĥ;aÞ), by:

PEAVaRðĥ; h0;aÞ ¼ EVaRðĥ;aÞ þ biasðh0; ĥ;aÞ: ð2Þ

As a general rule, the smaller a is, the better we forecast VaR and identify the bias function. The reason is that, for a given sam- ple size, the number of quantiles increases along with decreasing a, so the effective sample size used in the forecasting exercise increases. As the probabilities become more extreme, the accuracy of the VaR forecasts decreases, for example, because fewer obser- vations are used in the estimation. Consequently, it is harder to model the shape of the tail than the shape of the interior distribu- tion. For this reason, it might be tempting to forecast VaR slightly closer to the center of the distribution, perhaps at a ¼ 95%, and then use those estimation results to get at the VaR for more extreme probability levels, like a ¼ 99% or a ¼ 99:9%. This is often referred to as probability shifting.

2.1. Probability shifting

We can analyze the impact of probability shifting within our framework by defining two random probabilities, ~a� and ~a��, so that:

PEAVaRðĥ; h0;aÞ ¼ EVaRðĥ; ~a�Þ ¼ ThVaRðh0; ~a�Þ EVaRðĥ; ~a��Þ ¼ PEAVaRðĥ; h0;aÞ ¼ ThVaRðh0;aÞ;

( ð3Þ

or equivalently, with Fð�Þ and bFð�Þ representing, respectively, the theoretic and estimated cumulative density functions:

~a� ¼ F½bF�1ðaÞ� ~a�� ¼ bF ½F�1ðaÞ�;

( ð4Þ

with bF�1ðaÞ ¼ EVaRðĥ;aÞ and F�1ðaÞ ¼ ThVaRðh0;aÞ. If one were to use ~a� instead of a, the bias adjusted VaR results,

whilst ~a�� achieves the opposite, mapping the probability corresponding to the biased VaR, to the theoretic VaR.

It follows that if ~a� > a > ~a��, the estimated VaR is biased towards zero, whilst if ~a� < a < ~a��, it is biased towards minus infinity.

2.2. Monte Carlo examination

Many potential sources of error can significantly impact on the accuracy of risk forecasts. The sources one is most likely to encoun- ter in day-to-day risk forecasting, and certainly in most academic studies, are estimation and specification errors. For this reason, we investigate these two in detail by means of Monte Carlo experiments.

We consider below the distribution of the errors between the poorly estimated VaR and the true VaR when considering, alterna- tively, estimation risk, specification uncertainty or both. We first specify a DGP from which we generate data. We then treat the DGP as unknown and forecast VaR for the simulated data.

As before, the true parameters are h0, but we now also have the true parameters of the misspecified model, indicated by h1, as well as its estimate ĥ1. In this case, we indicate the estimated VaR by EVaRðĥ1;aÞ and define the perfect model risk adjusted VaR (denoted herein PMAVaR) by:

PMAVaRðĥ1;aÞ ¼ EVaRðĥ1;aÞ þ biasðh0; ĥ1;aÞ: ð5Þ

We first present the theoretical framework related to the correction procedure in a static setting for the sake of simplicity. However, in the subsequent empirical application, we also con- sider the dynamic properties of our correction procedure that is proposed at date t based on the conditional information available at date t � 1.

2.2.1. The true model The DGP needs to be sufficiently general to capture the salient

features of financial return data. Because we are not limited by the need to estimate a model, we can specify a DGP that might be difficult, to the point of impossible, to estimate in small samples. The DGP we employ is a second order Markov-switching generalized autoregressive conditionally heteroskedastic with Student-t disturbances (hereafter denoted MS(2)-GARCH(1,1)-t)5

as in Frésard et al. (2011) in a VaR context.6

More precisely, the DGP is:

rt ¼ lst þ rst zt ; ð6Þ

where the zt innovations series are independently and identically distributed as a standard Student distribution with t degrees of freedom (zt � iidStð0;1; tÞ), and r2st ¼ xst þ ast e

2 t�1 þ b

2 st r

2 st�1

, with st 2 f1;2g characterizes the state of the market, lst is the mean return and with t degrees of freedom, and where xst > 0; ast P 0; bst P 0 are the parameters of the GARCH(1,1) in the two states, and et ¼ rt � lst the return innovations with the fat tails of a Student density with a t degree of freedom.

The state is modelled with a Markov chain whose matrix of transition probabilities is defined by pij ¼ Prðst ¼ jjst�1 ¼ iÞ. Appro- priately chosen restrictions on the GARCH coefficients ensure that r2t is strictly positive.

Using this DGP, we first simulate a long artificial series of 360,000 daily returns with estimated parameters on the daily DJIA from the 1st January, 1990 to the 20th September, 2011.7 We then forecast various VaRs using 1000 observations, and finally compute main statistics of the forecast error, measured by differences between the asymptotic VaR (computed with the true simulated DGP on 360,000 observations) and empirical ones recovered from limited samples.

Table 1 Conditional simulated errors associated with the 95%, 99% and 99.5% VaR: GARCH (1,1) versus MS(2)-GARCH (1,1)-t.

Probability (%) Mean estimated VaR (%) Perfect VaR (%) Mean bias (%) Median bias (%) Min. bias (%) Max. bias (%)

Pair Panel A. GARCH (1,1) DGP and GARCH (1,1) VaR with estimation error a = 95.00 �36.16 �36.16 .00 .02 �19.53 19.60 a = 99.00 �59.70 �59.70 .00 .04 �32.66 32.02 a = 99.50 �70.99 �70.99 .00 .06 �38.35 38.03

Panel B. MS(2)-GARCH (1,1)-t DGP and GARCH (1,1) VaR with specification error a = 95.00 �30.78 �36.16 �5.38 �5.38 �5.38 �5.38 a = 99.00 �43.83 �59.70 �15.87 �15.87 �15.87 �15.87 a = 99.50 �48.61 �70.99 �22.38 �22.38 �22.38 �22.38

Panel C. MS(2)-GARCH (1,1)-t DGP and GARCH (1,1) VaR with specification and estimation errors a = 95.00 �28.97 �36.16 �7.19 �8.83 �21.70 18.99 a = 99.00 �41.28 �59.70 �18.42 �20.76 �38.88 18.02 a = 99.50 �45.78 �70.99 �25.20 �27.79 �47.84 15.03

Daily DJIA index from the 1st January, 1900 to the 20th September, 2011. These statistics were computed with the results of 360,000 simulated series of 1,000 daily returns according to a specific DGP (rescaled GARCH (1,1) for Panel A and MS(2)-GARCH (1,1)-t for Panels B and C) using an annualized Normal GARCH VaR (in all Panels). The columns represent, respectively, the average adjusted VaR according to specification and/or estimation errors, the theoretical VaR, the average, the minimum and the maximum value of the adjustment terms. A negative adjustment term indicates that the estimated VaR (negative return) should be more conservative (more negative). Panel A presents GARCH (1,1) DGP and/or estimated GARCH VaR; Panel B relates to a MS(2)-GARCH (1,1) DGP with estimated GARCH VaR; Panel C refers to an estimated MS(2)- GARCH (1,1) DGP with results from an estimated GARCH VaR.

76 C.M. Boucher et al. / Journal of Banking & Finance 44 (2014) 72–92

2.2.2. Misspecification and a parameter estimation uncertainty Our focus is on the annualized daily 95%, 99% and 99.5% VaR.

Table 1 illustrates the model risk of VaR estimates, defined as the implication of model misspecification and a parameter estimation uncertainty. We examine this model risk by comparing simulations and estimates corresponding to a normal GARCH(1,1) and a MS(2)- GARCH(1,1)-t. The columns represent respectively the average adjusted VaR according to specification and/or estimation errors, the theoretic VaR, the average, the minimum and maximum values of the adjustment terms. Note that a negative adjustment term indicates that the estimated VaR (which is a negative return) should be more conservative (more negative).

We present the estimation bias ðbiasðh0; ĥ;aÞÞ, in Panel A of Table 1, when we simulate a simple model (Normal GARCH(1,1)) and use the appropriate methodology for computing the VaR (Nor- mal GARCH VaR). This bias arises only due to the small estimation sample size (1000) and is zero for the full 360,000 sized sample. However, the dispersion of this estimation bias is quite large since the minimum and the maximum values of the bias (or adjustment term) represent about 50% of the true VaR. For example, with a ¼ 99%, the minimum and maximum biases are respectively equal to �33% and +32% for a true VaR of �60%.

The specification bias (biasðh0; h1;aÞ) is presented in Panel B of Table 1, where the quantiles were modelled by a GARCH(1,1) VaR. Within this specific illustration, the risk model is fully explained by the discrepancy between the DGP and the assumed simple risk model used (since the parameters are here known and the estimation bias is zero by definition); the specification bias is thus constant and depends upon the choice of the risk model specification. The average specification bias is large here; it is negative and increases in absolute terms with a, which indicates that extreme risks of the MS(2)-GARCH(1,1)-t DGP are generally underestimated by the GARCH(1,1) parametric VaR model.

The estimation and specification biases are captured simulta- neously in Panel C. These components of model risk are jointly considered and, in the worst cases, they merely add up in an inde- pendent manner. We compute the global error—denoted biasðh0; h1; ĥ1;aÞ in its most general formulation—as the difference between the true VaR and the estimated VaR according to a mis- specified VaR model estimated on a limited sample. As in Panel B, where a normal GARCH(1,1) VaR is used with a simulated MS(2)-GARCH(1,1)-t, the average bias is negative and increases in absolute terms with a. The mean errors are thus equivalent to

the specification bias component, but the dispersion of the model risk realizations is inflated by the estimation bias.

2.2.3. Probability shifting We illustrate the impact of probability shifting and model risk

in Table 2, which shows the two modified probability levels ~a�

and ~a��. The former is associated to the true density and corre- sponds to the (mis-) estimated ð1� aÞ-VaR, whilst the latter, asso- ciated to the estimated VaR, corresponds to the ð1� aÞ-VaR without model error.

The gap between ~a� and a can be interpreted as a measure of the model risk of the risk model. The gap between ~a�� and a can also be analyzed as the probability shift that we should apply using a specific model of VaR to reach the true VaR.

This alternative representation of the model risk of risk models shows that ~a�� is often unreachable and cannot be used for correct- ing the estimated VaR. For instance, the maximum associated with the 99.5% VaR in Panel C has to be superior to 100%, which cannot in practice be discriminated from the maximum, i.e. when associ- ated with the 100% probability. More generally, ~a�� is frequently superior to a, (and ~a� generally inferior to a) which can be inter- preted as an under-estimation of the risk using the proposed model of VaR (the estimated VaR is too aggressive).

This suggests that the recent call of some authorities for more extreme quantiles (see, e.g. FSA, 2006), i.e. VaR 99.5% or 99.9%, is not warranted since in some cases the real VaR appears below the worst estimated return.

Finally, our results show, surprisingly, that the mean bias is not a simple increasing function of the VaR and, accordingly, of the level of probability associated to the VaR. The expected adjustment associated to the 99.5% (99%) probability level is, for instance, four (two) times larger than the expected adjustment associated to the 95% probability level and represents an increase of nearly 15% (10%). The relation between the model risk and the probability associated to the VaR is not linear and depends on several components.

The implemented estimated VaR should be corrected by an adjustment corresponding to the global bias linked to the potential model risk error. However, the true perfect VaR is generally unknown by definition. The proposed adjustments are thus impos- sible to quantify accurately outside a pure academic simulation exercise.

Table 2 Probability shifts associated with 95%, 99% and 99.5% annualized VaR: GARCH (1,1) versus MS(2)-GARCH (1,1) quantiles.

Estimated VaR Probability ~a� associated to the true density corresponding to the (mis- )estimated VaR

Probability ~a�� associated to the biased empirical density corresponding to the perfect VaR

Mean shift Median shift Min shift Max shift Mean shift Median shift Min shift Max shift

Panel A. GARCH (1,1) DGP and GARCH (1,1) VaR with estimation error a = 95.00 94.19 94.24 90.37 99.31 94.51 94.26 94.36 99.88 a = 99.00 98.92 98.95 96.83 99.92 99.05 99.08 98.49 99.99 a = 99.50 99.25 99.38 98.71 99.97 99.47 99.09 99.98 N.R.

Panel B. MS(2)-GARCH (1,1)-t DGP and GARCH (1,1) VaR with specification error a = 95.00 95.81 95.81 95.81 95.81 97.29 97.29 97.29 97.29 a = 99.00 98.64 98.64 98.64 98.64 99.92 99.92 99.92 99.92 a = 99.50 99.07 99.07 99.07 99.07 99.99 99.99 99.99 99.99

Panel C. MS(2)-GARCH (1,1)-t DGP and GARCH (1,1) VaR with specification and estimation errors a = 95.00 94.15 94.29 82.43 99.44 97.44 98.47 85.69 N.R. a = 99.00 97.71 97.94 89.81 99.88 99.78 99.98 96.27 N.R. a = 99.50 98.35 98.56 91.71 99.92 99.93 N.R. 98.32 N.R.

Daily DJIA index from the 1st January, 1900 to the 20th September, 2011. These statistics were computed with the results of 360,000 simulated series of 1000 daily returns according to a specific DGP (rescaled GARCH (1,1) for Panel A and MS(2)-GARCH (1,1)-t for Panels B and C) using an annualized Normal GARCH VaR (in all Panels). The columns represent, respectively, the average Estimated VaR according to specification and/or estimation errors, the mean, the minimum and the maximum of the modified probability level ~a� , the mean, the minimum and the maximum of the modified probability level ~a�� . The letters N.R. stand for ‘‘Not Reached’’, i.e. condition on bounds is not met even for 100.00%. Panel A presents GARCH (1,1) DGP and/or estimated GARCH VaR; Panel B relates to a MS(2)-GARCH (1,1) DGP with estimated GARCH VaR; Panel C refers to an estimated MS(2)-GARCH (1,1) DGP with results from an estimated GARCH VaR.

8 Note that the Basel ‘‘traffic light’’ backtesting framework is directly inspired by this unconditional coverage test. However, Escanciano and Olmo (2009), Escanciano and Olmo, 2010, Escanciano and Olmo, 2011 and Escanciano and Pei (2012) note tha the Eq. (9) is ‘‘asymptotically correct’’ only if the in-sample size is infinitely large relative to the out-of-sample one. To deal with this issue, we present later on in this article (in Table 3), the results of our risk model correction method based on the bootstrapped version of several tests as proposed by Escanciano and Olmo (2009 2010, 2011).

C.M. Boucher et al. / Journal of Banking & Finance 44 (2014) 72–92 77

3. An economic valuation of model risk

While the illustration above is focused on the controlled exper- iment where the modeller knows the true model, in reality the true model is not known. To address this, we propose a practical method for dealing with model uncertainty, that makes use of past historical errors related to specific estimated models. While it is not possible to optimally adjust for biases, we can approximate them by adjusting the VaR forecasts by the model’s historical per- formance. More concretely, historical errors are used to adjust future forecasts by identifying the minimum correction factor needed to pass backtest criteria.

Recall the general Eq. (1) that defines the theoretical VaR as the estimated VaR plus an error term. We can rewrite this general for- mulation in another setting where we cannot be sure of the future DGP (in an uncertain context). Hence, we define the imperfect model adjusted VaR (IMAVaR) as (with previous notations):

IMAVaRðĥ1;aÞ ¼ EVaRðĥ1;aÞ þ adjðh0; h1; ĥ1;aÞ; ð7Þ

where EVaRð�Þ is an estimated VaR at the level a with a specific risk model, ĥ1 are model parameters estimated with T observations, and adjð�Þ the minimum VaR adjustment for the risk model, so that:

IMAVaRðĥ1;aÞ ¼ sup|{z} VaR2R

fVaRðaÞ�g; ð8Þ

where the symbol R refers to the real numbers set, VaRð�Þ� is a set of corrected VaR built from a model, and IMAVaRð�Þ is the highest limit VaR (the less conservative VaR) that can be validated by the super- visor (and all other more aggressive VaR rejected).

The IMAVaRð�Þ is thus the lowest acceptable VaR where the term ‘‘acceptable’’ means that this VaR has the main good expected qualities such as, for instance, a right hit frequency (and/or a fair dependence, and/or a reasonable magnitude) of hits. Imagine two polar cases. The VaR is �100.00% (the asset price is then equal to zero); in this case, there is no hit and then no bad properties of hits, but the estimated VaR is too conservative. If the VaR is +100.00% (in this case, the VaR is too aggressive), we have numerous hits with very bad properties. The IMAVaRð�Þ corresponds to a ‘‘model risk robust’’ VaR that serves to calculate the correction we apply to the Estimated VaR. Hence, the IMAVaRð�Þ is the highest VaR (the less conservative VaR) that can be both validated by the

regulator (regarding the properties of its hits) and accepted by the asset manager based on a consensual criterion.

3.1. General backtest procedures

A variety of tests have been proposed in the literature to gauge the accuracy of VaR estimates (see Pérignon and Smith, 2010). In our view, there are three desirable properties that should be met by a risk model: the expected frequency of violations, the absence of violation clustering and the consistency of exception magni- tudes to the underlying statistical model in the parametetric case.

3.1.1. Frequency The unconditional coverage test (Kupiec, 1995) is based on

comparing the observed number of violations to the expected. The hit variable, obtained from the ex post observation of EVaRð�Þ violations for threshold a and time t, denoted IEVaRt ðaÞ, is defined as:

IEVaRð�Þt ðaÞ ¼ 1 if rt < �EVaRðĥ;aÞt�1 0 otherwise;

( where rt is the return at time t, with t ¼ ½1;2; . . . ; T�.

If we assume that IEVaRt ð�Þ is iid, then, under the unconditional coverage hypothesis (Kupiec, 1995), the total number of VaR

exceptions, denoted HitEVaRt ðaÞ, follows a binomial distribution (Christoffersen, 1998), denoted BðT;aÞ:

HitEVaRð�Þt ðaÞ ¼ XT t¼1

IEVaRð�Þt ðaÞ � BðT;aÞ: ð9Þ

Under the null hypothesis, the likelihood ratio, LRuc, has the asymp- totic distribution8:

LRucI VaRð�Þ t ðaÞ ¼ 2 log baTI 1� baT�TI� �� �� log aTI 1� aT�TI� �� �� �!d v2ð1Þ;

ð10Þ

t ,

,

9 Theoretically, two interesting and limited situations may happen: an empty set of T ðaÞ (no correction can fit the output to the test) and an AT ðaÞ which is null (no rrection needed. In the empty case, no correction is acceptable for fulfilling the test ndition (the model is just so bad that it cannot be corrected). This could arise from a

tuation in which a numerical solution cannot be reached (because the grid-search is o large) or, more importantly, from a more annoying situation corresponding to a ilure of the VaR model under study. For instance, let us imagine a theoretical tuation in which the series of hits are exactly equal (all exceptions are of equal size): en there is no correction that leads to being in accordance with the confidence level ither the correction leaves the hits unchanged in terms of frequency – if not severe,

r makes all hits disappear – if too strict). However, in our estimation (see Fig. 4), a se of an empty set of AT ðaÞ never happened, and a nil correction (with a two digit

ccuracy) occurred only in fewer than 5% or so of the cases in the sample of 27,842 bservations (whatever the method of VaR computation). 0 We used a looped grid-search algorithm, adding successively a small increment

n the top of the VaR (+.1% of the EVaR at each step of the loop), starting from the aximum positive value and increasing until the test is finally passed at a given

robability threshold. 1 A generalization of the basic procedure leads to simple time-varying corrections,

here the original sequence is modified as VaRt1 ðĥ1;aÞ þ q1 : t1 ¼ ½1; . . . ; T1�; n

. ;VaRtk ðĥk;aÞ þ qk : tk ¼ ½k; . . . ; T1 þ k� 1�; . . .g and the optimization is done in all e arguments ðq1; . . . ; qk; . . .Þ, with the optimal adjustment at the end being the aximum of the sequence ðq1; . . . ; qk; . . .Þ.

78 C.M. Boucher et al. / Journal of Banking & Finance 44 (2014) 72–92

where the symbol!d denotes the convergence in distribution of the test statistic, TI ¼ T � E IEVaRð�Þt

h i is the number of exceptions andba ¼ TI=T is the unconditional coverage.

3.1.2. Independence Christoffersen (1998) proposed a test for the independence of

violations:

LRindI EVaR t ðaÞ ¼ 2 log LI

EVaR t ðaÞðp01;p11Þ � log LI

EVaR t ðaÞðp;pÞ

h i !d v2ð1Þ;

ð11Þ

where pij ¼ Pr IEVaRt ðaÞ ¼ jjI EVaR t�1 ¼ i

h i is a Markov chain that reflects

the existence of an order 1 memory in the process IEVaRt ðaÞ, LI

EVaR t ðaÞðp01;p11Þ ¼ ð1� p01ÞT00pT1001 ð1� p11Þ

T10pT1111 is thus the likeli- hood under the hypothesis of the first-order Markov dependence, LI

EVaR t ðaÞðp;pÞ is the likelihood under the hypothesis of independence,

such as p01 ¼ p11 ¼ p, with Tij the number of observations in the state j for the current period and at state i for the previous period, p01 ¼ T01=ðT00 þ T01Þ;p11 ¼ T11=ðT10 þ T11Þ and p ¼ ðT01 þ T11Þ=T .

3.1.3. Magnitude A third class of tests focuses on the magnitude of the losses

experienced when VaR limits are violated. While this is not relevant for methods such as historical simulation, it provides a useful evaluation of the parametric approaches. Berkowitz (2001), for instance, proposes a hypothesis test for determining whether the magnitudes of observed VaR exceptions are consistent with the underlying VaR model, such as:

LRmagctþ1 ¼ 2 Lctþ1magðl;rÞ � L ctþ1 magð0;1Þ

h i !d v2ð2Þ; ð12Þ

where ctþ1 is the magnitude variable of the observed VaR excep- tions, l and r are unconditional mean and standard deviation of ctþ1 series, and where

Lctþ1magðl;rÞ¼ X

fctþ1¼0g log 1�U U

�1ðaÞ�l r

( )( )

þ X

fctþ1–0g �1

2 logð2pr2Þ�ðctþ1�lÞ

2

2r2 � log U U

�1ðaÞ�l r

( )( )( ) :

For both unconditional and conditional coverage tests, Escanciano and Olmo (2009, 2010, 2011) alternatively approxi- mate the critical values of these tests by using a sub-sampling bootstrap methodology, since they show that the coverage VaR backtest is affected by model misspecification. Note that, interest- ingly, the bootstrapped versions of the tests always lead to lower VaR corrections, i.e. a dynamically corrected VaR which is less conservative.

3.2. A desirable VaR and the backtests

Under the H0 hypothesis, a desirable VaR passes each of these three test criteria:

LRucI VaRð�Þ t ðaÞ !d v2ð1Þ for the hit test;

LRindI VaRð�ÞðaÞ

t ! d v2ð1Þ for the independence test;

LRmagctþ1ðaÞt ! d v2ð2Þ for the exception magnitude test:

8>>><>>>: ð13Þ We now have to search for the minimal adjustment value q�

that allows us to pass all the tests (one-by-one, or jointly). For a given VaR forecast and the bounding range for the tests above, we can obtain the IMAVaR that respects conditions (10), (11) and/or (12) (or their sub-sampled versions). More precisely, given a sequence of predictions fVaRtðĥ;aÞ : t ¼ ½1; . . . ; T�g, we construct the set of values q 2 R such that the sequence

fVaRtðĥ;aÞ þ q : t ¼ ½1; . . . ; T�g passes several backtests. If we denote the set of accepted adjustments by ATðaÞ, the optimal adjustment is given by9:

q�T ¼ arg min|ffl{zffl} q2AT ðaÞ

fqg: ð14Þ

We use a numerical optimization technique to solve the program (14 ): During the adjustment process, we search for the optimal adjustment, starting with a large negative value of q�, increasing it slowly, until the adjusted VaR allows us to pass all the tests.10

The program (14) gives the optimal value of adjustment of the imperfect VaR estimation to become a desirable VaR. This means that the H0 hypothesis is true for the selected backtest method, so that the test statistic is lower than critical values for all tests at the threshold a. In what follows, in order to distinguish the effect of each test, we provide each correction separately, corre- sponding to each of the tests taken alone.11

As a first illustration, Fig. 3 provides the minimum adjustments (errors), denoted q� as solutions to the program (14). We first only consider the hit test, for the historical, the Gaussian and the GARCH VaRs computed on the DJIA over one century of daily data. The figure represents the minimal adjustment (in a percentage of the underlying VaR) necessary to respect the hit ratio criteria accord- ing to the VaR level of confidence (95–99.5%). This minimal adjust- ment is here considered as a proxy for the economic value of the model risk; it is expressed as a proportion of the observed average VaR.

In other words, we show the minimal constant that should be added to the quantile estimation for reaching a VaR sequence that passes the hit test at all times (here with full information at time T). We can see that the corrections range from (almost) 0% to 140% and increase with the quantile. The comparison between the three methods favors the GARCH method, since the error is lower for all quantiles and the difference between methods (with full information about the total sample) is quite similar and rather independent of the confidence level.

3.3. VaR model comparisons

We apply the general adjustment method presented above, obtained for the daily DJIA index from January 1st, 1900 until March 2nd, 2011 (29,002 daily returns). We use a moving window of four years (1040 daily returns) to re-estimate parameters dynamically for the various methods. Forecasted VaR are

A co co si to fa si th (e o ca a o

1

o m p

1

w

. .

th m

95.0% 95.5% 96.0% 96.5% 97.0% 97.5% 98.0% 98.5% 99.0% 99.5% −0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4 Historical Normal GARCH

Fig. 3. Minimum model risk adjustment factor for the hit test associated with historical, Gaussian and GARCH VaRs on the DJIA, for a range of probabilities. Daily DJIA index from the 1st January, 1900 to the 20th September, 2011. This figure represents on the y-axis the minimal adjustment (in a percentage of the underlying VaR) necessary to respect the hit ratio criterion according to the VaR level of confidence (x-axis). This minimal adjustment is here considered as a proxy of the economic value of the model risk; it is expressed as a proportion of the observed average VaR. The historical VaR is here computed on a weekly horizon as an empirical quantile using 5 years of past returns. The Gaussian and the GARCH VaRs are here computed on a weekly horizon as a parametric quantile using 5 years of past returns to estimate the parameters.

C.M. Boucher et al. / Journal of Banking & Finance 44 (2014) 72–92 79

computed dynamically for each method for the final 29,957 days (about 108 years). The out-of-sample exercise consists in a rolling forecast scheme with a window of four years (1040 daily returns) to re-estimate parameters dynamically. Then, we use one year of out-of-sample daily forecasts to calibrate the correction based on the backtesting procedures. The backtesting experiment to correct the risk model of VaR estimates is then based on a ratio of the out- of-sample to in-sample size equal to.24, i.e. 250/1040), which is sufficiently close to zero, as required for a valid out-of-sample exercise as shown by West (1996), McCracken (2000), Escanciano and Olmo (2010), Escanciano and Pei (2012). This comparison con- siders daily estimation of the 95%, 99% and 99.5% conditional VaR.

This leaves the choice of the VaR forecast method. While there is a large number of techniques that could be used, we restrict our- selves to the most common in practice, in particular historical sim- ulation and several parametric approaches based on Gaussian or Student-t return distributions, as well as the Cornish–Fisher VaR; see Cornish and Fisher, 1937; Favre and Galeano, 2002). We also employ three dynamic methods, EWMA, GARCH(1,1) and CAViaR (Engle and Manganelli, 2004). Finally, we complement these meth- ods by using two extreme densities for the returns, such as the GEV distribution and the GPD (see e.g. Engle and Manganelli, 2001).

Fig. 4 shows the optimal adjustment factor for the various risk models for a 95% VaR estimated with the DJIA, in particular the daily correction factors that pass the hit test over the past year of daily returns (over the period from t � 250 to t). The magnitude can sometimes be large (specifically around the 1929 and 2008 cri- ses), ranging from 0 to 15% EWMA or to more than 100% in some circumstances (for the Cornish–Fisher VaR). We also see that the most extreme VaR violations happened during the Great Depres- sion for all measures. Dynamic measures, such as EWMA, GARCH and CAViaR, also demonstrate some superiority over unconditional parametric methodologies.

Fig. 5 illustrates the evolution of the maximum required correc- tions for all VaR methods under consideration (maxima of the his- torical correction record needed from January 1st, 1900 to the current date t, which were already represented in Fig. 4).12 These corrections are for the hit test, from the general program aiming to correct today’s VaR with the historical maximum of the minimum correction that has been necessary since the beginning of the series (expressed here in relative terms compared to the level of VaR).

Fig. 6 illustrates the minimum dynamic adjustment required for

12 We did the same estimation and backtesting with a 10-year sample for VaR. We obtained the same qualitative results and saw that the choice of the size for VaR estimation is not crucial in our case. The results are available on demand.

passing the hit test for a randomly chosen first date of implemen- tation. More precisely, the exercise consists of choosing a first date and then computing the dynamic adjustment until the end of the sample; repeating this exercise 30,000 times, whilst ultimately keeping, for each horizon, the minimum correction obtained. The optimal adjustments are here expressed in terms of a percentage of their maximum value over the whole sample. For each horizon (x-axis in Fig. 6), the correction (on the y-axis) thus corresponds to the worst case scenario, i.e. the smallest correction required in the various samples of the same horizon).

The figure shows that, depending on the VaR method, the time period length for having almost all of the maximum correction fac- tors varies from 18 years (GEV) to 46 years (CAViaR). Moreover, regardless of the model, the major part (80% or so) of the correction factors is reached after 10 years. This means that, whatever the VaR model, most of the greatest surprises have been faced after a dec- ade of history (even in the worst scenario when the sample is amongst the least turbulent ones). In other words, at least ten years are needed to have a fairly good idea of the magnitude of the required correction factors.

We next consider the three main qualities of VaR models as a generalization of the approach by Kerkhof et al. (2010). Table 3 reports the various minimum required corrections related to the three main categories of tests, together with their Escanciano and Olmo (2009, 2010, 2011) bootstrapped corrected versions. We first note that the hit test is less permissive when the bootstrapped crit- ical values are used, whilst the tests of independence and magni- tude impose very severe corrections (to the order of 100% in relative terms for some tests).

According to the the unconditional coverage test at a 5% level, EWMA is the best model for estimating the DJIA index 95% VaR, fol- lowed by GARCH and then GEV. The independence test favors the conditional methods, with the best result for the GARCH model. Finally, when considering the magnitude of the violations—the most severe test—once again the dynamic measures show some superiority, whilst the extreme density VaR exhibits weakness.

3.4. Generalized model risk of model risk

Finally, we compare our method with classical stress-test exer- cises. We first present the extent to which the required calibrated correction factors can provide an insurance against major historical financial crises. Then, we compare the correction factors implied by the various backtests to correct the model risk of risk models, to a typical stress-test scenario.

1907 1922 1936 1951 1966 1981 1996 2011 −8.0% −4.0%

0.0% 4.0%

Historical

0.0% 4.0%

Normal −8.0% −4.0%

0.0% 4.0%

Student

−8.0% −4.0%

0.0% 4.0%

Cornish −Fisher −8.0% −4.0%

0.0% 4.0%

RiskMetrics

−8.0% −4.0%

0.0% 4.0%

GARCH −8.0% −4.0%

0.0% 4.0%

CAViaR

1908 1933 1959 1985 2011 −8.0% −4.0%

0.0% 4.0%

GEV

1908 1933 1959 1985 2011 −8.0% −4.0%

0.0% 4.0%

GPD

−8.0% −4.0%

Fig. 4. Dynamic optimal adjustment on the daily 95% VaR. Daily DJIA index from the 1st January, 1900 to the 20th September, 2011. We use a moving window of four years (1040 daily returns) to re-estimate parameters dynamically for the various methods.

1907 1922 1936 1951 1966 1981 1996 2011 0.0%

5.0% Historical

0.0%

5.0% Normal

0.0%

5.0% Student

0.0%

5.0% Cornish −Fisher

0.0%

5.0% RiskMetrics

0.0%

5.0% GARCH

0.0%

5.0% CAViaR

1908 1933 1959 1985 2011 0.0%

5.0% GEV

1908 1933 1959 1985 2011 0.0%

5.0% GPD

Fig. 5. Optimal dynamic absolute value of minimum negative adjustments for the hit test for different methods and the 95% VaR. Daily DJIA index from the 1st January, 1900 to the 20th September, 2011. We use a moving window of four years (1040 daily returns) to re-estimate parameters dynamically for the various methods.

80 C.M. Boucher et al. / Journal of Banking & Finance 44 (2014) 72–92

Three implicit levels of confidence are required: the probability level of the VaR under consideration, the thresholds in the various tests applied for computing the required correction and, finally, the degree of confidence we want to put on the solidity of the buffer. Typically, a high probability VaR focus will increase the model risk, whilst a more severe test level leads to a lower risk. Consequently, a high incremental buffer leads to a high protection against the model risk that is realized during extreme events in the market. By contrast, a reduced buffer decreases the insurance against these major turbulent episodes and, then, ultimately increases failures of (corrected) risk models.

Fig. 7 below illustrates this link between the level of the buffer, here translated into protection against the more severe historical crises, and the degree of confidence associated to the buffer. The Figure represents the cumulative density functions of required adjustments (in the last century of the DJIA) for, respectively, the historical and GARCH(1,1) VaR at a 95% confidence level, with a threshold for the hit test fixed at 5%. The series of dates stand for

years corresponding to the largest exceptions for the two VaR methods for certain levels of confidence (on the y-axis) and related corrections (on the x-axis). We note here that the GARCH VaR leads to smaller corrections in general. We also see that if we accept a 5% model risk, we are, unsurprisingly, not protected anymore against the 5% biggest shocks in the data (such as, for instance, those of 1929, 1930, 2008 and 2009 for the historical method).

We then compare the correction applied to assess the robust- ness of risk estimates with the correction implied by a typical stress test exercise for usual portfolio profiles by imposing hand- picked shocks for each investment class. We provide these compar- isons in terms of factor k used by regulators for determining capital (k being between 3 and 5).

Thus, we first present in Table 4 (Panel A and B) the various (model risk free) minimum corrections corresponding to the three tests (frequency, independence and magnitude) at a 5% confidence level for a 95% GARCH VaR applied to financial series of daily return on indexes and profiled portfolios in the period from December

0 10 20 30 40 50 0%

50%

100%

Historical

0 20 40 60 0%

50%

100%

Normal

0 20 40 60 0%

50%

100%

Student

0 20 40 60 0%

50%

100%

Cornish Fisher

0 20 40 60 0%

50%

100%

RiskMetrics

0 20 40 60 0%

50%

100%

GARCH

0 20 40 60 0%

50%

100%

CAViaR

0 20 40 60 0%

50%

100%

GEV

0 20 40 60 0%

50%

100%

GPD

Fig. 6. Optimal dynamic relative adjustment for the hit test for different starting dates and 95% VaR by horizon (in years). Daily DJIA index from the 1st January, 1900 to the 20th September, 2011. We use a moving window of four years (1040 daily returns) to dynamically re-estimate parameters for the various methods. This figure illustrates the dynamic negative adjustment required for passing the hit test (see Fig. 4), having randomly chosen the first date of implementation. Optimal relative negative adjustments are here expressed in terms of percentage of their maximum value over the whole sample.

Table 3 Minimum model risk for 95% daily VaR models for various validity tests with a 5% confidence level.

Method Mean VaR (%)

q1 (%) q�1 (%) q2 (%) q � 2 (%) q3 (%) q

� 3 (%)

Historical �1.60 �2.61 �2.03 �4.85 �3.24 �3.10 �5.90 Normal �1.68 �2.66 �1.86 �4.62 �2.76 �2.76 �5.49 Student �1.89 �2.49 �1.86 �4.25 �2.85 �3.11 �6.30 CF �1.26 �8.29 �7.48 �8.40 �8.86 �8.40 �8.86 EWMA �1.59 �.98 �.65 �2.03 �1.02 �1.02 �2.89 GARCH �1.61 �1.13 �.96 �2.57 �1.15 �1.20 �2.46 CAViaR �1.66 �1.87 �1.55 �2.59 �2.22 �2.08 �2.56 GEV �1.84 �2.42 �1.99 �4.47 �2.99 �2.80 �6.97 GPD �2.11 �2.35 �1.67 �4.43 �2.63 �2.71 �6.51

Daily DJIA index from the 1st January, 1900 to the 20th September, 2011. We use a moving window of four years (1040 daily returns) to dynamically re-estimate parameters for the various methods. The variable q1 refers to the hit test; q2 to the independence test; q3 to the magnitude test; and q�1; q

� 2, q

� 3 correspond to their

resampling versions, following Escanciano and Olmo (2009, 2010, 2011).

C.M. Boucher et al. / Journal of Banking & Finance 44 (2014) 72–92 81

31st, 1986 to November 28th, 2011. We consider four asset classes as well as three investment profiles combining these asset classes (defensive, balanced and aggressive portfolios).13

We express the outcomes as a percentage of VaR in Table 4 (Panel A), whilst presenting them as k ratios of corrected VaR out of estimated VaR in Panel B of Table 4. The correction factors in Panel A of Table 4 for single indexes range from �3.65% (for q3— magnitude correction for the commodity index) to �63.83% (for q�3—bootstrapped magnitude correction for the real estate index). For the various profiles, we see that the correction factor is lower than 1% for the defensive profile and goes to 10% or so for the aggressive one (and to �35.05% when considering the most severe test of magnitude). When these correction factors are expressed in terms of k ratios in Panel B of Table 4, they range from 1.01 to 3.66 which is in line with the official k ratio between 3 and 5.

We can now compare the correction factors, calibrated based on our framework, with a standard stress–test approach supposing some typical shocks on various asset classes. As underlined by Breuer and Csiszár (2014), stress tests with hand-picked scenarios are subject to two significant criticisms. First, arbitrary severe sce- narios may be too implausible. Second, some other stress scenarios leave open the question of whether there are more severe scenar- ios of similar plausibility. If the considered scenarios are harmless, either because stress testers lack proficiency or wish to hide risks, stress tests convey a feeling of safety which might be false. If they are merely unrealistic, they lead falsely to excessively high capital. Our proposed strategy can help to gauge the severity (and plausi- bility) of an ad hoc handpicked specific scenario.

Focusing indeed on the k ratios, Panel C of Table 4 reports the implied corrections on annual 95% GARCH VaR in the case of a hypothetical stress. With the given intensity of shocks considered here14 (�30% for the equity index, �40% for the real estate, �30% for commodity and �20% for bonds over a one-year horizon), k ratios

13 For the bonds, we use the ‘‘Merrill Lynch U.S. Treasuries/Agencies-Master AAA’ index before 01/01/1998 and the ‘‘J.P. Morgan EMU Global Aggregate Bond AAA Al Maturities’’ after; for the equity class we use a composite index ‘‘95% MSCI Europe Index +5% MSCI World Index’’; for the real estate class we get the ‘‘European Rea Estate Investment and Services Index’’ and for commodity, the ‘‘CRB Spot Index’’.

14 The amplitude of the shocks is directly inspired from recommendations of the Committee of European Insurance and Occupational Pensions Supervisors (CEIOPS).

’ l

l

vary from 1.90 (for q2 – independence correction for the equity index) to 4.99 (for q3—magnitude test for the real estate index) for the single indexes, and from 1.54 (for q2 – independence correction for the balanced profile) to 6.10 (for q3—magnitude test for the aggressive portfolio).

If we now compare the results in Panel C of Table 4 (ad hoc stress tests) to those in Panel B of Table 4 (calibrated empirical corrections), the arbitrary implied corrections of the stress test scenarios appear to be far more severe for almost all indexes and portfolios (except for the balanced one and the independence test). We thus conclude that this illustrative stress-test is very conserva- tive. In other words, because k ratios are almost higher in Panel C of Table 4 than in Panel B of Table 4 (on average by 80%), this stress- test seems to be relatively robust to the impact of model risk for the risky assets.

Taken altogether, our results suggest that some VaR models are preferred (e.g. the dynamic approaches such as the EWMA, CAViaR and GARCH models), whilst others should be avoided (e.g. the Cornish–Fisher VaR or extreme distribution based VaR) when com- paring the minimum correction to pass the frequency/hit test. Moreover, the independence and the magnitude tests lead to more

−4.0% −3.5% −3.0% −2.5% −2% −1.5% −1.0% 0.0%

1.0%

2.0%

3.0%

4.0%

5.0%

6.0%

7.0%

8.0%

9.0%

10.0%

1920; 1931; 1932; 1933; 1937; 1938; 1946; 1947; 1987; 1988

1929; 1930; 2008; 2009

1907; 1908; 1920; 1921; 1926; 1932; 1933; 1937; 1988; 2008; 2009

1920; 1929; 1930; 1938

Historical GARCH

Fig. 7. The empirical cumulative density function of optimal adjustment values for the hit test of a 95% daily historical and GARCH VaR. Daily DJIA index from the 1st January, 1900 to the 20th September, 2011. We use a moving window of four years (1040 daily returns) for computing the VaR. The threshold for the hit test is here fixed at 5% and we use a Gaussian kernel smoothing density (see Bowman and Azzalini, 1997).

Table 4 Minimum model risk for a 95 GARCH-VaR, k ratio model risk confidence levels for a 95 GARCH-VaR and 95% stress-VaR for 5% validity tests on various portfolios.

Portfolio q1 q�1 q2 q � 2 q3 q

� 3

Panel A. Minimum annualized model risk for a 95% GARCH-VaR Equity �10.15% �7.14% �9.86% �15.12% �44.80% �16.44% Real estate �12.65% �10.32% �16.53% �18.93% �63.83% �25.03% Commodity �6.39% �6.25% �5.29% �6.99% �13.76% �3.65% Bond �9.89% �9.62% �10.27% �10.54% �18.44% �13.62%

Defensive profile �.08% �.08% .00% �.21% �1.04% �.26% Balanced profile �4.63% �4.36% �5.88% �6.52% �15.79% �8.74% Aggressive profile �9.28% �8.38% �8.52% �11.62% �35.05% �12.72%

Panel B. Minimum k ratio model risk confidence levels for a 95% GARCH-VaR Equity 1.35 1.25 1.34 1.53 2.56 1.57 Real estate 1.40 1.33 1.53 1.60 3.03 1.80 Commodity 1.65 1.64 1.54 1.72 2.41 1.37 Bond 2.43 2.39 2.48 2.52 3.66 2.97

Defensive profile 1.15 1.15 1.01 1.40 3.00 1.50 Balanced profile 1.45 1.42 1.57 1.63 2.52 1.84 Aggressive profile 1.42 1.38 1.38 1.52 2.58 1.57

Panel C. Minimum k ratio model risk confidence levels of 95% stress-VaR Equity 2.54 2.25 1.90 2.71 4.81 3.29 Real estate 3.11 2.84 3.18 3.80 4.99 4.84 Commodity 2.89 2.89 3.27 2.92 3.83 3.19 Bond 3.51 3.50 3.52 3.56 4.04 3.76

Defensive profile 2.11 2.11 2.11 2.12 2.20 2.16 Balanced profile 2.63 2.50 1.54 2.63 5.10 3.81 Aggressive profile 3.08 2.94 1.83 3.78 6.10 4.94

Datasource: DataStream and Bloomberg. Daily data from the 31st December, 1986 to the 28th November, 2011; computations by the authors. The asset classes as detailed in Footnote 13. A moving window of four years (1040 daily returns) is used to re-estimate parameters dynamically for the various methods. ‘‘Defensive Profile’’ corresponds to a mixed portfolio compound with 10% bond + 90% Liquidity; ‘‘Balanced Profile’’ 30% equity + 10% Real Estate + 10% commodity + 40% bond + 10% liquidity; and ‘‘Aggressive Profile’’ 70% equity + 15% real estate + 15% commodity. The variable q1 refers to the hit test; q2 to the independence test; q3 to the magnitude test; and q�1; q

� 2, q

� 3 correspond

to their resampling versions, following Escanciano and Olmo (2009, 2010, 2011). Panel A gives the minimum annualized corrections for backtest at 5% confidence level on a 95% GARCH-VaR, Panel B the minimum k-ratio (adjustment/VaR) for a 95% GARCH-VaR and Panel C the minimum k ratio model in the stress-VaR context for 5% validity tests. The following shocks are considered for Panel C: �30% for the equity index, �40% for the real estate, �30% for commodity and �20% for bonds over a one-year horizon.

82 C.M. Boucher et al. / Journal of Banking & Finance 44 (2014) 72–92

severe corrections on the estimated VaR than the frequency test does. But whatever the model, the magnitude of the correction factors can be sometimes exceptionally large, especially during major financial crisis episodes such as the Great Depression of 1929 or the crisis of 2008. This is why there is a direct link between the confidence level on the required ex post correction (on the full historical sample), and the insurance against these major historical financial turmoils. However, we also show that a 10 year sample of

observations for calibrating the minimum correction to be added, is sufficient to have a fairly good idea of the magnitude of the model risk of risk models.

4. Conclusions

Standard risk measures failed to forecast extreme risks and reg- ulators require that financial institutions quantify this model risk

C.M. Boucher et al. / Journal of Banking & Finance 44 (2014) 72–92 83

of risk models. We propose to adjust risk forecasts for model risk by the historical performance of the model. In other words, the risk model learns from its past mistakes.

We first examine standard risk models by assessing how well they forecast risk from a simulated process, designed to realisti- cally capture the salient features of financial returns. The experi- ment shows that model risk is significant and ever present, in some cases, so large that it exceeds the actual risk forecast.

In our main contribution, we then propose a methodology for explicitly incorporating model risk corrections into risk forecasting by taking into account the models’ performance on a range of stan- dard back testing methodologies.

The general setup also enables us to evaluate the performance of standard risk forecast models, by applying the basic principle that the lower the model risk correction factor, the lower the model risk and, therefore, the better the model. The results show that dynamic methods, such as EWMA, CAViAR and GARCH VaR, have an advantage over static approaches such as Gaussian and extreme density approaches. Somewhat surprisingly, the very sim- ple historical simulation approach is, if not the best method, close to the best.

We conclude by proposing an approach that provides a tailored methodology for risk managers where they can explicitly relate the degrees of confidence in the correction factor to the distribution of past violations. In this, the manager addresses three concerns: the VaR probability, the severity of tests and the trust we want to put into the correction buffer. This can, for example, enable a risk man- ager to explicitly consider extreme events, such as 1929 and 2008, or alternatively disregard their impact on risk forecasts.

The Basel Committee has recently proposed (BCBS, 2013) the use of a stressed risk forecast as the main input into the current risk forecast. Such an approach is an improvement over the exist- ing methodology, and is partially consistent with our methodology. The Committee indeed proposes to rescale the risk forecasts by the ratio of the stressed and unstressed risk factors, such as the adjusted current risk forecast becoming more conservative and thus less prone to exceptions. However, our proposal deals with this in a more precise way. First, we adjust risk forecasts by their past errors, which mainly come from these distressed periods. Sec- ond, we consider a confidence level about the required correction factor, linked to the insurance against major financial stress episodes. Finally, we define proper criteria for adjusting the risk forecasts based on some properties of forecast errors such as their frequency, their independence and their magnitude. In our view, the Basel Committee proposal still ignores the model risk of risk forecasts and consists of an adjustment of the current risk without an explicit criterion.

Our work can be extended in several ways. Our general correc- tion framework can be used when comparing the various tests of a desirable VaR proposed in the literature (Berkowitz et al., 2011). The second extension could be to apply some specific VaR models when judging the riskiness of some non-linear products using, this time, several pricing models. In the same vein, evaluating the impact on asset allocation of integrating the model risk of risk measures could be of interest, especially for asset allocation para- digms depending on risk budgets, e.g. safety first criteria. The third extension could be found in generalizing the comparison consider- ing several time-horizons (e.g. Cheridito and Stadje, 2009; Hoogerheide et al., 2011) or several quantile levels (Colletaz et al., 2013). The fourth extension is about alternative backtests when calibrating our model risk correction (see Appendix for a list of tests), in particular when the VaR violations are clustered. For this purpose, the recent D-test of Escanciano and Pei (2012), the MCS-tests by Ziggel et al. (2013), the Geometric-VaR test by Pelletier and Wei (2014), and the Multi-level VaR test by Leccadito et al. (2014), because of their shown finite-sample size

and power properties, might be of interest as complementary backtest criteria for strengthen the safety of the buffer for model risk. Another approach would be to adopt the same methodology leading to an estimated multi-VaR, built as a portfolio of various VaR models (see Abdous and Remillard, 1995), directly aiming to minimize the model risk (McAleer et al., 2013). Finally, using the same metric of corrections, the quality of other VaR based mea- sures in a context of systemic risk measures (such as Marginal Expected Shortfall or CoVaR) would be worth considering (e.g. Daníelsson et al., 2014; Benoit et al., 2013; Löffler and Raupach, 2013).

Acknowledgements

We thank Carol Alexander, Arie Gozluklu, Monica Billio, Thomas Breuer, Massimiliano Caporin, Rama Cont, Christophe Hurlin, Christophe Pérignon, Michaël Rockinger, Thierry Roncalli and Jean–Michel Zakoïan for suggestions when preparing this article, as well as Benjamin Hamidi for research assistance and joint col- laborations on collateral subjects. Authors thank the Global Risk Institute for support; the second author gratefully acknowledges the support of the Economic and Social Research Council (UK) [Grant No.: ES/K002309/1] and the fourth author the support of the Risk Foundation Chair Dauphine–ENSAE–Groupama ‘‘Behav- ioral and Household Finance, Individual and Collective Risk Attitudes’’ (Louis Bachelier Institute). Some extra materials related to this article can be found at: www.riskresearch.org. The usual dis- claimer applies.

Appendix A. Model risk when forecasting risk

Financial risk forecast models, just like any other statistical model, are thus subject to model risk. In spite of this, almost all presentations of risk forecasts focus on point estimates, omitting any mention of model risk, not even mentioning estimation risk. They are, however, subject to the same basic elements of model risk as any other model, and are also subject to unique model risk factors because of the specific application.

In order to formally identify the model risk factors, we propose a five level classification scheme:

1. Parameter estimation error arises from uncertainty in the parameter values of the chosen model.

2. Specification error refers to the model risk stemming from inappropriate assumptions about the form of the data generating process (DGP) for the random variable.

3. Granularity error is based on the impact of undiversified idiosyncratic risk on the portfolio VaR.

4. Measurement error relates to the use of erroneous data when measuring the risks and testing the models.

5. Liquidity risk is defined as the consequence of both infre- quent quotes and the inability to conduct sometimes a transaction at current market prices because of the too large size of the transaction.

The ultimate objective is to forecast VaR, where we indicate the estimate by ‘‘estimated VaR’’ (denoted EVaR). It is a function of the portfolio size and the true model parameters h0. In what follows, VaR is the ð1� aÞth quantile (with a > :50Þ of the profit and loss distribution, so that the VaR is negative (and expressed hereafter as a return for the sake of simplicity). We also indicate the theoret- ical (or true) VaR by ThVaRðh0;aÞ. Thus, when comparing the esti- mated VaR with the theoretical VaR (i.e. EVaR and ThVaR respectively), we present both the buffer needed to directly adjust the EVaR and the probability (or quantile) shift required. Our

84 C.M. Boucher et al. / Journal of Banking & Finance 44 (2014) 72–92

objective is to approximate the errors or ‘‘biases’’ of VaR estimates since we do not know the ‘‘true’’ DGP with real data. Biases defined hereafter are ‘‘errors’’ (that can be repeated) that come mainly from the use of a wrong model and/or the wrong specification regarding the ‘‘true’’ (assumed DGP). Our proposed procedure consists of approximating these errors, based on the minimum cor- rection needed not to reject a predefined consensual backtest. In the following sub-sections, we detail these specific model risks that impact VaR forecasts and provide some examples.

A.1. Estimation risk

Estimation risk occurs in every estimation process. Relatively small changes in the estimation procedure or in the number of data observations can change the magnitude and even the sign of some important decision variables. Thus, estimation risk is the risk associated with an inaccurate estimation of parameters, due to the estimator quality and/or limited sample of data (past and/or future), and/or noise in the data.

If PEAVaR denotes the perfect estimation adjusted VaR, EVaRðĥ;aÞ the estimated VaR and biasðĥ; h0;aÞ the bias function, where ĥ are the estimated parameters, we have:

PEAVaRðĥ; h0;aÞ ¼ EVaRðĥ;aÞ þ biasðĥ; h0;aÞ: ðA:1Þ

Example 1. As an illustration, assuming an ARCH model, the estimation risk (denoted herein ERð�Þ) is expressed in Gouriéroux and Zakoïan (2013), as (with the previous notations):

EVaRðĥ;aÞ ¼ ThVaRðh0;aÞ þ ERðThVaRðh0;aÞ; ĥ;aÞ;

with

ER½ThVaRðh0;aÞ; ĥ;a� ¼ �ð2TÞ�1h½ThVaRðh0;aÞ; ĥ;a� þ oðT�1Þ;

where T is the length of the estimation period, oðT�1Þ converges to a term of order T�1 and:

h½ThVaRðh0;aÞ; ĥ;a� ¼ @2g�1

@r2 ðrt�1;h;rÞ�ThVaRðh0;aÞ

@g�1

@r ðrt�1;h;rÞ

2( )( )

� @g @h0

rt�1;h;ThVaRðh0;aÞ½ �Xðh0Þ @g @h0 ½rt�1;h;ThVaRðh0;aÞ�

þ@g �1

@r ðrt�1;h;rÞTr Xðh0Þ

@2g @h@h0

½rt�1;h;ThVaRðh0;aÞ� ( )

;

and r ¼ g½rt�1; h; ThVaRðh0;aÞ�;XðhÞ the variance–covariance of parameters in h; gð:Þ a continuous function, strictly increasing with respect to the VaR parameter and g�1ð:Þ its inverse.

A.2. Specification risk

Specification error arises from using inappropriate assumptions about the form of the DGP. We propose denoting the strong form of specification risk as the risk from using a risk model which cannot capture the true unknown DGP. The weak form of specification risk then corresponds to the risk of using a risk model inadequate with the assumed, and hence known, DGP.

Consider the special case of knowing the true model parame- ters, but not knowing the model. In this case, we can define the perfect specification adjusted VaR (PSAVaR) as:

PSAVaRðh0; h1;aÞ ¼ EVaRðh1;aÞ þ biasðh0; h1;aÞ; ðA:2Þ

where h1 are known parameters, defined so that we can link the misspecified model to the true model, with some mapping h0 ¼ f ðh1Þ.

Example 2. A simple measure of the specification risk (denoted as SRð�Þ) associated to the expansion of the unknown ‘‘true’’ theoret- ical model of VaR (denoted ThVaRðh;aÞ), can be written as:

EVaRðĥ;aÞ ¼ ThVaRðh;aÞ þ SR½ThVaRðh;aÞ; ĥ;a�;

with

SR½ThVaRðh;aÞ; ĥ;a� ¼r 6

AVaRðĥ;aÞ�l r

" #2 �1

8<: 9=;Sk

þ r 24

AVaRðĥ;aÞ�l r

" #3 �3 AVaRðĥ;aÞ�lr

" #8<: 9=;Ku

� r 36

2 AVaRðĥ;aÞ�l

r

" #3 �5 AVaRðĥ;aÞ�l

r

" #8<: 9=;Sk2

þoðT�1Þ;

where ThVaRðh;aÞ is the ‘‘true’’ theoretical model of VaR, AVaRðĥ;aÞ is the asymptotic a-quantile of the approximate model in use, SRð�Þ is the specification error associated to this specific model and parameters l, r; Sk and Ku stand, respectively, for the mean, the standard deviation, the skewness and the kurtosis of the return distribution.

A.3. Granularity error

Granularity error is caused by the bias resulting from a finite number of assets in portfolios and then by the resulting residual idiosyncratic risk, see e.g. Gordy (2003) and Wilde (2001). The granularity principle yields a decomposition of such risk measures that highlights the different effects of systematic and non-system- atic risks.

More precisely, any portfolio risk measure can be decomposed into the sum of an asymptotic risk measure corresponding to an infinite portfolio size and 1=n times an adjustment term where n is the portfolio size (number of assets). The asymptotic portfolio risk measure, called the cross-sectional asymptotic risk measure, captures the non-diversifiable effect of risks on the portfolio. The adjustment term, called granularity adjustment, summarizes the effect of the individual specific risks and their cross-effect with systematic risks, when the portfolio size is large, but finite.

Suppose the theoretical VaR is based on an asymptotic factorial model, valid asymptotically. In this case, we can apply a similar adjustment factor to arrive at the perfect granularity adjusted VaR (PGAVaR) so that:

PGAVaRðh0;a;nÞ ¼ EVaRðh0;a;NÞ þ biasðh0;a;nÞ; ðA:3Þ

where n is the number of assets in the portfolio under study and N a large number of assets for which the asymptotic model is valid.

Example 3. As an illustration, and following here Gagliardini and Gouriéroux (2013), in the special case of independent stochastic drift and volatility, the granularity risk (denoted below GRð�Þ) that impacts the estimated VaR can be expressed as (with the previous notations):

EVaRðh;a;NÞ ¼ ThVaRðh;a;nÞ þ ðn�1ÞGRðaÞ þ oðn�1Þ;

with

GRðaÞ ¼ �ð2�1ÞEfr2½q�g � dlogf ðqÞ dq

� � ;

where n is the number of assets in the portfolio under study, N a large number of assets for which the asymptotic model is valid,

Table B.1 A road map of the main risk model validation tests.

C.M. Boucher et al. / Journal of Banking & Finance 44 (2014) 72–92 85

q ¼ EVaRðh0;a;NÞ is the quantile of a factor G and f ð�Þ is its density function.

Exception frequency tests Intuition: test the violation frequency that should be equal to the probability

threshold An Unconditional Coverage Test – Kupiec (1995) A GMM Duration Test – Candelon et al. (2011) A Z-test – Jorion (2007) A Multi-variate Unconditional Coverage Test – Pérignon and Smith (2008) A D-test – Escanciano and Pei (2012) A MCS Test – Ziggel et al. (2013)

Exception independence tests Intuition: test the violations associated to the VaR forecasting that should be

independent (not clustered and/or no forecasting power via a time-series model for extremes) An Independence Test – Christoffersen (1998) A Violation Duration-based Test – Christoffersen and Pelletier (2004) A Discrete Violation Duration-based Test – Haas (2005) A Dynamic Quantile Test – Engle and Manganelli (2004) A Dynamic Quantile Test – Gaglianone et al. (2011) A GMM Duration Test – Candelon et al. (2011) A Multivariate Test of Zero-autocorrelation of Violations – Hurlin and Tokpavi

(2006) An Estimation-risk adjusted Test – Escanciano and Olmo (2009, 2010, 2011) A MCS Test – Ziggel et al. (2013)

Exception frequency and independence of violations tests Intuition: test jointly the hit ratio and the independence of VaR violations A Conditional Coverage Test – Christoffersen (1998) A GMM Duration Test – Candelon et al. (2011) A Dynamic Binary Response Test – Dumitrescu et al. (2012) A Geometric-VaR Test – Pelletier and Wei (2014) A MCS Test – Ziggel et al. (2013) A Multilevel Test – Leccadito et al. (2013)

Exception magnitude tests Intuition: test the amplitude of VaR violations (that should be small)

A.4. Measurement error

Financial data are prone to measurement errors caused by var- ious phenomena such as non-synchronous trading, rounding errors, infrequent trading, micro-structure noise or insignificant volume exchanges. In addition, observed data might be subject to manipulations (smoothing, extra revenues, fraudulent exchanges, informationless trading, etc).

Measurement error risk can strongly distort backtesting results and significantly affects the performance of standard statistical tests used to backtest VaR models. Frésard et al. (2011) extensively document the phenomena and report that a large fraction of banks artificially boost the performance of their models by polluting their ‘‘true’’ profit and loss with extra revenues that cause under-esti- mation of the true risk.

Example 4. Certain financial institutions report a contaminated P&L (denoted PLct ) with extraneous profits (denoted pt)) such as intraday revenues, fees, commissions, net interest incomes and revenues from market making or underwriting activities as such:

PLct ¼ PLt þ pt;

with PLt the true profit at time t. So, the estimated VaR is impacted by a contamination risk

(denoted CRð�Þ) that reads:

EVaRðh;a;pÞ ¼ ThVaRðh;aÞ þ CRðpÞ:

Degiannakis (2007) A Double-threshold Test – Colletaz et al. (2013)

Exceedances for expected shortfall test Intuition: Measure the observed ES, then compare to a local approximated value (and the difference should be small) A Saddlepoint Technique Test for ES – Wong (2008, 2010)

See, among others, Campbell (2007), Nieto and Ruiz (2008) and Berkowitz et al. (2011) for comprehensive surveys.

A.5. Liquidity risk

While liquidity has many meanings, from the point of view of risk forecasting, the most relevant are some aspects of market liquidity, as defined by the BCBS (2010), such as the ability to quickly trade large quantities, at a low cost, without impacting the price. These directly follow from Kyle’s (1985) three dimen- sions of liquidity: tightness, depth and resilience.

For portfolios of illiquid securities, reported returns will tend to be smoother than true economic returns, which will understate volatility and increase risk-adjusted performance measures such as the Sharpe ratio. As an extreme example of illiquidy, we can mention that the NY stock exchange remained shut for more than four months at the beginning of the First World War (from the 31st July, 1914 to the 12th December, 1914) and that the re-opening brought the largest one-day percentage drop in the DJIA (�24.4%).15

Getmansky et al. (2004) propose, for instance, an econometric model of illiquidity exposure and develop estimators for the smoothing profile as well as a smoothing-adjusted Sharpe ratio (that basically leads to the intensification of the measured smoothed volatility by a factor to recover a proxy of the true underlying volatility). Measures for gauging illiquidity exposure of several asset classes are presented in Chan et al. (2006). Liquidity aspects enter the Value-at-Risk methodology quite naturally. The VaR approach is built on the hypothesis that ‘‘market prices represent achievable transaction prices’’ (Jorion, 2007). In other words, the prices used to compute market returns in the VaR models have to be representative of market conditions and traded volume. Consequently, the price impact of portfolio liquidation has to be taken into account. Chordia et al. (2001) find a significant cross-sectional relation between stock returns and the

15 See e.g. Silber (2005).

variability of liquidity, which is approximated by measures of trad- ing activity such as volume and turnover. Giot and Grammig (2005), using a weighted spread in an intraday VaR framework, show that accounting for liquidity risk becomes a crucial factor and that the traditional (frictionless) measures severely underesti- mate the true VaR.

Example 5. As a simple illustration, we can formalize that risk using the following relation (with the previous notations):

P̂Lt ¼ PLt þ p1;t þ 1ILep2;t ;

where p1;t is a factor that contributes to the smoothing of the released prices and p2;t a liquidity risk premium that only occurs when a liquidity event happens (denoted Le, such as quotation interruption, due to large movement in the market related to an exogenous shock: war, terrorist attack, a large collapse . . .), mod- elled here thanks to a Heaviside function (1I�) that takes the value 1when the event happens, which leads to a biased estimated VaR with a liquidity risk (denoted LRð�Þ) as:

EVaRðh;a;p1;p2Þ ¼ ThVaRðh;aÞ þ LRðp1;p2Þ:

Table C.1 Illustrations of unconditional simulated errors associated to the 95%, 99% and 99.5% annualized VaR: Gaussian versus t-Student quantiles.

Probability Mean estimated VaR Perfect VaR Mean bias Median bias Min. bias Max. bias

Panel A. Gaussian DGP and Gaussian VaR with estimation error a = 95.00 �29.49 �29.49 .00 .00 �7.93 7.24 a = 99.00 �41.88 �41.88 .00 .00 �9.92 9.17 a = 99.50 �46.41 �46.41 .00 .00 �12.45 10.16

Panel B. t-Student(5) DGP and Gaussian VaR with specification error a = 95.00 �29.49 �36.22 �6.73 �6.73 �6.73 �6.73 a = 99.00 �41.88 �60.75 �18.87 �18.87 �18.87 �18.87 a = 99.50 �46.41 �72.87 �26.46 �26.46 �26.46 �26.46

Panel C. t-Student(5) DGP and Gaussian VaR with specification and estimation errors a = 95.00 �29.49 �36.22 �6.73 �6.73 �13.97 1.20 a = 99.00 �41.88 �60.75 �18.87 �18.87 �28.04 �8.95 a = 99.50 �46.41 �72.87 �26.46 �26.46 �36.62 �14.01

Source: Bloomberg; daily data of the DJIA index in USD from the 1st January, 1900 to the 20th September, 2011. These statistics were computed with the results of 100,000 simulated series of 250 daily returns according to a specific DGP (Gaussian for Panel A and t-Student(5) for Panel B and C) and using an annualized parametric VaR. The columns represent, respectively, the average Estimated VaR with specification or/and estimation errors, the Theoretical VaR, and the average-minimum–maximum of the adjustment terms of all samples. A positive adjustment term indicates that the Estimated VaR (negative return) should be more conservative (more negative).

Table C.2 Estimated annualized VaR and model-risk errors (%) in the Brownian case.

Probability (%) Mean estimated VaR (%) Perfect VaR (%) Mean bias (%) Median bias (%) Min. bias (%) Max. bias (%)

Panel A. Gaussian DGP and Gaussian VaR with estimation error a = 95.00 �24.78 �24.78 .00 .00 �8.69 10.16 a = 99.00 �35.74 �35.74 .00 .00 �14.21 20.70 a = 99.50 �39.95 �39.95 .00 .00 �16.04 28.92

Panel B. Brownian DGP and Gaussian VaR with specification error a = 95.00 �29.49 �36.22 �6.73 �6.73 �6.73 �6.73 a = 99.00 �41.88 �60.75 �18.87 �18.87 �18.87 18.87 a = 99.50 �46.41 �72.87 �26.46 �26.46 �26.46 26.46

Panel C. Brownian DGP and Gaussian VaR with specification and estimation errors a = 95.00 �29.49 �36.22 � 6.73 �6.73 �13.97 1.20 a = 99.00 �41.88 �60.75 �18.87 �18.87 �28.04 �8.95 a = 99.50 �46.41 �72.87 �26.46 �26.46 �36.62 �14.01

Three price processes of the asset returns are considered below, such as for t ¼ ½1; . . . ; T� and p ¼ ½1;2;3�:

dSt ¼ St ldt þ rdWt þ Jpt dNt � �

;

with J1t ¼ 0 for Brownian, where St is the price of the asset at time t; Wt is a standard Brownian motion, independent from the Poisson process Nt , governing the jumps of various intensities Jpt (null, constant or time-varying according to the process p). Source: simulations by the authors. Errors are defined as the differences between the ‘‘true’’ asymptotic simulated VaR and the Estimated VaR. These statistics were computed with a series of 250,000 simulated daily returns with specific DGP (Brownian), averaging the parameters estimated in Aït-Sahalia et al., 2013Aït-Sahalia et al. (2013, Tabel 2, i.e. b = 41.66%, k3 = 1.20% and c = 22.22%), and ex post recalibrated for sharing the same first two moments (i.e. l = .12% and r = 1.02%) and the same mean jump intensity (for the last two processes – which leads after rescaling here, for instance, to an intensity of the Lévy such as: k2=1.06%). Per convention, a negative adjustment term in the table indicates that the Estimated VaR (negative return) should be more conservative (more negative).

86 C.M. Boucher et al. / Journal of Banking & Finance 44 (2014) 72–92

Appendix B. Main backtest procedures

We present hereafter three tests proposed in the literature to gauge the accuracy of VaR estimates.

The first test for a good VaR is the so-called ‘‘traffic light’’ approach in the regulatory framework, related to the Kupiec (1995) Proportion of Failure Test. The Unconditional Coverage test (Kupiec, 1995) attempts to determine whether the observed fre- quency of exceptions is consistent with the expected frequency of exceptions according to a chosen VaR model and a confidence interval (an exception occurs when the ex post return is below the ex ante VaR).16 We define IEVaRt ðaÞ as the ‘‘hit variable’’ associated

16 Note that the Basel ‘‘traffic light’’ backtesting framework is directly inspired by this unconditional coverage test. Escanciano and Pei (2012) show, however, that this unconditional test is always inconsistent in detecting non-optimal VaR forecasts based on the historical method. In the following, nevertheless, we consider for our adjustment procedure three of the main tests (including the unconditional coverage test), as well as their bootstrapped corrected versions.

to the ex post observation of EVaRð�Þ exceptions at the threshold a at date t, so that (with previous notations):

IEVaRð�Þt ðaÞ ¼ 1 if rt < �EVaRðĥ;aÞt�1 0 otherwise;

( ðB:1Þ

where rt is the return on portfolio P at time t, with t ¼ ½1;2; . . . ; T�. If we assume that the IEVaRt ð�Þ variables are independently and

identically distributed, then, under the Unconditional Coverage hypothesis of Kupiec (1995), the cumulated number of VaR viola- tions follows a Binomial distribution, denoted BðT;aÞ, as (see Christoffersen, 1998):

HitEVaRð�Þt ðaÞ ¼ XT t¼1

IEVaRð�Þt ðaÞ � BðT;aÞ: ðB:2Þ

A perfect sequence of (corrected) empirical VaR in the sense of this test (not too aggressive, but not too confident), is such that it respects condition (B.2).

Table C.3 Estimated annualized VaR and model-risk errors (%) in the Lévy case.

Probability (%) Mean estimated VaR (%) Perfect VaR (%) Mean bias (%) Median bias (%) Min. bias (%) Max. bias (%)

Panel A. Gaussian DGP and Gaussian VaR with estimation error a = 95.00 �24.78 �24.78 .00 .00 �8.69 10.16 a = 99.00 �35.74 �35.74 .00 .00 �14.21 20.70 a = 99.50 �39.95 �39.95 .00 .00 �16.04 28.92

Panel B. Lévy DGP and Gaussian VaR with specification error a = 95.00 �29.49 �36.22 �6.73 �6.73 �6.73 �6.73 a = 99.00 �41.88 �60.75 �18.87 �18.87 �18.87 �18.87 a = 99.50 �46.41 �72.87 �26.46 �26.46 �26.46 �26.46 Panel C. Lévy DGP and Gaussian VaR with specification and estimation errors a = 95.00 �29.49 �36.22 �6.73 �6.73 �13.97 1.20 a = 99.00 �41.88 �60.75 �18.87 �18.87 �28.04 �8.95 a = 99.50 �46.41 �72.87 �26.46 �26.46 �36.62 �14.01

Three price processes of the asset returns are considered below, such as for t ¼ ½1; . . . ; T� and p ¼ ½1;2;3�:

dSt ¼ St ldt þ rdWt þ Jpt dNt � �

;

with J2t ¼ k2expð�k2tÞ for Lévy, where St is the price of the asset at time t; Wt is a standard Brownian motion, independent from the Poisson process Nt , governing the jumps of various intensities Jpt (null, constant or time-varying according to the process p), defined by parameters, k2, which is a positive constant. Source: simulations by the authors. Errors are defined as the differences between the ‘‘true’’ asymptotic simulated VaR and the Estimated VaR. These statistics were computed with a series of 250,000 simulated daily returns with specific DGP (Lévy), averaging the parameters estimated in Aït-Sahalia et al., 2013Aït-Sahalia et al. (2013, Table 2, i.e. b = 41.66%, k3 = 1.20% and c = 22.22%), and ex post recalibrated for sharing the same first two moments (i.e. l ¼ :12% and r = 1.02%) and the same mean jump intensity (for the last two processes – which leads after rescaling here, for instance, to an intensity of the Lévy with k2 = 1.06%). Per convention, a negative adjustment term in the table indicates that the Estimated VaR (negative return) should be more conservative (more negative).

Table C.4 Estimated annualized VaR and model-risk errors (%) in the Hawkes case.

Probability (%) Mean estimated VaR (%) Perfect VaR (%) Mean bias (%) Median bias (%) Min. bias (%) Max. bias (%)

Panel A. Gaussian DGP and Gaussian VaR with estimation error a = 95.00 �24.78 �24.78 .00 .00 �8.69 10.16 a = 99.00 �35.74 �35.74 .00 .00 �14.21 20.70 a = 99.50 �39.95 �39.95 .00 .00 �16.04 28.92

Panel B. Hawkes DGP and Gaussian VaR with specification error a = 95.00 �29.49 �36.22 �6.73 �6.73 �6.73 �6.73 a = 99.00 �41.88 �60.75 �18.87 �18.87 �18.87 �18.87 a = 99.50 �46.41 �72.87 �26.46 �26.46 �26.46 �26.46

Panel C. Hawkes DGP and Gaussian VaR with specification and estimation errors a = 95.00 �29.49 �36.22 �6.73 �6.73 �13.97 1.20 a = 99.00 �41.88 �60.75 �18.87 �18.87 �28.04 �8.95 a = 99.50 �46.41 �72.87 �26.46 �26.46 �36.62 �14.01

Three price processes of the asset returns are considered below, such as for t ¼ ½1; . . . ; T� and p ¼ ½1;2;3�:

dSt ¼ St ldt þ rdWt þ Jpt dNt � �

;

with J3t ¼ k3 þ bexp½�cðt � sÞ� for Hawkes, where St is the price of the asset at time t; Wt is a standard Brownian motion, independent from the Poisson process Nt , governing the jumps of various intensities Jpt (null, constant or time-varying according to the process p), defined by parameters, k3; b and c, which are positive constants with s the date of the last observed jump. Source: simulations by the authors. Errors are defined as the differences between the ‘‘true’’ asymptotic simulated VaR and the Estimated VaR. These statistics were computed with a series of 250,000 simulated daily returns with specific DGP (Hawkes), averaging the parameters estimated in Aït-Sahalia et al., 2013Aït-Sahalia et al. (2013, Table 2, i.e. b = 41.66%, k3 = 1.20% and c = 22.22%), and ex post recalibrated for sharing the same first two moments (i.e. l = .12% and r = 1.02%) and the same mean jump intensity. Per convention, a negative adjustment term in the table indicates that the Estimated VaR (negative return) should be more conservative (more negative).

C.M. Boucher et al. / Journal of Banking & Finance 44 (2014) 72–92 87

The second test for a good VaR concerns the independence of forecasting errors. The independence hypothesis is associated to the idea that if the VaR model is correct then violations associated to VaR forecasting should be independently distributed, it is also the independence of exceptions hypothesis. If the exceptions exhi- bit some type of ‘‘clustering’’, then the VaR model may fail to cap- ture the profit and loss variability under certain conditions, which could represent a potential problem down the road. Christoffersen (1998) supposes that, under the alternative hypothesis of VaR inef- ficiency, the process of IEVaRt ðaÞ violations is modelled with a Mar- kov chain whose matrix of transition probabilities is defined by:

P ¼ p00 p01 p10 p11

� ; ðB:3Þ

where pij ¼ Pr IEVaRt ðaÞ ¼ jjI EVaR t�1 ¼ i

h i . This Markov chain reflects the

existence of an order 1 memory in the process IEVaRt ðaÞ. The proba- bility of having a violation (not having one) for the current period depends on the occurrence or not of a violation (for the same level of coverage rate) in the previous period. Christoffersen (1998) shows that the likelihood ratio for the test is:

LRindI EVaR t ðaÞ ¼ 2 log LI

EVaR t ðaÞðp01;p11Þ � log LI

EVaR t ðaÞðp;pÞ

h i !d

> v2ð1Þ; ðB:4Þ

where LI EVaR t ðaÞðp01;p11Þ is thus the likelihood under the hypothesis of

the first-order Markov dependence, and LI EVaR t ðaÞðp;pÞ is the likeli-

hood under the hypothesis of independence p01 ¼ p11 ¼ p as:

Table C.5 Dates of the maximum adjustment for different 95% VaRs and backtest models at 5% confidence level.

VaR methods Dates q1 (%) Dates q�1 (%) Dates q2 (%) Dates q � 2 (%) Dates q3 (%) Dates q

� 3 (%)

Historical 1 02/09/2009 42.02 01/16/2009 32.80 09/08/1930 78.26 04/06/2009 52.18 08/24/2009 49.91 09/08/1930 95.06 2 11/28/2008 40.60 01/06/1930 31.94 04/06/2009 46.46 08/24/2009 42.02 09/08/1930 48.91 04/21/1930 78.26 3 10/16/1930 39.97 01/05/1933 21.73 07/25/1988 42.52 04/21/1930 40.15 04/06/2009 47.68 12/02/1929 72.61 4 12/11/1929 37.75 01/06/1931 19.87 03/07/1988 40.82 12/02/1929 37.79 11/17/2008 46.46 04/06/2009 65.07

Normal 1 11/28/2008 42.86 01/16/2009 30.07 09/08/1930 74.43 08/24/2009 44.46 04/06/2009 44.46 09/08/1930 88.47 2 12/11/1929 38.57 01/06/1930 25.55 07/25/1988 39.56 11/17/2008 42.86 11/17/2008 42.86 12/02/1929 74.43 3 09/14/2009 38.07 01/05/1933 17.88 12/02/1929 38.57 09/08/1930 42.45 12/02/1929 38.86 11/17/2008 62.83 4 11/11/1929 36.69 01/06/1938 14.92 03/07/1988 34.42 12/02/1929 38.86 04/21/1930 38.57 04/21/1930 59.24

Student 1 11/28/2008 40.16 01/16/2009 30.02 09/08/1930 68.50 04/06/2009 45.93 09/08/1930 50.18 09/08/1930 101.52 2 12/11/1929 33.06 01/06/1930 18.40 12/02/1929 68.18 08/24/2009 40.27 04/06/2009 44.50 11/17/2008 86.13 3 09/14/2009 32.85 01/05/1933 14.25 07/25/1988 35.52 12/02/1929 35.86 12/02/1929 33.57 03/07/1988 71.79 4 11/11/1929 31.43 01/13/1975 11.14 03/07/1988 30.26 09/08/1930 34.80 04/21/1930 33.06 12/02/1929 68.50

Cornish 1 05/13/1915 133.65 01/04/1916 120.56 02/14/1916 135.43 09/27/1915 142.87 09/27/1915 135.43 09/27/1915 142.87 Fisher 2 05/07/1915 133.36 01/04/1915 106.47 09/27/1915 133.86 05/10/1915 133.86 05/10/1915 133.86 05/10/1915 130.22

3 05/06/1915 131.48 01/03/1917 82.83 05/10/1915 133.65 02/14/1916 114.82 02/14/1916 117.67 04/09/1917 111.55 4 05/04/1915 130.22 01/14/1988 76.04 09/08/1930 93.47 07/03/1916 90.03 03/07/1988 90.73 09/08/1930 93.47

Risk 1 03/28/1938 15.85 01/04/1921 10.50 05/10/1915 32.76 12/02/1929 16.43 12/02/1929 16.43 03/21/1932 46.62 Metrics 2 10/28/1929 15.02 01/06/1938 9.14 12/02/1929 31.91 05/03/1920 16.33 05/09/1938 16.04 12/20/1937 32.36

3 03/15/1938 14.80 01/02/1908 7.88 04/21/1930 30.51 09/26/1938 15.85 12/20/1937 14.57 03/07/1988 31.92 4 01/25/1938 14.57 01/06/1930 5.64 07/25/1988 26.69 09/08/1930 15.02 05/03/1920 14.35 12/02/1929 31.91

GARCH 1 03/24/1938 18.24 01/06/1930 15.50 09/08/1930 41.44 05/09/1938 18.48 12/02/1929 19.42 05/09/1938 39.61 2 04/06/1938 18.15 01/06/1938 9.28 12/02/1929 34.12 12/20/1937 18.24 05/03/1920 18.55 11/02/1931 35.40 3 10/28/1929 17.17 01/02/1908 8.09 07/25/1988 32.49 09/26/1938 18.15 05/09/1938 18.48 12/20/1937 34.98 4 03/15/1938 16.78 01/17/2008 7.01 03/07/1988 30.73 12/02/1929 17.17 09/08/1930 17.17 03/07/1988 33.59

CAViaR 1 01/21/1994 30.10 01/17/2008 24.96 09/24/2007 41.75 02/11/2008 35.73 09/24/2007 33.48 03/21/1932 41.32 2 06/06/2007 29.81 01/14/1994 20.53 09/08/1930 41.44 04/25/1994 32.95 04/25/1994 31.72 11/30/1998 38.82 3 02/26/2008 28.11 01/17/2006 15.47 12/02/1929 34.12 09/24/2007 32.64 04/19/1999 23.31 10/19/1987 33.59 4 03/11/2008 26.93 01/15/1999 13.87 07/25/1988 32.49 09/12/1994 31.72 07/31/2006 19.50 04/19/1999 31.35

GEV 1 11/28/2008 39.05 01/06/1930 32.12 09/08/1930 72.11 08/24/2009 48.25 04/06/2009 45.07 04/21/1930 112.42 2 12/11/1929 36.82 01/16/2009 29.52 12/02/1929 61.01 04/06/2009 41.84 08/24/2009 41.84 09/08/1930 75.87 3 11/11/1929 35.24 01/05/1933 14.55 03/07/1988 34.35 04/21/1930 37.15 04/21/1930 39.52 12/02/1929 72.11 4 09/14/2009 33.32 01/08/1947 13.62 11/17/2008 29.52 12/02/1929 35.24 11/17/2008 39.05 11/17/2008 56.53

GPD 1 11/28/2008 37.89 01/16/2009 26.98 09/08/1930 71.38 11/17/2008 42.33 04/06/2009 43.75 04/21/1930 105.01 2 09/14/2009 34.58 01/06/1930 25.64 04/06/2009 42.33 04/21/1930 35.69 08/24/2009 42.33 09/08/1930 71.38 3 12/11/1929 32.86 01/06/1931 10.30 12/02/1929 32.86 12/02/1929 32.86 04/21/1930 32.86 12/02/1929 67.67 4 07/18/1930 32.80 01/05/1933 7.67 07/25/1988 31.95 05/09/1938 29.66 09/08/1930 32.80 11/17/2008 52.51

Source: Bloomberg; daily data of the DJIA index in USD from the 1st January, 1900 to the 20th September, 2011. We use a moving window of four years (1040 daily returns) to re-estimate parameters dynamically for the various methods. The variable q1 refers to the hit test; q2 to the independence test; q3 to the magnitude test; and q�1; q

� 2; q

� 3

correspond to their resampling versions, following Escanciano and Olmo (2009, 2010, 2011).

Table C.6 Minimum k ratio model risk for 95% annualized value-at-risk models for various validity tests with 5%.

VaR methods Mean VaR (%) q1 q�1 q2 q � 2 q3 q

� 3

Historical �25.78 2.37 2.08 3.29 2.53 2.59 3.79 Normal �27.09 2.26 1.84 2.84 2.31 2.32 3.18 Student �30.52 2.02 1.73 2.52 2.04 2.04 3.31 Cornish–Fisher �20.25 3.45 3.73 4.88 4.76 3.72 4.88 RiskMetrics �25.67 1.71 20.77 57.18 41.47 36.84 101.69 GARCH �25.99 1.59 1.75 2.65 1.85 1.94 2.35 CAViaR �26.84 10.95 9.82 23.89 8.9 8.55 7.6 GEV �29.71 2.01 1.72 2.37 2.13 2.06 3.24 GPD �33.97 2.04 1.69 2.64 2.21 2.01 3.63

Source: Bloomberg; daily data of the DJIA index in USD from the 1 st January, 1900 to the 20th September, 2011. We use a moving window of four years (1040 daily returns) to dynamically re-estimate parameters for the various methods. The variable q1 refers to the hit test; q2 to the independence test; q3 to the magnitude test; and q�1; q

� 2; q

� 3

correspond to their resampling versions, following Escanciano and Olmo (2009, 2010, 2011).

88 C.M. Boucher et al. / Journal of Banking & Finance 44 (2014) 72–92

LI EVaR t ðaÞðp01;p11Þ ¼ ð1� p01ÞT00pT1001 ð1� p11Þ

T10pT1111 ;

and

LI EVaR t ðaÞðp;pÞ ¼ ð1� pÞT00þT10pT01þT11 ;

with Tij the number of observations in the state j for the current per- iod and at state i for the previous period, p01 ¼ T01=ðT00 þ T01Þ; p11 ¼ T11=ðT10 þ T11Þ and p ¼ ðT01 þ T11Þ=T .

A perfect sequence of corrected (empirical) VaR in the sense of this test (i.e. not too reactive, but not too smooth) is such that it respects condition (B.4).

A third category of tests considers the magnitude or size of vio- lation. This class of tests is based on the intuition that VaR excep- tions are treated as continuous random variables. For this test, Berkowitz (2001) transforms the empirical series into a standard normal ztþ1 series. He defines the observed quantile qtþ1 with the distribution forecast ftþ1 for the observed portfolio return rt as:

Fig. C.1. Risk map for maximum annualized adjustment values at 5% confidence levels for tests for 95% and 99% value-at-risk models (see Colletaz et al., 2013). Source: Bloomberg; daily data of the DJIA index in USD from the 1st January, 1900 to the 20th September, 2011; computations by the authors. We use a moving window of four years (1040 daily returns) to dynamically re-estimate parameters for the various methods. The variable q1 refers to the hit test; q2 to the independence test; q3 to the magnitude test; and q�1; q

� 2; q

� 3 correspond to their resampling versions, following Escanciano and Olmo (2009, 2010, 2011).

1923 1938 1953 1968 1983 1998 2013

−2.0% 0.0% 2.0%

Historical

−2.0% 0.0% 2.0%

Normal −2.0% 0.0% 2.0%

Student

−2.0% 0.0% 2.0%

Cornish − Fisher −2.0% 0.0% 2.0%

RiskMetrics

−2.0% 0.0% 2.0%

GARCH −2.0% 0.0% 2.0%

CAViaR

1935 1961 1987 2013

−2.0% 0.0% 2.0%

GEV

1935 1961 1987 2013

−2.0% 0.0% 2.0%

GPD

Fig. C.2. Dynamic optimal adjustment on the daily 95% VaR related to the hit test for 10-year sample estimation data. Source: Bloomberg; daily data of the DJIA index in USD from the 1st January, 1900 to the 7th October, 2013; computations by the authors. We use a moving window of ten years (2600 daily returns) to re-estimate parameters dynamically for the various methods.

C.M. Boucher et al. / Journal of Banking & Finance 44 (2014) 72–92 89

qtþ1 ¼ Z rtþ1 �1

ftþ1ðrÞdr: ðB:5Þ

The ztþ1 values are then compared to the normal random vari- ables with the desired coverage level of the VaR estimates:

ztþ1 ¼ U�1ðqtþ1Þ; ðB:6Þ

where U�1ð�Þ is the quantile function of the standard normal density.

If the VaR model generating the empirical quantiles is correct, then the ctþ1 series should be identically distributed with the

unconditional mean and standard deviation, denoted ðl;rÞ and should equal ð0;1Þ, as such:

ctþ1 ¼ ztþ1 if ztþ1 < U�1ðaÞ 0 otherwise;

( ðB:7Þ

where Uð�Þ is the standard normal cumulative distribution function. Finally, the corresponding test statistic is:

LRmagctþ1 ¼ 2 Lctþ1magðl;rÞ � L ctþ1 magð0;1Þ

h i !d > v2ð2Þ; ðB:8Þ

where

1923 1938 1953 1968 1983 1998 2013 0.0%

2.5% Historical

0.0%

2.5% Normal

0.0%

2.5% Student

0.0%

2.5% Cornish − Fisher

0.0%

2.5% RiskMetrics

0.0%

2.5% GARCH

0.0%

2.5% CAViaR

1935 1961 1987 2013 0.0%

2.5% GEV

1935 1961 1987 2013 0.0%

2.5% GPD

Fig. C.3. Optimal dynamic absolute value of minimum negative adjustments for the hit test for different 95% VaR – 10-year sample estimation data. Source: Bloomberg; daily data of the DJIA index in USD from the 1st January, 1900 to the 7th October, 2013; computations by the authors. We use a moving window of ten years (2600 daily returns) to re-estimate parameters dynamically for the various methods.

0 10 20 30 40 50 50%

100%

Historical

0 10 20 30 40 50 50%

100%

Normal 0 10 20 30 40 50

50%

100%

Student

0 10 20 30 40 50 50%

100%

Cornish Fisher 0 10 20 30 40 50

50%

100%

RiskMetrics

0 10 20 30 40 50 50%

100%

GARCH 0 10 20 30 40 50

50%

100%

CAViaR

0 10 20 30 40 50 50%

100%

GEV 0 10 20 30 40 50

50%

100%

GPD

Fig. C.4. Optimal dynamic relative adjustment for the hit test for different starting dates and 95% VaR by horizon (in years) – 10 years sample estimation data. Source: Bloomberg; daily data of the DJIA index in USD from the 1st January, 1900 to the 7th October, 2013; computations by the authors. We use a moving window of ten years (2600 daily returns) to dynamically re-estimate parameters for the various methods. This figure illustrates the dynamic negative adjustment required for passing the hit test, having randomly chosen the first date of implementation. Optimal relative negative adjustments are here expressed in terms of the percentage of their maximum value over the whole sample.

90 C.M. Boucher et al. / Journal of Banking & Finance 44 (2014) 72–92

Lctþ1magðl;rÞ¼ X

fctþ1¼0g log 1�U U

�1ðaÞ�l r

( )( )

þ X

fctþ1–0g �1

2 logð2pr2Þ�ðctþ1�lÞ

2

2r2 � log U U

�1ðaÞ�l r

( )( )( ) :

A perfect sequence of (corrected) empirical VaR in the sense of this test ( i.e. not too conservative, but not too over-confident) is such that it respects condition (B.8).

For unconditional and conditional coverage tests, Escanciano and Olmo (2009, 2010, 2011) approximate the critical values. Thus, they propose to use robust sub-sampling techniques to approxi- mate the true distribution of these tests. However, they also show that although the estimation risk can be diversified by choosing a large in-sample size relative to an out-of-sample one, the risk asso- ciated to the model cannot be eliminated using sub-sampling.

Indeed, let GxðxÞ denote the cumulative distribution function of the test statistic k for any x 2 IR, and, kb;t ¼ Kðt; t þ 1; � � � ; t þ b� 1Þ, with t ¼ ½1;2; � � � ; T � bþ 1�, the test statistic computed with the subsample ½1;2; � � � ; T � bþ 1� of size b.

Hence, the approximated sampling cumulative distribution function of k, denoted Gkb ðxÞ, built using the distribution of the val- ues of kb;t computed over the ðT � bþ 1Þ different consecutive subsamples of size b is given by:

Gkb ðxÞ ¼ ðT � bþ 1Þ �1 XT�bþ1

t¼1 1Ifkb;t<xg: ðB:9Þ

The ð1� sÞth sample quantile of Gkb , is given by: (see Table. B.1)

ckb ;1�s ¼ inf|{z} x2IR

fxjGkb ðxÞP 1� sg: ðB:10Þ

0 10 20 30 40 50 50%

100%

Historical

0 10 20 30 40 50 50%

100%

Normal 0 10 20 30 40 50

50%

100%

Student

0 10 20 30 40 50 50%

100%

Cornish Fisher 0 10 20 30 40 50

50%

100%

RiskMetrics

0 10 20 30 40 50 50%

100%

GARCH 0 10 20 30 40 50

50%

100%

CAViaR

0 10 20 30 40 50 50%

100%

GEV 0 10 20 30 40 50

50%

100%

GPD

10 years 4 years

Fig. C.5. Optimal dynamic relative adjustment for the hit test for different starting dates and 95% VaR by horizon (in years) – 4 years and 10 years VaR estimation. Source: Bloomberg; daily data of the DJIA index in USD from the 1st January, 1900 to the 7th October, 2013; computations by the authors. We use a moving window of four and ten years to dynamically re-estimate parameters for the various methods. This figure illustrates the dynamic negative adjustment required for passing the hit test, having randomly chosen the first date of implementation. Optimal relative negative adjustments are here expressed in terms of the percentage of their maximum value over the whole sample.

0 10 20 30 40 50

0.7 0.8 0.9

Historical

0 10 20 30 40 50 0.5

1

1.5

Normal

0 10 20 30 40 50 0.5

1

1.5

Student

0 10 20 30 40 50 0.5

1

1.5

Cornish Fisher 0 10 20 30 40 50

0

2

4 RiskMetrics

0 10 20 30 40 50 1

2

3

GARCH

0 10 20 30 40 50 1

2

3

CAViaR

0 10 20 30 40 50 1

1.2

1.4

GEV

0 10 20 30 40 50 0

1

2

GPD

Fig. C.6. Ratio ten/four year optimal dynamic relative adjustment for the hit test for different starting dates and 95% VaR by Horizon (in years). Source: Bloomberg; daily data of the DJIA index in USD from the 1st January, 1900 to the 7th October, 2013; computations by the authors. We use a moving window of four and ten years to dynamically re- estimate parameters for the various methods. This figure illustrates the dynamic negative adjustment required for passing the hit test, having randomly chosen the first date of implementation. Optimal relative negative adjustments are here expressed in terms of the percentage of their maximum value over the whole sample.

C.M. Boucher et al. / Journal of Banking & Finance 44 (2014) 72–92 91

Appendix C. Miscellaneous complementary results

See Tables C.1, C.2, C.3, C.4, C.5 and C.6. See Figs. C.1, C.2, C.3, C.4, C.5 and C.6.

References

Abdous, B., Remillard, B., 1995. Relating quantiles and expectiles under weighted- symmetry. Annals of the Institute of Statistical Mathematics 47 (2), 371–384.

Aït-Sahalia, Y., Cacho-Diaz, J., Laeven, R., 2013. Modeling financial contagion using mutually exciting jump processes. Princeton Working Paper, 44p.

Angelidis, T., Degiannakis, S., 2007. Backtesting VaR models: a two-stage procedure. Journal of Risk Model Validation 1 (2), 27–48.

Alexander, C., Sarabia, J.M., 2012. Quantile uncertainty and value-at-risk model risk. Risk Analysis 32 (8), 1293–1308.

Bao, Y., Ullah, A., 2004. Bias of value-at-risk. Finance Research Letters 1 (4), 241– 249.

Basel Committee on Banking Supervision, 2010. Basel III: Towards a Safer Financial System. Bank for International Settlements, 8p.

Basle Committee on Banking Supervision, 2013. Fundamental Review of the Trading Book: A Revised Market Risk Framework. Bank for International Settlements, 127p.

Bauwens, L., Preminger, A., Rombouts, J., 2010. Theory and inference for a Markov switching GARCH model. Journal of Econometrics 13 (2), 218–244.

Beder, T., 1995. VaR: seductive but dangerous. Financial Analysts Journal 51 (5), 12– 24.

Benoit, S., Colletaz, G. Hurlin, Ch., Perignon, Ch., 2013. A Theoretical and Empirical Comparison of Systemic Risk Measures. SSRN Working Paper, 42p.

Berkowitz, J., 2001. Testing density forecasts with applications to risk management. Journal of Business and Economics Statistics 19 (4), 465–474.

Berkowitz, J., O’Brien, J., 2002. How accurate are value-at-risk models at commercial banks? Journal of Finance 57 (3), 1093–1111.

92 C.M. Boucher et al. / Journal of Banking & Finance 44 (2014) 72–92

Berkowitz, J., Christoffersen, P., Pelletier, D., 2011. Evaluating value-at-risk models with desk-level data. Management Science 57 (12), 2213–2227.

M., Billio, Casarin, R., Osuntuyi, A., 2012. Efficient Gibbs Sampling for Markov Switching GARCH Models, Working Paper 35, University of Ca’Foscari, 40p.

Bowman, A., Azzalini, A., 1997. Applied Smoothing Techniques for Data Analysis. Oxford University Press, 204p.

Breuer, T., Csiszár, I., 2013. Systematic stress tests with entropic plausibility constraints. Journal of Banking and Finance 37 (5), 1552–1559.

Breuer, T., Csiszár, I., 2014. Measuring distribution model risk. Mathematical Finance 19 (forthcoming).

Breuer, T., Jandacka, M., Mencía, J., Summer, M., 2012. A systematic approach to multi-period stress testing of portfolio credit risk. Journal of Banking and Finance 36 (2), 332–340.

Campbell, S., 2007. A review of backtesting and backtesting procedures. Journal of Risk 9 (2), 1–19.

Candelon, B., Colletaz, G., Hurlin, C., Tokpavi, S., 2011. Backtesting value-at-risk: a GMM duration-based test. Journal of Financial Econometrics 9 (2), 314–343.

Chan, N., Deng, S., Peng, L., Xia, Z., 2007. Interval estimation of value-at-risk based on GARCH models with heavy-tailed innovations. Journal of Econometrics 137 (2), 556–576.

Chan, N., Getmansky, M., Haas, S., Lo, A., 2006. Systemic risk and hedge funds. In: Carey, Stulz (Eds.), The Risks of Financial Institutions. University of Chicago Press, pp. 235–338.

Cheridito, P., Stadje, M., 2009. Time-inconsistency of VaR and time-consistent alternatives. Finance Research Letters 6 (1), 40–46.

Christoffersen, P., 1998. Evaluating interval forecasts. International Economic Review 39 (4), 841–862.

Christoffersen, P., Gonçalves, S., 2005. Estimation risk in financial risk management. Journal of Risk 7 (3), 1–28.

Christoffersen, P., Pelletier, D., 2004. Backtesting value-at-risk: a duration-based approach. Journal of Financial Econometrics 2, 84–108.

Colletaz, G., Hurlin, Ch., Pérignon, Ch., 2013. The risk map: a new tool for validating risk models. Journal of Banking and Finance 37 (10), 3843–3854.

Chordia, T., Roll, R., Subrahmanyam, A., 2001. Market liquidity and trading activity. Journal of Finance 56 (2), 501–530.

Cornish, E., Fisher, R., 1937. Moments and cumulants in the specification of distributions. Review of the International Statistical Institute 5 (4), 307–320.

Crouhy, M., Galai, D., Mark, R., 1998. Model risk. Journal of Financial Engineering 7, 267–288.

Daníelsson, J., 2002. The emperor has no clothes: limits to risk modelling. Journal of Banking and Finance 26 (4), 1273–1296.

Daníelsson, J., James, K., Valenzuela, M., Zer, I., 2014. Model Risk of Risk Models, SRC Working Paper, London School of Economics, 32p.

Derman, E., 1996. Model risk. Risk 9 (5), 34–37. Dumitrescu, E., Hurlin, Ch., Pham, V., 2012. Backtesting value-at-risk: from dynamic

quantile to dynamic binary tests. Finance 33 (1), 79–112. Engle, R., Manganelli, S., 2001. Value-at-Risk Models in Finance, ECB Working Paper

75, 40p. Engle, R., Manganelli, S., 2004. CAViaR: conditional autoregressive value-at-risk by

regression quantile. Journal of Business and Economic Statistics 22 (4), 367– 381.

Escanciano, J., Olmo, J., 2009. Specification tests in parametric value-at-risk models. In: Gouriéroux, Jeanblanc (Eds.), Financial Risks: New Developments in Structured Product & Credit Derivatives. Proceedings of the 1st International Financial Research Paris, FR, Economica, pp. 49–62.

Escanciano, J., Olmo, J., 2010. Backtesting parametric value-at-risk with estimation risk. Journal of Business and Economic Statistics 28 (1), 36–51.

Escanciano, J., Olmo, J., 2011. Robust backtesting test for value-at-risk. Journal of Financial Econometrics 9 (1), 132–161.

Escanciano, J., Pei, P., 2012. Pitfalls in backtesting historical simulation models. Journal of Banking and Finance 36 (8), 2233–2244.

Favre, L., Galeano, J., 2002. Mean modified value-at-risk optimization with hedge funds. The Journal of Alternative Investments 5 (2), 21–25.

Figlewski, S., 2004. Estimation Error in the Assessment of Financial Risk Exposure, Working Paper, New-York University, 48p.

Financial Services Authority, 2006. Solvency II: A New Framework for Prudential Regulation of Insurance in the EU, FSA Discussion Paper, pp. 1–64.

Frésard, L., Pérignon, Ch., Wilhelmsson, A., 2011. The pernicious effects of contaminated data in risk management. Journal of Banking and Finance 35 (10), 2569–2583.

Gagliardini, P., Gouriéroux, Ch., 2013. Granularity adjustment for VaR risk measures: systematic vs unsystematic risks. International Journal of Approximate Reasoning 54 (6), 717–747.

Gagliardini, P., Gouriéroux, Ch., Monfort, A., 2012. Micro-information, non-linear filtering and granularity. Journal of Financial Econometrics 10 (1), 1–53.

Gaglianone, W., Lima, L., Linton, O., Smith, D., 2011. Evaluating value-at- risk models via quantile regression. Journal of Business & Economic Statistics 29 (1), 150– 160.

Getmansky, M., Lo, A., Makarov, I., 2004. An econometric analysis of serial correlation and illiquidity in hedge-fund returns. Journal of Financial Economics 74 (3), 529–609.

Gibson, R., (Ed.), (2000). Model Risk: Concepts, Calibration and Pricing, Risk Books.

Gibson, R., L’Habitant, F.-S., Pistre, N., Talay, D., 1999. Interest rate model risk: an overview. Journal of Risk 1 (3), 37–62.

Giot, P., Grammig, J., 2005. How large is liquidity risk in an automated auction market? Empirical Economics 30 (4), 867–888.

Gordy, M., 2003. A risk-factor model foundation for rating-based bank capital rules. Journal of Financial Intermediation 12 (1), 199–232.

Gouriéroux, C., Zakoïan, J.-M., 2013. Estimation-adjusted VaR. Econometric Theory 29 (4), 735–770.

Gray, S., 1996. Modelling the conditional distribution of interest rates as a regime- switching process. Journal of Financial Economics 42 (1), 27–62.

Haas, M., 2005. Improved duration-based backtesting of value-at-risk. Journal of Risk 8 (2), 17–36.

Haas, M., Mittnik, S., Paolella, M., 2004. A new approach to Markov-switching GARCH models. Journal of Financial Econometrics 2 (4), 493–530.

Hamilton, J., Susmel, R., 1994. AutoRegressive conditional heteroskedasticity and changes in regime. Journal of Econometrics 64 (1-2), 307–333.

Hartz, C., Mittnik, S., Paolella, M., 2006. Accurate value-at-risk forecasting based on the normal-GARCH model. Computational Statistics and Data Analysis 51 (4), 2295–2312.

Hoogerheide, L., Ravazzolo, F., van Dijk, H., 2011. Backtesting Value-at-Risk using Forecasts for Multiple Horizons, A Comment on the Forecast Rationality Tests of A.J. Patton and A. Timmermann, Tinbergen Institute Discussion Paper TI 2011- 131/4, 15p.

Hurlin, C., Tokpavi, S., 2006. Backtesting value-at-risk accuracy: a simple new test. Journal of Risk 9 (2), 19–37.

Inui, K., Kijima, M., 2005. On the significance of expected shortfall as a coherent risk measure. Journal of Banking and Finance 29 (4), 853–864.

Jorion, P., 2007. Value-at-Risk: The New Benchmark for Managing Financial Risk. McGraw-Hill, 600p.

Jorion, P., 2009a. Financial Risk Manager Handbook. Wiley, 717p. Jorion, P., 2009b. Risk management lessons from the credit crisis. European

Financial Management 15 (5), 923–933. Kerkhof, J., Melenberg, B., Schumacher, H., 2010. Model risk and capital reserves.

Journal of Banking and Finance 34 (1), 267–279. Klaassen, F., 2002. Improving GARCH volatility forecasts with regime-switching

GARCH. Empirical Economics 27 (2), 363–394. Kupiec, P., 1995. Techniques for verifying the accuracy of risk measurement models.

Journal of Derivatives 3 (2), 73–84. Kyle, A., 1985. Continuous auctions and insider trading. Econometrica 53 (6), 1315–

1336. Leccadito, A., Boffelli, S., Urga, G., 2014. Evaluating the accuracy of value-at-risk

forecasts: new multi-level tests. International Journal of Forecasting 30 (2), 206–216.

Löffler, G., Raupach, P., 2013. Robustness and informativeness of systemic risk measures. Deutsche Bundesbank Discution Paper 4, 40.

Lönnbark, C., 2010. Uncertainty of multiple period risk measures. Umea Economic Studies 768, 37.

Lopez, J., 1998. Methods for evaluating value-at-risk estimates. Federal Reserve Bank of San Francisco 98–02, 3–17.

Lopez, J., 1999. Regulatory evaluation of value-at-risk models. Journal of Risk 1 (2), 37–64.

McAleer, M., Jiménez-Martín, J.-A., Pérez-Amaral, T., 2013. International evidence on GFC-robust forecast for risk management under the basel accord. Journal of Forecasting 32 (3), 267–288.

McCracken, M.W., 2000. Robust out-of-sample inference. Journal of Econometrics 99 (5), 195–223.

Nieto, M., Ruiz, E., 2008. Measuring Financial Risk: Comparison of Alternative Procedures to estimate VaR and ES. Statistics and Econometrics Working Paper – University Carlos III, 45p.

Pelletier, D., Wei, W., 2014. The Geometric-VaR Backtesting Method, Working Paper, 28p.

Pérignon, Ch., Smith, D., 2008. A new approach to comparing VaR estimation methods. Journal of Derivatives 16 (2), 54–66.

Pérignon, Ch., Smith, D., 2010. The level and quality of value-at-risk disclosure by commercial banks. Journal of Banking and Finance 34 (2), 362–377.

Pritsker, M., 1997. Evaluating value-at-risk methodologies: accuracy versus computational time. Journal of Financial Services Research 12 (2), 201–242.

RiskMetrics, 1996. Technical Document, Morgan Guaranty Trust Company of New York, 296p.

Silber, W., 2005. What happened to liquidity when world war I shut the NYSE? Journal of Financial Economics 78 (3), 685–701.

Talay, D., Zheng, Z., 2002. Worst case model risk management. Finance and Stochastics 6 (4), 517–537.

West, K., 1996. Asymptotic inference about predictive ability. Econometrica 64, 1067–1084.

Wilde, T., 2001. Probing granularity. Risk 18 (4), 103–106. Wong, W., 2008. Backtesting trading risk of commercial banks using expected

shortfall. Journal of Banking and Finance 32 (7), 1404–1415. Wong, W., 2010. Backtesting value-at-risk based on tail losses. Journal of Empirical

Finance 17 (3), 526–538. Ziggel, D., Berens, T., Weiss, G., Wied, D., 2013. A New Set of Improved Value-at-Risk

Backtests, Working Paper, 45p.

The-good-and-bad-news-about-the-new-liquidity-rules-of_2014_Journal-of-Banki.pdf

• The good and bad news about the new liquidity rules of Basel III in Western European countries
• 1 Introduction
• 2 Related literature
• 3 Data
• 4 Methodology and model specification
• 4.1 Multivariate modelling technique
• 4.2 Model 1 specification
• 4.2.1 Bank characteristics
• 4.2.2 Owner characteristics
• 4.2.3 Market/environmental characteristics
• 4.3 Model 2 specification
• 4.3.1 Dependent variables
• 4.3.2 Independent variables
• 5 Results
• 5.1 Descriptive statistics and univariate results
• 5.2 Multivariate results
• 5.2.1 Which factors influence the NSFR ratio?
• 5.2.2 What is the influence of NSFR on Bank performance measures?
• 6 Conclusions
• Acknowledgements
• References