FINANCE , STATISTICS, FORECASTING. ACCURACY TESTING AND ERROR MATRICES

International Journal of Forecasting 34 (2018) 105–116

Contents lists available at ScienceDirect

International Journal of Forecasting

journal homepage: www.elsevier.com/locate/ijforecast

Forecast-error-based estimation of forecast uncertainty when the horizon is increased✩ Malte Knüppel Malte Knüppel, Deutsche Bundesbank, Wilhelm-Epstein-Strasse 14, D-60431 Frankfurt am Main, Germany

a r t i c l e i n f o

Keywords: Multi-step-ahead forecasts Forecast error variance SUR

a b s t r a c t

Past forecast errors are employed frequently in the estimation of the unconditional forecast uncertainty, and several institutions have increased their forecast horizons in recent times. This work addresses the question of how forecast-error-based estimation can be performed if there are very few errors available for the new forecast horizons. It extends the results of Knüppel (2014) in order to relax the condition on the data structure that is required for the SUR estimator to be independent of unknown quantities. It turns out that the SUR estimator of the forecast uncertainty, which estimates the forecast uncertainty for all horizons jointly, tends to deliver large efficiency gains relative to the OLS estimator (i.e., the sample mean of the squared forecast errors for each individual horizon) in the case of increased forecast horizons. The SUR estimator is applied to the forecast errors of the Bank of England, the US Survey of Professional Forecasters, and the FOMC. © 2017 International Institute of Forecasters. Published by Elsevier B.V. All rights reserved.

1. Introduction

The forecast horizons of many important macroeco- nomic forecasts have increased in recent times. For exam- ple, since 2009, the members of the Federal Open Market Committee (FOMC) have been providing forecasts for the ‘‘longer run’’ in their Economic Projections, which might correspond to a horizon of about five years, in addition to their forecasts for the current year and the next two years.1 In the same year, the forecast horizons for the annual fore- casts of real GDP, the unemployment rate, and 3-month and 10-year Treasuries in the US Survey of Professional Forecasters (SPF), conducted by the Federal Reserve Bank of Philadelphia, were extended from one to three years ahead. In the SPF, conducted by the European Central Bank (ECB),

✩ The views expressed in this paper are those of the author and do not necessarily reflect the position of the Deutsche Bundesbank or its staff.

E-mail address: [email protected]. 1 According to the FOMC, ‘‘[t]he longer-run projections are the rates of

growth, inflation, and unemployment to which a policymaker expects the economy to converge over time–maybe in five or six years–in the absence of further shocks and under appropriate monetary policy.’’ See http:// www.federalreserve.gov/monetarypolicy/fomc_projectionsfaqs.htm.

respondents have been asked about their 2-year-ahead forecasts for annual real GDP growth, HICP inflation and the unemployment rate in the first two quarters of the current year since 2013. Before 2013, the forecasts for this horizon were requested only in the last two quarters.2 Since 2014, the ECB staff and the Eurosystem staff have been publishing 2-year-ahead forecasts of annual real GDP growth and HICP inflation in every quarter. Previously, such forecasts were made only in the last quarter of the current year, while the longest forecast horizon in the first three quarters was one year. Finally, the Bank of England (henceforth BoE) extended its forecast horizons for real GDP growth and CPI inflation from 8 to 12 quarters in 2004.

At the same time, there appears to be an increased in- terest in forecast uncertainty in the field of economics. Re- cent contributions include the papers by Jurado, Ludvigson, and Ng (2015), who estimate time-varying macroeconomic uncertainty, and Abel, Rich, Song, and Tracy (2016), Boero,

2 In addition, the ECB SPF also contains a longer-run-type forecast for a forecast horizon of four or five years. When the survey started in 1999, this forecast was surveyed only for the first quarter, but since 2001, it has been included in every quarter.

http://dx.doi.org/10.1016/j.ijforecast.2017.08.006 0169-2070/© 2017 International Institute of Forecasters. Published by Elsevier B.V. All rights reserved.

106 M. Knüppel / International Journal of Forecasting 34 (2018) 105–116

Fig. 1. Forecast errors of the Bank of England for quarterly real GDP growth in the UK for quarterly horizons of h = 1 to h = 13, where h = 1 corresponds to the nowcast.

Smith, and Wallis (2015) and Lahiri and Sheng (2010), who investigate time-varying uncertainty in the context of survey forecasts. Other studies have focused on topics related to unconditional forecast uncertainty by employ- ing empirical forecast errors.3 For example, Clements (2014) discovers a horizon-dependent mismatch between the unconditional ex-ante uncertainty of survey forecast- ers and the unconditional variability of their empirical forecast errors. Rossi and Sekhposyan (2015) construct an uncertainty index that is based on the quantiles of the empirical forecast errors in the unconditional distribution of those errors. Patton and Timmermann (2011) set up forecasting models whose implied unconditional forecast uncertainties match the unconditional volatilities of the empirical forecast errors of survey forecasts.

Empirical forecast errors also play an important role in producing measures of the unconditional forecast uncertainty for institutions which publish forecasts. Model-based approaches for the estimation of uncondi- tional forecast uncertainty, as described by Ericsson (2002) for instance, are used only rarely because, as Wallis (1989, pp. 55-56) noted, ‘‘This approach is of little help to the practitioner. It neglects the contribution of the fore- caster’s subjective adjustments [. . .]. More fundamentally, the model’s specification is uncertain. At any point in time competing models coexist, over time model specifications evolve, and there is no way of assessing this uncertainty. Thus, the only practical indication of the likely margin of future error is provided by the past forecast errors ’’ [emphasis added]. It is common for only the institution’s own forecast errors to be used for estimating its forecast uncertainty. This error-based forecast uncertainty measure is regarded as information about the unconditional uncertainty that can serve as a benchmark when statements about the current forecast uncertainty are being made. For example, the FOMC members must state whether the uncertainty

3 The terms ‘‘empirical forecast errors’’ and ‘‘past forecast errors’’ are used interchangeably in the literature.

attached to their current forecasts is larger than, smaller than, or broadly similar to that observed in the past.

The construction of uncertainty measures in the liter- ature for the forecasts of a specific model or institution based on its empirical forecast errors goes back to the work of Williams and Goodman (1971), who suggested that prediction intervals be constructed from the empirical distribution of forecast errors instead of being derived from the forecasting model. Recent contributions include those of Lee and Scholtes (2014), who study robustness issues concerning the prediction intervals proposed by Williams and Goodman (1971) and Jordà, Knüppel, and Marcellino (2013), who propose empirical prediction regions for fore- cast paths; and Knüppel (2014), who investigates ways of exploiting the information contained in the forecast errors of shorter forecast horizons for estimating the forecast un- certainty at longer horizons. Moreover, Clark, McCracken, and Mertens (2017) estimate the time-varying forecast uncertainty based on multi-step-ahead survey forecast er- rors. Studies that attempt to estimate the uncertainty of central bank forecasts include those by Reifschneider and Tulip (2007) and Tulip and Wallace (2012), for example, where the former focus on the Federal Reserve System in the US and the latter on the Reserve Bank of Australia.4 Finally, Hartmann, Herwartz, and Ulm (2017) evaluate different measures of ex-ante uncertainty according to their abilities to predict squared forecast errors at different horizons.

If the past unconditional forecast uncertainty is to be estimated, the question arises as to how this can be ac- complished in a reasonably precise manner if very few observations are available because a new forecast horizon was introduced only a short time ago. For example, the first ‘‘longer run’’ forecast error for the FOMC forecasts was observed in 2015 when the data for 2014 were released, if one assumes that the ‘‘longer run’’ corresponds to a forecast horizon of about five years. The first forecast errors of the US SPF 3-year-ahead forecasts became available in 2012. Even the sample of forecast errors for the new forecast horizons for the BoE, whose growth forecast errors are displayed in Fig. 1, appears relatively short because the large persistence of these forecast errors carries over to the squared errors, and many observations are required in order to obtain a reliable mean estimate for a persistent series. This work focuses on estimating the forecast uncer- tainty for such horizons as soon as the first forecast errors become available.

The SUR estimator used by Knüppel (2014) relies on the correlations between forecast errors from different hori- zons for the same period, which are a typical feature of em- pirical forecast errors such as those displayed in Fig. 1. This SUR estimator of the unconditional forecast uncertainty, which estimates the forecast uncertainty for all horizons jointly, tends to deliver efficiency gains relative to the OLS estimator (i.e., the sample mean of the squared forecast errors for each single horizon). However, in addition to the assumption of optimal forecasts, this SUR estimator

4 Reifschneider and Tulip’s approach differs from those of the other studies mentioned because they estimate their uncertainty measure from the errors of a panel of forecasters.

M. Knüppel / International Journal of Forecasting 34 (2018) 105–116 107

requires a certain data structure called the recent-forecast- errors structure, which makes it independent of unknown parameters. This data structure is present if forecasts for the same horizons are made in every period until the current period. Hence, it differs from the structure that is observed when additional forecast horizons are introduced after the first forecasts were produced.

This study relaxes these restrictions on the data struc- ture. It turns out that the SUR estimator is still independent of unknown parameters for optimal forecasts in circum- stances that have not been considered previously. These circumstances include the case where new, longer forecast horizons are introduced. In this case, the SUR estimator tends to deliver even larger efficiency gains than in the case of the recent-forecast-errors structure for both optimal and non-optimal forecasts. Other new findings include the fact that the SUR estimator of the unconditional forecast uncer- tainty also appears to be preferable to the OLS estimator in the case of stationary time-varying uncertainty, and that the SUR estimator can be employed for certain types of fixed-event forecasts as well.

In an empirical application, the SUR estimator is applied to the forecasts of the BoE, the US SPF, and the FOMC. If the true unconditional forecast uncertainty is a relatively smooth, non-decreasing function of the forecast horizon, the measures of the forecast uncertainty that are obtained using the SUR estimator tend to deliver patterns of fore- cast uncertainty that appear more plausible than those obtained using horizon-specific sample means of squared forecast errors. Moreover, the results of the SUR estimator are mostly more in line with other measures of the forecast uncertainty.

2. The estimation of forecast uncertainty

In what follows, the SUR estimator for the unconditional forecast uncertainty is derived under the assumptions of optimal forecasts and a constant forecast uncertainty. The robustness of the resulting estimator with respect to de- partures from these assumptions is studied in Section 3 us- ing Monte Carlo simulations. If the unconditional forecast uncertainty is non-stationary, due to a structural break for example, the OLS estimator may be preferable to the SUR estimator, depending on the bias—variance trade-off.5

2.1. Fixed-horizon forecasts

Much of what follows is taken from Knüppel (2014). Consider a stationary data-generating process with a Wold representation given by

yt = µ + ∞∑ i=0

biεt−i,

with E [εt] = 0, E [ ε2t ]

= σ2 and b0 = 1. E [•] is the expectation operator. It is assumed that the fourth moment of εt exists, so that the kurtosis

α = E [ ε 4 t

] /σ

4

5 This issue is investigated in the context of the recent-forecast-errors structure by Knüppel (2014).

is finite. The optimal h-step-ahead forecast is

yt+h,t = µ + ∞∑ i=0

bh+iεt−i,

with h = 1,2, . . . ,H, where H denotes the largest forecast horizon. Hence, the h-step-ahead forecast error equals

et+h,t := yt − yt+h,t = h−1∑ i=0

biεt+h−i. (1)

Thus, et+h,t is the error of the forecast made in period t for period t + h, and has a moving-average representation of order h − 1.

The variance of the h-step-ahead forecast error is given by

E [ e2t+h,t

] = σ

2 h = σ

2 h−1∑ i=0

b2i .

The variances for all H∗ forecast horizons of interest are collected in the (H∗ × 1) vector σ2 given by

σ 2

= ( σ

2 1 ,σ

2 2 , . . . ,σ

2 H

)′ ,

with H∗ ≤ H. The corresponding estimates of the forecast uncertainty will be denoted by

σ̂ 2 m =

( σ̂

2 m,1, σ̂

2 m,2, . . . , σ̂

2 m,H

)′ ,

where m will refer to the estimation method used. Since the DGP is stationary, the covariance between

the squared forecast errors e2t2,t1 and e 2 t4,t3

tends to be- come small when |t4 − t2| becomes large. Assuming that the covariance actually equals zero if t4 ̸= t2 opens up the possibility of estimating the forecast uncertainty by seemingly unrelated regressions (SUR).

Under this assumption, and defining p = t2 − t1 and q = t4 − t3, the covariance of the squared forecast errors equals

ωs = E [( e2t2,t1 − E

( e2t2,t1

))( e2t4,t3 − E

( e2t4,t3

))] =

⎧⎪⎪⎪⎨ ⎪⎪⎪⎩ 0 if t2 ̸= t4

σ 4 (α − 1)

s−1∑ i=0

b4i + 2σ 4

s−1∑ i=0

s−1∑ j=0,j̸=i

b2i b 2 j

if t2 = t4,

(2)

with s = min(p,q)and using the convention ∑0

i=0 ∑0

j=0,j̸=i xij = 0. If the kurtosis equals α = 3, the last term in Eq. (2)

simplifies to 2σ4 (∑s−1

i=0 b 2 i

)2 .

If the forecasts are not optimal, or the forecast uncer- tainty is not constant over time, the errors of forecasts for a certain period tend to be strongly correlated nevertheless, due to the shock in that period, making the SUR estimation of forecast uncertainty seem a promising approach.

Concerning the data structure, suppose that the first forecast was made in period t = 0 and that the last avail- able forecast errors are observed in period t = T . Define a (T × H) index matrix J, where the element in the tth row and jth column of J, jth, is determined by the existence of the h-step-ahead forecast error for period t, i.e., by the error of

108 M. Knüppel / International Journal of Forecasting 34 (2018) 105–116

the h-step-ahead forecast that was produced in period t−h. Thus, the element jth equals

jth = { 1 if et,t−h exists 0 else.

I assume that there is at least one 1-step-ahead forecast error and one H-step-ahead forecast error. The correspond- ing (T × H) matrix of squared forecast errors is given by

E2 =

⎡ ⎢⎢⎢⎢⎢⎢⎢⎣

ẽ21,0 ẽ 2 1,−1 · · · ẽ

2 1,1−H

ẽ22,1 ẽ 2 2,0 · · · ẽ

2 2,2−H

ẽ23,2 ẽ 2 3,1 · · · ẽ

2 3,3−H

.

.

. . . .

... . . .

ẽ2T,T −1 ẽ 2 T,T −2 · · · ẽ

2 T,T −H

⎤ ⎥⎥⎥⎥⎥⎥⎥⎦

with elements

ẽ2t,t−h = { e2t,t−h if e

2 t,t−h exists

c else,

where c is an arbitrary value that will receive a weight of zero in the estimation. If only the elements ẽ2t,t−h with t − h < 0 equal c, the data structure is equivalent to the recent- forecast-errors structure considered by Knüppel (2014).

Hence, for example, if 1- and 2-step-ahead forecasts were produced since t = 0, the last observation came from t = 7, additional 4-step-ahead forecasts were produced at t = 1 and t = 3, and no forecast errors were available for t = 3, then J and E2 would equal

J =

⎡ ⎢⎢⎢⎢⎢⎢⎣

1 0 0 0 1 1 0 0 0 0 0 0 1 1 0 0 1 1 0 1 1 1 0 0 1 1 0 1

⎤ ⎥⎥⎥⎥⎥⎥⎦ ,

E2 =

⎡ ⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣

e21,0 c c c

e22,1 e 2 2,0 c c

c c c c

e24,3 e 2 4,2 c c

e25,4 e 2 5,3 c e

2 5,1

e26,5 e 2 6,4 c c

e27,6 e 2 7,5 c e

2 7,3

⎤ ⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦ . (3)

In order to estimate a model with correlated error terms using SUR, one must construct a covariance matrix and a regressor matrix of the error terms collected in E2. If J consists only of ones, the(TH × TH)SUR covariance matrix of vec

( E2 ) is given by

ΩSUR =

⎡ ⎢⎢⎢⎢⎣ ω1 ω1 ω1 . . . ω1 ω1 ω2 ω2 . . . ω2 ω1 ω2 ω3 . . . ω3 . . .

.

.

. . . .

... . . .

ω1 ω2 ω3 . . . ωH

⎤ ⎥⎥⎥⎥⎦ ⊗ IT, (4)

where In denotes the (n × n) identity matrix and ⊗ de- notes the Kronecker product. To account for the possibility

that elements of J can equal zero, modifications of the SUR covariance matrix are required. Denote the number of columns of J with at least one element equal to 1 by H∗, and let the unity vector ei,H denote the ith column of IH. Then define the (H × H∗) selection matrix S as

S = [ e1,H e2,H . . . eH,H,

] where the vector ei,H is contained only if the ith column of J has at least one element equal to 1.

Moreover, define the (TH × TH) diagonal matrix

D = diag (vec (J)) ;

then, the(TH∗ × TH∗)SUR covariance matrix to be used for the estimation is given by

Ω ∗

SUR = ( S′ ⊗ IT

)( DΩSURD′ + ITH − D

)( S′ ⊗ IT

)′ .

The modification involving the D matrix is required in or- der to account for the zero elements in J. The modification using the selection matrix S deletes all elements related to forecast horizons for which no forecast errors exist.

Now define the (T × H∗) matrices

J∗ = JS E∗2 = E2S

and the (TH∗ × H∗) regressor matrix

X = (IH∗ ⊗ 1T) ⊙ ( 1H∗ ⊗ J∗

) ,

where 1n denotes an (n × 1) vector of ones, and ⊙ denotes the Hadamard product.

Then, the SUR estimator of the forecast uncertainty is calculated as

σ̂ 2 SUR =

( X′Ω∗−1SUR X

)−1 X′Ω∗−1SUR vec

( E∗2 ) , (5)

and the OLS estimator, which yields the sample means, is calculated as

σ̂ 2 OLS =

( X′X

)−1 X′vec

( E∗2 ) . (6)

For the following considerations, it is helpful to define the matrix

A = ( X′Ω∗−1SUR X

)−1 X′Ω∗−1SUR , (7)

so that XA is the projection matrix, and σ̂2SUR = Avec ( E∗2 ) .

Conjecture 1. If for all t and all h > 1, j∗th = 1 implies that j∗ti = 1 for every integer i with 0 < i < h, then A depends only on J.

This conjecture states that, if the forecast error et,t−h is available, then the availability of all other fore- cast errors for the same period from shorter horizons et,t−h+1,et,t−h+2, . . . implies that A depends only on known quantities. Neither the dynamics of yt nor the kurtosis of the shocks affect the estimator. This result also holds if et,t−h+i (with i > 0) is unavailable, but the(h − i)th column of J contains only zeros; i.e., if there are no forecast errors for the horizon h−i. A proof of the conjecture for the special case of H∗ = 2 can be found in the Appendix.

The condition of this conjecture is fulfilled in the case where a new, longer forecast horizon is introduced after

M. Knüppel / International Journal of Forecasting 34 (2018) 105–116 109

the first forecasts were produced. A simple example is given by the presence of T 1-step-ahead forecast errors et,t−1, with t = 1,2, . . . ,T , and one 2-step-ahead forecast error eT,T −2, which would lead to the matrix

J = J ∗ = [ 1T −1 0T −1 1 1

] . (8)

The condition of the conjecture is fulfilled here because j∗T2 = 1 and j

∗

T1 = 1 in the last row. The condition would be violated if j∗T1 = 0. The condition is also fulfilled in the example given by Eq. (3).

If the condition is fulfilled, the matrix A can be deter- mined without using Eq. (5).6 In order to do so, partition A into the H∗ submatrices Ah∗ of dimension (H∗ × T) with h∗ = 1,2, . . . ,H∗:

A = [ A1 A2 . . . AH∗

] ,

then, for each h∗, calculate

wh∗ = ( 1′T J

∗eh∗,H∗ )−1 ,

so that 1/wh∗ equals the number of h∗-step-ahead forecast errors.

If h∗ < H∗, determine

Oh∗ = ( J∗eh∗,H∗

) ⊙ ( J∗eh∗+1,H∗

) Nh∗ =

( J∗eh∗,H∗

) ⊙ ( 1T − J∗eh∗+1,H∗

) w

o h∗ = −wh∗

( 1′T Nh∗

)( 1′T Oh∗

)−1 .

Here, Oh∗ is a vector that equals 1 in those periods t where a forecast error exists in the (h∗)th and (h∗ + 1)th columns of E∗2 (periods with ‘‘overlapping’’ forecast errors), and 0 otherwise. Nh∗ is a vector that equals 1 in those periods t where a forecast error exists in the (h∗)th column of E∗2 but not in the (h∗ + 1)th column (periods with ‘‘non- overlapping’’ forecast errors), and 0 otherwise. Finally, woh∗ is a scalar weight with w

o h∗ < 0, defined such that(

woh∗ O ′

h∗ + wh∗ N ′

h∗ ) 1T = 0 holds. Use these quantities to

calculate Ah∗ as

Ah∗ =

⎡ ⎢⎢⎢⎢⎣

0(h∗−1)×T wh∗

( J∗eh∗,H∗

)′ w

o h∗ O

′

h∗ + wh∗ N ′

h∗ . . .

w o h∗ O

′

h∗ + wh∗ N ′

h∗

⎤ ⎥⎥⎥⎥⎦ . (9)

where 0(h∗−1)×T denotes an ((h∗ − 1) × T) matrix of zeros that is an empty matrix for h∗ = 1.

If h∗ = H∗, AH∗ equals

AH∗ = [

0(H∗−1)×T wH∗

( J∗eH∗,H∗

)′] . (10) Note that wh∗ is equal to the OLS weight for an observation belonging to the (h∗)th column of E∗2. Hence, for the ex- ample in Eq. (8) one obtains w1 = 1/T and w2 = 1. For the

6 Like the conjecture itself, the validity of the construction proposed has not been confirmed analytically for H∗ > 2. However, the construc- tion described always yielded the same result as in Eq. (7) in simulations considering many different matrices J where the condition of the conjec- ture is fulfilled, and considering many different values of α and the bis.

same example, A is given by

A = [ A1 A2

] with A1 =

[ (1/T) · 1′T −1 1/T (1/T) · 1′T −1 −(T − 1)/T

] ,

A2 = [ 0′T −1 0 0′T −1 1

] .

This example illustrates how the SUR estimator uses information from shorter forecast horizons to calculate the uncertainty for longer horizons. The 1-step-ahead error that is affected by the same shock as the 2-step-ahead error receives a negative weight in the estimation of the 2-step- ahead uncertainty. In order for the estimator to remain consistent, the T −1 other 1-step-ahead errors must receive weights which equal −1/(T −1) times the negative weight. Note here that for large values of T , σ̂2SUR,2 ≈ σ̂

2 OLS,1 −

e2T,T −1 + e 2 T,T −2.

The example also shows that, in rare cases, the SUR estimator could generate negative estimates for σ̂2SUR,h∗ . In the example, a necessary condition for such an outcome is e2T,T −1 > e

2 T,T −2. However, such results can be avoided by

considering the modified estimator

σ̂ 2 SUR = max

( 0H∗,Avec

( E∗2 )) ,

where max(·) is applied element-wise. This estimator would be expected to be upward biased, but to yield a lower mean squared error than σ̂2SUR.

2.2. Fixed-event forecasts

There are at least two important types of fixed-event forecasts. For example, the ECB forecasts changes in annual averages several times each year. In contrast, the FOMC forecasts are concerned with changes between the fourth quarter of a given year and the fourth quarter of the fol- lowing year (referred to henceforth as q4/q4 forecasts).7 If the fixed-event forecasts are concerned with annual av- erages, the SUR estimator is found to depend on unknown parameters, and therefore, in what follows, I focus on q4/q4 forecasts.8 Adapting the formulas to the case of monthly forecasts of changes from the 12th month of a given year to the 12th month of the following year is straightforward.

The variable of interest xs is defined as

xs = yt + yt−1 + yt−2 + yt−3,

where s refers to the year; t denotes the last quarter in year s; xs denotes the change from the fourth quarter of year s−1 to the fourth quarter of year s; and yt denotes the change from quarter t − 1 to quarter t. The corresponding forecast errors for xs are given by

us,t−1 = et,t−1 us,t−2 = et,t−2 + et−1,t−2

7 In both cases, the forecast for the unemployment rate concerns the level itself, not the change in the level.

8 It remains to be investigated whether efficiency gains can still be obtained in the case of forecasts of annual averages, simply by assuming certain values for the bis and for α; for example, by setting α = 3 and bi = c for i > 0, with c close to zero.

110 M. Knüppel / International Journal of Forecasting 34 (2018) 105–116

us,t−3 = et,t−3 + et−1,t−3 + et−2,t−3 us,t−4 = et,t−4 + et−1,t−4 + et−2,t−4 + et−3,t−4 us,t−5 = et,t−5 + et−1,t−5 + et−2,t−5 + et−3,t−5

.

.

.

us,t−H = et,t−H + et−1,t−H + et−2,t−H + et−3,t−H,

where H is typically a multiple of four. For example, if forecasts of xs are made for the current and the next year in the first, second and third quarters, and for the next year and the year after in the fourth quarter, and if the first forecast was made in the fourth quarter of the year s = 0, then the last realization is observed for the year s = 2 and the last quarter of the year s = 2 is denoted by t = 8, and one obtains

J = [ 1 1 1 1 0 0 0 0 1 1 1 1 1 1 1 1

] E2 =

[ u21,3 u

2 1,2 u

2 1,1 u

2 1,0 c c c c

u22,7 u 2 2,6 u

2 2,5 u

2 2,4 u

2 2,3 u

2 2,2 u

2 2,1 u

2 2,0

] .

In this example, H = 8. If J only contains ones, the SUR covariance matrix of

vec ( E2 ) is given in Box I.

While the SUR covariance matrix differs from its counter- part in the fixed-horizon case, all other formulas continue to apply. Most importantly, the conjecture made in the fixed-horizon case is valid in the fixed-event case as well, and the matrix A for the fixed-event case can be deter- mined in the same manner as in the fixed-horizon case, i.e., by using Eqs. (9) and (10).

The latter result points to a more general applicability of the conjecture. Indeed, simulations suggest that it always holds if ΩSUR can be expressed as

ΩSUR = Ψ ⊗ IT,

with Ψ having full rank and the elements of Ψ having the property

ψij = ψmin(i,j)min(i,j)

with ψij ∈ R. Obviously, the Ψ matrices in Eq. (4) and Box I have this property.

3. Monte Carlo simulations

As was shown by Grenander and Rosenblatt (1957), OLS estimation is asymptotically as efficient as SUR estimation if the regressors consist exclusively of constants. Thus, SUR estimation is helpful in small samples only. If a new fore- cast horizon is introduced, the first sample that can be used for estimating its forecast uncertainty contains only one observation for this horizon, and hence, can certainly be regarded as small. Monte Carlo simulations are employed below in order to study the potential efficiency gains of the SUR estimator. In these simulations, a forecaster produces forecasts for yt+h using the zero-mean first-order autore- gressive model (henceforth AR(1) model)

ŷt+h = ρ̂yt.

The data-generating processes (DGPs) considered for yt are described by

yt = µ + xt xt = ρxt−1 + εt.

with Var (yt) = 1. The four parameterizations investigated and their descriptions are as follows:

optimal: µ = 0 ρ̂ = ρ εt ∼ iid N

( 0,1 − ρ2

) biased mean: µ = 2 ρ̂ = ρ εt ∼ iid N

( 0,1 − ρ2

) biased AR(1) coefficient: µ = 0 ρ̂ = ρ − 0.2 εt ∼ iid N

( 0,1 − ρ2

) optimal, stochastic volatility: µ = 0 ρ̂ = ρ εt ∼ N

( 0, ( 1 − ρ2

) η 2 t

) ,

where in the last case, η2t is given by

ln ( η 2 t

) = −0.5/(1 − 0.9) + 0.9 ln

( η 2 t−1

) + ut,

with ut being iid N ( 0,1 − 0.92

) , resulting in E

[ η2t ]

= 1. The values 0.5, 0.8 and 0.95 are considered for ρ.

While the first DGP corresponds to the assumptions of optimal forecasts and constant forecast uncertainty, the next two DGPs consider non-optimal forecasts and a con- stant forecast uncertainty. Note that the biased-mean fore- cast in particular is severely misspecified, having a mean that differs from the true value by two standard deviations of yt . The last DGP considers optimal forecasts and a time- varying forecast uncertainty. The process forη2t is relatively persistent, but stationary, so that the unconditional first moment of the squared forecast errors exists.9

Forecasts of these processes are produced in the sim- ulations for the sample sizes T = 1,2, . . . ,40. Starting in period t = 0, forecasts for one to four periods ahead are produced, then, starting in period t = 10, forecasts for five periods ahead are also made. The 5-step-ahead mean- squared forecast error σ25 is estimated recursively as the sample size grows, using the SUR estimator in Eq. (5) and the OLS estimator in Eq. (6).

The measure of the efficiency gains of the SUR estimator for the 5-step-ahead forecasts is defined as

100 ln

(√ E [( σ̂2OLS,5 − σ

2 5

)2] ÷ E

[( σ̂2SUR,5 − σ

2 5

)2]) .

Hence, values larger than zero indicate efficiency gains of the SUR estimator. The unit is percentages (relative to the standard deviation of the OLS estimator).

As Fig. 2 shows, large efficiency gains are possible with the SUR estimator. For ρ = 0.5, they can exceed 100%

9 In macroeconomics, the stochastic volatility is often modeled as a unit-root process. However, this modeling choice may be motivated by the need for a parsimonious representation of a persistent process rather than by the assumption that the volatility is indeed non-stationary. Sim- ulations that are not reported here found the efficiency gains to decrease only moderately when the autoregressive parameter of the stochastic volatility equation was increased to 0.99. The signs of the efficiency gains reported below do not change due to the increase. Note that the unconditional kurtosis exists here, being equal to 3e for the 1-step-ahead forecast errors and decreasing for longer horizons.

M. Knüppel / International Journal of Forecasting 34 (2018) 105–116 111

ΩSUR =

⎡ ⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣

ω1 ω1 ω1 ω1 ω1 ω1 · · · ω1

ω1

2∑ i=1

ωi

2∑ i=1

ωi

2∑ i=1

ωi

2∑ i=1

ωi

2∑ i=1

ωi · · ·

2∑ i=1

ωi

ω1

2∑ i=1

ωi

3∑ i=1

ωi

3∑ i=1

ωi

3∑ i=1

ωi

3∑ i=1

ωi · · ·

3∑ i=1

ωi

ω1

2∑ i=1

ωi

3∑ i=1

ωi

4∑ i=1

ωi

4∑ i=1

ωi

4∑ i=1

ωi · · ·

4∑ i=1

ωi

ω1

2∑ i=1

ωi

3∑ i=1

ωi

4∑ i=1

ωi

5∑ i=2

ωi

5∑ i=2

ωi · · ·

5∑ i=2

ωi

ω1

2∑ i=1

ωi

3∑ i=1

ωi

4∑ i=1

ωi

5∑ i=2

ωi

6∑ i=3

ωi · · ·

6∑ i=3

ωi

.

.

. . . .

.

.

. . . .

.

.

. . . .

... . . .

ω1

2∑ i=1

ωi

3∑ i=1

ωi

4∑ i=1

ωi

5∑ i=2

ωi

6∑ i=3

ωi · · ·

H∑ i=H−3

ωi

⎤ ⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦

⊗ IT. (11)

Box I.

Fig. 2. Efficiency gains of the SUR estimator in Eq. (5) for the new forecast horizon h = 5, for which the first forecast is made at t = 10. The sample size increases from T = 1 to T = 40. The results are based on 400,000 Monte Carlo simulations in the case with stochastic volatility and 10,000 simulations for the rest.

when the sample size is small. They decrease as the sample size grows, but remain elevated even when the full sample is available. For ρ = 0.8, the efficiency gains are smaller

but still large for all sample sizes considered, and they also decrease as the sample size grows. For ρ = 0.95, the efficiency gains are still notable in almost all cases, and can

112 M. Knüppel / International Journal of Forecasting 34 (2018) 105–116

Fig. 3. Efficiency gains of the SUR estimator in Eq. (5) for the new forecast horizon h = 5, for which the first forecast is made at t = max(0,T − 5), meaning that at most one forecast error is available for this horizon. The sample size increases from T = 1 to T = 40. The results are based on 400,000 Monte Carlo simulations in the case with stochastic volatility and 10,000 simulations for the rest.

reach more than 40%. The efficiency gains first increase as the sample size grows, then start to decrease. The efficiency gains tend to be smaller if the forecasts are not optimal or the volatility is stochastic than in the cases of optimal forecasts and constant volatility, but the differences are relatively small.

However, since the SUR estimator neglects the autocor- relation of forecast errors, the OLS estimator can be more efficient in certain situations. These cases are characterized by strongly autocorrelated forecast errors in connection with a very small number of observations before the in- crease in the forecast horizon takes place, and a very small number of observations subsequently.

This situation is illustrated in Fig. 3, which uses the same settings as in the previous simulations but where the 5- step-ahead forecasts start at t = max(0,T − 5) instead of t = 10. Thus, only a single 5-step-ahead forecast error is available for every sample size with T ≥ 5 considered. If the sample size is equal to either T = 5 or T = 6 — that is, if the 5-step-ahead forecasts start at either t = 0 (the special case of the recent-forecast-errors structure) or t = 1 — efficiency losses can be observed for the SUR estimator if the DGP is sufficiently persistent. However, the efficiency gains are always positive and often large in the cases that are most relevant in the context of this paper, i.e., the cases where the forecast horizon is increased several periods after the first forecasts for lower horizons begin to be made. Hence, employing the SUR estimator is recommended even more highly in the case of an increased forecast horizon than in the case of the recent-forecast- errors structure analyzed by Knüppel (2014).

4. Applications

Three applications of the SUR estimator are presented below, based on the BoE’s forecasts of real GDP growth for the UK, of the SPF forecasts for the US, and of the FOMC forecasts for the US. A common feature of all forecasts is the presence of nowcasts; i.e., the smallest forecast horizon corresponds to a forecast for the current period. Thus, forecasts for H horizons are available when forecasts are made up to H − 1 periods ahead. The BoE forecasts have a fixed horizon, whereas the FOMC forecasts are fixed-event q4/q4 forecasts. Since the US SPF forecasts are fixed-event forecasts for annual averages, we only consider the fore- casts from the second quarter of each year, which yields fixed-horizon forecasts.

4.1. Bank of England forecasts

Concerning the data from the BoE, quarterly year-on- year real GDP growth rate forecasts have been available for up to eight quarters ahead since the first quarter of 1998 (henceforth 1998q1). Then, in 2004q3, the longest forecast horizon increased from 8 to 12 quarters. Thus, the first forecast errors for these horizons were for the period 2006q4 to 2007q3, whereas up to 35 forecast errors for the shorter horizons already existed in 2006q3. The forecast errors are calculated using second vintages from the BoE real-time database, and end in 2013q1.

In addition to the forecast-error-based measures of the forecast uncertainty, two other measures are also consid- ered. The BoE actually publishes density forecasts, so that

M. Knüppel / International Journal of Forecasting 34 (2018) 105–116 113

the expected variance of the error can be calculated for each forecast.10 Following Clements (2014), I refer to this quantity as the ex-ante uncertainty, and calculate the average over all forecasts available in order to obtain a measure of the unconditional ex-ante uncertainty.11 The same exercise is repeated using only forecasts from 2004q3 to 2013q1, i.e., from the sample in which the longest fore- cast horizon equals 12 quarters. Moreover, a model-based measure is determined. To this end, an autoregressive model for the quarter-on-quarter growth rates of real GDP is estimated, and the forecast error variance of this model is calculated using the bootstrap, as described by Clements and Taylor (2001, Section 2.5). However, since the object of interest here is given by an unconditional measure, there is no need to condition on the observed end-of-sample values.12 The sample used for the estimation of the model starts in 1988q1 and ends in 2013q1. The choice of the starting date is motivated by the fact that, when assessing the unconditional forecast uncertainty, the BoE considers forecast errors from the preceding ten years, as was men- tioned by Wallis (2004, p. 65), for example.13 The model contains three lags, as suggested by the Akaike information criterion (AIC).

The different measures of the forecast uncertainty are depicted in Fig. 4. Concerning the measures based on fore- cast errors, the OLS results for h ≥ 10 are affected by the fact that they are estimated based on a subset of the full sample in which the (squared) forecast errors tended to be large. Since the SUR estimator employs the entire sample for the estimation for all horizons, the forecast uncertainty evolves relatively smoothly as the forecast horizon increases. The measures of the ex-ante uncertainty suggest that the BoE is underconfident at small horizons and overconfident at large horizons, mirroring the results of Clements (2014) for the participants in the US SPF. The ex-ante uncertainty is slightly larger in the sample start- ing in 2004q3 than in the full sample. The model-based forecast uncertainty is similar to both forecast-error-based measures for h < 10, but only to the SUR estimates for h ≥ 10. Altogether, the results show that the SUR estimator

10 The BoE publishes asymmetric density forecasts, which emphasizes the possibility that the downside and upside risks to the forecast might be unbalanced. However, it is questionable whether these asymmetries actually contain valuable information about future risks, as Knüppel and Schultefrankenfeld (2012) document. Therefore, it seems appropriate to restrict the analysis of the BoE’s density forecasts to first and second moments only. 11 Clements (2014) uses the standard deviation as the measure of fore-

cast uncertainty, whereas we employ the variance. For the calculation of the variance based on the parameters published by the BoE, the formulas of Wallis (2004) are used. 12 Two additional differences with respect to Clements and Taylor

(2001) arise from the facts that the bootstrapped series are transformed from quarter-on-quarter growth rates to year-on-year growth rates be- fore calculating the object of interest, and that the object of interest is the mean-squared forecast error instead of the prediction interval. 13 Applying the tests of Andrews (1993) and Andrews and Ploberger

(1994) to the model parameters and to the mean of the squared residuals does not yield any evidence against the absence of structural breaks, which implies that the bootstrap approach is appropriate. The same result is obtained for the model estimated for US GDP growth in the following subsection.

Fig. 4. Estimates of the mean squared forecast errors of the Bank of England’s forecasts for quarterly year-on-year real GDP growth in the UK for the quarterly horizons h = 1 to h = 13, where h = 1 corresponds to the nowcast. The ex-ante uncertainty refers to the average forecast error variance implied by the BoE’s density forecasts. The model-based uncertainty is calculated using an autoregressive model and the bootstrap.

clearly delivers more plausible results than the OLS estima- tor. Comparing the results of the SUR estimator for h = 13 to those reported in Table 6 of Knüppel (2014) suggests that using the methodology presented here instead of that of Knüppel (2014) can lead to quantitatively important improvements.14

The dynamic behaviors of both forecast-error-based measures are illustrated in Fig. 5, which shows the re- cursive estimates of the forecast uncertainty starting in 2006q4. While the SUR and OLS estimates obtain similar values between 2006 and the beginning of 2008, the large forecast errors for the end of 2008 and 2009 lead to strong increases in the OLS estimates. The increases are more sub- dued for the SUR estimates because they employ a larger sample. The OLS estimates peak in 2009q4 and decrease strongly thereafter, while the SUR estimates remain rela- tively stable after 2009q4.

4.2. US SPF forecasts

With respect to the data from the US SPF, I consider the second-quarter consensus forecasts for annual averages of

14 Knüppel (2014) obtained a value of about 13 from the SUR estimator for the sample 2004q3 to 2010q4 and for h = 13. The pronouncedly smaller value of the SUR estimator reported here is due mainly to the fact that the sample can start in 1998q1 instead of 2004q3, now that the re- strictions on the data structure have been relaxed. Given the starting date of 1998q1, the results are very robust with respect to the two alternative sample ends. The statement that the estimate presented in Fig. 4 is better than that of Knüppel (2014) relies on the assumption that no structural break leading to a higher unconditional forecast uncertainty has occurred in the sample under study. In this respect, it might be interesting to note that the mean squared error of forecasts made since 2010q1 for h ≥ 7 is about 5, which is marginally smaller than the full-sample value of about 6 that can be calculated for h = 7,8,9.

114 M. Knüppel / International Journal of Forecasting 34 (2018) 105–116

Fig. 5. Evolution of the recursive estimates of the mean-squared forecast errors of the Bank of England’s forecasts for real GDP growth in the UK for the quarterly horizons h = 10,11,12,13.

real GDP from 1984q2 to 2014q2.15 The starting year is chosen because of the structural break in the volatility that is documented by McConnell and Perez-Quiros (2000). The second quarter is used because the first 2- and 3-year- ahead forecasts became available in 2009q2. The forecast for the current year requires the current quarter and the next two quarters to be nowcasted, implying a forecast horizon of h = 3 quarters. The 1-, 2- and 3-year-ahead forecasts imply horizons of 7, 11, and 15 quarters, re- spectively. The forecast errors are calculated using second vintages from the real-time database of the Federal Reserve Bank of Philadelphia (see Croushore & Stark, 2001), and end in 2014. Again, the ex-ante uncertainty and the model uncertainty are considered as additional measures. For the ex-ante uncertainty, I use the variance of the aggregate probability distribution of the US SPF, which is one of the measures employed by Clements (2014).16 Following Boero et al. (2015) and Giordani and Sderlind (2003), a normal distribution is fitted to the histograms of the SPF. I do so for the full sample as well as for the forecasts from 2009 to 2014, i.e., for the sample where the longest forecast horizon equals 15 quarters. The model-based measure is derived in the way described above, with the sample of US real GDP quarter-on-quarter growth starting in 1984q1

15 The consensus forecast is the mean of the point forecasts of all survey participants. 16 To be more precise, Clements (2014) employs the standard devia-

tion, which he denotes by σaggh , whereas I use the variance ( σ

agg h

)2 .

and ending in 2014q4, and the number of lags set to two based on the AIC.17

The different measures of the forecast uncertainty are depicted in Fig. 6. The (squared) forecast errors for 2011 to 2014 were relatively small, leading to OLS estimates of the forecast uncertainty for the horizons h = 11 and h = 15 that are about half as large as that for h = 7. In contrast, the SUR estimator indicates small increases in forecast un- certainty for the longer horizons. When considering the ex- ante uncertainty measures, the results of Clements (2014) hold. For the short horizon h = 3, the forecasters are underconfident, whereas for the longer horizon h = 7, they are overconfident. This result holds for both the full- sample ex-ante uncertainty and the ex-ante uncertainty since 2009. For the horizons of 11 and 15 quarters that were not considered by Clements (2014), one would find based on the OLS estimates that the forecasters are un- derconfident, whereas the SUR estimates imply that they also remain overconfident for these two large horizons, which appears more plausible. The model-based measure implies a moderately greater uncertainty than the SUR estimator, but the uncertainty evolves similarly over the forecast horizons.

17 Here, the bootstrapped series are transformed from quarter-on- quarter growth rates to growth rates of annual averages before calculating the mean-squared forecast error.

M. Knüppel / International Journal of Forecasting 34 (2018) 105–116 115

Fig. 6. Estimates of the mean-squared forecast errors of the US SPF’s con- sensus forecasts for real GDP growth of annual averages for the quarterly horizons h = 3,7,11,15, where h = 3 corresponds to the nowcast, i.e., the forecast made in the second quarter for the current year. The ex- ante uncertainty refers to the average forecast error variance implied by the SPF’s aggregate probability distribution. The model-based uncertainty is calculated using an autoregressive model and the bootstrap.

4.3. FOMC forecasts

Concerning the data from the FOMC, I use the quarterly forecasts for the q4/q4 growth rate that have been avail- able since October 2007.18 These forecasts, the so-called Summaries of Economic Projections, contain nowcasts and 1- and 2-year-ahead forecasts, as well as 3-year-ahead forecasts in some cases, depending on the quarter in which the forecast is made. Moreover, the FOMC has also been publishing ‘‘longer-run’’ forecasts since 2009q1, which are assumed to represent 5-year-ahead forecasts.19 Thus, this is a case where H∗ < H because there are no 4-year- ahead forecasts. Hence, the longest forecast horizon is 23 quarters, corresponding to the long-run forecast made in the first quarter of the year. The forecasts are given as ranges only. I calculate a point forecast as the mean of the lower and upper ends of the ‘‘central tendency’’(referred to as the mean forecast henceforth). The forecast errors and the model-based measure of the uncertainty are calculated in the same way as for the US SPF.20

18 Previously, the forecasts were made only on a semi-annual basis. 19 The timing of the FOMC forecasts changed in 2013. From October

2007 until mid-2012, forecasts were made in January, April, June, and October/November. Since mid-2012, they have been produced in March, June, September and December. One could try to construct March and September forecasts before mid-2012 by interpolation in order to achieve a better synchronization of the forecast dates, but I abstain from doing so here. Below, the pre-2012 forecasts from June are regarded as forecasts from the third quarter. There were actually five forecasts made in 2012, but the April forecasts are not included in the data set used here. 20 Here, the bootstrapped series are transformed from quarter-on-

quarter growth rates to quarterly year-on-year growth rates before cal- culating the mean-squared forecast error.

Fig. 7. Estimates of the mean-squared forecast errors of the FOMC’s fixed- event forecasts for q4/q4 real GDP growth in the US for the quarterly horizons h = 1 to h = 24, where h = 1 corresponds to the nowcast, i.e., the forecast made in the fourth quarter for the current year. The horizons h = 21,22,23,24 correspond to the long-run forecasts. The model-based uncertainty is calculated using an autoregressive model and the bootstrap.

The results depicted in Fig. 7 indicate that the OLS estimates of the long-run forecast uncertainty deliver un- reasonably small values, due to the fact that they are based on only one observation. The SUR long-run esti- mates are larger because they make use of all observations and are close to the model-based estimates. Nevertheless, both forecast-error-based estimators yield values that are smaller than many of the estimates for shorter horizons, which does not appear very plausible and indicates the need for a larger sample. The model-based measure sug- gests that the SUR approach might overestimate the fore- cast uncertainty for h = 6,7, . . . ,13, whereas the OLS approach might tend to underestimate it for these hori- zons.

5. Conclusion and outlook

This paper relaxes the restriction on the data struc- ture that Knüppel (2014) imposed for the SUR estimation of the unconditional forecast uncertainty for multi-step- ahead forecasts, labeled the recent-forecast-errors struc- ture. It turns out that the SUR estimator is independent of unknown parameters in the case of optimal forecasts in various circumstances that have not been considered previously. These circumstances include the empirically relevant case where new, longer forecast horizons are introduced. The SUR estimator is found to be robust with respect to non-optimal forecasts and the presence of sta- tionary time-varying uncertainty, and provides potentially large efficiency gains when new, longer forecast horizons are introduced. These efficiency gains exceed those ob- served in the recent-forecast-errors case. Applications to

116 M. Knüppel / International Journal of Forecasting 34 (2018) 105–116

the real GDP growth forecasts of the Bank of England, the US SPF and the FOMC illustrate the usefulness of the proposed SUR estimator.

Appendix

A.1. Proof of conjecture for H∗ = 2

Suppose that H∗ = 2. In this case, the proof of the con- jecture for parameter independence is very similar to that of Knüppel (2014), which follows the setup of Im (1994). It requires that all rows that contain only zeros be deleted from X, and that the remaining rows be reorganized (if necessary) such that the resulting matrix X̃ can be written as

˜X = [

1T 0T 0T −M 1T −M

] =

[ 1M 0M

1T −M 0T −M 0T −M 1T −M

] .

where M is the number of periods in which only 1-step errors are available, making T − M the number of 2-step errors available. As the corresponding SUR covariance ma- trix equals

Ω̃SUR =

[ g1IT

[ 0T −M g1IT −M

]′[ 0T −M g1IT −M

] g2IT −M

] =

[ g1IM 0′T −M 0

′

T −M 0T −M g1IT −M g1IT −M 0T −M g1IT −M g2IT −M

] ,

the corresponding matrix à is given by( X̃′Ω̃−1SURX̃

′ )−1

X̃′Ω̃−1SUR

=

⎡ ⎢⎣

1 T 1′M

1 T 1′T −M 0

′

T −M

1 T 1′M −

M T (T − M)

1′T −M 1

T − M 1′T −M

⎤ ⎥⎦ ,

which depends on neither g1 nor g2.

References

Abel, J., Rich, R., Song, J., & Tracy, J. (2016). The measurement and behavior of uncertainty: Evidence from the ECB survey of professional forecast- ers. Journal of Applied Econometrics, 31(3), 533–550. http://dx.doi.org/ 10.1002/jae.2430.

Andrews, D. W. K. (1993). Tests for parameter instability and structural change with unknown change point. Econometrica, 61(4), 821–856.

Andrews, D. W. K., & Ploberger, W. (1994). Optimal tests when a nuisance parameter is present only under the alternative. Econometrica, 62(6), 1383–1414.

Boero, G., Smith, J., & Wallis, K. F. (2015). The measurement and char- acteristics of professional forecasters’ uncertainty. Journal of Applied Econometrics, 30(7), 1029–1046.

Clark, T. E., McCracken, M. W., & Mertens, E. (2017). Modeling time- varying uncertainty of multiple-horizon forecast errors. mimeo.

Clements, M., & Taylor, N. (2001). Bootstrapping prediction intervals for autoregressive models. International Journal of Forecasting, 17(2), 247–267.

Clements, M. P. (2014). Forecast uncertainty — ex ante and ex post: U.S. inflation and output growth. Journal of Business & Economic Statistics, 32(2), 206–216.

Croushore, D., & Stark, T. (2001). A real-time data set for macroeconomists. Journal of Econometrics, 105(1), 111–130.

Ericsson, N. R. (2002). Predictable uncertainty in economic forecasting. In D. F. H. Michael P. Clements (Ed.), A companion to economic fore- casting (pp. 19–44). Blackwell.

Giordani, P., & Sderlind, P. (2003). Inflation forecast uncertainty. European Economic Review, 47(6), 1037–1059.

Grenander, U., & Rosenblatt, M. (1957). Statistical analysis of stationary time series. New York: John Wiley and Sons.

Hartmann, M., Herwartz, H., & Ulm, M. (2017). A comparative assessment of alternative ex ante measures of inflation uncertainty. International Journal of Forecasting, 33(1), 76–89.

Im, E. I. (1994). Unequal numbers of observations and partial efficiency gain. Economics Letters, 46, 291–294.

Jordà, O., Knüppel, M., & Marcellino, M. (2013). Empirical simultaneous prediction regions for path-forecasts. International Journal of Forecast- ing, 29(3), 456–468.

Jurado, K., Ludvigson, S. C., & Ng, S. (2015). Measuring uncertainty. American Economic Review, 105(3), 1177–1216.

Knüppel, M. (2014). Efficient estimation of forecast uncertainty based on recent forecast errors. International Journal of Forecasting, 30(2), 257–267.

Knüppel, M., & Schultefrankenfeld, G. (2012). How informative are central bank assessments of macroeconomic risks? International Journal of Central Banking, 8(3), 87–139.

Lahiri, K., & Sheng, X. (2010). Measuring forecast uncertainty by dis- agreement: the missing link. Journal of Applied Econometrics, 25(4), 514–538.

Lee, Y. S., & Scholtes, S. (2014). Empirical prediction intervals revisited. International Journal of Forecasting, 30(2), 217–234.

McConnell, M. M., & Perez-Quiros, G. (2000). Output fluctuations in the United States: What has changed since the early 1980s? American Economic Review, 90(5), 1464–1476.

Patton, A. J., & Timmermann, A. (2011). Predictability of output growth and inflation: a multi-horizon survey approach. Journal of Business & Economic Statistics, 29(3), 397–410.

Reifschneider, D., & Tulip, P. (2007). Gauging the uncertainty of the economic outlook from historical forecasting errors. In Finance and economics discussion series of the board of governors of the federal reserve system, No. 60.

Rossi, B., & Sekhposyan, T. (2015). Macroeconomic uncertainty indices based on nowcast and forecast error distributions. American Economic Review, 105(5), 650–655.

Tulip, P., & Wallace, S. (2012). Estimates of uncertainty around the RBA’s forecasts. In RBA Research Discussion Papers rdp2012-07. Reserve Bank of Australia.

Wallis, K. F. (1989). Macroeconomic forecasting: a survey. The Economic Journal, 99, 28–61.

Wallis, K. F. (2004). An assessment of Bank of England and national institute inflation forecast uncertainties. National Institute Economic Review, 189, 64–71.

Williams, W. H., & Goodman, M. L. (1971). A simple method for the construction of empirical confidence limits for economic forecasts. Journal of the American Statistical Association, 66(336), 752–754.

Malte Knüppel studied economics at the University of Osnabrück, the Humboldt-University in Berlin, and the ENSAE in Paris. He earned his Ph.D. in 2004 at the University of Hamburg. Since 2003, he has worked at the Research Centre of the Deutsche Bundesbank in Frankfurt. His major research interests are forecasting and non-linear time series analysis.