Review on Energy Resilience

Deep Learning for Energy Markets

Michael Polson∗and Vadim Sokolov†

First Draft: April 2018 This Draft: April 2019

Abstract

Deep Learning is applied to energy markets to predict extreme loads observed in en- ergy grids. Forecasting energy loads and prices is challenging due to sharp peaks and troughs that arise due to supply and demand fluctuations from intraday system con- straints. We propose deep spatio-temporal models and extreme value theory (EVT) to capture theses effects and in particular the tail behavior of load spikes. Deep LSTM architectures with ReLU and tanh activation functions can model trends and tempo- ral dependencies while EVT captures highly volatile load spikes above a pre-specified threshold. To illustrate our methodology, we use hourly price and demand data from 4719 nodes of the PJM interconnection, and we construct a deep predictor. We show that DL-EVT outperforms traditional Fourier time series methods, both in-and out-of- sample, by capturing the observed nonlinearities in prices. Finally, we conclude with directions for future research.

Keywords: Deep Learning, PJM Interconnection, EVT, Machine Learning, Locational Marginal Price (LMP), Peak prediction, Energy Pricing, Smart Grid, LSTM, ReLU.

∗Michael Polson is a researcher at Bates White. email:[email protected] †Vadim Sokolov is Assistant Professor in Operations Research at George Mason University.

email:[email protected]

1

ar X

iv :1

80 8.

05 52

7v 3

[ st

at .M

L ]

1 0

A pr

2 01

9

1 Introduction

Deep learning (DL) in conjunction with Extreme Value Theory (EVT) is proposed to pre- dict load and wholesale energy prices on the energy grid. This is essential for the eco- nomic operation of grid resources, as electricity grids operate without large amounts of storage, and generation of energy (supply) within the system must always match the de- mand of energy (load). Electricity price prediction is challenging due to a number of com- plex factors that impact the intraday grid conditions, which create highly volatile price spikes. Deep Learning multi-layer networks are developed to capture nonlinearities and the spatio-temporal patterns in energy prices and demand. Likelihood function defined by Extreme Value Theory allows to properly model spikes. As supply must constantly adapt to meet changes in load, accurate predictions are essential to making informed short and long-term generation decisions. Accurate anticipation of fluctuations in load, especially sharp fluctuations, would remove certain flexibility constraints, allowing for efficient deployment of generation and grid resources.

The traditional approach to electricity price prediction has been applying economic models based on firm behavior to the data. More recently, data-driven analytics, using large price data sets and machine learning techniques have been used to uncover price patterns. Forecasting supply and demand with a standard deep learning model, however, fails to address the importance of peak prediction. Deep learning models aim to predict the mean level of the dependent variable and typically do not capture any extreme jumps in the data. Furthermore, squared loss is used to fit the model, which implicitly assumes the Gaussian distribution of the errors. Therefore, in the context of electricity markets, a Gaussian model would be well suited for predicting the system’s average demand or energy price, but would not capture the peaks nor the true, fat-tailed distribution of the dependent variables. Until now, data-driven models were not flexible enough to cap- ture the extreme nonlinearities in the price the dynamics. Recently, deep learners (DL) have shown empirical success in large datasets forecasting problems with high dimen- sional nonlinearities. Long-short-term memory (LSTM) provides a framework for build- ing spatio-temporal model (Dixon et al., 2019; Polson and Sokolov, 2017b).

Modeling extremes has a long history in financial risk management Poon et al. (2003). For example, Hilal et al. (2011) develop an extreme value theory model for stock indexes. Like predicting price in equity markets, predicting price in wholesale electricity markets is challenging due to a number of complex factors that impact intraday supply and de- mand.

Davison et al. (2012) develop a spatial statistical model for the extremes of a natural process. Peaks are modeled as an exceedance of a certain threshold. EVT provides the framework for the prediction of these exceedances, it predicts the frequency of energy price exceeding a certain threshold Davison and Smith (1990). The exceedance over a threshold allows to measure risk associated with high prices Smith (2002). Incorporating extreme value theory (EVT) into deep learning allows us to capture the tail behavior of the price distribution. We develop deep learning multi-layer networks to capture nonlinear- ities and spatiotemporal patterns in the price distribution. In particular, likelihood func- tions, defined using the Extreme Value Theory framework, allow us to properly model price spikes. In the context of energy markets capturing the spikes is a crucial as these

2

are the central component of interest in the market. Furthermore, this approach provides an improvement over traditional deep learning approaches, which typically only focus on capturing the mean of a given distribution. Our work builds on that of Sigauke et al. (2013) who develop probabilistic EVT model and Shenoy and Gorinevsky (2014) who use generalized linear model, with EVT errors to model electricity demand.

The rest of our paper is organized as follows. Section 1.1 provides connections to previous work. Section 2 describes the energy market for electricity and the PJM inter- connection. Section 3 discusses traditional deep learning models. Section 4 extends DL models using extreme value theory (EVT). Section 5 provides algorithms for load and price prediction for PJM. Finally, Section 6 concludes with directions for future research.

1.1 Connection to Previous Work

Data-driven energy pricing models used to forecast hourly locational marginal prices (LMPs) have been studied previously (Catalão et al., 2007; Hong and Hsiao, 2002, 2001; Kim, 2015). Hong and Hsiao (2001) propose neural networks to predict LMPs in the PJM Interconnection. Mandal et al. (2007) use neural networks to improve performance, and Catalão et al. (2007); Kim (2015) predict LMPs in Nord Pool, an electricity spot market located across Northern Europe. Wang et al. (2017) predict prices at various hubs in the American Midwest with a stacked denoising auto-encoder, exploiting local information to improve its predictive performance. Modeling wind generated-electricity is considered by Hering and Genton (2010). Our analysis extends the functional data analysis approach for electricity pricing developed by Liebl (2013).

In another line of research, Cottet and Smith (2003) and Wilson et al. (2018) develop a random effects Bayesian framework to quantify uncertainty in wholesale electricity price projections. Jónsson et al. (2013) forecast electricity prices while accounting for wind power prediction. Christensen et al. (2012) forecast spikes in electricity prices. Heavy tails in electricity prices are modeled using multivariate skew t-distributions by Cottet and Smith (2003). Benth and Schmeck (2014) address the non-Gaussian nature of price data using a L’evy process. Dupuis (2017) develops a detrended correlation approach to capture price dynamics within the New York section of the grid. Garcia et al. (2005) ex- plain time-varying volatility in prices using GARCH effects for one-day price forecasting. Li et al. (2007) develop a fuzzy inference system to forecast prices in LMP spot markets. Subbayya et al. (2013) address the problem of model selection.

Our approach uses extreme value theory (EVT) in combination with deep learning. To our knowledge, this is the first time EVT-based deep learning model is developed and applied.

2 Energy Prices in PJM Interconnection

The PJM Interconnection is a regional transmission organization (RTO), that exists to cre- ate a competitive wholesale electricity market, coordinating numerous wholesale electric- ity producers and consumers in all or parts of 13 states located in America’s Mid-Atlantic and Great Lakes Regions as well as the District of Columbia.

3

PJM is divided into 20 transmission zones. Each zone is owned and operated by separate transmission owners who are responsible for designing and maintaining their portion of the system. Figure 1 shows PJM’s load nodes and zone boundaries. Individ- ual utilities within PJM plan their use of resources around peak loads. Predicting the strength and timing of these peaks is integral to improving both short- and long-term decision-making. Current methods used for short-term prediction focus on neural net- works (weather channel, PJM).

PJM acts as a guarantor of system reliability and is responsible for preventing out- ages within the system. PJM operates the system at a cost-efficient level by coordinating generating plant operations, which are owned by various entities, to match the system’s demand. Operating the system includes ensuring real-time demand is met, maintaining a reserve capacity of generation, and monitoring transmission lines to prevent overloaded lines, which could cause system failure (Cain et al., 2007).

Figure 1: PJM Zone boundaries and node locations

The PJM Interconnection contains over 11,000 nodes for which hourly day-ahead or real-time prices are reported. These nodes are specific generation or load locations, ag- gregates of various locations, regions, or points of interconnection with areas outside of PJM. We examine prices at the 4,700 load nodes across the system.

Within the PJM Interconnection, nearly all wholesale electricity is bought or sold through bilateral contracts. The remainder is bought or sold on the two bid-based electricity mar- kets PJM operates: day-ahead (DA) and real-time (RT) markets. In the day-ahead market, market participants submit bids or offers to buy or sell energy to the scheduling opera- tor (PJM). The operator uses the bid and offers to determine the day-ahead LMP, which

4

reflects the expected cost of energy, congestion and transmission loss needed to provide electricity at a location given the expected system constraints.

The real-time market operates in a similar way but reflects the actual cost of provid- ing electricity at a location given actual system constraints. Despite the comparatively smaller volume, the real-time market plays a central role in determining the price of all futures contracts as the futures contracts’ prices depends on the expected of the real-time market prices. The day-ahead market is a futures market that allows generators to enter agreements to provide electricity for the upcoming day.

Generators can fulfill obligations to provide energy by either physically producing electricity or purchasing it on the real-time market. Multiple factors, such as unexpected maintenance, may cause a generator to fulfill their obligation through purchases on the real-time market rather than generation. These factors, or risks, cause significant volatility in real-time markets compared to the day-ahead market (FERC, 2014).

Prices in the real-time market are a function of the cost to produce electricity and system constrains, such as congestion in transmission lines. When these constraints are binding, prices differ across locations in the PJM Interconnection to reflect the relative ease of delivering energy to a non-congested location and the relative difficulty of deliv- ering energy to a congested location. Therefore, each node (or location) has an associated locational marginal price (LMP), which reflects the price of the marginal unit of electricity delivered to that specific location. LMPs are important price signals in the day-ahead and real-time market, which inform short-term decisions, as well as long-term investments and bilateral agreements (Cain et al., 2007).

2.1 Local Marginal Price Data (LMP)

Locational Marginal Pricing is used to price energy on the PJM market in response to changes in supply and demand and the hardware’s physical constraints. LMP accounts for the cost to produce the energy, the cost to transmit this energy within PJM RTO, and the cost of energy lost due to resistance as the energy is transported across the system. LMP data is available at www.pjm.com (PJM). Our study uses price data, which includes real-time and day-ahead hourly prices from 1/1/2017 to 12/31/2017. Load prices rep- resent the cost of providing electricity at a given location. The price reflects the system’s load (demand), generation, and limits of the transmission system. The system’s con- straints can affect locations asymmetrically, causing variations in price across different locations. Hub prices are a collection of these locational prices and are intended to reflect the uncongested price of electricity.

LMPs have three components: energy, congestion, and marginal loss. The energy component reflects the price of electricity, called system marginal price (SMP). SMP is calculated based on the current dispatch (supply) and load (demand). SMP is calculated for both the day ahead and real time markets. The congestion component is greater than zero whenever congestion occurs at a given node. Constraints occur when delivery limi- tations prevent the use of least-cost generator, for example, a higher cost generator located closer to load must be used to meet the demand if transmission constraints are present. The congestion price is calculated using the shadow price, which is the value of the dual

5

variable (price of violating a binding constrain) in the optimization problem that governs the grid. When none of the constraints are active, all the congestion prices are zero.

The marginal loss component reflects the cost of transmission and other losses at a given location. Losses are priced according to marginal loss factors, which are calculated at a bus and represent the percentage increase in system losses caused by a small increase in power injection or withdrawal.

Variable name Description TotalLMP Total cost; reflects Energy + congestion + marginal loss

CongestionLMP Congestion component of the LMP; can be +ve or −ve Value is relative to the energy component

MarginalLMP marginal loss component of the LMP

Table 1: Description of LMP variables.

2.2 Nonlinearities in Prices and Demand

The dynamics of energy prices are nonlinear due to the congestion component in the Locational Marginal Price. The congestion price represents the cost of violating a binding constraint in the linear program that models optimal generation strategy. The congestion price is paid by the load (consumer) to the generator (the producer). The congestion prices are calculated in real time and a day ahead. The constraints are the results of several physical limits of the electric grid and include thermal limits due to the power system equipment’s thermal capability, voltage limits, and stability limits. Figure 2 shows the temporal patterns in the load data and the relationship between price and load variables. The figure shows that the relationships are non-linear.

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

● ●

● ●

●

●

●

●

●

●

●

●

●

● ●●

●

●

●

●

●

● ●●

●

●

● ●

●

●

●

●

●

●

●

●

●

● ●●●

●

●

●

●

●

●

●

●

●

●

●●

● ●

●

●

●

●

●

●● ●

●

●

●

●

●

● ● ●●●

●●●● ● ●

●

●

●

●

●

● ● ●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

● ●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

● ●●

●

●

● ●

●

●

●

●

●

●

● ● ● ●

●

●

●

●

●

●

●

●

● ●●

●

● ●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

● ●● ●●

●

● ●

●

●

●

●

●

● ● ●

●

●

●

●

●

●

● ●

●● ●

● ●●

● ●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

● ● ●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

● ● ●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

● ●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

● ●

●

● ●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

● ●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

● ●

● ●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

● ●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

● ●●

●

●

● ●

●

●

●

●

●

●

●● ●

●

●

●

●

●

●

●

●

●

● ●●

●

● ●

●

●

●

●

●

●

● ●●

●

●

●

●

●

●

●

●

●

●

● ●

● ●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

● ●

● ● ● ● ●

● ● ●

●

●

●

●

●

● ●

●

●

●

●

●

●

● ●●●●

●●● ●●●

●

●

●

●

●

● ● ●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

● ●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

● ●●●

●

● ●

●

●

●

20 40 60 80 100 1202 0

0 0

0 3

0 0

0 0

4 0

0 0

0 5

0 0

0 0

Price

L o

a d

( M

W )

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

● ●

● ●

●

●

●

●

●

●

●

●

●

● ● ●

●

●

●

●

●

● ●●

●

●

●●

●

●

●

●

●

●

●

●

●

● ● ● ●

●

●

●

●

●

●

●

●

●

●

●●

● ●

●

●

●

●

●

● ● ●

●

●

●

●

●

● ●●●●

●●●● ●●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

● ● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

● ●●

●

●

● ●

●

●

●

●

●

●

● ●

● ●

●

●

●

●

●

●

●

●

● ●●

●

● ●

●

●

●

●

●

●

●

● ● ●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

● ●●●●

●

●●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

● ●

●● ●

● ●●

●●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

● ●●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

● ●

● ●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

● ● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

● ●

●

● ●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

● ●●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

● ●

●●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●●

●

● ●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

● ●●

●

●

●●

●

●

●

●

●

●

● ● ●

●

●

●

●

●

●

●

●

●

● ●●

●

● ●

●

●

●

●

●

●

● ● ●

●

●

●

●

●

●

●

●

●

●

● ●

● ●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

● ●

● ●

●●●

● ●●

●

●

●

●

●

● ●

●

●

●

●

●

●

● ●●●●

●●● ●●●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

● ●●●

●

●

20000 25000 30000 35000 40000 45000 500002 0

0 0

0 3

0 0

0 0

4 0

0 0

0 5

0 0

0 0

Load at time t−5

L o

a d

a t

tim e

t

Figure 2: Temporal patterns and nonlinearities in price-demand relationship

The key to efficient electric grid management is understanding peak loads. At the day- to-day level, over- or underestimating peak loads can be costly. Overestimating peaks will cause the system to have too much generation in reserve. Underestimating peaks will

6

cause the system to call upon costly, but flexible, sources of energy to quickly meet the de- mand. Day-to-day prediction is further complicated by the increase in renewable energy, whose pattern of generation does not always match the system’s pattern of demand. This imbalance in supply and demand patterns adds to the volatility of the system’s energy prices, further complicating prediction (tiaraiya et al., 2011; Hogg and Klugman, 1983).

3 Deep Learning

Let y denote a low dimensional output and x = (x1, . . . , xp) a high dimensional input. One of the advantages of DL models is the ability to analyze inputs with p being in millions. For example, in image analysis 1 1 megapixel input image will correspond to p = 106.

DL uses a composite of univariate semi-affine rather than traditional additive func- tions. Deep learning models can efficiently approximate high-dimensional functions y = F(x). A deep network prediction, denoted by ŷ(x), is defined by hierarchical layers

z0 = x, z1 = a1(W1z0 + b1), . . . , zL = aL(WLzL−1 + bL) ŷ(x) = WL+1zL + bL+1 (1)

where Wl ∈ Rnl×nl−1 is the weight matrix, bl ∈ R is the bias term, and nl is the number of neurons in layer l. Here, we apply non-linear activation function al element-wise to the activation vectors (Wl zl−1 + bl). Typical activation functions are rectified linear unit (ReLU) a(x) = max(x, 0) and sigmoid a(x) = 1/(1 + e−x).

Specifically, the deep learning approach employs a series of hierarchical predictors comprising L nonlinear transformations applied to the input x. Each of the L transforma- tions is referred to as a layer, where the original input is x, the output of the first transfor- mation is the first layer, and so on, with the output ŷ as the last layer. Layers 1 to L are called hidden layers. The number of layers, L, represents the depth of our routine. Note that a linear regression is a particular case of a deep learning model with no hidden layers, e.g. L = 0.

The fact that DL forms a universal “basis” which we recognize in this formulation dates to Poincare and Hilbert. From a practical perspective, given a large enough dataset of “test cases”, we can empirically learn an optimal predictor. Similar to a classic basis decomposition, the deep approach uses univariate activation functions to decompose a high dimensional x.

It is well known that shallow networks are universal approximators and thus can be used to identify any input-output relations. The first result in this direction was obtained by Kolmogorov (1957) who showed that any multivariate function can be exactly repre- sented using operations of addition and superposition on univariate functions. Formally, there exist continuous functions ψpq, defined on [0, 1] such that each continuous real func- tion F defined on [0, 1]n is represented as

F(x1, . . . , xn) = 2n+1

∑ q=1

aq

( n

∑ p=1

ψpq(xp)

) ,

7

where each aq is a continuous real function. This representation is a generalization of earlier results Kolmogorov (1956); Arnold (1963). Kolmogorov (1956) showed that every continuous multivariate function can be represented in the form of a finite superposition of continuous functions of not more than three variables.

Recurrent Neural Networks (RNNs) is a specific type of architecure designed to an- alyze sequences, e.g. time series data. RNNs can capture electricity prices’ time series properties. Recurrent layers capture long term dependencies without much increase in the number of parameters. They learn temporal dynamics by mapping an input sequence to a hidden state sequence and outputs via a recurrent relationships. Let yt denote the ob- served data and ht a hidden state, then

yt = σ(W1ht + bz) ht = σ(W2[xt, ht−1] + bh).

Here σ(x) = 1/(1 + e−x) is the sigmoid function applied component-wise and is used for calculating both the hidden vector ht and the output vector yt. The main difference between RNNs and feed-forward deep learning is the use of a hidden layer with an auto- regressive component, here ht−1. It leads to a network topology in which each layer represents a time step, and we index it by t to highlight its temporal nature.

xt

ht−1

σ

ht

ŷ t

σ ht

Figure 3: Hidden layer of a Recurrent Neural Network

A particular type of RNN, called LSTM (Long short-term memory), was proposed to address the issue of vanishing or exploding gradients in plain RNNs during training. A memory unit used in LSTM networks allows a network to learn which previous states can be forgotten (Hochreiter and Schmidhuber, 1997; Schmidhuber and Hochreiter, 1997b).

The hidden state will be generated via another hidden cell state ct that allows for long term dependencies to be “remembered”. Then, we generate

Output: ht = ot ? tanh(ct) kt = tanh(Wc[ht−1, xt] + bc) ct = ft ? ct−1 + it ? kt

State equaitons:

  ftit

ot

  = σ(W[ht−1, xt] + b).

Where ? denotes point-wise multiplication. Then, ft ? ct−1 introduces the long-range de-

8

pendence. The states (it, ft, ot) are input, forget, and output states. Figure 4 shows the network architecture.

σ

ct−1

σ tanh σ

ht−1

xt

ht

ct

ŷ t

ot it gt ft tanh

Figure 4: Hidden layer of an LSTM model. Input (ht−1, xt) and state output (ht, ct).

The key addition, versus RNN is the cell state ct, the information is added or removed from the memory state via gates defined via the activation function σ(x) and point-wise multiplication ?. The first gate ft ? ct−1, called the forget gate, allows to throw away some data from the previous cell state. The next gate it ? kt, called the input gate, decides which values will be updated. Then, the new cell state ct is a sum of the previous cell state ct−1, passed through the forgot gate plus selected components of the kt vector, which is a filtered version of inputs (ht−1, xt). Thus, the vector ct provides a mechanism for dropping irrelevant information from the past, and adding relevant information from the current time step. At the last output layer, the filtered version of of the previous hidden state and input vectors ot is then combined with tanh applied to the cell state ot ? tanh(ct). The forget gate resolves the problem of vanishing gradient, which is the case when values of the gradient vector are close to zero. SGD optimization algorithm is straight forward to implement. See Section A for discussion.

Deep rectified linear units (ReLU) with long short term memory (LSTM) cells have be- came popular architectures as they can capture long-range dependencies and nonlinear- ities. They can efficiently approximate highly multivariate functions with small number of neurons at each layer (Bach, 2017; Schmidt-Hieber, 2017; Yarotsky, 2017).

4 Deep Learning Extreme Value Theory (DL-EVT)

A traditional deep learning regression model uses least squared loss to estimate model parameters (weights and biases of each of the neural network layers). This model is not appropriate for quantifying large values of y (spikes) that are a rare but very crucial to the stable operations of electric grids. Extreme value theory approach allows to model the tail behavior of the distribution of electricity loads.

Let each observation follows a common distribution yi ∼ G(yi) = Pr(Y ≤ yi) and let Mn = max{y1, . . . , yn}. The central result of the extreme value theory is that regardless

9

of the distribution G, the scaled value of Mn follows a limiting distribution K

Pr {

Mn − bn an

≤ y }

= Gn(any + bn) → K(y)

Here an > 0 and bn are normalizing constants. Gnedenko (1943) provided a rigorous mathematical proof of existence of this limiting distribution and characterized its func- tional form (Davison and Smith, 1990). Modeling the extreme values Mn limits the num- ber of samples that can be used for estimation. For example, we can use monthly max- imum loads but then we will have to discard most of the samples. Further, traditional extreme value theory does not allow for covariates (predictor inputs). It makes this ap- proach impractical. A more practical approach was suggested by Smith (1989) who pro- posed to model values of y that exceed some fixed threshold value u. The distribution over the excess values has a limiting distribution as u + y approaches the right-hand end- point of the underlying distribution. Specifically, as u + y approaches the right-hand side endpoint of distribution G, for some normalizing constant cn > 0, we have

Pr{Y ≤ cu(u + y) | Y > u}→ H(y)

where

H(y | σ, ξ) = 1 − (

1 + ξ y − u

σ

)−1/ξ +

, ξ 6= 0.

Distribution H(y) is called the generalized Pareto (GP) distribution. The corresponding density function is given by

h(y | σ, ξ) = 1 σ

( 1 + ξ

y − u σ

)−1/ξ−1 , 1 + ξ

y − u σ

> 0, ξ 6= 0

Here (u, σ, ξ) are the location, scale and shape parameters, σ > 0 and z+ = max(z, 0). The Exponential distribution is obtained by continuity as ξ → 0, and we have

lim ξ→0

h(y | σ, ξ) = σ exp (−σ(y − u))

Under this distribution, the mean value of the y is σ + u. We fit the GP distribution using observations that exceed the threshold. Compared

to the the the classical extreme value theory that only models maximum values, each ex- ceedance is associated with a specific time point and it allows us to incorporate covariates, e.g. when parameters σ and ξ depend on input variable x.

Suppose, that we have data denoted by y(si, tj) at spatial locations si, 1 ≤ i ≤ n and time tj, 1 ≤ j ≤ T and. We build an EVD-DL input-output model for each location si and estimate it using the pairs {(y(si, tk), xk)}k∈C, where C = {j | Y(si, tj) > u} and xk = (y(si, tk), y(si, tk−1), . . . , y(si, tk−h)) are the recent observations of the output for the given location.

We assume that observations follow GP distribution with parameters being functions

10

of the input variables y(si, tk) | xk ∼ GP(σ(xk), ξ(xk))

We model the functions σ(x | W, b) and ξ(x | W, b) using a deep learning model parametrized by weight matrices W = (W1, . . . , WL+1) and biases b = (b1, . . . , bL+1). Linear regression GP were developed in (Davison and Smith, 1990; Beirlant et al., 2006). To complete our specification for exceedance sizes we assume a functional form for σ(x | W, b) that is a deep neural network. As shown in Equation 1, we introduce

(ξ(x), σ(x)) = F (x | W, b) , where F = fl ◦ . . . ◦ f L, fl(z) = σ(Wl z + bl)

Here F is a deep learner constructed via superposition of semi-affine univariate functions, see Polson and Sokolov (2017a, 2015); Dixon et al. (2019) for further discussion.

To estimate the weights and bias parameters of the deep leanring model, we use the the negative log-likelihood loss function. Under, the the assumption of GP distribution for our dependent variable, for a single observation, the negative log-likelihood is given by

l(yi | xi, W, b) = log σ(xi | W, b) − (1/ξ(xi | W, b) + 1) log (1 + σ(xi | W, b)ξ((xi | W, b))(yi − u)) .

Then, the loss function for our deep learning model model, which is the negative log- likelihood for a training data sample, becomes

L(W, b) = 1 |C| ∑i∈C

l(yi | xi, W, b).

The weights W and offsets b are learned by minimizing the loss function, using stochastic gradient descent (SGD) algorithm.

5 Empirical Results

In this section we compare temporal neural network architecture with more traditional Fourier and ARIMA models to predict electricity prices. Further, we demonstrate our DL-EVT approach to predict peak loads on the PJM interconnect.

5.1 PJM Price Forecasting

We predict the price at each load point using the historical prices, demand and weather observations. There are a total of 4719 generating nodes in the system. Plot 5 shows that there are strong spatial correlation among prices at different zones. Zone aggregates several nodes. Thus, prices at nodes will be correlated as well. We use this spatial pattern to build our model.

11

E K

P C

A E

P

D A

Y

D E

O K

C O

M E

D

D O

M

P E

P C

O

B G

E

P JM

− R

T O

A T

S I

P E

N E

L E

C

D U

Q

A P

S

M E

T E

D

M ID

− A T

L /A

P S

D P

L

R E

C O

P P

L

P E

C O

A E

C O

JC P

L

P S

E G

EKPC

AEP

DAY

DEOK

COMED

DOM

PEPCO

BGE

PJM−RTO

ATSI

PENELEC

DUQ

APS

METED

MID−ATL/APS

DPL

RECO

PPL

PECO

AECO

JCPL

PSEG

−1 −0.5 0 0.5 1

Value

0 1

0 2

0 3

0 4

0

Color Key and Histogram

C o

u n

t

Figure 5: Correlation in Marginal Prices Among Zones

Further, Figure 6 shows the relations between price and weather as well as price and the demand. We see that demand is not alway an accurate predictor for the price. This is due to the nonlinearities present in the system. High demand not necessarily leads to high prices. We only observe demand variable for the overall system and not on individual nodes. Thus, we cannot use spatial patterns in the demand variable to predict the prices.

0 20 40 60 80

2 0

3 0

4 0

5 0

6 0

Time [h]

P ri

ce [

$ ]

0 20 40 60 80

2 0

2 2

2 4

2 6

2 8

3 0

3 2

Price

Weather

0 50 100 150 200

2 5

0 0

0 3

0 0

0 0

3 5

0 0

0 4

0 0

0 0

4 5

0 0

0 5

0 0

0 0

Time [h]

D e

m a

n d

0 50 100 150 200

1 0

2 0

3 0

4 0

5 0

6 0

Demand

Price

(a) (b)

Figure 6: (a) Price vs Temperature [C]. (b) Demand vs Price

We demonstrate our forecasting approach for a node named ”KULLERRD138 KV T- 2”, with id 48667, which is locate in Clifton, NJ.

First, we try traditional model for electricity prices, which uses Fourier series to de- scribe the seasonal patterns and short-term time series dynamics modeled as an ARIMA terms. Here yt is decomposed as a sum, a deterministic Fourier term f (t), and a stochastic component, Nt, leading to

yt = a + f (t) + Nt, where f (t) = K

∑ k=1

[αk sin(2πkt/m) + βk cos(2πkt/m)] ,

12

where Nt is an ARIMA process. The number of terms of K can be chosen by minimizing cross-validation. This allows: (i) any length seasonality, (ii) several seasonality periods. Smoothness of the seasonal term is governed by value K. The short-term dynamics is han- dled with an ARMA error. The only real disadvantage (compared to a seasonal ARIMA model) is that the seasonality is assumed to be fixed the pattern is not allowed to change over time. In practice, seasonality is usually remarkably constant, so assumption gener- ally holds except in applications with very long time series.

The in-sample fit of the Fourier model is shown on Figure 9.

Time

yt

0 5 10 15 20 25 30

20 40

60 80

10 0

Figure 7: In-sample prediction

This model captures the cyclical patterns in the prices but does not accurately capture the levels of the peak prices.

Figure 8 shows the out-of-sample prediction for the 32 hours of price observations for Fourier model with weather and demand predictors. Inclusion of predictors does not change the quality of forecasting peak prices. As we noted in our exploratory plots, demand is not a good predictor of a peak price.

13

ARIMA + Fourier + Predictors

30.0 30.5 31.0 31.5 32.0

0 1

0 2

0 3

0 4

0

ARIMA + Fourier + Weather

30.0 30.5 31.0 31.5 32.0

0 1

0 2

0 3

0 4

0 5

0

(a) Demand + Weather (b) Only weather

Figure 8: Out-of-sample Prediction from linear model with ARIMA(2,0,0) errors and Fourier predictors. Yellow is data and blue is the forecast with confidence intervals.

We compare the Furies model with temporal neural network (LSTM) model. Table 2 shows several out-of-sample goodness-of fit metrics.

mse mrse mae mape ARIMA + Fourier 26.6 5.1 4 0.19

LSTM 16.8 4.1 2.4 0.09

Table 2: Out-of-sample performance of DL and Fourier models

0 10 20 30 40

0 1

0 2

0 3

0 4

0 5

0 6

0

Time [h]

P ri

ce [

$ ]

ARIMA Data ARIMA Conf. Interval LSTM

Figure 9: Comparison of Fourier and DL models

LSTM model shows an improved performance when compared to traditional ARIMA model with Fourier predictors. However, as shown in Figure 9 both, the traditional ARIMA and LSTM neural network model are not able to capture the peak in the price value at 5pm. Further, the peak price lies outside of the 95% confidence interval of our

14

ARIMA model. On the other hand, prediction of the peak values is of high importance. In the next section we show how EVT combined with DL (DL-EVT) addresses this problem and captures the peak values of the demand time series.

5.2 Demand Forecasting

Electricity load forecasting is essential for designing operational strategies for electric grids. In presence of renewable energy sources short-term forecasts are becoming increas- ingly important. Many decisions, such as dispatch scheduling and demand management strategies are based on load forecasts Taylor and McSharry (2007). One hour-ahead fore- casts are a key input for transmission companies on a self-dispatching markets Garcia and Kirschen (2006). Hourly behavior of electricity load is known to be non-stationary Alme- shaiei and Soltan (2011). Since there is not much of a change in meteorological variables, it is typical to use univariate time series data for short term load forecasting Bunn (1982).

In this section, we analyze an hourly electricity load observations on the PJM inter- connection. The data is available at https://www.dropbox.com/sh/1dczb673bx9kxzl/ AABII5ePMWdFhAk-dEcRGS1La?dl=0 We use data for January 2016 - May 2018 period year of observations and use last 10 days of observations for testing. We use a local trend model that takes previous 24 observations (one day) to predict load in five hours.

(a) Jan-Feb 2016 (b) Jan 2016 - Apr 2018

Figure 10: Hourly electricity load on PJM interconnect in MW.

Figure 10(b) shows the hourly PJM interconnect load series from January 2016 to April 17, 2018. This data was used to train our DL-EVT model. The graph shows seasonal cycles. Figure 10(a) shows the shorter period (Jan-Feb 2016) of the same data and shows daily and weekly cycles. We can see that weekends have lower load levels compared to work days.

The following architecture is used to model the relations between previous load ob- servations (X) and the scale parameter of the Generalize Pareto distribution σ.

15

3/26/19, 23)48dl-evt.xml

Page 1 of 1about:blank

x z(1) exp(tanh( + ))W2 z(1) b2 z(2)1tanh( x + )W1 b1 z(2)2

σ(x) tanh( )z(2)2 ξ(x)

where W1 ∈ Rp×3 and W2 ∈ R2×3, and x ∈ Rp is the vector of recent observations of electricity demand, we used p = 24 (1 day). We use tanh to constrain values of ξ to be in the (-1,1) interval. We require ξ < 1 to guarantee that we have a finite mean, which we use in our prediction. Further, we require ξ > −1 to guarantee that the likelihood function is bounded.

To train the EVT model we only used the observations yi > u with u = 31000. We used the mean σ/(1 − ξ) + u of the GP distribution as the point estimate for plotting Figure 11(b).

Our DL-EVT model is compared with a vanilla deep learning model with standard mean squared error loss function. Figure 11 shows the resulting out-of-sample forecasts.

(a) DL (MSE Loss) (b) DL-EVT (GP Loss)

Figure 11: One our electricity load and its forecast for the period from Friday, April 27, 2018 to Monday, May 7, 2018

We can see that while a standard DL model captures both ups and downs in the load levels, the DL-EVT model does capture the location and level of the peak loads more accurately compared to the standard DL model.

6 Discussion

Deep learning, combined with extreme value theory, can predict peaks in electricity prices and demand. With the availability of real-time data, computational power, and machine learning pattern recognition tools, such as deep learning, we have the ability to more accurately predict and manage energy generation and distribution. One of our goals is to demonstrate that an EVT extension of the standard DL framework is a viable option and is applicable to electricity data. DL-EVT performed well on in- and out-of-sample forecasting of electricity prices and load.

16

We demonstrated our DL-EVT model is more accurate at forecasting peak values that exceed a given threshold when compared to a Gaussian likelihood-based model. The EVT model predicts peak values conditional on the exceedance over the threshold. One of the artifacts of our model is that prediction for time points when a threshold is not expected to be exceeded is a constant value. An extension of our approach could include a binary classifier that predicts the probability of crossing a specific threshold. Another possible extension is to include a Gaussian likelihood-based model to forecast values be- low a threshold. Naveau et al. (2016) demonstrate that a similar approach can be used for successful environmental modeling.

Forecasting electricity prices is challenging because they can spike due to supply- demand imbalances, yet have long-range dependence. Deep ReLU LSTM models capture spikes with non-linear activation functions, are scalable, and can efficiently fit using SGD. For a grid of 4,786 electricity load nodes, we show how such models can fit in-sample with better accuracy than traditional time series models. There are a number for directions of future research. For extensions to multivariate time series data with spatiotemporal dy- namics, see Dixon et al. (2019).

References

Martı́n Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, and Michael Isard. Tensorflow: A system for large-scale machine learning. In OSDI, volume 16, pages 265–283, 2016.

Eisa Almeshaiei and Hassan Soltan. A methodology for electric power load forecasting. Alexandria Engineering Journal, 50(2):137–144, 2011.

tiladimir I Arnold. On functions of three variables. Dokl. Akad. Nauk SSSR, 14(4):679–681, 1963. [ English translation: American Mathematical Society Translation, 2(28) (1963), pp. 51-54].

Francis Bach. Breaking the curse of dimensionality with convex neural networks. Journal of Machine Learning Research, 18(19):1–53, 2017.

Jan Beirlant, Yuri Goegebeur, Johan Segers, and Jozef L Teugels. Statistics of extremes: theory and applications. John Wiley & Sons, 2006.

Fred Espen Benth and Maren Diane Schmeck. Pricing futures and options in electricity markets. In The Interrelationship Between Financial and Energy Markets, pages 233–260. Springer, 2014.

Derek W Bunn. Short-term forecasting: a review of procedures in the electricity supply industry. Journal of the Operational Research Society, 33(6):533–545, 1982.

Collin Cain, Jonathan Lesser, and Bates White. A common sense guide to wholesale elec- tric markets. Technical report, Bates White, Tech. Rep., Apr. 2007.[Online]. Available: http://www. bateswhite. com/media/publication/55 media. 741. pdf, 2007.

17

Pedro Emanuel Almeida Cardoso. Deep learning applied to PMU data in power systems, 2017.

João Paulo da Silva Catalão, Sı́lvio José Pinto Simões Mariano, tiMF Mendes, and LAFM Ferreira. Short-term electricity prices forecasting in a competitive market: A neural network approach. Electric Power Systems Research, 77(10):1297–1304, 2007.

Tim M Christensen, A Stan Hurn, and Kenneth A Lindsay. Forecasting spikes in electricity prices. International Journal of Forecasting, 28(2):400–411, 2012.

Remy Cottet and Michael Smith. Bayesian modeling and forecasting of intraday electric- ity load. Journal of the American Statistical Association, 98(464):839–849, 2003.

Anthony C Davison and Richard L Smith. Models for exceedances over high thresholds. Journal of the Royal Statistical Society. Series B (Methodological), pages 393–442, 1990.

Anthony C Davison, Simone A Padoan, and Mathieu Ribatet. Statistical modeling of spatial extremes. Statistical science, 27(2):161–186, 2012.

Matthew F. Dixon, Nicholas G. Polson, and Vadim O. Sokolov. Deep learning for spatio- temporal modeling: Dynamic traffic flows and high frequency trading. Applied Stochas- tic Models in Business and Industry, 0(0), 2019.

Debbie J Dupuis. Electricity price dependence in New York state zones: A robust de- trended correlation approach. The Annals of Applied Statistics, 11(1):248–273, 2017.

Federal Energy Regulatory Commission FERC. Operator-initiated commitments in rto and iso markets. FERC Docket No, pages D14–14, 2014.

Maria P Garcia and Daniel S Kirschen. Forecasting system imbalance volumes in compet- itive electricity markets. IEEE Transactions on Power Systems, 21(1):240–248, 2006.

Reinaldo C Garcia, Javier Contreras, Marco van Akkeren, and João Batista C Garcia. A garch forecasting model to predict day-ahead electricity prices. IEEE transactions on power systems, 20(2):867–874, 2005.

Boris Gnedenko. Sur la distribution limite du terme maximum d’une serie aleatoire. An- nals of mathematics, pages 423–453, 1943.

Amanda S Hering and Marc G Genton. Powering up with space-time wind forecasting. Journal of the American Statistical Association, 105(489):92–104, 2010.

Sawsan Hilal, Ser-Huang Poon, and Jonathan Tawn. Hedging the black swan: Condi- tional heteroskedasticity and tail dependence in s&p500 and tiix. Journal of Banking & Finance, 35(9):2374–2387, 2011.

G. E. Hinton and R. R. Salakhutdinov:. Reducing the dimensionality of data with neural networks. Science, 313(5786):504–507, 2006.

18

Sepp Hochreiter and Jürgen Schmidhuber. LSTM can solve hard long time lag problems. In Advances in neural information processing systems, pages 473–479, 1997.

Robert V Hogg and Stuart A Klugman. On the estimation of long tailed skewed distribu- tions with actuarial applications. Journal of Econometrics, 23(1):91–102, 1983.

Y-Y Hong and C-Y Hsiao. Locational marginal price forecasting in deregulated electricity markets using artificial intelligence. IEE Proceedings-Generation, Transmission and Distri- bution, 149(5):621–626, 2002.

Ying Yi Hong and Chuan-Yo Hsiao. Locational marginal price forecasting in deregulated electric markets using a recurrent neural network. In Power Engineering Society Winter Meeting, 2001. IEEE, volume 2, pages 539–544. IEEE, 2001.

Katarzyna Janocha and Wojciech Marian Czarnecki. On loss functions for deep neural networks in classification. arXiv:1702.05659, 2017.

Tryggvi Jónsson, Pierre Pinson, Henrik Aalborg Nielsen, Henrik Madsen, and Tor- ben Skov Nielsen. Forecasting electricity spot prices accounting for wind power pre- dictions. IEEE Transactions on Sustainable Energy, 4(1):210–218, 2013.

Mun-Kyeom Kim. A new approach to short-term price forecasting strategy with an arti- ficial neural network approach: Application to the nord pool. Journal of Electric Engi- neering and Technology, 10:709–718, 2015.

Andrei Nikolaevich Kolmogorov. On the representation of continuous functions of sev- eral variables by superpositions of continuous functions of a smaller number of vari- ables. Dokl. Akad. Nauk SSSR, 108:179–182, 1956. [ English translation: American Mathe- matical Society Translation, 17 (1961), pp. 369-373].

Andrei Nikolaevich Kolmogorov. On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition. Dokl. Akad. Nauk SSSR, 114(5):953–956, 1957. [English translation: American Mathematical Society Translation, 28 (2) (1963), pp. 55–59].

Guang Li, Chen-Ching Liu, Chris Mattson, and Jacques Lawarrée. Day-ahead electricity price forecasting in a grid environment. IEEE Transactions on Power Systems, 22(1):266– 274, 2007.

Dominik Liebl. Modeling and forecasting electricity spot prices: A functional data per- spective. The Annals of Applied Statistics, 7(3):1562–1592, 2013.

P. Mandal, T. Senjyu, N. Urasaki, T. Funabashi, and A. K. Srivastava. A novel approach to froecast electricity price for PJM using neural network and similar days method. IEEE Transactions on Power Systems, 22:4, 2007.

Philippe Naveau, Raphael Huser, Pierre Ribereau, and Alexis Hannart. Modeling jointly low, moderate, and heavy rainfall intensities without a threshold selection. Water Re- sources Research, 52(4):2753–2769, 2016.

19

PJM. Daily real-time lmp: August 1, 2017 through September 7, 2017. URL http://www. pjm.com/markets-and-operations/energy/real-time/lmp.aspx.

Nicholas Polson and Vadim Sokolov. Bayesian analysis of traffic flow on interstate I-55: The LWR model. The Annals of Applied Statistics, 9(4):1864–1888, 2015.

Nicholas G Polson and Vadim Sokolov. Deep Learning: A Bayesian Perspective. Bayesian Analysis, 12(4):1275–1304, 2017a.

Nicholas G. Polson and Vadim O. Sokolov. Deep learning for short-term traffic flow pre- diction. Transportation Research Part C: Emerging Technologies, 79:1–17, June 2017b. ISSN 0968-090X. doi: 10.1016/j.trc.2017.02.024.

Ser-Huang Poon, Michael Rockinger, and Jonathan Tawn. Extreme value dependence in financial markets: Diagnostics, models, and financial implications. The Review of Financial Studies, 17(2):581–610, 2003.

Jürgen Schmidhuber and Sepp Hochreiter. Long Short-Term Memory. Neural Comput, 9 (8):1735–1780, 1997b.

Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with relu activation function. arXiv:1708.06633, 2017.

Saahil Shenoy and Dimitry Gorinevsky. Risk adjusted forecasting of electric power load. In American Control Conference (ACC), 2014, pages 914–919. IEEE, 2014.

Caston Sigauke, Andréhette Verster, and Delson Chikobvu. Extreme daily increases in peak electricity demand: Tail-quantile estimation. Energy Policy, 53:90–96, 2013.

Richard L Smith. Extreme value analysis of environmental time series: an application to trend detection in ground-level ozone. Statistical Science, 4(4):367–377, 1989.

Richard L Smith. Measuring risk with extreme value theory. Risk management: value at risk and beyond, 224, 2002.

N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929–1958, 2014.

Sankaranarayanan Subbayya, Jorjeta G Jetcheva, and Wei-Peng Chen. Model selection criteria for short-term microgrid-scale electricity load forecasts. In Innovative Smart Grid Technologies (ISGT), 2013 IEEE PES, pages 1–6. IEEE, 2013.

James W Taylor and Patrick E McSharry. Short-term load forecasting methods: An eval- uation based on european data. IEEE Transactions on Power Systems, 22(4):2213–2219, 2007.

Pravin P tiaraiya, Felix F Wu, and Janusz W Bialek. Smart operation of smart grid: Risk- limiting dispatch. Proceedings of the IEEE, 99(1):40–57, 2011.

20

Long Wang, Zijun Zhang, and Jieqiu Chen. Short-term electricity price forecasting with stacked denoising autoencoders. IEEE Transactions on Power Systems, 32(4):2673–2681, 2017.

AL Wilson, CJ Dent, and M Goldstein. Quantifying uncertainty in wholesale electricity price projections using bayesian emulation of a generation investment model. Sustain- able Energy, Grids and Networks, 13:42–55, 2018.

Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94:103–114, 2017.

A Stochastic Gradient Descent for Deep Learning (SGD)

Once the activation functions, depth L and size n1, . . . , nL of the learner have been chosen, the parameters, Ŵ and b̂ are found by solving the following optimization problem

minimizeW,b T

∑ i=1

L(yi, ŷ(xi | W, b)) + φ(W), (2)

Which is a penalized loss function, where (yi, xi)Ni=1 is training data of input-output pairs, and φ(W) is a regularization penalty on the network weights. Most architectures em- ploy regularization techniques to prevent the model from over-fitting training set data (Hinton and Salakhutdinov:, 2006). This improves the model’s predictive performance on data outside of the training set. Normally, a regularization penalty allows to improve convergence rate of the optimization algorithms and to avoid over-fitting. Dropout, the technique of removing input dimensions in x randomly with probability p, can also be used to further reduce the change of over-fitting during the training process (Srivastava et al., 2014).

A typical choice is L(yi, Ŷ(xi)) being an L2-norm, then we have a traditional least squares problem (Janocha and Czarnecki, 2017) and φ(W) = λ||W||2.

The common numerical approach to find the solution to this optimizaiton probelm 2 is stochastic gradient descent (SGD). It iteratively updates the current iterated by taking a step in the direction opposite to the gradient vector

(W, b)+ = (W, b)− η∇(L(yi, ŷ(xi | W, b)) + φ(W))

SGD then uses back-propagation algorithm to calculate the gradient at each iteration. Back-propagation is an implementaiton of chain rule applied to a function defined by a neural network. One caveat of SGD is the complexity of the system to be optimized, resulting in slow convergence rates. As a result, deep learning methods rely heavily on large computational power (Abadi et al., 2016; Cardoso, 2017).

21