Review on Energy Resilience

2498 IEEE TRANSACTIONS ON SMART GRID, VOL. 11, NO. 3, MAY 2020

Markov Decision Process-Based Resilience Enhancement for Distribution Systems: An

Approximate Dynamic Programming Approach Chong Wang , Member, IEEE, Ping Ju , Senior Member, IEEE, Shunbo Lei , Member, IEEE,

Zhaoyu Wang , Member, IEEE, Feng Wu, Member, IEEE, and Yunhe Hou , Senior Member, IEEE

Abstract—Because failures in distribution systems caused by extreme weather events directly result in consumers’ outages, this paper proposes a state-based decision-making model with the objective of mitigating loss of load to improve the distri- bution system resilience throughout the unfolding events. The system topologies including on/off states of feeder lines are mod- eled as Markov states, and the probabilities from one Markov state to another Markov state throughout the unfolding events are determined by the component failure caused by the unfolding events. A recursive optimization model based on Markov deci- sion processes (MDP) is developed to make state-based actions, i.e., system reconfiguration, at each decision time. To overcome the curse of dimensionality caused by enormous states and actions, an approximate dynamic programming (ADP) approach based on post-decision states and iteration is used to solve the proposed MDP-based model. IEEE 33-bus system and IEEE 123-bus system are used to validate the proposed model.

Index Terms—Approximate dynamic programming, distribution systems, Markov decision processes, resilience enhancement.

NOMENCLATURE

Indices and Sets

c Index of components. l Index of lines. k, k′ Index of terminal buses of line l. t, τ Index of time periods. i, i′, j, j′ Index of states.

Manuscript received March 23, 2019; revised July 11, 2019 and September 11, 2019; accepted November 8, 2019. Date of publication December 2, 2019; date of current version April 21, 2020. This work was supported in part by the National Natural Science Foundation of China under Grant 51907050, in part by the Fundamental Research Funds for the Central Universities under Grant 2018B05514, in part by the “111” Project of Renewable Energy and Smart Grid under Grant B14022, and in part by the Research Grant Council, Hong Kong, under Grant GRF 17207818. Paper no. TSG-00430-2019. (Corresponding author: Ping Ju.)

C. Wang, P. Ju, and F. Wu are with the College of Energy and Electrical Engineering, Hohai University, Nanjing 211100, China (e-mail: [email protected]; [email protected]; [email protected]).

S. Lei is with the Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI 48109 USA (e-mail: [email protected]).

Z. Wang is with the Department of Electrical and Computer Engineering, Iowa State University, Ames, IA 50011 USA (e-mail: [email protected]).

Y. Hou is with the Department of Electrical and Electronic Engineering, University of Hong Kong, Hong Kong (e-mail: [email protected]).

Color versions of one or more of the figures in this article are available online at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TSG.2019.2956740

A Set of actions. B Set of buses. B̄i,t Set of non-islanded buses under the state

Si,t. B̃ Set of substation nodes. Cfi,t Set of repaired components under state Si,t. Ft Set of all possible failure components at t. F̃t Set of actual failure components at t. Li,t Set of non-islanded lines under state Si,t. Li′,t Set of dispatchable lines under post-

decision state Sati′,t. Ldi,t Set of dispatchable lines under state Si,t. Lndi,t Set of non-dispatchable lines under state

Si,t. Nk,i,t Set of nodes connected to bus k under state

Si,t. R̃τ Set of repaired components at τ . S Set of Markov states. Sposti Set of post-decision states of state Si,t. T Set of time periods.

Notation for Solution Method

bi Binary-coded matrix for post-decision states.

E Expected value. m Number of dispatchable lines. n Number of iterations. N Maximum number of iterations. Sati,t Post-decision after Si,t with action at. S

at−1 j′,t−1 Post-decision after Sj′,t−1 with action at−1.

T Number of decision periods. vatt Value function of post-decision state. vnt Value function of state at n

th iteration at t. ṽat ,nt , ṽ

at−1,n t−1 Approximated value function of post-

decision state at nth iteration. Vi′ Known values of post-decision states. x1, x2 Binary variables. yl,i′,t Binary variable. � A coefficient.

Notation for Markov Decision Process-Based Model

at Action at t. Ct Immediate cost at t.

1949-3053 c© 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

Authorized licensed use limited to: Penn State University. Downloaded on August 11,2020 at 22:55:01 UTC from IEEE Xplore. Restrictions apply.

WANG et al.: MDP-BASED RESILIENCE ENHANCEMENT FOR DISTRIBUTION SYSTEMS: ADP APPROACH 2499

Cl Operational cost of line l at t. F

p kk′,i,t Active power flow on line k − k′ under Si,t

at t. F

q kk′,i,t Reactive power flow on line k −k′ under Si,t

at t. Fskk′ Apparent power capacity of line k − k′. �L

p k,i,t, �L

q k,i,t Loss of active/reactive load of bus k under

state Si,t. L

p k,t Active load of bus k at t.

L q k,t Reactive load of bus k at t.

M Large positive number. okk′,i,t Binary variable, the value is 1 if bus k

′ is the parent bus for bus k under state Si,t, otherwise 0.

rkk′ /xkk′ Resistance/reactance of line k − k′. Si,t, Sj,t+1 Markov state i and j at t and t + 1,

respectively. sc,t On-off state of component c at t. �T Duration of each time period. T

f c Time period from normal state to failure

state of component c. �T rc Repair duration for component c. Uk,i,t Squared voltage magnitude of bus k. V k, V k Low/upper limits of voltage value of bus k. vt, vt+1 values functions at t and t + 1. Pr Transition probability. βl,i,t, βc,i,t Binary variables representing on-off states

of line l and c, respectively. 1 denotes on state, and 0 denotes off state.

ηt, ηt−1 Penalty due to loss of load at t and t − 1 ($/MWh).

ξt Uncertainty of extreme event at t.

I. INTRODUCTION

B ECAUSE distribution systems are directly connectedto commercial and residential customers with radial topologies, any failures in distribution systems will lead to outages. Climate change increases the frequency and inten- sity of severe weather, which is a major cause of severe system failures. For example, weather events caused roughly 679 power outages, each of which affected at least 50000 customers, between 2003 and 2012. Although transmission system outages did occur, a major portion of outages occurred along distribution systems [1]. The severe consequences have required distribution systems to have resilience against these extreme weather events, and this has been identified by the United States Electric Power Research Institute (EPRI) [2] and the North American Electric Reliability Corporation (NERC) [3].

A power system may reside in different stages when it is imposed to natural disasters or extreme weather events. It is necessary to define these stages to enable systematic enhance- ment power systems against to these events. Reference [4] analyzes the notion of resilience in power systems from a fundamental viewpoint and thoroughly examines its practical implications. Usually, three critical stages, i.e., prior to events, during events, and after events, need to be included [5], [6].

Different stages have different requirements. Prior to events, it requires the ability to keep operating or to stay standing in the face of disasters. During events, it requires to manage the dis- asters as they unfolds. It includes identifying options and prior- itizing what should be done to mitigate damages. After events, it requires to get the power systems back to normal states as quickly as possible after a disaster. To enhance the system resilience, different strategies can be done in three stages. Prior to weather events, the historical data-based models [7]–[9] are used to estimate outages that help the system operators to make preventive actions such as system maintenance [10] and system hardening [11]. System hardening makes physical infrastructural changes to systems so that they are less suscep- tible to extreme events. For example, a coordinated hardening and distributed generator (DG) allocation strategy has been developed in [11]. An analytical method is proposed in [12] to offer a quick way for getting knowledge about adverse impacts of an approaching windstorm and taking preven- tive measures accordingly. References [13] and [14] propose proactive scheduling for resilience enhancement of microgrids (MGs) [15]–[18] ahead of extreme windstorms and floods, respectively. Prior to events, preparing enough blackstart gen- erators and emergency generators after potential failures is also a critical measure to improve the system resilience. To this end, a Generic Restoration Milestones (GRMs)-based algorithm is developed to assess blackstart capacities [19], and a procure- ment plan with a minimal cost while guaranteeing sufficient blackstart capacities is proposed to provide enough blackstart resources at right locations [20]. To effectively isolate pos- sible failures and connect blackstart/emergency generators to systems, a resilience-based model for switch placement in dis- tribution systems is developed prior to events [21]. In addition to physical power systems, hardening communication systems in charge of monitoring/controlling the physical power systems play an important role in enhancing the system resilience [22]. Even though many preventive actions are performed prior to events, it is impossible to avoid outages completely. When out- ages occur after events, it is necessary to recover outages as quickly as possible to improve the system resilience. A con- ventional power system restoration includes three stages, i.e., preparation, system restoration and load restoration [23]–[25]. Some algorithms such as expert systems [26] and heuris- tic approaches [27] are proposed to accelerate load recov- ery. However, there are unique characteristics associated with outages caused by weather-related events, leading to different restoration strategies such as microgrid-based restoration strategies [28] and decentralized restoration schemes [29].

The above studies mainly focus on strategies prior to events and after events. There are also some studies on operational strategies during events. A concept of operational resilience is proposed in [30], but the detailed models are not estab- lished. In [31], generator re-dispatch, topology switching and load shedding are considered as emergency responses, which are modeled as a two-stage robust mixed-integer optimization model. In consideration of high controllability of microgrids, [32] presents a two-stage stochastic program- ing approach to obtain the optimal scheduling of a resilient

Authorized licensed use limited to: Penn State University. Downloaded on August 11,2020 at 22:55:01 UTC from IEEE Xplore. Restrictions apply.

2500 IEEE TRANSACTIONS ON SMART GRID, VOL. 11, NO. 3, MAY 2020

Fig. 1. Markov Decision Processes.

microgrid, and a robust optimization model with survivabil- ity of critical loads during the emergency period is proposed. With the development of integrated natural gas and power grids, [33] establishes a two-stage robust model to accom- modate random outages caused by natural disasters in both natural gas and power systems. A stochastic programming approach for increasing resilience of a distribution system exposed to wildfires is investigated in [34]. The uncertainties associated with wind speeds and wind directions that affect the progression of the wildfire are represented by simulated scenarios. Even though the above studies have focused on the strategies during the unfolding events, few studies investigate state-based strategies, which mean that the strategies should be made based on observed states during the unfolding events. One difficulty in generating the strategies during the unfold- ing events is to map sequentially varying states caused by the unfolding events to the optimal strategies. The commonly used scenario-based stochastic programming [32], [34] and the robust stochastic programming [33] are not suitable for map- ping sequentially real-time varying states to optimal strategies. To address this difficulty, Markov decision processes (MDP) can be employed to make state-based decisions on a stochastic environment caused by weather events. Some applications of MDP in power systems have been investigated [35], [36]. For the resilience enhancement, [37] proposes sequentially proac- tive MDP-based strategies to improve the transmission system resilience, and a linear scalarization method based on the state tree is used to solve the proposed model. However, distri- bution systems and transmission systems differ in topologies and allowable actions, the developed model and the solution in [37] cannot be applied to distribution systems directly. To fill the gap, it is necessary to develop state-based decision- making models for distribution systems considering their own characteristics.

This paper proposes MDP-based resilience enhancement for distribution systems during the events. Because minimizing loss of load is considered in the established model, the strate- gies contribute to mitigate damages during the events. The con- tributions of this paper are three-fold: 1) The system topologies including on/off states of feeder lines are modeled as Markov states. Transition probabilities between Markov states are determined by component failure probabilities caused by the unfolding event. 2) A recursive optimization model for each Markov state is constructed to map states to optimal strategies. The allowable action for each state is system reconfiguration.

3) An approximate dynamic programming approach based on post-decision states and value function approximation is employed to solve the proposed model to deal with the curse of dimensionality caused by a mass of states and allowable actions.

The remainder of this paper is organized as fol- lows. Section II shows extreme events’ impacts on system states. Section III presents the mathematical formulation, and Section IV introduces the solution method. The case studies are demonstrated in Section V, and the work is concluded in Section VI.

II. MODELING INFLUENCES OF EVENTS ON DISTRIBUTION SYSTEMS

This section first introduces the concept of MDP, and then presents Markov states on the trajectories of extreme events, and finally shows transition probabilities between different Markov states under allowable actions.

A. Introduction of MDP

A Markov decision process is an extension of a Markov chain that includes an agent that makes decisions that affect the evolution of the system over time. MDP provides a mathe- matical framework for modeling real-time state-based decision making in situations where there are sequential uncertainties. Fig. 1 illustrates the processes of MDP. At t1, the decision maker observes the state S1, and performs the action A1, result- ing in a reward R1. With the action A1 and the uncertainties caused by environments, the state S1 reaches the state S2. At t2, the decision maker observes the state S2 and performs the action A2, resulting in a reward R2. Then, repeat this process for other decision periods. During this process, the decision maker expects the optimal action for each state considering the current cost and the future cost.

The goal of this research is to minimize loss of load in consideration of operational costs by means of system recon- figuration based on the observed states in consideration of the uncertainties caused by the extreme weather event. The cost of loss of load and the operational cost are considered as the reward at each decision time. A system topology is defined as a Markov state. System reconfiguration works as the action. The environment denotes the uncertain impacts of extreme weather events on power systems, and the uncertain impacts are repre- sented by transition probabilities. Different system topologies maybe change into the same system topology under different actions in consideration of the uncertain impacts. One system topology may also change into different topologies under the same actions in consideration of the uncertain impacts. This shows that a system topology is defined as a Markov state regardless of the action and the uncertainties that cause the feeder disconnection.

B. Markov States on the Event’s Trajectories

Usually, the impacts of a weather-related event on a distri- bution system are sequential due to the sequential trajectory, indicating that the components with different locations in the system may be in failure in different time periods. This results

Authorized licensed use limited to: Penn State University. Downloaded on August 11,2020 at 22:55:01 UTC from IEEE Xplore. Restrictions apply.

WANG et al.: MDP-BASED RESILIENCE ENHANCEMENT FOR DISTRIBUTION SYSTEMS: ADP APPROACH 2501

Fig. 2. An example of a distribution system under an unfolding event.

in the sequential changes of system states such as on-off states of distribution lines. The system topologies including on/off states of feeder lines are modeled as Markov states. Define Ht as the set of disconnected lines due to system configuration but not in failure caused by the unfolding event at t. Define Ft as the set of all possible failure components due to the unfolding event at t, and F̃t as the set of the actual failure components at t, and we have F̃t ⊆ Ft. Take the scenario in Fig. 2 as an example. Ft2 = {b1−2, b1−4} and F̃t2 = {b1−4}. The failure scenarios of the lines b1−2 and b1−4 are uncertain before the time period t2, and the actual state can only be observed at t2 and the actual failure on b1−4 occurs. Because the system topologies are modeled as the Markov states, we use the set of the disconnected lines of different system topologies to describe the different Markov states.

Si,t = Ht t⋃

τ =1

( F̃τ − R̃τ

) (1)

where (1) shows that the Markov state Si,t at t. R̃τ is the set of components repaired at time τ .

The extreme weather events only have probabilities to result in system outages, and we hope to implement system recon- figuration to minimize the potential system outages during the typhoon. However, proactive load shedding will directly and deterministically result in power outages for customers in dis- tribution systems. Therefore, proactive load shedding will not be implemented during system reconfiguration. It means that system reconfiguration is considered as the allowable action and load shedding is not considered as the allowable action. For the load level, it is based on the predicted load curve, which is assumed to be known in this paper.

C. Transition Probability Between Markov States

On the trajectory of the event, the current Markov state at t has a probability of reaching each future Markov state at t + 1, and the probability is called as a transition probability, which is determined by component failure rates caused by the event. The transition probability can be expressed as follows.

Pr ( Sj,t+1|Si,t, at, ξt

) = ∏

c∈Ft+1 Pr

( sc,t+1|sc,t, at, ξt

) (2)

where Pr(Sj,t+1|Si,t, at, ξt) means the probability from the state Si,t to the state Sj,t+1 under the action at with the uncertainty ξt, and Pr(sc,t+1|sc,t, at, ξt) represents the probability from the

cth component’s on-off state sc,t to the on-off state sc,t+1 under the action at with the uncertainty ξt. Three causes can change the on-off states, and they are listed as follows.

• Component failure caused by extreme events: Since the on-off states are uncertain in the next time period due to extreme events, it is necessary to calculate the probabil- ity of each scenario. At present, there are many existing studies on components’ failure probabilities caused by extreme events [38].

• System reconfiguration: After system reconfiguration, the line state, i.e., on state or off state, is determined. This indicates that the corresponding probability from sk,t to sk,t+1 is 0 or 1.

• Repair: Before the failure components are repaired, the state is off. After repaired, the state is on. This shows that the corresponding probability from sk,t to sk,t+1 is 0 or 1.

III. OPTIMIZATION MODEL BASED ON MARKOV DECISION PROCESSES

This section first introduces a recursive model to map each Markov state to its optimal strategy, and then lists the operational constraints for distribution systems.

A. Markov Decision Processes-Based Recursive Model

Transition probabilities in (2) show that the current actions associated with uncertainties caused by extreme events impact the current states and future states. Since different states result in different operational costs, it is necessary to make decisions based on not only current states but also future states impacted by transition probabilities. Take the scenario in Fig. 2 as an example. Since the line b1−2 will be impacted by the typhoon at t1, the line b1−2 may be in failure at t1, resulting in the outages of the lines b1−4 and b1−7. If we can disconnect the line b1−2 and connect the line b3−4 before the line b1−2 is impacted by the typhoon, we can avoid the outages of the lines b1−4 and b1−7 at t1. For this case, we only consider one time period ahead and there is only one line impacted by the typhoon at t1. If all time periods over the unfolding typhoon and numerous components impacted in each time period are considered, we need to develop a model to help make deci- sions to ensure the minimum operational cost. Considering the sequential time periods and numerous components on the tra- jectory, we establish a recursive model with the current cost and the expected future cost listed as follows.

vt ( Si,t

) = min at ∈A

( Ct

( Si,t, at

)

+ ∑Sj,t+1∈S Pr ( Sj,t+1|Si,t, at, ξt

) · vt+1 ( Sj,t+1

) )

(3)

where vt(Si,t) and vt+1(Sj,t+1) are the value functions of the states Si,t and Sj,t+1 at t and t + 1, respectively. The second term on the right side of (3) shows the expected future cost. Ct(Si,t, at) is the current cost caused by the action at under the state Si,t at t, and this cost in this study is defined as the sum of the cost of loss of load and the operational cost of

Authorized licensed use limited to: Penn State University. Downloaded on August 11,2020 at 22:55:01 UTC from IEEE Xplore. Restrictions apply.

2502 IEEE TRANSACTIONS ON SMART GRID, VOL. 11, NO. 3, MAY 2020

controllable lines. It is expressed as follows.

Ct ( Si,t, at

) = ∑

k∈B

( ηt · �Lpb,i,t · �T

) +

∑

l∈Ldi,t

( βl,i,t · Cl

) (4)

where the first term on the right side of (4) is the cost of loss of load, and the second term is the operational cost of controllable lines. The breaker will be triggered when a fault occurs in distribution systems because of the relay protection and the corresponding downstream feeders will be out of ser- vice. This is considered as loss of load. In the proposed model, the downstream feeders without faults will be reconnected to the system by means of system configuration.

B. Operational Constraints

For each time period during the unfolding event, the opera- tional constraints, i.e., radial topologies, power balance, power flow, voltage limits, and line capacity, should be satisfied.

1) Radiality Constraint: Different from transmission systems, distribution systems should operate in radial topolo- gies [39]. When performing system reconfiguration under the state Si,t, the spanning tree constraints are used to guarantee the network radiality.

okk′,i,t + ok′k,i,t = βl,i,t l ∈ Li,t, t ∈ T (5a)∑ k′∈Nk,i,t

okk′,i,t = 1 k ∈ B̄i,t, t ∈ T (5b)

okk′,i,t = 0 k ∈ B̃, k′ ∈ Nk,i,t, t ∈ T (5c) where (5a) and (5b) constrain that the two terminals of a con- nected line only have one parent bus. In practice, islanded buses, to which power cannot be supplied by the grid, maybe exist due to component failures caused by extreme events, and these islanded buses are not included in the spanning tree constraint. Equation (5c) indicates that the substation bus (i.e., the bus connected to the external system) has no parent buses. We assume that the dispatchable lines are remotely- controlled. Remotely-controlled dispatchable lines could be reconfigurated within a short time by utilizing more advanced distribution management systems [40], [41]. Therefore, the operational time of reconfiguration is not considered.

It is possible that only some lines can be dispatched under the state Si,t. In this case, we can add an constraint with regard to non-dispatched lines.

βl,i,t = 1 l ∈ Lndi,t , t ∈ T . (6) 2) Repair Constraint: Detailed repair scheduling after the

extreme weather event is a complicated optimization that needs to consider equipment/crew constraints, and there are many research studies on it [42], [43]. Because this paper mainly focuses on dispatch strategies during the extreme weather event and repair scheduling belongs to a task after the extreme weather event, the repair is simply constrained by the repair duration as follows.

βc,i,t = 0 c ∈ Cfi,t, T fc ≤ t ≤ T fc + �T rc (7) where �T rc represents the repair duration for the component c. Equation (7) means that the state of the failure component c is

set to 0 during the repair periods, and it is a generic constraint. Therefore, whether a failure line is repaired or not depends on the constraint of the corresponding repair duration.

3) Power Flow Constraint: The power flow of each line has relations to bus voltages of the two terminal buses of each line, and can be expressed as follows.

Uk,i,t − Uk′,i,t ≤ ( 1 − βl,i,t

) · M + 2 (

rkk′ · Fpkk′,i,t + xkk′ · F q kk′,i,t

)

l ∈ Li,t, t ∈ T (8a) Uk,i,t − Uk′,i,t ≥

( βl,i,t − 1

) · M + 2 (

rkk′ · Fpkk′,i,t + xkk′ · F q kk′,i,t

)

l ∈ Li,t, t ∈ T (8b) where (8a)-(8b) are derived from the DsitFlow model [44]. The quadratic terms in the accurate power flow model are ignored [45]. The big M is a disjunctive parameter. With a sufficiently large M, (8a)-(8b) are redundant when distribu- tion lines are disconnected or outages. Non-islanded buses are included in these constraints.

4) Power Balance Constraint: When reaching the state Si,t at t, the out-flow/in-flow power of each non-islanded bus in the system should be equal. The constraint can be expressed as follows.

L p k,t +

∑

k′∈Nk,i,t F

p kk′,i,t = 0 k ∈ B̄i,t, t ∈ T (9a)

L q k,t +

∑

k′∈Nk,i,t F

q kk′,i,t = 0 k ∈ B̄i,t, t ∈ T (9b)

where (9a) and (9b) represent real power balance and reac- tive power balance, respectively. Only non-islanded buses are included in the constraint. The load connected to the islanded buses in the system is directly considered as loss of load in (4).

5) Line Capacity Constraint: The power through each line should be within the limit for the state Si,t with the action at. The constraint can be expressed as follows. (

F p kk′,i,t

)2 +

( F

q kk′,i,t

)2 ≤ βl,i,t ·

( Fskk′

)2 l ∈ Li,t, t ∈ T (10)

where (10) is a nonlinear constraint, resulting in compu- tational intractability. To facilitate the model solution, the constraint (10) is relaxed to a group of linear constraints [46], and are rewritten as follows.

− βl,i,t · Fskk′ ≤ F p kk′,i,t ≤ βl,i,t · Fskk′

l ∈ Li,t, t ∈ T (11a) −βl,i,t · Fskk′ ≤ F

q kk′,i,t ≤ βl,i,t · Fskk′

l ∈ Li,t, t ∈ T (11b) −

√ 2βl,i,t · Fskk′ ≤ F

p kk′,i,t + F

q kk′,i,t ≤

√ 2βl,i,t · Fskk′

l ∈ Li,t, t ∈ T (11c) −

√ 2βl,i,t · Fskk′ ≤ F

p kk′,i,t + F

q kk′,i,t ≤

√ 2βl,i,t · Fskk′

l ∈ Li,t, t ∈ T . (11d) 6) Voltage Constraint: The voltage limits under the state

Si,t with the action at should be satisfied.

V 2k ≤ Uk,i,t ≤ V 2 k k ∈ B̄i,t, t ∈ T . (12)

Authorized licensed use limited to: Penn State University. Downloaded on August 11,2020 at 22:55:01 UTC from IEEE Xplore. Restrictions apply.

WANG et al.: MDP-BASED RESILIENCE ENHANCEMENT FOR DISTRIBUTION SYSTEMS: ADP APPROACH 2503

Fig. 3. Markov state-based decision at (a) t1, (b) t2, and (c) t3.

C. MDP-Based Optimization Model

The constructed MDP-based optimization model can be represented as follows.

Obj. (3)

s.t. (4) − (12) (13) This is a recursive model for each Markov state which is

computationally intractable. The next section will introduce the solution method.

IV. MODEL SOLUTION BASED ON APPROXIMATE DYNAMIC PROGRAMMING

This section first introduces the challenges of solving the proposed model, and then presents the basic idea of approxi- mate dynamic programming (ADP) in solving the MDP-based model, and finally shows how to solve the proposed model by using the ADP approach.

A. Challenges of Model Solution

When using the conventional stochastic programming to deal with a sequential decision-making problem with uncer- tainties, one commonly used approach is to generate some scenarios to represent the uncertainties. Based on these gen- erated scenarios, we optimize a model with an expected objective, and then we can obtain the optimal strategy. The conventional stochastic programming cannot be used to deal with the proposed model because the influences of actions on scenario transitions are not included and the real scenario may not be included in the generated scenarios.

The decision processes used in this paper can be illustrated by using the case in Fig. 3 (a), (b), and (c), which represent the decision making processes at t1, t2, and t3, respectively. At t1, the state transition tree under the impacts of actions is constructed, as shown in Fig. 3 (a), and the optimal action can be obtained by optimizing the recursive model (3). After performing the optimal action, the state reaches a new state at t2 under uncertainty, as shown in Fig. 3 (b). For this new state, the state transition tree under the impacts of actions needs to be updated again because some state transitions may be invalid. With the new state transition tree, the optimal action for the new state at t2 can be obtained by optimizing the recursive model (3) again. The optimal action for the new state at t3 can be obtained with the similar processes. According to the decision processes, constructing the state transition tree under

Fig. 4. Decision Processes with Post-decision States.

actions in consideration of uncertainties is one critical step to solve the MDP-based model. However, it is a difficult task to construct the state transition tree of the proposed model in consideration of various actions and the resulting compli- cated state transitions. In addition, a large-scale problem with numerous states is possibly intractable due to “curse of dimen- sionality”. There are three curses of dimensionality: (i) the state space S may be too huge to calculate the value function vt(Si,t) for each state within acceptable time, (ii) the decision space A is too large to obtain the optimal action for each state, (iii) the outcome space may be too large to calculate the expectation of future cost.

B. Approximate Dynamic Programming

Approximate dynamic programming is a modeling frame- work offering some techniques for dealing with the curses of dimensionality in multi-period, large, and stochastic MDP- based models. Two important techniques, the post-decision states and the forward dynamic algorithm in approximate dynamic programming, are used to address the curses of dimensionality for multi-period, large-scale, and stochastic MDP-based models.

• Post-Decision States: A post-decision state is defined as a state immediately after the action but before the arrival of a new state at the next decision time due to uncer- tainties. The information embedded in the post-decision states can be used to estimate the downstream future cost. As shown in Fig. 4, S′1, S

′ 2, and S

′ 3 are the post-decision

states of S1, S2, and S3 after the actions A1, A2, and A3, respectively. If we can assign the future cost to the post- decision states, the multi-period and large-scale stochastic

Authorized licensed use limited to: Penn State University. Downloaded on August 11,2020 at 22:55:01 UTC from IEEE Xplore. Restrictions apply.

2504 IEEE TRANSACTIONS ON SMART GRID, VOL. 11, NO. 3, MAY 2020

MDP-based model becomes a one-period deterministic model for each state in each decision period, which can be solved easily.

• Forward Dynamic Algorithm: To solve the one-period deterministic model for each state in each decision period, the estimated value of each post-decision state should be known. A sample path-based forward algorithm is employed to solve the recursive MDP-based model, and then iteratively repeat this procedure to obtain the estimated value of each post-decision state.

1) Post-Decision States: The post-decision state, defined as Sati,t, is a state immediately after the action at but before the arrival of a new state in consideration of uncertainties. To apply the ADP approach, a more generic form of the proposed model in (3) is an expectational form listed as follows.

vt ( Si,t

) = min at ∈A

( Ct

( Si,t, at

) + E{vt+1 ( Sj,t+1|Si,t, at, ξt

)}) (14)

where (14) can be rewritten as (15) with the post-decision state Sati,t.

vt ( Si,t

) = min at ∈A

( Ct

( Si,t, at

) + E{vt+1 ( Sj,t+1|Sati,t, ξt

)}) (15)

Define E{vt+1(Sj,t+1|Sati,t, ξt)} by vatt (Sati,t), we have the fol- lowing optimality equations.

vt ( Si,t

) = min at∈A

( Ct

( Si,t, at

) + vatt ( Sati,t

)) (16a)

v at−1 t−1

( S

at−1 j′,t−1

) = E

{ vt

( Si,t|Sat−1j′,t−1, ξt−1

)} (16b)

Substituting (16a) into (16b) results in the optimality equa- tions of the post-decision states as follows.

v at−1 t−1

( S

at−1 j′,t−1

) = E

{ min at ∈A

( Ct

( Si,t, at

) + vat (

Sai,t|Sat−1j′,t−1, ξt−1 ))}

(17)

where (17) can be rewritten as the form at t as follows.

vatt ( Sati,t

) = E {

min at+1∈A

( Ct+1

( Sj,t+1, at+1

) + vat+1t+1 (

S at+1 j,t+1|Sati,t, ξt

))}

(18)

where (18) shows the value of the post-decision state. 2) Forward Dynamic Algorithm: With the post-decision

states, it would be easy to solve the optimization model (16a) if we know the value of vatt (S

at i,t) in (16a). Based on this idea,

the forward dynamic algorithm in ADP is to use a deter- ministic optimization model (16a) with an initial estimation of ṽatt (S

at i,t) of v

at t (S

at i,t) to make decisions for each state, and

then employ the resulting observations to update an estimation ṽatt (S

at i,t) thereby approximating the expected value in (18). To

deal with iterations, we add a superscript n and n − 1 to the value functions, and (16a) can be expressed as follows.

vnt ( Si,t

) = min at∈A

( Ct

( Si,t, at

) + ṽat ,n−1t ( Sati,t

)) (19)

where the decision that minimizes (19) at nth iteration is shown as follows.

at = arg min at∈A

( Ct

( Si,t, at

) + ṽat ,n−1t ( Sati,t

)) (20)

TABLE I FOUR POST-DECISION STATES

The estimated values of the post-decision states ṽat ,nt (S at i,t)

in the nth iteration is updated by

ṽat ,nt ( Sati,t

) = (1 − �) · ṽat ,n−1t ( Sati,t

) + � · vnt+1 ( Sj,t+1

) (21)

where the first term on the right side of (21) represents the estimate of the post-decision state Sati,t at the (n − 1)th iteration, and the second term represents the value of the result- ing observations from the post-decision state Sati,t at the nth iteration.

With the known estimated values of the post-decision states at the (n − 1)th iteration, we can get the estimated value of the states Si,t by using (19) associated with operational con- straints. This is a one-period deterministic optimization model. Based on (21), we can obtain the estimated values of the post- decision states Sati,t at the nth iteration. With enough iterations, we can obtain the converged estimated values of the post- decision states, and the iterative process is offline. Based on the converged estimated values of the post-decision states, the online one-period deterministic optimization is performed to obtain the optimal strategies for one observed system state Si,t in practice.

C. Reformulation of the Proposed Model

Based on (19), we just need to solve a deterministic model. In the model, the term Ct(Si,t, at) is an explicit objective (4) with regard to variables associated with constraints (5)-(12), however, the term ṽat ,nt (S

at i,t) is just a value with regard to the

post-decision state Sati,t but has no relations to the variables and actions. In this case, it is not possible to optimize the model (19). Therefore, it is necessary to relate ṽat ,nt (S

at i,t) to

the variables and actions. The Markov state in the study is determined by the on-

off states of distributed lines. Since a failure is an observed event and the repair is an activity with continuous time period, whether a failure component is repaired or not at the cur- rent period is known. Furthermore, the post-decision states at the current period are defined as states before arrival of uncertainties in the next time period. Therefore, system recon- figuration is the cause of changing the current state Si,t to S

at i,t.

It is assumed that there are two reconfigurable lines and the corresponding binaries are x1 and x2. We will have four post- decision states listed in Table I. In this case, the second term ṽat ,nt (S

at i,t) in (19) can be expressed as (1 − x1)(1 − x2)V1 +

(1 − x1)x2V2 + x1(1 − x2)V3 + x1x2V4 which relates the values of post-decision states to decision variables.

Based on this technique, we can rewrite the second term ṽat ,nt (S

at i,t) in (19) in a generic form. For the state Si,t, there

are m reconfigurable lines, resulting in 2m post-decision states

Authorized licensed use limited to: Penn State University. Downloaded on August 11,2020 at 22:55:01 UTC from IEEE Xplore. Restrictions apply.

WANG et al.: MDP-BASED RESILIENCE ENHANCEMENT FOR DISTRIBUTION SYSTEMS: ADP APPROACH 2505

associated with the corresponding estimates (represented as V1, V2, . . . , V2m ) at the nth iteration. The 2

m post-decision states are binary-coded with a 2m × m matrix bi, in which the (i′, l)th entry denotes the on-off state of the line l under the i′th post-decision states. The generic form of the term ṽat ,nt (S

at i,t)

at the nth iteration is listed as follows.

∑

i′∈Sposti

⎧ ⎪⎨

⎪⎩

∏

l∈Ldi,t

( 1 − βl,i′,t − bi(i′, l)

)( 1 − 2bi(i′, l)

) Vi′

⎫ ⎪⎬

⎪⎭ (22)

where βl,i′,t is a binary representing on-off states of the line l under post-decision states. bi(i′, l) is a known value with 0 or 1, making (22) a sum of multilinear functions. The optimization model (19) in forward dynamic algorithm can be rewritten as follows.

min (4) + (22) s.t. (5), (6), (7), (8), (9), (10), (11), (12) (23)

For multilinear functions in (22), McCormick proposed a recursive procedure in which additional variables and con- straints are added to obtain a formulation of the problem having only bilinear equations, which can be represented by four binary inequations. In (22), the multilinear func- tion with most variables is β1,i′,tβ2,i′,t · · · βm,i′,t, which can be represented by additional variables

y2,i′,t = β1,i′,tβ2,i′,t y3,i′,t = y2,i′,tβ3,i′,t

· · · ym,i′,t = ym−1,i′,tβm,i′,t (24)

and additional constraints

y2,i′,t ≥ β2,i′,t + β1,i′,t − 1 y2,i′,t ≤ β1,i′,t yl,i′,t ≥ 0 (l = 2, . . . , m) yl,i′,t ≤ βl,i′,t (l = 2, . . . , m) yl,i′,t ≥ βl,i′,t + yl−1,i′,t − 1 (l = 3, . . . , m) yl,i′,t ≤ yl−1,i′,t (l = 3, . . . , m) (25)

where (25) is an exact reformulation of β1,i′,tβ2,i′,t · · · βm,i′,t since βl,i′,t, l = 1, 2, . . . , m are binary variables. Based on the additional variables and the additional constraints, the optimization model is a mixed integer linear programming, which can be solved by many solvers such as CPLEX and GUROBI.

D. ADP Algorithm

Based on the previous sections, the recursive MDP-based model is solved only for one state in each time period, by using the estimation of the post-decision state and performing iterations to update the estimations of the post-decision states on the sample paths. For each iteration, the optimization model is a mixed integer linear programming model, which is solved by the CPLEX solver. The detailed procedures are listed in Algorithm 1.

Algorithm 1 ADP Algorithm 1: Step 1. Set the iteration counter n = 1 and the maximum number

of iterations N, respectively.

2: Step 2. Set the initial approximation ṽat ,nt (S at i,t) for each state.

3: Step 3. Do for t = 1, · · · , T 4: Step 3.1. Solve (19) and (20) to get vnt (Si,t) and at at the

iteration n.

5: Step 3.2. Update the approximation ṽ at ,n t (S

at i,t) for the post-

decision S at i,t with (21).

6: Step 3.3. Obtain the post-decision S at i,t from the state Si,t

under at .

7: Step 3.4. According to uncertainties of the extreme event, generate a new state Sj,t+1 at t + 1 from the post-decision state S

at i,t at t.

8: Step 4. Set n=n+1. If n ≤ N go to Step 3. 9: Step 5. If n = N, return ṽat ,Nt (Sati,t) and the corresponding action

at for the state Si,t .

Fig. 5. Topology of IEEE 33-bus system.

V. CASE STUDIES

In this section, two test systems are used to verify the proposed model and the algorithm. The first system is the IEEE 33-bus system, and the second system is the IEEE 123- bus system. The cases are tested in MATLAB 2017a using the CPLEX 12.6 solver on computers with 3.1 GHz i5 processors and 8 GB RAMS.

A. IEEE 33-Bus System

1) Data Description: Fig. 5 shows the topology of the IEEE 33-bus system. The typhnoon trajectory is also shown in Fig. 5. In this paper, we assume that the trajectory and the duration of the typhnoon are known. Currently, there are many research studies on information of a typhnoon [47]–[49]. In addition, the system operators can obtain the information of a weather-related event from the meteorological depart- ments. The duration between two decisions in the case study is assumed to be 15 minutes, and six decisions need to be made on the typhnoon trajectory. In practice, the duration of a typhoon depends on its real development. For the original topology, the lines 8-21, 12-22, 1-18, 9-15, and 25-29 are dis- connected to ensure the radial topology. It is assumed that the lines 10-11, 12-13, 25-29, 1-18, 14-15, 12-22, 8-21, and

Authorized licensed use limited to: Penn State University. Downloaded on August 11,2020 at 22:55:01 UTC from IEEE Xplore. Restrictions apply.

2506 IEEE TRANSACTIONS ON SMART GRID, VOL. 11, NO. 3, MAY 2020

Fig. 6. Iterations for estimated values of post-decision states.

TABLE II POST-DECISION STATES S1,2 , S2,2 , S3,2 AND S4,2

9-15 are dispatchable and the other lines are non-dispatchable. The operational costs for the lines 10-11, 12-13, 25-29, 1-18, 14-15, 12-22, 8-21, and 9-15 are 1000, 1200, 1300, 1300, 1400, 1400, 1400, and 1400 $/period, respectively. The penalty cost for loss of load is 35000 $/kW in each period.

2) Estimated Values of Post-Decision States: Because post- decision states are introduced to make the proposed recursively state-based model easily to be solved, one important task is to first estimate the values of these post-decision states accord- ing to the ADP algorithm. 1500 iterations were performed to get the estimated values of post-decision states, and the iteration process took about 4.7 × 104s, which are accept- able because this process is offline. Due to a large number of post-decision states, we only show the estimated values of some post-decision states for the sake of exposition. Fig. 6 shows the estimated values of four post-decision states S1,2, S2,2, S3,2 and S4,2, shown in Table II, in the second deci- sion period. The estimated values of the post-decision states S1,2, S2,2, S3,2 and S4,2 converge to 1.29 × 106$, 1.37 × 106$, 1.45×106$, and 1.24×106$, respectively. The estimated value of each post-decision state can be interpreted as the cost for the post-decision state. Because the cost in the paper is quanti- fied by dollars, the estimated value of each post-decision state is is quantified by dollars. When having the estimated values of the post-decision states, the term vatt (S

at i,t) is known when

optimizating (16a). In this case, the problem is transformed into an online one-period deterministic optimization problem, which can be solved within about 5.5s.

Different intensity of severe weather will result in different failure rates, which have great impacts on dispatch strategies. From the perspective of the mathematical model, different failure rates cause different estimated values of each post- decision state. Fig. 7 shows the estimated values of several post-decision states with different failure rates. For example,

Fig. 7. Iterations for estimated values of post-decision states with different failure rates.

Fig. 8. Iterations for estimated values of post-decision states with different values of �.

the estimated values of the post-decision state S1,1 in the first period are 1.01 × 106$, 1.58 × 106$, and 1.81 × 106$ when the failure rates are 0.02, 0.05, and 0.08, respectively. The estimated values of the post-decision state S1,2 in the second period are 1.27 × 106$, 1.89 × 106$, and 2.19 × 106$ when the failure rates are 0.02, 0.05, and 0.08, respectively. It is observed that a higher failure rate cause a larger estimated value of a post-decision state.

When updating the estimated values of post-decision states by using (21), � is artificially set. Fig. 8 shows the impacts of different values of � on the estimated values of the post- decision state S2,1. It is observed that the estimated value are close even when � has different values.

3) Dispatch Strategies With Estimated Values of Post- Decision States: With the estimated values of each post- decision state based on ADP, the strategy corresponding to one observed real-time state can be obtained by a one-period deterministic optimization problem. Table III and Table IV show the state-based strategies, and the original topology has the disconnected lines 10-11, 8-21, 9-15, 1-18, and 25-29. It is observed that the strategies make that the feeders impacted

Authorized licensed use limited to: Penn State University. Downloaded on August 11,2020 at 22:55:01 UTC from IEEE Xplore. Restrictions apply.

WANG et al.: MDP-BASED RESILIENCE ENHANCEMENT FOR DISTRIBUTION SYSTEMS: ADP APPROACH 2507

TABLE III THE FIRST CASE OF STATE-BASED STRATEGY

TABLE IV THE SECOND CASE OF STATE-BASED STRATEGY

by the typhoon are downstream. This is reasonable because downstream feeders cause smaller outages even they are in failure due to the typhoon. When the system operators do nothing during the unfolding event, the values of post-decision states S1,2, S2,2, S3,2 and S4,2 are 2.13 × 106$, 2.56 × 106$, 2.71 × 106$, and 2.01 × 106$, respectively. They are larger than the values 1.27 × 106$, 1.89 × 106$, and 2.19 × 106$ by using the proposed method, respectively. Because the values of post-decision states are expected costs in the subsequent periods, smaller values indicates better strategies.

B. IEEE 123-Bus System

1) Data Description: Fig. 9 shows the topology of the IEEE 123-bus system and the trajectory of a typhoon. The original topology has the disconnected lines 16-96, 92-120, 115-116, 42-120, 38-43, 39-57, 56-76, 46-65, 51-108, and 71-85. The lines 16-96, 92-120, 56-76, 39-57, 38-43, 42-120, 46-65, 51-108, 71-85, 52-53, 60-57, 60-117, 101-119, 63-64, and 67-117 are dispatchable. The operational costs for the dis- patchable lines are 1200 $/period. The penalty cost for loss of load is 35000 $/kW in each period.

2) Simulated Results: Based on the ADP algorithm, the estimated values of post-decision states can be obtained, and then the state-based strategies can be optimized. 1500 iter- ations were performed to get the estimated values of the post-decision states, and the iteration process took about 2.02 × 105s, which are acceptable because this process is offline. With the estimated values of the post-decision states, the problem is transformed into an online one-period deter- ministic optimization problem, which can be solved within about 15s. Table V shows the state-based strategies on the trajectory of the typhoon, and the states on the trajectory are assumed to be generated stochastically based on failure rates

Fig. 9. Topology of IEEE 123-bus system.

TABLE V STATE-BASED STRATEGY FOR IEEE 123-BUS SYSTEM

caused by the typhoon. Fig. 10 (a) and (b) show the topologies after implementing the state-based strategies in the 3th and 6th

periods, respectively. In the 3th period, the line 52-53 is dis- connected and the line 46-65 is connected to avoid balck out of downstream feeders if the typhoon fails the line 52-53. In the 6th period, three lines (57-60, 60-117, 101-119) are discon- nected and three lines (52-53, 56-76, 51-108) are connected to reduce possible black-out areas. It is observed that the state- based strategies try to make the feeders on the trajectory locate the terminal of the whole network to reduce potential loss of load. Define the topologies in Fig. 10 (a) and (b) as the post-decision states S1 and S2. When the system operators do nothing during the unfolding event, the values of S1 and S2 are 2.79 × 106$ and 2.03 × 106$, respectively. When using the proposed method, the values of S1 and S2 are 2.04 × 106$ and 1.79 × 106$, respectively. Smaller values indicates better strategies.

C. Discussion

In the proposed model, the failure rates under extreme weather events are parameter inputs. Once we have the fail- ure rates under extreme weathers, we can obtain the optimal strategies based on the proposed model. In our paper, the com- ponents’ failure probabilities during the event are assumed to

Authorized licensed use limited to: Penn State University. Downloaded on August 11,2020 at 22:55:01 UTC from IEEE Xplore. Restrictions apply.

2508 IEEE TRANSACTIONS ON SMART GRID, VOL. 11, NO. 3, MAY 2020

Fig. 10. System Topologies (a) in the 3rd period and (b) in the 6th period.

be known. Currently, there are many studies on the compo- nents’ failure probabilities during the event. For example, the failure probabilities of lines, towers, and poles caused by wind speed are investigated in the paper [50]–[52]. In practice, it is possible that some data about components’ failure probabili- ties may not be available and we may only know a range of the failure probability. In this case, we can extrrend our MDP- based model to a robust MDP-based model, which could be a future research topic.

When scheduling the system, the accuracy of the obtained strategy is related to the accuracy of load forecast. Because load forecast is a traditional topic, there are many related research studies [53], [54], in which the corresponding meth- ods for load forecast have high accuracy. Furthermore, many existing studies on system scheduling assume that the load forecast is known and its accuracy is acceptable [55], [56]. Therefore, the assumption on load forecast in this paper is reasonable. If the uncertainty of load should be considered, we can extend our MDP-based model to a robust MDP-based model.

The DistFlow model in the power flow constraint is only applicable to the balanced distribution networks. The distribu- tion system can be divided into the high-voltage distribution system, the medium voltage distribution system, and the low voltage distribution system. Usually, the high-voltage distribu- tion system can be considered as a balanced network. In our paper, we only consider the balanced network. The investiga- tion on resilience enhancement for the unbalanced distribution networks is a future research topic.

In practice, there exist some distribution systems that may cover large geography scales. Reference [11] focuses on the scenario that the distribution system is impacted sequentially in different time intervals. If the geography scale of a distribu- tion system is rather small and it is impacted by a hurricane at the same time, the proposed method is also available. In this case, all states of the lines are considered in the proposed model. The post-decision states are introduced in the proposed model to transform the multi-period stochastic model to a one-period deterministic model. The values of post-decision states are estimated by means of the iterative approach, which is offline. With the values of post-decision states, online optimization is a one-period deterministic optimization that has no issue of curse of dimensionality.

VI. CONCLUSION

This paper proposed a Markov state-based decision-making model with dispatching system topology to improve the dis- tribution system resilience throughout the unfolding events. The sequentially states of system topologies changed by the unfolding events and actions are modeled as Markov states, and the uncertainties between different Markov states are rep- resented as transition probabilities that are determined by the component failure rates caused by the unfolding events. Based on Markov states, a recursive optimization model based on Markov decision processes, including the current cost and the expected cost in the future, is developed to make state-based actions at each decision time. To deal with ‘curse of dimen- sionality’ caused by uncertainties, an approximate dynamic programming (ADP) approach with post-decision states and iteration is employed to solve the proposed model. With the estimated values of post-decision states, the stochastic problem with sequential multi-period stochastic optimization problem is transformed into a one-period deterministic problem. Case studies demonstrate that the state-based strategies try to make the feeders on the trajectory locate the terminal of the whole network to reduce potential loss of load, and in consequence to improve system resilience.

REFERENCES

[1] Economic Benefits of Increasing Electric Grid Resilience to Weather Outages, Executive Office President, Washington, DC, USA, Aug. 2013. [Online]. Available: https://www.energy.gov/sites/prod/files/ 2013/08/f2/Grid%20Resiliency%20Report_FINAL.pdf

[2] Enhancing Distribution Resiliency: Opportunities for Applying Innovative Technologies, Elect. Power Res. Inst., Palo Alto, CA, USA, 2013. [Online]. Available: http://www2.epri.com/abstracts/ Pages/ProductAbstract.aspx?ProductId=000000000001026889

[3] Severe Impact Resilience: Considerations and Recommendations, North Amer. Elect. Rel. Corporation, Atlanta, GA, USA, 2012. [Online]. Available: http://www.nerc.com/comm/OC/SIRTF%20Related %20Files%20DL/SIRTF_Final_May_9_2012-Board_Accepted.pdf

[4] A. Gholami, T. Shekari, M. H. Amirioun, F. Aminifar, M. H. Amini, and A. Sargolzaei, “Toward a consensus on the definition and taxonomy of power system resilience,” IEEE Access, vol. 6, pp. 32035–32053, 2018.

[5] M. Panteli and P. Mancarella, “The grid: Stronger, bigger, smarter?: Presenting a conceptual framework of power system resilience,” IEEE Power Energy Mag., vol. 13, no. 3, pp. 58–66, May/Jun. 2015.

[6] National Research Council, Disaster Resilience: A National Imperative. Washington, DC, USA: Nat. Acad. Press, 2012.

[7] H. Liu, R. A. Davidson, J. R. Stedinger, and D. V. Rosowsky, “Negative binomial regression of electric power outages in hurricanes,” J. Infrastruct. Syst., vol. 11, no. 4, pp. 258–267, Dec. 2005.

Authorized licensed use limited to: Penn State University. Downloaded on August 11,2020 at 22:55:01 UTC from IEEE Xplore. Restrictions apply.

WANG et al.: MDP-BASED RESILIENCE ENHANCEMENT FOR DISTRIBUTION SYSTEMS: ADP APPROACH 2509

[8] S. D. Guikema, R. Nateghi, S. M. Quiring, A. Staid, A. C. Reilly, and M. Gao, “Predicting hurricane power outages to support storm response planning,” IEEE Access, vol. 2, pp. 1364–1373, 2014.

[9] R. Nateghi, S. D. Guikema, and S. M. Quiring, “Forecasting hurricane- induced power outage durations,” Nat. Hazards, vol. 74, no. 3, pp. 1795–1811, Dec. 2014.

[10] C. Wang, Y. Hou, Z. Qin, C. Peng, and H. Zhou, “Dynamic coordinated condition-based maintenance for multiple components with external conditions,” IEEE Trans. Power Del., vol. 30, no. 5, pp. 2362–2370, Oct. 2015.

[11] W. Yuan, J. Wang, F. Qiu, C. Chen, C. Kang, and B. Zeng, “Robust optimization-based resilient distribution network planning against natural disasters,” IEEE Trans. Smart Grid, vol. 7, no. 6, pp. 2817–2826, Nov. 2016.

[12] M. H. Amirioun, F. Aminifar, H. Lesani, and M. Shahidehpour, “Metrics and quantitative framework for assessing microgrid resilience against windstorms,” Int. J. Elect. Power Energy Syst., vol. 104, pp. 716–723, Jan. 2019.

[13] M. H. Amirioun, F. Aminifar, and H. Lesani, “Resilience-oriented proac- tive management of microgrids against windstorms,” IEEE Trans. Power Syst., vol. 33, no. 4, pp. 4275–4284, Jul. 2018.

[14] M. H. Amirioun, F. Aminifar, and H. Lesani, “Towards proactive scheduling of microgrids against extreme floods,” IEEE Trans. Smart Grid, vol. 9, no. 4, pp. 3900–3902, Jul. 2018.

[15] L. Zhang, K. Sun, Y. W. Li, X. Lu, and J. Zhao, “A distributed power control of series-connected module-integrated inverters for PV grid-tied applications,” IEEE Trans. Power Electron., vol. 33, no. 9, pp. 7698–7707, Sep. 2018.

[16] F. Iqbal and A. S. Siddiqui, “Optimal configuration analysis for a campus microgrid—A case study,” Protect. Control Mod. Power Syst., vol. 2, no. 23, pp. 1–12, Dec. 2017.

[17] W. Guo and L. Mu, “Control principles of micro-source inverters used in microgrid,” Protect. Control Mod. Power Syst., vol. 1, no. 5, pp. 1–7, Dec. 2016.

[18] C. Wang, B. Cui, and Z. Wang, “Analysis of solvability boundary for droop-controlled microgrids,” IEEE Trans. Power Syst., vol. 33, no. 5, pp. 5799–5802, Sep. 2018.

[19] W. Sun, C.-C. Liu, and S. Liu, “Black start capability assessment in power system restoration,” in Proc. IEEE Power Energy Soc. Gen. Meeting, Detroit, MI, USA, Jul. 2011, pp. 1–7.

[20] F. Qiu, J. Wang, C. Chen, and J. Tong, “Optimal black start resource allocation,” IEEE Trans. Power Syst., vol. 31, no. 3, pp. 2493–2494, May 2016.

[21] M. Zare-Bahramabadi, A. Abbaspour, M. Fotuhi-Firuzabad, and M. Moeini-Aghtaie, “Resilience-based framework for switch placement problem in power distribution systems,” IET Gener. Transm. Distrib., vol. 12, no. 5, pp. 1223–1230, Mar. 2018.

[22] S. Zhang and V. Vittal, “Wide-area control resiliency using redun- dant communication paths,” IEEE Trans. Power Syst., vol. 29, no. 5, pp. 2189–2199, Sep. 2014.

[23] M. M. Adibi and L. H. Fink, “Overcoming restoration challenges associated with major power system disturbances—Restoration from cascading failures,” IEEE Power Energy Mag., vol. 4, no. 5, pp. 68–77, Sep./Oct. 2006.

[24] Y. Hou, C.-C. Liu, K. Sun, P. Zhang, S. Liu, and D. Mizumura, “Computation of milestones for decision support during system restora- tion,” IEEE Trans. Power Syst., vol. 26, no. 3, pp. 1399–1409, Aug. 2011.

[25] W. Sun, C.-C. Liu, and L. Zhang, “Optimal generator start-up strategy for bulk power system restoration,” IEEE Trans. Power Syst., vol. 26, no. 3, pp. 1357–1366, Aug. 2011.

[26] C.-C. Liu, S. J. Lee, and S. S. Venkata, “An expert system operational aid for restoration and loss reduction of distribution systems,” IEEE Trans. Power Syst., vol. PWRS-3, no. 2, pp. 619–626, May 1988.

[27] S. Toune, H. Fudo, T. Genji, Y. Fukuyama, and Y. Nakanishi, “Comparative study of modern heuristic algorithms to service restora- tion in distribution systems,” IEEE Trans. Power Del., vol. 17, no. 1, pp. 173–181, Jan. 2002.

[28] A. Castillo, “Microgrid provision of blackstart in disaster recovery for power system restoration,” in Proc. IEEE Int. Conf. Smart Grid Commun., Vancouver, BC, Canada, Oct. 2013, pp. 534–539.

[29] F. Ren, M. Zhang, D. Soetanto, and X. Su, “Conceptual design of a multi-agent system for interconnected power systems restora- tion,” IEEE Trans. Power Syst., vol. 27, no. 2, pp. 732–740, May 2012.

[30] M. Panteli, P. Mancarella, D. N. Trakas, E. Kyriakides, and N. D. Hatziargyriou, “Metrics and quantification of operational and infrastructure resilience in power systems,” IEEE Trans. Power Syst., vol. 32, no. 6, pp. 4732–4742, Nov. 2017.

[31] G. Huang, J. Wang, C. Chen, J. Qi, and C. Guo, “Integration of preven- tive and emergency responses for power grid resilience enhancement,” IEEE Trans. Power Syst., vol. 32, no. 6, pp. 4451–4463, Nov. 2017.

[32] A. Gholami, T. Shekari, F. Aminifar, and M. Shahidehpour, “Microgrid scheduling with uncertainty: The quest for resilience,” IEEE Trans. Smart Grid, vol. 7, no. 6, pp. 2849–2858, Nov. 2016.

[33] M. Yan, Y. He, M. Shahidehpour, X. Ai, Z. Li, and J. Wen, “Coordinated regional-district operation of integrated energy systems for resilience enhancement in natural disasters,” IEEE Trans. Smart Grid, vol. 10, no. 5, pp. 4881–4892, Sep. 2019.

[34] D. N. Trakas and N. D. Hatziargyriou, “Optimal distribution system operation for enhancing resilience against wildfires,” IEEE Trans. Power Syst., vol. 33, no. 2, pp. 2260–2271, Mar. 2018.

[35] E. Byon and Y. Ding, “Season-dependent condition-based maintenance for a wind turbine using a partially observed Markov decision process,” IEEE Trans. Power Syst., vol. 25, no. 4, pp. 1823–1834, Nov. 2010.

[36] S. K. Abeygunawardane, P. Jirutitijaroen, and H. Xu, “Adaptive main- tenance policies for aging devices using a Markov decision process,” IEEE Trans. Power Syst., vol. 28, no. 3, pp. 3194–3203, Aug. 2013.

[37] C. Wang, Y. Hou, F. Qiu, S. Lei, and K. Liu, “Resilience enhancement with sequentially proactive operation strategies,” IEEE Trans. Power Syst., vol. 32, no. 4, pp. 2847–2857, Jul. 2017.

[38] M. Ouyang and L. Dueñas-Osorio, “Multi-dimensional hurricane resilience assessment of electric power systems,” Struct. Safety, vol. 48, pp. 15–24, May 2014.

[39] S. Lei, C. Chen, Y. Song, and Y. Hou, “Radiality constraints for resilient reconfiguration of distribution systems: Formulation and application to microgrid formation,” arXiv preprint:1907.04951, 2019.

[40] Y. Xu, C.-C. Liu, K. P. Schneider, and D. T. Ton, “Placement of remote-controlled switches to enhance distribution system restoration capability,” IEEE Trans. Power Syst., vol. 31, no. 2, pp. 1139–1150, Mar. 2016.

[41] Insights Into Advanced Distribution Management Systems, U.S. Dept. Energy, Washington, DC, USA, 2015. [Online]. Available: https://www.energy.gov/

[42] A. Arif, Z. Wang, J. Wang, and C. Chen, “Power distribution system outage management with co-optimization of repairs, reconfiguration, and DG dispatch,” IEEE Trans. Smart Grid, vol. 9, no. 5, pp. 4109–4118, Sep. 2018.

[43] S. Lei, C. Chen, Y. Li, and Y. Hou, “Resilient disaster recovery logistics of distribution systems: Co-optimize service restoration with repair crew and mobile power source dispatch,” IEEE Trans. Smart Grid, vol. 10, no. 6, pp. 6187–6202, Nov. 2019.

[44] M. E. Baran and F. F. Wu, “Network reconfiguration in distribution systems for loss reduction and load balancing,” IEEE Trans. Power Del., vol. 4, no. 2, pp. 1401–1407, Apr. 1989.

[45] J. A. Taylor and F. S. Hover, “Convex models of distribution system reconfiguration,” IEEE Trans. Power Syst., vol. 27, no. 3, pp. 1407–1413, Aug. 2012.

[46] X. Chen, W. Wu, and B. Zhang, “Robust restoration method for active distribution networks,” IEEE Trans. Power Syst., vol. 31, no. 5, pp. 4005–4015, Sep. 2016.

[47] L. Liu, L. Padilla, S. H. Creem-Regehr, and D. H. House, “Visualizing uncertain tropical cyclone predictions using representative samples from ensembles of forecast tracks,” IEEE Trans. Vis. Comput. Graphics, vol. 25, no. 1, pp. 882–891, Jan. 2019.

[48] J. B. Elsner and T. H. Jagger, “A hierarchical Bayesian approach to seasonal hurricane modeling,” J. Climate, vol. 17, no. 14, p. 2813–2827, 2004.

[49] P. J. Vickery and L. A. Twisdale, “Prediction of hurricane wind speeds in the united states,” J. Struct. Eng., vol. 121, no. 11, pp. 1691–1699, 1995.

[50] M. Panteli, C. Pickering, S. Wilkinson, R. Dawson, and P. Mancarella, “Power system resilience to extreme weather: Fragility modeling, proba- bilistic impact assessment, and adaptation measures,” IEEE Trans. Power Syst., vol. 32, no. 5, pp. 3747–3757, Sep. 2017.

[51] M. J. P. C. D. Canham and E. F. Latty, “Interspecific variation in sus- ceptibility to windthrow as a function of tree size and storm severity for northern temperate tree species,” Can. J. Forest Res., vol. 31, no. 1, pp. 1–10, Jan. 2001.

[52] A. M. Salman, Y. Li, and M. G. Stewart, “Evaluating system reliability and targeted hardening strategies of power distribution systems subjected to hurricanes,” Rel. Eng. Syst. Safety, vol. 144, pp. 319–333, Dec. 2015.

Authorized licensed use limited to: Penn State University. Downloaded on August 11,2020 at 22:55:01 UTC from IEEE Xplore. Restrictions apply.

2510 IEEE TRANSACTIONS ON SMART GRID, VOL. 11, NO. 3, MAY 2020

[53] J. W. Taylor and P. E. McSharry, “Short-term load forecasting meth- ods: An evaluation based on European data,” IEEE Trans. Power Syst., vol. 22, no. 4, pp. 2213–2219, Nov. 2007.

[54] S. Fan and L. Chen, “Short-term load forecasting based on an adaptive hybrid method,” IEEE Trans. Power Syst., vol. 21, no. 1, pp. 392–401, Feb. 2006.

[55] S. Lei, Y. Hou, F. Qiu, and J. Yan, “Identification of critical switches for integrating renewable distributed generation by dynamic network recon- figuration,” IEEE Trans. Sustain. Energy, vol. 9, no. 1, pp. 420–432, Jan. 2018.

[56] S. Gill, I. Kockar, and G. W. Ault, “Dynamic optimal power flow for active distribution networks,” IEEE Trans. Power Syst., vol. 29, no. 1, pp. 121–131, Jan. 2014.

Chong Wang (M’16) received the B.E. and M.S. degrees in electrical engineering from Hohai University, Nanjing, China, in 2009 and 2012, and the Ph.D. degree in electrical engineering from the University of Hong Kong, Hong Kong, in 2016.

He was a Post-Doctoral Researcher with the University of Hong Kong in 2016, and Iowa State University from 2017 to 2018. He is currently an Associate Professor with the College of Energy and Electrical Engineering, Hohai University. His research interests include integrated energy systems and power system resilience.

Ping Ju (M’95–SM’10) received the B.S. and M.S. degrees in electrical engineering from Southeast University, Nanjing, China, in 1982 and 1985, respectively, and the Ph.D. degree in electrical engineering from Zhejiang University, Hangzhou, China.

From 1994 to 1995, he was an Alexander-von Humboldt Fellow with the University of Dortmund, Germany. He is currently a Professor of electri- cal engineering with Hohai University, Nanjing, China, and Zhejiang University. His research interest

include modeling and control of power system with integration of renewable generation.

Shunbo Lei (M’17) received the B.E. degree in elec- trical engineering from the Huazhong University of Science and Technology, Wuhan, China, in 2013, and the Ph.D. degree in electrical and electronic engineering from the University of Hong Kong, Hong Kong, in 2017.

He was a Visiting Scholar with Argonne National Laboratory, Argonne, IL, USA, from 2015 to 2016. He was a Post-Doctoral Researcher with the University of Hong Kong from 2017 to 2019. He is currently a Research Fellow with the University of

Michigan, Ann Arbor, MI, USA. His research interests include power system operation, resilience, demand response, convex optimization, and machine learning.

Zhaoyu Wang (S’13–M’15) received the B.S. and M.S. degrees in electrical engineering from Shanghai Jiaotong University in 2009 and 2012, respectively, and the M.S. and Ph.D. degrees in electrical and computer engineering from the Georgia Institute of Technology in 2012 and 2015, respectively.

He is the Harpole-Pentair Assistant Professor with Iowa State University. He was a Research Aid with Argonne National Laboratory in 2013 and an Electrical Engineer Intern with Corning, Inc., in 2014. His research interests include power distribu-

tion systems, microgrids, renewable integration, power system resilience, and power system modeling.

Feng Wu (M’15) received the B.Eng. and M.S. degrees in electrical engineering from Hohai University, China, in 1998 and 2002, respec- tively, and the Ph.D. degree from the University of Birmingham, U.K., in 2009. He is currently a Professor of electrical engineering with the College of Energy and Electrical Engineering, Hohai University. His research interests are modeling and control of renewable power generation systems. He is the winner of the National Outstanding Youth Science Foundation of China.

Yunhe Hou (M’08–SM’15) received the B.E. and Ph.D. degrees in electrical engineering from the Huazhong University of Science and Technology, Wuhan, China, in 1999 and 2005, respectively. He was a Post-Doctoral Research Fellow with Tsinghua University, Beijing, China, from 2005 to 2007, and the Post-Doctoral Researcher with Iowa State University, Ames, IA, USA, and University College Dublin, Dublin, Ireland, from 2008 to 2009. He was also a Visiting Scientist with the Laboratory for Information and Decision Systems, Massachusetts

Institute of Technology, Cambridge, MA, USA, in 2010. He joined the faculty with the University of Hong Kong, Hong Kong, in 2009, where he is cur- rently an Associate Professor with the Department of Electrical and Electronic Engineering.

Authorized licensed use limited to: Penn State University. Downloaded on August 11,2020 at 22:55:01 UTC from IEEE Xplore. Restrictions apply.