Literature review

Contents lists available at ScienceDirect

Optical Switching and Networking

Optical Switching and Networking 23 (2017) 225–240

http://d 1573-42

n Corr 305-701

E-m dkkang chyoun

journal homepage: www.elsevier.com/locate/osn

Energy and QoS aware resource allocation for heterogeneous sustainable cloud datacenters

Yuyang Peng, Dong-Ki Kang, Fawaz Al-Hazemi, Chan-Hyun Youn n

Department of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea

a r t i c l e i n f o

Article history: Received 31 August 2015 Received in revised form 1 February 2016 Accepted 19 February 2016 Available online 27 February 2016

Keywords: Sustainable cloud datacenters Renewable energy Virtual machine allocation Heterogeneity

x.doi.org/10.1016/j.osn.2016.02.001 77/& 2016 Elsevier B.V. All rights reserved.

esponding author at: 373-1 Guseong-dong, , Korea. Tel.: +82 42 350 3495; fax: +82 42 ail addresses: [email protected] (Y. Pen @kaist.ac.kr (D.-K. Kang), [email protected] ( @kaist.ac.kr (C.-H. Youn).

a b s t r a c t

As the demand on Internet services such as cloud and mobile cloud services drastically increases recently, the energy consumption consumed by the cloud datacenters has become a burning topic. The deployment of renewable energy generators such as Pho- toVoltaic (PV) and wind farms is an attractive candidate to reduce the carbon footprint and, achieve the sustainable cloud datacenters. However, current studies have focused on geographical load balancing of Virtual Machine (VM) requests to reduce the cost of brown energy usage, while most of them have ignored the heterogeneity of power consumption of each cloud datacenter and the incurred performance degradation by VM co-location. In this paper, we propose Evolutionary Energy Efficient Virtual Machine Allocation (EEE- VMA), a Genetic Algorithm (GA) based metaheuristic which supports a power hetero- geneity aware VM request allocation of multiple sustainable cloud datacenters. This approach provides a novel metric called powerMark which diagnoses the power efficiency of each cloud datacenter in order to reduce the energy consumption of cloud datacenters more efficiently. Furthermore, performance degradation caused by VM co-location and bandwidth cost between cloud service users and cloud datacenters are considered to avoid the deterioration of Quality-of-Service (QoS) required by cloud service users by using our proposed cost model. Extensive experiments including real-world traces based simulation and the implementation of cloud testbed with a power measuring device are conducted to demonstrate the energy efficiency and performance assurance of the pro- posed EEE-VMA approach compared to the existing VM request allocation strategies.

& 2016 Elsevier B.V. All rights reserved.

1. Introduction

The electric energy consumption of datacenters is accounted to be 1.5% of the worldwide electricity usage in 2010, and the energy cost is a primary fraction of a data- center's maintenance expenditure [1,2]. Therefore, there is a growing push to improve the energy efficiency of the

Yuseong-gu, Daejeon 350 7260. g), F. Al-Hazemi),

data centers behind cloud computing [3,4]. Traditionally, datacenters get their power supply from the utility grid which is generated by dirty energy generators, such as coal, or nuclear plants [15]. These conventional energy generators not only produce much carbon but also increase the operation cost for datacenters. Towards addressing this inefficiency, a promising solution receiving spotlights is the incorporation of renewable energy gen- erators such as PhotoVoltaic (PV) and wind turbines into the design of datacenters (i.e., achieving “sustainable” datacenters which reduce not only the electricity cost but also the carbon footprint). Renewable energy generator is becoming drastically an attractive candidate for designing

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240226

green datacenters in academia. Recently, researchers have proposed several studies to integrate renewable energy sources into cloud datacenters. The cost optimization model considering both of renewable energy source and cooling infrastructure is proposed to realize the potential of sustainable cloud datacenters [9]. They propose the demand shifting which schedules non-interactive work- load to maximize the utilization of renewable power source. The energy storage management of sustainable cloud datacenters has been proposed to minimize the cloud service provider's electricity cost [10,11]. The sche- duling scheme for parallel batch jobs has been proposed in order to maximize the utilization of green energy con- sumption while ensuring the Service Level Agreements (SLAs) of requests [16]. However, there are still remaining challenges to achieve the energy efficient sustainable cloud datacenters.

First, each cloud datacenter have heterogeneous server architecture, i.e., they require different power consump- tion even for serving of the same amount of workload. The server heterogeneity is caused by hardware upgrades, capacity extension, and the replacement of peripheral devices [6–8]. However, traditional cloud datacenter management schemes assume that all the cloud data- centers have homogeneous server architecture with same power efficiency although this assumption is unrealistic for most cloud resource providers. Second, two issues of greening cloud datacenters and Quality-of-Service (QoS) assurance are conflicting goals in the resource manage-

Fig. 1. Cloud environment consists of multiple cloud datacenters and Cloud

ment. Especially, the performance degradation might be induced by VM co-location interference when multiple VM instances are running on common physical server in cloud datacenters [12]. As more VM instances are packed into common servers, the required number of active servers is decreased, while the resource contention is deteriorated. This means that the energy consumption is reduced with sacrificing the QoS assurance of the processing for VM requests. It is important to find a desirable tradeoff between above two goals corresponding to the dynamic workload level.

To solve these challenges, we propose an Evolutionary Energy Efficient Virtual Machine Allocation (EEE-VMA) approach which depends on an energy optimization model for sustainable cloud datacenters having hetero- geneous power efficiency with renewable energy gen- erators. This paper proposes four contributions as belows.

First, our approach tries to find a near optimized solu- tion of VM request allocation by applying Genetic Algo- rithm (GA) with consideration for both of renewable energy cost and traditional utility grid cost. The funda- mental strategy adopted in EEE-VMA as an energy saving scheme is Dynamic Right Sizing (DRS) which is for making cloud datacenters be power-proportional (i.e., consumes power only in proportion to the workload level) by adjusting the number of active servers in response to actual workload (i.e., to adaptively “right-size” the data- center) [3,5]. In DRS, the energy saving can be achieved through allowing idle servers that do not have any running

Request Brokers (CRBs) with Cloud Request Broker Manager (CRBM).

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240 227

VM instances to be low-power mode (e.g., sleep or hiber- nation). Note that our proposed energy consumption model for the EEE-VMA approach includes a switching cost of DRS which is incurred by toggling a server from low-power mode into active mode (i.e., awaken transi- tion). This makes our proposed approach more practical for energy efficient cloud datacenter in real world.

Second, in our proposed EEE-VMA approach, in order to adopt the heterogeneous power efficiency of each cloud datacenters, we propose a novel metric called powerMark to quantize the power efficiency of servers by measuring their power consumption at each utilization level of resources such as CPU, memory, and I/O bandwidth. Especially, we compute powerMark for serveral types of server by measuring their power consumption for pro- cessing CPU-intensive applications.Through powerMark, we are able to determine the allocation priority of each cloud datacenter based on their power efficiency so as to improve the performance of energy saving.

Third, we achieve the significant energy saving of cloud datacenters while minimizing the performance degrada- tion caused by VM co-location interference through our EEE-VMA approach. The workload model including both of the number of co-located VM instances and the resource utilization which are key factors reflecting VM co-location interference is applied to the cost model of the EEE-VMA approach. Moreover, we consider the bandwidth cost between cloud service users and cloud datacenters as an additional part contributing the QoS deterioration of VM request processing [31,32]. The desirable cloud datacenter selection for each VM request assignment are conducted with consideration for both of energy saving and QoS assurance corresponding to the dynamic workload level.

Finally, we conduct extensive experiments through simulations at various workload levels based on real-world traces such as dynamic capacity of renewable energy and electricity prices of traditional grid power [9,18–20], and the implementation of testbed with a power measuring

Table 1 Set of key notations.

Notation Description

DC The set of cloud datacenters CRB The set of CRBs F The set of flavor types of VM request supported by cloud resour RC The set of resource components such as CPU and memory Λi tð Þ The set of VM requests arrived at whole CRBs at time t X tð Þ Resource allocation plan of VM requests from CRBs to cloud datac

each VM request M tð Þ DRS plan of cloud datacenters at time t, which determines the n S tð Þ A solution including resource allocation plan X tð Þ and DRS plan Dj tð Þ Performance degradation of cloud datacenter DCj by CPU resour UR The set of predetermined resource utilization levels pwMjrch i

An average power consumption per an unit level of utilization o

pivotS The predetermined pivot server used as a criterion of resource c ej tð Þ The energy consumption of cloud datacenter DCj at time t ctotal tð Þ The total cost of whole cloud datacenters at time t f EEE �VMA Uð Þ The objective function to get ctotal tð Þ in the EEE-VMA solver

device called Yocto-Watt to measure a real power con- sumption of several cloud server types [21].

The rest of the paper is organized as follows. Section 2 gives an overview of the proposed system architecture of multiple cloud datacenters and cloud request brokers. In Section 3, the objective cost model including workload and energy consumption model with powerMark are for- mulated. Our EEE-VMA approach based on Genetic Algo- rithm is proposed to obtain the approximated optimal solution minimizing the total cost of cloud datacenters in Section 4. Section 5 shows the various experimental results that demonstrate the effectiveness of our proposed approach based on real-world traces. The conclusion is given in Section 5.

2. System architecture and design

Our considered cloud environment including multiple Cloud Request Brokers (CRBs) which support mesh net- working with distributed multiple cloud datacenters is depicted in Fig. 1. There are h CRBs and m cloud data- centers with h�m communication links. In each cloud datacenter, the information of resource utilization, the available renewable energy, and the power consumption of each server are collected through monitoring modules, power measuring devices, and reported to the Cloud Request Broker Manager (CRBM) which is responsible for solving the allocation of VM requests submitted to CRBs. The CRBM has two modules: the powerMark analyzer and the EEE-VMA solver. The powerMark analyzer is respon- sible for capturing the power efficiency of each cloud datacenter through our proposed novel metric called powerMark. We describe this metric in detail in Section 3. The EEE-VMA solver is responsible for finding a near optimal solution of VM request allocation from CRB to the cloud datacenter. The solution derived by the EEE-VMA solver based on the amount of submitted VM requests in

ce provider

enters at time t, which determines the destined cloud datacenters for

umber of active servers of cloud datacenters M tð Þ ce contention at time t

f resource component rcARC of servers in the cloud datacenter DCj apacity

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240228

each CRB and the reported information from each cloud datacenter is delivered to whole CRBs and all the submitted VM requests are allocated to their destined cloud datacenters.

The owner of cloud datacenters has to minimize costs for resource operation while boosting benefits which can be realized since cloud service users have a good reputation on observed QoS by cloud services. In this paper, our EEE-VMA solver tries to find a solution to minimize the total cost of resource operation including three sub cost models: energy consumption cost, band- width cost, and performance degradation cost. In the perspective of energy consumption cost, the EEE-VMA solver tries to maximize the utilization of renewable energy with consideration on the dynamic capacity of each renewable energy generator since the price of renewable energy is much cheaper than the one of grid energy.

VM requests from CRBs are preferably allocated to cloud datacenters which have the higher capacity of renewable energy and the higher power efficiency (i.e., the lower powerMark value). In the perspective of bandwidth cost, the EEE-VMA tends to route VM requests to cloud datacenters having the cheaper bandwidth cost. Obviously, different pairs of CRB and cloud datacenter have different bandwidth cost according to the hop distance and the amount of transferred data of routed VM requests. Therefore, it is clear that VM requests need to be allocated to the closest cloud datacenter to their source CRB in order to minimize the bandwidth cost. To simplify our model, we assume that the transferred data size of each VM request is known to the EEE-VMA solver in CRBM beforehand.

In the perspective of performance degradation cost, the EEE-VMA solver tries to spread whole VM requests over multiple cloud datacenters in order to avoid QoS dete- rioration of VM request processing. In cloud datacenters, the VM co-location interference is the key factor that makes servers undergo severe performance degradation [12,22]. The VM co-location interference is caused by resource contention which can be reflected mainly by the number of co-located VM instances and resource utiliza- tion of them. In brief, the VM co-location interference grows bigger as more VM instances are co-located on the common server and the higher resource utilization is occurred. Therefore, VM requests have to be scattered in order to try its hardest to avoid performance degradation by VM co-location interference. Because of the complexity of optimization for aggregated cost model, the EEE-VMA solver adopts metaheuristic based on GA to obtain near optimal solution of VM requests allocation within the acceptable computation time. In next section, we propose a mathematical model to describe the cost of cloud data- center and describe the metric powerMark in detail. The set of involved key notations are shown in Table 1.

3. Problem formulation

3.1. Workload model

There are many different kinds of workloads in cloud datacenters which can be classified into two categories: interactive or transactional (delay-sensitive) and non- interactive or batch (delay-tolerant) workload [9]. The inter- active workloads such as Internet web services and multi- media streaming services have to be processed within a cer- tain response time defined by service users. They are often network I/O intensive jobs which have less impact to the power consumption of servers. In contrast, the batch work- loads such as scientific applications and big data analysis can be scheduled to process anytime as long as the whole tasks are finished before the predetermined deadline. They are usually computation intensive jobs that require a lot of CPU utilization causing a significant power consumption of ser- vers. In this paper, we are interested in the computation intensive batch workloads since they have a greater influence to server power consumption than interactive workloads. We assume that all the VM requests have computation intensive workloads, and the resource contention is always occurred in CPU resource. A workload λki tð ÞAΛi tð Þ denotes the number of arrived VM requests with a required flavor type (e.g., instance type such as m3.medium or c4.large in Amazon EC2) Fk AF at the CRBi ACRB at time t [29]. We use rkrch i to denote the required amount of resource component rcARC by a VM request with flavor type Fk where RC ¼ rcCPU; rcMEMf g. For example, rkrch isuchth at Fk ¼ m3:medium and rc ¼ rcCPU represents the required number of CPU cores by a VM request of which the flavor type is m3:medium. When multiple VM requests are arrived at the CRB, then the CRB would decide in which cloud datacenters each VM request should be routed for processing. We assume no data buffering at the CRB so that whenever a VM request arrives at the CRB, it would be routed to a cloud datacenter for processing immediately [11]. We denote the number of VM requests with the flavor type Fk routed from the CRBi to DCj at time t as x

i;k j tð Þ, which is

derived by a resource allocation plan for cloud datacenters, X tð Þ. Then we have the following constraints:X

8DCj ADC xi; kj tð Þ ¼ λ

k i tð Þ; 8CRBi ACℛℬ; 8Fk Aℱ; 8t ð1Þ

0rxi;kj tð Þrλ k i tð Þ; 8CRBi ACRB; 8Fk AF ; 8DCj ADC; 8t ð2Þ

Above Eq. (1) means that the total number of VM requests arrived at CRBs must agree with the one of whole VM requests allocated to cloud datacenters. Another con- straint we should consider is a resource capacity of the cloud datacenter. Each cloud datacenter only can accom- modate VM requests within their resource capacity (e.g., the total number of CPU cores). Then, we have the fol- lowing constraintsX

8CRBi ACℛℬ X

8Fk Aℱ rk⟨rc ¼ rcCPU⟩ Ux

i; k j tð Þrscp

j ⟨rc ¼ rcCPU⟩

Umj tð Þ; 8DCj ADC; 8t ð3Þ X

8CRBi ACℛℬ X

8Fk Aℱ rk⟨rc ¼ rcmem⟩ Ux

i; k j tð Þrscp

j ⟨rc ¼ rcmem⟩

Umj tð Þ; 8DCj ADC; 8t ð4Þ

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240 229

0rmj tð ÞrN DCj � �

; 8DCj ADC; 8t ð5Þ where rkrc ¼ rcCPUh i and r

k rc ¼ rcmemh i are required CPU cores

and memory size of VM request with flavor type Fk AF . scp jrch i is the physical capacity of resource component rcARC of an arbitrary server in the cloud datacenter DCj. Constraints (3) and (4) represent that allocated VM requests can not always exceed the capacity of resource provided by cloud datacenter DCj. We use mj tð Þ to denote the number of active servers in cloud datacenter DCj at time t and it is determined by a DRS plan M tð Þ, and its upper bound is N DCj

� � which is the number of total phy-

sical servers in the cloud datacenter DCj. The constraint (5) represents that mj tð Þ can be determined in the range of 0 to N DCj

� � through the DRS plan. mj tð Þ ¼ 0 means that

whole servers in the cloud datacenter DCj are in the sleep state, while mj tð Þ ¼ N DCj

� � means that they are in the

active state at time t. Next, we consider a VM co-location interference to

build a performance degradation model of resource allo- cation in cloud datacenter [12]. The VM co-location inter- ference implies that the virtualization of cloud supports resource isolation explicitly when multiple VM requests are running simultaneously on common PM, but it does not mean the assurance of performance isolation between VM requests internally. In the perspective of CPU resource, physical CPU cores of the server are not pinned to each running VM request, but assigned dynamically. The switching overhead by the dynamical CPU assignment policy might cause the undesirable performance degra- dation of allocated VM requests. Moreover, the CPU resource contention aggravates the performance degra- dation since it is very difficult to isolate the cache space of CPU. There is a strong relationship between VM co- location interference and the number of co-located VMs in PM [12]. The more co-located VM instances, the more severe VM co-location interference is occurred. Based on [12], we estimate the performance degradation Dj tð Þ of the cloud datacenter DCj ADC by the CPU resource contention at time t as follows,

Dj tð Þ ¼ P

8CRBi ACℛℬ P

8Fk Aℱx i; k j tð ÞUrk⟨rc ¼ rcCPU⟩ U vu⟨rc ¼ CPU⟩ tð Þ þtsj tð Þ

� � scp j⟨rc ¼ rcCPU⟩ Umj tð Þ

;

8DCj ADC; 8t ð6Þ where tsj tð Þ is an average allocated time slice deter-

mined by Hypervisor [25,28] for VM requests allocated to the cloud datacenter DCj at time t. We use vu rch i tð Þ to denote an average utilization of assigned virtual resources of whole VM requests allocated to cloud datacenters at time t. Note that in Eq. (6), tsj tð Þ and rkrc ¼ CPUh i can be known in advance, while vu rch i tð Þ can not be recognized beforehand, until the utilization of CPU resource is mea- sured through the internal monitoring module of each server in cloud datacenters at time t [24]. Therefore it is required to use the historical information of CPU resource utilization of VM requests to find optimal solution of resource allocation for the current workload. As shown in

Fig. 1, the data repository module is responsible for col- lecting and storing the monitoring information of resource utilization of each VM request to estimate the future demand. Our EEE-VMA solver uses the historical data of the resource utilization from the data repository module in each cloud datacenter to estimate the expected perfor- mance degradation of solution candidates.

3.2. Energy consumption model

3.2.1. The renewable energy model The renewable energy such as the PV and wind energy

is more sustainable than the traditional grid power, and its price is low and the less carbon is emitted [9]. There are two models to achieve the sustainable cloud data- centers by deploying the renewable energy generation. One is on-site deployment of renewable energy genera- tion at the datacenter facility itself. For example, Apple has built its own local biogas fuel cells and two 20-MW solar arrays in Maiden, NC and they have been powered by 100% renewable energy sources [26,27]. Such on-site renewable energy generator can alleviate energy losses due to the transmission and distribution of generated energy, but its energy potential depends greatly on the location of the cloud datacenter. Another model is building the renewable energy generator at off-site facilities. It has the flexibility to locate the generator in a location with good weather (e.g., strong wind speed or bright sunshine), but the significant transmission losses of energy can be occurred. In this paper, we use the first model which has been adopted by most major datacenter owners.

We denote rwej tð Þ and rpej tð Þ as dynamic capacity of renewable wind energy and renewable photovoltaic energy of the cloud datacenter DCj ADC at time t, respec- tively. Obviously, it is required to forecast the future capacity of renewable energy to achieve energy efficient resource management of cloud datacenters since they are usually intermittent and irregular. Therefore, we estimate the future capacity of renewable energy generation by using the historical data from the data repository module in the cloud datacenter through calculating an Exponen- tially Weighted Moving Average (EWMA) values. The detailed descriptions of the EWMA based forecasting scheme for estimated capacity of renewable energy is omitted in this paper.

3.2.2. Heterogeneous power consumption model We propose a novel power efficient metric called

powerMark to evaluate the heterogeneous power con- sumption of cloud datacenters. Servers consist of each cloud datacenter have heterogeneous architecture, which implies that the specification of their resources are dif- ferent, consequently, even though they process the same application, for which each required power consumption might be different [8,13]. To describe powerMark in detail, we propose Definition 1 and 2 as belows,

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240230

Definition 1. (powerMark):

The powerMark pwMjrch i is an average power con- sumption per an unit level of utilization of resource component rcARC of servers in the cloud datacenter DCj.

Definition 2. (pivot server):

The predetermined pivot server pivotS used as a cri- terion of resource capacity for normalizing powerMark of each cloud datacenter.

Moreover, we propose the novel concept pivot server pivotS in Definition 2 to normalize the powerMark of each cloud datacenter. For simplicity, we assume that servers in the same cloud datacenter has power-Homogeneity to each other. To obtain the powerMark value, we predetermined the set of resource utilization levels UR ¼ ur1; ur2; …; urk

� � .

The powerMark pwMjrch i represents the power efficiency of servers in the cloud datacenter DCj with respect to the certain resource component rcAR by calculating an arith- metic mean of power consumption measured at each resource utilization level urk AUR; 8urk 40. For example, we set UR ¼ ur1 ¼ 0:1; ur2 ¼ 0:2; …; ur9 ¼ 0:9f g and rc ¼ rcCPU, then the power consumption of server is measured at each CPU utilization level 0:1; 0:2; …; 0:9 respectively. Based on the data of measured power consumption, the power- Mark pwMjrch i is given by

pwMj⟨rc⟩ ¼ 1

Uℛj j X

8urk AUℛ pw j⟨rc⟩; urk

urk : ð7Þ

npwMj⟨rc⟩ ¼ 1

Uℛj j X

8urk AUℛ

scppivot ⟨rc⟩

scpj ⟨rc⟩

Upw j⟨rc⟩;urk

urk : ð8Þ

where pw jrch i;urk is the power consumption of servers in the cloud datacenter DCj at the utilization level urk of the

resource component rc. npwMjrch i is the normalized value

of pwMjrch i based on the capacity of resource component

rc of the pivot server pivotS where scppivotrch i scpjrch i

Upw jrch i;urk is the

normalized value of pwjrch i;urk . The lower powerMark represents the higher power efficiency of the cloud datacenter and, with larger URjj , powerMark can accu- rately describe the power efficiency of the cloud data- center. In order to investigate the availability of power- Mark, we conduct the preliminary experiment to obtain

Table 2 Three server types for an experiment to investigate powerMark.

Server types CPU architecture CPU cores CPU

Server-1 Intel i5-760 4 2.8 Server-2 Intel i5-4590 4 3.3 Server-3 Intel i7-3770 8 3.4

power consumption of heterogeneous servers with run- ning VM requests processing computation-intensive jobs on a real test bed. There are 3 server types to investigate the heterogeneity of power consumption. The hardware specifications of each server type are shown in Table 2. Each server has two Ethernet interface cards with 1 Gbps and uses Ubuntu 14.04. The test application for the experiment is mProject module (m108 with range 1.7) of Montage Project to make astronomy image files of space galaxy, which is the computation-intensive application [14]. Fig. 2 shows curves of power consumption of each server type as the resource utilization of CPU is increased by mProject running when Server-1 type is pivotS. As mentioned earlier, each server requires different power consumption even at same resource utilization level. The Server-1 has the largest amount of power consumption than others, which means that this server type has the worst performance in terms of energy consumption. Fig. 3 shows the calculated normalized powerMark npwM values with rc ¼ rcCPU of Server-1, 2, and 3 based on Eq. (8). Note that the difference of normalized powerMark npwM values among servers of Fig. 3 is bigger than the one of powerMark pwM values of Fig. 2. The Server-3 has the smallest value of npwM, which means that this server has the best power efficiency among three servers, and this is in concordance with the results in Fig. 2. Based on result curves in Figs. 2 and 3, we conclude that our pro- posed metric powerMark is simple and useful to represent the relative power efficiency of heterogeneous cloud datacenters in practice.

3.2.3. Dynamic right sizing model To achieve power-proportional cloud datacenter

which consumes power only in proportion to the work- load, we consider DRS approach which adjusts the number of active servers by turning them on or off dynamically [3]. Obviously, there is no need to turn all the servers in cloud datacenter on when the total workload is low. In DRS approach, the state of servers which have no running applications can be transit to the power saving mode (e.g., sleep or hibernation) in order to avoid wasting energy as shown in Fig. 4. In order to successfully deploy DRS approach onto our system, we should consider the switching overhead for adjusting the number of active servers (i.e., for turning sleep ser- vers on again).

The switching overhead includes: (1) additional energy consumption by transition from sleep to active state (i.e.,

clocks (GHz) Cache size (kB) Memory size (GB)

8192 3 6144 8 8192 16

Fig. 2. Normalized power consumption results of Server-1, 2, 3 under execution of Montage applications as an example.

Fig. 3. Results of normalized powerMark of Server-1, 2, 3 based on Eq. (7).

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240 231

awaken transition); (2) wear-and-tear cost of server; (3) fault occurrence by turning sleep servers on when toggled is high [3]. We only consider the energy con- sumption as the overhead by DRS execution. Therefore, we define a constant αaWaken to denote the amount of energy consumption for awaken transition of servers. Then the total energy consumption ej tð Þ of cloud datacenter DCj at time t is defined as follows,

ej tð Þ ¼ X

8rcAℛ pwM j ⟨rc⟩ U

P 8CRBi ACℛℬ

P 8Fk Aℱ r

k ⟨rc⟩ Ux

i; k j tð ÞUvurc tð Þ

scpjrc Umj tð Þ

!

þαaWaken � mj tð Þ �mj t�1ð Þ � �þ

; 8DCj ADC; 8t ð9Þ where xð Þþ ¼ max 0; xð Þ. The first term of the right hand

in (9) represents an energy consumption for using servers to serve VM requests allocated to the cloud datacenter DCj at time t and the second term represents an energy consumption for awaken transition of sleeping servers. Especially, the second term implies that a frequent changes in the number of active servers might increase the undesirable waste of energy. Note that the overhead by transition from active to sleep state (i.e., asleep tran- sition) is ignored in our model since a time required for asleep transition is relatively short compared to the one for awaken transition.

3.3. The cloud datacenter cost minimization problem

We build a cost model based on workload model and energy consumption model proposed in Sections 3.1 and 3.2. We focus on minimizing the total cost including three sub costs: (1) energy cost; (2) performance degradation cost; (3) bandwidth cost. In our energy cost model, to simplify it, we assume that the price for renewable energy usage is zero in this paper (strictly, the real price is not zero since the investment expense and the maintenance expenditure for renewable energy generation equipments

are required to deploy the renewable energy generator onto the cloud datacenter). Generally, the price of power grid and the capacity of the generated renewable energy are time-varying according to the electricity market and the location of the cloud datacenter [17,19]. We use cej tð Þ to denote the energy cost of cloud datacenter DCj at time t as follows,

cej tð Þ ¼ ρgrid tð ÞU ej tð Þ�rwej tð Þ�rpej tð Þ � �þ

; 8DCj ADC; 8t ð10Þ

where ρgrid tð Þ denotes the time-varying price of power grid at time t. Next, the performance degradation cost can be determined by the total performance degradation of the cloud datacenter based on Eq. (6). When we use ρperf to denote the constant of penalty price for performance degradation, then the perfor- mance degradation cost of the cloud datacenter DCj at time t, cperfj tð Þ is given by,

cperfj tð Þ ¼ ρ perf � Dj tð Þ; 8DCj ADC; 8t ð11Þ

Note that ρperf is a constant in contrast with ρgrid tð Þ which is dynamically changed according to time. Third, the bandwidth cost is the one for the data transfer between the cloud service users closed to CRBs and VM requests allocated on servers in cloud datacenters. Obviously, different links between CRB and cloud data- center require the different bandwidth cost. The band- width cost is determined by the network distance (e.g., hop distance) and the transferred data size. We use cbwj tð Þ to denote the bandwidth cost of the cloud datacenter DCj at time t as given by,

cbwj tð Þ ¼ X

8CRBi ACℛℬ X

8DCj ADC ρbwi; j U

X 8Fk Aℱ

xi; kj tð ÞUds k

� � ;

8DCj ADC; 8t ð12Þ where ρbwi;j denotes is the bandwidth cost coefficient of

the communication link between the cloud request broker CRBi and the cloud datacenter DCj, and ds

k denotes the transferred data size of VM request with flavor type Fk.

Fig. 4. Illustration of Dynamic Right Sizing procedure.

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240232

Obviously, as the hop distance between the CRBi and DCj grows longer, ρbwi;j is also increased. (12) implies that the allocation of more VM requests to the cloud datacenter which is far away (i.e., has long hop distance) from the source CRB increases the bandwidth cost cbwj tð Þ. It is advantageous for bandwidth cost saving to allocate VM requests to the nearest cloud datacenter to their source CRB.

Consequently, we focus on minimizing the total cost of whole cloud datacenters through our proposed approach of the EEE-VMA solver. We use ctotal tð Þ to denote the total cost of whole cloud datacenters at time t, which includes the energy costs, the performance degradation cost and the bandwidth cost. Then we define the objective function f EEE �VMA S tð Þð Þ to calculate the total cost determined by the solution S tð Þ ¼ X tð Þ ¼ x1; 11 tð Þ;

nn x1;1 2 tð Þ; …; x

Cℛℬj j; ℱj j DCj j tð Þ g;

M tð Þ ¼ m1 tð Þ; m2 tð Þ; …; m DCj j tð Þ � �g at time t as belows,

f EEE �VMA S tð Þð Þ : ctotal tð Þ ¼ X

8DCj ADC cej tð Þþc

perf j tð Þþc

bw j tð Þ

� � s:t 1ð Þ � ð5Þ

ð13Þ To solve this function, we propose the EEE-VMA

approach based on GA in order to find an approximated optimal solution for VM request allocation. In next section, we describe our algorithm in detail.

3.4. Evolutionary energy efficient virtual machine allocation

In this section, we propose EEE-VMA approach based on GA which is one of efficient metaheuristics to solve a complex optimization problem. In order to successfully deploy GA onto the EEE-VMA, we should define accu- rate strategies for GA and set their appropriate para- meters. To do this, we consider five basic steps of GA as follows.

3.4.1. Encoding scheme A chromosome (i.e., individuals in the population)

features the solution S tð Þ of our datacenter management scheme in cloud datacenters. The format of genes in the chromosome is described as an integer value. The chro- mosome includes multiple genes which are divided into two parts: the first part is for the VM request allocation plan X tð ÞAS tð Þ; and the second part is for the DRS plan M tð ÞAS tð Þ. This detailed structure of the chromosome is shown in the Fig. 5.

In the first part of the chromosome, gene values repre- sent the number of VM requests allocated to the cloud datacenter at time t. For example, as shown in Fig. 5, the gene 1; 210ð Þ in the chromosome represents x1;11 tð Þ ¼ 210 which means that the number of allocated VM requests with flavor type F1 from the CRB1 to DC1 is 210 at time t. In the second part of the chromosome, gene values represent the number of active servers in the cloud datacenter at time t. In Fig. 5, the gene CRB � F � DC þ1; 3200j Þ

����������� in the chromosome means that the number of active servers in the cloud datacenter DC1 is 3200 at time t.

3.4.2. Initialization In the first generation g ¼ 1, GA in the EEE-VMA

approach begins with randomly generated populations according to submitted VM requests at each CRB. To reduce the computation time for GA execution, the range of value for each gene can be predetermined based on (2), and (5).

3.4.3. Evaluation In EEE-VMA approach, we use (13) to evaluate the

performance of each chromosome (i.e., solution) in the population. The fitness value of a chromosome is inversely

Fig. 5. Encoding example of VM allocation with chromosome.

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240 233

related to the cost value. The higher fitness function value implies the higher performance of the chromosome. Note that if a certain chromosome violates any of constraints (1)–(5), then its cost value is counted as “positive infinity”. Otherwise, the chromosome which has the smallest cost value among all the chromosomes in the generated population at g ¼ gMax (a max step of generation) is cho- sen as an optimal solution S� tð Þ finally.

3.4.4. Selection There are several candidate schemes for selection of

appropriate solutions in GA. Especially, we adopt the roulette-wheel selection which determines the probability of each chromosome to be chosen according to their fit- ness function value. This scheme tends to preserve superior solutions and evolve them in the next generation [30].

3.4.5. Crossover The role of crossover is to generate offspring from

two parents by cutting certain genes of parents and conducting recombination of each gene fragment. The offspring inherits characteristics of each parent. Our EEE-VMA approach adopts the simple crossover scheme by which the first half of the first parent and the second half of the second parent are aggregated to genes of their offspring. Note that crossover has to be conducted separately on each part of the chromosome since it has two parts of the VM request allocation plan and the DRS plan.

3.4.6. Mutation It is necessary to ensure the diversity of the generated

population at all generation steps in order to avoid local minima problem in GA. At each generation step g, gene values constituting chromosome can be modified ran- domly according to the predetermined probability pbmt. If pbmt is too large, the superior genes inherited from parents can be loss, otherwise, the diversity of popula- tion might be lower when pbmt is too small. It is

important to determine the appropriate pbmt in order to maintain the diverse and superior population. However, we do not consider this issue since it is out of scope in this paper.

The proposed GA for EEE-VMA approach is described in Algorithm 1. In order to get the near optimal solution x�t of datacenter management for VM requests arrived at CRBs at time t, the state information of servers in all the data- centers at time t�1 is required. If the current time t ¼ 0, then we assume that the previous state of all the servers is active (i.e., all the servers are switched on). In line 02, we initialize the candidate population cand_popg tð Þ, g ¼ 1ð Þ with the population size ps (represents the limit number of chromosomes in the population) randomly. The popu- lation cand_popg tð Þ evolves until g ¼ gMax to generate the final population popg ¼ gMax tð Þ to search the near optimal solution x�t as shown from line 03 to 29. Two parent chromosomes Sg;i tð Þ and Sg;j tð Þ are released from temp_popg tð Þ to produce an offspring Sg;k tð Þ from line 06 to 10. In line 11, each offspring in the set offspringg tð Þ is mutated by modifying each gene according to the prob- ability prmt to maintain the diversity of the population. We check constraints (i.e., Eqs. (1)–(5)) of each solution Sg;i tð Þ in cand_popg tð Þ are whether violated or not in line 14. If they are violated, the corresponding solution has the cost value counted as “positive_infinity”. Otherwise, the objective function value of the solution is calculated through f EEE �VMA Uð Þ in line 17. If we find the solution Sg;i tð Þ having the objective function value cg;i tð Þ which is smaller than the predetermined threshold value cthr, then we count Sg;i tð Þ as the near optimal solution S� tð Þ and the algorithm 1 is finished. Otherwise, chromosomes to be preserved until the next generation are chosen from the current population through the Iterative Roulette-wheel Selection (Algorithm 2) procedure as shown in line 23. When reaching the max step of generation gMax, the near optimal solution S� tð Þ having minimum cost value of f EEE �VMA Uð Þ in popg ¼ gMax tð Þ is found and returned to the EEE-VMA solver in our system in line 30.

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240234

The IterativeRwSelection for Algorithm 1 is described in Algorithm 2. From line 02 to 07, the cost values which have “positive_infinity” are released from cand_Cg tð Þ and added to illeg_Cg tð Þ since solutions which do not violate constraints (i.e., (1)–(5)) are preferentially considered as candidates to be preserved until the next generation. The maximum (worst) and minimum (best) objective function

Algorithm 1.

values are found from cand_Cg tð Þ in line 09 and 10. Fitness values of each solution are calculated as shown in line 13. Through this equation, the fitness value of the best solu- tion comes out as α times of one of the worst solution. The selection pressure which represents the difference between fitness values of superior solutions and inferiors is increased as α is increased. The sum of fitness values of each solution, SF is updated in line 15.

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240 235

Algorithm 2.

This value represents the total size of roulette-wheel, and each solution is assigned to spaces on the roulette-wheel. That means that the selection probability of each solution is proportional to the size of their assigned spaces. The selec- tion procedure of the roulette-wheel is described from line 19 to 26. At every step, the cumulative summation QS is

updated according to the fitness value fvg;i tð Þ in FVg tð Þ. If the selection point SP is smaller than QS with the latest update

by fvg;i tð Þ (i.e., Pi�1 k ¼ 1

fvg;k tð ÞrSPr Pi k ¼ 1

fvg;k tð Þ), then the index

i of Sg;i tð Þ in cand_popg tð Þ is added to chIdxSetg tð Þ. If the total

Fig. 6. Wind speed (m=s) and its corresponding amount of generated wind energy (kW) at Oak Ridge National Lab (a and b), Univ of Arizona (c and d), and Univ of Nevada (e and f) at EST 05:20–17:54 on September 9, 2015 [33].

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240236

number of chosen chromosomes by the roulette-wheel procedure is not sufficient (i.e., the cardinality of chIdxSetg tð Þ is smaller than the predetermined size of the population ps), then we supplement chIdxSetg tð Þ by ran- domly putting out the indices of chromosomes from illeg_Cg tð Þ. After all the procedures are finished, then chIdxSetg tð Þ is finally returned to the Algorithm 1 in line 32.

4. Performance evaluation

In this section, we evaluate the performance of our proposed EEE-VMA approach based on both of simulation analysis and experiments on real testbeds. To highlight the

benefits of our design for renewable and QoS aware workload management, we perform a numerical simula- tion based on real-world traces of renewable energy capacity.

4.1. Dynamic capacity of renewable energy

We consider three locations to employ the raw data in order to build a capacity trace of renewable energy including wind energy: Oak Ridge National Lab (Eastern Tennessee); University of Arizona (Tucson, Arizona); Uni- versity of Nevada, Las Vegas (Paradise, Nevada) [11,33]. We obtain the capacity traces of wind energy at those three locations baced on [33] that collects data of wind speed every day. The capacity traces of each location at EST

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240 237

05:20–17:54 on September 9, 2015 are shown in Fig. 6. Fig. 6(a), (c), and (e) shows the wind speed of each location and we can find that it is fluctuated a lot even during short period. We assume that each generator has 30 wind tur- bines and the amount of generated wind energy is esti-

Fig. 7. Real time price of power grid at three regions.

Fig. 8. CPU utilization of the running VM request including pbzip2, iozone3, and netperf.

Fig. 9. Total cost of datacenters including servers with heter

mated based on the wind power prediction scheme from [34]. Then the curves of the amount of available wind energy are shown in Fig. 6(d), (e), and (f).

4.2. Energy price description

As mentioned earlier, only the grid power price is considered since we assume that the renewable energy price is free. The grid power price is dynamically changed according to the electricity consuming time. We use the electricity price information in our simulation based on the real time pricing during 24 h in the electricity market which is shown in Fig. 7 [23,35]. Note that the electricity price is high from 6 a.m. to 14 p.m., and from 19 p.m. to 21 p.m. The electricity usage is usually increased during these periods due to the needs of industrial and household appliances. In our simulation, each cloud datacenter ran- domly has the electricity pricing curve among datacenter 1, 2, and 3 in Fig. 7.

4.3. Cloud resource description

The total number of multiple cloud datacenters is nine, and each datacenter owns 2 � 103 homogeneous servers in this paper. In perspective of VM instance specifications, we adopt the policy of Amazon EC2 Web Services (AWS), our cloud datacenters support the set of flavor types F ¼ F1 ¼ CPU ¼ 2cores; mem ¼ 4 GBð Þ; F2 ¼ 4; 8ð Þ; F3 ¼ 8; 16ð Þ;

� F4 ¼ 16; 32ð Þg and each VM request has an arbitrary flavor type Fk AF randomly [29]. As mentioned in Section 3, each cloud datacenter has heterogeneous server architecture, they have different powerMark value in the range of 200– 500 based on results in Fig. 3.

4.4. Workload scenario

Our considered workload includes two parts: the number of VM requests tð Þ , and their required resource utilization vu rch i tð Þ at time t. The number of VM requests Λ tð Þ can be defined as from 3 � 103 to 100 � 103 in this paper. Obviously, as Λ tð Þ is increased, both of energy con- sumption and performance degradation are also increased.

ogeneous (a) and homogeneous power efficiency (b).

Fig. 10. Active server ratio of cloud datacenters including servers with heterogeneous (a) and homogeneous power efficiency (b).

Fig. 11. Performance degradation of CPU contention by co-located VM requests in Server-1 (a), 2 (b), and 3 (c).

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240238

In perspective of resource utilization, we only consider the resource component rc ¼ rcCPU and ignore the resource component rcmem since the energy consumption and per- formance degradation caused by rcmem is negligible com- pared to ones by rcCPU. We use the real traces of CPU

resource utilization measured by the monitoring module with serveral benchmark applications on the physical machines. Fig. 8 shows the CPU resource utilization of running benchmarks including a mixture of pbzip2, iozone3, netperf on VM instances.

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240 239

4.5. GA Parameters for EEE-VMA approach

We consider a population size ps with a range from 102

to 104, the max step of generation gMax in the range of 100 to 1000, and the mutation probability 0.001, 0.005 and 0.01 in the Algorithm 1 and 2. As the parameters such ps and gMax are increased, the performance of the derived solution is increased, but its computation need is also deteriorated.

4.6. Traditional resource management schemes

To demonstrate that our proposed approach outper- forms existing resource management schemes, we com- pare the EEE-VMA approach to both of VM consolidation and VM balancing based allocation approaches. The VM consolidation approach tries to pack multiple VM requests as many as possible in the common physical server. This scheme tends to reduce the number of active servers. Therefore, the energy saving performance is increased, while the performance degradation is deteriorated. In contrast, the VM balancing approach splits VM requests over multiple cloud datacenters. This scheme avoids the performance degradation of resource contention by VM request co-location, but causes the large energy con- sumption due to a lot of active servers.

Figs. 9, 10, and 11 show the performance of our pro- posed EEE-VMA approach and existing VM balancing and consolidation approaches at ps ¼ 102, gMax ¼ 500, and mutation probability is 0.001. Fig. 9 shows the total cost in Eq. (13) of the VM balancing, VM consolidation and our proposed EEE-VMA approach at different workload offered load level. Fig. 9(a) shows the curves of total cost of all the approaches assuming that each cloud datacenter has het- erogeneous power efficiency. Our proposed approach achieves the improvements of the cost saving performance about 8% and 53% compared to VM consolidation and VM balancing approaches, respectively. However, the differ- ence of the cost saving performance between traditional approaches and our EEE-VMA approach in Fig. 9 (b) assuming that each cloud datacenter has homogeneous power efficiency is relatively small compared to the one in Fig. 9(a). The EEE-VMA approach achieves the improve- ments of the cost saving performance about 10% and 15% compared to VM consolidation and VM balancing approaches, respectively. Note that our EEE-VMA approach further improves the performance of energy saving in the heterogeneous cloud datacenters since it uses powerMark value which can rank the power efficiency of each cloud datacenter to maximize the energy efficiency of resource allocation. However, our proposed approach still has the better performance than ones of existing approaches even under the assumption of homogeneous power efficiency of each cloud datacenter. Fig. 10 shows the active server ratio of cloud datacenters by our EEE-VMA approach and existing resource management approaches. In Fig. 10(a), the average active server ratio of the EEE-VMA approach is under 30%, while the ones of VM balancing is closed to 60%. Our EEE-VMA approach considers both of energy consumption and performance degradation of VM requests, while the VM balancing only focuses on the

performance degradation. Note that the energy saving performance of VM consolidation is worse than the one of the EEE-VMA approach even though the VM consolidation focuses on the energy consumption of cloud datacenters. This is because our EEE-VMA approach allocates VM requests to power efficient cloud datacenters pre- ferentially based on their powerMark values, while the VM consolidation randomly assigns VM requests to cloud datacenters. In Fig. 10(b), the active server ratio of VM consolidation is lower than the one of our EEE-VMA approach, this is because the VM consolidation approach only focuses on the energy consumption of cloud data- center, but the EEE-VMA approach avoids unacceptable performance degradation of running VM requests through Eq. (11).

Fig. 11 shows the performance degradation of allocated VM requests in each cloud datacenter by the EEE-VMA, VM consolidation, and VM balancing approaches based on the server types. The performance degradation is calculated by Eq. (6). In the perspective of performance degradation, the VM balancing approach outperforms the others including our proposed EEE-VMA approach. The VM balancing tries to spread submitted VM requests over whole cloud data- centers as fair as possible, therefore the CPU resource contention of co-located VM requests can be minimized. In Server-1 type, the performance degradation of VM balan- cing is lower than the ones of the EEE-VMA approach and the VM consolidation by 40% and 60%, respectively. In Server-2 type, the performance degradation of VM balan- cing is lower than ones of the EEE-VMA approach and the VM consolidation by 55% and 62%, respectively. Finally, in Server-3 type, the VM balancing approach can improve the performance degradation about 39% and 55% compared to the EEE-VMA approach and the VM consolidation, respectively.

5. Conclusions

In this paper, we introduced the EEE-VMA approach for greening cloud datacenters with renewable energy gen- erators. We proposed a novel energy efficient metric powerMark to classify the power efficiency of hetero- geneous servers in cloud datacenters and built a con- siderate cost model considering switching overheads in order to reduce efficiently the energy consumption of servers without a significant performance degradation by co-located VM requests and DRS execution. We deployed the iterative roulette-wheel algorithm for GA of the EEE- VMA approach in order to solve the complex objective function of our cost model. Through various experimental results based on simulation and Openstack platform justify that our proposed algorithms are supposed to be deployed for prevalent cloud data centers. In the perspective of total cost, our EEE-VMA approach can improve the average cost by 28% compared to existing resource management schemes at all the workload level. With the increase of the computation investment for GA in EEE-VMA approach, our proposed approach can get arbitrarily close to the optimal value.

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240240

Acknowledgments

This work was supported by 'The Cross-Ministry Giga KOREA Project' of the Ministry of Science, ICT and Future Planning, Korea [GK13P0100, Development of Tele- Experience Service SW Platform based on Giga Media].

References

[1] J. Hamilton, Cost of Power in Large-Scale Data Centers, Nov. 2009. Available Online: ⟨http://perspectives.mvdirona.com/⟩.

[2] J. Koomey, Growth in Data Center Electricity Use 2005–2010, Ana- lytics Press, Burlingame, CA, USA, 2011.

[3] M. Lin, A. Wierman, L. Lachlan, H. Andrew, E. Thereska, Dynamic right-sizing for power-proportional data centers, IEEE/ACM Trans. Netw. 21 (5) (2013) 1378–1391.

[4] L.A. Barroso, U. Holzle, The case for energy-proportional computing, Computer 40 (12) (2007) 33–37.

[5] T. Lu, M. Chen, L. Lachlan, H. Andrew, Simple and effective dynamic provisioning for power-proportional data centers, IEEE Trans. Par- allel Distrib. Syst. 24 (6) (2013) 1161–1171.

[6] Z. Ou, H. Zhuang, J. K. Nurminen, A. Yla-Jaaski, and P. Hui, Exploiting hardware heterogeneity within the same instance type of Amazon EC2, In: Proceedings of the 4th USENIX Workshop on HotCloud, 2012.

[7] Z. Ou, H. Zhuang, A. Lukyanenko, J.K. Nurminen, P. Hui, V. Mazalov, A. Yla-Jaaski, Is the same instance type created equal? Exploiting heterogeneity of public clouds, IEEE Trans. Cloud Comput. 1 (2) (2013) 201–214.

[8] A. Beloglazov, R. Buyya, Optimal online deterministic algorithms and adaptive heuristics for energy and performance efficient dynamic consolidation of virtual machines in cloud data centers, Concurr. Comput.: Pract. Exp. 24 (2012) 1397–1420, http://dx.doi.org/ 10.1002/cpe.1867.

[9] Z. Liu, Y. Chen, C. Bash, A. Wierman, D. Gmach, Z. Wang, M. Marwah, C. Hyser, Renewable and cooling aware workload management for sustainable data centers, In: Proceedings of the ACM SIGMETRICS, London, UK, 2012.

[10] Y. Guo, Y. Fang, Electricity cost saving strategy in data centers by using energy storage, IEEE Trans. Parallel Distrib. Syst. 24 (6) (2013) 1149–1160.

[11] Y. Guo, Y. Gong, Y. Fang, P.P. Khargonekar, X. Geng, Energy and network aware workload management for sustainable data centers with thermal storage, IEEE Trans. Parallel Distrib. Syst. 25 (8) (2014) 2030–2042.

[12] F. Xu, F. Liu, L. Liu, H. Jin, B. Li, B. Li, iAware: making live migration of virtual machines interference-aware in the cloud, IEEE Trans. Com- put. 63 (12) (2014) 3012–3025.

[13] X. Wang, B. Li, B. Liang, Dominant Resource Fainess in Cloud Com- puting Systems with Heterogeneous Servers, In: Proceedings of the IEEE INFOCOM, Toronto, Canada, 2014.

[14] Montage, ⟨http://montage.ipack.caltech.edu/⟩. [15] S.K. Garg, R. Buyya, Green cloud computing and environmental

sustainability, in: S.M. A. G. G. (Ed.), Harnessing Green IT: Principles and Practices, Wiley Press, UK, 2012, pp. 315–340.

[16] I. Goiri, M.E. Haquc, K. Le, R. Beauchea, T.D. Nguyen, J. Guitart, J. Torres, R. Bianchini, Matching renewable energy supply and demand in green datacenters, Elsevier Ad Hoc Netw. 25 (2015) 520–534.

[17] C. Wu, H. Mohsenian-Rad, J. Huang, A. Y. Wang, Demand side management for wind power integration in microgrid using dynamic potential game theory, In: Proceedings of the IEEE CLO- BECOM Workshop on Smart Grid Communications and Networking, Houston, Tx, Dec. 2001.

[18] Solar Anywhere, Solar anywhere overview, Clean Power Research, Web. 17 Apr. 2012. ⟨http://www.solaranywhere.com/Public/Over view.aspx⟩.

[19] R. Huang, T. Huang, R. Gadh, Solar Generation Prediction using the ARMA Model in a Laboratory-level Micro-grid, In: Proceedings of the IEEE SmartGridComm Symposium, Tainan, Taiwan, Nov. 2012.

[20] Openstack, ⟨http://www.openstack.org/⟩. [21] YOCTO-WATT, ⟨http://www.yoctopuce.com/EN/products/usb-elec

trical-sensors/yocto-watt⟩. [22] D. Gupta, L. Cherkasove, R. Gardner, A. Vahdata, Enforcing perfor-

mance isolation across virtual machines in Xen, In: Proceedings of the ACM/IFIP/USENIX 2006 International Conference on Middle- ware, Nov. 2006.

[23] A. Qureshi, R. Weber, H. Balakrishnan, J. Guttag, B. Maggs, Cutting the electric bill for internet-scale systems, In: Proceedings of the ACM SIGCOMM Computer Communications Review, vol. 39(4), Aug. 2009, pp. 123–134.

[24] T. Wood, P. Shenoy, A. Venkataramani, M. Yousif, Sandpiper: black- box and gray-box resource management for virtual machines, Elsevier Comput. Netw. 53 (17) (2009) 2923–2938.

[25] Credit Scheduler, ⟨http://wiki.xen.org/wiki/Credit_scheduler⟩. [26] C. Ren, D. Wang, B. Urgaonkar, A. Sivasubramaniam, Carbon-aware

energy capacity planning for datacenters, In: Proceedings of the IEEE MASCOTS, 2012, pp. 391–400.

[27] Apple Environmental Responsibility, ⟨http://www.apple.com/envir onment.renewable-resources/⟩.

[28] H. Fawaz, Y. Peng, C. Youn, A MISO mode for power consumption in virtualized servers, Clust. Comput. 18 (2) (2015) 847–863.

[29] Amazon Web Services, ⟨https://aws.amazon.com⟩. [30] E.D. Dasgupta, Z. Michalewicz, Evolutionary Algorithms in Engi-

neering Applications, Springer, Berlin, Germany, 1997. [31] M. Chen, Y. Wen, H. Jin, V. Leuna, Enaling technologies for future

data center networking: a primer, IEEE Netw. 27 (4) (2013) 8–15. [32] M. Chen, Y. Zhana, L. Hu, T. Taleb, Z. Shena, Cloud-based wireless

network: virtualized, reconfigurable, smart wireless network to enable 5G technologies, ACM/Springer Mob. Netw. Appl. 20 (6) (2015) 704–712.

[33] Measurement and Instrumentation Data Center (MIDC), ⟨http:// www.nrel.gov/midc/⟩.

[34] C. Wu, H. Mohsenian-Rad, J. Huang, A.Y. Wang, Demand side man- agement for wind power integration in microgrid using dynamic potential game theory, In: Proceedings of the IEEE GLOBECOM Workshop on Smart Grid Communications and Networking, Hous- ton, Tx, Dec. 2001.

[35] H. Mohsenian-Rad, A. Leon-Garcia, Optimal residential load control with price prediction in real-time electricity pricing environments, IEEE Trans. Smart Grid 1 (2) (2010) 120–133.