Literature review

Renewable and Sustainable Energy Reviews 62 (2016) 195–214

Contents lists available at ScienceDirect

Renewable and Sustainable Energy Reviews

http://d 1364-03

n Corr E-m

wasimra

journal homepage: www.elsevier.com/locate/rser

Sustainable Cloud Data Centers: A survey of enabling techniques and technologies

Junaid Shuja a, Abdullah Gani a,n, Shahaboddin Shamshirband b, Raja Wasim Ahmad a, Kashif Bilal c

a Centre for Mobile Cloud Computing Research (C4MCCR), FSKTM, University of Malaya, Kuala Lumpur 50603, Malaysia b Faculty of Computer Science and Information Technology, University of Malaya, Malaysia c Department of Computer Science, COMSATS Institute of Information Technology, Pakistan

a r t i c l e i n f o

Article history: Received 19 June 2015 Received in revised form 15 February 2016 Accepted 16 April 2016 Available online 4 May 2016

Keywords: Cloud Data Centers Energy efficiency Renewable energy Waste heat utilization Modular data centers VM migration

x.doi.org/10.1016/j.rser.2016.04.034 21/& 2016 Elsevier Ltd. All rights reserved.

esponding author. Tel.: þ60 0379676300; fax ail addresses: [email protected] ( [email protected] (R.W. Ahmad), kashifbil

a b s t r a c t

Cloud computing services have gained tremendous popularity and widespread adoption due to their flexible and on-demand nature. Cloud computing services are hosted in Cloud Data Centers (CDC) that deploy thousands of computation, storage, and communication devices leading to high energy utilization and carbon emissions. Renewable energy resources replace fossil fuels based grid energy to effectively reduce carbon emissions of CDCs. Moreover, waste heat generated from electronic components can be utilized in absorption based cooling systems to offset cooling costs of data centers. However, data centers need to be located at ideal geographical locations to reap benefits of renewable energy and waste heat recovery options. Modular Data Centers (MDC) can enable energy as a location paradigm due to their shippable nature. Moreover, workload can be transferred between intelligently placed geographically dispersed data centers to utilize renewable energy available elsewhere with virtual machine migration techniques. However, adoption of aforementioned sustainability techniques and technologies opens new challenges, such as, intermittency of power supply from renewable resources and higher capital costs. In this paper, we examine sustainable CDCs from various aspects to survey the enabling techniques and technologies. We present case studies from both academia and industry that demonstrate favorable results for sustainability measures in CDCs. Moreover, we discuss state-of-the-art research in sustainable CDCs. Furthermore, we debate the integration challenges and open research issues to sustainable CDCs.

& 2016 Elsevier Ltd. All rights reserved.

Contents

1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 2. Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

2.1. Renewable energy in CDC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 2.2. Waste heat utilization in CDC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 2.3. Modular CDC designs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 2.4. VM migrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

3. Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 3.1. Parasol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 3.2. Free lunch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 3.3. Aquasar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 3.4. MDC with free cooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 3.5. Facebook Arctic CDC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 3.6. Green House Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200

4. Renewable Energy based CDCs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 4.1. Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

: þ60 379579249. J. Shuja), [email protected] (A. Gani), [email protected], [email protected] (S. Shamshirband), [email protected] (K. Bilal).

J. Shuja et al. / Renewable and Sustainable Energy Reviews 62 (2016) 195–214196

4.2. State-of-the-Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

4.2.1. Dynamic load balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 4.2.2. Follow the renewables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 4.2.3. Renewable based power capping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202

5. Waste heat utilization in CDCs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 5.1. Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204 5.2. State-of-the-Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

6. Modular data centers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 6.1. Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 6.2. State-of-the-art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207

7. VM migration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208 7.1. Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208 7.2. State-of-the-Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209

8. Research Issues and Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 8.1. Renewable energy-CDC integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 8.2. Waste heat utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 8.3. MDC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 8.4. VM WAN migrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212

9. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212 Acknowledgments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212

1. Introduction

Cloud Data Centers (CDC) are increasingly being deployed by Information Technology (IT) service providers, such as Google, Amazon, and Microsoft to cater for world's digital needs. CDCs provide an efficient infrastructure to store large amount of data along with enormous processing capabilities. Business objectives and Service Level Agreements (SLA) demand that the storage and compute facilities be replicated redundantly to provide fault tol- erance and minimal service delay. Therefore, IT service providers run data centers 24/7 with thousands of servers, storage, and networking devices to ensure 99.99% availability of cloud services [1,2]. Our digital activities such as social media, search, file sharing, and streaming are creating huge amount of data. Each bit of data created needs to be processed, stored, and transmitted, adding to energy costs and leaving environmental impact in the form of Greenhouse Gas (GHG) emissions [3]. While sustainable energy economy is one of the major challenges faced by the world com- munity, CDCs have emerged as a major consumer of electricity. The number and of size of data centers has been increasing exponentially over the past decade to keep pace with the growing number of cloud based applications and users. CDCs are estimated to consume more than 2.4% of electricity worldwide with a global economic impact of $30 billion [4]. Despite advancements in IT equipment efficiencies, data center electricity consumptions are expected to grow 15-20% annually [5]. Additionally, CDCs are responsible for emission of GHG produced during the process of electricity generation, IT equipment manufacturing, and disposal [6,7]. It is estimated that the data centers were responsible for 78.7 million metric ton of CO2 emissions equal to 2% of global emissions in 2011 [8]. These figures advocate application of innovative and disruptive measures in CDCs for energy and carbon efficiency. Power Usage Efficiency (PUE) and Carbon Usage Efficiency (CUE) are commonly applied sustainability indicators in CDCs. PUE is defined as the ratio of total CDC energy usage to IT equipment energy usage [9,10]. Energy wasted in measures other than com- puting, such as cooling, leads to poor PUE values. CUE is the ratio of total CO2 emissions caused by CDC power consumption to total power used by the CDC. Complete dependency on fossil fuel based grid energy in CDCs leads to poor CUE values [11].

Sustainable and green CDCs necessitate application of multiple techniques and technologies to achieve lower energy costs and GHG emissions. The main elements of sustainable CDCs are [9]:

� On/off-site renewable energy generation techniques to reduce GHG emission. Renewable energy resource powered CDCs lead to lower GHG emissions while eliminating fossil fuel based energy resources.

� Waste heat recovery and free cooling techniques to lower cooling costs. Cooling costs which make up 40% of total CDC energy consumption on average. Both renewable energy and waste heat utilization techniques in CDCs are dependent on geo-dispersed MDC designs and virtualization based workload migrations.

� Transportable Modular Data Center (MDC) designs that facil- itate exploitation of renewable energy, waste heat, and free cooling opportunities in geo-dispersed locations.

� Virtualization based workload migrations that enable workload and resource management across geo-dispersed CDC nodes.

Renewable energy generation and free cooling techniques require ideal climatic conditions which are dependent on the location of CDC. Similarly, waste heat utilization requires co- location of CDC with places suitable for waste heat recovery opportunities, such as district heating. As MDC designs are based on shipping containers, they enable relocation of CDC nodes to places where sustainable computing opportunities are abundant. Hence, the opportunistic relocation of CDCs nodes is based on two factors: (a) on-site availability of renewable energy resource and (b) proximity to free cooling resources and waste heat recovery opportunities [12,13]. Sustainable CDCs are supported by and dependent on geo-dispersed MDC designs and virtualization based workload migration techniques. MDC shippable containers allow distribution of CDC nodes to optimal locations with sustainable computing opportunities.

Moreover, virtualization of CDC resources allows efficient migration of workloads between geo-dispersed data center nodes to pursue sustainable computing opportunities across the globe [14]. IT service providers, such as, Google and Facebook have also emphasized on migration from grid energy resources to renewable energy resources in geo- dispersed configurations [15,16]. Fig. 1

Fig. 1. Elements of sustainable Cloud Data Center Model.

J. Shuja et al. / Renewable and Sustainable Energy Reviews 62 (2016) 195–214 197

presents a model of green CDCs with application of techniques and technologies for sustainability.

To the best of our knowledge, this is the first survey on sus- tainable CDCs that covers all major factors of sustainability and green economy in the cloud. Previous surveys have largely focused on a single aspect of sustainable CDCs. For instance, Oro et al. [17] reviewed renewable energy integration schemes for CDCs. Ebra- himi et al. [5] presented a survey on waste heat opportunities in CDCs. Similarly, Ahmad et al. [18] surveyed the Virtual Machine (VM) based workload consolidation schemes in CDCs. A compre- hensive survey covering the major techniques and technologies of sustainable CDCs is not present in the literature. Furthermore, open research issues and challenges in context of sustainable CDCs need to be investigated in detail. The major contributions of this article are: (a) we classify state-of-the-art techniques and tech- nologies enabling sustainable CDCs, (b), we detail cases studies from IT industry and research community that advocate the application of sustainability measures for CDCs, (c) we present a survey of existing studies in sustainable CDCs, and (d) we highlight research challenges and issues in adoption of sustainable and green energy techniques and technologies among geo- dispersed CDCs.

The rest of the paper is organized as follows. Section 2 provides background knowledge to sustainable CDC techniques and tech- nologies. Section 3 presents case studies from leading IT compa- nies and research community that demonstrate the benefits of the integration of renewable energy, waste heat recovery, geo- dispersed MDC designs, and VM migration techniques in CDCs. In Section 4, we examine adoption of renewable in CDCs with corresponding taxonomy of solutions and summary of research issues. Section 5 investigates waste heat utilization opportunities in CDCs. In Section 6, we elaborate on MDC architectures and the corresponding server, network, and cooling designs. Section 7 reviews Wide Area Network (WAN) VM migration techniques in context of geo-dispersed CDCs. In Section 8, we debate on future research directions and open challenges in the field of sustainable CDCs. Section 9 provides the concluding findings of our study.

2. Background

In this section, we provide basic knowledge of sustainable and green CDCs. We provide brief summary of enabling techniques and technologies for sustainable CDCs, namely, renewable energy, waste heat utilization, modular CDC designs, and VM migration.

2.1. Renewable energy in CDC

Sustainable and green computing requires application of both energy efficiency measures and renewable energy resources to lower energy and carbon footprint [6,19]. Brown energy generated from fossil fuels, such as coal, gas, and oil results in large amount of CO2 emissions. On the other hand, green energy produced from renewable resources, such as water, wind, and sun results in almost zero CO2 emissions [20]. Hydroelectricity, although cate- gorized as green energy, is available only through grid electricity supplied by government corporations. On the contrary, solar and wind energy can be generated with both on-site installations or purchased from off-site corporations. The capital cost and unpre- dictability of renewable energy resources are barriers to their widespread adoption [21]. However, cost/Watt of renewable energy resources is estimated to half in the next decade [22]. The reduction in the cost/Watt of renewable energy is based on (a) advancements in capacity of materials, such as photovoltaic arrays, (b) increase in storage capacity of rechargeable batteries, and (c) monetary incentives by governmental organizations for the inte- gration of renewable energy resources [23]. The issue of unpre- dictability in renewable energy supply can be addressed by power- supply load balancing and workload migration techniques among geo-dispersed CDCs [24,25]. Moreover, hybrid grid designs that draw power from both steady grid resources and variable on-site renewable energy sources are essential to guarantee 100% avail- ability of cloud services [17]. However, abundant renewable energy resources are often located away from commercial CDC sites. Therefore, transportable MDC designs need to be utilized to locate CDC nodes near renewable energy resources [23]. The integration of renewable energy in CDC results in lower CUE metric. Higher capital costs and intermittency of renewable energy

J. Shuja et al. / Renewable and Sustainable Energy Reviews 62 (2016) 195–214198

resources remain a challenge for widespread adoption in CDCs [26].

2.2. Waste heat utilization in CDC

Fossil fuel deposits are diminishing at rapid pace calling for reuse of waste heat in all type of energy conversion systems. Most of the electric energy supplied to the CDC servers is converted into heat energy requiring deployment of large scale cooling systems to keep server rack temperatures in operational range [9]. As a result, 40-50% of the electricity consumed by CDCs is used to cool heat dissipating servers [5]. With advent of multi-core and stacked server designs, power densities of servers have increased, resulting in increased cooling costs. Minimizing the energy used in cooling can lead to significant impact on energy efficiency in CDCs [27]. However, reduction in cooling cost requires relocation of CDCs to places where free cooling resources are available in the form of lower environment temperatures [15]. Multiple geographically dispersed locations are also exploited for variable electricity prices [28]. Moreover, as most of the power supplied to the servers is dissipated as heat, CDCs can act as heat generators for many waste heat recovery techniques [5]. Waste heat can be ideally applied to vapor-absorption based CDC cooling systems. When heat is sup- plied to a refrigerant in vapor-absorption based cooling, it eva- porates while taking away heat from the system. In this manner, application of waste heat utilization and free cooling techniques results in ideal PUE values by neutralizing cooling costs while powering vapor-absorption based CDC cooling systems [29]. Heat generated by CDCs can also be supplied to district heating facilities in areas with lower temperatures. However, CDCs are often not located in proximity of waste heat recovery locations. Therefore, either CDCs have to apply waste heat to internal vapor-absorption based cooling system, or relocate to proximity of a waste heat recovery site. MDC shippable nodes are ideal to tap into waste heat recovery opportunities in geo-dispersed sites. Moreover, VM based workload migrations are also necessary to balance CDC load between geo-dispersed computing nodes [22]. The main challenge to waste heat utilization is low heat quality in CDCs and higher capital costs of heat exchange interfaces.

2.3. Modular CDC designs

CDCs need to intelligently tap into renewable energy resources and waste heat utilization opportunities present at sites that are often remote to commercial CDC buildings [30,13]. Modular Data Centers (MDC) enable location as an energy efficiency measure as they are built inside shipping containers that can be transported to remote locations. The container based MDC design offers two desirable properties for sustainable CDCs. Firstly, the shippable nature of MDC allows cloud providers to relocate their compute facilities to geo-dispersed locations abundant with sustainability opportunities. Secondly, the container based closed looped system of MDC is ideal for application of free cooling and waste heat utilization measures [12]. The container design can efficiently perform hot-aisle containment so that high grade waste heat can be captured from the servers. Hot-aisle containment also leads to better cooling efficiency resulting in lower operational costs. In a generic MDC design, computing and cooling devices are setup inside the container before shipment to a remote location. MDC nodes provide flexibility to cloud service providers with service- free design as computing resources are setup before deployment and not repaired or upgraded upon failure. The MDC node is kept in service until the assembled components provide a minimum level of computational output [31,32]. MDC nodes can be operated as continuously moving entities searching for opportunistic

sustainability options, or static entities that are operated from a location that has redundant sustainability options for CDCs [30].

2.4. VM migrations

Virtualization technology lies at the core of CDC infrastructure while providing resource management, resource consolidation, and migration for energy efficiency and fault-tolerance [1,9]. Vir- tualization adeptly manages existing cloud resources through highly dynamic resource provisioning to significantly reduce operational costs. Intermittent nature of renewable energy resources and decentralized MDC nodes necessitate workload migration while balancing workload among multiple geo- dispersed nodes. Virtual Machine (VM) migration techniques enable migration of workloads when on-site renewable energy generation is low and available elsewhere in geo-dispersed sites. Similarly, virtualization also enables workload migration between distributed MDC nodes where some nodes leverage on-site renewable energy while other nodes employ nearby waste heat utilization opportunities [33,34]. Moreover, VM based workload migration and consolidation techniques are utilized to pack a set of VMs to fewer number of physical devices to balance renewable power generation and workload demand [35]. Researchers have leveraged both MDC designs and VM migration techniques to efficiently harness renewable energy resources and waste heat utilization alternatives in green CDCs [36–38]. However, the cost, in terms of network delay and energy consumption, between geo- dispersed nodes is the foremost challenge to VM based workload migrations in CDCs.

3. Case Studies

The relationship between sustainable CDCs techniques and technologies is established and complemented by several case studies carried out by the IT industry and published in scholarly articles. Many IT companies including Apple, Google, and Facebook have added green and sustainable CDC nodes to their expanding infrastructure [39]. In this section, we will present the case studies that report significantly efficient PUE values while leveraging multiple sustainability measures, such as renewable energy, MDC design, and waste heat recovery. Table 1 summarizes the cases studies of sustainable CDCs discussed in this section.

3.1. Parasol

Parasol [23] is a green CDC prototype based on four key tech- niques and technologies of sustainable CDCs, namely, MDC design, on-site renewable energy generated through solar panels, free cooling, and net-metering. A one year case study was conducted with a MDC powered by on-site solar panels set on roof-top of a building located in New Jersey. Parasol works on dynamic load balancing of CDC power between renewable and grid energy defined by the CDC workload. The MDC container consists of two server racks that are free cooled whenever possible. The workload and power source scheduling is based on workload and power predictions, existing power stored in batteries, analytical models of power consumption, peak power, and power costs. Excessive renewable energy is either stored in batteries or net-metered to the grid. The experimental results show 36% and 13% error respectively in workload and solar power generation prediction in a 1 hour prediction time frame. The total grid electricity cost was reduced by 75% in the parasol design. Moreover, the Parasol design can amortize the capital cost of a solar setup without batteries in 4.8 to 7.1 years with 60% government incentives. The study esti- mated that the efficiency of the photovoltaic technology (multi-

Ta b le

1 C as e st u d ie s o f su

st ai n ab

le C D C s.

C a se

st u d y

O b je ct iv e

Su st a in a b il it y m ea

su re s

A p p ro

a ch

A ss es sm

en t

P ar as o l

Sc h ed

u le

w o rk lo ad

b as ed

o n w ea

th er , tr ac e,

an d

b at te ry

le ve

l in p u ts

M D C , re n ew

ab le

en er g y,

h y b ri d g ri d ,

fr ee

co o li n g , n et -m

et er in g

O p er at e a M D C o n a ro o ft o p fo r m ax

im u m

so - la r

co ve

ra g e

P U E va

lu e 1. 1,

7 5 % lo w er

en er g y co

st s, ca p it al

co st s o f so la r

en er g y m ay

h al f in

n ex

t 15

ye ar s

Fr ee

Lu n ch

U ti li ze

re n ew

ab le

en er g y av

ai la b le

in ge

o -d is -

p er se d M D C n o d es

p la ce d in

d if fe re n t ti m e

zo n es

R en

ew ab

le en

er g y,

ge o - d is p er se d C D C ,

M D C d e-

si g n , V M

m ig ra ti o n s

M ig ra te

V M s w h er e fr ee

re n ew

ab le

en er g y is av

ai l-

ab le

u si n g d ed

ic at ed

n et - w o rk

an d li m it u se

o f

g ri d en

er g y

W h en

re n ew

ab le

en er g y is

lo w

at o n e C D C n o d e,

m o st

o f

th e ti m es

th e w o rk lo ad

ca n b e m ig ra te d to

an o th er

n o d e

A q u as ar

A n al y ze

w at er

an d ai r co

o le d Su

p er co

m p u te r fo r

w as te

h ea

t re co

v er y

W as te

h ea

t re u se , M D C d es ig n , fr ee

co o li n g

C o o l a M D C w it h w at er

th at

is h ea

te d b y th e se rv er

h ea

t d is si p at io n

P U E 1. 15

, 3 4 % in cr ea

se s in

ex er g et ic

ef fi ci en

cy an

d 8 0 %

d is si p at ed

h ea

t re u se d fo r co

o li n g

M D C w it h fr ee

co o li n g

C o m p ar e M D C an

d co

n v en

ti o n al

C D C d es ig n s fo r

fr ee

co o li n g an

d en

er g y ef fi ci en

cy M D C d es ig n , fr ee

co o li n g , ge

o -d is -

p er se d , v ir tu al iz at io n

Si m u la te

si n g u la r an

d m u lt ip le

ge o -d is p er se d d at a

ce n te r w it h d if fe re n t d es ig n s fo r ev

al u at io n

P U E o f 1. 3 5 , 3 4 % in cr ea

se s in

ex er ge

ti c ef fi ci en

cy , M D C

ac h ie ve

s u p to

4 4 % en

er g y sa v in g as

co m p ar ed

to co

n ve

n -

ti o n al

C D C d es ig n

Fa ce b o o k A rc ti c

C D C

O p er at e a C D C in

A rc - ti c re g io n to

lo w er

co o li n g

co st s

R en

ew ab

le en

er g y (h y d ro ), M D C

d es ig n , fr ee

co o li n g , ge

o -d is p er se d

U ti li ze

A rc ti c ai r an

d w at er

fr o m

n ea

rb y ri ve

r fo r

fr ee

co o li n g w h il e u si n g g ri d en

er g y b as ed

o n

h y d ro -p o w er

P U E va

lu e 1. 0 7 an

d 7 0 % re d u ct io n in

n u m b er

o f b ac k u p

ge n er at o rs

d u e to

h y d ro -p o w er

G re en

H o u se

D at a

O p er at e at

10 0 % re n ew

- ab

le en

er g y

R en

ew ab

le en

er g y,

M D C ge

o -d is p er se d

n o d es , fr ee

co o li n g , v ir tu al iz at io n

P ro v id e 10

0 % su

st ai n ab

le cl o u d se rv ic es

P U E va

lu e o f 1. 14

, 6 4 .5 % re d u ct io n in

en er g y co

st s,

el im

- in at io n re d u ce d ca r-

b o n em

is si o n s

J. Shuja et al. / Renewable and Sustainable Energy Reviews 62 (2016) 195–214 199

crystalline silicon) will increase from 15% to 25% by 2030. More- over, it is estimated that on current space and capacity values, the space required by the solar panels to power a CDC is 47 times larger than that occupied by the racks. However, with the increasing capacity factor of solar technologies, the space requirement can decrease to 24 times by 2020-2030. According to the case study, the installed cost of solar energy will decrease by 50% by 2030.

These forecasts depict that the cost and space requirements of sustainable CDCs will decrease significantly over the next decade.

3.2. Free lunch

Free Lunch [40] is a MDC architecture evaluated to experiment with the viability of sustainable CDC elements. Free lunch is based on three principles of sustainability: (a) utilization of on-site renewable energy through remote geo- dispersed CDCs, (b) dedi- cated high speed network connectivity between two CDC nodes, and (c) VM based workload migrations. The study identified vir- tualization, MDC architecture, and renewable energy as key enabling technologies for sustainable CDCs. The authors chose two locations (near the Red Sea and the Southwest of Australia) ideal for harvesting solar energy that are situated in different time and climatic zones to complement each other. Moreover, wind turbines of 1.5 MW power were modeled with year average climatic con- ditions. The study assumed 10 Km2 of solar cells with 10% effi- ciency. It was found that throughout the year the power generated by the renewable energy sources dropped 615 times below the average demand (150 W per server). At 331 of these instances, excessive power was available at the other CDC node. However, on the remaining 284 instances, the servers have to be powered off. To migrate the workload between CDC nodes during renewable energy outages, a worst case of 677ms downtime can occur while considering a 10 Gbps wide-area network spanning 10,000 miles. A total of 416.4 s downtime/year resulted from the workload migrations which leads to 99.5% uptime/year. Moreover, each VM migration costs additional 0.16 kWh of energy. The results show that it is feasible to operate CDCs solely on renewable energy resources in geo- dispersed configurations while migrating work- loads over dedicated high-speed networks.

3.3. Aquasar

IBM carried out a case study on hot water cooled CDC prototype to evaluate waste heat reuse in minimal carbon footprint CDCs [41]. Water cooled IBM blade server based supercomputer was analyzed for waste heat recovery and exergy profile. The super- computer consisted of 42 blade servers equally placed in three chassis of which two are cooled by hot water. The third chassis utilizes air based cooling to facilitate the comparison of water and air based cooling techniques. It was found that for water based cooling chip/coolant temperature differential of only 15◦C was sufficient while for air based cooling it had to be increased to 35◦C. Hot water at 60◦C flowing in micro-channels attached to electronic chips of the supercomputer is used as coolant. The hot water further gains exergy from the system and drops some portion of it in the waste heat recovery cycle that heats a building. As a result, cooler water is returned back to the Supercomputer. It was found that utilizing hot water based cooling results in 34% increases in exergetic efficiency. Moreover, the higher exergetic efficiency can increase the possibility of waste heat utilization to 80% of the recovered heat for space heating. The PUE of 1.15 for the Aquasar design shows an ideal approach for energy efficiency in CDCs.

J. Shuja et al. / Renewable and Sustainable Energy Reviews 62 (2016) 195–214200

3.4. MDC with free cooling

Researchers [42] carried out a case study to quantitatively compare MDC and bricks and mortar CDC designs for cooling efficiency. The study was carried out to investigate CDC tempera- ture profiles, characterize power efficiency, and study impact of virtualization and consolidation on cooling power in singular and multiple interconnected MDC configurations. The data center profiles were based on multiple data center locations, namely, NASA, ClarkNet, UC Berkeley, Calgary, and Saskatchewan. The MDC design reduces inter-rack air recirculation with hot-aisle contain- ment that allows raising the maximum inlet air temperature to 32 C. On the other hand, conventional bricks and mortar CDC design does not provide efficient air containment resulting in undesired air recirculation. As a result, the inlet air temperature needs to be lowered while higher energy is consumed in the cooling systems. The case study found that workload consolidation on the least number of MDC containers leads to 81% and 44% more efficiency than raised- floor bricks and mortar data centers in terms of cooling and power consumption respectively. An average PUE of 1.35 is observed for both singular and multiple MDC designs. Overall, the study found that MDCs achieve 44% and 13.7% power savings at low and high workloads respectively, as com- pared to raised-floor bricks and mortar CDC.

3.5. Facebook Arctic CDC

In 2013, Facebook added a new CDC node near the Arctic circle in Sweden to its growing number of CDCs [43]. The CDC is built with the aim to be one of the most sustainable CDC in the world. The arctic CDC is powered by locally generated hydro-electric energy that is nearly 100% renewable. The hydro-electric energy leads to more reliable power generation as compared to solar and wind powered CDCs. As a result, the design leads to 70% reduction in number of backup generators at the CDC. The arctic CDC opportunistically utilizes cool water from a nearby river and chilly Nordic air for free cooling. The water is directly pumped from the river and used in water based cooling systems. Most of the hard- ware equipment used inside the CDC is based on Open Compute Project designs which where developed for high energy efficiency. Experiments show that the arctic CDC has an average PUE of 1.07 by almost eliminating the cooling costs which is a high indicator of sustainability.

3.6. Green House Data

Green House Data1 is the leading example in industrial and commercial utilization of sustainable CDC techniques and tech- nologies. The main objective of the Green House Data is to operate its CDC with wind energy while benefiting from complimentary techniques and technologies, such as, opportunistic free cooling, virtualization, MDC design, geo-dispersed nodes, and Hot/cold aisle containment. Taking advantage of the aforementioned mea- sures, the CDC operates at near ideal PUE of 1.14. Virtualization based resource consolidations leads to lower energy costs while increasing CPU utilization by 40-60%. MDC design and hot/cold aisle containment ensures the cold air entering server rows and the hot air exiting them do not mix, increasing the cooling effi- ciency. It is estimated that the Green House Data center saves up to 64.5% of energy as compared to an average CDC. Moreover, assuming a 10 MW facility, 100,000 metric ton of CO2 emissions can be reduced with the Green House Data center design.

1 https://www.greenhousedata.com/green-data-centers.

4. Renewable Energy based CDCs

Renewable energy resources, such as, wind and solar energy are considered as a substitute to conventional brown energy resources that result in GHG emissions. Researchers have addres- sed various challenges to adoption of renewable energy resources for IT industry in general, and CDCs in particular. The main chal- lenges faced in adoption of renewable energy for CDC are: (a) uncertain and intermittent nature of renewable energy resources, (b) capital costs and space requirements of renewable energy generation and storage, (c) and proximity of renewable energy based CDCs to end users [22,20]. While the space and capital cost of renewable energy are explicitly high, future trends in this regard are encouraging. Both space and capital cost of renewable energy is expected to half in the next decade [23]. Moreover, over a time period of 7-12 years, the capital cost of on-site renewable energy deployments is estimated to be amortized due to lower operational costs [44]. These estimates further fortify the vision that green energy resources should be integrated into CDC designs. Furthermore, sustainability options are often located in remote areas while cloud service providers prefer to host their applica- tions in the proximity of dense metropolitan areas. For example, relocating cloud services to South Australia (plenty of solar energy opportunities) will increase the end-to-end delay for majority of the users. Therefore, trade-off exists between adoption of sus- tainability measures and proximity of cloud services to end users [45].

Sustainable CDCs apply workload scheduling techniques to address the issue of instability of renewable energy generation. Renewable energy schemes in CDCs can be classified on workload scheduling techniques, namely, dynamic workload-power balan- cing, follow the renewable approach, and server power-capping. Dynamic load balancing technique balances the workload within a CDC according to the available on-site renewable energy [20]. In follow the renewable approach, workload is migrated between geo-dispersed CDCs based on the availability of renewable energy. Power-capping based workload scheduling techniques apply vol- tage scaling and power transitions to match available renewable energy with CDC workload. These workload scheduling techniques are sub-classified based on energy source and energy locality with respect to the CDC. Renewable energy sources are mainly solar, wind, grid or a hybrid of these sources. Depending entirely on renewable energy can lead to power outages and consequent service downtime. Therefore, most of CDCs deploy a hybrid grid of brown and renewable of these. There exist three methods to acquire renewable energy for green CDCs: (a) co-location of CDC with renewable energy generation plant, (b) on-site self-genera- tion of renewable energy, and (c) purchase of renewable energy in the form of renewable energy certificates and power purchase agreements from renewable power vendors [46,36]. The advan- tage of co-location and on-site self-generation is that energy transmission and storage losses are minimized. However, while on-site self-generation incurs high capital costs, co-locating might not necessarily satisfy other CDC objectives, such as end user proximity and network latency. Purchase of renewable energy products avoids capital and operational costs of on-site self-gen- eration. However, such products are only available at few geo- graphic locations. Moreover, power transmission and storage los- ses occur along with higher costs of renewable energy than con- ventional brown energy [20,47]. Excessive renewable energy produced on-site can be saved either by (a) installing costly energy storage devices or by (b) net-metering. Net-metering allows addition and distribution of excessive energy generated on-site to local power distribution system. The taxonomy of renewable energy based CDCs is illustrated in Fig. 2.

Fig. 2. Taxonomy of renewable energy based CDC.

J. Shuja et al. / Renewable and Sustainable Energy Reviews 62 (2016) 195–214 201

4.1. Design

Energy generated from resources, such as, wind and sun is intermittent and variable. There are multiple techniques to ensure power quality and stability of renewable energy in sustainable CDCs. These techniques are; (a) Combined cooling, heating and power systems (CCHP), (b) Automatic Transfer Switches (ATS), and (c) dynamic workload scheduling techniques. CCHP systems have been deployed in CDCs with positive impact on power quality and overall efficiency [48]. CDCs have stable electrical and thermal demands which can be met by properly sized CCHP systems with high efficiency. CCHP systems produce electricity and heat at the same time. CCHP systems can provide high reliability in on-site power generation, thereby greatly reducing the risk of power outages. Moreover, if CCHP systems are used as the energy supply systems for data centers, the cooling costs can be reduced by recovering the waste heat from the power generation through an absorption cycle [49].

Hybrid power supply designs are essential to ensure availability of cloud services while catering intermittency of renewable energy resources. Integration of on-site renewable energy requires addi- tion of ATS to CDC power delivery infrastructure [50]. ATS manage power from grid and renewable resources based on pre-configured power-transfer threshold. The ATS switches to a secondary energy source if the power supply from the primary source falls below the pre-configured threshold. The utilization of renewable energy depends on the CDC workload and total renewable power gen- eration. In case the renewable energy does not meet the power demand of CDC, ATS are required to shift power supply of the CDC to grid electricity. On the contrary, light CDC workloads may not utilize all of the available renewable energy. In such scenarios, excessive renewable energy can be saved either by energy storage devices or by net-metering [51].

Conventional CDC designs include multiple power backups in the form of uninterrupted power supply (UPS) and diesel gen- erators. However, such measures are expensive and provide power backup for short and medium power outages. Sustainable CDC designs can result in reduction of power backup systems due to availability of alternate power resources in the form of renewable energy [36]. Moreover, net-metering can offset grid energy costs of the CDC [46]. Additionally, renewable energy resources can sup- plement traditional backup power plans, such as diesel generators.

However, there are drawbacks to both renewable energy storage and net-metering [52–54]. Storage batteries have limited lifetime and incur power losses due to self-discharge. Moreover, capital cost of renewable energy can double due to integration of storage batteries [54]. On the other hand, while net-metering might not be available option in most parts of the world, it also incurs power lose during voltage transformations and transmission [44]. Fig. 3 depicts a hybrid CDC power generation and delivery architecture that is based on both on-site renewable energy and grid electricity.

4.2. State-of-the-Art

Contrary to the conventional grid energy that is constant, renewable energy is variable and depends on weather conditions, such as, wind speed and solar irradiance. Similarly, CDC workloads are variable and require dynamic power provisioning [55]. Therefore, there is an opportunity to solely depend on renewable energy resources while matching variability of renewable energy resource to CDC workloads. However, the peak workload and renewable power generation may occur in different time-frames requiring efficient storage of power or/and supplementary power from conventional electricity grid. For example, wind energy can peak at night while the CDC workload is light. The approaches to renewable energy aware workload scheduling of CDCs can be categorized as: (a) load balancing between grid and renewable energy based on workload demand and power supply [44], (b) following the renewable resources across geo-dispersed CDCs [56], and (c) Dynamic Voltage and Frequency Scaling (DVFS) based power capping based on available renewable energy [54].

4.2.1. Dynamic load balancing Dynamic load balancing between grid and renewable energy is

the most commonly applied technique for green scheduling of cloud resources [44,57–59,54,60]. Dynamic load balancing tech- nique matches renewable energy supply to CDC workload. This involves: (a) supply side prediction while predicting the amount of renewable energy based on weather data and (b) demand side prediction while characterizing the CDC workload based on traces [44]. Deadline sensitive jobs are aggressively perused during high green energy availability. However, grid energy has to be applied for such jobs if renewable energy is not available. Li et al. [57] proposed iSwitch; a cooperative renewable supply-load balancing

Fig. 3. Hybrid energy generation model for CDC.

J. Shuja et al. / Renewable and Sustainable Energy Reviews 62 (2016) 195–214202

scheme. iSwitch applies wind energy and advocates against con- tinuous supply-load tuning activities that degrade system perfor- mance. Gird electricity is utilized instead of pushing servers to low power states during low green power generation periods. The design employs a hybrid grid-renewable energy power supply mechanism where server racks are either powered by grid energy or renewable energy. During low renewable energy phase, VM migrations are utilized to shift workload from renewable energy powered racks to grid powered racks.

Researchers have proposed a CDC design that leads to net zero grid energy utilization during CDC life-cycle while utilizing on-site solar energy for: (a) powering the CDC, and (b) adding redundant renewable energy to the grid to offset grid electricity costs [58]. The Life-cycle Assessment (LCA) based approach limits both grid energy utilization to reduce operational costs and renewable energy storage to reduce capital costs of storage batteries [61]. CDC workloads with majority of jobs categorized as deadline insensi- tive are suitable for dynamic grid-renewable based load balancing schemes. The drawbacks of green load balancing techniques are [20]:

� CDC tasks are composed of characteristically dynamic jobs usually requiring time critical response. Fluctuations in renew- able energy supply can affect deadline constrains and violate SLAs.

� Hybrid power utilization techniques are based on single CDC. Such techniques miss the opportunity to utilize renewable energy that may be available elsewhere in multiple geo- dispersed CDC environments.

� While CDC workloads may be scheduled at daytime to take advantage of redundant solar energy, cooling costs are also increased as daytime temperatures are high. Similarly, although windy nights can generate redundant renewable energy, CDC workloads usually peak in day hours.

� Hybrid power architecture does not eliminate the dependency on grid electricity. Therefore, carbon emissions are reduced but not eradicated.

4.2.2. Follow the renewables Second approach for green workload scheduling is migrating

tasks between geo-dispersed CDCs while following the renewable

resources [56,50,62,63]. This approach requires that dynamic load balancing mechanism be aware of the availability of renewable energy at geo-dispersed CDCs. Dynamic request routing is required to route user requests towards CDC with abundant renewable energy. Moreover, the renewable aware routing mechanism can be guided by fine-grained request level energy profiles [50]. Liu et al. [56] proposed follow the renewable energy model based on availability of wind and solar energy at geographically distributed CDCs. The model optimizes a social objective that is a trade-off between the green energy utilization and delay in user response. However, the proposed model does not consider electricity prices and workload migration costs across geo-dispersed CDCs. Elec- tricity price variation corresponding to geographical locations adds another factor to multi-objective sustainable cloud service placement model. Researchers [62] proposed a workload sche- duling algorithm for CDCs located in different time-frames based on availability of solar energy and cooling costs for delay-tolerant workloads. The simulation results show that as the number of CDCs increases, the likelihood of finding CDC that meets con- straints of abundant renewable energy, minimal delay, and low cooling cost increases. Moreover, as the migration time between geo-dispersed CDCs increases, the number of tasks suitable for migration decreases. The drawbacks of follow the renewables approach are [56]:

� The approach consumes more energy due to overhead of workload migrations over the WAN between geo-dispersed CDC nodes.

� Dedicated communication links are required between geo- dispersed CDC. Otherwise the VM migration time is prohibitive for delay sensitive workloads.

� Factors, such as, user proximity and electricity prices might not be in favor of follow the renewables approach in geo-dispersed scenarios.

4.2.3. Renewable based power capping Third approach to renewable energy aware workload schedul-

ing in CDC is power capping based on available renewable energy [54,64,65]. The power capping approach applies DVFS and server power state transitions to match CDC power demand-supply. Li et al. [54] proposed Solarcore, a multi-core power capping scheme

te ch

n iq u es

in C D C s.

O b je ct iv e

E n er g y so

u rc e

W o rk

lo a d

Se tu

p G eo

- d is p er se d

Fr ee

co o li n g

B al an

ce D C p er fo rm

an ce

an d su

p p ly -d em

an d lo ad

b al an

ci n g

H y b ri d : W

in d , g ri d ,

b at te ry

M ix ed

- cr it ic al it y

4 8 0 0 se rv er

p ro to ty p e

N o

N o

P re ci se

en er g y p re d ic ti o n to

re d u ce

d ea

d li n e v io la ti o n s

H y b ri d : So

la r, w in d , an

d g ri d

M ix ed

-c ri ti ca li ty

2 0 0 se rv er

p ro to ty p e

N o

N o

K ee

p tr an

si en

t se rv er s to

ac t as

b ac k u p in

ca se

o f lo w

re - n ew

ab le

en er g y

ge n er at io n

H y b ri d : So

la r, w in d , g ri d

M ix ed

- cr it ic al it y

2 0 b la d e se rv er

cl u st er

N o

N o

D em

an d -r es p o n se

b as ed

en -e rg y p ri ci n g an

d o p ti m iz at io n

H y b ri d , n et - m et er in g

In te ra ct iv e w eb

-b as ed

Tr ac e d ri ve

n si m u la ti o n s

Y es

N o

M in im

iz e b ro w n en

er g y co

n si d er in g b at ch

jo b s

So la r

M ix ed

-c ri ti ca li ty

Si m u la ti o n s

Y es

N o

O p ti m iz at io n b as ed

o n SL

A s,

re n ew

ab le

av ai la b il it y,

an d m o n th ly

b u d g et

W in d , so la r

W eb

re q u es ts

Si m u la ti o n s

Y es

N o

M ax

im u m

re n ew

ab le

en er g y u ti li za ti o n ac ro ss

m u lt i- co

re s w it h w o rk lo ad

o p ti m iz at io n

H y b ri d : So

la r, g ri d

M ix ed

- cr it ic al it y

Si m u la ti o n s

N o

N o

Tr an

si ti o n fr o m

h ig h to

lo w

p o w er

b as ed

o n g re en

en er g y av

ai la b il it y

So la r, w in d , b at te ri es

W eb

b as ed

st o re -k ey

re q u es ts

P ro to ty p e

N o

N o

A ch

ie v e b es t ra ti o o f re n ew

-a b le

en er g y to

w o rk lo ad

H y b ri d g ri d an

d n et -

m et er in g

M ix ed

- cr it ic al it y

Si m u la to r

N o

N o

J. Shuja et al. / Renewable and Sustainable Energy Reviews 62 (2016) 195–214 203

where Photovoltaic (PV) arrays are operated at maximum power and multi-core server powers are adjusted accordingly. Direct coupled solar PV panels are used to avoid costly storage batteries. The server power supply units draw power from both grid and solar panels. The Solarcore design maximizes solar power utiliza- tion while enforcing DVFS along with monitoring workload per- formance. Researchers proposed Blink, a server power state tran- sition design to cater intermittent renewable energy power gen- eration [64]. Rather than applying workload driven state transi- tions, Blink design advocates renewable energy driven state transition. The Blink design is applied to a memcached application which stores key-value pairs on distributed servers, thus, allowing for selection of active state servers to service requests. There are several drawbacks of power capping approach that focus on higher renewable utilization [66,9]:

� System performance and response time degrades due to power capping mechanism.

� Power capping is only applicable to IT equipment while cooling units can consume as high as 40% energy on average in CDCs.

� Deadline sensitive workloads are not suitable for such approach as renewable energy outages are usually of long duration.

Table 2 provides a comparison of the aforementioned renew- able energy aware workload scheduling techniques. Most of the studies utilize a hybrid renewable-grid energy based power gen- eration. Moreover, state-of-the-art studies har- ness MDCs, free cooling opportunities, and geo-dispersed locations to assist CDCs to achieve optimal sustainability metrics.

Integrating renewable energy resources to CDC designs leads to sustainable operations. Renewable energy can offset the carbon emission caused by utilization of brown energy resources. More- over, redundant on-site renewable energy generation can be uti- lized in net-metering to lower the grid energy costs. However, ATS and CCHP systems need to be integrated to conventional CDC power supply designs to carter for variance in renewable energy generation through hybrid power supply mechanisms [54]. Dynamic load balancing, follow the renewables, and renewable power generation based server power capping approaches are essentially applied to sustainable CDC designs due to unpredict- able nature of renewable energy generation [20]. Similarly, con- ventional commercial CDC locations are of- ten not suitable for renewable energy generation. For example, major IT firms and their CDCs are located in Silicon Valley. However Silicon Valley experiences only 5.8 hours of sunshine each day on average [67]. In such cases, either the CDC node needs to be relocated to a remote location with redundant renewable energy, or off-site renewable energy purchased without relocation [30]. Transpor- table MDC designs enable relocation of CDC nodes to remote lo- cations for on-site renewable power generation. Moreover, work- loads migrations between geo-dispersed CDC nodes are facilitated by the virtualization technology [14].

Ta b le

2 R en

ew ab

le en

er g y aw

ar e w o rk lo ad

sc h ed

u li n g

Te ch

n iq u e

St u d ie s

D y n am

ic lo ad

b al an

ci n g

iS w it ch

[5 7 ]

R ef . [5 9 ]

Y an

k [6 0 ]

Fo ll o w

th e

re n ew

ab le s

G eo

-l o ad

b al - an

ci n g

[5 6 ]

G re en

-a w ar e [6 2 ]

G re en

w ar e [6 3 ]

P o w er

ca p p in g

So la rc o re

[5 4 ]

B li n k [6 4 ]

R eR

ac k [6 5 ]

5. Waste heat utilization in CDCs

The concept of waste heat utilization or energy re-use is another innovative direction for sustainable CDCs. Waste heat utilization calls for measures that capture the heat dissipated by the electronic components and utilize it into some useful work [29]. Waste heat utilization can provide benefits, such as: (a) smaller carbon footprint while replacing other heat production processes, such as gas heater that emit carbon, (b) reduced load on cooling equipment as heat energy can be applied to vapor- absorption based cooling process, and (c) lesser operational costs with additional revenue generation from selling waste heat energy

J. Shuja et al. / Renewable and Sustainable Energy Reviews 62 (2016) 195–214204

[68, 5]. The Green Grid defined a metric, called Energy Re-use Effectiveness (ERE) to measure the waste heat utilization of a CDCs [69,70]. ERE supplements the PUE metric while quantifying the power saved by utilizing CDC waste heat.

The law of conservation of energy states that energy can change form but can neither be created nor destroyed. To better capture the state of a system in terms of useful energy, exergy is defined. The amount of useful work a system can perform before coming into a state of equilibrium is called exergy [71]. From a CDCs point of view, exergy is the amount of work that can be done before the heat dissipating electronic components and the cooling equipment drive the environment into a state of equilibrium. Therefore, the heat dissipated from high power density compo- nents can be partially utilized to perform some useful work. The amount of work done in such scenario depends on the quality of the waste heat or the exergy of the system [29]. The main objec- tive of waste heat utilization techniques in CDC is to either heat a co-located building or utilize the heat in vapor-absorption based cooling systems. Waste heat utilization techniques can be based on air cooled, fluid cooled, or hybrid cooling based mechanisms. Fluid and hybrid systems have been proved as more efficient for cooling in CDCs [9]. Moreover, the heat exchange mechanism between the CDC and the heat recovery entity can be air to air, air to fluid, fluid to air, or fluid to fluid. Similarly, the waste heat recovery entity can be co-located with the CDC or the heat can be transferred to an off-shore site from the CDC. Substantial amount of heat is lost while during transfer from CDC node to heat recovery site. The taxonomy of waste heat utilization techniques in CDCs is depicted in Fig. 4.

5.1. Design

Waste heat generated from CDCs is categorized as low grade heat [3]. The low grade waste heat of CDCs can be utilized in eight waste heat recovery techniques: district heating, absorption refrigeration, thermal Rankine cycles, organic Rankine cycles, piezoelectric, thermoelectric, bio-mass processing, and water

Fig. 4. Taxonomy of waste heat u

desalination [5,3]. Each of the aforementioned waste heat recovery technique requires a minimal threshold of heat quality. However, waste heat quality is limited to 85◦C in most CDC configurations due to operational requirements of electronic equipment [5]. Operating CDC servers at higher thermal profile results in device failures and malfunctioning [9,72]. Therefore, the main application of waste heat in CDCs is vapor-absorption based refrigeration systems due to low quality of waste heat [29]. Moreover, the applicability of heat recovery techniques also depends on their proximity to the CDC and potential economic value [41]. The optimal solution is to utilize waste heat on-site to avoid further degradation of heat quality.

Over the past decade, CDC power densities have increased ten folds from 450-860 W/m2 to 6400-11000 W/m2 [3,5]. As a result, CDCs are moving away from conventional air cooled systems to water cooled and hybrid cooling systems due to higher thermal conductance, heat capacity, and density of water [58]. The thermal conductivity and density of water are 25 time and 800 more than that of air [29]. Efficient cooling of CDCs leads to better good put in terms of server performance. Water cooled servers deploy inno- vative designs in which micro-channel heat sinks are attached to electronic boards to dissipate heat from the chips [41]. While cold water circulates in the micro-channels, it takes away heat. As a result, the hot water can be used as low grade heat recovery source. On the contrary, hot air can be captured in chilling towers of air cooled CDC. Similarly, the overall workload of the CDC effects waste heat utilization. If the CDC workload is low, the servers have lower utilization and do not dissipate high amount of heat. Otherwise, if the CDC servers operate on full utilization, large amount of heat is dissipated. Hence, high workloads can lead to better waste heat quality. The main implication of workload fluc- tuations on cloud services is that higher workloads lead to higher resource contention. As a result, end to end delay in cloud services can increase [73,74]. Fig. 5 explains the common working of a hybrid cooling system with district waste heat recovery technique.

tilization techniques in CDC.

Fig. 5. Hybrid Cooling with waste heat utilization.

J. Shuja et al. / Renewable and Sustainable Energy Reviews 62 (2016) 195–214 205

5.2. State-of-the-Art

A case study of Cray XE6 Supercomputer named Lindgren located in Stockholm that provides waste heat energy for heating of neighboring building was presented in [3]. The case study was carried out to evaluate the feasibility of a supercomputer heating adjacent building in winters. Lindgren supercomputer consisted of 36,384 CPU cores that were cooled by pumping air at 16◦C from raised floor underneath the Cray XE6 racks. The hot air (40◦C) is captured in large air ducts that are attached on top of the racks. The air ducts are connected to air-water heat exchangers that receive cold water as input. The heat exchanger output air at room temperature (21◦C) back to the Computer Room Air Conditioning (CRAC) and warm water (35◦C) to the neighboring building for district heating. The system takes advantage of the ideal location of Cray XE6 Supercomputer neighboring a building requiring dis- trict heating while facing low winter temperatures most of the year. Lie et al. [68] coined the term of Data Furnace that constitutes of heat dissipating servers in a CDC. A mid-sized CDC with 100 kW power density acts as a Data Furnace while being relocated to an office building that requires heating during winters. In this man- ner the problem of heat dissipation that requires energy wasted in cooling measures can be turned into an advantage. Moreover, relocation of CDC to residential proximity improves quality of service to the end user by reducing network costs. However, the disadvantages of relocating a CDC to residential area are [33],

� Higher cost of commodities such as electricity in residential locations.

� Need of physically secured perimeter for each geo-dispersed CDC node.

� Limitation of network bandwidth and power supply infra- structure in residential areas.

Haywood et al. [29] proposed a CDC with model that can achieve ideal PUE (r 1) while applying waste heat utilization techniques and redundant on-site renewable energy generation to offset the energy consumed from public utility grid. The proposed CDC design utilizes water based cooling which captures 85% of heat generated by the servers and efficiently transfers it to a lithium bromide-water chiller. The waste heat activated lithium bromide- water chiller helps in limiting amount of energy spent on cooling CDC while achieving a very efficient PUE value. Solar thermal energy is added to the chiller to increase the waste heat

quality and drive the PUE below the ideal value of 1. However, this design requires the servers to operate in thermal range of 70-90◦C which is higher than ASHRAE recommendations and can lead to higher device failures rates [75]. Moreover, the coefficient of heat capture fraction from servers is set for ideal scenario that requires expensive thermal interfaces. The applicability of waste heat uti- lization techniques has been proved by several such CDC powered district heating implementations in Scandinavian countries [76,77].

Woodruff et al. [78] proposed Environmentally Opportunistic Computing (EOC) model that realizes containerized data centers as geo-disbursed heat providers. To study the results of EOC, an MDC node prototype was integrated to a local greenhouse at University of Notre Dame, Indiana, USA. The MDC node utilizes free cooling from environment and exhausts warm air to the greenhouse. The MDC node is able to deliver 15-40 kW of waste heat to the greenhouse that offsets heating costs in cold months. The study also modeled EOC with distributed CDC nodes opportunistically placed near office buildings. The study modeled a 3 MW CDC with an average PUE value of 1.8. Distributed CDC nodes are modeled having 10% of the CDC load. The model shows that 2.5-17% energy savings in heating expenditure can be made while relocating CDC nodes to commercial buildings. Researchers [73] studied the triad relationship between CPU utilization, CPU temperature, and ther- mal power for waste heat recovery. The authors evaluated the feasibility of mineral oil as coolant in this study. It was found that a 123 servers based CDC encapsulated in mineral oil can power a 10- ton chiller with a design point of 50.2 kWth. The experiments showed that as CPU utilization increased, the CPU temperatures also increased along with the thermal power resulting in better waste heat quality. However, if the thermal power is not properly utilized, higher temperature profiles can lead to device failures.

Table 3 provides comparison of various waste heat utilization studies in CDCs. Most of the state-of-the-art studies in waste heat utilization based CDC exploit liquid based heat exchange mechanisms. Moreover, CDC nodes are co- located to places where waste heat utilization opportunities are available. Furthermore, free cooling is applied to lower the cooling costs of CDCs.

Waste heat utilization techniques can offset operational costs while achieving ideal PUE. Moreover, waste heat utilization tech- niques can be used in vapor-absorption based cooling mechanism to supplement the cooling process and eliminate energy spent on cooling CDCs. The insulated MDC design provides ideal waste heat recovery mechanisms with minimal energy losses as compared to

J. Shuja et al. / Renewable and Sustainable Energy Reviews 62 (2016) 195–214206

large scale Bricks and Mortar CDC design. Moreover, MDCs also facilitate relocation of CDC to vicinity of heat recovery opportu- nities. Various waste heat recovery opportunities, such as district heating can also be availed while relocating container based MDCs to residential locations. However, the complexity of CDC design increases as a result of integration of waste heat recovery com- ponents, such as, heat exchangers, air/liquid ducts, and thermal interfaces. Moreover, the process of relocation of CDC to proximity of waste heat recovery opportunities incurs additional cost. This necessitates a technological migration from Bricks and Mortar design to innovative MDC designs [33,78,61]. Furthermore, while waste heat utilization techniques lower the cost of operating CDCs, capital cost of such designs is always higher than traditional designs [5,80].

6. Modular data centers

CDCs housed inside a single large warehouse size building with raised flooring are known as”Bricks and Mortar” CDC designs [81]. Recently, CDC operators have shifted from Bricks and Mortar paradigm to the Modular Data Center (MDC) paradigm. The next- generation of CDCs are shipping container based, assembled at

Table 3 Waste heat utilization schemes in CDCs.

Study Objective Setup

Ref. [68] Heat a small building with micro-CDC N/A

Ref. [29] Drive enough heat from CDC to power a vapor-absorption chiller

320 blade serv

Ref. [3] To heat a building with waste heat from a Supercomputer 36,384 core Su (Cray XE6)

Ref. [79] Extract more exergy from hot water cooled 3D servers CDC prototype Ref. [78] Environmentally opportunistic computing: distribute CDC

nodes to buildings requiring heating A MDC and co greenhouse

Fig. 6. Taxonomy of modular da

factory site, setup and wired, ready to run, and transported to the CDC operator site anywhere in the world. CDC nodes built inside generic shipping containers have two favorable characteristics for sustainability. Firstly, modular designs take advantage of the fact that shipping containers can be transported to those geographic regions that offer cheap electricity, renewable energy resources, free cooling, and waste energy utilization opportunities [61,81,82]. Secondly, the MDC is a closed loop that provides better opportu- nities for waste heat recovery and free cooling. Once transported to the site, the shipping container based CDC needs to be provided with power, network connectivity, and chilled water for operation, while it becomes a pluggable entity. Moreover, MDC designs promise to lower the Total Cost of Ownership (TCO) of a CDC, which is sum of capital and operational expenses [31,9,83]. As Google [84,85], IBM [86], and Microsoft [87] have recently inves- ted in portable MDC designs, their further widespread adoption can be predicted. Fig. 6 illustrates the taxonomy of modular data centers.

6.1. Design

The shift to MDC building paradigm can be contributed mainly to (a) size scaling of CDC servers from hundreds to thousands of

Heat Exchange Mechanism

Co-located Free Heat-ing/ Cooling

Air to water Yes and geo- dispersed

Yes

er micro-CDC Air to lithium-bromide liquid

Yes Yes

-per computer Air to water Yes Yes

Water to water No Yes - located Air to air Yes Yes

ta centers for sustainability.

Fig. 7. Container based MDC architecture.

J. Shuja et al. / Renewable and Sustainable Energy Reviews 62 (2016) 195–214 207

computing devices, (b) ability of MDC to be transported to ideal geographic location, and (c) energy and space efficiency of rack mounted servers used in MDC designs [81,88]. MDCs provide a scalable solution to carter workload fluctuations of cloud services. Scalability can result from incremental expansion in the number of MDCs having geographically distributed locations with added advantage of electricity price, energy resource, and user proximity variations [37]. Modular architectures also decrease cooling cost as: (a) insulated containers provide better temperature control while bringing the CRAC systems close to racks, (b) rackable sys- tems provide better air flow mechanisms and reduced hot aisle cold aisle air recirculation, and (c) free cooling from nearby cool air and water resources can be applied to insulated containers [27,42,89]. Fig. 7 gives cross-sectional view of a container built MDC architecture.

A single container module consists of multiple racks with several computing modules including blade servers, memory drives, and an access (top-of-the rack) switch mounted inside the rack. The racks usually run up to the roof of the container so that more computing power can be assembled in relatively small space. The dense assembly of computing devices requires careful tem- perature control. On the contrary, the insulated container envir- onment provides better applicability of free cooling and waste heat recovery techniques [5,90]. A blower fan may be installed over the top of each rack to: (a) expel warm air out of the con- tainer from the racks, (b) blow cool air from outside the container for free cooling, and (c) to capture waste heat from racks into heat exchange ducts for waste heat recovery mechanisms. Free cooling resources such as river water or cool air can also be used for condensing the fluid in the CRAC systems. PV panels and wind turbines can be deployed adjacent to the MDC container for gen- eration of renewable energy [36]. For high scalability, several MDC nodes can be interconnected with each other and housed within a Bricks and Mortar CDC building [84,91].

6.2. State-of-the-art

Researchers [32] carried out a case study to evaluate the fea- sibility of free cooling, waste heat utilization, and evaporative cooling in a MDC prototype located in Tokyo suburban area. The main focus of the study was to examine the characteristics of fresh air throughout an year and the effects on free cooling of a MDC. As the fresh air characteristics, such as, humidity and temperature change, filters are required to operate the servers in desirable environment. Moreover, waste heat from the IT equipment is applied to cold air to remove humidity if either the Relative Humidity (RH) value is greater than 85% or the air temperature is less than 10◦C. During the yearlong evaluation, annual mean air

temperature of 16.9◦C and the annual mean RH of 60.7% is observed. The evaluated system saves 21% energy compared to conventional CRAC based cooling system. The average PUE value of 1.058 is reported for the MDC. Researchers also investigated dif- ferent air inlet louver angles for efficient air circulation and lower cooling costs [27]. Their analysis shows that uniform air distribu- tion among rack components is possible at 45 degree inlet louver angle, thereby resulting in significant energy savings.

MDC designs have also attracted research on innovative cooling techniques [32,82,92,38]. The application of free cooling elim- inates the energy spent in CRAC systems. However, the CDC equipment requires specific temperature and humidity profiles to operate efficiently with lower device failure rates. Climatic chan- ges make it difficult to apply the same free cooling technique to a MDC throughout the year. Some geographical location may not be suitable for application of free cooling techniques, while some locations might require supplemental CRAC systems. A study found that MDC with air side economizer achieves 70% savings in cooling costs as compared to baseline CRAC systems [92]. To scale up the CDC, new modular containers having inter-connection to other containers can be commissioned. Brocanelli et al. [37] pro- posed ExContainer, a strategy that commissions geo-distributed MDCs while reducing the operational and capital costs of building a CDC. The portable MDC containers are commissioned based on fluctuating workload demands of cloud services. Geographical location for the portable container is selected based on abundance of green energy resources, lower electricity prices, and ambient temperatures for free cooling. The containers are built to utilize free cooling opportunities from air side economizer, thus, achiev- ing very low PUE. The strategy results in lower operational expenses while employing two level optimization. A coarse time scale policy dispatches portable modules to various geo- distributed locations while a fine time scale policy performs dynamic work- load scheduling based on electricity cost and network delay parameters. To reduce capital costs, long term upgrades are differed by commissioning new container modules and handling short-term workload peaks within each module.

A similar study proposing intelligent selection and optimiza- tion of geographical CDC locations based on constraints of green energy availability, free cooling environment, user proximity, and response time has been discussed in [12]. A similar concept of Nano CDC design examines relocation of servers to Internet Ser- vice Provider (ISP) centers in order achieve user proximity and energy efficiency [93]. Moreover, researchers [36,23] have utilized container based MDC prototype to dynamically schedule workload based on available green energy resources.

MDC designs have attracted research with respect to their network architectures. The issue of interconnecting a few

J. Shuja et al. / Renewable and Sustainable Energy Reviews 62 (2016) 195–214208

thousand servers inside a MDC is investigated by BCube archi- tecture [94]. BCube architecture utilizes hybrid packet forwarding where both switch and server elements act as forwarders. As MDCs are designed to be service-free once deployed, graceful performance degradation is required in case of component fail- ures. BCube achieves graceful performance degradation while maintaining a constant aggregate bottleneck throughput by con- necting components with multiple switches on different levels of the architecture. However, the study does not address the issues of load balancing between switches and interconnection of multiple MDCs. Moreover, as servers also perform data forwarding, they cannot be switched off for energy efficiency [2]. Wu et al. [95] discuss the issue of interconnecting hundreds of BCube based MDC nodes inside a Bricks and Mortar CDC design. The proposed design, MDCube, investigates three objectives of inter-container connec- tions: (a) high inter-container bandwidth, (b), lower inter- connection cost, and (c) lower cabling complexity. High inter- container bandwidth is achieved by directly connecting high- speed up- link interfaces of commodity switches inside each BCube architecture based container through optical fiber cables. Utilizing up-link interfaces of commodity switches for inter- container connections reduces the network and cabling cost. Table 4 show energy efficiency comparison of CDC entities for raised floor and MDC designs [42,81,82].

MDC design can be regarded as the ideal candidate for sus- tainable and green cloud computing. MDC paradigm allows ample opportunities to leverage renewable energy resources, free cool- ing, and waste heat utilization techniques. MDC enable relocation and incremental expansion of cloud services while lowering the TCO. Portable MDC containers can be transported to geographic locations that provide favorable conditions for sustainable and green CDC design. The micro-sized MDC designs allow distribution and de-centralization of CDC services. Each such distributed MDC node can be relocated to a site with one of the sustainability options, such as waste heat utilization or abundant renewable energy. The cumulative effect of distributed MDC nodes feeding on renewable energy, waste heat utilization, and free cooling choices is near ideal PUE values and lower carbon footprint. However, ideal geographic locations for MDC might not always provide optimal cloud user proximity. Both conventional CRAC and inno- vative free cooling techniques work efficiently in MDC as com- pared to Bricks and Mortar design. As a result, MDC achieve near ideal PUE values with minimal carbon footprint while utilizing renewable energy resources.

7. VM migration

Virtualization techniques complement sustainable CDC model by; (a) migrating workloads between geo-dispersed nodes based on renewable energy and waste heat utilization parameters (b) consolidating CDC workload on fewer physical servers within a CDC node for energy efficiency [9,96]. Virtualization efficiently manages the underlying CDC resources through abstraction. Vir- tualization adds a management layer known as Virtual Machine

Table 4 Comparison of Bricks and Mortar DC with MDC design.

Data center attribute Bricks and Mortar data center

Modular data center

Relative cooling costs 1 44–80% less Relative administrative costs 1 20–50% less Relative space efficiency 1 50% less Time to Development Long – more than an year Short – months PUE Average 1.7 E1.1

Monitor (VMM) over the physical hardware resources. The VMM executes multiple operating system instances (VMs) over shared physical resources to increase resource utilization. The VM instances are used to host high level applications that serve end users. Moreover, multiple instances of the VMs are also created to provide fault tolerance and redundancy for cloud resources [1,97]. CDCs utilize virtualization to offer pay-as-you-go multi-tenant business model. Virtualization technology exploits VM migration methods to shift the server workload for either: (a) consolidating applications and workload on fewer physical resources or (b) deploying redundant images of an application across multiple physical servers for fault tolerance [98]. In this manner, virtuali- zation imparts flexibility to CDC business model by catering fluc- tuating workloads. The resource flexibility imparted by virtuali- zation also helps avoid SLA violations. Recently, virtualization has been increasingly adopted to reduce energy consumption of CDCs through workload consolidation schemes. Due to over provision- ing, the average resource utilization of CDC physical re- sources remains around 30% [9,99]. Virtualization enables resource con- solidation via VM migration. As a result, fewer numbers of physical resources host VM instances with higher resource utilization. Remaining physical resources are transitioned to sleep/low-power state for energy efficiency [18].

VM migration techniques can be broadly classified on the migration mode. VM migration mode describes the migration methodology employed by reported schemes to effectively trans- fer memory pages across the geo-dispersed servers. The reported literature has either considered Pre-copy or post-copy migration model during VM migration phase. Objective function highlights the main aim of the renewable energy techniques. Reported schemes aim to; (a) minimize grid energy, (b) maximize good-put or (c) minimize renewable energy operational overhead. The attributes of network model describes the underlying network employed to migrate VM across the servers. The network model chosen by state-of-the-art renewable schemes is either LAN, WAN, or hybrid. KVM, Xen, and VMware are the virtualization technol- ogies chosen by renewable sustainable CDC studies. VM migration is a resource intensive operation. The optimization of the migra- tion process includes techniques, such as compression and write- throttling of the VM contents to efficiently utilize network band- width to achieve objective function under a set of constraints. Fig. 8 depicts the taxonomy of VM migration schemes in sustainable CDCs.

7.1. Design

VM migrations are commonly used to move workloads between physical servers or racks within a CDC (intra- CDC). However, virtualization also enables inter-CDC migration of workloads among geo-dispersed CDC nodes. Virtualization is enabler of sustainable CDC model in both scenarios [98]. Sus- tainable CDCs deploy hybrid power generation architectures that rely on both grid electricity and on-site renewable resources. In such hybrid power generation designs, some server racks are powered from renewable energy resource while other racks are powered by grid electricity. When renewable energy generation does not match the rack workload, workload is migrated with the help of virtualization to racks powered by grid energy [60]. Such intra-CDC workload migrations guarantee that job deadlines are met while optimizing utilization of renewable energy resources on hybrid grid. Similarly, in geo-dispersed CDCs, VM migrations facilitate follow the renewable approach [56]. Cloud service pro- viders manage multiple geo-dispersed CDC nodes for purposes, such as, renewable energy and waste heat utilization, user proxi- mity, service redundancy, variable energy resources, and pricing [20]. Workload migration among geo-dispersed CDC is executed

Fig. 8. Taxonomy of VM migrations techniques in sustainable CDC.

Fig. 9. Sustainable CDC based workload migrations.

J. Shuja et al. / Renewable and Sustainable Energy Reviews 62 (2016) 195–214 209

over diverse high bandwidth WAN technologies, such as, dedi- cated communication networks [40] and optical networks [100]. However, the performance cost of inter-CDC VM migrations in terms of network delay, energy, and communication overhead is always more than intra-CDC migrations. While intra-CDC network delay is minimal due to small network paths of Local Area Net- works (LAN) and shared storage, VM migration over WAN is often deemed performance prohibitive for user centric cloud services [14]. Network delay between geo-dispersed CDC locations is often intolerable for interactive and web based workloads. Moreover, migrating VM disk space that can be in range of tens or hundreds of gigabytes represents significant overhead in VM migrations over WAN [101]. However, compute-intensive scientific workloads can take advantage of follow the renewable approach while migrating

deadline insensitive jobs over WAN [50]. Fig. 9 illustrates workload migration between geo-dispersed MDC nodes over WAN.

7.2. State-of-the-Art

VM migration over WAN links is expensive due to larger VM image size and requirement of storage migration over inter- mediate links. Optimization of VM migration over WAN has been rigorously examined in literature [101,14]. Most of VM migration optimizations over WAN have focused on limiting data redun- dancy during live migration. Researchers proposed CloudNet [14] that implements three optimizations for the VM migration over low bandwidth WAN. Firstly, the VM live migration is adapted to the application memory access pattern. VM migration iterations

J. Shuja et al. / Renewable and Sustainable Energy Reviews 62 (2016) 195–214210

are stopped as soon as the dirty page ratio becomes stable. Sec- ondly, redundant data in the form of similar or empty blocks is eliminated. Thirdly, modification to data after first migration iteration is not transferred as whole. Rather deltas, the difference between cached and current page state are transferred.

Aikema et al. [102] conducted an experimental study critically investigating the power consumed by VM migration process while following the renewable approach. The authors considered the VM migration among the servers located at San Diego (at California), Cyberia (at Calgary), and rack-force (at Kelowna) while hosting high performance computing applications. The study revealed that power consumption during VM migration is directly influenced by unpredictable workload nature, size of VM memory, and under- lying communication infrastructure. The migration cost in terms of power consumption using live and non-live migration pattern is high unless attended by smart migration termination criterion, fine granular memory compression, and dedicated network com- munication links. However, the study lacks in considering the relationship between VM migration optimization and SLA viola- tion while optimizing the trade-off among power-performance. Researchers [60] studied a CDC prototype that hosted transient servers on renewable energy and stable servers on grid energy. Based on weather data, advance warnings are issued on avail- ability of renewable energy. VMs and their data hosted on tran- sient servers are migrated to stable servers on occurrence of advance warning. Two approaches are adopted to migrate VMs from transient servers. If the advance warning shows large migration window, VMs are live migrated along with their data. Otherwise, a live copy of transient VM is continuously updated on stable servers. However, the latter approach results in a 1:1 ratio between stable and transient server, nullifying benefits of tran- sient servers. In the former approach, the snapshot manager within the VM periodically sends dirty pages to stable servers. Upon receiving an advance warning, the VM at transient server is paused and all dirty pages are committed to stable server. In order to optimize VM migration, upper and lower thresholds are forced on rate of dirty page writing. Table 5

Stewart et al. [50] proposed a request routing scheme that is aware of renewable energy availability in multiple geo-dispersed CDC scenario. While computational workloads are assigned to a CDC, discontinuity of renewable energy supply demands workload migration to CDC nodes with redundant renewable energy. Researchers proposed iSwitch [57], a lightweight server power- management framework that minimizes the operational perfor- mance over- head of renewable energy powered CDC by leveraging cooperative load migration schemes. iSwitch offers automatic power-load balancing between conventional utility grids and renewable energy to achieve the best power delivery- demand matching. During low renewable energy availability period, iSwitch applies limited power throttling until the system throughput is constant. However, as renewable energy power generation further diminishes, iSwitch migrates the computing workload across the

Table 5 State-of-the-art Modular Data Center studies.

Study Objective Setu

Ref. [42] Computational fluid dynamics (CFD) analysis to compare MDC and raised-floor designs

Simu

Ref. [27] Optimize the inlet louver for better air distribution CFD Ref. [32] Utilize direct fresh air to as free cooling in evaporative cooling

systems MDC

Ref. [37] Geo-distribute MDC nodes to locations pro-viding lesser operational costs

Simu

Ref. [94] Utilize server to server connections for scalability in a MDC Simu

hybrid energy sources. VM migration enables the load switching in the iSwitch framework. To overcome associated network delays of VM migration, researchers [40] proposed Free lunch which utilizes VM based opportunistic workload migrations. Free lunch is a computation architecture that improves the CDC power consump- tion by co-locating the CDCs operated by heterogeneous energy resources linked through high speed dedicated network links. The proposed system exploits the renewable energy resource if the renewable energy is cheaper than CDC hosting the services. The diversity in seasons and solar power generation stimulates migra- tion of workload from one CDC to another prior to shutting down the server if the power capacity is enough at receiver end. Free lunch exploits live VM migration methods to transfer the workload among the CDCs. Further, VM migration operation is not free and consumes sufficient system resources. To handle number of migration, the resource capacity of the target server is investigated prior to migration. The free lunch exploited greedy based VM migration pattern which do not perform well when the workload nature is varying. Table 6 summarizes the aforementioned studies for renewable energy based VM migration techniques.

The overall utilization of renewable energy resources in a CDC is based on their limited daily availability. To maximize renewable energy utilization, geo-dispersed CDCs located in different time zones have been proposed [50]. Virtualization technology allows CDC resources to be managed and shared efficiently among mul- tiple end users. Inter and intra-CDC workload migrations enabled by virtualization facilitate follow the renewable approach. In this manner geo-dispersed CDCs can be fully operated on renewable energy. Otherwise, to fully operate the CDCs on renewable energy, costly energy storage batteries are required. Moreover, with dedicated communication links, cloud operators can overcome intra-CDC network delay while relocating CDC nodes to optimal sustainable locations across the globe. In this manner geo- dispersed CDC nodes can avail renewable energy, free cooling, and waste heat utilization options located in different time zones [50].

8. Research Issues and Challenges

Several issues need to be addressed for sustainable CDC operations. The capital costs of the advocated green measures, such as renewable energy and waste heat utilization techniques are always high than conventional CDC designs. CDCs require constant power supply, while the power generated by renewable resources is often inconsistent due to weather conditions, such as wind speed and solar irradiance. This calls for capacity planning and hybrid power delivery architectures that balance between grid and renewable energy resources. Automatic power transfer switches are required to switch power from renewable energy to grid to cater renewable energy intermittency. Redundant renew- able power can be stored in batteries. However, energy storage

p Cooling Mechanism Opportunistic location

Free Heat-ing/ Cooling

lation Hybrid: Water and air No No

simulation Air No Yes prototype Air Yes Yes

lation Air Yes Yes

lation NA No No

Table 6 Comparison of renewable based VM migration schemes.

Study VM migra- tion model

Objective Network model Storage migration

Virtualization technology

Optimization model

Compression Write throttling

Ref. [102] Pre and post- copy

To investigate energy consumption during VM migration

Hybrid: LAN and WAN

No KVM No Yes

Ref. [60] Pre-copy Transient server abstraction based CDC ser- vice availability

WAN Yes Xen Yes No

Ref. [62] NA Minimize the use of off-grid renewable LAN NA NA No No Ref. [57] Pre-copy Minimizing operational overhead of renew-

able powered CDC WAN Yes NA No No

Ref. [40] Pre-copy Investigate VM migration down-time during follow the renewable approach

WAN Yes Xen Yes No

Ref. [103] NA Maximizing system good-put using batch migration method

WAN No VMware No No

J. Shuja et al. / Renewable and Sustainable Energy Reviews 62 (2016) 195–214 211

batteries lead to increased capital costs and incur voltage loses [52,53]. The major challenge in waste heat reuse is the fact that CDC waste heat is of low quality bounded by temperature limits of various IT equipment [5]. Moreover, waste heat utilization requires modifications to cooling systems and server rack configurations that can be ideally handled with MDC designs [68]. Similarly, workload migrations over long-haul networks are expensive in terms of service delay and bandwidth consumption. Dedicated communication networks are often required to meet requirements of SLAs while migrating workloads in geo-dispersed CDCs [40]. Geographical locations with abundance of green computing opportunities might not always meet business objectives, such as, user proximity, commodity and electricity prices. Electricity price can be both in favor of and against the relocation of CDCs. Trade- offs lie between the degree of green computing adoption, meeting business objectives with low capital costs, and migrating from conventional CDC designs [36,104].

8.1. Renewable energy-CDC integration

Research issues that need to be addressed for fruitful integra- tion of renewable energy to CDCs are:

� The major challenge that needs to be addressed with respect to integration of renewable energy to CDCs is the intermittency of renewable energy. Hybrid grid-renewable power supply archi- tectures for CDCs need to be designed to ensure availability of cloud services and non-violation of SLAs [20].

� Life-cycle based assessment and capacity planning of CDCs in terms of energy cost and business models need to be investi- gated for sustainable CDCs. Although work has been done, it does not cover all aspects of sustainable CDC designs [61,105].

� Precise prediction models for renewable energy generation and CDC workload need to be formalized. These models depend on weather forecasting and prediction. Similarly, location-aware data center workload prediction models also need to be investigated. Renewable power generation and CDC power demand balancing techniques need to be developed based on aforementioned prediction models [106].

� Reliable business and service model need to be explored in wake of intermittent renewable energy supply for sustainable CDCs [107].

8.2. Waste heat utilization

There are many challenges to incorporating waste heat utili- zation techniques to CDCs. The foremost challenge is to shift

conventional CDCs from air based cooling systems to water based or hybrid cooling systems. Air based cooling systems have several disadvantages in terms of waste heat recovery.

� Hot aisle cold aisle assembly of server racks has complex ther- modynamics and exhaust air can mix with cold supply before being captured as waste heat. As a result, exergy and waste heat quality of the air cooled CDC remains low [68].

� Water is more efficient heat exchange medium with higher Capture Fraction (CF) than air [108,29].

� Water cooled systems have smaller flow path than air cooled systems resulting in smaller insulation area and lower pump power [29].

� Due to lower density and thermal conductivity, air losses cap- tured heat very quickly. On the contrary, water is able to transmit heat to higher distances due to higher thermal con- ductivity and density [109].

Therefore, water cooled and hybrid cooling systems are deemed as ideal candidates for waste heat utilization techniques [5]. However, water and hybrid cooled CDCs are more complex with concerns of water leakages into electronic equipment and cost of water micro-channels and thermal heat exchange inter- faces. Moreover, to capture high grade waste heat, CDC servers are often operated on higher temperatures, resulting in higher fre- quency of device failures [29]. Furthermore, relocating CDC nodes in the proximity of waste heat recovery opportunities requires shift from conventional CDC bricks and mortar design to MDC paradigm [30].

8.3. MDC

MDC designs are gaining popularity among CDC providers. The idea of utilizing location-as-a-service with MDCs has been pro- posed for sustainable CDC operations [30]. However, essential challenges exist to widespread adoption of MDCs [110,111]. The foremost challenge faced by MDC designs is service-free operation of MDCs in remote locations in wake of server and network fail- ures. Further challenges are:

� MDC nodes are envisioned to provide a minimum throughput for 3-7 years without device repair services. There- fore, MDC servers need to be interconnected such that device failure does not adversely affect the overall MDC throughput [31].

� MDC scalability from single node to a mega CDC with thousands of MDC nodes requires high speed interconnection high cabling and networking complexity [95]. Connecting MDCs over WAN

J. Shuja et al. / Renewable and Sustainable Energy Reviews 62 (2016) 195–214212

for high inter-container bandwidth with lower cabling com- plexity and cost efficiency is a challenge.

� Placement of MDC nodes while considering parameters such as, lower TCO, end user proximity, renewable energy and waste heat recovery opportunities is a multi-factor optimization problem [31].

8.4. VM WAN migrations

The main challenge to VM based workload migrations is the resultant network and service delay between geo- dispersed CDC nodes. Content compressing techniques have been proposed to decrease the amount of data transfer and service delay [60]. However, the VM size, often in the range of 10X Gbytes, leads to crucial overhead in terms of delay and additional energy con- sumption. Moreover, expensive dedicated point-to-point commu- nicational links are required for efficient migration of VMs over WAN [40]. Further challenges to VM migration techniques enabling sustainable CDCs are:

� VM migration is resource rigorous process as it requires abun- dant resources including CPU cycles, bandwidth capacity, sys- tem cache, and memory, during VM migration process across geo-dispersed CDCs. Live VM migration schemes reduce total migration time and downtime by employing write-throttling, deduplication, and pre-paging heuristics. However, incorporat- ing optimization strategies effects co-hosted application perfor- mance in terms of SLA violation and extended end-to-end delay if migration process is not managed properly [18].

� VM migration over WAN links is trivial due to need of storage migration in addition to VM memory transfer. For VM migration among Geo-dispersed CDC nodes, migration controller exploits shared bandwidth to transfer VM memory image and storage which resultantly increases total migration time [35].

� During VM migration process, securing contents of migrating VM is an important concern of the CDC operator as VM has to migrate on vulnerable data links. Hijacking VM during migra- tion can enable the intruders to access the running kernel state, hosted applications data, and hardware state for malicious activities [112].

9. Conclusion

In this article, we advocated innovative and disruptive techni- ques and technologies for sustainable and green CDCs. On-site renewable energy generation, waste heat utilization, geo- dispersed modular CDCs, and VM migrations are the techniques and technologies that enable sustainable CDCs. Renewable energy power generation eliminates the carbon emissions from CDC operations. However, abundant renewable energy resources are often available at remote geo- graphical locations. Similarly, waste heat utilization techniques offset energy spent on cooling, thus, achieving higher PUE. Still, waste heat utilization opportunities, such as district heating are found mostly at places away from CDC deployments. Shippable MDCs enable CDC operators to take ben- efits from remote renewable energy and waste heat utilization opportunities. Moreover, virtualization facilitates workload migration between geo-dispersed CDC nodes to overcome inter- mittency in renewable power generation. Furthermore, portable MDCs nodes and VM migrations also empower geo-dispersed CDC model in which modular nodes are relocated to sites with sus- tainable and green computing possibilities. This article sum- marizes studies that focus on the sustainable and green CDC paradigm with vision to utilize renewable energy and waste heat utilization techniques. Further, the article also examines the

enabling technologies, such as, shippable MDCs, distributed geo- dispersed CDC nodes, free cooling, and VM migration. The case studies examined in this study portrayed favorable forecast regarding integration of sustainability measures in CDCs. The case studies also revealed that utilization of multiple sustainability techniques leads to a successful green CDC model. Through a comprehensive survey, we found that sustainability measures can both lower energy costs and carbon footprint in CDCs. The issues regarding renewable energy capital costs are predicted to be amortized over the next decade. The taxonomies presented in the article categorize the parameters of sustainability measures in CDCs.

The study also identified several research challenges to sus- tainable CDCs. Modern CDC designs need to be developed to address instability of renewable energy supplies. Moreover, abundant renewable energy resources often lay at remote geo- graphical sites. However, sustainable geo-dispersed CDCs often compromise on user proximity and workforce availability. Studies need to examine user performance metrics for locating CDCs at remote locations. VM migrations over WAN are not suitable for deadline sensitive tasks. Researchers need to optimize VM migrations over long distances for WAN technologies to enable sustainable operations among geo-dispersed CDCs. To document the resultant benefits of sustainable CDCs, formal studies, proto- type projects, and real-time commercial deployments need to be carried out. Life-Cycle Assessment (LCA) studies need to be implemented that examine the sustainability and environmental impact of the CDCs across various life-cycle stages. The LCA studies need to examine the effect of sustainable CDC techniques and technologies discussed in this article both individually and colla- boratively. Moreover, a comparison based on LCA of both tradi- tional CDC design and sustainable CDC design should be made.

Acknowledgments

This work is partially funded by the Malaysian Ministry of Education under the High Impact Research Grant of University of Malaya UM.C/625/1/HIR/MOE/FCSIT/03.

References

[1] Shuja J, Madani S, Bilal K, Hayat K, Khan S, Sarwar S. Energy-efficient data centers. Computing 2012;94:973–94.

[2] Bilal K, Malik SUR, Khalid O, Hameed A, Alvarez E, Wijaysekara V, et al. A taxonomy and survey on green data center networks. Future Gener Comput Syst 2014;36:189–208.

[3] Romero M, Hasselqvist H, Svensson G. Supercomputers keeping people warm in the winter. In: Proceedings of the 2nd International Conference on ICT for Sustainability (ICTS), August 24–27, 2014. Stockholm, Sweden: Bok- förlaget Atlantis; 2014. p. 324–32.

[4] Meijer G. Cooling energy-hungry data centers. Science 2010;328(5976):318– 9.

[5] Ebrahimi K, Jones GF, Fleischer AS. A review of data center cooling tech- nology, operating conditions and the corresponding low-grade waste heat recovery opportunities. Renew Sustain Energy Rev 2014;31:622–38.

[6] Masanet E, Shehabi A, Koomey J. Characteristics of low-carbon data centres. Nat Clim Change 2013;3(7):627–30.

[7] Ahmad RW, Gani A, Hamid SHA, Xia F, Shiraz M. A review on mobile application energy profiling: Taxonomy, state-of-the-art, and open research issues. J Netw Comput Appl 2015;58:42–59.

[8] Brown R. Report to congress on server and data center energy efficiency public law 109-431. Environ Prot 2007;109:431.

[9] Shuja J, Bilal K, Madani S, Othman M, Ranjan R, Balaji P, Khan S. Survey of techniques and architectures for designing energy-efficient data centers. Syst J IEEE 2014;99:1–13. http://dx.doi.org/10.1109/JSYST.2014.2315823.

[10] Uddin M, Rahman AA. Energy efficiency and low carbon enabler green it framework for data centers con- sidering green metrics. Renew Sustain Energy Rev 2012;16(6):4078–94.

[11] Schomaker G, Janacek S, Schlitt D. The energy demand of data centers. In: ICT Innovations for sustainability. Springer; 2015. p. 113–24.

J. Shuja et al. / Renewable and Sustainable Energy Reviews 62 (2016) 195–214 213

[12] Goiri ´I, Le K, Guitart J, Torres J, Bianchini R, Intelligent placement of data- centers for internet services, In: Distributed Computing Systems (ICDCS), 2011 31st International Conference on, 2011, pp. 131–142.

[13] Al-Ayyoub M, WardatM, Jararweh Y, Khreishah A.A, Optimizing expansion strategies for ultrascale cloud computing data centers, Simulation Modelling Practice and Theory-doi:http://dx.doi.org/10.1016/j.simpat.2015.03.002.

[14] Wood T, Ramakrishnan K, Shenoy P, Van der Merwe J. Cloudnet: dynamic pooling of cloud resources by live wan migration of virtual machines. In: SIGPLAN ACM, editor. Notices, vol. 46. ACM; 2011. p. 121–32.

[15] Google, Google's green data centers: Network pop case study (2012). URL ⟨https://www.google.com/about/datacenters/efficiency/external⟩/.

[16] Wang U, Google, facebook and apple lead on green data centers (April 2014). URL ⟨http://www.theguardian.com/sustainable-business/greenpeace-report- google-facebook-apple-green-data-centers⟩.

[17] Or o´ E, Depoorter V, Garcia A, Salom J. Energy efficiency and renewable energy integration in data centres. strategies and modelling review. Renew Sustain Energy Rev 2015;42:429–45.

[18] Ahmad RW, Gani A, Hamid SHA, Shiraz M, Yousafzai A, Xia F. A survey on virtual machine migration and server consolidation frameworks for cloud data centers. J Netw Comput Appl 2015;52:11–25.

[19] Uchechukwu A, Li K, Shen Y. Improving cloud computing energy efficiency, in: Cloud Computing Congress (APCloudCC). 2012 IEEE Asia Pacific. IEEE 2012:53–8.

[20] Deng W, Liu F, Jin H, Li B, Li D. Harnessing renewable energy in cloud datacenters: opportunities and challenges. IEEE Netw 2014;28(1):48–55.

[21] Jaradat M, Jarrah M, Jararweh Y, Al-Ayyoub M, Bousselham A, Integration of renewable energy in demand- side management for home appliances, in: Renewable and Sustainable Energy Conference (IRSEC), 2014 International, IEEE, 2014, pp. 571–576.

[22] Bianchini R, Leveraging renewable energy in data centers: present and future, in: Proceedings of the 21st international symposium on High- Performance Parallel and Distributed Computing, 2012, pp. 135–136.

[23] Goiri ´I, Katsak W, Le K, Nguyen TD, Bianchini R. In: SIGARCH ACM, editor. Parasol and greenswitch: Managing datacenters powered by renewable energy, Vol. 41. Computer Architecture News; 2013. p. 51–64.

[24] Qureshi A. Power-demand routing in massive geo-distributed systems. Ph.D. thesis. Massachusetts Institute of Technology; 2010.

[25] Mustafa S, Nazir B, Hayat A, Madani S.A, et al., Resource management in cloud computing: Taxonomy, prospects, and challenges, Computers & Elec- trical Engineering.

[26] Westphall CB, Westphall CM, Koch FL, Geronimo GA, Werner J. Operation, management, security and sustainability for cloud computing. Rev Sist Informac¸a~o da FSMA 2014;13:30–50.

[27] Gebrehiwot B, Aurangabadkar K, Kannan N, Agonafer D, Sivanandan D, Hendrix M, Cfd analysis of free cooling of modular data centers, in: Semi- conductor Thermal Measurement and Management Symposium (SEMI- THERM), 2012 28th Annual IEEE, 2012, pp. 108–111.

[28] Wang H, Huang J, Lin X, M. Rad H, Proactive demand response for data centers: A win-win solution, CoRR abs/1511.00575. URL http://arxiv.org/abs/ 1511.00575.

[29] Haywood A, Sherbeck J, Phelan P, Varsamopoulos G, Gupta SK. Thermo- dynamic feasibility of harvesting data center waste heat to drive an absorption chiller. Energy Convers Manag 2012;58:26–34.

[30] Depoorter V, Or o´ E, Salom J. The location as an energy efficiency and renewable energy supply measure for data centres in europe. Appl Energy 2015;140:338–49.

[31] Vishwanath K. V, Greenberg A, Reed D.A, Modular data centers: how to design them?, in: Proceedings of the 1st ACM workshop on Large-Scale system and application performance, 2009, pp. 3–10.

[32] Endo H, Kodama H, Fukuda H, Sugimoto T, Horie T, Kondo M, Effect of cli- matic conditions on energy consumption in direct fresh-air container data centers, Sustainable Computing: Informatics and Systems.

[33] Ward E, Goedke M. L, Brenner P, Go D. B, A simplified thermodynamic model for waste heat utilization from a containerized data center experimental platform, in: 13th IEEE Intersociety Conference on Thermal and Thermo- mechanical Phenomena in Electronic Systems (ITherm), 2012, 2012, pp. 521– 529.

[34] Bilal K, Khan S. U, Zomaya A. Y, Green data center networks: Challenges and opportunities, in: Frontiers of Information Technology (FIT), 2013 11th International Conference on, IEEE, 2013, pp. 229–234.

[35] Ahmad RW, Gani A, Hamid SHA, Shiraz M, Xia F, Madani SA. Virtual machine migration in cloud data centers: a review, taxonomy, and open research issues. J Supercomput 2015:1–43.

[36] Goiri ´I, Katsak W, Le K, Nguyen TD, Bianchini R. Designing and managing datacenters powered by renewable energy. IEEE Micro 2014:1.

[37] Brocanelli M, Zheng W, Wang X. Reducing the expenses of geo-distributed data centers with portable con- tainerized modules. Perform Eval 2014;79:104–19.

[38] Weerts B, Gallaher D, Weaver R, Van Geet O, Green data center cooling: Achieving 90% reduction: Airside economization and unique indirect eva- porative cooling, in: Green Technologies Conference, 2012 IEEE, IEEE, 2012, pp. 1–6.

[39] Clancy H, 12 green data centers worth emulating from apple to verne (2013). URL ⟨http://www.greenbiz.com/article/12-green-data-centers-worth- emu lating-apple-verne⟩.

[40] Akoush S, Sohan R, Rice A, Moore AW, Hopper A. Free lunch: exploiting renewable energy for computing. In: Proceedings of HotOS; 2011 17–17.

[41] Zimmermann S, Meijer I, Tiwari MK, Paredes S, Michel B, Poulikakos D. Aquasar: A hot water cooled data center with direct energy reuse. Energy 2012;43(1):237–45.

[42] Qouneh A, Li C, Li T, A quantitative analysis of cooling power in container- based data centers, in: 2011 IEEE International Symposium on Workload Characterization (IISWC), 2011, pp. 61–71.

[43] Lule goes live (2013). URL http://tinyurl.com/pxn97r5. [44] Goiri ´I, Beauchea R, Le K, Nguyen TD, Haque ME, Guitart J, Torres J, Bianchini

R. Greenslot: schedul- ing energy consumption in green datacenters. in: Proceedings of 2011 International Conference for High Per- formance Com- puting. Networking, Storage and Analysis; 2011. p. 20.

[45] Kong F, Liu X. A survey on green-energy-aware power management for datacenters. ACM Computing Surveys (CSUR) 2015;47(2):30.

[46] Ren C, Wang D, Urgaonkar B, Sivasubramaniam A, Carbon-aware energy capacity planning for datacenters, in: 2012 IEEE 20th International Sympo- sium on Modeling, Analysis & Simulation of Computer and Tele- communication Systems (MASCOTS), 2012, pp. 391–400.

[47] Geronimo G, Werner J, Westphall C, Westphall C, Defenti L, Provisioning and resource allocation for green clouds, in: ICN 2013, The Twelfth International Conference on Networks, 2013, pp. 81–86.

[48] Sevencan S, Lindbergh G, Lagergren C, Alvfors P. Economic feasibility study of a fuel cell-based combined cooling, heating and power system for a data centre. Energy Build 2016;111:218–23.

[49] Xu D, Qu M. Energy, environmental, and economic evaluation of a cchp system for a data center based on operational data. Energy Build 2013;67:176–86.

[50] Stewart C, Shen K, Some joules are more precious than others: Managing renewable energy in the datacenter, in: Proceedings of the Workshop on Power Aware Computing and Systems, 2009.

[51] Kiani A, Ansari N. Toward low-cost workload distribution for integrated green data centers. Commun Lett IEEE 2015;19(1):26–9.

[52] Wang D, Ren C, Sivasubramaniam A, Urgaonkar B, Fathy H. Energy storage in datacenters: what, where, and how much? ACM SIGMETRICS Perform Eval Rev 2012;40(1):187–98.

[53] Beaudin M, Zareipour H, Schellenberglabe A, Rosehart W. Energy storage for mitigating the variability of renewable electricity sources: an updated review. Energy Sustain Dev 2010;14(4):302–14.

[54] Li C, Zhang W, Cho C.-B, Li T, Solarcore: Solar energy driven multi-core archi- tecture power management, in: High Performance Computer Architecture (HPCA), 2011 IEEE 17th International Symposium on, 2011, pp. 205–216.

[55] Chandio AA, Bilal K, Tziritas N, Yu Z, Jiang Q, Khan SU, Xu C-Z. A comparative study on resource allocation and energy efficient job scheduling strategies in large-scale parallel computing systems. Cluster Comput 2014;17(4):1349–67.

[56] Liu Z, Lin M, Wierman A, Low S.H, Andrew L.L, Greening geographical load balancing, in: Proceedings of the ACM SIGMETRICS joint international con- ference on Measurement and modeling of computer systems, 2011, pp. 233– 244.

[57] Li C, Qouneh A, Li T, iswitch: coordinating and optimizing renewable energy powered server clusters, in: Computer Architecture (ISCA), 2012 39th Annual International Symposium on, 2012, pp. 512–523.

[58] Arlitt M, Bash C, Blagodurov S, Chen Y, Christian T, Gmach D, Hyser C, Kumari N, Liu Z, Marwah M, et al., Towards the design and operation of net-zero energy data centers, in: Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm), 2012 13th IEEE Intersociety Conference on, 2012, pp. 552–561.

[59] Aksanli B, Venkatesh J, Zhang L, Rosing T. Utilizing green energy prediction to schedule mixed batch and service jobs in data centers. ACM SIGOPS Oper Syst Rev 2012;45(3):53–7.

[60] Singh R, Irwin DE, Shenoy PJ, Ramakrishnan KK. Yank: Enabling green data centers to pull the plug. in:. NSDI 2013:143–55.

[61] Meza J, Shih R, Shah A, Ranganathan P, Chang J, Bash C. Lifecycle-based data center design. ASME 2010 International Mechanical Engineering Congress and Exposition. In: American Society of Mechanical Engineers; 2010. p. 217–26.

[62] Chen C, He B, Tang X. Green-aware workload scheduling in geographically distributed data centers. in: CloudCom 2012:82–9.

[63] Zhang Y, Wang Y, Wang X. Greenware: Greening cloud-scale data centers to maximize the use of renewable energy. Middleware 2011. In: Springer; 2011. p. 143–64.

[64] Sharma N, Barker S, Irwin D, Shenoy P. Blink: managing server clusters on intermittent power. in:. ACM SIGPLAN Notices 2011;Vol. 47:185–98.

[65] Brown M, Renau J. Rerack: Power simulation for data centers with renewable energy generation. ACM SIGMETRICS Perform Eval Rev 2011;39(3):77–81.

[66] Li C, Qouneh A, Li T. Characterizing and analyzing renewable energy driven data centers. ACM SIGMETRICS Perform Eval Rev 2011;39(1):323–4.

[67] California energy commission tracking progress (December 2015). URL ⟨http://www.energy.ca.gov/renewables/tracking_progress/documents/renew able.pdf⟩.

[68] Liu J, Goraczko M, James S, Belady C, Lu J, Whitehouse K, The data furnace: heating up with cloud computing, in: The 3rd USENIX Workshop on Hot Topics in Cloud Computing (HotCloud’11), 2011.

[69] Patterson M, Tschudi B, Vangeet O, Cooley J, Azevedo D, Ere: a metric for measuring the benefit of reuse energy from a data center, White Paper 29.

J. Shuja et al. / Renewable and Sustainable Energy Reviews 62 (2016) 195–214214

[70] Li Z, Kandlikar SG. Current status and future trends in data-center cooling technologies. Heat Transf Eng 2015;36(6):523–38.

[71] Shah AJ, Carey VP, Bash CE, Patel CD, Sharma RK. Exergy analysis of data center thermal manage- ment systems. Springer; 2012.

[72] Chaudhry MT, Ling TC, Manzoor A, Hussain SA, Kim J. Thermal-aware scheduling in green data centers. ACM Comput Surv (CSUR) 2015;47(3):39.

[73] Haywood AM, Sherbeck J, Phelan P, Varsamopoulos G, Gupta SK. The rela- tionship among cpu utilization, temperature, and thermal power for waste heat utilization. Energy Convers Manag 2015;95:297–303.

[74] Chaudhry MT, Ling TC, Hussain S, Lu X-z. Thermal-aware relocation of ser- vers in green data centers. Front Inform Technol Electron Eng 2015;16 (2):119–34.

[75] ASHRAE T, Design considerations for datacom equipment centers - expanding the recommended environmen- tal envelope, American Society of Heating, Refrigerating and Air-conditioning Engineers, Inc.

[76] Vela J, Helsinki data centre to heat homes (2010). URL ⟨http://www.the guardian.com/environment/2010/jul/20/helsinki-data-centre-heat-homes⟩.

[77] Niccola J, Swedish data center saves 1 million a year using seawater for cooling (2013). URL ⟨http://www.pcworld.com/article/2038934/swedish- data-center-saves-1-million-a-year-using-seawater-for-cooling.html⟩,.

[78] Zachary Woodruff J, Brenner P, Buccellato AP, Go DB. Environmentally opportunistic computing: A distributed waste heat reutilization approach to energy-efficient buildings and data centers. Energy Build 2014;69:41–50.

[79] Tiwari M, Zimmermann S, Sharma C, Alfieri F, Renfer A, Brunschwiler T, Meijer I, Michel B, Poulikakos D, Waste heat recovery in supercomputers and 3d integrated liquid cooled electronics, in: Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm), 2012 13th IEEE Intersociety Conference on, 2012, pp. 545–551.

[80] Lu X, Lu T, Remes M, Viljanen M, Energy efficiency assessment for data center in finland: Case study, in: Distributed Computing Systems Workshops (ICDCSW), 2011 31st International Conference on, 2011, pp. 54–60.

[81] Hamilton, Architecture for modular data centers, arXiv preprint cs/0612110. [82] Shah N, Cfd analysis of direct evaporative cooling zone of air-side econo-

mizer for containerized data center, Master's thesis (2012). [83] Church K, Greenberg AG, Hamilton JR. On delivering embarrassingly dis-

tributed cloud services. In: HotNets. Citeseer; 2008. p. 55–60. [84] Whitted W, Aigner G, Modular data center, uS Patent 7,278,273 (Oct. 9 2007). [85] Miller R, Google unveils its container data center (2009). URL ⟨http://www.

datacenterknowledge.com/archives/2009/04/01/google-unveils-its-con tainer-data-center⟩/.

[86] IBM, Portable modular data centersadd capacity where and when you need, White Paper.

[87] Miller R, Microsofts 1 billion data center (2013). URL ⟨http://www.data centerknowledge.com/archives/2013/01/31/microsofts-1-billion-roofless- data-center⟩/.

[88] Masanet E, Brown R, Shehabi A, Koomey J, Nordman B. Estimating the energy use and efficiency potential of U.S. data centers. Proc IEEE 2011;99(8):1440– 53.

[89] Bramfitt M, Coles H, Modular/container data centers procurement guide: optimizing for energy efficiency and quick deployment, Department of Energy, Lawrence Berkley National Laboratory Report.

[90] Greenberg S, Mills E, Tschudi B, Rumsey P, Myatt B. Best practices for data centers: Lessons learned from benchmarking 22 data centers, Proceedings of the ACEEE Summer Study on Energy Efficiency in Buildings in Asilomar. August 3. CA. ACEEE 2006:76–87.

[91] Ham S-W, Kim M-H, Choi B-N, Jeong J-W. Energy saving potential of various air-side economizers in a modular data center. Appl Energy 2015;138:258– 75.

[92] Anubhav K, Yogendra J, Use of airside economizer for data center thermal management, in: Thermal Issues in Emerging Technologies, 2008. ThETA’08. Second International Conference on, IEEE, 2008, pp. 115–124.

[93] Valancius V, Laoutaris N, Massouli e´ L, Diot C, Rodriguez P, Greening the internet with nano data centers, in: Proceedings of the 5th international conference on Emerging networking experiments and technologies, CoN- EXT’09, ACM, New York, NY, USA, 2009, pp. 37–48.

[94] Guo C, Lu G, Li D, Wu H, Zhang X, Shi Y, Tian C, Zhang Y, Lu S. Bcube: a high performance, server- centric network architecture for modular data centers, SIGCOMM. Comput Commun Rev 2009;39(4):63–74.

[95] Wu H, Lu G, Li D, Guo C, Zhang Y. Mdcube: a high performance network structure for modular data center interconnection. Proceedings of the 5th international conference on Emerging networking experiments and tech- nologies. In: ACM; 2009. p. 25–36.

[96] Liaqat M, Ninoriya S, Shuja J, Ahmad RW, Gani A. Virtual machine migration enabled cloud resource management: A challenging task. arXiv preprint arXiv 1601:03854.

[97] Ahmed E, Gani A, Sookhak M, Ab Hamid SH, Xia F. Application optimization in mobile cloud computing: Motivation, taxonomies, and open challenges. J Netw Comput Appl 2015;52:52–68.

[98] Hammadi A, Mhamdi L. A survey on architectures and energy efficiency in data center networks. Comput Commun 2014;40:1–21.

[99] Shuja J, Bilal K, Madani S, Khan S. Data center energy efficient resource scheduling. Cluster Comput 2014;17(4):1265–77.

[100] Velasco L, Asensio A, Berral J, Castro A, L o´pez V. , Towards a carrier sdn: an example for elastic inter- datacenter connectivity. Opt Express 2014;22 (1):55–61.

[101] Medina V, Garcia JM. A survey of migration mechanisms of virtual machines. ACM Comput Surv (CSUR) 2014;46(3):30.

[102] Aikema D, Mirtchovski A, Kiddle C, Simmonds R. Green cloud vm migration: Power use analysis. in: Green Computing Conference (IGCC), 2012 Interna- tional. IEEE 2012:1–6.

[103] Cheng D, Jiang C, Zhou X. Heterogeneity-aware workload placement and migration in distributed sustainable datacenters. in: Parallel and Distributed Processing Symposium, 2014 IEEE 28th International. IEEE 2014:307–16.

[104] Mustafa H, Baveja B, Vijayan S, Merchant S, Desai U. Replicating the geo- graphical cloud: Provisioning omnipresence, omniscience and omnipotence. Future Gener Comput Syst 2015;47:1–15.

[105] Whitehead B, Andrews D, Shah A. The life cycle assessment of a uk data centre. Int J Life Cycle Assess 2015;20(3):332–49.

[106] Weing ärtner R, Bräscher GB, Westphall CB. Cloud resource management: a survey on forecasting and profiling models. J Netw Comput Appl 2015;47:99– 106.

[107] Gmach D, Rolia J, Bash C, Chen Y, Christian T, Shah A, Sharma R, Wang Z, Capacity planning and power management to exploit sustainable energy, in: Network and Service Management (CNSM), 2010 International Conference on, 2010, pp. 96–103.

[108] Gao T, Sammakia B, Geer J, Ortega A, Schmidt R, Transient effectiveness characteristics of cross flow heat exchangers in data center cooling systems, in: Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm), 2014 IEEE Intersociety Conference on, IEEE, 2014, pp. 688–697.

[109] Fulpagare Y, Bhargav A. Advances in data center thermal management. Renew Sustain Energy Rev 2015;43:981–96.

[110] Farrington N, Porter G, Radhakrishnan S, Bazzaz H, Subramanya V, Fainman Y, Papen G, Vahdat A. Helios: a hybrid electrical/optical switch architecture for modular data centers, ACM SIGCOMM. Comput Commun Rev 2011;41 (4):339–50.

[111] Hamilton J, Cooperative expendable micro-slice servers (cems): low cost, low power servers for internet-scale services, in: Conference on Innovative Data Systems Research (CIDR09)(January 2009), 2009.

[112] Ali M, Khan SU, Vasilakos AV. Security in cloud computing: opportunities and challenges. Inf Sci 2015;305:357–83.