Economic arguments for cloud services migration

Reference Notes – Additional AWS Cloud Management Tools and Services

As part of the process of managing compliant and secure cloud systems, AWS provides a number of

useful tools and services. The attributes of the prime ones are outlined in the sections that follow.

For a fuller treatment of these tools or services, you should refer to the AWS User Guides.

AWS CloudFront:

CloudFront is a global Content Delivery Network (CDN) service. It integrates with other AWS

products to give developers and businesses an easy way to distribute content to end users with low

latency, high data transfer speeds and no minimum usage commitments.

A Content Delivery Network (CDN) is a globally distributed network of caching servers that speed up

the downloading of web pages and other content. CDNs use Domain Name System (DNS) geo-

location to determine the geographic location of each request for a web page or other content, then

they serve that content from edge caching servers closest to that location instead of the original web

server. CDNs allow organisations to increase the scalability of their websites or mobile applications

easily in response to peak traffic spikes.

Figure 1: Sample CloudFront Delivery-to-Edge Location Flows

In most cases, using a CDN is completely transparent – end users simply experience better website

performance, while the load on the organisation’s original website is reduced. Amazon CloudFront is

AWS’ CDN. It can be used to deliver web content using Amazon’s global network of edge locations.

When a user requests content that is being served with Amazon CloudFront, the user is routed to

the edge location that provides the lowest latency (time delay), so content is delivered with the best

possible performance. If the content is already in the edge location with the lowest latency, Amazon

CloudFront delivers it immediately.

If the content is not currently in that edge location, Amazon CloudFront retrieves it from the origin

server, such as an Amazon Simple Storage Service (Amazon S3) bucket or a web server, which stores

the original, definitive versions of the files. Amazon CloudFront is optimised to work with other AWS

cloud services as the origin server, including Amazon S3 buckets, Amazon S3 static websites, Amazon

Elastic Compute Cloud (Amazon EC2) and Elastic Load Balancing. Amazon CloudFront works

seamlessly with any non-AWS origin server, such as an existing on-premises web server.

Figure 2: Sample CloudFront Edge Location Flows

Amazon CloudFront also integrates with Amazon Route 53. Amazon CloudFront supports all content

that can be served over HTTP or HTTPS. This includes any popular static files that are a part of web

applications, such as HTML files, images, JavaScript and CSS files, and also audio, video, media files

or software downloads. Amazon CloudFront also supports serving dynamic web pages, so it can

actually be used to deliver entire websites. Finally, Amazon CloudFront supports media streaming,

using both HTTP and RTMP.

AWS Storage Gateway:

AWS Storage Gateway is a service connecting an on-premises software appliance with cloud-based

storage to provide seamless and secure integration between an organisation’s on-premises IT

environment and AWS storage infrastructure. The service enables one to store data securely on the

AWS cloud in a scalable and cost-effective manner.

AWS Storage Gateway supports industry-standard storage protocols that work with the

organisation’s existing applications. It provides low-latency performance by caching frequently

accessed data on-premises while encrypting and storing all of the data in Amazon S3 or Amazon

Glacier.

Figure 3: AWS Storage Gateway Scheme Components

AWS Storage Gateway’s software appliance is available for download as a Virtual Machine (VM)

image that one installs on a host in the data center and then registers with the site’s AWS account

through the AWS Management Console. The storage associated with the appliance is exposed as an

iSCSI device that can be mounted by the organisation’s on-premises applications.

There are three configurations for AWS Storage Gateway: Gateway-Cached volumes, Gateway-

Stored volumes, and Gateway-Virtual Tape Libraries (VTL).

Gateway-Cached volumes allow expansion of local storage capacity into Amazon S3. All data stored

on a Gateway-Cached volume is moved to Amazon S3, while recently read data is retained in local

storage to provide low-latency access. While each volume is limited to a maximum size of 32TB, a

single gateway can support up to 32 volumes for a maximum storage of 1 PB. Point-in-time

snapshots can be taken to back up the AWS Storage Gateway.

These snapshots are performed incrementally, and only the data that has changed since the last

snapshot is stored. All Gateway-Cached volume data and snapshot data is transferred to Amazon S3

over encrypted Secure Sockets Layer (SSL) connections. It is encrypted at rest in Amazon S3 using

Server-Side Encryption (SSE). However, the data cannot be directly accessed with the Amazon S3 API

or other tools such as the Amazon S3 console; instead it must be accessed through the AWS Storage

Gateway service.

Gateway-Stored volumes allow storage of data on the organisation’s on-premises storage. The

stored data is then asynchronously backed up to Amazon S3. This provides low-latency access to all

data, while also providing off-site backups taking advantage of the durability of Amazon S3. The data

is backed up in the form of Amazon Elastic Block Store (Amazon EBS) snapshots. While each volume

is limited to a maximum size of 16TB, a single gateway can support up to 32 volumes for a maximum

storage of 512TB.

Gateway Virtual Tape Libraries (VTLs) offer a durable, cost-effective solution to archive the

organisation’s data on the AWS cloud. The VTL interface allows leveraging of existing tape-based

backup application infrastructure to store data on virtual tape cartridges that are created on the

Gateway-VTL. A virtual tape is analogous to a physical tape cartridge, except the data is stored on

the AWS cloud. Tapes are created blank through the console or programmatically and then filled

with backed up data. A gateway can contain up to 1,500 tapes (1 PB) of total tape data.

Virtual tapes appear in the organisation’s gateway’s VTL, a virtualised version of a physical tape

library. Virtual tapes are discovered by the backup application using its standard media inventory

procedure. When the tape software ejects a tape, it is archived on a Virtual Tape Shelf (VTS) and

stored in Amazon Glacier. A site is allowed 1 VTS per AWS region, but multiple gateways in the same

region can share a VTS.

AWS CloudTrail:

AWS CloudTrail provides visibility into user activity by recording API calls made on the organisation’s

account. AWS CloudTrail records important information about each API call, including the name of

the API, the identity of the caller, the time of the API call, the request parameters and the response

elements returned by the AWS service. This information helps the organisation to track changes

made to AWS resources and to troubleshoot operational issues.

AWS CloudTrail makes it easier to ensure compliance with internal policies and regulatory standards.

It captures AWS API calls and related events made by or on behalf of an AWS account and delivers

log files to an Amazon S3 bucket that is pre-specified. Optionally, one can configure AWS CloudTrail

to deliver events to a log group monitored by Amazon CloudWatch Logs.

One can also choose to receive Amazon Simple Notification Service (Amazon SNS) notifications each

time a log file is delivered to a specified bucket. One can create a trail with the AWS CloudTrail

console, the AWS Command Line Interface (CLI) or the AWS CloudTrail API. A trail is a configuration

that enables logging of the AWS API activity and related events in the organisation’s account.

AWS CloudTrail typically delivers log files within 15 minutes of an API call. In addition, the service

publishes new log files multiple times an hour, usually about every five minutes. These log files

contain API calls from all of the account’s services that support AWS CloudTrail.

AWS Kinesis:

Kinesis is a platform for handling massive streaming data on AWS, offering services to make it easier

to load and analyse streaming data and also providing the ability for one to build custom streaming

data applications for specialised needs. It has similar functionality to Apache-Hadoop Kafka and

Storm.

Figure 4: AWS Kinesis Data Streams High-Level Architecture

Amazon Kinesis is a streaming data platform consisting of three services addressing different real-

time streaming data challenges:

1. Kinesis Firehose: A service enabling the loading of massive volumes of streaming data into AWS.

Kinesis Firehose receives stream data and stores it in Amazon S3, Amazon Redshift or Amazon

Elasticsearch. One does not need to write any code – the client just creates a delivery stream

and configures the destination for the data. Clients write data to the stream using an AWS API

call, and the data is automatically sent to the proper destination.

2. Kinesis Streams: A service enabling the building of custom applications for more complex

analysis of streaming data in real time. Kinesis Streams enables one to collect and process large

streams of data records in real time. Using AWS SDKs, one can create an Amazon Kinesis Streams

application that processes the data as it moves through the stream. Because response time for

data intake and processing is in near real time, the processing is typically lightweight. Amazon

Kinesis Streams can scale to support nearly limitless data streams by distributing incoming data

across a number of shards. If any shard becomes too busy, it can be further divided into more

shards to distribute the load further. The processing is then executed on consumers, which read

data from the shards and run the Amazon Kinesis Streams application.

3. Kinesis Analytics: A service enabling analysis of streaming data real time with standard SQL.

Each of these services can scale to handle virtually limitless data streams.

AWS Elastic MapReduce (EMR):

EMS provides the organisation with a fully managed, on-demand Hadoop framework. Amazon EMR

reduces the complexity and up-front costs of setting up Hadoop and, combined with the scale of

AWS, gives the ability to spin up large Hadoop clusters instantly and start processing within minutes.

When an AWS EMR cluster is launched, several options are specified, the key ones being:

• The instance type of the nodes in the cluster

• The number of nodes in the cluster

• The version of Hadoop to be run (EMR supports several recent versions of Apache Hadoop and

also several versions of MapReduce.)

• Additional tools or applications like Hive, Pig, Spark or Kafka

Figure 5: Typical EMR Cluster Master-Slave Node Layout

AWS EMR data can be stored in HDFS (Hadoop Distributed Filesystem) or EMRFS (EMR File System).

EMRFS allows the data to be held in lower-cost Amazon S3 storage.

We will discuss more fully the attributes and operational features of the Hadoop cluster and

supporting ecosystem in the Big Data Module.

AWS CloudFormation:

AWS CloudFormation is a service that helps one model and set up AWS resources so that the

organisation can spend less time managing those resources and more time focusing on the

applications that run in AWS. AWS CloudFormation allows organisations to deploy, modify and

update resources in a controlled and predictable way, in effect applying version control to AWS

infrastructure the same way one would do with software.

AWS CloudFormation gives developers and systems administrators an easy way to create and

manage a collection of related AWS resources, provisioning and updating them in an orderly and

predictable fashion. When AWS CloudFormation is used, cloud administrators work with templates

and stacks. They create AWS CloudFormation templates to define AWS resources and their

properties. A template is a text file whose format complies with the JSON standard. AWS

CloudFormation uses these templates as blueprints for building the organisations’s AWS resources.

When AWS CloudFormation is used, cloud administrators manage related resources as a single unit

called a stack. They create, update and delete a collection of resources by creating, updating and

deleting stacks. All of the resources in a stack are defined by the stack’s AWS CloudFormation

template.

Figure 6: AWS CloudFormation Workflow

For example, one may set up a template that includes an Auto Scaling group, Elastic Load Balancing

load balancer and an Amazon RDS database instance. To create those resources, it will be necessary

to create a stack by submitting a template that defines those resources, and AWS CloudFormation

handles all of the provisioning based on this.

After all the resources have been created, AWS CloudFormation reports that the stack has been

created. One can then start using the resources in the stack. If stack creation fails, AWS

CloudFormation rolls back the changes by deleting the resources that it created. Often one will need

to launch stacks from the same template, but with minor variations, such as within a different

Amazon VPC or using AMIs from a different region.

These variations can be addressed using parameters. One can use parameters to customise aspects

of the template at runtime, when the stack is built. For example, one can pass the Amazon RDS

database size, EC2 instance types, database and web server port numbers to AWS CloudFormation

when the stack is being created. By leveraging template parameters, one can use a single template

for many infrastructure deployments with different configuration values.

AWS provides an extensive library of sample templates, which cloud administrators and developers

can use as desired (viewable at url: https://aws.amazon.com/cloudformation/aws-cloudformation-

templates/). The figure below shows CloudFormation template library items for creating a variety of

VPCs –

Amazon Virtual Private Cloud

Template Name Description View View in

Designer

A single Amazon EC2 in an Amazon VPC Creates a VPC and adds an Amazon EC2 instance

with an Elastic IP address and a security group.

View View in

Designer

Amazon VPC with static routing to an

existing VPN

Creates a private subnet with a VPN connection that

uses static routing to an existing VPN endpoint.

View View in

Designer

Autoscaling and load-balancing website

in an Amazon VPC

Creates a load balancing, auto scaling sample

website in an existing VPC.

View View in

Designer

Amazon VPC with DNS and public IP

addresses

Creates a VPC with DNS support and public IP

addresses enabled.

View View in

Designer

Publicly accessible Amazon EC2

instances that are in an Auto Scaling

group

Creates a load balancing, autoscaling group with

instances that are directly accessible from the

Internet.

View View in

Designer

Amazon EC2 with multiple dynamic IP

addresses in an Amazon VPC

Creates an Amazon EC2 instance with multiple

dynamic IP addresses in a VPC.

View View in

Designer

Amazon EC2 with multiple static IP

addresses in an Amazon VPC

Creates an Amazon EC2 instance with multiple static

IP addresses in a VPC.

View View in

Designer

Table 7: AWS CloudFormation Sample Templates

AWS Elastic Beanstalk:

AWS Elastic Beanstalk is the fastest and simplest way to get an application up and running on AWS.

Developers can simply upload their application code, and the service automatically handles all of the

details, such as resource provisioning, load balancing, Auto Scaling and monitoring.

Figure 8: AWS Elastic Beanstalk Multi-Programming Language Support

AWS comprises dozens of building block services, each of which exposes an area of functionality.

While the variety of services offers flexibility for how organisations want to manage their AWS

infrastructure, it can be challenging to figure out which services to use and how to provision them.

With AWS Elastic Beanstalk, one can quickly deploy and manage applications on the AWS cloud

without worrying about the infrastructure that runs those applications. AWS Elastic Beanstalk

reduces management complexity without restricting choice or control.

There are key components that comprise AWS Elastic Beanstalk and work together to provide the

necessary services to deploy and manage applications easily in the cloud. An AWS Elastic Beanstalk

application is the logical collection of these AWS Elastic Beanstalk components, which includes

environments, versions and environment configurations. In AWS Elastic Beanstalk, an application is

conceptually similar to a folder.

An application version refers to a specific, labeled iteration of deployable code for a web application.

An application version points to an Amazon S3 object that contains the deployable code.

Applications can have many versions, and each application version is unique. In a running

environment, organisations can deploy any application version they already uploaded to the

application, or they can upload and immediately deploy a new application version.

Organisations might upload multiple application versions to test differences between one version of

their web application and another. An environment is an application version that is deployed onto

AWS resources. Each environment runs only a single application version at a time; however, the

same version or different versions can run in as many environments at the same time as needed.

When an environment is created, AWS Elastic Beanstalk provisions the resources needed to run the

application version that is specified.

An environment configuration identifies a collection of parameters and settings that define how an

environment and its associated resources behave. When an environment’s configuration settings are

updated, AWS Elastic Beanstalk automatically applies the changes to existing resources or deletes

and deploys new resources depending on the type of change.

When an AWS Elastic Beanstalk environment is launched, the environment tier, platform and

environment type are specified. The environment tier that is chosen determines whether AWS

Elastic Beanstalk provisions resources to support a web application that handles HTTP(S) requests or

an application that handles background-processing tasks.

An environment tier whose web application processes web requests is known as a web server tier.

An environment tier whose application runs background jobs is known as a worker tier. Currently,

AWS Elastic Beanstalk provides platform support for the programming languages Java, Node.js, PHP,

Python, Ruby and Go with support for the web containers Tomcat, Passenger, Puma and Docker.

AWS Config:

AWS Config is a fully managed service that provides cloud administrators with an AWS resource

inventory, configuration history and configuration change notifications to enable security and

governance. With AWS Config, one can discover existing and deleted AWS resources, determine the

organisation’s overall compliance against rules and delve into configuration details of a resource at

any point in time.

Figure 9: AWS Config Workflow

These capabilities enable compliance auditing, security analysis, resource change tracking and

troubleshooting. AWS Config provides a detailed view of the configuration of AWS resources in the

AWS account. This includes how the resources are related and how they were configured in the past

so that one can see how the configurations and relationships change over time. AWS Config defines

a resource as an entity that works within AWS, such as an Amazon EC2 instance, an Amazon EBS

volume, a security group or an Amazon VPC.

When one turns on AWS Config, it first discovers the supported AWS resources that exist in the

account and generates a configuration item for each resource. A configuration item represents a

point-in-time view of the various attributes of a supported AWS resource that exists in one’s

account. The components of a configuration item include metadata, attributes, relationships,

current configuration and related events. AWS Config will generate configuration items when the

configuration of a resource changes, and it maintains historical records of the configuration items of

the resources from the time one starts the configuration recorder. The configuration recorder stores

the configurations of the supported resources in the account as configuration items.

By default, AWS Config creates configuration items for every supported resource in the region. If it is

not wanted to have AWS Config create configuration items for all supported resources, one can

specify the resource types to be tracked.

Organisations often need to assess the overall compliance and risk status from a configuration

perspective, view compliance trends over time and pinpoint which configuration change caused a

resource to drift out of compliance. An AWS Config Rule represents desired configuration settings

for specific AWS resources or for an entire AWS account. While AWS Config continuously tracks the

resource configuration changes, it checks whether these changes violate any of the conditions in the

organisation’s rules.

If a resource violates a rule, AWS Config flags the resource and the rule as noncompliant and notifies

the administrators through Amazon SNS. AWS Config makes it easy to track resource configuration

without the need for up-front investments and while avoiding the complexity of installing and

updating agents for data collection or maintaining large databases. Once AWS Config is enabled,

organisations can view continuously updated details of all configuration attributes associated with

AWS resources.