Since I have a background and experience in cloud operations, I started my
research by looking at the tools available from the major cloud service
providers who compete with IBM. I reviewed Big Data service offerings and
tools from Alibaba, Amazon, Google, and Microsoft.
Alibaba has a solution called Hologres, a real-time data warehouse compatible
with Postgres SQL. They also offer an analytics product called MaxCompute,
enabling one to query and analyze data with low latency and high
concurrency. One can utilize MaxCompute to analyze logs, e-commerce
transactions, exploration of user profile data and behavior, and other data
analytics projects f (What is MaxCompute? 2022). Alibaba also has a product
named QuickBI to perform data analytics, exploration, and reporting.
Amazon offers several different data analytics services that support
computing with a large volume of unstructured and structured data with low
latency. One appealing offering is Amazon EMR. This offering enables one to
run Big Data workloads such as Apache Spark, Hive, and Presto using
complementary Amazon services such as Amazon EC2 clusters (Amazon
EMR, n.d.).
Since we have not studied Presto in this class, further research revealed that
Presto is an open-source distributed SQL query engine. Presto is a community-
owned project by the Presto Foundation. (What is Presto? n.d.). I found it
interesting that one can use Presto to query data where it resides (such as
Cassandra, Hive, and traditional SQL databases), and Facebook also uses
Presto to run queries against its 300 PB (petabyte) data warehouse (What is
Presto? n.d.). From a glance at the Presto documentation, it looks like one can
use this tool as an alternative to Apache Pig and Hive (Use Cases, n.d.). One
benefit of using this tool rather than Apache Pig or Hive is that one is not
limited to just HDFS queries. Even though this tool originated at Facebook,
the documentation indicates that Facebook and the open-source community
continue developing the tool (Use Cases, n.d.).
There are other computing service offerings that one can leverage in Big Data
analytics projects, such as Amazon Athena (to query Amazon S3 storage),
Amazon RedShift (Data Warehousing), and Amazon Kinesis (real-time video
and stream analysis). Amazon QuickSight (business analytics), Amazon
OpenSearch (text and unstructured data search), and AWS Glue DataBrew
(data cleaning and normalization) round out their data analytics offering
(Analytics on AWS, n.d.).
I also noticed that Amazon provides several tutorials on using the service that
one could find helpful since a big part of Big Data analytics is getting the data
ingested into a data store to analyze the data. I liked that Amazon's
documentation appears to be thorough, mature, and full of examples where
one can take advantage of their service offerings. Amazon also provides use
cases and customer success stories to help explain how their service offerings
fit together to provide a complete Data Analytics stack. As an example,
Amazon explains how Pearson used Amazon OpenSearch to analyze and gain
insights from log data that it collects to improve the company's learning
platform (Pearson Boosts Security and Productivity Using Amazon
OpenSearch Service, 2020).
Not surprisingly, Google also has many service offerings that provide
compute capacity to handle a large volume of unstructured and structured data
with low latency. The centerpiece of their Big Data service offering seems to
be Google BigQuery, which Google claims to have a 26 – 34% lower Total
Cost of Ownership over three years than cloud data warehouse alternatives
(BigQuery, n.d.). BigQuery also has various client libraries available. A data
analyst can leverage familiar programming languages such as Python, Java,
Javascript, and Go to write and execute queries to gain valuable insights from
Big Data (What is BigQuery? n.d.).
One complimentary offering from Google that sounds interesting is their
BigQuery Omni product, which enables data analysis across clouds such as
Microsoft Azure and Amazon Web Services. One tremendous benefit to
Microsoft and Amazon customers is that they do not have to move their data to
Google Cloud to take advantage of Google BigQuery (BigQuery Omni, n.d.).
As a former Microsoft Azure administrator, I can certainly appreciate
Google's interest in serving customers to operate resources and services in one
or more public clouds.
Last but not least, Microsoft has several service offerings to support
computing with a large volume of unstructured and structured data with low
latency. It seems like their service offerings focus on either Azure HDInsight
or Azure Databricks. Azure HDInsight essentially enables one to leverage a
customizable environment of Hadoop components distributed in Microsoft's
cloud (What is Azure HDInsight? 2021). I found this offering appealing since
one can scale a workload up and down and only pay for what one uses (similar
to IBM's Watson Studio cloud offering). Depending on business need, one can
also select from any number of cluster types to deploy in Microsoft Azure. For
example, one could create an Apache Spark cluster to support a Big Data
analytics project where in-memory processing is vital for performance
reasons. Alternatively, one could create an Apache Storm cluster if large data
streams need to be ingested and analyzed quickly.
I found it challenging to review some of these cloud-based service offerings
because it is hard to understand what Big Data technology is behind each
offering. It seems like some of the cloud vendors have repurposed open-source
tools for the cloud to make it easy for customers to consume capacity on a pay-
as-you-go basis, while others have built their own proprietary tools to handle
Big Data needs.
I think one needs to be naturally curious and determined to learn more to
understand the underpinnings of each one. One could spend hours reading
about the different features and experimenting with capabilities. For example,
I spent some time reading about Amazon Kinesis which provides some
exciting real-time data analysis capabilities with other Amazon service
offerings. I learned that Zillow uses Kinesis Data Streams to collect public
data and Multiple Listing Service (MLS) services which helps the company
keep buyers and sellers updated on estimated home values (Amazon Kinesis,
n.d.). This capability sounds like an Apache Kafka or Flume feature, so I spent
some time reading the Kinesis Data Streams developer documentation to see
what one can do with the service.
It isn't easy to render an informed opinion on which one is best with all the
available tools choices. My favorite answer, as a former consultant, is - "it
depends". I think one needs to carefully assess the business requirements for
any Big Data analytics project and then evaluate the ability of various Big
Data platforms, tools, and service providers to meet the requirements. All
kinds of questions come to mind: Does one need to analyze data in real-time or
using batch processing? How much disk storage is required to store the data
now and in the future? Who needs to access the data, and what is the data
governance strategy to ensure the data is properly secured? Does the data need
to be encrypted at rest? How responsive do queries need to be? What kind of
reports will one need to generate and distribute? Is the data structured, semi-
structured, unstructured, or some combination of all three? Will a data
warehouse suffice to store the data, or is a more extensible data lake with data
originating from disparate data sources required? The list goes on and on, but
those are a few questions one may need to answer before selecting Big Data
tools, platforms, and a cloud service provider.
Lastly, my experience with product/tool/platform evaluations has been that
some vendors will practically fall over themselves to earn your business and
trust. For example, our agency was interested in determining how well one
vendor's machine learning platform would identify fraudulent transactions in a
dataset we had available. We invited the vendor to work with us side by side in
a sort of "hackathon" where we hacked through the business problem and
developed a machine learning model using the vendor's platform. That
exercise gave us a good understanding of the vendor's capabilities and a
baseline against which we could compare products from other vendors. When
it comes to vendors wanting to show off shiny new toys coming out of their
respective research labs, there is typically no shortage of enthusiasm from the
vendor's product managers.
I would be interested in hearing what tools others in our class have discovered
for computing with a large volume of unstructured and structured with low
latency.
Very Respectfully,
Michael Goddard
References:
"Amazon EMR" (n.d.). Amazon.com. Retrieved from
https://aws.amazon.com/emr/?c=a&sec=srv
"Amazon Kinesis" (n.d.). Amazon.com. Retrieved from
https://aws.amazon.com/kinesis/?c=a&sec=srv
"BigQuery Omni" (n.d.) Google.com. Retrieved from
https://cloud.google.com/bigquery-omni/docs/introduction
"Pearson Boosts Security and Productivity Using Amazon OpenSearch
Service" (2020). Amazon.com. Retrieved from
https://aws.amazon.com/solutions/case-studies/pearson-elasticsearch-case-
study/
"Use Cases" (n.d.). Presto.io. Retrieved from https://prestodb.io/
"What is Azure HDInsight?" (2021, November 18). Microsoft.com. Retrieved
from https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-overview
"What is BigQuery?" (n.d.). Google.com Retrieved from
https://cloud.google.com/bigquery/docs/introduction
"What is MaxCompute?" (2022, January 28). Alibaba. Retrieved from
https://www.alibabacloud.com/help/en/doc-detail/27800.htm
"What is Presto?" (n.d.). Presto.io. Retrieved from https://prestodb.io/
