Since I have a background and experience in cloud operations, I started my
research by looking at the tools available from the major cloud service
providers who compete with IBM. I reviewed Big Data service offerings and
tools from Alibaba, Amazon, Google, and Microsoft.
Alibaba has a solution called Hologres, a real-time data warehouse
compatible with Postgres SQL. They also offer an analytics product called
MaxCompute, enabling one to query and analyze data with low latency and
high concurrency. One can utilize MaxCompute to analyze logs, e-
commerce transactions, exploration of user profile data and behavior, and
other data analytics projects g (What is MaxCompute? 2022). Alibaba also
has a product named QuickBI to perform data analytics, exploration, and
reporting.
Amazon offers several different data analytics services that support
computing with a large volume of unstructured and structured data with low
latency. One appealing offering is Amazon EMR. This offering enables one
to run Big Data workloads such as Apache Spark, Hive, and Presto using
complementary Amazon services such as Amazon EC2 clusters (Amazon
EMR, n.d.).
Since we have not studied Presto in this class, further research revealed that
Presto is an open-source distributed SQL query engine. Presto is a
community-owned project by the Presto Foundation. (What is Presto? n.d.).
I found it interesting that one can use Presto to query data where it resides
(such as Cassandra, Hive, and traditional SQL databases), and Facebook also
uses Presto to run queries against its 300 PB (petabyte) data warehouse
(What is Presto? n.d.). From a glance at the Presto documentation, it looks
like one can use this tool as an alternative to Apache Pig and Hive (Use
Cases, n.d.). One benefit of using this tool rather than Apache Pig or Hive is
that one is not limited to just HDFS queries. Even though this tool originated
at Facebook, the documentation indicates that Facebook and the open-source
community continue developing the tool (Use Cases, n.d.).
There are other computing service offerings that one can leverage in Big
Data analytics projects, such as Amazon Athena (to query Amazon S3
storage), Amazon RedShift (Data Warehousing), and Amazon Kinesis (real-
time video and stream analysis). Amazon QuickSight (business analytics),
Amazon OpenSearch (text and unstructured data search), and AWS Glue
DataBrew (data cleaning and normalization) round out their data analytics
offering (Analytics on AWS, n.d.).
I also noticed that Amazon provides several tutorials on using the service
that one could find helpful since a big part of Big Data analytics is getting
the data ingested into a data store to analyze the data. I liked that Amazon's
documentation appears to be thorough, mature, and full of examples where
one can take advantage of their service offerings. Amazon also provides use
cases and customer success stories to help explain how their service
offerings fit together to provide a complete Data Analytics stack. As an
example, Amazon explains how Pearson used Amazon OpenSearch to
analyze and gain insights from log data that it collects to improve the
company's learning platform (Pearson Boosts Security and Productivity
Using Amazon OpenSearch Service, 2020).
Not surprisingly, Google also has many service offerings that provide
compute capacity to handle a large volume of unstructured and structured
data with low latency. The centerpiece of their Big Data service offering
seems to be Google BigQuery, which Google claims to have a 26 – 34%
lower Total Cost of Ownership over three years than cloud data warehouse
alternatives (BigQuery, n.d.). BigQuery also has various client libraries
available. A data analyst can leverage familiar programming languages such
as Python, Java, Javascript, and Go to write and execute queries to gain
valuable insights from Big Data (What is BigQuery? n.d.).
One complimentary offering from Google that sounds interesting is their
BigQuery Omni product, which enables data analysis across clouds such as
Microsoft Azure and Amazon Web Services. One tremendous benefit to
Microsoft and Amazon customers is that they do not have to move their data
to Google Cloud to take advantage of Google BigQuery (BigQuery Omni,
n.d.). As a former Microsoft Azure administrator, I can certainly appreciate
Google's interest in serving customers to operate resources and services in
one or more public clouds.
Last but not least, Microsoft has several service offerings to support
computing with a large volume of unstructured and structured data with low
latency. It seems like their service offerings focus on either Azure HDInsight
or Azure Databricks. Azure HDInsight essentially enables one to leverage a
customizable environment of Hadoop components distributed in Microsoft's
cloud (What is Azure HDInsight? 2021). I found this offering appealing
since one can scale a workload up and down and only pay for what one uses
(similar to IBM's Watson Studio cloud offering). Depending on business
need, one can also select from any number of cluster types to deploy in
Microsoft Azure. For example, one could create an Apache Spark cluster to
support a Big Data analytics project where in-memory processing is vital for
performance reasons. Alternatively, one could create an Apache Storm
cluster if large data streams need to be ingested and analyzed quickly.
I found it challenging to review some of these cloud-based service offerings
because it is hard to understand what Big Data technology is behind each
offering. It seems like some of the cloud vendors have repurposed open-
source tools for the cloud to make it easy for customers to consume capacity
on a pay-as-you-go basis, while others have built their own proprietary tools
to handle Big Data needs.
I think one needs to be naturally curious and determined to learn more to
understand the underpinnings of each one. One could spend hours reading
about the different features and experimenting with capabilities. For
example, I spent some time reading about Amazon Kinesis which provides
some exciting real-time data analysis capabilities with other Amazon service
offerings. I learned that Zillow uses Kinesis Data Streams to collect public
data and Multiple Listing Service (MLS) services which helps the company
keep buyers and sellers updated on estimated home values (Amazon Kinesis,
n.d.). This capability sounds like an Apache Kafka or Flume feature, so I
spent some time reading the Kinesis Data Streams developer documentation
to see what one can do with the service.
It isn't easy to render an informed opinion on which one is best with all the
available tools choices. My favorite answer, as a former consultant, is - "it
depends". I think one needs to carefully assess the business requirements for
any Big Data analytics project and then evaluate the ability of various Big
Data platforms, tools, and service providers to meet the requirements. All
kinds of questions come to mind: Does one need to analyze data in real-time
or using batch processing? How much disk storage is required to store the
data now and in the future? Who needs to access the data, and what is the
data governance strategy to ensure the data is properly secured? Does the
data need to be encrypted at rest? How responsive do queries need to be?
What kind of reports will one need to generate and distribute? Is the data
structured, semi-structured, unstructured, or some combination of all three?
Will a data warehouse suffice to store the data, or is a more extensible data
lake with data originating from disparate data sources required? The list goes
on and on, but those are a few questions one may need to answer before
selecting Big Data tools, platforms, and a cloud service provider.
Lastly, my experience with product/tool/platform evaluations has been that
some vendors will practically fall over themselves to earn your business and
trust. For example, our agency was interested in determining how well one
vendor's machine learning platform would identify fraudulent transactions in
a dataset we had available. We invited the vendor to work with us side by
side in a sort of "hackathon" where we hacked through the business problem
and developed a machine learning model using the vendor's platform. That
exercise gave us a good understanding of the vendor's capabilities and a
baseline against which we could compare products from other vendors.
When it comes to vendors wanting to show off shiny new toys coming out of
their respective research labs, there is typically no shortage of enthusiasm
from the vendor's product managers.
I would be interested in hearing what tools others in our class have
discovered for computing with a large volume of unstructured and structured
with low latency.
Very Respectfully,
Michael Goddard
References:
"Amazon EMR" (n.d.). Amazon.com. Retrieved from
https://aws.amazon.com/emr/?c=a&sec=srv
"Amazon Kinesis" (n.d.). Amazon.com. Retrieved from
https://aws.amazon.com/kinesis/?c=a&sec=srv
"BigQuery Omni" (n.d.) Google.com. Retrieved from
https://cloud.google.com/bigquery-omni/docs/introduction
"Pearson Boosts Security and Productivity Using Amazon OpenSearch
Service" (2020). Amazon.com. Retrieved from
https://aws.amazon.com/solutions/case-studies/pearson-elasticsearch-case-
study/
"Use Cases" (n.d.). Presto.io. Retrieved from https://prestodb.io/
"What is Azure HDInsight?" (2021, November 18). Microsoft.com.
Retrieved from https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-
overview
"What is BigQuery?" (n.d.). Google.com Retrieved from
https://cloud.google.com/bigquery/docs/introduction
"What is MaxCompute?" (2022, January 28). Alibaba. Retrieved from
https://www.alibabacloud.com/help/en/doc-detail/27800.htm
