Since I have a background and experience in cloud operations, I started my research by looking at the
tools available from the major cloud service providers who compete with IBM. I reviewed Big Data
service offerings and tools from Alibaba, Amazon, Google, and Microsoft.
Alibaba has a solution called Hologres, a real-time data warehouse compatible with Postgres SQL.
They also offer an analytics product called MaxCompute, enabling one to query and analyze data
with low latency and high concurrency. One can utilize MaxCompute to analyze logs, e-commerce
transactions, exploration of user profile data and behavior, and other data analytics projects i (What
is MaxCompute? 2022). Alibaba also has a product named QuickBI to perform data analytics,
exploration, and reporting.
Amazon offers several different data analytics services that support computing with a large volume
of unstructured and structured data with low latency. One appealing offering is Amazon EMR. This
offering enables one to run Big Data workloads such as Apache Spark, Hive, and Presto using
complementary Amazon services such as Amazon EC2 clusters (Amazon EMR, n.d.).
Since we have not studied Presto in this class, further research revealed that Presto is an open-
source distributed SQL query engine. Presto is a community-owned project by the Presto
Foundation. (What is Presto? n.d.). I found it interesting that one can use Presto to query data where
it resides (such as Cassandra, Hive, and traditional SQL databases), and Facebook also uses Presto to
run queries against its 300 PB (petabyte) data warehouse (What is Presto? n.d.). From a glance at the
Presto documentation, it looks like one can use this tool as an alternative to Apache Pig and Hive
(Use Cases, n.d.). One benefit of using this tool rather than Apache Pig or Hive is that one is not
limited to just HDFS queries. Even though this tool originated at Facebook, the documentation
indicates that Facebook and the open-source community continue developing the tool (Use Cases,
n.d.).
There are other computing service offerings that one can leverage in Big Data analytics projects, such
as Amazon Athena (to query Amazon S3 storage), Amazon RedShift (Data Warehousing), and
Amazon Kinesis (real-time video and stream analysis). Amazon QuickSight (business analytics),
Amazon OpenSearch (text and unstructured data search), and AWS Glue DataBrew (data cleaning
and normalization) round out their data analytics offering (Analytics on AWS, n.d.).
I also noticed that Amazon provides several tutorials on using the service that one could find helpful
since a big part of Big Data analytics is getting the data ingested into a data store to analyze the data.
I liked that Amazon's documentation appears to be thorough, mature, and full of examples where
one can take advantage of their service offerings. Amazon also provides use cases and customer
success stories to help explain how their service offerings fit together to provide a complete Data
Analytics stack. As an example, Amazon explains how Pearson used Amazon OpenSearch to analyze
and gain insights from log data that it collects to improve the company's learning platform (Pearson
Boosts Security and Productivity Using Amazon OpenSearch Service, 2020).
Not surprisingly, Google also has many service offerings that provide compute capacity to handle a
large volume of unstructured and structured data with low latency. The centerpiece of their Big Data
service offering seems to be Google BigQuery, which Google claims to have a 26 – 34% lower Total
Cost of Ownership over three years than cloud data warehouse alternatives (BigQuery, n.d.).
BigQuery also has various client libraries available. A data analyst can leverage familiar programming
languages such as Python, Java, Javascript, and Go to write and execute queries to gain valuable
insights from Big Data (What is BigQuery? n.d.).
One complimentary offering from Google that sounds interesting is their BigQuery Omni product,
which enables data analysis across clouds such as Microsoft Azure and Amazon Web Services. One
tremendous benefit to Microsoft and Amazon customers is that they do not have to move their data
to Google Cloud to take advantage of Google BigQuery (BigQuery Omni, n.d.). As a former Microsoft
Azure administrator, I can certainly appreciate Google's interest in serving customers to operate
resources and services in one or more public clouds.
Last but not least, Microsoft has several service offerings to support computing with a large volume
of unstructured and structured data with low latency. It seems like their service offerings focus on
either Azure HDInsight or Azure Databricks. Azure HDInsight essentially enables one to leverage a
customizable environment of Hadoop components distributed in Microsoft's cloud (What is Azure
HDInsight? 2021). I found this offering appealing since one can scale a workload up and down and
only pay for what one uses (similar to IBM's Watson Studio cloud offering). Depending on business
need, one can also select from any number of cluster types to deploy in Microsoft Azure. For
example, one could create an Apache Spark cluster to support a Big Data analytics project where in-
memory processing is vital for performance reasons. Alternatively, one could create an Apache
Storm cluster if large data streams need to be ingested and analyzed quickly.
I found it challenging to review some of these cloud-based service offerings because it is hard to
understand what Big Data technology is behind each offering. It seems like some of the cloud
vendors have repurposed open-source tools for the cloud to make it easy for customers to consume
capacity on a pay-as-you-go basis, while others have built their own proprietary tools to handle Big
Data needs.
I think one needs to be naturally curious and determined to learn more to understand the
underpinnings of each one. One could spend hours reading about the different features and
experimenting with capabilities. For example, I spent some time reading about Amazon Kinesis which
provides some exciting real-time data analysis capabilities with other Amazon service offerings. I
learned that Zillow uses Kinesis Data Streams to collect public data and Multiple Listing Service (MLS)
services which helps the company keep buyers and sellers updated on estimated home values
(Amazon Kinesis, n.d.). This capability sounds like an Apache Kafka or Flume feature, so I spent some
time reading the Kinesis Data Streams developer documentation to see what one can do with the
service.
It isn't easy to render an informed opinion on which one is best with all the available tools choices.
My favorite answer, as a former consultant, is - "it depends". I think one needs to carefully assess the
business requirements for any Big Data analytics project and then evaluate the ability of various Big
Data platforms, tools, and service providers to meet the requirements. All kinds of questions come to
mind: Does one need to analyze data in real-time or using batch processing? How much disk storage
is required to store the data now and in the future? Who needs to access the data, and what is the
data governance strategy to ensure the data is properly secured? Does the data need to be
encrypted at rest? How responsive do queries need to be? What kind of reports will one need to
generate and distribute? Is the data structured, semi-structured, unstructured, or some combination
of all three? Will a data warehouse suffice to store the data, or is a more extensible data lake with
data originating from disparate data sources required? The list goes on and on, but those are a few
questions one may need to answer before selecting Big Data tools, platforms, and a cloud service
provider.
Lastly, my experience with product/tool/platform evaluations has been that some vendors will
practically fall over themselves to earn your business and trust. For example, our agency was
interested in determining how well one vendor's machine learning platform would identify
fraudulent transactions in a dataset we had available. We invited the vendor to work with us side by
side in a sort of "hackathon" where we hacked through the business problem and developed a
machine learning model using the vendor's platform. That exercise gave us a good understanding of
the vendor's capabilities and a baseline against which we could compare products from other
vendors. When it comes to vendors wanting to show off shiny new toys coming out of their
respective research labs, there is typically no shortage of enthusiasm from the vendor's product
managers.
I would be interested in hearing what tools others in our class have discovered for computing with a
large volume of unstructured and structured with low latency.
Very Respectfully,
References:
"Amazon EMR" (n.d.). Amazon.com. Retrieved from https://aws.amazon.com/emr/?c=a&sec=srv
"Amazon Kinesis" (n.d.). Amazon.com. Retrieved from
https://aws.amazon.com/kinesis/?c=a&sec=srv
"BigQuery Omni" (n.d.) Google.com. Retrieved from https://cloud.google.com/bigquery-
omni/docs/introduction
"Pearson Boosts Security and Productivity Using Amazon OpenSearch Service" (2020). Amazon.com.
Retrieved from https://aws.amazon.com/solutions/case-studies/pearson-elasticsearch-case-study/
"Use Cases" (n.d.). Presto.io. Retrieved from https://prestodb.io/
"What is Azure HDInsight?" (2021, November 18). Microsoft.com. Retrieved from
https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-overview
"What is BigQuery?" (n.d.). Google.com Retrieved from
https://cloud.google.com/bigquery/docs/introduction
"What is MaxCompute?" (2022, January 28). Alibaba. Retrieved from
https://www.alibabacloud.com/help/en/doc-detail/27800.htm
"What is Presto?" (n.d.). Presto.io. Retrieved from https://prestodb.io/
