The Sentiment Analysis of Twitter Data:
Using DB2 on IBM Cloud to Analyze the First GOP Debate (Twitter) Dataset
Date: 03/15/2022
Introduction:
The United States of America conducts a presidential election every four years. The winner in
the presidential election emerges as the president of the nation. There are two major parties, the
Republicans (GOP) and the Democrats. The registration and campaign for the election usually
start early, almost a year or two before the election date.
Twitter, which was created in March 2006, was new to many. It was barely two years after the
creation of Twitter, a black president was elected to office in 2008. President Obama continued
in the office after winning again in 2012. The last term of President Obama will end in January
2017, which will give room for another election in 2016.
So, in 2015, the game was on. For the Democrats, it was quite easy to nominate their candidate
in the primaries. Hillary Clinton was nominated to run against the Republican opponent. But for
the Republicans, the case was not the same. About 19 candidates were coming out to run.
Everyone believes he or she has the chance to win. Especially, when the outgoing president
broke the jinx of the minorities getting a chance in such national position. Whomever the
Republicans nominate will face the Democratic party nominee, a woman, who was the first
woman in history to be nominated as a presidential candidate.
The presidential election that is usually political, and boring for many, suddenly became a
national reality show. Everyone, inside and outside America became interested in the USA
presidential election. Social media, Twitter especially, has become popular as at then. Anyone
can voice out opinion about the election through Twitter. The GOP activities became the talking
point.
Hence, the sentiment analysis of public opinion became necessary and useful information to
obtain. Sentiment Analysis is the aspect of data analytics that studies public sentiment. It is a
fast-growing area in today’s analytics. Government agencies and businesses are so interested in
insights from such analysis. This special area will enhance the speed and improve the decisions
of any business, or government policy. For the government, public opinion will guide the
enacting of policies to impact and benefit the masses. Businesses are interested in the opinion of
the customer they serve to improve services and minimize loss. For this assignment, sentiment
analysis will be used to analyze the First GOP Debate (Twitter) Dataset, using the DB2 on the
IBM Cloud service – the RStudio and SQL.
Dataset Description:
The First GOP Debate (Twitter) dataset was provided for the analysis from the source -
https://data.world/socialmediadata/first-gop-debate-twitter, in a csv file. The table below shows
the attributes and description of the dataset.
A
Id
Row id
B
CANDIDATE
Candidate mentioned
C
CANDIDATE_CONFIDENCE
Confidence of the candidate mentioned
D
RELEVANT_YN
"no" means that the tweet was meant to be part of the dataset
but was not available when contributors went to judge it
E
RELEVANT_YN_CONFIDENCE
confidence in the existence/non-existence of the tweet
F
SENTIMENT
Tweet Sentiment
G
SENTIMENT_CONFIDENCE
Confidence of the sentiment
H
SUBJECT_MATTER
Tweet subject
I
SUBJECT_MATTER_CONFIDENCE
Confidence of the subject matter
J
CANDIDATE_GOLD
whether the candidate was included in the gold standard for
the model
K
NAME
the user who tweeted
L
RELEVANT_YN_GOLD
whether the tweet yn value is golden
M
RETWEET_COUNT
number of times the user has retweeted
N
SENTIMENT_GOLD
if the profile is golden, what is the sentiment
O
SUBJECT_MATTER_GOLD
whether the subject matter was included in the gold
standard for the model
P
TEXT
the text of the tweet
Q
TWEET_COORD
if the user in column K has location turned on, the
coordinates as a string with the format "[latitude, longitude]"
R
TWEET_CREATED
When tweet was created
S
TWEET_ID
Tweet identification number
T
TWEET_LOCATION
User’s country, city, state
U
USER_TIMEZONE
User time zone
Table 1: The Attributes and Description of the First GOP Debate Twitter Dataset
The .csv file was loaded successfully into the DB2, but not without editing some of the columns
to ensure that the data type lengths, number of characters, are enough to accommodate the data.
The figure below shows the database name (i.e., Target) and other information after the load was
completed. The status section on the left-hand side of the page shows that there were 13,871 rows
that was read, and all were loaded successfully. There were no errors nor warning recorded after
completion. There was zero rejection, and the start time and the end time was captured. At the top
of the page, there is a confirmation that the correct file was uploaded
Figure 1: The Load Status Page
Dataset Validation:
In addition to the load status page, SQL and/or R can be used to validate the number of rows in
the table. Also, the ‘systable’ can be queried to confirm that the right database instance is
running, and the table structure can be seen by querying the ‘syscolumns’. Figure 2 shows the
select statement and the result that confirms the total number of rows in the table. 13,871 rows
were returned, which is the with the number of rows loaded.
• The number of Rows read
• The number of Rows loaded
• The number of Rows rejected
• Start Time
• End Time
Figure 2: The Query and the Result Showing the Total Number of Rows
The information about the table was seen after the system table, ‘systable’, was quarried. The
information in the figure 3 below contains the table created and the admin username. The table
name is the ‘FIRST_GOP_DEBATE’, the administrative username is ‘SWW42202’, the time the
table was created, and the number of columns in the table were returned.
Figure 3: The Query and the Result Showing the Metadata
Dataset Manipulation - SQL:
The following query will return the number of times each candidate is mentioned. To derive this
information, the ‘CANDIDATE’ column will be selected, counted, and grouped by candidate.
The figure 4 below shows the number of times each candidate is mentioned. The output reveals
that Donald Trump is the most mentioned candidate, with total of 2,813 times, while John Kasich
was mentioned the least, with 242 times. With the margin between Donald and the rest of the
candidates, it is obvious that Donald Trump was super popular as at the time of the first GOP
debate. Being popular is of great importance and advantage to anyone running for any position, in
an election.
Figure 4: The Number of times Each Candidate is Mentioned.
In these Twitter feeds analysis, the public sentiment is of great importance. So, the next query
will return a five-column output, to show these sentiments according to the candidates. Figure 5a
and 5b show the select statement and the result, respectively.
Figure 5a: The SQL Query to Show the Sentiments Per Candidate
Figure 5b: The Result of SQL Query to Show the Sentiments Per Candidate
Dataset Manipulation - R:
Similarly in R, the programmatic approaches used can still be verified. Having loaded the data, R
commands can be used to generate the total number of rows, view the data frame structure, and
check for any missing value. The result of figure 6 below confirms that the dimension of the
table. The total number of the rows are 13,871, and the number of columns are 21.
Figure 6: The R Command Showing the Dimension of the Table.
To check if there is any missing value, the colSums command was ran, and the output shows that
there were no missing values in the table. See figure 7 below.
Figure 7: The colSums Command
It will be interesting to note that these 13,871 numbers of twit feeds were just created in 2 days.
Data frame was created on the ‘TWEET_CREATED’ and the frequency of the dates shows that
the whole tweets were created in just 2 days. The figure 8 below shows the frequency of dates.
Figure 8: The Frequency of the TWEET_CREATED – Dates
Pie Chart:
As earlier seen that Donald Trump was the most popular candidate, the pie chart below will
immediately show the margin at which he is higher than other candidates. Also, more than half
the number of the tweets do not have any candidate name.
Figure 9: The Pie Chart Showing the Number of Times Each Candidate is Mentioned
Bar Chart:
The first bar chart here shows the sentiments in the tweet feeds. The Negative sentiment stood
out among the three – Positive, Negative, and Neutral. More than half of the tweet has negative
sentiment while, positive sentiment were the least.
Figure 10: The Bar Chart Showing the Tweet Sentiment
The next chart gives additional information about the candidates. Here the sentiments were taken
into consideration. The popularity of the candidate Trump is still obvious here. One thing is
significant while, the negative sentiment tops the chart in the overall sentiment, Donald Trump
receives more of the positive sentiment than any other candidate.
Figure 11: The Sentiments Per Candidate.
Apriori:
The Apriori method is used for the Association Rule. A subset of the First GOP Debate data
frame was defined with 'CANDIDATE', 'SENTIMENT', 'SUBJECT_MATTER',
'RETWEET_COUNT', and 'USER_TIMEZONE' columns. Having done all that, the columns’
data type were changed to factor, before generating the rule, using apriori algorithm, to show the
confidence limit of 0.8 (i.e., 80%) and 0.1 (i.e., 10%) of support. In all, 1387 is the absolute
minimum support count which is 10% of the entries. Two (2) rules were generated. Figure 12a
shows the output of the Apriori rule. And figure 12b shows the gragh.
Figure 12a: The Apriori Rule
Figure 12b: The Apriori Rule Plot
WordCloud – Using the TWEET_LOCATIONS
In other to generate a WordCloud, a subset of the targeted column should be defined. Then the
single column will be converted to a vector source, after which a corpus document will be
created. This document must be free from all noise. Punctuations and stopwords must be
removed, upper letters cases must be set to lower, and the document must be stripped from all
white spaces. Only after then, that the matrix will be created and set up the frequency count.
Washington, New York, Texas, and California were seen to the frequent locations where tweets
are originating from. See figure 13. These are the key cities in the United States of America.
Figure 13: The WordCloud per Tweet Location
K-Means Cluster – Using the Positive Sentiment Tweet Text:
Like WordCloud, K-means follows the same process to produce the corpus document. After the
document were created, then k-means will be fitted and the total squared distance between
clusters and within clusters by k value will be plotted. Figure 14a and 14b the cluster and the
squared distance, respectively.
Figure 14a: The Cluster diagram
Figure 14b: The Squared Distance Between Clusters and Within Clusters
Accomplishments and Limitations:
In the process of this assignment, I was able to use this fast-growing technique – sentiment
analysis to analyze the First_GOP_Debate dataset. The insight from the analysis shows that
against all odds, negative sentiments, Donald Trump is the man of the people. I know that I
improve on the use of Cloud resources.
Beside the personal limitation I faced during the period of this assignment, I figured out that
basic knowledge of SQL and R is not enough in this type of analysis.
