For most of the data preparation and data cleansing steps R would
be used. R has been chosen as the main instrument as it serves as a
solid framework to perform tasks like handling of missing values as
well as feature creation. In addition to R, SAS Enterprise Miner
would also be used for some data preparation activities. SAS
Enterprise Miner has been chosen as it would offer a simple and
easy-to-use interface that would support fast calculations on
multiple models. Thus while working on the storm dataset,it would
be possible to handle outliers, transform the variables, and divide the
data.
While preparing the dataset, the first step that would be performed
involves the three datasets including the NWS Storm events
database, storm fatalities for 2017 and specific information on
location and individual storm episodes to R. Then the packages used
in the project would be initialized. The packages that would be used
in the text mining activity of the analysis are ‘SnowballC’, ‘tm’ and
‘wordcloud’. After the initialization of the data, the data cleansing
activities such as handling of missing
values, correlated variables and outliers would be performed.The
first step would be handling of missing variables in the data. The
storm details dataset has approximately 57,000 and many variables
have a large missing volume of missing data. For example in property
damage and crop damage, about 10,000 cases are missing.For
handling missing data, I will eliminate the variables from the analysis
that have over 20,000 missing variables. If the missing number is
less than 20,000 then suitable measures would be taken depending
on whether the variables are numeric or categorical in nature. If it is
categorical then the rows with missing values would be removed.
The examination of the database has revealed that within the same
17,566 rows the variables have missing values. Thus these rows
would be removed from the data. In the fatalities database, two
variables have missing values. In ‘victim’s age’ there is 104 missing
values and in ‘gender’ of the victim 68 values are missing. As only
775 cases exist in the fatalities dataset, the missing value issue is
relatively small.I would replace the missing age of the victim by using
the mean. As the age gender is categorical, I would remove the rows
with the missing value.
Determined which variable to keep and which to discard from each
data source as some of the variables were not necessary for the
project completion, identification of Predictor (Input) and Target
(output) variables. Next, identify the data type and category of the
variables and finally renamed some variables for the ease of dataset
merge.
Because of the different sources of the dataset, there were a lot of
null values were identified which were treated with the method of
imputation. Also, outliers were detected and treated with
visualizations. correlation was examined between the target variable
and some features. Variable age was binned and categorical
variables were encoded
