For most of the data preparation and data cleansing steps R would be
used. R has been chosen as the main instrument as it serves as a solid
framework to perform tasks like handling of missing values as well as
feature creation. In addition to R, SAS Enterprise Miner would also be
used for some data preparation activities. SAS Enterprise Miner has
been chosen as it would offer a simple and easy-to-use interface that
would support fast calculations on multiple models. Thus while
working on the storm dataset,it would be possible to handle outliers,
transform the variables, and divide the data.
While preparing the dataset, the first step that would be performed
involves the three datasets including the NWS Storm events database,
storm fatalities for 2017 and specific information on location and
individual storm episodes to R. Then the packages used in the project
would be initialized. The packages that would be used in the text
mining activity of the analysis are ‘SnowballC’, ‘tm’ and ‘wordcloud’.
After the initialization of the data, the data cleansing activities such as
handling of missing
values, correlated variables and outliers would be performed.The first
step would be handling of missing variables in the data. The storm
details dataset has approximately 57,000 and many variables have a
large missing volume of missing data. For example in property damage
and crop damage, about 10,000 cases are missing.For handling missing
data, I will eliminate the variables from the analysis that have over
20,000 missing variables. If the missing number is less than 20,000
then suitable measures would be taken depending on whether the
variables are numeric or categorical in nature. If it is categorical then
the rows with missing values would be removed. The examination of
the database has revealed that within the same 17,566 rows the
variables have missing values. Thus these rows would be removed
from the data. In the fatalities database, two variables have missing
values. In ‘victim’s age’ there is 104 missing values and in ‘gender’ of
the victim 68 values are missing. As only 775 cases exist in the
fatalities dataset, the missing value issue is relatively small.I would
replace the missing age of the victim by using the mean. As the age
gender is categorical, I would remove the rows with the missing value.
Determined which variable to keep and which to discard from each
data source as some of the variables were not necessary for the
project completion, identification of j Predictor (Input) and Target
(output) variables. Next, identify the data type and category of the
variables and finally renamed some variables for the ease of dataset
merge.
Because of the different sources of the dataset, there were a lot of
null values were identified which were treated with the method of
imputation. Also, outliers were detected and treated with
visualizations. correlation was examined between the target variable
and some features. Variable age was binned and categorical variables
were encoded
