Need help fixing grammatical error for statistic reports
emjay16
statistics.docxIntroduction & Executive Summary Statistics is a useful science in many fields. In this paper, it used this fantastic science to make linear regression models that predicts the price of a diamond from its weight and other variables. This paper builds 3 different models and all models confirm that the 4C are the most important factor for the diamonds, which are Weight (Carat), Clarity, Colour, and Cut. Based on the 3 different models, which also show that the importance for diamond is Weight (Carat) > Clarity > Colour> Cut. These analyses and models are utilized the statistical software Minitab. In the following report, it talks about whole processes to build these three different linear regression models and how well models predict prices with test samples. The below model is the final model selected.
Brief background of the problem Diamonds, one of the most luxurious jewellery, already become a mature industry. Based on nearly 800 diamonds available sample, that selected from nearly 600,000 round-cut diamonds that are certified by the GIA (Gemmological Institute of America). This paper used around 60% sample to create linear regression model that enable to predict the price of a diamond. Meanwhile, it will also use left 40% of sample to test these models. The prices for different standards of diamonds are extremely different because of different variables. In general, when people think about diamonds, they only know Cut, Colour, Clarity, and weight. However, the other factors, such as table, depth, measurements and etc, should also be recognized. Therefore, the purpose for this paper is to use the following variables chart, Table 1, to create the linear regression models in order to predict the price of a diamond.
Log10_Price = -3.278 + 3.883 Log10_Weight (mg) + 0.4538 Log 10_Product of CCC - 1.974 Log 10_Measurement Value
Data Preparation In appendix A, it displays 466 sample data I used to find the linear regression model. For the first step, I created the unique simple ID column as the 1st column for these 466 diamonds from 1 to 466. Then, I checked whether the missing values have been corrected or not. In this process, some “0” are found in the columns of table and depth, which are missing values and not converting correctly. Therefore, I replace them with “*”, a missing value code, in order to ignore these values for model building and ensure the accuracy for analysis. The next step, I made the correct orders for the dummy variables (cut, colour, and clarity) from worst to the best. In the Appendix A, it shows the orders for these different dummy variables.
Data Transformation After I prepared data, I used scatter plot and box plots graphs to test my data in order to check data and find relationships between these variables. In the appendix B, it shows the graphs based on these prepared data, which cannot show any linear relationships. Therefore, I decide to transform the data in order to “ improve the visualization linearity and the interpretability”(1). In this case, the price and weight are two important numerical factors, thus that I am going to do the data transformation for these two variables. The transformation I used is logarithmic transformation, Log base 10, because “the data will be spread more uniformly in the graph”(1). However, after the transformation, the weight became a negative value, therefore, I decided to change the unit for the weights from Carat to Milligram (1Carat=200mg) first; and then transform them with logarithmic transformation. In the appendix C, it compares the data before and after transformation that makes the data spread more uniformly.
Model Building After prepared the data, I draw a Matrix Plot Graph with all the variables except dummy variables (Cut, Color, and Clarity). In the graph 1, it shows the strong linear associations between variables therefore a very large R^2 should be expected and linear regression model should be used.
Model-1 The purpose for model 1 is to find the relationship between different variables in order to upgrade and develop the new models (Model 2 and 3). The Fit Regression Model was used to build model 1 in the Minitab. The stepwise method with backward elimination was used because “it has a modest- sized set of potential variables and the purpose is to eliminate a few” (2). All the variables were put into the models as predictors that includes Log10 Price, Log10 Weight (mg), table, depth, x, y, z, cut, colour, and clarity. In appendix D, it shows all the regression analyses for model 1. In the analysis, it shows a high R^2 that around 98% as expected. As well, in the coefficients analysis part, it also shows the P-Value for table, depth, x, y, and z, are not significant because they are greater than 5.000%. With the coefficients of Log10 weight (mg), cut, colour, clarity are positive and highly significant because P=0.000=0.0% so<0.05%. Therefore, it provides strong support for the price has a positive effect on these individual variables if other variables are held constant. Therefore, I changed stepwise method back to none in Minitab and put Log10 weight (mg), cut, colour, and clarity as predictors, which got the following equation. However, this model was built on around 30 residual perform as outliers that will be discussed later. Next, I am going to upgrade the data and develop new models, model 2 and 3, based on the finding provided from model 1.
Log10_Price = -1.5118 + 1.9596 Log10_Weight (mg) + 0.0 cut_Good + 0.0426 cut_V.Good + 0.0756 cut_Ideal + 0.0 colour_L + 0.0198 colour_K + 0.0748 colour_J + 0.1383 colour_I + 0.2373 colour_H + 0.2617 colour_G + 0.3100 colour_F + 0.3453 colour_E + 0.3692 colour_D + 0.0 clarity_I2 + 0.2421 clarity_I1 + 0.3619 clarity_SI2 + 0.4131 clarity_SI1 + 0.4567 clarity_VS2 + 0.4955 clarity_VS1 + 0.5400 clarity_VVS2 + 0.5946 clarity_VVS1 + 0.6020 clarity_IF
Upgrade Variables Based on the previous finding that shows weight, cut, colour, and clarity have positive relationships with price. Therefore, I am going to convert the categorical predictor (Cut, Colour, Clarity) into continuous predictors and build the connection between these variables to make an easy use model. On the other hand, the other factors should also be considered again, such as table, depth, and x, y, z. Measurement is a good variable to represent x, y, z, which combines x, y, z (Measurement=x*y*z). Therefore, it is going to be converted to a new numerical variable.
Firstly, the dummy variables (Cut, Colour, and Clarity) were recoded. The core idea is to build the connection between these variables by scoring these variable’s levels from the lowest to the highest.
To be more specific, the score for each level will calculate by 100 (Full Score)/ N (Number of Levels in different categories) and will be cumulated. In the Appendix E, it shows the detail of setting.
After recoded these variables, I calculated the products for these three variables (Cut* Colour* Clarity) and change the measurements (the product of x*y*z) from the text variable to the numerical variable. The last step for upgrading variables was to take the logarithmic transformation, Log base 10, for table, depth, measurement (Numerical variable), and the products for CCC (Cut* Colour* Clarity) in order to make variables more uniformly.
Model Improvement After developed new variables, two new models, model 2 and 3, were created. For model 2 variables selection, it put the Log10 price as the response and upgraded variables as predictors, which are Log10 Weight (mg), Log Recoded Cut, Log Recoded Colour, Log Recoded Clarity, Log10 Table, Log10 Depth, and Log 10 Measurement. For model 3 variables selection, it put Log10 price as the response and other variables as predictors, which are Log10 Weight (mg), Log Product of CCC (Cut, Colour, Clarity), Log 10 Measurement Value, Log10 Table, and Log10 Depth.
For these two different selections for regression models, I used the Fit Regression Model with three different stepwise methods (Stepwise, Forward selection, and Backward Elimination) to find the better model in these two different logic assumptions for regression models. For the stepwise method with forward selection, I used 0.25 instead of 0.15 for the default entry significance level that is larger; therefore more variables could be introduced. Then, the best subsets method was used to compare the model I found. As well, I also changed the setting for the best subsets methods, which changed the number for models of each size to 3 in order to improve the accuracy.
Model-2 Selection In appendix F, it shows all the regression analysis results for different methods used for model-2. By comparing these results, they show different stepwise methods created different final models but all results show that Log10 Table, Log10 Depth, and Log 10 Measurement are not significant because the P-values are greater than 5.00%. Meanwhile, the coefficients of Log10 Weight (mg), Log Recoded Cut, Log Recoded Colour, and Log Recoded Clarity are positive and highly significant because P=0.000=0.0% so<0.05%. Therefore, it provides strong support for the price has a positive effect on these individual variables if other variables are held constant.
At the same time, by checking the result from the best subsets method at the end of appendix F, it confirms model contains Log10 Weight (mg), Log Recoded Cut, Log Recoded Colour, and Log Recoded Clarity has the lowest Cp and highest R^2. To be more precise, in the graph 2, by calculating P (number of variable plus 1), it shows the model 10 that contains Log10 Weight (mg), Log Recoded Cut, Log Recoded Colour, and Log Recoded Clarity is the best model because it has the lower P and nearest to the 45-degree line.
Model-2 Checking After analysis these results, I changed stepwise method back to none and reduced the variables as predictors that were Log10 Weight (mg), Log Recoded Cut, Log Recoded Colour, and Log Recoded Clarity. As well, I stored the fitted values and standardised residuals. In the graph 3, it shows the majority of residual plots are random and broadly satisfactory but there are around 30 outliers that stand out on the plot graph. Although these are around 30 outliers in the samples, they should be included and the details will be discussed later (Discussion for Large Residuals and Unusual X Checking).
Graph 3
Meanwhile, in the appendix G, it also shows these variables are essential because P=0.000=0.0% so<0.05%. Therefore, it provides strong support for the price has a positive effect on these individual variables if other variables are held constant. The final model 2’s regression equation is below.
Model-3 Selection In appendix H, it shows all the regression analysis results for model-3 selection processes. By comparing these results, all of these different methods show same result that Log10 Weight (mg), Log Product of CCC, Log 10 Measurement Value are highly significant because P=0.000=0.0%, so <0.05%. Therefore, it provides strong support for the price will be affected on these individual variables if other variables are held constant.
At the same time, by checking the result from the best subsets method at the end of appendix G, it confirms model-7 contains Log10 Weight (mg), Log Product of CCC, and Log 10 Measurement has the lowest Cp and highest R^2. In the graph 4, it also shows the same result, because the model 7 is not only has the lowest Cp but has the lower P and nearest to the 45-degree line.
Graph 4
Log10_Price = -2.9604 + 1.9460 Log10_Weight (mg) + 0.1729 Log 10_Recoded Cut + 0.5429 Log 10_Recoded Colour + 0.5437 Log 10_Recoded Clarity
Model-3 Checking
After analysis these results, all of them show that Log 10 Measurement Value has a negative coefficient that will negatively affect the price. In the graph 5, it shows Log 10 Measurement Value and Log10 Weight (mg) are highly correlated because of large VIF (variance inflation factors) and they have a positive relationship. Therefore, it is against the model. However, based on the P- Value, which these variables are highly significant; and if ignore one of the variable, it might lead to a big problem.
Therefore, it is better to keep these two correlated variables. The Log 10 Measurement Value has a negative coefficient because if both two correlated variables have positive coefficient, the prediction may go very wrong therefore it adjust to a negative coefficient, but it is the possible problem that might arise due this correlation.
After check models, I changed stepwise method back to none and set predictors to Log10 Weight (mg), Log Product of CCC, and Log 10 Measurement Value. And then, I stored the fitted values and standardised residuals. In the graph 6, it shows the majority of residual plots are also random and broadly satisfactory but there are around 26 outliers that stand out on the plot graph.
Graph 5
Graph 6
In appendix I, it also confirms these variables are essential because P=0.000=0.0% so<0.05%. Therefore, it provides strong support that Log 10 Weight (mg) and Log 10 Product of CCC have positive effect on price if other variables are held constant; Log 10 Measurement Value has a negative effect on price based on this model. The regression equation is shown below.
Large Residuals and Unusual X Checking
By checking the large residuals and unusual x in different models (model 1, model 2, and model 3), I recorded these 3 different models’ large residuals and unusual observations into Excel. Therefore, in appendix J, it shows that lots of fits and diagnostics for unusual observations are the same. At the same time, I also recorded the values for Fits, Standardized Residuals, Leverages and Cook’s Distance, and then I draw the matrix plot for standardized residuals and Cook’s Distance for these three different models, which show in Graph 7. The purpose for using Cook’s Distance is to find the most influential observations. At the same time, I also highlight the significant residuals that are outliers in appendix J.
Graph 7
After analysis these important residuals, the prices are in the rational range. Meanwhile, it shows that the majority of these residuals always happen in the low price range or high price range which means either have an extremely high price or extremely low price. As well, the majority of these important residuals always happen for the diamonds have high price with a high level either in Colour or Clarity. Or, the diamonds has a low price with low weight, Cut, Colour and Clarity. And then, I also used regression model to record the residuals first between Log 10 price and Log 10 weight and then
draw the boxplot graph with different levels of cut, colour, and clarity, which showed in the graph 8. 8
Log10_Price = -3.278 + 3.883 Log10_Weight (mg) + 0.4538 Log 10_Product of CCC - 1.974 Log 10_Measurement Value
As a result, it also provides the higher level of Cut, Colour, and Clarity brings the higher residuals between price and weights. As well, it also explains these residuals should be included in these data samples.
Graph 8
Therefore, the three models do not contain any residuals that should not include in these samples.
Prediction
After checking the accuracy for the data and models, I followed the same steps as Diamond A file for data preparation data transformation, and then converted to the new variables for the Diamond B files. And then, I used these three different models to do the predictions with the predict function in the Minitab. Based on the missing value for measurements in Diamond B, there are two diamonds are ignored during the analysis for Model 3. At the same time, I also stored the predicted fits that was the predict value for price. At last, I used these values to draw the scatter plot of the transformed price against the predicted. In the graph 9, it shows that Model 1 and Model 3 have better predictions than Model 2 because they are evenly scattered around the 45-degree line.
Graph 9
After the graph analysis, I transformed the log price back to the real price and calculated the mean error, root mean square error, and mean absolute error for Model 1 and Model 3, which shows in the Table 2. The result shows Model 3 has better RMSE and ME; and Model 1 has a better MAE. Although Model 3 is around 13% higher MAE than Model 1, but it is less errors on other measurement and it is easy to use. Therefore the final model I choose is Model 3. The reason is that it is not only easy to use, but also has a similar or a little bit better prediction result.
Table 2
Limitation: The range of diamond’s price is extremely large; the error will increase with higher level of predictors (Cut, Colour or Clarity); and the diamond has large errors with one extremely high or low level for predictor combine with extremely low or high weight. In order to improve the accuracy, models can be improved by adding more diamond samples in low price range and high price range. To be more specific, by adding more diamonds sample with unbalanced combination, such as low weight with one or two extremely high level predictors (Colour, Cut, and Clarity) or high weight with one or two extremely low level predictors (Colour, Cut, and Clarity). At the same time, by adding more samples, models will be upgraded with more precise coefficients for these different predictors and more accurate price will be predicted. However, these models might lead some problems by ignoring other variables, such as table, depth, x, y, and z.
Conclusion
