Create a PowerPoint presentation of 10 to 15 slides

Running head: MODEL EVALUATION

Model Evaluation 2

Model Evaluation

Description of the Model Evaluation in Supply Chain Management:

Predicting the delivery of packages is a key metric in the logistics industry. If delivery times are predicted, company can achieve and make effective associate, cost allocations. This study focuses on building a predictive analytics model, specifically regression based on the recent and historical datasets. The developed model uses explanatory variables with statistical significance to result in significant resource reduction. Application of regression model can deliver value in predictive power specifically in supply chain management (SCM) industry to play strategic competitor by employing predictive analytic techniques. Predicting accurately with certainty, companies can make profits in logistics field. In this case, by predicting the delivery time, number of packages metric with some degree of certainty, then Amazon can have better planning of associates allocation and efforts can be focused on delivery resources like delivery providers, delivery schedules, delivery associates, delivery batches by segmenting the zip codes, prioritized deliveries etc. With a high degree of transactional data availability, an exhaustive list of variables which have great impact on the delivery of packages and delivery times are compiled. This study involves thorough selection of variables with high statistical significance were chosen. A predictive model selection formed scope to make a choice of regression model. (Gibbs, Y., Kanyongo, 2006).

Firstly, the delivery service requirement of supply chain is high due to the remarkable costs and disadvantages for trained resources shortage. However, the demand for product deliveries is periodic and hard to predict which bring high risks of late delivery. Secondly, due to the decrease of the buffer time and delivery methods in the supply chain and production systems, streamlining the delivery process logistics is under the pressure. As a multiple criteria decision-making problem, warehouse evaluation can have several quantitative and qualitative criteria. The relationship between these criteria and associate performance could be complex.

For predicting resources delivery time deviation, it is a warehouse evaluation task which focuses specifically on associate delivery precision performance. Delivery precision or delivery reliability refers to the ability to delivery according to schedules or promises made to the customer. The higher the delivery precision, the lower the deviation of delivery time. Supply chain companies use regression models to understand how changes in a set of independent variables affect a dependent one and can predict customer satisfaction with an increase efficiency of associates and balance the operating costs.

In the supply chain industry, distance is considered one of the main operating cost. Distance is defined as the total miles traveled to deliver a product within promised time. Basically, delivery and pick up routes are established in a shipment transportation process; enabling the user to modify the delivery route as per their convenience. and cost efficiency. Delivery Frequency is also factored in the distance because it will drive the total miles driven to deliver a product on a daily basis. Depending on the nature of business and on how the distribution routes are established, distance can play an important role on operating cost reduction and associate efficiency.

In the delivery systems, the distance is strongly related to operating cost. For example, fuel charge is commonly based on distance multiplied by a fuel rate. It is commonly a one-to-one relationship. Where the distance increases, transportation cost will rise. This means that whatever reduction is made on distribution distances will lead to reduction on transportation cost and in turn reduces the operating costs.

Linear Regression analysis is most important statistical methods and it takes us one large step further by serving three purposes like description, control and prediction. In regression model if one variable describes the variation it is due to change of one or more variables and such inference on models is described as regression analysis. A regression is a test to see if we can predict one variable's value as we know the value of another one or more variables. In case of linear regressions, which fit the data to a straight line and are really Cartesian geometry, based on the formula y=mx+b and the X and Y axis, where, the X axis is the independent variable, and the Y axis is the dependent variable. Linear regression analyzes the relationship between dependent and independent variables, which are X and Y. If we know both X and Y and want to find the best straight line through the data and in certain cases, the slope and/or intercept have a scientific meaning. In other cases, we use the linear regression line as a standard or normal curve to find values of X from Y, or values of Y from X variables to make sure how good the model fits the data. (Gibbs, Y., Kanyongo, 2006).

The methodology applied in building and evaluation of regression model:

The model evaluation is about determining if the model is good fit in predicting the outcome on new and future data. In case of future instances if there are any unknown target values, it is required to check the accuracy of the model based on current data and access the predictive accuracy on future data. In order to properly evaluate the model, we need to hold on to labeled data which was not used during training. The data used in the evaluation phase need to have the same schema structure as the data used in training phase and include actual values for the target variables used. We can split the dataset by first 70% for training and the remaining 30% for an evaluation dataset. Building and training of a model, here the goal is to select a model which results in best prediction outcomes. The selection of model parameter configurations will result the best predictive performance on the data used for evaluation, might overfit the model. Overfitting may result when a model run into memorized patterns that show up in training and evaluation datasets, which has failed to streamline the patterns in the data. It may pop-up the case where the training data may include all of the data used in the evaluation. An overfitted model does well during evaluations, but will be unsuccessful to make accurate predictions on unnoticed data. In order to avoid such selections of overfitted model, we can backup additional data to re-validate the model performance.

The regression analysis output is categorized into the regression statistics, the ANOVA to analyze the variance, the regression equation & statistics and the residuals. The regression output includes the variance measures SSE. There are a set of numbers that we consider when evaluating a regression model where R2 represents the amount of the total variability between the data points that is described by the regression model where numbers closer to 1 indicate a better fit between the regression line and the points. Standard error is the basis for constructing confidence intervals for predictions created from the model and Significance F gives a sense of how the regression model compares with the alternative model of using just the average of the Yi as a predictor of another Yi. (Gibbs, Y., Kanyongo, 2006).

Most important performance parameters are listed below for evaluating regression analysis includes:

· Correlation: Which defines the relationship between two sets of variables used to describe or predict the on-time delivery of the products. The closer the number is to positive one, the stronger the positive correlation and the closer the number to zero is weaker correlation. Zero means there is no correlation between variables.

· R2: It is proportion of variance of the dependent variable which is described by the regression model. If the model is perfect then R2 is 1 and if R2 is zero then model is a complete failed one i.e., no variance is explained.

· Accuracy: The accuracy is calculated as percentage deviation of predicted target with actual target.

· Prediction Time: Time taken for making predictions or forecasts with a regression model, where a quantity is being predicted.

· Error (RMSE, MAE, MSE, etc): The Mean Squared Error, Root Mean Squared Error, Mean Absolute Error are list of errors, e is the error between independent x and dependent y variables.

The on-time delivery time and number of packages delivered metric calculations depend upon various factors. However, the total delivery time is split into two sections, preparation time and shipping time. The approach was here to build model for order prediction time and then sub-model for prediction of shipping time so that corresponding model will depend upon the factors that are most relevant to their specific problems.

Validation Process:

The model has been validation by using few techniques like resampling method by rearranging the data samples to inspect the model performance, random split by changing the percentage of data used during training, testing & validation sets, time-based splits to make sure in prediction of delivery times during seasonal periods and k-fold cross validations which makes random shuffling of data and then splitting the data into k-groups.

Approach of Model validation:

Cross validation technique is widely used for model evaluations by training on subsets of data and then evaluating them on created data subsets. In k-fold we split the input data into k-folds and then we train on all except one of the subsets (k-1), and then evaluate the subset which is left unused during training. Performing a 6-fold cross-validation generates six models, six datasets to train the models, six datasets to evaluate the models, and 6 evaluations, one for each model. After the cross validation of the models, we can adjust the settings for the next model depending upon the model performance whether it is overfitting, underfitting, or model fit. (Richard, R., Picard, 2012).

The approach of improving model accuracy involves below steps that we took:

· Collect the data and choose a set of hyperparameters with increase in number of training examples and config changes can be done via random sampling

· In case of feature processing, adding more variables and better feature processing

· Split the training set into two data sets, one on which we trained the model and one on which we evaluated its performance. Repeated this process by varying the split and then average the resulting performances. (Astrid, Schneider, 2010).

· Model parameter tuning by considering alternate values to training parameters to learn the model and reiterate the process till the final outcome was achieved as desired.

Summary Model Validation Results

We ran different models on our dataset and we determined the approach to measure the predictive performance of each. Methods to evaluate a model’s performance includes adjusted R-2 is most often used and also the goal is to predict accurate outcome. Overall, there has been positive correlation between sorting time and delivery time. (Richard, R., Picard, 2012). The distribution of customer_care_calls, Customer_rating, Cost_of_the_Product, Prior_purchases 

look normal, because the mean and the median are close, while discount_offered and weight_in_grams are indicated skewed. The overall recommendation of the model for shipment data are as follows based on the feature importance:

· The operation team should add more manpower when there is a sale program, especially for the discount more than 10% and the parcel weight is 1 - 4 Kg.

· The parcel should not be centralized in the warehouse block F, so that the handling is not too crowded which can cause the late shipment.

· Adding more features can improve model performance, such as delivery time estimation, delivery date, customer address, and courier.

Evaluation Metrics for Training Data:

REGRESSOR MODEL

MSE = 7.79

R2 score = 0.68

MAE = 2.09

DECISION TREE MODEL

MSE = 11.91

R2 score = 0.51

MAE = 2.58

RANDOM FOREST

MSE = 6.02

R2 score = 0.75

MAE = 1.93

ADABOOST

MSE = 7.22

R2 score = 0.71

MAE = 2.12

GRADIENT BOOSTING GRADIENT BOOSTING TWEAKED WITH GRIDSEARCHCV

MSE = 5.42 MSE = 6.81

R2 score = 0.78 MSE = 0.72

MAE = 1.67 MSE = 1.53

We found outliers in discount_offered & prior_purchases with almost 30% of data. We didn't remove the outliers, but replacing with upper bound and lower bound. And we can see in the visualization above, there is no outliers detected.

ier is not found in column

Inferences:

Not used Test-train split here as number of records are very less (21 rows) So if we do test-train split then while training the data the model may miss out some pattern which may be present in the test split.

The overall accuracy for Gradient Boosting Model is highest (i.e. 78%) for training data Also the MSE & MAE is the least for the Gradient Boosting Model.

The accuracy for linear regression is the least (i.e. 68%) for training data.

Now, to decide which model will perform better with unseen data we will check with k-fold CV, we can't use train test split here for the above-mentioned reason.

From k fold validation we observed that the mean accuracy was 37% for Linear Regression model Mean accuracy was 3% for Random Forest model Mean accuracy was in negatives for all other models. Thus, in production line Linear Regressor model will be best for prediction.

So, Linear Regression is a better model for predicting delivery time based on the sorting time. (Astrid, Schneider, 2010).

Screenshots

Raw Materials