PYTHON PROGRAMMING

The purpose of the assignment is to analyze data from somewhere in order to predict the number of something. This will be accomplished by generating simple and multivariate regression models using different variable selection techniques including, but not limited to, Forward, Stepwise, and Backward regression. From these techniques, the best model will be selected. This best model will then be further analyzed to determine if it is an adequate model to predict or if further analysis is necessary.

Make sure you follow the assignment instructions. To get points for each of these sections, you have to show them in your report. Each assignment will require a different type report. This template is fairly generic so adjust to assignment instructions.

1. Data Exploration

 Mean / Standard Deviation / Median

 Do many charts, Bar Charts, Box Plots, Scatter Plots of the data

 Is the data correlated to the target variable (or to other variables?)

 Are any of the variables missing or out of range and need to be imputed “fixed”?

 Don’t delete records that will cause test records to be deleted, fix them.

2. Data Preparation

 Fix missing values (maybe with a Mean or Median value)

 Fix outliers

 Create flags to suggest if a variable was missing

 Transform data by putting it into buckets

 Try mathematical transforms such as log or square root

 Combine variables (such as ratios or adding or multiplying) to create new variables

3. Build Models

These are instructions from Assignment 1 but will be similar in the other assignments.

Build at least two different LINEAR REGRESSION models using different variables. Show all of your models and the statistical significance of the input variables.

Discuss the coefficients in the model, do they make sense? Are you keeping the model even though it is counter intuitive? Why?

Display the Python results for your assignment and comment on the results. Your discussion of the results should be intertwined with (or linked to) the Python output, i.e. the discussion should be on or near the page containing the output. You should not be showing a lot of unnecessary Python output.

Discuss the results thoroughly. Include such discussion points as:  What is observed in the graph / table / output

 Put the results in understandable (real world terms)

 Are the results in keeping with theory?

 Do the results make sense?

 Should something different be done?

GOOD DESCRIPTION OF A DIAGRAM

The analysis continues by examining the plot of the residual values versus the predicted variables given in Figure 1. In this type of analysis, a visual inspection of the chart is conducted to determine whether or not any patterns exist in the residuals. Some patterns might include errors that increase or decrease with larger predictive variables or some other type of pattern such as a curve. In an ideal situation, the data will appear to be random. An inspection of Figure 1 suggests that the data points are randomly distributed and no obvious patterns exist in the data. Therefore, there are no immediate concerns with the distribution of the errors.

Figure 1 Housing Data Predicted vs Residual Graph

GOOD DESCRIPTION OF AN EQUATION

The model chosen from the different candidates was the XXX model because it had the highest Adjusted R-Squared value and the lowest AIC and SBC values. Using these metrics, it was far superior to the other models. The formula given for the predicted sale price is:

p_saleprice = 50000

+ 5000 * X1 LotFrontage

+ 6000 * X2 LotArea

+ 3000 * X3 OverallCond 5

The formula makes intuitive sense for the most part because sale price coefficients reflect that size and condition add to the value of a property.

However, the data should be analyzed for multi-collinearity which can result in sign changes. Also, it might be wise to remove the variable from the model if no explanation can be found.

4. Select Models

Decide on the criteria for selecting the “Best Model”. Will you use a metric such as Adjusted R-Square or AIC? Will you select a model with slightly worse performance if it makes more sense or is more parsimonious? Discuss why you selected your model. Put the results in a table to display and discuss.

5. Model Formula

Write python code that will score new data and predict the sale price. The variable with the predicted sale price should be named:

p_saleprice

6. Scored Data File

Make sure you submit as a csv file.

Use the stand alone program that you wrote in the previous section. Score the data file ames_test. Create a file that has only TWO variables for each record:

index

p_saleprice

CONCLUSION

Several models were developed to predict the sale price of a home using Ames Housing data. The best model was derived using XXXX. Although there were no problems with the model from a statistical standpoint, the winning model did have a …

CODE:

Attach as a separate file or paste your code in at the end.

BONUS

Place all bonus work at the end of the document. Clearly identify what you are doing and how many points you are trying to earn.