stats assignment.

Learning Outcomes

LO2 fully implement data mining/machine learning projects, focused on problem analysis, data pre-processing, data post-processing by choosing and implementing appropriate algorithms;

LO4 fully implement encode and test data mining and machine learning algorithms using the programming language (such as Python) and standard packages and toolkits (such as R);

LO6 perform critical evaluation of performance metrics for data mining and machine learning algorithms for a given domain/application.

Part 1: Data Set

In this assignment, we consider a set of observations on a number of red and white wine varieties involving their chemical properties and ranking by tasters. Wine industry shows a recent growth spurt as social drinking is on the rise. The price of wine depends on a rather abstract concept of wine appreciation by wine tasters. Pricing of wine depends on such a volatile factor to some extent. Another key factor in wine certification and quality assessment is physicochemical tests which are laboratory-based and takes into account factors like acidity, pH level, presence of sugar and other chemical properties. For the wine market, it would be of interest if human quality of tasting can be related to the chemical properties of wine so that certification and quality assessment and assurance process is more controlled. 
Two datasets are available of which one dataset is on red wine and have 1599 different varieties and the other is on white wine and has 4898 varieties. All wines are produced in a particular area of Portugal. Data are collected on 12 different properties of the wines one of which is Quality (i.e. the last column), based on sensory data, and the rest are on chemical properties of the wines including density, acidity, alcohol content etc. All chemical properties of wines are continuous variables. Quality is an ordinal variable with possible ranking from 1 (worst) to 10 (best). Each variety of wine is tasted by three independent tasters and the final rank assigned is the median rank given by the tasters.



Description of attributes:


1. fixed acidity: most acids involved with wine or fixed or non-volatile (do not evaporate readily)
2. volatile acidity: the amount of acetic acid in wine, which at too high of levels can lead to an unpleasant, vinegar taste
3. citric acid: found in small quantities, citric acid can add ‘freshness’ and flavour to wines
4. residual sugar: the amount of sugar remaining after fermentation stops, it’s rare to find wines with less than 1 gram/litre and wines with greater than 45 grams/litre are considered sweet
5. chlorides: the amount of salt in the wine
6. free sulfur dioxide: the free form of SO2 exists in equilibrium between molecular SO2 (as a dissolved gas) and bisulfite ion; it prevents microbial growth and the oxidation of wine
7. total sulfur dioxide: amount of free and bound forms of S02; in low concentrations, SO2 is mostly undetectable in wine, but at free SO2 concentrations over 50 ppm, SO2 becomes evident in the nose and taste of wine
8. density: the density of water is close to that of water depending on the percent alcohol and sugar content
9. pH: describes how acidic or basic a wine is on a scale from 0 (very acidic) to 14 (very basic); most wines are between 3-4 on the pH scale
10. sulphates: a wine additive which can contribute to sulfur dioxide gas (S02) levels, wich acts as an antimicrobial and antioxidant
11. alcohol: the percent alcohol content of the wine
12. Output variable (based on sensory data): quality (score between 0 and 10)
In this assignment you need to use the first 11 attributes to your calculations. 

Part 2: Data Clustering

1st Objective:
Merge the two excel sheets (red and white) into one. Create a code in R Studio to conduct the k-means clustering analysis of this combined sheet and try to distinguish red and white wines. Obviously, the number of clusters in this objective is equal to 2. Plot a table with the results and produce in your report a confusion matrix in order to provide information regarding classification accuracy and sensitivity (Hint: you know which sample belongs to each category). Before conducting the k-means, please investigate if you need to add in your code any pre-processing task (justify your answer).
[5 Marks]

2nd Objective (only white wines):

You need to conduct the k-means clustering analysis of the white wine sheet. Find the ideal number of clusters (please justify your answer). Choose the best two possible numbers of clusters and perform the k-means algorithm for both candidates. Validate which clustering test is more accurate. For the winning test, get the mean of the each attribute of each group. Before conducting the k-means, please investigate if you need to add in your code any pre-processing task (justify your answer). Write a code in R Studio to address all the above issues. In your report, check the consistency of those produced clusters, with information obtained from column 12.
[10 Marks]


3rd Objective (only white wines):


You need to conduct the hierarchical clustering (agglomerative) clustering analysis of the white wine sheet. Investigate the hclust() function for single, complete, average methods. Create the visualization of all methods using a dendrogram. Look at the cophenetic correlation between each clustering result using cor.dendlist. Discuss the produced results after using the coorplot function. Write a code in R Studio to address all the above issues.
[10 Marks]






Part 3: Data Classification


1st Objective:
Review the predictor variables and comment whether they are sensible or not? Explore the integrated wine dataset and develop your own strategy in analysing the data. 						          [5 Marks]


2nd Objective:


Fit a classification decision tree model to classify a wines according to their quality. Experiment with different stopping criteria to enhance the performance of the model. State briefly the logical steps taken to arrive at such a model, and report its performance. Start with building a decision tree for red and white wines with package ‘party’ using the built tree in R studio for classification, followed by another way to build decision trees with package ‘rpart’. After that present a decision tree on training a random forest model with package ‘randomForest’ in R studio.
[10 Marks]

3rd Objective:


Explain how overfitting can be avoided in the context of decision tree induction          		            [5 Marks]


4th Objective:


It is claimed that white wine is affected by physicochemistry attributes (alcohol, density, free sulfur dioxide, chlorides, citric acid and volatile acidity). Meanwhile red wine quality is highly correlated to alcohol, sulphates, total sulfur dioxide and volatile acidity. Justify the correctness of the above claim.          		            	            	             [5 Marks]



  Total [50 Marks]