python knn data

{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Chapter 7: k-Nearest Neighbors (kNN)\n", "\n", "> (c) 2019 Galit Shmueli, Peter C. Bruce, Peter Gedeck \n", ">\n", "> Code included in\n", ">\n", "> _Data Mining for Business Analytics: Concepts, Techniques, and Applications in Python_ (First Edition) \n", "> Galit Shmueli, Peter C. Bruce, Peter Gedeck, and Nitin R. Patel. 2019.\n", "\n", "# CIS 4321\n", "# Dr. Mohammad Salehan\n", "\n", "## Import required packages" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "\n", "from pathlib import Path\n", "\n", "import pandas as pd\n", "from sklearn import preprocessing\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.metrics import accuracy_score\n", "from sklearn.neighbors import NearestNeighbors, KNeighborsClassifier\n", "import matplotlib.pylab as plt\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load data" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "mower_df = pd.read_excel('RidingMowers.xlsx', 'Data')\n", "#add a row number = row index + 1\n", "mower_df['Number'] = mower_df.index + 1\n", "mower_df.shape" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "mower_df.head(24)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data partitioning" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "trainData, validData = train_test_split(mower_df, test_size=0.4, random_state=26)\n", "print(trainData.shape, validData.shape)\n", "\n", "#Make up an artificial record named newHouseHold to resemble future cases\n", "newHousehold = pd.DataFrame([{'Income': 60, 'Lot_Size': 20}])\n", "newHousehold" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Scatter plot of training data" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots()\n", "\n", "subset = trainData.loc[trainData['Ownership']=='Owner']\n", "ax.scatter(subset.Income, subset.Lot_Size, marker='o', label='Owner', color='C1')\n", "\n", "subset = trainData.loc[trainData['Ownership']=='Nonowner']\n", "ax.scatter(subset.Income, subset.Lot_Size, marker='D', label='Nonowner', color='C0')\n", "\n", "ax.scatter(newHousehold.Income, newHousehold.Lot_Size, marker='*', label='New household', color='black', s=150)\n", "\n", "plt.xlabel('Income') # set x-axis label\n", "plt.ylabel('Lot_Size') # set y-axis label\n", "for _, row in trainData.iterrows():\n", " ax.annotate(row.Number, (row.Income + 2, row.Lot_Size))\n", " \n", "handles, labels = ax.get_legend_handles_labels()\n", "ax.set_xlim(40, 115)\n", "ax.legend(handles, labels, loc=4)\n", "\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Scatter plot of all data" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def plotDataset(ax, data, showLabel=True, **kwargs):\n", " subset = data.loc[data['Ownership']=='Owner']\n", " ax.scatter(subset.Income, subset.Lot_Size, marker='o', label='Owner' if showLabel else None, color='C1', **kwargs)\n", "\n", " subset = data.loc[data['Ownership']=='Nonowner']\n", " ax.scatter(subset.Income, subset.Lot_Size, marker='D', label='Nonowner' if showLabel else None, color='C0', **kwargs)\n", "\n", " plt.xlabel('Income') # set x-axis label\n", " plt.ylabel('Lot_Size') # set y-axis label\n", " for _, row in data.iterrows():\n", " ax.annotate(row.Number, (row.Income + 2, row.Lot_Size))\n", "\n", "fig, ax = plt.subplots()\n", "\n", "plotDataset(ax, trainData)\n", "plotDataset(ax, validData, showLabel=False, facecolors='none')\n", "\n", "ax.scatter(newHousehold.Income, newHousehold.Lot_Size, marker='*', label='New household', color='black', s=150)\n", "\n", "plt.xlabel('Income') # set x-axis label\n", "plt.ylabel('Lot_Size') # set y-axis label\n", " \n", "handles, labels = ax.get_legend_handles_labels()\n", "ax.set_xlim(40, 115)\n", "ax.legend(handles, labels, loc=4)\n", "\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Preprocessing\n", "Initialize normalized training, validation, and complete data frames. Use the training data to learn the transformation.<br>\n", "There are no categorical variables so there is no need to create dummies." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "scaler = preprocessing.StandardScaler()\n", "scaler.fit(trainData[['Income', 'Lot_Size']]) # Note the use of an array of column names\n", "\n", "# Transform the full dataset\n", "mowerNorm = pd.concat([pd.DataFrame(scaler.transform(mower_df[['Income', 'Lot_Size']]), \n", " columns=['zIncome', 'zLot_Size']),\n", " mower_df[['Ownership', 'Number']]], axis=1)\n", "\n", "#Then repartition into train and test using row indexs\n", "trainNorm = mowerNorm.iloc[trainData.index]\n", "validNorm = mowerNorm.iloc[validData.index]\n", "\n", "#Transform newHouseHold record\n", "newHouseholdNorm = pd.DataFrame(scaler.transform(newHousehold), columns=['zIncome', 'zLot_Size'])\n", "trainNorm.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## NearestNeighbors\n", "sklearn.neighbors.NearestNeighbors class can find nearest neighbors of a record. It does not conduct classification so it is an usupervised learner." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#create an instance of knn\n", "knn = NearestNeighbors(n_neighbors=3)\n", "#fit to training set independent variables\n", "knn.fit(trainNorm[['zIncome', 'zLot_Size']])\n", "#find 3 nearest neighbors for newHouseholdNorm\n", "distances, indices = knn.kneighbors(newHouseholdNorm)\n", "print(trainNorm.iloc[indices[0], :]) # indices is a list of lists, we are only interested in the first element" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## KNeighborsClassifier\n", "Initialize a data frame with two columns: `k` and `accuracy`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "train_X = trainNorm[['zIncome', 'zLot_Size']]\n", "train_y = trainNorm['Ownership']\n", "valid_X = validNorm[['zIncome', 'zLot_Size']]\n", "valid_y = validNorm['Ownership']\n", "\n", "# Train a classifier for different values of k (1-14)\n", "results = []\n", "for k in range(1, 15):\n", " knn = KNeighborsClassifier(n_neighbors=k).fit(train_X, train_y)\n", " results.append({\n", " 'k': k,\n", " #test performance on validation set\n", " 'accuracy': accuracy_score(valid_y, knn.predict(valid_X))\n", " })\n", "\n", "# Convert results to a pandas data frame\n", "results = pd.DataFrame(results)\n", "print(results)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Retrain with full dataset" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Retrain with full dataset\n", "mower_X = mowerNorm[['zIncome', 'zLot_Size']]\n", "mower_y = mowerNorm['Ownership']\n", "knn = KNeighborsClassifier(n_neighbors=4).fit(mower_X, mower_y)\n", "distances, indices = knn.kneighbors(newHouseholdNorm)\n", "#Predict ownership for newHouseholdNorm\n", "print(knn.predict(newHouseholdNorm))\n", "print('Distances',distances)\n", "print('Indices', indices)\n", "print(mowerNorm.iloc[indices[0], :])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ROC Curve" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.metrics import roc_curve, auc\n", "\n", "#Train on training set\n", "knn = KNeighborsClassifier(n_neighbors=4).fit(train_X, train_y)\n", "\n", "#valid_y is in the form (Owner, Nonowner). We have to convert it to (1, 0)\n", "valid_y_binary = valid_y.apply(lambda x: 1 if x=='Owner' else 0)\n", "\n", "#get probability of belonging to each class\n", "y_scores = knn.predict_proba(valid_X)\n", "#get false positive rate and true positive rate values \n", "fpr, tpr, threshold = roc_curve(valid_y_binary, y_scores[:, 1])\n", "\n", "roc_auc = auc(fpr, tpr) # area under the curve\n", "plt.figure()\n", "lw = 2\n", "plt.plot(fpr, tpr, color='darkorange',\n", " lw=lw, label='ROC curve (area = %0.2f)' % roc_auc)\n", "plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--') #naive rule\n", "plt.xlim([0.0, 1.0])\n", "plt.ylim([0.0, 1.05])\n", "plt.xlabel('False Positive Rate')\n", "plt.ylabel('True Positive Rate')\n", "plt.title('Receiver operating characteristic (ROC) curve')\n", "plt.legend(loc=\"lower right\")\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.5" } }, "nbformat": 4, "nbformat_minor": 2 }