Python Code

{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "### Assignment 2: Python for Analytics, Winter" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* covers lectures 4-6\n", "* due: Feb 25th by 9am.\n", "* Points will be deducted if:\n", " * Problems are not completed.\n", " * Portions of problems are not completed.\n", " * Third party modules where used when the question specified not to do so.\n", " * The problem was solved in a very inefficient manner. For instance, copying and pasting the same block of code 10 times instead of using a for loop or using a for loop when a comprehension would work." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Question 1 (15 points)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using the Iris data, sum the 4 numeric features and find out how many rows have a sum greater than 10. Do this in two ways. First using Numpy, then using Pandas.\n", "\n", "Print the shape for both the Pandas and Numpy solution." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "from sklearn.datasets import load_iris\n", "\n", "iris = load_iris()\n", "iris_df = pd.DataFrame(data= np.c_[iris['data'], iris['target']],\n", " columns= iris['feature_names'] + ['target'])\n", "iris_df['species'] = pd.Categorical.from_codes(iris.target, iris.target_names)\n", "iris_df = iris_df.drop('target', 1)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "<div>\n", "<style scoped>\n", " .dataframe tbody tr th:only-of-type {\n", " vertical-align: middle;\n", " }\n", "\n", " .dataframe tbody tr th {\n", " vertical-align: top;\n", " }\n", "\n", " .dataframe thead th {\n", " text-align: right;\n", " }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", " <thead>\n", " <tr style=\"text-align: right;\">\n", " <th></th>\n", " <th>sepal length (cm)</th>\n", " <th>sepal width (cm)</th>\n", " <th>petal length (cm)</th>\n", " <th>petal width (cm)</th>\n", " <th>species</th>\n", " </tr>\n", " </thead>\n", " <tbody>\n", " <tr>\n", " <th>0</th>\n", " <td>5.1</td>\n", " <td>3.5</td>\n", " <td>1.4</td>\n", " <td>0.2</td>\n", " <td>setosa</td>\n", " </tr>\n", " </tbody>\n", "</table>\n", "</div>" ], "text/plain": [ " sepal length (cm) sepal width (cm) petal length (cm) petal width (cm) \\\n", "0 5.1 3.5 1.4 0.2 \n", "\n", " species \n", "0 setosa " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "iris_df.head(1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Question 2 (10 points)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Consider the below two arrays. The first will be actual values and the second predicted values. Calculate the below:\n", "\n", "* MAE: Mean Absolute Error, defined as the average absolute error.\n", "* MSE: Mean Squared Error, defined as taking the difference between the two arrays, squaring the errors, summing and finding the mean.\n", "* MAPE: Mean Absolute Percentage Error, defined as the mean percentage difference between the two arrays.\n", "\n", "Solve each using one line of code, making use of Numpy array elementwise operations.\n", "\n", "Print out each metric." ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [], "source": [ "a = np.array([1,4,5,2,4,6,1])\n", "b = np.array([5,2,3,4,5,6,1])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# mape = abs(y - yhat)/y" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Question 3 (10 points)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using the describe method and loc, find the standard deviation and mean for Sepal Length and Petal Length. Create a new DataFrame that is subetted to include only rows where the Sepal Length and Petal Length are greater than one standard deviation from the mean. Find the pairwise correlations for each feature for the subsetted DataFrame and the number of rows left after subsetting. Do the same process but switch the and to an or when subsetting the DataFrame.\n", "\n", "Print the row count and correlation matrix for both the and subsetting and or subsetting." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Question 4 (15 points)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "from sklearn.datasets import load_boston" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "boston_data = load_boston()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Load Boston Housing dataset from Scikit-Learn. Put the data into a Pandas DataFrame using the data and feature_names attributes from the boston_data object. Find the IQR (interquartile range) for AGE, which is defined as the 75th quartile - the 25th quartile. Remove observations with an AGE that are not within 1.5 IQR of the median. Find the strongest correlated feature with AGE, not including itself, and plot the two features as a scatter plot. Note strongest correlated could mean positive or negative.\n", "\n", "Print the IQR, the highest correlating feature, the correlation itself and the scatter plot." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Question 5 (10 points)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For rating_df, do the folling 3 data transformations to each column:\n", "\n", "* min_max: 0-1 scale, defined as (x - min(x))/(max(x) - min(x))\n", "* mean_centered: x - mean(x)\n", "* z_score: (x - mean(x))/std(x)\n", "\n", "This means, for instance, each column for min max should be scaled to where the max is 1 and the min is 0.\n", "\n", "Hint, this should be done using 1 line, making use of broadcasting and rows and columnwise mean, min, max and standard deviation calculations.\n", "\n", "Print out the 3 scaled dataframes." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "<div>\n", "<style scoped>\n", " .dataframe tbody tr th:only-of-type {\n", " vertical-align: middle;\n", " }\n", "\n", " .dataframe tbody tr th {\n", " vertical-align: top;\n", " }\n", "\n", " .dataframe thead th {\n", " text-align: right;\n", " }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", " <thead>\n", " <tr style=\"text-align: right;\">\n", " <th></th>\n", " <th>star_wars</th>\n", " <th>harry_potter</th>\n", " <th>avengers</th>\n", " </tr>\n", " </thead>\n", " <tbody>\n", " <tr>\n", " <th>user_1</th>\n", " <td>4</td>\n", " <td>2</td>\n", " <td>5</td>\n", " </tr>\n", " <tr>\n", " <th>user_2</th>\n", " <td>1</td>\n", " <td>5</td>\n", " <td>4</td>\n", " </tr>\n", " <tr>\n", " <th>user_3</th>\n", " <td>2</td>\n", " <td>4</td>\n", " <td>2</td>\n", " </tr>\n", " <tr>\n", " <th>user_4</th>\n", " <td>3</td>\n", " <td>5</td>\n", " <td>4</td>\n", " </tr>\n", " </tbody>\n", "</table>\n", "</div>" ], "text/plain": [ " star_wars harry_potter avengers\n", "user_1 4 2 5\n", "user_2 1 5 4\n", "user_3 2 4 2\n", "user_4 3 5 4" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import numpy as np \n", "import pandas as pd \n", "\n", "user_1 = np.array([4,2,5])\n", "user_2 = np.array([1,5,4])\n", "user_3 = np.array([2,4,2])\n", "user_4 = np.array([3,5,4])\n", "\n", "rating_matrix = np.array([user_1, user_2, user_3, user_4])\n", "\n", "columns = [\"star_wars\", \"harry_potter\", \"avengers\"]\n", "index = [\"user_1\", \"user_2\", \"user_3\", \"user_4\"]\n", "\n", "rating_df = pd.DataFrame(rating_matrix, columns = columns, index = index)\n", "\n", "rating_df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Quesiton 6 (15 points)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Add a column to rating_df called \"most_similar_user\" that has the user_id of the most similar user for that given observation. Define similarity using the euclidean distance between two rating vectors. Note, when making a distance matrix, the min distance is going to be the distance between each user and themselves. Make sure the most_similar_user is not the user themself.\n", "\n", "For instance, for user_1, the most similar user, not including themself, is user_4.\n", "\n", "Print out the dataframe with the new column." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "from scipy.spatial.distance import cdist" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "<div>\n", "<style scoped>\n", " .dataframe tbody tr th:only-of-type {\n", " vertical-align: middle;\n", " }\n", "\n", " .dataframe tbody tr th {\n", " vertical-align: top;\n", " }\n", "\n", " .dataframe thead th {\n", " text-align: right;\n", " }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", " <thead>\n", " <tr style=\"text-align: right;\">\n", " <th></th>\n", " <th>star_wars</th>\n", " <th>harry_potter</th>\n", " <th>avengers</th>\n", " </tr>\n", " </thead>\n", " <tbody>\n", " <tr>\n", " <th>user_1</th>\n", " <td>4</td>\n", " <td>2</td>\n", " <td>5</td>\n", " </tr>\n", " <tr>\n", " <th>user_2</th>\n", " <td>1</td>\n", " <td>5</td>\n", " <td>4</td>\n", " </tr>\n", " <tr>\n", " <th>user_3</th>\n", " <td>2</td>\n", " <td>4</td>\n", " <td>2</td>\n", " </tr>\n", " <tr>\n", " <th>user_4</th>\n", " <td>3</td>\n", " <td>5</td>\n", " <td>4</td>\n", " </tr>\n", " </tbody>\n", "</table>\n", "</div>" ], "text/plain": [ " star_wars harry_potter avengers\n", "user_1 4 2 5\n", "user_2 1 5 4\n", "user_3 2 4 2\n", "user_4 3 5 4" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "user_1 = np.array([4,2,5])\n", "user_2 = np.array([1,5,4])\n", "user_3 = np.array([2,4,2])\n", "user_4 = np.array([3,5,4])\n", "\n", "rating_matrix = np.array([user_1, user_2, user_3, user_4])\n", "\n", "columns = [\"star_wars\", \"harry_potter\", \"avengers\"]\n", "index = [\"user_1\", \"user_2\", \"user_3\", \"user_4\"]\n", "\n", "rating_df = pd.DataFrame(rating_matrix, columns = columns, index = index)\n", "\n", "rating_df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Question 7 (10 points)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Use a for loop to make a 2,3 and 4 period rolling mean column for each user. Making sure to add each column to the dataframe.\n", "\n", "Print the dataframe out." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "<div>\n", "<style scoped>\n", " .dataframe tbody tr th:only-of-type {\n", " vertical-align: middle;\n", " }\n", "\n", " .dataframe tbody tr th {\n", " vertical-align: top;\n", " }\n", "\n", " .dataframe thead th {\n", " text-align: right;\n", " }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", " <thead>\n", " <tr style=\"text-align: right;\">\n", " <th></th>\n", " <th>id</th>\n", " <th>metric</th>\n", " </tr>\n", " </thead>\n", " <tbody>\n", " <tr>\n", " <th>0</th>\n", " <td>a</td>\n", " <td>5</td>\n", " </tr>\n", " <tr>\n", " <th>1</th>\n", " <td>a</td>\n", " <td>3</td>\n", " </tr>\n", " <tr>\n", " <th>2</th>\n", " <td>a</td>\n", " <td>2</td>\n", " </tr>\n", " <tr>\n", " <th>3</th>\n", " <td>a</td>\n", " <td>4</td>\n", " </tr>\n", " <tr>\n", " <th>4</th>\n", " <td>a</td>\n", " <td>5</td>\n", " </tr>\n", " <tr>\n", " <th>5</th>\n", " <td>b</td>\n", " <td>1</td>\n", " </tr>\n", " <tr>\n", " <th>6</th>\n", " <td>b</td>\n", " <td>4</td>\n", " </tr>\n", " <tr>\n", " <th>7</th>\n", " <td>b</td>\n", " <td>1</td>\n", " </tr>\n", " <tr>\n", " <th>8</th>\n", " <td>b</td>\n", " <td>4</td>\n", " </tr>\n", " <tr>\n", " <th>9</th>\n", " <td>b</td>\n", " <td>2</td>\n", " </tr>\n", " <tr>\n", " <th>10</th>\n", " <td>c</td>\n", " <td>5</td>\n", " </tr>\n", " <tr>\n", " <th>11</th>\n", " <td>c</td>\n", " <td>3</td>\n", " </tr>\n", " <tr>\n", " <th>12</th>\n", " <td>c</td>\n", " <td>1</td>\n", " </tr>\n", " <tr>\n", " <th>13</th>\n", " <td>c</td>\n", " <td>2</td>\n", " </tr>\n", " <tr>\n", " <th>14</th>\n", " <td>c</td>\n", " <td>3</td>\n", " </tr>\n", " </tbody>\n", "</table>\n", "</div>" ], "text/plain": [ " id metric\n", "0 a 5\n", "1 a 3\n", "2 a 2\n", "3 a 4\n", "4 a 5\n", "5 b 1\n", "6 b 4\n", "7 b 1\n", "8 b 4\n", "9 b 2\n", "10 c 5\n", "11 c 3\n", "12 c 1\n", "13 c 2\n", "14 c 3" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "metric = np.array([5,3,2,4,5,1,4,1,4,2,5,3,1,2,3])\n", "ids = np.array([\"a\",\"a\",\"a\",\"a\",\"a\",\"b\",\"b\",\"b\",\"b\",\"b\",\"c\",\"c\",\"c\",\"c\",\"c\"])\n", "\n", "df = pd.DataFrame({\n", " \"id\":ids,\n", " \"metric\":metric\n", "})\n", "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Question 8 (15 points)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Find pairwise correlations for each ids period of data, meaning for each id a,b,c, treat metric for periods 1-5 as a vector, and find the correlations. The result should be a 3 x 3 correlation matrix.\n", "\n", "Find the highest correlating users (negative or positive) using the correlation matrix. Create a dataframe with two columns, the first being an id (a,b,c) and the second column the highest correlating id.\n", "\n", "Print the 3 by 3 correlation matrix and the two column dataframe with the most similar ids." ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/html": [ "<div>\n", "<style scoped>\n", " .dataframe tbody tr th:only-of-type {\n", " vertical-align: middle;\n", " }\n", "\n", " .dataframe tbody tr th {\n", " vertical-align: top;\n", " }\n", "\n", " .dataframe thead th {\n", " text-align: right;\n", " }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", " <thead>\n", " <tr style=\"text-align: right;\">\n", " <th></th>\n", " <th>id</th>\n", " <th>metric</th>\n", " <th>period</th>\n", " </tr>\n", " </thead>\n", " <tbody>\n", " <tr>\n", " <th>0</th>\n", " <td>a</td>\n", " <td>5</td>\n", " <td>1</td>\n", " </tr>\n", " <tr>\n", " <th>1</th>\n", " <td>a</td>\n", " <td>3</td>\n", " <td>2</td>\n", " </tr>\n", " <tr>\n", " <th>2</th>\n", " <td>a</td>\n", " <td>2</td>\n", " <td>3</td>\n", " </tr>\n", " <tr>\n", " <th>3</th>\n", " <td>a</td>\n", " <td>4</td>\n", " <td>4</td>\n", " </tr>\n", " <tr>\n", " <th>4</th>\n", " <td>a</td>\n", " <td>5</td>\n", " <td>5</td>\n", " </tr>\n", " </tbody>\n", "</table>\n", "</div>" ], "text/plain": [ " id metric period\n", "0 a 5 1\n", "1 a 3 2\n", "2 a 2 3\n", "3 a 4 4\n", "4 a 5 5" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "metric = np.array([5,3,2,4,5,1,4,1,4,2,5,3,1,2,3])\n", "periods = [1,2,3,4,5] * 3\n", "ids = np.array([\"a\",\"a\",\"a\",\"a\",\"a\",\"b\",\"b\",\"b\",\"b\",\"b\",\"c\",\"c\",\"c\",\"c\",\"c\"])\n", "\n", "df = pd.DataFrame({\n", " \"id\":ids,\n", " \"metric\":metric,\n", " \"period\": periods\n", "})\n", "\n", "df.head()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 4 }