python business analytics

{ "cells": [ { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "2a202824e5fcc69886b716fff925e902", "grade": false, "grade_id": "intro", "locked": true, "schema_version": 3, "solution": false } }, "source": [ "## Homework 11: Regression and Regularization\n", "\n", "#### Due Date: Saturday November 21, 2020 at 11:59PM EST\n", "\n", "We want to study models for prices in real estates markets. We will use regression to predict sale prices from features \n", "like number of bathrooms or size of garage. While we want to add many features into the model, we need to avoid overfitting. We will use Ridge regression. Since we need adjust the the amount of regularization, we will need to incorporate validation alongside training and testing to determine the extra parameters of the model. \n", "\n", "The questions guide you step-by-step through the assignment. Please post to the #homework-11 channel on Slack with any questions. \n", "\n", "#### Collaboration Policy\n", "\n", "Data analysis is a collaborative activity. While you may discuss the homework with classmates, you should answer the questions by yourself. If you discuss the assignments with other students, then please **include their names** below." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Name:** *list name here*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**NetId:** *list netid here*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Collaborators:** *list names here*" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "9ea5ef85106c429ff09bf496330b8a4c", "grade": false, "grade_id": "cell-f10559bf72db5412", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "### Rubric\n", "\n", "Question | Points\n", "--- | ---\n", "[Question 1](#q1) | 3\n", "[Question 2](#q2) | 3\n", "[Question 3](#q3) | 1\n", "[Question 4.1](#q4) | 1\n", "[Question 4.2](#q4) | 2\n", "[Question 5](#q5) | 0\n", "[Question 6.1](#q6) | 1\n", "[Question 6.2](#q6) | 0\n", "[Question 7](#q7) | 1\n", "[Question 8.1](#q8a) | 2\n", "[Question 8.2](#q8b) | 1\n", "Total | 14" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "30face9aaa807e2e0cc5c33a12b7f6b8", "grade": false, "grade_id": "cell-3030f1c9f2ae459a", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "\n", "from sklearn.linear_model import Ridge\n", "\n", "# change some settings\n", "\n", "plt.rcParams['figure.figsize'] = (9,6)\n", "pd.options.display.max_rows = 15 \n", "pd.options.display.max_columns = 10\n", "\n", "# specify path to files\n", "\n", "import os, sys\n", "\n", "path_home = os.environ[\"HOME\"] + \"/shared/homework-11\"\n", "path_training = f\"{path_home}/training.csv\"\n", "path_testing = f\"{path_home}/testing.csv\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "d3de5afaafc52543b6d38ddb80bb9917", "grade": true, "grade_id": "cell-de812e3e350107bd", "locked": true, "points": 0, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "# TEST \n", "\n", "assert 'pandas' in sys.modules and \"pd\" in locals()\n", "assert 'numpy' in sys.modules and \"np\" in locals()\n", "assert 'matplotlib' in sys.modules and \"plt\" in locals()\n", "assert 'sklearn' in sys.modules and \"Ridge\" in locals()" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "0817ced6c687bde9fc7ab040f7759f81", "grade": false, "grade_id": "cell-f68729731e7fe39d", "locked": true, "schema_version": 3, "solution": false } }, "source": [ "The [Ames dataset](http://jse.amstat.org/v19n3/decock.pdf) consists of 2928 records taken from the Assessor’s Office in Ames, Iowa. The records describe individual residential properties sold in Ames, Iowa from 2006 to 2010. \n", "\n", "The data set has 82 features in total\n", "\n", "- 46 Qualitative Variables\n", " * 23 nominal\n", " * 23 ordinal\n", "- 34 Quantitative Variables\n", " * 14 discrete \n", " * 20 continuous variables \n", "\n", "along with 2 additional identifiers. Please see `description_of_features.txt` for an explanation of each feature. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "b24093cc6c452725d7269e7888592ef6", "grade": false, "grade_id": "cell-e8fea30adc9d489b", "locked": true, "schema_version": 3, "solution": false } }, "outputs": [], "source": [ "training_data = pd.read_csv(path_training)\n", "testing_data = pd.read_csv(path_testing)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "a9c5bae40d66cd54391ae2d390ab3b27", "grade": false, "grade_id": "cell-9d6d509b6e854e10", "locked": true, "schema_version": 3, "solution": false } }, "source": [ "We split the data into\n", "\n", "- training set with 1998 records \n", "- test set with 930 records \n", "\n", "We shold verify that the data shape matches the description." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "99d200514d598fb55501da97ce9866a0", "grade": true, "grade_id": "cell-c841a2de55691502", "locked": true, "points": 0, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "# TEST \n", "\n", "# 2000 observations and 82 features in training data\n", "assert training_data.shape == (1998, 82)\n", "\n", "# 930 observations and 81 features in test data\n", "assert testing_data.shape == (930, 82)\n", "\n", "# training and testing should have the same columns\n", "assert len(training_data.columns.intersection(testing_data.columns)) == 82" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "84c9449a33fd5cc48e45e22b38126542", "grade": false, "grade_id": "cell-ce9acc2f62c96e59", "locked": true, "schema_version": 3, "solution": false } }, "source": [ " The Ames data set contains information that typical homebuyers would want to know. A more detailed description of each variable is included in `description_of_features.txt`. You should take some time to familiarize yourself with some of the features." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "f542e637735a9ff2be37b5e2f2ea35b1", "grade": false, "grade_id": "cell-4e60a7a0cda5eecf", "locked": true, "schema_version": 3, "solution": false } }, "outputs": [], "source": [ "training_data.columns.values" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "63119111a0cd0e21c36dd7431a919c25", "grade": false, "grade_id": "cell-ba0f6926b0dafefb", "locked": true, "schema_version": 3, "solution": false } }, "source": [ "### Question 1 <a name=\"q1\"></a>\n", "\n", "We will generate a couple visualizations to understand the relationship between `SalePrice` and other features. Note that we will examine the training data so that information from the testing data does not influence our modeling decisions.\n", "\n", "We generate a [raincloud plot](https://micahallen.org/2018/03/15/introducing-raincloud-plots/amp/?__twitter_impression=true) of `SalePrice`. Here we combine a histogram with density, a scatter-plot, and a box plot. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "4ae7a72607f894bd860ebf12dafd3b7f", "grade": false, "grade_id": "cell-15d483a695655cea", "locked": true, "schema_version": 3, "solution": false } }, "outputs": [], "source": [ "fig, axs = plt.subplots(nrows=2)\n", "\n", "sns.distplot(\n", " training_data['SalePrice'], \n", " ax=axs[0]\n", ")\n", "sns.stripplot(\n", " training_data['SalePrice'], \n", " jitter=0.4, \n", " size=3,\n", " ax=axs[1],\n", " alpha=0.3\n", ")\n", "sns.boxplot(\n", " training_data['SalePrice'],\n", " width=0.3, \n", " ax=axs[1],\n", " showfliers=False,\n", ")\n", "\n", "# Align axes\n", "spacer = np.max(training_data['SalePrice']) * 0.05\n", "xmin = np.min(training_data['SalePrice']) - spacer\n", "xmax = np.max(training_data['SalePrice']) + spacer\n", "axs[0].set_xlim((xmin, xmax))\n", "axs[1].set_xlim((xmin, xmax))\n", "\n", "# Remove some axis text\n", "axs[0].xaxis.set_visible(False)\n", "axs[0].yaxis.set_visible(False)\n", "axs[1].yaxis.set_visible(False)\n", "\n", "# Put the two plots together\n", "plt.subplots_adjust(hspace=0)\n", "\n", "# Adjust boxplot fill to be white\n", "axs[1].artists[0].set_facecolor('white')" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "375c05bbfe50f2bef3318ede6a7c54e2", "grade": false, "grade_id": "cell-89d2b18ca0a7115e", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Having visualized the data, we should try to summarize it with statistics like the mean. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "b4a085bd906dbca5edcdda315738e85f", "grade": false, "grade_id": "cell-45e5037c06db70f0", "locked": true, "schema_version": 3, "solution": false } }, "outputs": [], "source": [ "training_data['SalePrice'].describe()" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "a99dd4c9e68417524c7658c8a8693aad", "grade": false, "grade_id": "cell-592d5f41ebd67ee2", "locked": true, "schema_version": 3, "solution": false } }, "source": [ "To check your understanding of the graph and summary statistics above, answer the following `True` or `False` questions:\n", "\n", "1. The distribution of `SalePrice` in the training set is skewed to the left. In other words, most of the data clusters to the right.\n", "1. The mean of `SalePrice` in the training set is greater than the median.\n", "1. At least 25% of the houses in the training set sold for more than \\$200,000.00." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "nbgrader": { "cell_type": "code", "checksum": "20b6f605e76c6ba628f4420943e05cd1", "grade": false, "grade_id": "q1-answer", "locked": false, "schema_version": 3, "solution": true } }, "outputs": [], "source": [ "q1statement1 = ...\n", "q1statement2 = ...\n", "q1statement3 = ...\n", "\n", "# YOUR CODE HERE\n", "raise NotImplementedError()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "287764f31fe64d5636842b837a16a52b", "grade": true, "grade_id": "q1-tests", "locked": true, "points": 1, "schema_version": 3, "solution": false } }, "outputs": [], "source": [ "# TEST\n", "assert set([q1statement1]).issubset({False, True})\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "cf1911fef8816b8427fa37b8e1c754a1", "grade": true, "grade_id": "cell-9b79e81ae96ac518", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "# TEST\n", "\n", "assert set([q1statement2]).issubset({False, True})\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "7b40e0854fffd4d3dcfa17e2e3a27a0d", "grade": true, "grade_id": "cell-13fe5aaa378ebbf6", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "# TEST\n", "\n", "assert set([q1statement3]).issubset({False, True})\n" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "6fc869c6962d3ffa0b9c3b0f1e31b668", "grade": false, "grade_id": "cell-cea4653369b2c468", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "### Question 2 <a name=\"q2\"></a>\n", "\n", "We know that Total Bathrooms can be calculated as:\n", "\n", "$$ \\text{TotalBathrooms}=(\\text{BsmtFullBath} + \\text{FullBath}) + \\dfrac{1}{2}(\\text{BsmtHalfBath} + \\text{HalfBath})$$\n", "\n", "Write a function `add_total_bathrooms(data)` that returns a copy of `data` with an additional column called `TotalBathrooms` computed by the formula above. Please avoid loops." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "nbgrader": { "cell_type": "code", "checksum": "44d27aa775120b91a31b42b1a8262c00", "grade": false, "grade_id": "cell-31b41cddaaf9a22a", "locked": false, "schema_version": 3, "solution": true, "task": false } }, "outputs": [], "source": [ "def add_total_bathrooms(data):\n", " \"\"\"\n", " Input:\n", " data: a table containing at least four columns of numbers \n", " Bsmt_Full_Bath, Full_Bath, Bsmt_Half_Bath, and Half_Bath\n", " \n", " Output: \n", " Copy of the table with additional column TotalBathrooms \n", " \"\"\"\n", " \n", " # make a copy\n", " with_bathrooms = data.copy()\n", " \n", " \n", " # fill missing values with 0 \n", " bath_vars = ['Bsmt_Full_Bath', 'Full_Bath', 'Bsmt_Half_Bath', 'Half_Bath']\n", " with_bathrooms = with_bathrooms.fillna({var: 0 for var in bath_vars})\n", " \n", " # add the TotalBathrooms column\n", " weights = np.array([1, 1, 0.5, 0.5])\n", " \n", " # YOUR CODE HERE\n", " raise NotImplementedError()\n", " \n", " return with_bathrooms\n", "\n", "training_data = add_total_bathrooms(training_data)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "75ab4fe0729de650c42e15db86ad32b5", "grade": true, "grade_id": "cell-abe3fc20dd98fddf", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "# TEST\n", "\n", "assert not training_data['TotalBathrooms'].isnull().any() \n" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "e72bffba940ed0b12a6c6cc01f2c5224", "grade": false, "grade_id": "cell-1ee9ad2b25f73986", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "We us `sns.boxplot` to generate side-by-side boxplots showing the range of prices for different numbers of bathrooms. Take " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "8816931e824c12bdd7ba76c259e72a3b", "grade": false, "grade_id": "cell-7f6e24311beb4dc9", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "x = 'TotalBathrooms'\n", "y = 'SalePrice'\n", "data = training_data\n", "\n", "sns.boxplot(x=x, y=y, data=data, whis=5)\n", "\n", "plt.title('SalePrice distribution for each value of TotalBathrooms');" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "f0a8102bb82c4e0f14494f6346d29cc1", "grade": false, "grade_id": "cell-d954a0c14a1fdb4c", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "To check your understanding of the chart, answer the following `True` or `False` questions:\n", "\n", "1. Based on the trend in the chart, we should take houses with 5, 6, or 7 bathrooms to be outliers? \n", "1. We find a positive correlation between price and number of bathrooms for houses with 1 to 4.5 bathrooms?\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "nbgrader": { "cell_type": "code", "checksum": "bb5d1a0cd6871e588264f113d5750690", "grade": false, "grade_id": "cell-5e0f61e7ed4322cb", "locked": false, "schema_version": 3, "solution": true, "task": false } }, "outputs": [], "source": [ "q2statement1 = ...\n", "q2statement2 = ...\n", "\n", "# YOUR CODE HERE\n", "raise NotImplementedError()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "737ef5258cc43e5b6f86e823f05c17b0", "grade": true, "grade_id": "cell-542c8e6aa6d2ebc4", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "# TEST\n", "\n", "assert set([q2statement1]).issubset({False, True})\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "0e15080a6bf1cbd67622cf1b199f88e7", "grade": true, "grade_id": "cell-ea5e2771a0ca2b39", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "# TEST\n", "\n", "assert set([q2statement2]).issubset({False, True})\n" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "42056578d792ab17f6df72e614a7731c", "grade": false, "grade_id": "cell-5641434834f7cb9e", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "### Question 3 <a name=\"q3\"></a>\n", "\n", "We will create new features out of old features through transformations. We will focus on size of the house and garage.\n", "\n", "\n", "We can visualize the association between `SalePrice` and `Gr_Liv_Area`. The `description_of_features.txt` file tells us that `Gr_Liv_Area` measures \n", "\n", "> above grade (ground) living area square feet \n", "\n", "This variable represents the square footage of the house excluding anything underground. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "3b748ddb9fa07d8924eaf3575315f7f4", "grade": false, "grade_id": "cell-02a467f45464654e680", "locked": true, "schema_version": 3, "solution": false } }, "outputs": [], "source": [ "sns.jointplot(\n", " x='Gr_Liv_Area', \n", " y='SalePrice', \n", " data=training_data,\n", " stat_func=None,\n", " kind=\"reg\",\n", " ratio=4,\n", " space=0,\n", " scatter_kws={\n", " 's': 3,\n", " 'alpha': 0.25\n", " },\n", " line_kws={\n", " 'color': 'black'\n", " }\n", ");" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "46dd6e1aa93409e709313d53e3e58725", "grade": false, "grade_id": "cell-886c368c8574dab0", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Since `Gr_Liv_Area` excludes the garage space, we visualize the association between `SalePrice` and `Garage_Area`. We learn that `Gr_Liv_Area` measures \n", "\n", "> Size of garage in square feet\n", "\n", "from `description_of_features.txt`" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "e4d76c8b62e5db5f08ec336a31e1f273", "grade": false, "grade_id": "cell-02a46546546450ee680", "locked": true, "schema_version": 3, "solution": false } }, "outputs": [], "source": [ "sns.jointplot(\n", " x='Garage_Area', \n", " y='SalePrice', \n", " data=training_data,\n", " stat_func=None,\n", " kind=\"reg\",\n", " ratio=4,\n", " space=0,\n", " scatter_kws={\n", " 's': 3,\n", " 'alpha': 0.25\n", " },\n", " line_kws={\n", " 'color': 'black'\n", " }\n", ");" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "223bdcd00d135e4049810cb3c0ddf383", "grade": false, "grade_id": "cell-1851424400e94f88", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "We write a function called `add_power` that inputs\n", " - a table `data`\n", " - a column name `column_name` of the table \n", " - positive integer `degree`\n", "\n", "and outputs \n", "\n", " - a copy of `data` with an additional column called `column_name<degree>` (without the angle brackets) containing all entries of `column_name` raised to power `degree`. \n", " \n", "For instance, `add_power(training_data, \"Garage_Area\", 2)` should add a new column named `Garage_Area2`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "24a510b74986db02a707ba9f60d52a86", "grade": false, "grade_id": "cell-83c6bdc78e920b7a", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "def add_power(data, column_name, degree):\n", " \"\"\"\n", " Input:\n", " data : a table containing column called column_name\n", " column_name : a string indicating a column in the table\n", " degree: positive integer \n", "\n", " Output: \n", " copy of data containing a column called column_name<degree> with entries of column_name to power degree\n", " \"\"\"\n", " with_power = data.copy()\n", " \n", " new_column_name = column_name + str(degree)\n", " new_column_values = with_power[column_name]**(degree)\n", " \n", " with_power[new_column_name] = new_column_values\n", " \n", " return with_power\n", "\n", "training_data = add_power(training_data, \"Garage_Area\", 2)\n", "training_data = add_power(training_data, \"Gr_Liv_Area\", 2)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "c0bedfa4d0d3e222b1a5c0f1b80ac879", "grade": false, "grade_id": "cell-2b72bb0280995798", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Among `Gr_Liv_Area`, `Gr_Liv_Area2`, `Garage_Area`, `Garage_Area2` which has the largest correlation with `SalePrice`? Remember to use the function `corr` to compute correlations." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "nbgrader": { "cell_type": "code", "checksum": "e06b91266b11c937355e9767298ea3db", "grade": false, "grade_id": "cell-b65fd142c75fe753", "locked": false, "schema_version": 3, "solution": true, "task": false } }, "outputs": [], "source": [ "highest_variable = ...\n", "\n", "# YOUR CODE HERE\n", "raise NotImplementedError()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "00d5b26c7691253b1db4fd54ed0044e8", "grade": true, "grade_id": "cell-b5dec259f620cae0", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "# TEST\n", "\n", "assert highest_variable in ['Gr_Liv_Area', 'Gr_Liv_Area2', 'Garage_Area', 'Garage_Area2']\n" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "c2641c52d766f2b0878d2796271eb067", "grade": false, "grade_id": "cell-77136c9c83bb89a6", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "### Question 4 <a name=\"q4\"></a>\n", "\n", "Let's take a look at the relationship between neighborhood and sale prices of the houses in our data set." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "252b7aa438509f5e1f8e2b100d93edbb", "grade": false, "grade_id": "cell-8da235a7c4ceaa9b", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "fig, axs = plt.subplots(nrows=2)\n", "\n", "sns.boxplot(\n", " x='Neighborhood',\n", " y='SalePrice',\n", " data=training_data.sort_values('Neighborhood'),\n", " ax=axs[0]\n", ")\n", "\n", "sns.countplot(\n", " x='Neighborhood',\n", " data=training_data.sort_values('Neighborhood'),\n", " ax=axs[1]\n", ")\n", "\n", "# Draw median price\n", "axs[0].axhline(\n", " y=training_data['SalePrice'].median(), \n", " color='red',\n", " linestyle='dotted'\n", ")\n", "\n", "# Label the bars with counts\n", "for patch in axs[1].patches:\n", " x = patch.get_bbox().get_points()[:, 0]\n", " y = patch.get_bbox().get_points()[1, 1]\n", " axs[1].annotate(f'{int(y)}', (x.mean(), y), ha='center', va='bottom')\n", " \n", "# Format x-axes\n", "axs[1].set_xticklabels(axs[1].xaxis.get_majorticklabels(), rotation=90)\n", "axs[0].xaxis.set_visible(False)\n", "\n", "# Narrow the gap between the plots\n", "plt.subplots_adjust(hspace=0.01)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "a263cabe4c81e50a0f1105acc27b8bca", "grade": false, "grade_id": "cell-5175b8efe79d8990", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "We find a lot of variation in prices across neighborhoods. Moreover, the amount of data available is not uniformly distributed among neighborhoods. North Ames, for example, comprises almost 15% of the training data while Green Hill has only 2 observations in this data set.\n", "\n", "One way we can deal with the lack of data from some neighborhoods is to create a new feature that bins neighborhoods together. Let's categorize our neighborhoods in a crude way: we'll take the top 3 neighborhoods measured by median `SalePrice` and identify them as \"expensive neighborhoods\"; the other neighborhoods are not marked." ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "eb40421e61101bf1a51ee059a8acbf51", "grade": false, "grade_id": "cell-42d6f37b41e944ea", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "#### Question 4.1 <a name=\"q4b\"></a> \n", "We write a function that returns a list of the top `n` neighborhoods by `SalePrice` as measured by our choice of aggregating function. For example, in the setup above, we would want to call `find_expensive_neighborhoods(training_data, 3, np.median)` to find the top 3 neighborhoods measured by median `SalePrice`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "94eef48db81667fe69e54cfa6983fa6f", "grade": false, "grade_id": "cell-54d7e9d9b69e7b7f", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "def find_expensive_neighborhoods(data, n, summary_statistic):\n", " \"\"\"\n", " Input:\n", " data : table containing at a column Neighborhood and a column SalePrice \n", " n : integer indicating the number of neighborhood to return\n", " summary_statistic : function used for aggregating the data in each neighborhood.\n", " \n", " Output:\n", " a list of the top n richest neighborhoods as measured by the summary statistic\n", " \"\"\"\n", " \n", " neighborhoods = (training_data.groupby(\"Neighborhood\")\n", " .agg({\"SalePrice\" : summary_statistic})\n", " .sort_values(\"SalePrice\", ascending = False)\n", " .index[:n])\n", " \n", " return list(neighborhoods)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "f4c4a668e0f146dd435c0cb4a9eb63dc", "grade": false, "grade_id": "cell-72be39d2ead46f49", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "For example, if we want to find the top 5 neighborhoods in terms of average price, then we would could use `find_expensive_neighborhoods`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "91ec8809c818e1a12b9382a207a5424b", "grade": false, "grade_id": "cell-cb3cbd267a3fab6c", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "find_expensive_neighborhoods(training_data, 5, np.mean)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "59cb43fd3f65e608a8ab39cafed1273a", "grade": false, "grade_id": "cell-25c2d83e94432220", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Use `find_expensive_neighborhoods` to determine the top 3 neighborhoods in terms of median price. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "nbgrader": { "cell_type": "code", "checksum": "0efd9e8384b4efaf2b86b3dfa123f481", "grade": false, "grade_id": "cell-175246c52b088c70", "locked": false, "schema_version": 3, "solution": true, "task": false } }, "outputs": [], "source": [ "expensive_neighborhood_1 = ...\n", "expensive_neighborhood_2 = ...\n", "expensive_neighborhood_3 = ...\n", "\n", "# YOUR CODE HERE\n", "raise NotImplementedError()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "d1336107df55d155143c1c8e28230674", "grade": true, "grade_id": "cell-773252e4b8bb5da8", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "# TEST\n", "\n", "expensive_neighborhoods = [expensive_neighborhood_1, expensive_neighborhood_2, expensive_neighborhood_3]\n", "\n", "assert set(expensive_neighborhoods).issubset(training_data['Neighborhood'].unique())\n" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "6748e4fdde34be5bc665c076df74a98e", "grade": false, "grade_id": "cell-2c80ae6667de7296", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "#### Question 4.2 <a name=\"q4b\"></a> \n", "\n", "We now have a list of three expensive neighborhoods from Question 4a. We want to add a feature `in_expensive_neighborhood` to the training set. \n", "\n", "Write a function `add_expensive_neighborhood` that adds a column `in_expensive_neighborhood` to the table. The values should be 0 or 1. \n", "\n", "- if the house is in an `expensive_neighborhoods` then the value is 1\n", "- if the house is not in an `expensive_neighborhoods` then the value is 0\n", "\n", "Instead of loops, try to use the pandas function `isin`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "nbgrader": { "cell_type": "code", "checksum": "21005e0a1e8fea22537598844a79b282", "grade": false, "grade_id": "cell-4daaeaf8ccc0e97e", "locked": false, "schema_version": 3, "solution": true, "task": false } }, "outputs": [], "source": [ "def add_expensive_neighborhood(data, neighborhoods):\n", " \"\"\"\n", " Input:\n", " data : a table containing a 'Neighborhood' column \n", " neighborhoods : list of strings with names of neighborhoods\n", " Output:\n", " A copy of the table with an additional column in_expensive_neighborhood\n", " \"\"\"\n", " with_additional_column = data.copy()\n", " \n", " with_additional_column['in_expensive_neighborhood'] = ...\n", "\n", " # YOUR CODE HERE\n", " raise NotImplementedError()\n", " \n", " return with_additional_column\n" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "3488e1d3939c2e296e98b528304d643d", "grade": false, "grade_id": "cell-78c335dfeb70ded6", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Using the neighborhoods `expensive_neighborhoods` from Question 4a, we will add a column to the training set." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "39f8440046434c2b6bff47511ac4dd6c", "grade": false, "grade_id": "cell-07ea29e3bfce5d82", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "training_data = add_expensive_neighborhood(training_data, expensive_neighborhoods)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "4064789dd16494b9a4c081fbd21bf88c", "grade": true, "grade_id": "cell-5e8706298e68ab62", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "# TEST\n", "\n", "assert sum(training_data.loc[:, 'in_expensive_neighborhood']) == 191\n", "assert sum(training_data.loc[:, 'in_expensive_neighborhood'].isnull()) == 0\n" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "c6bee927435cfbc603dcb2bb1b0db970", "grade": false, "grade_id": "cell-5ffdfab3f8801658", "locked": true, "schema_version": 3, "solution": false } }, "source": [ "### Question 5 <a name=\"q5\"></a> \n", "\n", "We can use the features from Question 2, Question 3, and Question 4 to determine a model. \n", "\n", "Remember that we need to normalize features for regularization. If the features have different scales, then regularization will unduly shrink the weights for features with smaller scales.\n", "\n", "For example, if we want to normalize the features `Garage_Area` and `Gr_Liv_Area` then we could use the following approach.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "eede07142c66e09954bbcbdf9d7e0f47", "grade": false, "grade_id": "cell-61e978aad71c8255", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "Z = training_data[['Garage_Area','Gr_Liv_Area']].values\n", "\n", "Z_normalized = (Z - Z.mean(axis = 0)) / Z.std(axis = 0)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "b1b8d8ed33e77e31901e326a154025f0", "grade": false, "grade_id": "cell-889177bcc5e52c22", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Following the transformation, each column has mean 0 and standard deviation 1.\n", "\n", "We write a function called `normalize` that inputs either a 1 dimensional array or a 2 dimensional array `Z` of numbers and outputs a copy of `Z` where the columns have been transformed to have mean 0 and standard deviation 1.\n", "\n", "To avoid dividing by a small number, we can add 0.00001 to the standard deviation in the denominator. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "0283ce1dd1ed8657f9acd25f4745b64f", "grade": false, "grade_id": "cell-68b8ae82677f7d1f", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "def standardize(Z):\n", " \"\"\"\n", " Input:\n", " Z: 1 dimensional or 2 dimensional array \n", " Outuput\n", " copy of Z with columns having mean 0 and variance 1\n", " \"\"\"\n", " \n", " Z_normalized = (Z - Z.mean(axis=0)) / (Z.std(axis=0) + 0.00001)\n", " \n", " return Z_normalized " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "d1283b9e16d34fb58e62aae567122f97", "grade": true, "grade_id": "cell-95bf29f600a86031", "locked": true, "points": 0, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "# TEST \n", "\n", "Z = training_data[['Garage_Area','Gr_Liv_Area']].values\n", "\n", "assert np.all(np.isclose(standardize(Z).mean(axis = 0), [0,0]))\n", "assert np.all(np.isclose(standardize(Z).std(axis = 0), [1,1]))" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "494d127b5e6cbe67b80fdfe90d6a7c29", "grade": false, "grade_id": "cell-9ace041b1a38f4d8", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "### Question 6 <a name=\"q6\"></a>\n", "\n", "Let's split the training set into a training set and a validation set. We will use the training set to fit our model's parameters. We will use the validation set to estimate how well our model will perform on unseen data. If we used all the data to fit our model, we would not have a way to estimate model performance on unseen data." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "c7002a8e7529f68b8621274c36f2f3e6", "grade": false, "grade_id": "cell-a1fd669f1541027e", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "# Run to make a copy of the original training set\n", "\n", "training_data_copy = pd.read_csv(path_training)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "9d5923ef304080b0ee489365916a061d", "grade": false, "grade_id": "cell-45b4cc89c5f3ca95", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "#### Question 6.1\n", "\n", "We will split the data in `training_data_copy` into two tables named `training_data` and `validating_data`. \n", "\n", "First we need to shuffle the indices of the table. Note that the training set has 1998 rows. We want to generate an array containing the number 0,1,...,1997 in random order." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "2c3a0f1681d4afe451194855753da8f9", "grade": false, "grade_id": "cell-700027e321313c3c0adc57", "locked": true, "schema_version": 3, "solution": false } }, "outputs": [], "source": [ "length_of_training_data = len(training_data_copy)\n", "\n", "RANDOM_STATE = 47\n", "\n", "shuffled_indices = np.random.RandomState(seed=RANDOM_STATE).permutation(length_of_training_data)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "918f263132d16cbfffdb9aaaf44ce990", "grade": false, "grade_id": "cell-b82d90adaa3434c6", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Note that we set a seed to allow for reproducible random numbers. See Lab 12 for more information about random numbers.\n", "\n", "Second, we want to split the indices into two pieces\n", "\n", "- `train_indices` containing 80% of the shuffled indices \n", "- `validate_indices` containing 20% of the shuffled indices\n", "\n", "Here we want to leave 20% of the data for validation." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "984b4a6099dbdd9199355bbe610f1ee7", "grade": false, "grade_id": "cell-700027ec34464313c0adc57", "locked": true, "schema_version": 3, "solution": false } }, "outputs": [], "source": [ "train_indices = shuffled_indices[:int(length_of_training_data * 0.8)] \n", "validate_indices = shuffled_indices[int(length_of_training_data * 0.8):] " ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "a6ee447b54b04fcf1bd182a85e28f8a9", "grade": false, "grade_id": "cell-4911ffd22c3df571", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Third we use the indices `training_data` and `validating_data` to access the corresponding rows in `training_data_copy` to generate the training set and validation set. Try to use `iloc` to access the rows.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "nbgrader": { "cell_type": "code", "checksum": "7de498d930e4e56c4e9b2cb9fb34053a", "grade": false, "grade_id": "cell-700027ec3c687687630adc57", "locked": false, "schema_version": 3, "solution": true, "task": false } }, "outputs": [], "source": [ "training_data = ...\n", "validating_data = ...\n", "\n", "# YOUR CODE HERE\n", "raise NotImplementedError()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "a2a91a6d7e581d7c9f3d596ff9f28195", "grade": true, "grade_id": "cell-df6b91d36b95e732", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "# TEST\n", "\n", "assert training_data.shape == (1598, 82)\n", "assert validating_data.shape == (400, 82) \n", "assert len(set(train_indices).intersection(set(validate_indices))) == 0 \n" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "820c7fe30d5386ef78494a1c7b4bdb2b", "grade": false, "grade_id": "cell-acdc861fd11912e9", "locked": true, "schema_version": 3, "solution": false } }, "source": [ "#### Question 6.2\n", "\n", "We want to try a couple different models. For each model, we will have to process the data. By chaining the transformations together, we can repeatedly fit models to data. We write a function called `process_data` that combines the transformation from \n", "\n", "- Question 2 : calculate total number of bathrooms \n", "- Question 3 : add square of `Gr_Liv_Area` and `Garage_Area`\n", "- Question 4 : indicate expensive neighborhoods\n", "- Question 5: standardize the columns\n", "\n", "We use the `pandas` function pipe to chain together these transformations in order." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "126139a980eb95725bd6c3e5f94f2ee1", "grade": false, "grade_id": "cell-2fe1d82b2c19d1fa", "locked": true, "schema_version": 3, "solution": false } }, "outputs": [], "source": [ "def select_columns(data, columns):\n", " \"\"\"Select only columns passed as arguments.\"\"\"\n", " return data.loc[:, columns]\n", "\n", "def process_data(data):\n", " \"\"\"Process the data for a guided model.\"\"\"\n", " \n", " nghds = find_expensive_neighborhoods(data, n=3, summary_statistic=np.median)\n", " \n", " data = ( data.pipe(add_total_bathrooms)\n", " .pipe(add_power,'Gr_Liv_Area', 2)\n", " .pipe(add_power,'Garage_Area', 2)\n", " .pipe(add_expensive_neighborhood, nghds)\n", " .pipe(select_columns, ['SalePrice', \n", " 'Gr_Liv_Area', \n", " 'Garage_Area',\n", " 'Gr_Liv_Area2', \n", " 'Garage_Area2',\n", " 'TotalBathrooms',\n", " 'in_expensive_neighborhood']) )\n", " \n", " \n", " data.dropna(inplace = True)\n", " X = data.drop(['SalePrice'], axis = 1)\n", " X = standardize(X)\n", " y = data.loc[:, 'SalePrice']\n", " y = standardize(y)\n", " \n", " return X, y" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "572eff8d30282310edfff6224bbfb2da", "grade": false, "grade_id": "cell-ab87af3560375733", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Note that we split our data into a table of explantory variables `X` and an array of response variables `y`.\n", "\n", "We can use `process_data` to transform the training set and validation set from Question 5 along with the testing set from Question 0." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "8ea7c6ba36faaa8e76265a41f834f294", "grade": false, "grade_id": "cell-3b3e1ca24e5e95ff", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "X_train, y_train = process_data(training_data)\n", "X_validate, y_validate = process_data(validating_data)\n", "X_test, y_test = process_data(testing_data)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "52fdde5cd31b649725fd0f23cfa9ed24", "grade": false, "grade_id": "cell-41994ca254356415b31660e", "locked": true, "schema_version": 3, "solution": false } }, "source": [ "### Question 7 <a name=\"q7\"></a>\n", "\n", "We are ready to fit a model. The model we will fit can be written as follows:\n", "\n", "$$\\begin{align*}\n", "\\text{SalePrice} = \\theta_1 \\cdot \\text{Gr_Liv_Area} + \\theta_2 \\cdot \\text{Gr_Liv_Area2} \\\\ + \\theta_3 \\cdot \\text{Garage_Area} + \\theta_4 \\cdot \\text{Garage_Area2} \\\\ + \\theta_5 \\cdot \\text{is_in_expensive_neighborhood} \\\\ + \\theta_6 \\cdot \\text{TotalBathrooms}\n", "\\end{align*}\n", "$$\n", "\n", "Here `Gr_Liv_Area`, `Gr_Liv_Area2`, `Garage_Area`, and `Garage_Area2` are continuous variables and `is_in_rich_neighborhood` and `TotalBathrooms` are discrete variables. While `is_in_expensive_neighborhood` is a one-hot encoding of categories, `TotalBathrooms` can be understsood as a number.\n", "\n", "We will use a [`sklearn.linear_model.Ridge`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html) to implement Ridge Regression. Note that `alpha` is the extra parameter needed to specify the emphasis on regularization. Large values of `alpha` mean greater emphasis on regularization. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "d852cc7d254dc5719806ccc6c0c55e37", "grade": false, "grade_id": "cell-b2011b2b4d7b20ae", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "ridge_regression_model = Ridge(alpha = 1)\n", "\n", "ridge_regression_model.fit(X_train, y_train)\n", "\n", "ridge_regression_model.coef_" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "202c8340e954b67c7733e3f7030723d0", "grade": false, "grade_id": "cell-f91dc615a8ba097f", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "We want to try many different values for the extra parameter. Some values will give better models than other values. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "20a4119710757f37876e9078ef318c78", "grade": false, "grade_id": "cell-86a22da52bddaa0a", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "models = dict()\n", "alphas = np.logspace(-4,4,10)\n", "\n", "for alpha in alphas: \n", " ridge_regression_model = Ridge(alpha = alpha)\n", " models[alpha] = ridge_regression_model" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "043ef5815e7f4dfd0febebfd41ccec80", "grade": false, "grade_id": "cell-3429745411bd5f00", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "We have generated a dictionary called `models` with \n", "- key : the value of the extra parameter `alpha`\n", "- value : a model for Ridge regression with the corresponding `alpha`\n", "\n", "Fit each of the models to the training data `X_train`, `y_train`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "nbgrader": { "cell_type": "code", "checksum": "2f5375b4bd2710859139b52d28f6518f", "grade": false, "grade_id": "cell-1be99eea86f6cf57", "locked": false, "schema_version": 3, "solution": true } }, "outputs": [], "source": [ "for alpha, model in models.items():\n", " # YOUR CODE HERE\n", " raise NotImplementedError()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "63c521e16a6a4aca017d6e5a23ce2bb3", "grade": true, "grade_id": "cell-3d0539848edab3f9", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "# TEST \n", "\n", "assert all([len(model.coef_) == 6 for model in models.values()])\n" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "f7b89bfa61e581ad6db755509d2cc671", "grade": false, "grade_id": "cell-226ce9b56132c20a", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "We can plot the slopes determined from the data for each value of `alpha`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "053bd2841ae0b83122ca46d927cd7fb2", "grade": false, "grade_id": "cell-9be1964911af3b30", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "labels = ['Gr_Liv_Area', \n", " 'Garage_Area',\n", " 'Gr_Liv_Area2', \n", " 'Garage_Area2',\n", " 'TotalBathrooms',\n", " 'in_rich_neighborhood']\n", "\n", "coefs = []\n", "for alpha, model in models.items():\n", " coefs.append(model.coef_)\n", "\n", "coefs = zip(*coefs)\n", "\n", "fig, ax = plt.subplots(ncols=1, nrows=1)\n", "\n", "for coef, label in zip(coefs, labels):\n", " plt.plot(alphas, coef, label = label)\n", "\n", "ax.set_xscale('log')\n", "ax.set_xlim(ax.get_xlim()) # reverse axis\n", "plt.xlabel('alpha')\n", "plt.ylabel('weights')\n", "plt.title('Ridge Regression Weights')\n", "plt.legend();" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "5ef0c28767b3c487715ef762fbe210c9", "grade": false, "grade_id": "cell-d3e7096271477ccd", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "### Question 8\n", "\n", "Is our linear model any good at predicting house prices? Let's measure the quality of our model by calculating the Mean Square Error between our predicted house prices and the observed prices.\n", "\n", "$$\\text{MSE} = \\dfrac{\\sum_{\\text{houses in test set}}(\\text{actual price of house} - \\text{predicted price of house})^2}{\\text{# of houses in data set}}$$\n", "\n", "Here we have a function called `mse` that calculates the error." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "60d286b09ac5d3c7b0a3ca71842e6058", "grade": false, "grade_id": "cell-96600fa98a6c2e97", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "def mse(observed, predicted):\n", " \"\"\"\n", " Calculates RMSE from actual and predicted values\n", " Input:\n", " observed (1D array): vector of actual values\n", " predicted (1D array): vector of predicted/fitted values\n", " Output:\n", " a float, the root-mean square error\n", " \"\"\"\n", " return np.sqrt(np.mean((observed - predicted)**2)) " ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "b636482d8c68fddb0a220edd1e9f850d", "grade": false, "grade_id": "cell-226f076c74a6d1fd", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "#### Question 8.1 : Mean Square Error <a name=\"q8a\"></a>\n", "\n", "For each `alpha`, we use `mse` to calculate the training error and validating error. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "2f3ad654e58d4ec86059a3e257ae8140", "grade": false, "grade_id": "cell-2514b49029afbc92", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "mse_training = dict()\n", "mse_validating = dict()\n", "\n", "for alpha, model in models.items():\n", " y_predict = model.predict(X_train)\n", " mse_training[alpha] = mse(y_predict, y_train)\n", "\n", " y_predict = model.predict(X_validate)\n", " mse_validating[alpha] = mse(y_predict, y_validate)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "76d7f5a9d2f0d1ae41122b78fa0bb5c3", "grade": false, "grade_id": "cell-b8fe76b99f302e95", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "We store the calculations in dictionaries `mse_training` and `mse_validating`. Here \n", "\n", "- key : a value for the extra parameter `alpha`\n", "- value : mean square error of the corresponding model \n", "\n", "Which value of `alpha` has the smallest mean square error on the training set?" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "nbgrader": { "cell_type": "code", "checksum": "a1e4d94e6a0b533be5e38b80c8ee0e26", "grade": false, "grade_id": "cell-4e9913b57485651d", "locked": false, "schema_version": 3, "solution": true, "task": false } }, "outputs": [], "source": [ "alpha_training_min = ...\n", "\n", "# YOUR CODE HERE\n", "raise NotImplementedError()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "d6c300362cfabbc86f659c47b09ea706", "grade": true, "grade_id": "cell-d65f5a45ceb1c686", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "# TEST \n", "assert alpha_training_min in alphas\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "620f016ed449940322fc0651c4453114", "grade": false, "grade_id": "cell-501e08ee7dc85810", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Which value of `alpha` has the smallest mean square error on the validation set?" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "nbgrader": { "cell_type": "code", "checksum": "e523b069ddb199d5fb57abc01d09cb39", "grade": false, "grade_id": "cell-f88091853e32314f", "locked": false, "schema_version": 3, "solution": true, "task": false } }, "outputs": [], "source": [ "alpha_validating_min = ...\n", "\n", "# YOUR CODE HERE\n", "raise NotImplementedError()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "c2549e52679982fca178217590632271", "grade": true, "grade_id": "cell-4737faaed77a791c", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "# TEST \n", "assert alpha_validating_min in alphas\n" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "7ce67235c15cfa1d45020c1d5fdb5268", "grade": false, "grade_id": "cell-a359da232321465dda38fcdd", "locked": true, "schema_version": 3, "solution": false } }, "source": [ "#### Question 8.2 <a name=\"q8b\"></a>\n", "\n", "Using the `alpha` from Question 8a with the smallest mean square error on the validating set, predict `SalePrice` on the testing set. For the prediction, you can use the corresponding model fit to the data from Question 7." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "nbgrader": { "cell_type": "code", "checksum": "f95e9ec206f64bc2e7b37591c61a019a", "grade": false, "grade_id": "cell-b0d3da602049f797", "locked": false, "schema_version": 3, "solution": true, "task": false } }, "outputs": [], "source": [ "model = models[alpha_validating_min]\n", "\n", "y_predict = ...\n", "\n", "# YOUR CODE HERE\n", "raise NotImplementedError()" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "b7de26707f0c7e9ac5d1d13629e6d2ab", "grade": false, "grade_id": "cell-a359da2d33213214da38fcdd", "locked": true, "schema_version": 3, "solution": false } }, "source": [ "One way of understanding the appropriateness of a model is through a residual plot. Run the cell below to plot the actual sale prices against the residuals of the model for the test data." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "c47d22b7ee6644eb2df0be31a2437ed5", "grade": false, "grade_id": "cell-4d79f42d60b94fca", "locked": true, "schema_version": 3, "solution": false } }, "outputs": [], "source": [ "residuals = y_test - y_predict\n", "\n", "plt.axhline(y = 0, color = \"red\", linestyle = \"dashed\")\n", "plt.scatter(y_test, residuals, alpha=0.5);\n", "\n", "plt.xlabel('Sale Price (Test Data)')\n", "plt.ylabel('Residuals (Actual Price - Predicted Price)')\n", "plt.title(\"Residuals vs. Sale Price on Test Data\")\n", "\n", "q8b_gca = plt.gca(); " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "505932bff156083d623245fe9fe95944", "grade": true, "grade_id": "cell-af9cf949964b0549", "locked": true, "points": 1, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "# TEST \n", "\n", "assert \"q8b_gca\" in locals()\n" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "865cc13ce0e4058685ef287f776a708a", "grade": false, "grade_id": "cell-72244721136632e2", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "We want the residuals to be close to zero. Moreover, we do not want a pattern in the scatter-plot. However the most expensive homes are always more expensive than our prediction. We would have to modify the features to improve its accuracy and lower the test error." ] } ], "metadata": { "kernelspec": { "display_name": "Python [conda env:mg-gy-8413]", "language": "python", "name": "conda-env-mg-gy-8413-py" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.7" }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false } }, "nbformat": 4, "nbformat_minor": 4 }