{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure you fill in any place that says `YOUR CODE HERE` or \"YOUR ANSWER HERE\", as well as your name below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "NAME = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "caa430dda06f98c4f063377f034dbff4",
     "grade": false,
     "grade_id": "cell-e0f9b2de18190d9d",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "# Homework 6: Predicting Housing Prices\n",
    "\n",
    "### Collaboration Policy\n",
    "\n",
    "Data science is a collaborative activity. While you may talk with others about the homework, we ask that you **write your solutions individually**. If you do discuss the assignments with others please **include their names** at the top of your solution.\n",
    "\n",
    "## Due Date: 11:59pm Tuesday, April 10\n",
    "\n",
    "## Introduction\n",
    "In this homework, we will go through the iterative process of specifying, fitting, and refining a model.  \n",
    "\n",
    "In the first portion of the assignment, we will guide you through some basic EDA, laying out the thought process that leads to certain modeling decisions.  We will then specify and fit two linear models, providing an example of the type of code we expect from you in the open-response.\n",
    "\n",
    "The second part of the assignment is purposefully left open-ended.  You will be allowed to build a linear model of your choice to compete against your peers in an in-class Kaggle competition.  **DO NOT PUT THIS ASSIGNMENT OFF TO THE LAST MOMENT AS THERE IS A CAP ON DAILY KAGGLE SUBMISSIONS**\n",
    "\n",
    "After this homework, you should feel comfortable with the following:\n",
    "\n",
    "1. Working with a messy data set that requires a moderate amount of cleaning and wrangling\n",
    "1. Using sklearn to build models\n",
    "1. Using several different transformations on your data\n",
    "1. Building a data pipeline using pandas\n",
    "1. Using cross-validation for model selection\n",
    "\n",
    "### You may find some standoffish empty cells with only a comment in them in this assignment.  They are there on purpose; please do not attempt to modify them.\n",
    "\n",
    "## Score breakdown\n",
    "\n",
    "Question | Points\n",
    "--- | ---\n",
    "[Question 1a](#q1a) | 5\n",
    "[Question 1b](#q1b) | 5\n",
    "[Question 2](#q2) | 6\n",
    "[Question 3](#q3) | 6\n",
    "[Question 4a](#q4a) | 3\n",
    "[Question 4b](#q4b) | 3\n",
    "[Question 5](#q5) | 6\n",
    "[Question 6](#q6) | 6\n",
    "[Question 7](#q7) | 6\n",
    "[Question 8](#q8) | 6\n",
    "[Question 9](#q9) | 6\n",
    "[Question 10a](#q10a) | 3\n",
    "[Question 10b](#q10b) | 3\n",
    "[Question 11](#q11) | 6\n",
    "[Question 12](#q12) | 25\n",
    "[Question 13](#q13) | 5\n",
    "Total | 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework 6: Predicting Housing Prices\n",
    "### Collaboration Policy\n",
    "Data science is a collaborative activity. While you may talk with others about the homework, we ask that you **write your solutions individually**. If you do discuss the assignments with others please **include their names** at the top of your solution.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "caa430dda06f98c4f063377f034dbff4",
     "grade": false,
     "grade_id": "cell-e0f9b2de18190d9d",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "# Homework 6: Predicting Housing Prices\n",
    "\n",
    "### Collaboration Policy\n",
    "\n",
    "Data science is a collaborative activity. While you may talk with others about the homework, we ask that you **write your solutions individually**. If you do discuss the assignments with others please **include their names** at the top of your solution.\n",
    "\n",
    "## Due Date: 11:59pm Tuesday, April 10\n",
    "\n",
    "## Introduction\n",
    "In this homework, we will go through the iterative process of specifying, fitting, and refining a model.  \n",
    "\n",
    "In the first portion of the assignment, we will guide you through some basic EDA, laying out the thought process that leads to certain modeling decisions.  We will then specify and fit two linear models, providing an example of the type of code we expect from you in the open-response.\n",
    "\n",
    "The second part of the assignment is purposefully left open-ended.  You will be allowed to build a linear model of your choice to compete against your peers in an in-class Kaggle competition.  **DO NOT PUT THIS ASSIGNMENT OFF TO THE LAST MOMENT AS THERE IS A CAP ON DAILY KAGGLE SUBMISSIONS**\n",
    "\n",
    "After this homework, you should feel comfortable with the following:\n",
    "\n",
    "1. Working with a messy data set that requires a moderate amount of cleaning and wrangling\n",
    "1. Using sklearn to build models\n",
    "1. Using several different transformations on your data\n",
    "1. Building a data pipeline using pandas\n",
    "1. Using cross-validation for model selection\n",
    "\n",
    "### You may find some standoffish empty cells with only a comment in them in this assignment.  They are there on purpose; please do not attempt to modify them.\n",
    "\n",
    "## Score breakdown\n",
    "\n",
    "Question | Points\n",
    "--- | ---\n",
    "[Question 1a](#q1a) | 5\n",
    "[Question 1b](#q1b) | 5\n",
    "[Question 2](#q2) | 6\n",
    "[Question 3](#q3) | 6\n",
    "[Question 4a](#q4a) | 3\n",
    "[Question 4b](#q4b) | 3\n",
    "[Question 5](#q5) | 6\n",
    "[Question 6](#q6) | 6\n",
    "[Question 7](#q7) | 6\n",
    "[Question 8](#q8) | 6\n",
    "[Question 9](#q9) | 6\n",
    "[Question 10a](#q10a) | 3\n",
    "[Question 10b](#q10b) | 3\n",
    "[Question 11](#q11) | 6\n",
    "[Question 12](#q12) | 25\n",
    "[Question 13](#q13) | 5\n",
    "Total | 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "856da775ee07c6cbc82a9cec213da27d",
     "grade": false,
     "grade_id": "cell-03936654c721e35b",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "## Short Proofs\n",
    "\n",
    "#### Question 1 \n",
    "We have data $(x_1, y_1) ... (x_n, y_n)$ drawn independently from the same distribution $P(x, y)$.  Our goal is to use a linear model to describe the relationship between $x$ and $y$.\n",
    "\n",
    "$$ f_\\theta(x) = \\theta_1 + \\theta_2 x $$\n",
    "\n",
    "We fit $\\hat{\\theta}_1$ and $\\hat{\\theta}_2$ by minimizing average $L_2$ loss $$ \\hat\\theta = \\text{argmin}_{\\theta_1, \\theta_2}\\frac{1}{n}\\sum_{i=1}^n (y_i - f_\\theta(x_i))^2$$.\n",
    "\n",
    "##### Part a <a name=\"q1a\"></a>\n",
    "The fitted/predicted y-values are given by:\n",
    "\n",
    "$$ \\hat{y}_i = f_\\hat\\theta(x_i) = \\hat\\theta_1 + \\hat\\theta_2 x_i $$\n",
    "\n",
    "The **residuals** are defined as:\n",
    "\n",
    "$$ e_i = \\text{observed } y_i - \\text{predicted } y_i = y_i - \\hat y_i $$\n",
    "\n",
    "Show that the sum of the residuals is equal to 0.  In other words, prove that\n",
    "\n",
    "$$\\sum_{i=1}^n e_i = 0$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "758b9812603684b5845444c273510180",
     "grade": true,
     "grade_id": "cell-d2d817d54d0a3432",
     "locked": false,
     "points": 5,
     "schema_version": 2,
     "solution": true
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "5e9cdf7119ec4315781c48b846929f62",
     "grade": false,
     "grade_id": "cell-272af0dc6e360a42",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "#### Part b <a name=\"q1b\"></a>\n",
    "\n",
    "You can **center** a variable by subtracting the global mean of the variable from each observation:\n",
    "\n",
    "In other words, for each $x_i$, you subtract off $\\bar{x} = \\frac{1}{n}\\sum_{j=1}^n x_j$:\n",
    "\n",
    "$$ x^{centered}_i = x_i - \\bar{x} $$\n",
    "\n",
    "Let $y_1, ..., y_n$ and $x_1, ..., x_n$ be centered training data.  Again, we want to fit a linear model.\n",
    "\n",
    "$$ {y}_i = f_\\theta(x_i) = \\theta_1 + \\theta_2 x_i $$\n",
    "\n",
    "Show that $\\hat\\theta_1 = 0$ minimizes average $L_2$ loss on the training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "8353e81c86c397a91b55e62c6bffda62",
     "grade": true,
     "grade_id": "cell-20f31ba24d5bb4a3",
     "locked": false,
     "points": 5,
     "schema_version": 2,
     "solution": true
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "0096907d76711d88037433d3b241fc65",
     "grade": false,
     "grade_id": "cell-62cfd21463535cac",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "import pandas as pd\n",
    "from pandas.api.types import CategoricalDtype\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from sklearn import linear_model as lm\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.model_selection import KFold\n",
    "\n",
    "# Plot settings\n",
    "plt.rcParams['figure.figsize'] = (12, 9)\n",
    "plt.rcParams['font.size'] = 12"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "24790d95c59f0f3cc4e49438bf7a2ad2",
     "grade": false,
     "grade_id": "cell-f68729731e7fe39d",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "# The Data\n",
    "\n",
    "The Ames data set consists of 2930 records taken from the Ames Assessor’s Office.  The data set has 23 nominal, 23 ordinal, 14 discrete, and 20 continuous variables (and 2 additional observation identifiers) --- 82 features in total.  An explanation of each variable can be found in the included `codebook.txt` file.  The information was used in computing assessed values for individual residential properties sold in Ames, Iowa from 2006 to 2010.  **Some noise has been added to the actual sale price, so prices will not match official records.**\n",
    "\n",
    "The data are split into training and test sets with 2000 and 930 observations, respectively.  The actual sale price is withheld from you in the test set.  In addition, the test data are further split into public and private test sets.  When you upload a test set prediction onto Kaggle for validation, the score you receive will be calculated using the public test set.  The private test set will be used in a final evaluation of this homework assignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "1b79779bf31b1461baf5704b731c0672",
     "grade": false,
     "grade_id": "cell-e8fea30adc9d489b",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "training_data = pd.read_csv(\"ames_train.csv\")\n",
    "test_data = pd.read_csv(\"ames_test.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "528d80c0381c45f44c1734ba04056d63",
     "grade": false,
     "grade_id": "cell-9d6d509b6e854e10",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "As a good sanity check, we should at least verify that the data shape matches the description."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e245b08e8687f865bfcc986e06adda5a",
     "grade": false,
     "grade_id": "cell-c841a2de55691502",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# 2000 observations and 82 features in training data\n",
    "assert training_data.shape == (2000, 82)\n",
    "# 930 observations and 81 features in test data\n",
    "assert test_data.shape == (930, 81)\n",
    "# SalePrice is hidden in the test data\n",
    "assert 'SalePrice' not in test_data.columns.values\n",
    "# Every other column in the test data should be in the training data\n",
    "assert len(np.intersect1d(test_data.columns.values, \n",
    "                          training_data.columns.values)) == 81"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "e7e804b381f5c3ba5ea29b2df70bc92c",
     "grade": false,
     "grade_id": "cell-ce9acc2f62c96e59",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "The next order of business is getting a feel for the variables in our data.  The Ames data set contains information that typical homebuyers would want to know.  A more detailed description of each variable is included in `codebook.txt`.  **You should take some time to familiarize yourself with the codebook before moving forward.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f542e637735a9ff2be37b5e2f2ea35b1",
     "grade": false,
     "grade_id": "cell-4e60a7a0cda5eecf",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "training_data.columns.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "4afa74663e97e31e7faa39253d03d0c3",
     "grade": false,
     "grade_id": "cell-ba0f6926b0dafefb",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "# Guided Modeling\n",
    "\n",
    "In the first portion of the assignment, we will take you step-by-step through one cycle of the modeling process.  Along the way, we will provide commentary to give you a sense of the thought process that goes into building a model.\n",
    "\n",
    "## EDA\n",
    "Naturally, the first thing we want to do is get a feel for our data.  In this section, we will make a series of exploratory visualizations.  The plots we ask you to reproduce here are far from exhaustive.  **When you build your own model in the second part of this assignment, you will want to delve deeper into the data.**\n",
    "\n",
    "Note that we will perform EDA on the **training data** so that information from the test data does not influence our modeling decisions.\n",
    "\n",
    "### Sale Price\n",
    "We begin by examining a [raincloud plot](https://micahallen.org/2018/03/15/introducing-raincloud-plots/amp/?__twitter_impression=true) (yet another name for a combination of a KDE, a boxplot, and a boxplot all-in-one) of our target variable `SalePrice`.  At the same time, we also take a look at some descriptive statistics of this variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "4ae7a72607f894bd860ebf12dafd3b7f",
     "grade": false,
     "grade_id": "cell-15d483a695655cea",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "fig, axs = plt.subplots(nrows=2)\n",
    "\n",
    "sns.distplot(\n",
    "    training_data['SalePrice'], \n",
    "    ax=axs[0]\n",
    ")\n",
    "sns.stripplot(\n",
    "    training_data['SalePrice'], \n",
    "    jitter=0.4, \n",
    "    size=3,\n",
    "    ax=axs[1],\n",
    "    alpha=0.3\n",
    ")\n",
    "sns.boxplot(\n",
    "    training_data['SalePrice'],\n",
    "    width=0.3, \n",
    "    ax=axs[1],\n",
    "    showfliers=False,\n",
    ")\n",
    "\n",
    "# Align axes\n",
    "spacer = np.max(training_data['SalePrice']) * 0.05\n",
    "xmin = np.min(training_data['SalePrice']) - spacer\n",
    "xmax = np.max(training_data['SalePrice']) + spacer\n",
    "axs[0].set_xlim((xmin, xmax))\n",
    "axs[1].set_xlim((xmin, xmax))\n",
    "\n",
    "# Remove some axis text\n",
    "axs[0].xaxis.set_visible(False)\n",
    "axs[0].yaxis.set_visible(False)\n",
    "axs[1].yaxis.set_visible(False)\n",
    "\n",
    "# Put the two plots together\n",
    "plt.subplots_adjust(hspace=0)\n",
    "\n",
    "# Adjust boxplot fill to be white\n",
    "axs[1].artists[0].set_facecolor('white')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b4a085bd906dbca5edcdda315738e85f",
     "grade": false,
     "grade_id": "cell-45e5037c06db70f0",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "training_data['SalePrice'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "f4350a05fd5155b27a777a5e7764be91",
     "grade": false,
     "grade_id": "cell-592d5f41ebd67ee2",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "#### Question 2 <a name=\"q2\"></a>\n",
    "To check your understanding of the graph and summary statistics above, answer the following `True` or `False` questions:\n",
    "\n",
    "1. The distribution of `SalePrice` in the training set is left-skew.\n",
    "1. The mean of `SalePrice` in the training set is greater than the median.\n",
    "1. 75% of the houses in the training set sold for less than \\$213,125.00."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "d1adce782bbff16e05c935faf1a08681",
     "grade": false,
     "grade_id": "q1-answer",
     "locked": false,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# These should be True or False\n",
    "q1statement1 = ... \n",
    "q1statement2 = ...\n",
    "q1statement3 = ...\n",
    "\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "dc33cd6a58bd639f5b7a4b92a018fd74",
     "grade": true,
     "grade_id": "q1-tests",
     "locked": true,
     "points": 2,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Two of these statements are True.\n",
    "assert sum([q1statement1, q1statement2, q1statement3]) == 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "94b869c07a4c9b149c44dc1984761749",
     "grade": true,
     "grade_id": "cell-843b895e8c775560",
     "locked": true,
     "points": 4,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Yes, this is a cell with just a comment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "4aceb93fe5b43aa55ebc86fa2ff548b1",
     "grade": false,
     "grade_id": "cell-9e22aac9b45f88e3",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### SalePrice vs Gr_Liv_Area\n",
    "\n",
    "Next, we examine `SalePrice` vs `Gr_Liv_Area`.  Now it's not entirely obvious what `Gr_Liv_Area` should be, so we'll need to consult `codebook.txt`.  We find:\n",
    "\n",
    "```\n",
    "Gr Liv Area (Continuous): Above grade (ground) living area square feet\n",
    "```\n",
    "\n",
    "Ok, so this variable represents the square footage of the house excluding anything built underground.  Some additional research (into real estate conventions) reveals that this value also excludes the garage space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "dc280af9e494f5dba2ffcb7e89efb3e0",
     "grade": false,
     "grade_id": "cell-02a467f8950ee680",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "sns.jointplot(\n",
    "    x='Gr_Liv_Area', \n",
    "    y='SalePrice', \n",
    "    data=training_data,\n",
    "    stat_func=None,\n",
    "    kind=\"reg\",\n",
    "    ratio=4,\n",
    "    space=0,\n",
    "    scatter_kws={\n",
    "        's': 3,\n",
    "        'alpha': 0.25\n",
    "    },\n",
    "    line_kws={\n",
    "        'color': 'black'\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "66698bd61b5caefa52eaf789cad52b3a",
     "grade": false,
     "grade_id": "cell-e69fbfdd6101f836",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "We notice that there is a plausible linear relationship between house size and sale price, but the spread is wider at larger sale prices and above grade living areas.  There seem to be two particularly suspicious houses above 5000 square feet.\n",
    "\n",
    "#### Question 3 <a name=\"q3\"></a>\n",
    "What are the Parcel Indentification Numbers for the two houses with `Gr_Liv_Area` greater than 5000 sqft?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "bfef71ce6fe36475adfc42ab0e0f6c77",
     "grade": false,
     "grade_id": "cell-eb0c9f329767dfc2",
     "locked": false,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# Hint: You can answer this question purely through pandas\n",
    "# q2house1 and q2house2 should be integers\n",
    "q2house1 = ...\n",
    "q2house2 = ...\n",
    "\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "538ea55f321a24ba709592bc46f96a4d",
     "grade": true,
     "grade_id": "cell-c54dffdb6bbe776f",
     "locked": true,
     "points": 2,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert isinstance(q2house1, int)\n",
    "assert isinstance(q2house2, int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "a2511a214befac6ba7f5de3d65d5a0f4",
     "grade": true,
     "grade_id": "cell-b28c0d0507c81282",
     "locked": true,
     "points": 4,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Yes, this is a cell with just a comment. Watcha gonna do?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "cc2a1dc8afa94ee0f0ded6cd7783b70d",
     "grade": false,
     "grade_id": "cell-bf7fe5dcd37df6f9",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "#### Question 4a <a name=\"q4a\"></a>\n",
    "\n",
    "The codebook actually tells us how to manually inspect the houses using an online database called Beacon. These two houses are true outliers in this data set.  They were partial sales that were priced much under market value.  We will remove them in the guided model, but perhaps you will want to address them differently in your own model.  To make sure you know how to view the online database, please answer the following question:\n",
    "\n",
    "What are the gross values of the two houses with `Gr_Liv_Area` greater than 5000 as of 2017?  You will find this under the section titled \"Valuation (Ames)\".  Your answers should be integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "cdff910b5425d07f2878097e04d115e1",
     "grade": false,
     "grade_id": "cell-469a522df10e9e14",
     "locked": false,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "q3house1 = ...\n",
    "q3house2 = ...\n",
    "\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "13a4789c8b403fc274351ee1084c004e",
     "grade": true,
     "grade_id": "cell-ae81e3401465bed2",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Make sure your answer is integer-valued\n",
    "assert isinstance(q3house1, int)\n",
    "assert isinstance(q2house2, int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "bf8ac8def4ffbf67205e16828a9d3e89",
     "grade": true,
     "grade_id": "cell-bd546f3b116e7935",
     "locked": true,
     "points": 2,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Go away."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "512135c2fff1c1ce6fc1bbdadba03571",
     "grade": false,
     "grade_id": "cell-e6273150cc398987",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "#### Question 4b <a name=\"q4b\"></a>\n",
    "\n",
    "Write a function `remove_outliers` that removes outliers from a data set based off a threshold value of a variable.  For example, `remove_outliers(training_data, 'Gr_Liv_Area', upper=5000)` should return a data frame with only observations that satisfy `Gr_Liv_Area` less than or equal to 5000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "598e4b13a8977017cbd67901444dcd83",
     "grade": false,
     "grade_id": "cell-9186ec2ca053d0aa",
     "locked": false,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def remove_outliers(data, variable, lower=-np.inf, upper=np.inf):\n",
    "    \"\"\"\n",
    "    Input:\n",
    "      data (data frame): the table to be filtered\n",
    "      variable (string): the column with numerical outliers\n",
    "      lower (numeric): observations with values lower than this will be removed\n",
    "      upper (numeric): observations with values higher than this will be removed\n",
    "    \n",
    "    Output:\n",
    "      a winsorized data frame with outliers removed\n",
    "    \"\"\"\n",
    "        \n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n",
    "    return data\n",
    "\n",
    "training_data = remove_outliers(training_data, 'Gr_Liv_Area', upper=5000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ceb68283aa877b0dd5a890c7421624a1",
     "grade": true,
     "grade_id": "cell-1b16eb9d9ed74f3e",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Make sure that two observations were removed\n",
    "assert training_data.shape[0] == 1998"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "0413a2294b2c572dde737686cc8f86a1",
     "grade": true,
     "grade_id": "cell-f1aeeb0f6c025fe4",
     "locked": true,
     "points": 2,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# No admittance.  Except on party business."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "5d8f0e56c7d26637bb993441a552ee26",
     "grade": false,
     "grade_id": "cell-7b287f5749ce8d38",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Neighborhood vs Sale Price"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "05a0ed781525c94d30df37a56eccd5e8",
     "grade": false,
     "grade_id": "cell-692437a76f5a0651",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "fig, axs = plt.subplots(nrows=2)\n",
    "\n",
    "sns.boxplot(\n",
    "    x='Neighborhood',\n",
    "    y='SalePrice',\n",
    "    data=training_data.sort_values('Neighborhood'),\n",
    "    ax=axs[0]\n",
    ")\n",
    "\n",
    "sns.countplot(\n",
    "    x='Neighborhood',\n",
    "    data=training_data.sort_values('Neighborhood'),\n",
    "    ax=axs[1]\n",
    ")\n",
    "\n",
    "# Draw median price\n",
    "axs[0].axhline(\n",
    "    y=training_data['SalePrice'].median(), \n",
    "    color='red',\n",
    "    linestyle='dotted'\n",
    ")\n",
    "\n",
    "# Label the bars with counts\n",
    "for patch in axs[1].patches:\n",
    "    x = patch.get_bbox().get_points()[:, 0]\n",
    "    y = patch.get_bbox().get_points()[1, 1]\n",
    "    axs[1].annotate(f'{int(y)}', (x.mean(), y), ha='center', va='bottom')\n",
    "    \n",
    "# Format x-axes\n",
    "axs[1].set_xticklabels(axs[1].xaxis.get_majorticklabels(), rotation=90)\n",
    "axs[0].xaxis.set_visible(False)\n",
    "\n",
    "# Narrow the gap between the plots\n",
    "plt.subplots_adjust(hspace=0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "ab673704ee06cd646391a2671bda0ea1",
     "grade": false,
     "grade_id": "cell-b1a7c4a82cbad2d7",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "From the plot above, it becomes clear that there is quite some variation in prices across neighborhoods.  Moreover, the amount of data available is not uniformly distributed among neighborhoods.  North Ames, for example, comprises almost 15% of the training data while Green Hill has a scant 2 observations in this data set.\n",
    "\n",
    "One way we can deal with the lack of data from some neighborhoods is to create a new feature that bins neighborhoods together.  Let's dichotomize our neighborhoods in a very crude way: we'll take the top 3 neighborhoods measured by median `SalePrice` and identify them as \"rich neighborhoods\"; the other neighborhoods are not marked.\n",
    "\n",
    "#### Question 5 <a name=\"q5\"></a>\n",
    "\n",
    "Write a function that returns list of the top n most pricy neighborhoods as measured by our choice of aggregating function.  For example, in the setup above, we would want to call `find_rich_neighborhoods(training_data, 3, np.median)` to find the top 3 neighborhoods measured by mdian `SalePrice`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "933e3ae5f7a1f93f900ebc53e9ed1abc",
     "grade": false,
     "grade_id": "cell-0b6e60ab464a87ca",
     "locked": false,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def find_rich_neighborhoods(data, n=3, metric=np.median):\n",
    "    \"\"\"\n",
    "    Input:\n",
    "      data (data frame): should contain at least a string-valued Neighborhood\n",
    "        and a numeric SalePrice column\n",
    "      n (int): the number of top values desired\n",
    "      metric (function): function used for aggregating the data in each neighborhood.\n",
    "        for example, np.median for median prices\n",
    "    \n",
    "    Output:\n",
    "      a list of the top n richest neighborhoods as measured by the metric function\n",
    "    \"\"\"\n",
    "    neighborhoods = ...\n",
    "    \n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n",
    "    \n",
    "    return neighborhoods\n",
    "\n",
    "rich_neighborhoods = find_rich_neighborhoods(training_data, 3, np.median)\n",
    "rich_neighborhoods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "fd74d3f486ee5a5515b77e0d4f9df2c6",
     "grade": true,
     "grade_id": "cell-56545f1d10f9699e",
     "locked": true,
     "points": 2,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Check to see if the n argument works\n",
    "assert len(find_rich_neighborhoods(training_data, 5, np.median)) == 5\n",
    "assert isinstance(rich_neighborhoods, list)\n",
    "# Check to see if the list contains only strings\n",
    "assert all([isinstance(neighborhood, str) for neighborhood in rich_neighborhoods])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e8b237c510ac6a8fa225b9ded441ac97",
     "grade": true,
     "grade_id": "cell-f58f1e4866635bf7",
     "locked": true,
     "points": 4,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Shoo."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Missing Data\n",
    "\n",
    "#### Question 6 <a name=\"q6\"></a>\n",
    "Let's see if our data set has any missing values.  Create a Series object containing the counts of missing values in each of the columns of our data set sorted from greatest to least.  The Series should be indexed by the variable names.  For example, `missing_counts['Fireplace_Qu']` should return 975."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "1d1e58af147fdf527f92a153eae97bc5",
     "grade": false,
     "grade_id": "cell-95890767879a9a12",
     "locked": false,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "missing_counts = ...\n",
    "\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "60a61507b65dfe5dda8b08d8f5c39d1b",
     "grade": true,
     "grade_id": "cell-93d71894d6084ce9",
     "locked": true,
     "points": 2,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Make sure your answer is a Series\n",
    "assert isinstance(missing_counts, pd.Series)\n",
    "# Make sure all columns are represented\n",
    "assert missing_counts.size == 82\n",
    "# Make sure your index values match column names\n",
    "assert set(missing_counts.index.values) == set(training_data.columns.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e83289ae96a09980b5feb008c1a9f594",
     "grade": true,
     "grade_id": "cell-bec86d2c1e1fede0",
     "locked": true,
     "points": 4,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Well go on now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "9715c998d7e40cc5b31d68660a0a6324",
     "grade": false,
     "grade_id": "cell-f1045c22c76ca88f",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "It turns out that if we look at the codebook carefully, some of these \"missing values\" aren't missing at all! The Assessor's Office just used `NA` to denote a special value or that the information was truly not applicable for one reason or another.  One such example is the `Fireplace_Qu` variable.\n",
    "```\n",
    "FireplaceQu (Ordinal): Fireplace quality\n",
    "\n",
    "       Ex\tExcellent - Exceptional Masonry Fireplace\n",
    "       Gd\tGood - Masonry Fireplace in main level\n",
    "       TA\tAverage - Prefabricated Fireplace in main living area or Masonry Fireplace inbasement\n",
    "       Fa\tFair - Prefabricated Fireplace in basement\n",
    "       Po\tPoor - Ben Franklin Stove\n",
    "       NA\tNo Fireplace\n",
    "```\n",
    "\n",
    "#### Question 7 <a name=\"q7\"></a>\n",
    "\n",
    "An `NA` here actually means that the house had no fireplace to rate.  Let's fix this in our data set.  Write a function that replaces the missing values in `Fireplace_Qu` with `'No Fireplace'`.  In addition, it should replace each abbreviated condition with its full word.  For example, `'TA'` should be changed to `'Average'`.  Hint: the [DataFrame.replace](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.replace.html) method may be useful here.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ef3e1cecb38b05bf316992120bcf8a62",
     "grade": false,
     "grade_id": "cell-0fc3d70040e13894",
     "locked": false,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def fix_fireplace_qu(data):\n",
    "    \"\"\"\n",
    "    Input:\n",
    "      data (data frame): a data frame containing a Fireplace_Qu column.  Its values\n",
    "                         should be limited to those found in the codebook\n",
    "    Output:\n",
    "      data frame identical to the input except with a refactored Fireplace_Qu column\n",
    "    \"\"\"\n",
    "    \n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n",
    "    \n",
    "    return data\n",
    "    \n",
    "training_data = fix_fireplace_qu(training_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "70cd761f2a5d5bc44891a148f538836c",
     "grade": true,
     "grade_id": "cell-37e22ea673b500be",
     "locked": true,
     "points": 2,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Make sure you've replaced all the missing values with 'No Fireplace'\n",
    "assert sum(training_data['Fireplace_Qu'] == 'No Fireplace') == 975"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "3651a601ae113fba30d2809547f78054",
     "grade": true,
     "grade_id": "cell-2ecd79a12548f1f7",
     "locked": true,
     "points": 4,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Bah humbug."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "85daafa1cb14549709fcecd762651343",
     "grade": false,
     "grade_id": "cell-03bb29004f9a9837",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "It turns out that simply fixing these missing values isn't sufficient for using `Fireplace_Qu` in our model.  Since `Fireplace_Qu` is a categorical/nominal variable, we will have to one-hot-encode the data.  Notice in the example code below that we have to pre-specify the categories.  Why? Imagine what would happen if we automatically generated the categories only from the training data.  What would happen if the testing data contained a category not found in the training set?  For more information on categorical data in pandas, refer to this [link](https://pandas-docs.github.io/pandas-docs-travis/categorical.html).  **Note that `get_dummies` removes the original column.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "7f72f1c3dce38a4752e37b1d31847a2d",
     "grade": false,
     "grade_id": "cell-990396e0b4792eab",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "def ohe_fireplace_qu(data):\n",
    "    \"\"\"\n",
    "    One-hot-encodes fireplace quality.  New columns are of the form fpq_QUALITY\n",
    "    \"\"\"\n",
    "    cats = [\n",
    "        'Excellent',\n",
    "        'Good',\n",
    "        'Average',\n",
    "        'Fair',\n",
    "        'Poor',\n",
    "        'No Fireplace'\n",
    "    ]\n",
    "    \n",
    "    cat_type = CategoricalDtype(categories=cats)\n",
    "    \n",
    "    data['Fireplace_Qu'] = data['Fireplace_Qu'].astype(cat_type)\n",
    "    data = pd.get_dummies(data,\n",
    "                          prefix='fpq',\n",
    "                          columns=['Fireplace_Qu'], \n",
    "                          drop_first=True)\n",
    "    \n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "fe6b6037064e80980bfa3a887340aa4b",
     "grade": false,
     "grade_id": "cell-d77d7326fbf4aa85",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "training_data = ohe_fireplace_qu(training_data)\n",
    "training_data.filter(regex='fpq').head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "b42fe33c8067797a14195356b3b75d0f",
     "grade": false,
     "grade_id": "cell-63f48a31637fa052",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "## Feature Engineering\n",
    "\n",
    "In this section we will create a new feature out of existing ones through a simple data transformation.  When you move on to create your own model, you may want to try out more complex transformations.\n",
    "\n",
    "### Bathrooms\n",
    "\n",
    "We will create a groundbreaking new feature.  Due to recent advances in Universal WC Enumeration Theory, we now know that Total Bathrooms can be calculated as:\n",
    "\n",
    "$$ TotalBathrooms=(BsmtFullBath + FullBath) + \\dfrac{1}{2}(BsmtHalfBath + HalfBath)$$\n",
    "\n",
    "The actual proof is beyond the scope of this class, but we will use the result in our model.\n",
    "#### Question 8 <a name=\"q8\"></a>\n",
    "\n",
    "Write a function `add_total_bathrooms(data)` that returns the input data frame with a new column called `total_bathrooms` as calculated above.  **Treat missing values as 0s**.  Remember that you can make use of vectorized code here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "6bd6e1b0b887d5c5226c23a737487e2b",
     "grade": false,
     "grade_id": "cell-116902ce0d5e9f37",
     "locked": false,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def add_total_bathrooms(data):\n",
    "    \"\"\"\n",
    "    Input:\n",
    "      data (data frame): a data frame containing at least 4 numeric columns \n",
    "            Bsmt_Full_Bath, Full_Bath, Bsmt_Half_Bath, and Half_Bath\n",
    "    Output:\n",
    "      data frame identical to the input with the addition of a total_bathrooms column\n",
    "    \"\"\"\n",
    "    \n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n",
    "    return data\n",
    "\n",
    "training_data = add_total_bathrooms(training_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "9c53366ad56e5981bb852c5d4b64e559",
     "grade": true,
     "grade_id": "cell-77e71ef4c3305b6c",
     "locked": true,
     "points": 2,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Check that missing values are dealt with\n",
    "assert ~training_data['total_bathrooms'].isnull().any()\n",
    "# Check that the values are as expected\n",
    "assert training_data['total_bathrooms'].sum() == 4421.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "3f9fd8079e5139968dffd833d977c81d",
     "grade": true,
     "grade_id": "cell-f3794a6cf09068dc",
     "locked": true,
     "points": 4,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# **leers**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "ea24da8e33ea7256bb1df8a6ecd7803c",
     "grade": false,
     "grade_id": "cell-7f7419ac0a98e696",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Rich Neighborhoods\n",
    "\n",
    "#### Question 9 <a name=\"q9\"></a>\n",
    "From before, we have a list of neighborhoods we've deemed as richer than others.  Let's use that information to make a new variable `in_rich_neighborhood`.  Write a function `add_rich_neighborhood` that adds an indicator variable which takes on the value 1 if the house is part of `rich_neighborhoods` (question 4) and the value 0 otherwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "25ac6d1e8e8d99c33593a782c8535ce8",
     "grade": false,
     "grade_id": "cell-c48789abe9e04e3a",
     "locked": false,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def add_in_rich_neighborhood(data, neighborhoods):\n",
    "    \"\"\"\n",
    "    Input:\n",
    "      data (data frame): a data frame containing a 'Neighborhood' column with values\n",
    "        found in the codebook\n",
    "      neighborhoods (list of strings): strings should be the names of neighborhoods\n",
    "        pre-identified as rich\n",
    "    Output:\n",
    "      data frame identical to the input with the addition of a binary\n",
    "      in_rich_neighborhood column\n",
    "    \"\"\"\n",
    "\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n",
    "    \n",
    "    return data\n",
    "\n",
    "rich_neighborhoods = find_rich_neighborhoods(training_data, 3, np.median)\n",
    "training_data = add_in_rich_neighborhood(training_data, rich_neighborhoods)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "77cd5d714b480e5f270fafa99181a10c",
     "grade": true,
     "grade_id": "cell-5ebcf403adaba07b",
     "locked": true,
     "points": 2,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Check to see if you have identified the correct number of rich neighborhoods\n",
    "assert sum(training_data['in_rich_neighborhood']) == 191\n",
    "# Check to see if you've introduced any missing values\n",
    "assert sum(training_data['in_rich_neighborhood'].isnull()) == 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "78cbefe32609b859ea95f7da5f6a4bc0",
     "grade": true,
     "grade_id": "cell-aea0ebbf97f0c3be",
     "locked": true,
     "points": 4,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Repello Muggletum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "b93c0348a3cb01723699a8996cd46c3d",
     "grade": false,
     "grade_id": "cell-5ffdfab3f8801658",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "## Modeling\n",
    "\n",
    "We've finally gotten to a point where we can specify a simple model.  But first, we need to create perform a test-train split of our data.  We begin by loading a fresh copy of the data in at this point just in case our code above produced any undesired side-effects.  At this point, we will begin to treat `ames_train.csv` as our complete data set.  We will use `train_test_split` from `sklearn` to split the data into `test` and `train` sets.\n",
    "\n",
    "Remember: The reason we have to do a train-test split on `ames_train.csv` here is because we want to evaluate how well our model might perform on future data (`ames_test.csv`), but that data set does not have `SalePrice` in it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "cae7b5b9b8cefaa31603e6a52377c865",
     "grade": false,
     "grade_id": "cell-700027ec3c0adc57",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Load a fresh copy of the data\n",
    "full_data = pd.read_csv(\"ames_train.csv\")\n",
    "\n",
    "# This makes the train-test split in this section reproducible across different runs \n",
    "# of the notebook.  You do not need this line to run train_test_split in general\n",
    "np.random.seed(1337) \n",
    "\n",
    "# Split the data \n",
    "train, test = train_test_split(full_data, test_size=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "0c62535814a2477f247b6f14b17427ca",
     "grade": false,
     "grade_id": "cell-8006b45a1ad7725b",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Something has gone awry in the cell above if these do not pass\n",
    "assert test.shape == (400, 82)\n",
    "assert train.shape == (1600, 82)\n",
    "assert train.loc[887, 'PID'] == 902402260"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "292955f8fe0d49f2db32d47916229158",
     "grade": false,
     "grade_id": "cell-acdc861fd11912e9",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Reusable Pipeline\n",
    "\n",
    "Throughout this assignment, you should notice that your data flows through a single processing pipeline several times.  From a software engineering perspective, this should be sufficient motivation to abstract parts of our code into reusable functions/methods.  We will now encapsulate our entire pipeline into a single function `process_data_gm`.  gm is shorthand for \"guided model\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "6f65708807b775a088d2550bfa491b9e",
     "grade": false,
     "grade_id": "cell-2fe1d82b2c19d1fa",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "def select_columns(data, *columns):\n",
    "    return data.loc[:, columns]\n",
    "\n",
    "def process_data_gm1(data):\n",
    "    # Clean Data\n",
    "    data = remove_outliers(data, 'Gr_Liv_Area', upper=5000)\n",
    "    data = fix_fireplace_qu(data)\n",
    "    \n",
    "    # Transform Data\n",
    "    data = add_total_bathrooms(data)\n",
    "    data = add_in_rich_neighborhood(data, rich_neighborhoods)\n",
    "    data = select_columns(data, \n",
    "                          'SalePrice', \n",
    "                          'Gr_Liv_Area', \n",
    "                          'total_bathrooms',\n",
    "                          'Fireplace_Qu',\n",
    "                          'in_rich_neighborhood'\n",
    "                         )\n",
    "    data = ohe_fireplace_qu(data)\n",
    "    \n",
    "    # Return predictors and response variables separately\n",
    "    X = data.drop(['SalePrice'], axis = 1)\n",
    "    y = data['SalePrice']\n",
    "    \n",
    "    return X, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "1a717417f8ba4b81535d7e7fe8d0b941",
     "grade": false,
     "grade_id": "cell-dcea887f3db333c0",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "An alternative way of writing the same code above explicitly allows us to think about our data flowing through a [pipeline](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.pipe.html) where the output of one function is the input of the next.  Carefully thought out function names make the code self-documenting: you can just read off the intended high-level processing steps from top to bottom.\n",
    "\n",
    "You are not required to use this style of coding.  We just wanted to point out that it exists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "6c7a5414e0d8d14a86c746a00225ebc5",
     "grade": false,
     "grade_id": "cell-d7de4796ff0ab97d",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "def process_data_gm1(data):\n",
    "\n",
    "    data = (\n",
    "        data\n",
    "        # Clean Data\n",
    "        .pipe(remove_outliers, 'Gr_Liv_Area', upper=5000)\n",
    "        .pipe(fix_fireplace_qu)\n",
    "        \n",
    "        # Transform data\n",
    "        .pipe(add_total_bathrooms)\n",
    "        .pipe(add_in_rich_neighborhood, rich_neighborhoods)\n",
    "        .pipe(select_columns, \n",
    "              'SalePrice',           \n",
    "              'Gr_Liv_Area',            \n",
    "              'total_bathrooms',             \n",
    "              'Fireplace_Qu',            \n",
    "              'in_rich_neighborhood'\n",
    "             )\n",
    "        .pipe(ohe_fireplace_qu)\n",
    "    ) \n",
    "    \n",
    "    # Return predictors and response variables separately\n",
    "    X = data.drop(['SalePrice'], axis = 1)\n",
    "    y = data['SalePrice']\n",
    "    \n",
    "    return X, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "fc1078fbaf631bd50ebc349161483a41",
     "grade": false,
     "grade_id": "cell-41994ca25b31660e",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Fitting our first model\n",
    "\n",
    "We are finally going to fit a model!  This part is slightly unceremonious since we did much of the heavy lifting in the previous sections.  The model we will fit can be written as follows (with the caveat that one of the fireplace qualities is actually removed to avoid collinearity):\n",
    "\n",
    "$$\\begin{align} SalePrice = &\\theta_0 + \\theta_1 \\times GrLivArea + \\theta_2 \\times TotalBathrooms \\\\\n",
    "&+ \\theta_3 \\times InRichNeighborhood + \\sum_{quality \\in FireplaceQuality} \\theta_{quality} \\times Quality\n",
    "\\end{align}$$\n",
    "\n",
    "#### Question 10a <a name=\"q10a\"></a>\n",
    "Remove the commenting and fill in the ellipses `...` below with `X_train`, `y_train`, `X_test`, or `y_test`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "4cf94f1f1fb794a32ecf7ab61ba5fe18",
     "grade": false,
     "grade_id": "cell-1be99eea86f6cf57",
     "locked": false,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# Pre-process our training and test data in exactly the same way\n",
    "# Our functions make this very easy!\n",
    "X_train, y_train = process_data_gm1(train)\n",
    "X_test, y_test = process_data_gm1(test)\n",
    "guidedmodel1 = lm.LinearRegression(fit_intercept=True)\n",
    "\n",
    "# Fill in the ... below with X_train, y_train, X_train, or X_test.\n",
    "# Remember to uncomment\n",
    "# guidedmodel1.fit(..., ...)\n",
    "# y_fitted = guidedmodel1.predict(...)\n",
    "# y_predicted = guidedmodel1.predict(...)\n",
    "\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "c4618cb37f44c899c05ef718b5799be8",
     "grade": true,
     "grade_id": "cell-374e010658179ed3",
     "locked": true,
     "points": 3,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert 181100 <= y_fitted.mean() <= 181400\n",
    "assert 177700 <= y_predicted.mean() <= 177800"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "b4d15666eb0b2af71bc8e6a20da031f6",
     "grade": false,
     "grade_id": "cell-e963e93c7b4f9bad",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "In this assignment, we will use Root-Mean-Square Error (RMSE) to measure the quality of our models.  As a reminder, this quantity is defined as:\n",
    "\n",
    "$$RMSE = \\sqrt{\\dfrac{\\sum_{\\text{houses in test set}}(\\text{actual price of house} - \\text{predicted price of house})^2}{\\text{# of houses in test set}}}$$\n",
    "\n",
    "#### Question 10b <a name=\"q10b\"></a>\n",
    "\n",
    "Write a function `rmse` that calculates the RMSE of a model.  Again, make sure you are taking advantage of vectorized code.  This can be solved without any iteration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "de2c8737b6b9dac7f3a963708f61f46b",
     "grade": false,
     "grade_id": "cell-96600fa98a6c2e97",
     "locked": false,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def rmse(actual, predicted):\n",
    "    \"\"\"\n",
    "    Calculates RMSE from actual and predicted values\n",
    "    Input:\n",
    "      actual (1D array-like): vector of actual values\n",
    "      predicted (1D array-like): vector of predicted/fitted values\n",
    "    Output:\n",
    "      a float, the root-mean square error\n",
    "    \"\"\"\n",
    "    \n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n",
    "    \n",
    "    return rmse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "5038d16892edf7e473974c7556417578",
     "grade": true,
     "grade_id": "cell-0684404647cc924f",
     "locked": true,
     "points": 3,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert 43000 <= rmse(y_test, y_predicted) <= 44000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "666865df7f66c4b84cee8ff57f063900",
     "grade": false,
     "grade_id": "cell-a359da2dda38fcdd",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Residual Plots\n",
    "\n",
    "One way of diagnosing a model is through a residual plot.  Here we plot the actual sale prices against the residuals of the model.  Ideally, we would see a horizontal line of points at 0 (perfect prediction!).  The next best thing would be a homogenous set of points centered at 0.  But alas, our simple model is probably too simple.  We notice that we are really underfitting the more expensive homes in the test set.  In fact, it looks like our model tends to underprice the more expensive homes!  You will probably want to address this in your own work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "26bf8da4daa6b4fceefca67e3f961cc3",
     "grade": false,
     "grade_id": "cell-4d79f42d60b94fca",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "gm1_residuals = y_test - y_predicted\n",
    "ax = sns.regplot(y_test, gm1_residuals)\n",
    "ax.set_xlabel('Sale Price (Test Data)')\n",
    "ax.set_ylabel('Residuals (Actual Price - Predicted Price)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "18d527386ae2b92ff3ce55c9d0d14b41",
     "grade": false,
     "grade_id": "cell-4638dca2c51b40f0",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Regularizing our model\n",
    "\n",
    "Ok, so let's make our modeling a little more fancy by regularizing the coefficients.  This second model will use the Lasso, but you are free to use Ridge or ElasticNet in your work.  First, we need to add a step to our pre-processing.  In order for regularization to be fair to all the variables in our model, we need to standardize our predictor columns (otherwise it would unfairly penalize variables with inherently small values).\n",
    "\n",
    "#### Question 11 <a name=\"q11\"></a>\n",
    "\n",
    "Write a function that standardizes the columns of a data frame containing only numeric columns.  Be sure to make use of vectorization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "7558f1490e4147444c8eccc3ca600bf9",
     "grade": false,
     "grade_id": "cell-7331e38e38968cb9",
     "locked": false,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def standardize_columns(data):\n",
    "    '''\n",
    "    Input:\n",
    "      data (data frame): contains only numeric columns\n",
    "    Output:\n",
    "      data frame, the same data, except each column is standardized \n",
    "      to have 0-mean and unit variance\n",
    "    '''\n",
    "    standardized_data = ...\n",
    "    \n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n",
    "    \n",
    "    return standardized_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "2c923e7b76988d0a811187ef759e6eea",
     "grade": true,
     "grade_id": "cell-af48dec574258015",
     "locked": true,
     "points": 6,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_standardize_df = standardize_columns(training_data[['SalePrice', 'Gr_Liv_Area']])\n",
    "# Make sure the mean is correct\n",
    "assert -0.001 < test_standardize_df.mean().sum() < 0.001\n",
    "# Make sure the standard deviation is correct\n",
    "assert 1.9 < test_standardize_df.std().sum() < 2.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "1fb620524b21aa564cb592870d051da8",
     "grade": false,
     "grade_id": "cell-2b0b86e3a1743a83",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "We'll now use this function to describe our new processing for the regularized model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "9a5fc41c2d6689b3278e8bf98a483bf8",
     "grade": false,
     "grade_id": "cell-50de09444fbd962d",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "def process_data_gm2(data):\n",
    "\n",
    "    data = (\n",
    "        data\n",
    "        # Clean Data\n",
    "        .pipe(remove_outliers, 'Gr_Liv_Area', upper=5000)\n",
    "        .pipe(fix_fireplace_qu)\n",
    "        \n",
    "        # Transform data\n",
    "        .pipe(add_total_bathrooms)\n",
    "        .pipe(add_in_rich_neighborhood, rich_neighborhoods)\n",
    "        .pipe(select_columns, \n",
    "              'SalePrice',           \n",
    "              'Gr_Liv_Area',            \n",
    "              'total_bathrooms',             \n",
    "              'Fireplace_Qu',            \n",
    "              'in_rich_neighborhood'\n",
    "             )\n",
    "        .pipe(ohe_fireplace_qu)\n",
    "    ) \n",
    "    \n",
    "    # Return predictor and response variables separately\n",
    "    X = standardize_columns(data).drop(['SalePrice'], axis = 1)\n",
    "    y = data['SalePrice']\n",
    "    \n",
    "    return X, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "104f1875d38b3231b1ca650397eb1354",
     "grade": false,
     "grade_id": "cell-d15ce882bb659e94",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "It may be instructive to see the cross-validation procedure explicitly once.  You should be able to understand what each part of the code is doing below, but we do not expect you to use this code for your own model (use [LassoCV](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoCV.html) instead)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "cd16dcff03097f91b02409a841179214",
     "grade": false,
     "grade_id": "cell-093735333c641eab",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Process our data\n",
    "X_train, y_train = process_data_gm2(train)\n",
    "X_test, y_test = process_data_gm2(test)\n",
    "\n",
    "# Specify our model\n",
    "guidedmodel2 = lm.Lasso(copy_X=True)\n",
    "\n",
    "# Specify CV method and alpha grid\n",
    "five_fold_cv = KFold(n_splits = 5)\n",
    "alphas = np.arange(0.1, 200.1, .1)\n",
    "rmses = np.zeros(len(alphas))\n",
    "\n",
    "# Grid search over alphas\n",
    "for i, alpha in enumerate(alphas):\n",
    "    guidedmodel2.set_params(alpha=alpha)\n",
    "    model_rmse = 0\n",
    "    \n",
    "    # Fit each fold using the other four as training data\n",
    "    for train_index, test_index in five_fold_cv.split(X_train):\n",
    "        X_fold_train = X_train.iloc[train_index]\n",
    "        y_fold_train = y_train.iloc[train_index]\n",
    "        X_fold_test = X_train.iloc[test_index]\n",
    "        y_fold_test = y_train.iloc[test_index]\n",
    "        \n",
    "        guidedmodel2.fit(X_fold_train, y_fold_train)\n",
    "        y_fold_predicted = guidedmodel2.predict(X_fold_test)\n",
    "        model_rmse += rmse(y_fold_test, y_fold_predicted)\n",
    "    \n",
    "    # Average RMSE over the five folds for alpha_i\n",
    "    rmses[i] = model_rmse / 5\n",
    "\n",
    "optimal_alpha = alphas[rmses == np.min(rmses)]\n",
    "guidedmodel2.set_params(alpha=optimal_alpha)\n",
    "guidedmodel2.fit(X_train, y_train)\n",
    "y_predicted = guidedmodel2.predict(X_test)\n",
    "\n",
    "print(f'The validation RMSE for this model with '\n",
    "      f'alpha={float(optimal_alpha)} is {rmse(y_test, y_predicted)}.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "8a30ca6527d79e826f0a2f7b4f40f4d6",
     "grade": false,
     "grade_id": "cell-7e699b2561e846e5",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Lasso Path\n",
    "\n",
    "Let's take a look at how RMSE varied across different choices of the regularization hyperparameter ($\\lambda$ in lecture, `alpha` in `sklearn`).  This is often called the Lasso or Regularization Path.  The dashed red line marks the alpha that minimizes RMSE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "6ff96a7c92298ee4c945bc2bfc5214db",
     "grade": false,
     "grade_id": "cell-031b47ffdfd586a3",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "plt.plot(alphas, rmses)\n",
    "plt.axvline(x=optimal_alpha, color='red', linestyle='dashed')\n",
    "ax = plt.gca()\n",
    "ax.set_title('Average LASSO RMSE Path')\n",
    "ax.set_xlabel('Regularization Level')\n",
    "ax.set_ylabel('RMSE')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "a42e18f50628f81d09a97eec9ed2a25f",
     "grade": false,
     "grade_id": "cell-1a2d8b21df5a43ff",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### A simplified approach to Lasso and CV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "4a9d3635f2158a88c99deda33ca780af",
     "grade": false,
     "grade_id": "cell-29756b33edf57bc8",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "Here we perform another L1-regularized regression but using `LassoCV`.  This is more in line with what we expect from you code-wise.  That being said, you should still understand the concepts from the previous section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "558ebaf1a0faea23d0f73abe872d0dd0",
     "grade": false,
     "grade_id": "cell-e7875ce8a7c2435a",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Process the data\n",
    "X_train, y_train = process_data_gm2(train)\n",
    "X_test, y_test = process_data_gm2(test)\n",
    "\n",
    "# Specify the model, alphas, and number of folds for CV\n",
    "alphas = np.arange(0.1, 200.1, .1)\n",
    "guidedmodel2 = lm.LassoCV(alphas=alphas, cv=5)\n",
    "\n",
    "# Fit and predict\n",
    "guidedmodel2.fit(X_train, y_train)\n",
    "y_predicted = guidedmodel2.predict(X_test)\n",
    "\n",
    "print(f'The validation RMSE for this model with '\n",
    "      f'alpha={round(float(guidedmodel2.alpha_), 2)} is '\n",
    "      f'{round(rmse(y_test, y_predicted), 2)}.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "de766f4e7e50e37a60265f9463c7cdfc",
     "grade": false,
     "grade_id": "cell-d2b63d56e68f43c1",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "Again, we can map out the path that the Lasso algorithm took."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "8675620d991efb902e1439252754c090",
     "grade": false,
     "grade_id": "cell-9a316c221f111bc4",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "plt.plot(guidedmodel2.alphas_, np.sqrt(np.apply_along_axis(np.mean, 1, guidedmodel2.mse_path_)))\n",
    "plt.axvline(x=optimal_alpha, color='red', linestyle='dashed')\n",
    "ax = plt.gca()\n",
    "ax.set_title('Average LASSO RMSE Path')\n",
    "ax.set_xlabel('alpha')\n",
    "ax.set_ylabel('RMSE')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "3df1f08f72305cd59937bf627a81b9bc",
     "grade": false,
     "grade_id": "cell-a56a300a011993b3",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Lasso Residual Plot\n",
    "\n",
    "Looking at the residual plot for our L1 regularized linear model, it's clear the regularization did not solve the problems we saw in the simple model.  It seems you have your work cut out for you!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "7c63e9fe12ee5c32c12a83c5e4a0d9bf",
     "grade": false,
     "grade_id": "cell-aaa612b22458ba79",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "gm2_residuals = y_test - y_predicted\n",
    "ax = sns.regplot(y_test, gm2_residuals)\n",
    "ax.set_xlabel('Sale Price (Test Data)')\n",
    "ax.set_ylabel('Residuals (Actual Price - Predicted Price)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "fb19bb32ea8b1f8cc0610866b31e7e57",
     "grade": false,
     "grade_id": "cell-eaf5f48a7709a3e5",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "# Open-Response\n",
    "\n",
    "The second assignment is purposefully left nearly open-ended.  The Ames data in your possession comes from a larger data set.  Your goal is to provide a linear model (linear regression, Lasso, Ridge, or ElasticNet) that accurately predicts the prices of the held-out homes, measured by root mean square error.  That is, the score you will see on the Kaggle leaderboard is calculated as follows:\n",
    "\n",
    "$$score = \\sqrt{\\dfrac{\\sum_{\\text{houses in public test set}}(\\text{actual price for house} - \\text{predicted price for house})^2}{\\text{# of houses}}}$$\n",
    "\n",
    "Perfect prediction of house prices would have a score of 0, so you want your score to be as low as possible!\n",
    "\n",
    "**Kaggle Submission Site:** https://inclass.kaggle.com/c/ds100-s2018-hw6  \n",
    "**Max number of submissions per day:** 2  \n",
    "**Max number of final submissions:** 1 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "c3b7819312f92e8a964a97dfd6cfd8f8",
     "grade": false,
     "grade_id": "cell-22c456dc06025600",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Grading Scheme\n",
    "\n",
    "Your grade for the open-response section will be based on 4 things: your training RMSE, your public test set RMSE (seen on Kaggle), and your private test set RMSE (hidden until the end of the competition).  The thresholds are as follows:\n",
    "\n",
    "#### RMSE\n",
    "Points | 10 | 8 | 6 | 4 | 2 | 0\n",
    "--- | --- | --- | --- | --- | --- | ---\n",
    "Training RMSE | Less than 26k | 26k - 28k | 28k - 30k | 30k-35k | More than 35k | No work\n",
    "\n",
    "Points | 10 | 8 | 6 | 4 | 2 | 0\n",
    "--- | --- | --- | --- | --- | --- | ---\n",
    "Public Test Set RMSE | Less than 27k | 27k - 30k | 30k - 33k | 33k-36k | More than 36k | No work\n",
    "\n",
    "Points | 5 | 4 | 3 | 2 | 1 | 0\n",
    "--- | --- | --- | --- | --- | --- | ---\n",
    "Private Test Set RMSE | Less than 29k | 29k - 32k | 32k - 35k | 35k-38k | More than 38k | No work "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "93dcbebc9655308fdc691e0f80b77d54",
     "grade": false,
     "grade_id": "cell-3f5b5d8565d36d05",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Deliverables\n",
    "\n",
    "#### Question 12 <a name=\"q12\"></a>\n",
    "\n",
    "Just as in the guided model above, you should encapsulate as much of your workflow into functions as possible.  Define `process_data_fm` and `final model` in the cell below.  In order to calculate your final model's RMSE, we will run the code in the cell after that.\n",
    "\n",
    "**It is your duty to make sure that the code runs.** We will **NOT** accept regrade requests that require us to go back and run code that require typo/bug fixes. You can expect the data to be formatted like `ames_train.csv`.  `public_test_set.csv` and `private_test_set.csv` are both subsets of `ames_test.csv` (except that they both have the `SalePrice` column that was omitted from `ames_test.csv`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b3e553412af205894c0a206759e16b23",
     "grade": false,
     "grade_id": "cell-b5888b8cd104dcd4",
     "locked": false,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def process_data_fm(data):\n",
    "    ...\n",
    "    return data\n",
    "\n",
    "# This needs to be an object with a fit and predict method\n",
    "# The linear_model objects from sklearn all satisfy this\n",
    "# e.g. final_model = lm.LinearRegression(fit_intercept=False)\n",
    "# You could be fancy and define your own class.\n",
    "# Do this at your own peril.\n",
    "\n",
    "final_model = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "03345616c43103a3d79ac35096fef1da",
     "grade": false,
     "grade_id": "cell-4c29935005045f02",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# training_data = pd.read_csv('ames_train.csv')\n",
    "# public_test_data = pd.read_csv('public_test_set.csv')\n",
    "# private_test_data = pd.read_csv('private_test_set.csv')\n",
    "\n",
    "# X_train, y_train = process_data_fm(training_data)\n",
    "# X_public, y_public = process_data_fm(public_test_data)\n",
    "# X_private, y_private = process_data_fm(private_test_data)\n",
    "\n",
    "# final_model.fit(X_train, y_train)\n",
    "# y_predicted_train = final_model.predict(X_train)\n",
    "# y_predicted_public = final_model.predict(X_public)\n",
    "# y_predicted_private = final_model.predict(X_private)\n",
    "\n",
    "# training_score = rmse(y_predicted_train, y_train)\n",
    "# public_score = rmse(y_predicted_public, y_public)\n",
    "# private_score = rmse(y_predicted_private, y_private)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f477076fcbde845e2baca48f39e72ba4",
     "grade": true,
     "grade_id": "cell-64fa94746f0fd183",
     "locked": true,
     "points": 0,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# NO TOUCH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "4b60fb383e0aab3af059281b923eee2e",
     "grade": true,
     "grade_id": "cell-fd20db43e21281c2",
     "locked": true,
     "points": 2,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# NOH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b235ad05cb0cd447a3a6d00c97488cb3",
     "grade": true,
     "grade_id": "cell-dde1980abb5265eb",
     "locked": true,
     "points": 2,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# STAHP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e27afeb776e2bf6f541e08a54196317c",
     "grade": true,
     "grade_id": "cell-4720d6f913f2e2a8",
     "locked": true,
     "points": 2,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# NO MOLESTE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "97f7651a62f9408f892bdc515d7eca99",
     "grade": true,
     "grade_id": "cell-becfc4e39f8d613d",
     "locked": true,
     "points": 2,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# VA-T'EN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f70a1c01cbfb70718951dbd41b85f76a",
     "grade": true,
     "grade_id": "cell-66c75927b3162384",
     "locked": true,
     "points": 2,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# NEIN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f83b99fffeb32b5eaceb6d2a680d4298",
     "grade": true,
     "grade_id": "cell-480ee1c73e5aeed2",
     "locked": true,
     "points": 2,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# PLSNO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f8742a88fe2a2b0dba291db16fccf20b",
     "grade": true,
     "grade_id": "cell-5c13060e51df5fbf",
     "locked": true,
     "points": 2,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# THIS SPACE IS NOT YOURS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e14b1a962cbcf2b8efd3f4f34a16fbc8",
     "grade": true,
     "grade_id": "cell-9594161a0c8ada1b",
     "locked": true,
     "points": 2,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# TAWDEETAW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "fb3d42d88c662d386250832154cae0c0",
     "grade": true,
     "grade_id": "cell-7ff15da448905075",
     "locked": true,
     "points": 2,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# MAU LEN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "dbfb9c3dc9ca0e9d41681d097a44d0be",
     "grade": true,
     "grade_id": "cell-5799ae0e2e02e4ea",
     "locked": true,
     "points": 2,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# ALMOST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b71274e2a3a0876812af46e7fadda22a",
     "grade": true,
     "grade_id": "cell-4e8f93c8d961fdc3",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# TO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "2017f2b6d777ef593da976d61ccfa819",
     "grade": true,
     "grade_id": "cell-cb843d9e9defdb3f",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# THE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f4cd698d64eab1bfb873811b9d7cf3ea",
     "grade": true,
     "grade_id": "cell-a83a8dbeab2377b8",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# END"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f1df2786662d79bb122a671d432f6287",
     "grade": true,
     "grade_id": "cell-c11ba085e96a4d43",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Hmph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "9dcf92d155de3c51788e9f94c9b684d2",
     "grade": true,
     "grade_id": "cell-d83490b23ffd0d29",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Good riddance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "99022cdd710b881129649d6f74b4c5e3",
     "grade": false,
     "grade_id": "cell-9462be06538c26fb",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "#### Question 13 <a name=\"q13\"></a>\n",
    "\n",
    "In addition, please submit one visualization from your EDA with 2-5 sentences describing why you thought the plot was interesting and what decisions it led to in your model specification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "5852472798fd64080a4e51980583c337",
     "grade": true,
     "grade_id": "cell-9c4eca5ee103ab36",
     "locked": false,
     "points": 3,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "2566c0c8d1fcf21013adef5c00364d2c",
     "grade": false,
     "grade_id": "cell-ae16ef963b92e45a",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "Commentary about your plot goes in the cell below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "f8139e204d2657a14fca0069c8ba2d67",
     "grade": true,
     "grade_id": "cell-b7e08136b402bce9",
     "locked": false,
     "points": 2,
     "schema_version": 2,
     "solution": true
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "c607a993b7d7223a071454892200733c",
     "grade": false,
     "grade_id": "cell-a29ddf6d68aa973d",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Submitting to Kaggle\n",
    "\n",
    "The following code will write your predictions on the test dataset to a CSV, which you can submit to kaggle.  You may need to modify it a little to suit your needs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "c340fa932da1c594fbf57d290104d19c",
     "grade": false,
     "grade_id": "cell-b59b8204e11be6e4",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "GENERATE_SUBMISSION = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "a2714382cb176aa36d5ba57a3fbd1d8e",
     "grade": false,
     "grade_id": "cell-a4d6ff7cab1c1a7c",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "if GENERATE_SUBMISSION:\n",
    "    \n",
    "    test_data = pd.read_csv(\"ames_test.csv\")\n",
    "\n",
    "    submission_df = pd.DataFrame(\n",
    "        {\n",
    "        \"PID\": ..., # This will come from ames_test.csv\n",
    "        \"SalePrice\": ... # This will come from your model\n",
    "        }\n",
    "    )\n",
    "\n",
    "    timestamp = datetime.isoformat(datetime.now()).split(\".\")[0]\n",
    "\n",
    "    submission_df.to_csv(f'submission_{timestamp}', index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
