{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a href=\"http://www.datascience-paris-saclay.fr\">Paris Saclay Center for Data Science</a>\n",
    "# <a href=https://www.ramp.studio/problems/air_passengers>RAMP</a> on predicting the number of air passengers\n",
    "\n",
    "<i> Balázs Kégl (LAL/CNRS), Alex Gramfort (Inria), Djalel Benbouzid (UPMC), Mehdi Cherti (LAL/CNRS) </i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "The data set was donated to us by an unnamed company handling flight ticket reservations. The data is thin, it contains\n",
    "<ul>\n",
    "<li> the date of departure\n",
    "<li> the departure airport\n",
    "<li> the arrival airport\n",
    "<li> the mean and standard deviation of the number of weeks of the reservations made before the departure date\n",
    "<li> a field called <code>log_PAX</code> which is related to the number of passengers (the actual number were changed for privacy reasons)\n",
    "</ul>\n",
    "\n",
    "The goal is to predict the <code>log_PAX</code> column. The prediction quality is measured by RMSE. \n",
    "\n",
    "The data is obviously limited, but since data and location informations are available, it can be joined to external data sets. <b>The challenge in this RAMP is to find good data that can be correlated to flight traffic</b>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import os\n",
    "import importlib\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "pd.set_option('display.max_columns', None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the dataset using pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training and testing data are located in the folder `data`. They are compressed `csv` file (i.e. `csv.bz2`). We can load the dataset using pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(\n",
    "    os.path.join('data', 'train.csv.bz2')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 8902 entries, 0 to 8901\n",
      "Data columns (total 6 columns):\n",
      "DateOfDeparture     8902 non-null object\n",
      "Departure           8902 non-null object\n",
      "Arrival             8902 non-null object\n",
      "WeeksToDeparture    8902 non-null float64\n",
      "log_PAX             8902 non-null float64\n",
      "std_wtd             8902 non-null float64\n",
      "dtypes: float64(3), object(3)\n",
      "memory usage: 417.4+ KB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So as stated earlier, the column `log_PAX` is the target for our regression problem. The other columns are the features which will be used for our prediction problem. If we focus on the data type of the column, we can see that `DateOfDeparture`, `Departure`, and `Arrival` are of `object` dtype, meaning they are strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DateOfDeparture</th>\n",
       "      <th>Departure</th>\n",
       "      <th>Arrival</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2012-06-19</td>\n",
       "      <td>ORD</td>\n",
       "      <td>DFW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2012-09-10</td>\n",
       "      <td>LAS</td>\n",
       "      <td>DEN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2012-10-05</td>\n",
       "      <td>DEN</td>\n",
       "      <td>LAX</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011-10-09</td>\n",
       "      <td>ATL</td>\n",
       "      <td>ORD</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2012-02-21</td>\n",
       "      <td>DEN</td>\n",
       "      <td>SFO</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  DateOfDeparture Departure Arrival\n",
       "0      2012-06-19       ORD     DFW\n",
       "1      2012-09-10       LAS     DEN\n",
       "2      2012-10-05       DEN     LAX\n",
       "3      2011-10-09       ATL     ORD\n",
       "4      2012-02-21       DEN     SFO"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[['DateOfDeparture', 'Departure', 'Arrival']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While it makes `Departure` and `Arrival` are the code of the airport, we see that the `DateOfDeparture` should be a date instead of string. We can use pandas to convert this data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.loc[:, 'DateOfDeparture'] = pd.to_datetime(data.loc[:, 'DateOfDeparture'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 8902 entries, 0 to 8901\n",
      "Data columns (total 6 columns):\n",
      "DateOfDeparture     8902 non-null datetime64[ns]\n",
      "Departure           8902 non-null object\n",
      "Arrival             8902 non-null object\n",
      "WeeksToDeparture    8902 non-null float64\n",
      "log_PAX             8902 non-null float64\n",
      "std_wtd             8902 non-null float64\n",
      "dtypes: datetime64[ns](1), float64(3), object(2)\n",
      "memory usage: 417.4+ KB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you will create a submission, `ramp-workflow` will load the data for you and split into a data matrix `X` and a target vector `y`. It will also take care about splitting the data into a training and testing set. These utilities are available in the module `problem.py` which we will load."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `get_train_data()` loads the training data and returns a pandas dataframe `X` and a numpy vector `y`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = problem.get_train_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check the information of the data `X`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 8902 entries, 0 to 8901\n",
      "Data columns (total 5 columns):\n",
      "DateOfDeparture     8902 non-null object\n",
      "Departure           8902 non-null object\n",
      "Arrival             8902 non-null object\n",
      "WeeksToDeparture    8902 non-null float64\n",
      "std_wtd             8902 non-null float64\n",
      "dtypes: float64(2), object(3)\n",
      "memory usage: 347.9+ KB\n"
     ]
    }
   ],
   "source": [
    "X.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, this is important to see that `ramp-workflow` does not convert the `DateOfDeparture` column into a `datetime` format. Thus, keep in mind that you might need to make a conversion when prototyping your machine learning pipeline later on. Let's check some statistics regarding our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2011-09-01\n",
      "2013-03-05\n"
     ]
    }
   ],
   "source": [
    "print(min(X['DateOfDeparture']))\n",
    "print(max(X['DateOfDeparture']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['ORD', 'LAS', 'DEN', 'ATL', 'SFO', 'EWR', 'IAH', 'LAX', 'DFW',\n",
       "       'SEA', 'JFK', 'PHL', 'MIA', 'DTW', 'BOS', 'MSP', 'CLT', 'MCO',\n",
       "       'PHX', 'LGA'], dtype=object)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X['Departure'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 5092 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_ = plt.hist(y, bins=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 7440 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_ = X.hist('std_wtd', bins=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 8188 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_ = X.hist('WeeksToDeparture', bins=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>WeeksToDeparture</th>\n",
       "      <th>std_wtd</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>8902.000000</td>\n",
       "      <td>8902.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>11.446469</td>\n",
       "      <td>8.617773</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>2.787140</td>\n",
       "      <td>2.139604</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>2.625000</td>\n",
       "      <td>2.160247</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>9.523810</td>\n",
       "      <td>7.089538</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>11.300000</td>\n",
       "      <td>8.571116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>13.240000</td>\n",
       "      <td>10.140521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>23.163265</td>\n",
       "      <td>15.862216</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       WeeksToDeparture      std_wtd\n",
       "count       8902.000000  8902.000000\n",
       "mean          11.446469     8.617773\n",
       "std            2.787140     2.139604\n",
       "min            2.625000     2.160247\n",
       "25%            9.523810     7.089538\n",
       "50%           11.300000     8.571116\n",
       "75%           13.240000    10.140521\n",
       "max           23.163265    15.862216"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8902, 5)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10.99904767212102\n",
      "0.9938894125318564\n"
     ]
    }
   ],
   "source": [
    "print(y.mean())\n",
    "print(y.std())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing dates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting dates into numerical columns is a common operation when time series data is analyzed with non-parametric predictors. The code below makes the following transformations:\n",
    "\n",
    "- numerical columns for year (2011-2012), month (1-12), day of the month (1-31), day of the week (0-6), and week of the year (1-52)\n",
    "- number of days since 1970-01-01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make a copy of the original data to avoid writing on the original data\n",
    "X_encoded = X.copy()\n",
    "\n",
    "# following http://stackoverflow.com/questions/16453644/regression-with-date-variable-using-scikit-learn\n",
    "X_encoded['DateOfDeparture'] = pd.to_datetime(X_encoded['DateOfDeparture'])\n",
    "X_encoded['year'] = X_encoded['DateOfDeparture'].dt.year\n",
    "X_encoded['month'] = X_encoded['DateOfDeparture'].dt.month\n",
    "X_encoded['day'] = X_encoded['DateOfDeparture'].dt.day\n",
    "X_encoded['weekday'] = X_encoded['DateOfDeparture'].dt.weekday\n",
    "X_encoded['week'] = X_encoded['DateOfDeparture'].dt.week\n",
    "X_encoded['n_days'] = X_encoded['DateOfDeparture'].apply(lambda date: (date - pd.to_datetime(\"1970-01-01\")).days)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DateOfDeparture</th>\n",
       "      <th>Departure</th>\n",
       "      <th>Arrival</th>\n",
       "      <th>WeeksToDeparture</th>\n",
       "      <th>std_wtd</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>day</th>\n",
       "      <th>weekday</th>\n",
       "      <th>week</th>\n",
       "      <th>n_days</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8897</th>\n",
       "      <td>2011-10-02</td>\n",
       "      <td>DTW</td>\n",
       "      <td>ATL</td>\n",
       "      <td>9.263158</td>\n",
       "      <td>7.316967</td>\n",
       "      <td>2011</td>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>39</td>\n",
       "      <td>15249</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8898</th>\n",
       "      <td>2012-09-25</td>\n",
       "      <td>DFW</td>\n",
       "      <td>ORD</td>\n",
       "      <td>12.772727</td>\n",
       "      <td>10.641034</td>\n",
       "      <td>2012</td>\n",
       "      <td>9</td>\n",
       "      <td>25</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>15608</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8899</th>\n",
       "      <td>2012-01-19</td>\n",
       "      <td>SFO</td>\n",
       "      <td>LAS</td>\n",
       "      <td>11.047619</td>\n",
       "      <td>7.908705</td>\n",
       "      <td>2012</td>\n",
       "      <td>1</td>\n",
       "      <td>19</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>15358</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8900</th>\n",
       "      <td>2013-02-03</td>\n",
       "      <td>ORD</td>\n",
       "      <td>PHL</td>\n",
       "      <td>6.076923</td>\n",
       "      <td>4.030334</td>\n",
       "      <td>2013</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>15739</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8901</th>\n",
       "      <td>2011-11-26</td>\n",
       "      <td>DTW</td>\n",
       "      <td>ATL</td>\n",
       "      <td>9.526316</td>\n",
       "      <td>6.167733</td>\n",
       "      <td>2011</td>\n",
       "      <td>11</td>\n",
       "      <td>26</td>\n",
       "      <td>5</td>\n",
       "      <td>47</td>\n",
       "      <td>15304</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     DateOfDeparture Departure Arrival  WeeksToDeparture    std_wtd  year  \\\n",
       "8897      2011-10-02       DTW     ATL          9.263158   7.316967  2011   \n",
       "8898      2012-09-25       DFW     ORD         12.772727  10.641034  2012   \n",
       "8899      2012-01-19       SFO     LAS         11.047619   7.908705  2012   \n",
       "8900      2013-02-03       ORD     PHL          6.076923   4.030334  2013   \n",
       "8901      2011-11-26       DTW     ATL          9.526316   6.167733  2011   \n",
       "\n",
       "      month  day  weekday  week  n_days  \n",
       "8897     10    2        6    39   15249  \n",
       "8898      9   25        1    39   15608  \n",
       "8899      1   19        3     3   15358  \n",
       "8900      2    3        6     5   15739  \n",
       "8901     11   26        5    47   15304  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_encoded.tail(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will perform all preprocessing steps within a scikit-learn [pipeline](https://scikit-learn.org/stable/modules/compose.html) which chains together tranformation and estimator steps. This offers offers convenience and safety (help avoid leaking statistics from your test data into the trained model in cross-validation) and the whole pipeline can be evaluated with `cross_val_score`.\n",
    "\n",
    "To perform the above encoding within a scikit-learn [pipeline](https://scikit-learn.org/stable/modules/compose.html) we will a function and using `FunctionTransformer` to make it compatible with scikit-learn API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import FunctionTransformer\n",
    "\n",
    "def _encode_dates(X):\n",
    "    # With pandas < 1.0, we wil get a SettingWithCopyWarning\n",
    "    # In our case, we will avoid this warning by triggering a copy\n",
    "    # More information can be found at:\n",
    "    # https://github.com/scikit-learn/scikit-learn/issues/16191\n",
    "    X_encoded = X.copy()\n",
    "\n",
    "    # Make sure that DateOfDeparture is of datetime format\n",
    "    X_encoded.loc[:, 'DateOfDeparture'] = pd.to_datetime(X_encoded['DateOfDeparture'])\n",
    "    # Encode the DateOfDeparture\n",
    "    X_encoded.loc[:, 'year'] = X_encoded['DateOfDeparture'].dt.year\n",
    "    X_encoded.loc[:, 'month'] = X_encoded['DateOfDeparture'].dt.month\n",
    "    X_encoded.loc[:, 'day'] = X_encoded['DateOfDeparture'].dt.day\n",
    "    X_encoded.loc[:, 'weekday'] = X_encoded['DateOfDeparture'].dt.weekday\n",
    "    X_encoded.loc[:, 'week'] = X_encoded['DateOfDeparture'].dt.week\n",
    "    X_encoded.loc[:, 'n_days'] = X_encoded['DateOfDeparture'].apply(\n",
    "        lambda date: (date - pd.to_datetime(\"1970-01-01\")).days\n",
    "    )\n",
    "    # Once we did the encoding, we will not need DateOfDeparture\n",
    "    return X_encoded.drop(columns=[\"DateOfDeparture\"])\n",
    "\n",
    "date_encoder = FunctionTransformer(_encode_dates)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Departure</th>\n",
       "      <th>Arrival</th>\n",
       "      <th>WeeksToDeparture</th>\n",
       "      <th>std_wtd</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>day</th>\n",
       "      <th>weekday</th>\n",
       "      <th>week</th>\n",
       "      <th>n_days</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ORD</td>\n",
       "      <td>DFW</td>\n",
       "      <td>12.875000</td>\n",
       "      <td>9.812647</td>\n",
       "      <td>2012</td>\n",
       "      <td>6</td>\n",
       "      <td>19</td>\n",
       "      <td>1</td>\n",
       "      <td>25</td>\n",
       "      <td>15510</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>LAS</td>\n",
       "      <td>DEN</td>\n",
       "      <td>14.285714</td>\n",
       "      <td>9.466734</td>\n",
       "      <td>2012</td>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>37</td>\n",
       "      <td>15593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>DEN</td>\n",
       "      <td>LAX</td>\n",
       "      <td>10.863636</td>\n",
       "      <td>9.035883</td>\n",
       "      <td>2012</td>\n",
       "      <td>10</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>40</td>\n",
       "      <td>15618</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ATL</td>\n",
       "      <td>ORD</td>\n",
       "      <td>11.480000</td>\n",
       "      <td>7.990202</td>\n",
       "      <td>2011</td>\n",
       "      <td>10</td>\n",
       "      <td>9</td>\n",
       "      <td>6</td>\n",
       "      <td>40</td>\n",
       "      <td>15256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>DEN</td>\n",
       "      <td>SFO</td>\n",
       "      <td>11.450000</td>\n",
       "      <td>9.517159</td>\n",
       "      <td>2012</td>\n",
       "      <td>2</td>\n",
       "      <td>21</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>15391</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Departure Arrival  WeeksToDeparture   std_wtd  year  month  day  weekday  \\\n",
       "0       ORD     DFW         12.875000  9.812647  2012      6   19        1   \n",
       "1       LAS     DEN         14.285714  9.466734  2012      9   10        0   \n",
       "2       DEN     LAX         10.863636  9.035883  2012     10    5        4   \n",
       "3       ATL     ORD         11.480000  7.990202  2011     10    9        6   \n",
       "4       DEN     SFO         11.450000  9.517159  2012      2   21        1   \n",
       "\n",
       "   week  n_days  \n",
       "0    25   15510  \n",
       "1    37   15593  \n",
       "2    40   15618  \n",
       "3    40   15256  \n",
       "4     8   15391  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "date_encoder.fit_transform(X).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random Forests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tree-based algorithms requires less complex preprocessing than linear-models. We will first present a machine-learning pipeline where we will use a random forest. In this pipeline, we will need to:\n",
    "\n",
    "- encode the date to numerical values (as presented in the section above);\n",
    "- oridinal encode the other categorical values to get numerical number;\n",
    "- keep numerical features as they are.\n",
    "\n",
    "Thus, we want to perform three different processes on different columns of the original data `X`. In scikit-learn, we can use [`make_column_transformer`](https://scikit-learn.org/stable/modules/generated/sklearn.compose.make_column_transformer.html) to perform such processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OrdinalEncoder\n",
    "from sklearn.compose import make_column_transformer\n",
    "\n",
    "date_encoder = FunctionTransformer(_encode_dates)\n",
    "date_cols = [\"DateOfDeparture\"]\n",
    "\n",
    "categorical_encoder = OrdinalEncoder()\n",
    "categorical_cols = [\"Arrival\", \"Departure\"]\n",
    "\n",
    "preprocessor = make_column_transformer(\n",
    "    (date_encoder, date_cols),\n",
    "    (categorical_encoder, categorical_cols),\n",
    "    remainder='passthrough',  # passthrough numerical columns as they are\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can combine our preprocessor with an estimator (`RandomForestRegressor` in this case), allowing us to make predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.pipeline import make_pipeline\n",
    "\n",
    "n_estimators = 10\n",
    "max_depth = 10\n",
    "max_features = 10\n",
    "\n",
    "regressor = RandomForestRegressor(\n",
    "    n_estimators=n_estimators, max_depth=max_depth, max_features=max_features\n",
    ")\n",
    "\n",
    "pipeline = make_pipeline(preprocessor, regressor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can cross-validate our `pipeline` using `cross_val_score`. Below we will have specified `cv=5` meaning KFold cross-valdiation splitting will be used, with 8 folds. The mean squared error regression loss is calculated for each split. The output `score` will be an array of 5 scores from each KFold. The score mean and standard deviation of the 5 scores is printed at the end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 0.6277 +/- 0.0186\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "scores = cross_val_score(\n",
    "    pipeline, X, y, cv=5, scoring='neg_mean_squared_error'\n",
    ")\n",
    "rmse_scores = np.sqrt(-scores)\n",
    "\n",
    "print(\n",
    "    f\"RMSE: {np.mean(rmse_scores):.4f} +/- {np.std(rmse_scores):.4f}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear regressor\n",
    "\n",
    "When dealing with a linear model, we need to one-hot encode categorical variables instead of ordinal encoding and standardize numerical variables. Thus we will:\n",
    "\n",
    "- encode the date;\n",
    "- then, one-hot encode all categorical columns, including the encoded date as well;\n",
    "- standardize the numerical columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "date_encoder = FunctionTransformer(_encode_dates)\n",
    "date_cols = [\"DateOfDeparture\"]\n",
    "\n",
    "categorical_encoder = OneHotEncoder(handle_unknown=\"ignore\")\n",
    "categorical_cols = [\n",
    "    \"Arrival\", \"Departure\", \"year\", \"month\", \"day\",\n",
    "    \"weekday\", \"week\", \"n_days\"\n",
    "]\n",
    "\n",
    "numerical_scaler = StandardScaler()\n",
    "numerical_cols = [\"WeeksToDeparture\", \"std_wtd\"]\n",
    "\n",
    "preprocessor = make_column_transformer(\n",
    "    (categorical_encoder, categorical_cols),\n",
    "    (numerical_scaler, numerical_cols)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now combine our `preprocessor` with the `LinearRegression` estimator in a `Pipeline`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "regressor = LinearRegression()\n",
    "\n",
    "pipeline = make_pipeline(date_encoder, preprocessor, regressor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can evaluate our linear-model pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 0.6117 +/- 0.0149\n"
     ]
    }
   ],
   "source": [
    "scores = cross_val_score(\n",
    "    pipeline, X, y, cv=5, scoring='neg_mean_squared_error'\n",
    ")\n",
    "rmse_scores = np.sqrt(-scores)\n",
    "\n",
    "print(\n",
    "    f\"RMSE: {np.mean(rmse_scores):.4f} +/- {np.std(rmse_scores):.4f}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Merging external data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The objective in this RAMP data challenge is to find good data that can be correlated to flight traffic. We will use some weather data (saved in `submissions/starting_kit`) to provide an example of how to merge external data in a scikit-learn pipeline.\n",
    "\n",
    "Your external data will need to be included in your submissions folder - see [RAMP submissions](#RAMP-submissions) for more details.\n",
    "\n",
    "First we will define a function that merges the external data to our feature data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'submissions/starting_kit/external_data.csv'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# when submitting a kit, the `__file__` variable will corresponds to the\n",
    "# path to `estimator.py`. However, this variable is not defined in the\n",
    "# notebook and thus we must define the `__file__` variable to imitate\n",
    "# how a submission `.py` would work.\n",
    "__file__ = os.path.join('submissions', 'starting_kit', 'estimator.py')\n",
    "filepath = os.path.join(os.path.dirname(__file__), 'external_data.csv')\n",
    "filepath"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>AirPort</th>\n",
       "      <th>Max TemperatureC</th>\n",
       "      <th>Mean TemperatureC</th>\n",
       "      <th>Min TemperatureC</th>\n",
       "      <th>Dew PointC</th>\n",
       "      <th>MeanDew PointC</th>\n",
       "      <th>Min DewpointC</th>\n",
       "      <th>Max Humidity</th>\n",
       "      <th>Mean Humidity</th>\n",
       "      <th>Min Humidity</th>\n",
       "      <th>Max Sea Level PressurehPa</th>\n",
       "      <th>Mean Sea Level PressurehPa</th>\n",
       "      <th>Min Sea Level PressurehPa</th>\n",
       "      <th>Max VisibilityKm</th>\n",
       "      <th>Mean VisibilityKm</th>\n",
       "      <th>Min VisibilitykM</th>\n",
       "      <th>Max Wind SpeedKm/h</th>\n",
       "      <th>Mean Wind SpeedKm/h</th>\n",
       "      <th>Max Gust SpeedKm/h</th>\n",
       "      <th>Precipitationmm</th>\n",
       "      <th>CloudCover</th>\n",
       "      <th>Events</th>\n",
       "      <th>WindDirDegrees</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011-09-01</td>\n",
       "      <td>ATL</td>\n",
       "      <td>35</td>\n",
       "      <td>29</td>\n",
       "      <td>24</td>\n",
       "      <td>21</td>\n",
       "      <td>18</td>\n",
       "      <td>14</td>\n",
       "      <td>79</td>\n",
       "      <td>56</td>\n",
       "      <td>32</td>\n",
       "      <td>1022</td>\n",
       "      <td>1019</td>\n",
       "      <td>1017</td>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "      <td>11</td>\n",
       "      <td>19</td>\n",
       "      <td>6</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>129</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011-09-02</td>\n",
       "      <td>ATL</td>\n",
       "      <td>36</td>\n",
       "      <td>29</td>\n",
       "      <td>22</td>\n",
       "      <td>17</td>\n",
       "      <td>15</td>\n",
       "      <td>14</td>\n",
       "      <td>61</td>\n",
       "      <td>46</td>\n",
       "      <td>30</td>\n",
       "      <td>1019</td>\n",
       "      <td>1016</td>\n",
       "      <td>1014</td>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "      <td>24</td>\n",
       "      <td>7</td>\n",
       "      <td>34.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011-09-03</td>\n",
       "      <td>ATL</td>\n",
       "      <td>35</td>\n",
       "      <td>29</td>\n",
       "      <td>23</td>\n",
       "      <td>17</td>\n",
       "      <td>16</td>\n",
       "      <td>14</td>\n",
       "      <td>64</td>\n",
       "      <td>47</td>\n",
       "      <td>30</td>\n",
       "      <td>1015</td>\n",
       "      <td>1013</td>\n",
       "      <td>1011</td>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "      <td>19</td>\n",
       "      <td>7</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>147</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011-09-04</td>\n",
       "      <td>ATL</td>\n",
       "      <td>27</td>\n",
       "      <td>24</td>\n",
       "      <td>22</td>\n",
       "      <td>22</td>\n",
       "      <td>19</td>\n",
       "      <td>16</td>\n",
       "      <td>93</td>\n",
       "      <td>72</td>\n",
       "      <td>51</td>\n",
       "      <td>1014</td>\n",
       "      <td>1012</td>\n",
       "      <td>1011</td>\n",
       "      <td>16</td>\n",
       "      <td>14</td>\n",
       "      <td>4</td>\n",
       "      <td>21</td>\n",
       "      <td>9</td>\n",
       "      <td>26.0</td>\n",
       "      <td>6.10</td>\n",
       "      <td>6</td>\n",
       "      <td>Rain</td>\n",
       "      <td>139</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2011-09-05</td>\n",
       "      <td>ATL</td>\n",
       "      <td>26</td>\n",
       "      <td>24</td>\n",
       "      <td>22</td>\n",
       "      <td>23</td>\n",
       "      <td>22</td>\n",
       "      <td>20</td>\n",
       "      <td>94</td>\n",
       "      <td>91</td>\n",
       "      <td>87</td>\n",
       "      <td>1010</td>\n",
       "      <td>1005</td>\n",
       "      <td>999</td>\n",
       "      <td>16</td>\n",
       "      <td>13</td>\n",
       "      <td>3</td>\n",
       "      <td>32</td>\n",
       "      <td>16</td>\n",
       "      <td>45.0</td>\n",
       "      <td>16.00</td>\n",
       "      <td>8</td>\n",
       "      <td>Rain-Thunderstorm</td>\n",
       "      <td>149</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Date AirPort  Max TemperatureC  Mean TemperatureC  Min TemperatureC  \\\n",
       "0  2011-09-01     ATL                35                 29                24   \n",
       "1  2011-09-02     ATL                36                 29                22   \n",
       "2  2011-09-03     ATL                35                 29                23   \n",
       "3  2011-09-04     ATL                27                 24                22   \n",
       "4  2011-09-05     ATL                26                 24                22   \n",
       "\n",
       "   Dew PointC  MeanDew PointC  Min DewpointC  Max Humidity  Mean Humidity  \\\n",
       "0          21              18             14            79             56   \n",
       "1          17              15             14            61             46   \n",
       "2          17              16             14            64             47   \n",
       "3          22              19             16            93             72   \n",
       "4          23              22             20            94             91   \n",
       "\n",
       "   Min Humidity  Max Sea Level PressurehPa  Mean Sea Level PressurehPa  \\\n",
       "0            32                       1022                        1019   \n",
       "1            30                       1019                        1016   \n",
       "2            30                       1015                        1013   \n",
       "3            51                       1014                        1012   \n",
       "4            87                       1010                        1005   \n",
       "\n",
       "   Min Sea Level PressurehPa  Max VisibilityKm  Mean VisibilityKm  \\\n",
       "0                       1017                16                 16   \n",
       "1                       1014                16                 16   \n",
       "2                       1011                16                 16   \n",
       "3                       1011                16                 14   \n",
       "4                        999                16                 13   \n",
       "\n",
       "   Min VisibilitykM  Max Wind SpeedKm/h  Mean Wind SpeedKm/h  \\\n",
       "0                11                  19                    6   \n",
       "1                16                  24                    7   \n",
       "2                16                  19                    7   \n",
       "3                 4                  21                    9   \n",
       "4                 3                  32                   16   \n",
       "\n",
       "   Max Gust SpeedKm/h Precipitationmm  CloudCover             Events  \\\n",
       "0                26.0            0.00           3                NaN   \n",
       "1                34.0            0.00           2                NaN   \n",
       "2                26.0            0.00           4                NaN   \n",
       "3                26.0            6.10           6               Rain   \n",
       "4                45.0           16.00           8  Rain-Thunderstorm   \n",
       "\n",
       "   WindDirDegrees  \n",
       "0             129  \n",
       "1             185  \n",
       "2             147  \n",
       "3             139  \n",
       "4             149  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(filepath).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def _merge_external_data(X):\n",
    "    filepath = os.path.join(\n",
    "        os.path.dirname(__file__), 'external_data.csv'\n",
    "    )\n",
    "    \n",
    "    X = X.copy()  # to avoid raising SettingOnCopyWarning\n",
    "    # Make sure that DateOfDeparture is of dtype datetime\n",
    "    X.loc[:, \"DateOfDeparture\"] = pd.to_datetime(X['DateOfDeparture'])\n",
    "    # Parse date to also be of dtype datetime\n",
    "    data_weather = pd.read_csv(filepath, parse_dates=[\"Date\"])\n",
    "\n",
    "    X_weather = data_weather[['Date', 'AirPort', 'Max TemperatureC']]\n",
    "    X_weather = X_weather.rename(\n",
    "        columns={'Date': 'DateOfDeparture', 'AirPort': 'Arrival'})\n",
    "    X_merged = pd.merge(\n",
    "        X, X_weather, how='left', on=['DateOfDeparture', 'Arrival'], sort=False\n",
    "    )\n",
    "    return X_merged\n",
    "\n",
    "data_merger = FunctionTransformer(_merge_external_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double check that our function works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DateOfDeparture</th>\n",
       "      <th>Departure</th>\n",
       "      <th>Arrival</th>\n",
       "      <th>WeeksToDeparture</th>\n",
       "      <th>std_wtd</th>\n",
       "      <th>Max TemperatureC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2012-06-19</td>\n",
       "      <td>ORD</td>\n",
       "      <td>DFW</td>\n",
       "      <td>12.875000</td>\n",
       "      <td>9.812647</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2012-09-10</td>\n",
       "      <td>LAS</td>\n",
       "      <td>DEN</td>\n",
       "      <td>14.285714</td>\n",
       "      <td>9.466734</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2012-10-05</td>\n",
       "      <td>DEN</td>\n",
       "      <td>LAX</td>\n",
       "      <td>10.863636</td>\n",
       "      <td>9.035883</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011-10-09</td>\n",
       "      <td>ATL</td>\n",
       "      <td>ORD</td>\n",
       "      <td>11.480000</td>\n",
       "      <td>7.990202</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2012-02-21</td>\n",
       "      <td>DEN</td>\n",
       "      <td>SFO</td>\n",
       "      <td>11.450000</td>\n",
       "      <td>9.517159</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  DateOfDeparture Departure Arrival  WeeksToDeparture   std_wtd  \\\n",
       "0      2012-06-19       ORD     DFW         12.875000  9.812647   \n",
       "1      2012-09-10       LAS     DEN         14.285714  9.466734   \n",
       "2      2012-10-05       DEN     LAX         10.863636  9.035883   \n",
       "3      2011-10-09       ATL     ORD         11.480000  7.990202   \n",
       "4      2012-02-21       DEN     SFO         11.450000  9.517159   \n",
       "\n",
       "   Max TemperatureC  \n",
       "0                34  \n",
       "1                33  \n",
       "2                22  \n",
       "3                27  \n",
       "4                16  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_merger.fit_transform(X).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use `FunctionTransformer` to make our function compatible with scikit-learn API:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now assemble our pipeline using the same `data_merger` and `preprocessor` as above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OrdinalEncoder\n",
    "from sklearn.compose import make_column_transformer\n",
    "\n",
    "date_encoder = FunctionTransformer(_encode_dates)\n",
    "date_cols = [\"DateOfDeparture\"]\n",
    "\n",
    "categorical_encoder = OrdinalEncoder()\n",
    "categorical_cols = [\"Arrival\", \"Departure\"]\n",
    "\n",
    "preprocessor = make_column_transformer(\n",
    "    (date_encoder, date_cols),\n",
    "    (categorical_encoder, categorical_cols),\n",
    "    remainder='passthrough',  # passthrough numerical columns as they are\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_estimators = 10\n",
    "max_depth = 10\n",
    "max_features = 10\n",
    "\n",
    "regressor = RandomForestRegressor(\n",
    "    n_estimators=n_estimators, max_depth=max_depth, max_features=max_features\n",
    ")\n",
    "\n",
    "pipeline = make_pipeline(data_merger, preprocessor, regressor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 0.6330 +/- 0.0108\n"
     ]
    }
   ],
   "source": [
    "scores = cross_val_score(\n",
    "    pipeline, X, y, cv=5, scoring='neg_mean_squared_error'\n",
    ")\n",
    "rmse_scores = np.sqrt(-scores)\n",
    "\n",
    "print(\n",
    "    f\"RMSE: {np.mean(rmse_scores):.4f} +/- {np.std(rmse_scores):.4f}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature importances\n",
    "\n",
    "We can check the feature importances using the function [`sklearn.inspection.permutation_importances`](https://scikit-learn.org/stable/modules/generated/sklearn.inspection.permutation_importance.html). Since the first step of our pipeline adds the new external feature `Max TemperatureC`, we want to apply this transformation after adding `Max TemperatureC`, to check the importances of all features. Indeed, we can perform `sklearn.inspection.permutation_importances` at any stage of the pipeline, as we will see later on.\n",
    "\n",
    "\n",
    "The code below:\n",
    "\n",
    "* performs `transform` on the first step of the pipeline (`pipeline[0]`) producing the transformed train (`X_train_augmented`) and test (`X_test_augmented`) data\n",
    "* the transformed data is used to fit the pipeline from the second step onwards\n",
    "\n",
    "Note that pipelines can be sliced. `pipeline[0]` obtains the first step (tuple) of the pipeline. You can further slice to obtain either the transformer/estimator (first item in each tuple) or column list (second item within each tuple) inside each tuple. For example `pipeline[0][0]` obtains the transformer of the first step of the pipeline (first item of the first tuple)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5939918926708109"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, random_state=42\n",
    ")\n",
    "\n",
    "merger = pipeline[0]\n",
    "X_train_augmented = merger.transform(X_train)\n",
    "X_test_augmented = merger.transform(X_test)\n",
    "\n",
    "predictor = pipeline[1:]\n",
    "predictor.fit(X_train_augmented, y_train).score(X_test_augmented, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the fitted pipeline, we can now use `permutation_importance` to calculate feature importances:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.inspection import permutation_importance\n",
    "\n",
    "feature_importances = permutation_importance(\n",
    "    predictor, X_train_augmented, y_train, n_repeats=10\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we plot the permutation importance using the training set. The higher the value, more important the feature is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 12228 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sorted_idx = feature_importances.importances_mean.argsort()\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "ax.boxplot(feature_importances.importances[sorted_idx].T,\n",
    "           vert=False, labels=X_train_augmented.columns[sorted_idx])\n",
    "ax.set_title(\"Permutation Importances (train set)\")\n",
    "fig.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can replicate the same processing on the test set and see if we can observe the same trend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.inspection import permutation_importance\n",
    "\n",
    "feature_importances = permutation_importance(\n",
    "    predictor, X_test_augmented, y_test, n_repeats=10\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 12452 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sorted_idx = feature_importances.importances_mean.argsort()\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "ax.boxplot(feature_importances.importances[sorted_idx].T,\n",
    "           vert=False, labels=X_test_augmented.columns[sorted_idx])\n",
    "ax.set_title(\"Permutation Importances (test set)\")\n",
    "fig.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the current version of scikit-learn, it is not handy but still possible to check the feature importances at the latest stage of the pipeline (once all features have been preprocessed).\n",
    "\n",
    "The difficult part is to get the name of the features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocessor = pipeline[:-1]\n",
    "predictor = pipeline[-1]\n",
    "\n",
    "X_train_augmented = preprocessor.transform(X_train)\n",
    "X_test_augmented = preprocessor.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's find out the feature names (in the future, scikit-learn will provide a `get_feature_names` function to handle this case)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "date_cols_name = (date_encoder.transform(X_train[date_cols])\n",
    "                              .columns.tolist())\n",
    "categorical_cols_name = categorical_cols\n",
    "numerical_cols_name = (pipeline[0].transform(X_train)\n",
    "                                  .columns[pipeline[1].transformers_[-1][-1]]\n",
    "                                  .tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['year', 'month', 'day', 'weekday', 'week', 'n_days', 'Arrival',\n",
       "       'Departure', 'WeeksToDeparture', 'std_wtd', 'Max TemperatureC'],\n",
       "      dtype='<U16')"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_names = np.array(\n",
    "    date_cols_name + categorical_cols_name + numerical_cols_name\n",
    ")\n",
    "feature_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can repeat the previous processing at this finer grain, where the transformed date columns are included."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.inspection import permutation_importance\n",
    "\n",
    "feature_importances = permutation_importance(\n",
    "    predictor, X_train_augmented, y_train, n_repeats=10\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we plot the permutation importance using the training set. Basically, higher the value, more important is the feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 15524 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sorted_idx = feature_importances.importances_mean.argsort()\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "ax.boxplot(feature_importances.importances[sorted_idx].T,\n",
    "           vert=False, labels=feature_names[sorted_idx])\n",
    "ax.set_title(\"Permutation Importances (train set)\")\n",
    "fig.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can replicate the same processing on the test set and see if we can observe the same trend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.inspection import permutation_importance\n",
    "\n",
    "feature_importances = permutation_importance(\n",
    "    predictor, X_test_augmented, y_test, n_repeats=10\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 15216 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sorted_idx = feature_importances.importances_mean.argsort()\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "ax.boxplot(feature_importances.importances[sorted_idx].T,\n",
    "           vert=False, labels=feature_names[sorted_idx])\n",
    "ax.set_title(\"Permutation Importances (test set)\")\n",
    "fig.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submission\n",
    "\n",
    "To submit your code, you can refer to the [online documentation](https://paris-saclay-cds.github.io/ramp-docs/ramp-workflow/stable/using_kits.html)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}