{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predict equipment failure\n",
    "\n",
    "According to a study by [Aberdeen](https://www.aberdeen.com/blogposts/stat-of-the-week-the-rising-cost-of-downtime/), unplanned equipment failure can cost up to $260,000 an hour. They also introduce unintended consequences such as downtime, health and safety risks, and inventory damages.\n",
    "\n",
    "Using AI to automate the predictive maintenance process allows operators to proactively identify subtle or unknown issues with equipment operation in the collected sensor data and schedule maintenance when maintenance is truly needed, or automatically receive notifications to intervene when a sudden failure is imminent. DataRobot models can empower your maintenance staff by providing the information they need to keep operations running without any disruption.\n",
    "\n",
    "Before proceeding, review the [API quickstart guide](https://docs.datarobot.com/en/docs/api/api-quickstart/index.html) to ensure that your client and API credentials have been properly configured.\n",
    "\n",
    "To run this notebook on your own, click the download icon in the top right of the page."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": "auto"
   },
   "outputs": [],
   "source": [
    "!pip install datarobot umap-learn nbformat openpyxl hdbscan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "autoscroll": "auto"
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import datarobot as dr\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as mtick\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "light_blue = \"#598fd6\"\n",
    "grey_blue = \"#5f728b\"\n",
    "orange = \"#dd6b3d\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connect to DataRobot\n",
    "\n",
    "Read more about different options for [connecting to DataRobot from the client](https://docs.datarobot.com/en/docs/api/api-quickstart/api-qs.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If the config file is not in the default location described in the API Quickstart guide, '~/.config/datarobot/drconfig.yaml', then you will need to call\n",
    "# dr.Client(config_path='path-to-drconfig.yaml')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import data\n",
    "\n",
    "DataRobot hosts the dataset used in this notebook via the URL used in the snippet below. Read the data directly from the URL into a [pandas DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) and display the results to verify all of the data looks correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "autoscroll": "auto"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lifetime</th>\n",
       "      <th>broken</th>\n",
       "      <th>pressureInd</th>\n",
       "      <th>moistureInd</th>\n",
       "      <th>temperatureInd</th>\n",
       "      <th>team</th>\n",
       "      <th>provider</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>56</td>\n",
       "      <td>0</td>\n",
       "      <td>92.178854</td>\n",
       "      <td>104.230204</td>\n",
       "      <td>96.517159</td>\n",
       "      <td>TeamA</td>\n",
       "      <td>Provider4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>81</td>\n",
       "      <td>1</td>\n",
       "      <td>72.075938</td>\n",
       "      <td>103.065701</td>\n",
       "      <td>87.271062</td>\n",
       "      <td>TeamC</td>\n",
       "      <td>Provider4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>96.272254</td>\n",
       "      <td>77.801376</td>\n",
       "      <td>112.196170</td>\n",
       "      <td>TeamA</td>\n",
       "      <td>Provider1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>86</td>\n",
       "      <td>1</td>\n",
       "      <td>94.406461</td>\n",
       "      <td>108.493608</td>\n",
       "      <td>72.025374</td>\n",
       "      <td>TeamC</td>\n",
       "      <td>Provider2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>34</td>\n",
       "      <td>0</td>\n",
       "      <td>97.752899</td>\n",
       "      <td>99.413492</td>\n",
       "      <td>103.756271</td>\n",
       "      <td>TeamB</td>\n",
       "      <td>Provider1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>92.453609</td>\n",
       "      <td>92.495352</td>\n",
       "      <td>140.160091</td>\n",
       "      <td>TeamB</td>\n",
       "      <td>Provider2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>112.047415</td>\n",
       "      <td>89.849594</td>\n",
       "      <td>81.100953</td>\n",
       "      <td>TeamB</td>\n",
       "      <td>Provider2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>65</td>\n",
       "      <td>1</td>\n",
       "      <td>125.032692</td>\n",
       "      <td>101.163482</td>\n",
       "      <td>126.024077</td>\n",
       "      <td>TeamB</td>\n",
       "      <td>Provider3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>80</td>\n",
       "      <td>1</td>\n",
       "      <td>96.008785</td>\n",
       "      <td>84.587554</td>\n",
       "      <td>95.287253</td>\n",
       "      <td>TeamA</td>\n",
       "      <td>Provider1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>93</td>\n",
       "      <td>1</td>\n",
       "      <td>109.671138</td>\n",
       "      <td>104.600332</td>\n",
       "      <td>102.881957</td>\n",
       "      <td>TeamB</td>\n",
       "      <td>Provider2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    lifetime  broken  pressureInd  moistureInd  temperatureInd   team  \\\n",
       "0         56       0    92.178854   104.230204       96.517159  TeamA   \n",
       "1         81       1    72.075938   103.065701       87.271062  TeamC   \n",
       "2         60       0    96.272254    77.801376      112.196170  TeamA   \n",
       "3         86       1    94.406461   108.493608       72.025374  TeamC   \n",
       "4         34       0    97.752899    99.413492      103.756271  TeamB   \n",
       "..       ...     ...          ...          ...             ...    ...   \n",
       "95        41       0    92.453609    92.495352      140.160091  TeamB   \n",
       "96        12       0   112.047415    89.849594       81.100953  TeamB   \n",
       "97        65       1   125.032692   101.163482      126.024077  TeamB   \n",
       "98        80       1    96.008785    84.587554       95.287253  TeamA   \n",
       "99        93       1   109.671138   104.600332      102.881957  TeamB   \n",
       "\n",
       "     provider  \n",
       "0   Provider4  \n",
       "1   Provider4  \n",
       "2   Provider1  \n",
       "3   Provider2  \n",
       "4   Provider1  \n",
       "..        ...  \n",
       "95  Provider2  \n",
       "96  Provider2  \n",
       "97  Provider3  \n",
       "98  Provider1  \n",
       "99  Provider2  \n",
       "\n",
       "[100 rows x 7 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_path = \"https://s3.amazonaws.com/datarobot-use-case-datasets/maintenance_training.csv\"\n",
    "\n",
    "pathfinder_df = pd.read_csv(data_path, encoding=\"ISO-8859-1\")\n",
    "\n",
    "pathfinder_df.head(100)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize data\n",
    "\n",
    "Below, view several examples of charts that visualize the dataset in different ways, such as grouping by provider and part condition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "autoscroll": "auto"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lifetime</th>\n",
       "      <th>broken</th>\n",
       "      <th>pressureInd</th>\n",
       "      <th>moistureInd</th>\n",
       "      <th>temperatureInd</th>\n",
       "      <th>team</th>\n",
       "      <th>provider</th>\n",
       "      <th>broken_str</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>56</td>\n",
       "      <td>0</td>\n",
       "      <td>92.178854</td>\n",
       "      <td>104.230204</td>\n",
       "      <td>96.517159</td>\n",
       "      <td>TeamA</td>\n",
       "      <td>Provider4</td>\n",
       "      <td>Not Broken</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>81</td>\n",
       "      <td>1</td>\n",
       "      <td>72.075938</td>\n",
       "      <td>103.065701</td>\n",
       "      <td>87.271062</td>\n",
       "      <td>TeamC</td>\n",
       "      <td>Provider4</td>\n",
       "      <td>Broken</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>96.272254</td>\n",
       "      <td>77.801376</td>\n",
       "      <td>112.196170</td>\n",
       "      <td>TeamA</td>\n",
       "      <td>Provider1</td>\n",
       "      <td>Not Broken</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>86</td>\n",
       "      <td>1</td>\n",
       "      <td>94.406461</td>\n",
       "      <td>108.493608</td>\n",
       "      <td>72.025374</td>\n",
       "      <td>TeamC</td>\n",
       "      <td>Provider2</td>\n",
       "      <td>Broken</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>34</td>\n",
       "      <td>0</td>\n",
       "      <td>97.752899</td>\n",
       "      <td>99.413492</td>\n",
       "      <td>103.756271</td>\n",
       "      <td>TeamB</td>\n",
       "      <td>Provider1</td>\n",
       "      <td>Not Broken</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>907</th>\n",
       "      <td>88</td>\n",
       "      <td>1</td>\n",
       "      <td>88.589759</td>\n",
       "      <td>112.167556</td>\n",
       "      <td>99.861456</td>\n",
       "      <td>TeamB</td>\n",
       "      <td>Provider4</td>\n",
       "      <td>Broken</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>908</th>\n",
       "      <td>88</td>\n",
       "      <td>1</td>\n",
       "      <td>116.727075</td>\n",
       "      <td>110.871332</td>\n",
       "      <td>95.075631</td>\n",
       "      <td>TeamA</td>\n",
       "      <td>Provider4</td>\n",
       "      <td>Broken</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>909</th>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>104.026778</td>\n",
       "      <td>88.212873</td>\n",
       "      <td>83.221220</td>\n",
       "      <td>TeamB</td>\n",
       "      <td>Provider1</td>\n",
       "      <td>Not Broken</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>910</th>\n",
       "      <td>78</td>\n",
       "      <td>0</td>\n",
       "      <td>104.911649</td>\n",
       "      <td>104.257296</td>\n",
       "      <td>83.421491</td>\n",
       "      <td>TeamA</td>\n",
       "      <td>Provider4</td>\n",
       "      <td>Not Broken</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>911</th>\n",
       "      <td>63</td>\n",
       "      <td>0</td>\n",
       "      <td>116.901354</td>\n",
       "      <td>99.998694</td>\n",
       "      <td>47.641493</td>\n",
       "      <td>TeamB</td>\n",
       "      <td>Provider1</td>\n",
       "      <td>Not Broken</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>912 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     lifetime  broken  pressureInd  moistureInd  temperatureInd   team  \\\n",
       "0          56       0    92.178854   104.230204       96.517159  TeamA   \n",
       "1          81       1    72.075938   103.065701       87.271062  TeamC   \n",
       "2          60       0    96.272254    77.801376      112.196170  TeamA   \n",
       "3          86       1    94.406461   108.493608       72.025374  TeamC   \n",
       "4          34       0    97.752899    99.413492      103.756271  TeamB   \n",
       "..        ...     ...          ...          ...             ...    ...   \n",
       "907        88       1    88.589759   112.167556       99.861456  TeamB   \n",
       "908        88       1   116.727075   110.871332       95.075631  TeamA   \n",
       "909        22       0   104.026778    88.212873       83.221220  TeamB   \n",
       "910        78       0   104.911649   104.257296       83.421491  TeamA   \n",
       "911        63       0   116.901354    99.998694       47.641493  TeamB   \n",
       "\n",
       "      provider  broken_str  \n",
       "0    Provider4  Not Broken  \n",
       "1    Provider4      Broken  \n",
       "2    Provider1  Not Broken  \n",
       "3    Provider2      Broken  \n",
       "4    Provider1  Not Broken  \n",
       "..         ...         ...  \n",
       "907  Provider4      Broken  \n",
       "908  Provider4      Broken  \n",
       "909  Provider1  Not Broken  \n",
       "910  Provider4  Not Broken  \n",
       "911  Provider1  Not Broken  \n",
       "\n",
       "[912 rows x 8 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pathfinder_df[\"broken_str\"] = pathfinder_df[\"broken\"].replace(0, \"Not Broken\").replace(1, \"Broken\")\n",
    "\n",
    "pathfinder_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "autoscroll": "auto"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>provider</th>\n",
       "      <th>broken</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Provider1</td>\n",
       "      <td>0.446903</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Provider2</td>\n",
       "      <td>0.358025</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Provider3</td>\n",
       "      <td>0.479452</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Provider4</td>\n",
       "      <td>0.334821</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    provider    broken\n",
       "0  Provider1  0.446903\n",
       "1  Provider2  0.358025\n",
       "2  Provider3  0.479452\n",
       "3  Provider4  0.334821"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1 = pathfinder_df.where(pathfinder_df[\"broken\"] == 1).groupby(\"provider\").agg({\"broken\": \"count\"})\n",
    "df2 = pathfinder_df.where(pathfinder_df[\"broken\"] == 0).groupby(\"provider\").agg({\"broken\": \"count\"})\n",
    "\n",
    "df_perc = df1 / (df2 + df1)\n",
    "df_perc = df_perc.reset_index()\n",
    "\n",
    "df_perc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "autoscroll": "auto"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>team</th>\n",
       "      <th>broken</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>TeamA</td>\n",
       "      <td>0.369637</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>TeamB</td>\n",
       "      <td>0.425532</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>TeamC</td>\n",
       "      <td>0.414286</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    team    broken\n",
       "0  TeamA  0.369637\n",
       "1  TeamB  0.425532\n",
       "2  TeamC  0.414286"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1 = pathfinder_df.where(pathfinder_df[\"broken\"] == 1).groupby(\"team\").agg({\"broken\": \"count\"})\n",
    "df2 = pathfinder_df.where(pathfinder_df[\"broken\"] == 0).groupby(\"team\").agg({\"broken\": \"count\"})\n",
    "\n",
    "df_perc = df1 / (df2 + df1)\n",
    "df_perc = df_perc.reset_index()\n",
    "\n",
    "df_perc"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modeling\n",
    " \n",
    "For this use case, create a DataRobot project and initiate modeling by running Autopilot in Quick mode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "autoscroll": "auto"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In progress: 0, queued: 0 (waited: 0s)\n"
     ]
    }
   ],
   "source": [
    "EXISTING_PROJECT_ID = None  # If you've already created a project, replace None with the ID here\n",
    "if EXISTING_PROJECT_ID is None:\n",
    "    # Create project and pass in data\n",
    "    project = dr.Project.create(sourcedata=pathfinder_df, project_name=\"Predict equipment failure\")\n",
    "\n",
    "    # Set the project target to the appropriate feature. Use the LogLoss metric to measure performance\n",
    "    project.analyze_and_model(target=\"broken\", worker_count=\"-1\")\n",
    "else:\n",
    "    # Fetch the existing project\n",
    "    project = dr.Project.get(EXISTING_PROJECT_ID)\n",
    "\n",
    "project.wait_for_autopilot(check_interval=30)\n",
    "\n",
    "# Get the project metric (i.e LogLoss, RMSE, etc.)\n",
    "metric = project.metric"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View project in UI\n",
    "\n",
    "If you want to view the project in the DataRobot UI, use the following snippet to retrieve the project's URL and use it to navigate to the application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "autoscroll": "auto"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://app.datarobot.com/projects/62f2c49067ba04bafdda8972/models\n"
     ]
    }
   ],
   "source": [
    "# Get project URL\n",
    "project_url = project.get_uri()\n",
    "print(project_url)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate model performance \n",
    "\n",
    "To measure model performance, first select the top-performing model based on a specific performance metric (i.e., `LogLoss`) and then evaluate several different types of charts, such as Lift Chart, ROC Curve, and Feature Importance. There are two helper functions that you need to build in order to simplify producing these model insights.\n",
    "\n",
    "You can reference more information about model evaluation tools [here](https://docs.datarobot.com/en/docs/modeling/analyze-models/evaluate/index.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "autoscroll": "auto"
   },
   "outputs": [],
   "source": [
    "def sorted_by_metric(models, test_set, metric):\n",
    "    models_with_score = [model for model in models if model.metrics[metric][test_set] is not None]\n",
    "\n",
    "    return sorted(models_with_score, key=lambda model: model.metrics[metric][test_set])\n",
    "\n",
    "\n",
    "# Note that this code only works for metrics where lower values indicate better performance\n",
    "# Reverse to correctly display for metrics where greater values indicate better performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "autoscroll": "auto"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The top performing model is Model('Elastic-Net Classifier (L2 / Binomial Deviance)') using metric, LogLoss\n"
     ]
    }
   ],
   "source": [
    "models = project.get_models()\n",
    "\n",
    "# Uncomment if this is not set above in the create project cell\n",
    "metric = project.metric\n",
    "\n",
    "# Get top performing model\n",
    "model_top = project.get_top_model(metric)\n",
    "\n",
    "print(\n",
    "    \"\"\"The top performing model is {model} using metric, {metric}\"\"\".format(\n",
    "        model=str(model_top), metric=metric\n",
    "    )\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lift chart\n",
    "\n",
    "A [lift chart](https://docs.datarobot.com/en/docs/modeling/analyze-models/evaluate/lift-chart.html#lift-chart) shows you how close model predictions are to the actual values of the target in the training data. The lift chart data includes the average predicted value and the average actual values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set styling\n",
    "dr_dark_blue = \"#08233F\"\n",
    "dr_blue = \"#1F77B4\"\n",
    "dr_orange = \"#FF7F0E\"\n",
    "dr_red = \"#BE3C28\"\n",
    "\n",
    "# Create function to build Histograms\n",
    "\n",
    "\n",
    "def rebin_df(raw_df, number_of_bins):\n",
    "    cols = [\"bin\", \"actual_mean\", \"predicted_mean\", \"bin_weight\"]\n",
    "    new_df = pd.DataFrame(columns=cols)\n",
    "    current_prediction_total = 0\n",
    "    current_actual_total = 0\n",
    "    current_row_total = 0\n",
    "    x_index = 1\n",
    "    bin_size = 60 / number_of_bins\n",
    "    for rowId, data in raw_df.iterrows():\n",
    "        current_prediction_total += data[\"predicted\"] * data[\"bin_weight\"]\n",
    "        current_actual_total += data[\"actual\"] * data[\"bin_weight\"]\n",
    "        current_row_total += data[\"bin_weight\"]\n",
    "\n",
    "        if (rowId + 1) % bin_size == 0:\n",
    "            x_index += 1\n",
    "            bin_properties = {\n",
    "                \"bin\": ((round(rowId + 1) / 60) * number_of_bins),\n",
    "                \"actual_mean\": current_actual_total / current_row_total,\n",
    "                \"predicted_mean\": current_prediction_total / current_row_total,\n",
    "                \"bin_weight\": current_row_total,\n",
    "            }\n",
    "\n",
    "            new_df = new_df.append(bin_properties, ignore_index=True)\n",
    "            current_prediction_total = 0\n",
    "            current_actual_total = 0\n",
    "            current_row_total = 0\n",
    "    return new_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create function to build lift charts\n",
    "\n",
    "\n",
    "def matplotlib_lift(bins_df, bin_count, ax):\n",
    "    grouped = rebin_df(bins_df, bin_count)\n",
    "    ax.plot(\n",
    "        range(1, len(grouped) + 1),\n",
    "        grouped[\"predicted_mean\"],\n",
    "        marker=\"+\",\n",
    "        lw=1,\n",
    "        color=dr_blue,\n",
    "        label=\"predicted\",\n",
    "    )\n",
    "    ax.plot(\n",
    "        range(1, len(grouped) + 1),\n",
    "        grouped[\"actual_mean\"],\n",
    "        marker=\"*\",\n",
    "        lw=1,\n",
    "        color=dr_orange,\n",
    "        label=\"actual\",\n",
    "    )\n",
    "    ax.set_xlim([0, len(grouped) + 1])\n",
    "    ax.set_facecolor(dr_dark_blue)\n",
    "    ax.legend(loc=\"best\")\n",
    "    ax.set_title(\"Lift chart {} bins\".format(bin_count))\n",
    "    ax.set_xlabel(\"Sorted Prediction\")\n",
    "    ax.set_ylabel(\"Value\")\n",
    "    return grouped"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "autoscroll": "auto"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAAI4CAYAAABndZP2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABvcElEQVR4nO3deXyU5b3//9dnJpN9hZAQCLuIAioCbgjuVtS6tWpFj63Wfj3tT1tPj7W1tZv12P20Pa1aq1Wxda1ad1yq1n0DFxAQkD1hT8i+zkyu3x8zYIghG5ncM5P38/Hgwczc2+ceJby5ruu+LnPOISIiIpJMfF4XICIiItLfFHBEREQk6SjgiIiISNJRwBEREZGko4AjIiIiSUcBR0RERJKOAo7IIGZmc8xsZbv3k8zsAzOrM7Nv7eO5Xzazr+17ld4ws/VmdtJetu3xvYlI/FHAERkE9vaXtXPuNefcpHYffRd42TmX45z7Y1d/yQ80M5tvZv/TzT43mNlHZhYys592sv1CM9tgZg1m9piZDelLLZ18byISZxRwRKS9McAyr4voyMz8Pdx1NZGQ9nQn55gC/AW4GCgGGoFb+qtGEYkvCjgig5iZHWdm5dHXLwHHAzeZWb2Z3Q+MBp6Mvv/uXs5xlpl9aGa1ZrbGzOa22zzGzN6Idnk9b2aF7Y57yMy2mlmNmb0aDSC7ts03sz+b2QIzawAuAy4Cvhut5cnOanHO3e2cewao62TzRcCTzrlXnXP1wI+AL5hZThdf0WFmttzMqszsLjNL7/i9Rd+vN7PvmNmS6P082G7fQjN7ysyqzWynmb1mZvrZKxJj+kMmIgA4504AXgOudM5lO+fmARuBM6Lvf93xGDM7HPgbcA2QDxwDrG+3y4XApUARkAp8p922Z4CJ0W3vA/d2OP2FwI1ATvQa9wK/jtZyRh9ucQqwuN39rgFagf27OOYi4BRgQnS/H3ax7/nAXGAccDBwSfTzq4FyYBiRlqMfAFojRyTGFHBEZF9cBtzpnPuXc67NObfJObei3fa7nHOrnHNNwD+Aabs2OOfudM7VOedagJ8Ch5hZXrtjH3fOvRE9b3M/1JoN1HT4rIZIgNqbm5xzZc65nUTC1rwu9v2jc25zdN8n+fReg0AJMMY5F4yO31HAEYkxBRwR2RejgDVdbN/a7nUjkZCBmfnN7JfRLq1aPm31KWy3f1l/FgrUA7kdPsul8+6szmrYAIzoYt9O7xX4DZGxQc+b2Vozu7Zn5YrIvlDAEZGudNfSUEak+6a3LgTOAk4C8oCx0c+ti2vva6vHMuCQXW/MbDyQBqzq4phR7V6PBjb39qLRVqqrnXPjgTOA/zazE3t7HhHpHQUckcEjYGbp7X6l9OCYbcD4LrbfAVxqZieamc/MRprZAT04bw7QAlQCmcDP+6EWzCwQHdzrA1Ki97nrCax7gTOic9hkAT8D/umc66oF5wozK40+Tv4D4MEe1Nmxps+b2X5mZkAtEI7+EpEYUsARGTwWAE3tfv20B8f8Avhh9Amg73Tc6Jx7l8gg4t8TGc/yCpFHzbvzNyJdPpuA5cDbPTjmDmBytJbH9rLP7UTubR5wXfT1xdFalwFfJxJ0thMJWf9fN9e8D3geWBv91eU8PHsxEXiBSBfZW8AtzrmX+3AeEekF01g3ERERSTZqwREREZGko4AjIiIiSUcBR0RERJKOAo6IiIgknZ48JhpXzB9wFkj3ugwREREZIK6lvsI5N6w3xyRewAmkk1Y6w+syREREZIA0r3llQ2+PUReViIiIJB0FHBEREUk6CjgiIiKSdBJuDE5n8nOzue6KeUwYXYLPrPsDpFNtzrFm4xZuvPl+qmvrvS5HRESkz5Ii4Fx3xTwOmzaVlEAaKOD0nXMMGTKU666YxzW/uN3rakRERPosZl1UZnanmW03s6V72W5m9kczW21mS8xsel+vNWF0icJNfzAjJZDGhNElXlciIiKyT2I5Bmc+MLeL7acSWWV3InA58Oe+XshnpnDTX8zUzSeSIIZnBHn+5FUUpwe9LmXQSOTvPJFr74uYBRzn3KvAzi52OQv4m4t4G8g3MzUdiIj00LVTtzCrqIHvH7TF61IGjUT+zhO59r7wcgzOSKCs3fvy6GeD45vvwqKF73DP3Xfyh5v+wiv/fol1a1dzyWWXd7pvXW0tzy54ivMuuLBX1/jLLX8iMzOTiy+5rD9KFpEBtPOCD8lIcbvf/+ekSv5zUiVNIWPIA9O8KyyJVc37kHT/Z79z56C8MeBhZd0rzQzu0ckxWP5/8TLgdNYP4jr5DDO7nEg3FqSk9VsBDy6p5EsHD+2383UnHA7j9/t7dcyxx5/AscefsNftdXW1PPTg/b0OOCKSuCY/PoVfTN/EOaOrSfM7WtugPuinzcFfjtrAE2X5vLglh+awZgLZF8XpQU4vreHMUdUEw1DZkkJhWpg0v6MpZDy/OZdffVRMRUscBhyfH0vLwJ+WSVGWj2v2W8UJBZvJ8IVpDBmPb8zn+++P9LrKmPIy4JQDo9q9LwU2d7ajc+424DYAX3pOpyGoL/7xUVW/BZzNm8r55jf+H1MPOpiVKz5m9Jix/OzGX3HeOadz5tlf5O233uD8Cy4iLy+Pv9zyJ1pbg5SOGsVPbvg5mZlZvPn6a/zvr39OfkEBBxw4efd5n3z8nyxftpTv/eDHVFZW8Isbfsqm8kjD17U//AkP3HcPm8o3cuF5Z3PEkbO46urv8re77uCF55+htbWV4084if+84lsA3HHbrTz95GMMH15CfkEBB06e0i/3LiIDa2tTgNqgj1SfI9gGfoOH1ufzu+XD+XxpNVcesJ07Zq3n31tzeKIsn2c25VLdmhQPzcbc+OwWzhpdzRmlNRyQ18y/tuTw9zVDufj1cfzPoZu4bGKk5SPV79jalMIHVVkDV5zPjy89C196Fv60zN2vfZ28Np+ftuYG2loaqA2kU5X3KGlWTrMLkO4PERo9g/rqSbD8zYGrf4B5+X/8E8CVZvYAcARQ45xL6O6pDevX8aPrb2TaodO5/sc/4KEH7wMgNTWNO+6+j+qqKq759je55ba7yMjMZP6dt3Pv3+bz5Uu/xo3X/4g//3U+o0aP4fvXfLvT8//2lzcyfcZh/PYPNxEOh2lqbOSbV/03az75hPseegyAt998nbKN67n7vodwzvHf3/oG7y9aSEZmBs8/+zT3/eNRQuEw//GlLyjgiCSwovQQO1v9XL2wlCOH1TM8I8TGhlRuWVnELSuLGJIa4tTSGs4ZXc0fDitjUWUmT5bl82R5HpsaU70uP444Dh3SxBmjqjlzVA1D0kI8VZbHLz4azivbsmlt+7QVrCg9xO2rhnJ/20nM873A8IzQvl/e598joOw1uOwKLS2NkeDS3EBbSyPh5gZCdTtp21G2xzYXat3jMtnHrOW2piE8PvY7nLX+twzPWEjD8sp9rz+OxSzgmNn9wHFAoZmVAz8BAgDOuVuBBcBpwGqgEbi0v679xXtX9/u+j1y0X7f7FA8vYdqhkafdTzv9TB647+8AfG7uqQB8tORD1q5dzWVfiXQnBYNBDjpkGuvXrWXEyFJGjxkLwKmnn8mjjzz4mfMvfPdtrr/xVwD4/X6yc3Kora3ZY5+333yDt996g4vOPweAxsZGNm7cQGNDA8efeDLpGRkAHHPc8T26bxGJT1e+M5rlZy3j4Q0FPLh+yGe272xN4d61Q7l37VAy/WFOGlHHGaU1/PCQLayvT+PxjXk8WZ7Hipp0Oh8xkLz85phdVM+Zo2o4Y1Q1zWEfT5TlccXbo3m3IhO3l+9j3qvjASg+9yy+/fCqvV/AfNFQEg0oaZHX/rSsz7a0pASioaSRtpZPg0uovoq2ivI9Q0uwpc/3vLv2MWN4eeHoPp8nkcQs4Djn5nWz3QFXxOLaPQkjEAk3Pd23J6zD49W73mdkZALgnOOII2fx81//bo/9Vq74uN+ecnc4Lrnscr543gV7fH7f3+8eZD/CRJLb0UX1vF2RRdh1/ye7MeznibJ8nijLJ8UcR0f/cn/ihDU0hX08WZbHE2X5LOziL/dEl+Fv46SSWs4cVcPckTVsaEjjibI8znxpQo9DnqWk4s/KAyB97FR8aVn4dwWZ3eElE0tJbRdMPg0u4YYaWis3R983RkNLc4zvfE/1y98Y0Ot5SZ2y/Wjrls0sWfwBBx9yKM898zTTDp3OyhXLd28/6OBp/OrnN1C2cQOjRo+huamJbdu2MnbceDZt2kR52UZKR43muWee6vT8hx1xFA8/eD8XXvyVSBdVUxOZWVk0Njbs3ueoWbP5801/5NTTzyAzM4vt27aRkpLC9Bkz+emPvs9XLruccDjEa6+8zBfOPT/m34mIxMaconpe25bd6+NCznhlWw6vbMvh6kUjOXRIE2eOquaWIzcyJDXEk+X5PFmWxyvbsgm2JfYg5V3ddGeW1nDs8Drei3bTXb+4hPJOuul86Vn4M3PxZebiz8zDn5kTfZ9HSk4B5v/0r8y8mZGW+Zat62jesIy2lgbC0UDjWpsG7B57qyGJx9x0NKgDzvkHFfTr+caNn8BTTzzGz3/2E0aNHsO558/jwfvv2b29YMgQfnrDL7jue1fT2hrpH/3Glf/FmLHjuO7HP+OqK/6T/IICph06gzWrP9v8+Z3v/YAbr/8xjz/6MH6/n2t/+BMOPuRQDpl2KOefcwZHz57DVVd/l3Vr13Lpf0RacDIzM7nhF7/hgMlTOPmU07jwvLMpGTGCadNn9Ou9i8jAml1cz7ffLd3Hsxgf7Mzkg52ZXL94BBNymjlzVA3XHbyVv+U28/zmXJ4sy+O5zbnUh3r3BKhXRmW2csaoas4YVcOhQxp5eWsOj5Xl8/V3x1HrL8CflYuvKJeszFz80V++zFz8GTm0BVtoa6wlHP0Vqq+idfuGyPuG2t2tLcXnXsO2h3/j8Z1KdyzSU5Q4fOk5Lq10z7+cH7v1JxQWj/CooojNm8r5ryu/wT8efdLTOvpDxbbNnP31670uQ0T2Ii8QYvUXljHioYNi1soyPCPIaSMjj0gfNayBN3Zk82RZHk+X57G9OZ4ei3ZMzmvmzLGRbrfRmS08Vz2GZxoO4PXwVFoyhkZCTGo64aZ62poiYSXcWLtHmAk31kFbzwYNK+AMvOY1r7znnJvZm2MGdQuOiEgimlXUwMKKzJh2IW1tCnDn6kLuXF1ITiDM3BG1nDGqmp9P38zy6nSeiI7bWVff97nJsibP6mGXiUW6j7J2tbrkcOTQOk4bspFTs1cS8DmeC03nxvpJvL2pkNbGhmiAWU+4cQnhhlramhvYy1RrvTaYxrEkMgWcfjJiZGlStN6ISPybU1zPa9t7P/6mr+qCfh7aUMBDGwpI9bVx3PA6zhxVw0unrGJHcwpPluXzRFkei6sy6M0TWdmTj44EHJ9/z+6iz3QfZZPSWseR4feZG/iAUzJXsKM1jae3D+c/Fu/HB1sdLlgBVMTsO2hvMI1jSWQKOCIiCWZ2UT3XfeBNt3xrm4/nN+fx/OY8vvWu4/ChDZw5uoZ7j1lHwBxPlufzeFkeb27P/swTXv6sPFJyh5GSV0hKbiEAhZ////AF0gg31e3RZdRaUUZWSyUn5azn9OItnDS8luU16Ty5Oo9flo1l7e6Wo74/Oi3JTQFHRCSBZKeEOTCvmYUVAziD7l60OePtimzersjmB++PYHJeM2eMquFXM7cwKivI8zVjea75IF73H0YwdziupQnXFiYlZwjDqOKm1J9xJd9iB1k0b1xOw/I3P10eYdynY3+eKMvjOwtHxtnYH4l3CjgiIgnkyGENfLgzIy7WmbJAGim5hbtbZLbkDePO3ELuxFG84xNOsne5LOcN/pj1OC9vzeHxjbk8uymXba0pfP+wjRy2/06+tuFH/GlFEReOquHMU2qYFH166+9rhvIfr41LmKe3JP4o4IiIJJDZRfW8PoDjbwDwpZCSO4SUvGF7BBoLpBOqrSBcW0GwZgctmz8hVFNBW0sjO4ClwB8oYUjqME4treHsUdXcMWtDpytbh9rgnH9PSIr5dyQ+KOB4YNHCdwgEAhwybXqfzzHniOm89s77/ViViCSCOcX13LhkeGxOboY/uyAaYj4dK+PPzCFUV0WotoJQTQWNaz4kVLODtsbaHp22/bIRY7NauG3WBo4a1kCKD5rDxqMb8vj++6VsUxeU9KNBG3BSmioY/+51rD3iRkLphQN67fcWvktmZuY+BRwRGXwy/G0cMqSJt3fs+/gbX2bu7gCze+BvdgHhpvpokNlBc9kKQjUVhOurwLX1wx3A+oY0Pq5JZ1ZRw+5VuWuDfoUb6XeDNuCUfHwH2RUfUrL8Dsqmf69fznn1VVewbesWWltbueCii/nCuV/izddf4+Y//p62tjD5+QX86Pr/4Z8PPYjP7+OZp5/kmmt/yOOPPszsY47jpM/NBT5tnWlsbODqb11BbW0toVCQb3zzvzju+BP7pVYRSTyHFTawrCqdxvCn41K6m0vG0jL36FaKdDMNxYVaCdVUEKqtoHX7ehpXLyJUWwnhflghuxu7VuW+85NCvjqxon9W5RbpYNAFnEMfnY2v7dNl5IvWPULRukdo86XywTmv79O5f/yzG8nLy6e5uZkvzzuPY48/kRuv/xG33XUPI0tLqampJi8vny+c9yUyMzO5+JLLAHj80Yc7PV9qahq/+cNNZGdnU11VxSX/8SWOPe6EzyzqKSKDQ2fz3+yaS8ZSUvcIMv68QgK5heDzE6rZEWmVqd5O88ZlhGoqB3yRx/Z2rWwN8O1BsrK1DLykDDgzHjm818f42lq7PO69L77b7TkeuPfvvPzSCwBs27aFRx/+B4fOmMnI0sh6MXl5+b0ryjlu/uPv+OC9Rfh8PnZs30ZlZQWFhcN6dx4RSQpziur5/fKi3e/NH+nWKTz1cnxpGYRqK3ePk2nZujYyTqa5YW+nE0lqSRlwugsjo9//JYXrHsX5AlhbkB3jvrDP3VSLFr7Du++8xV1/f4D0jAwu/+rFTNz/ADasX9ftsX5/CrvWBHPOEQwGAXhmwZNUV1VxzwOPkBIIcMbcE2ht0aRWIoNRqq+NGUMbeWtHNlmTZ5E9+ejd2/xZeQC0bF2rWXZFopIy4HQnpWUnO8Z/gYpx51C47lECzZX7fM76+npyc3JJz8hg/bq1LF2ymGCwlfcXLWRTefkeXVRZWVk01NfvPrZk5Eg+Xr6Mk085lVf+/SKhUCTg1NfVUzBkCCmBAIvefZstmzfvc50ikphmDm1kVW0atUE/LH+ThuVvknvYaWSMmaKFH0U6MSgDztqjfr37ddmh/TPAeNbRc/jnPx7ggi+eyZix45h68CHkFxTwgx//jGv++5u4tjYKhgzlltvuZM6xx/O9q6/ilZdf4pprf8g5XzyPq791BV++8DwOP+JIMjIyATj19DP49je/zsUXfJH9Jx3I2HHju6lCRJLV7OIO89+YkTZ8nHcFicQ529U1kih86TkurXTGHp89dutPKCz2Zl2WZFSxbTNnf/16r8sQkXaeOGE1t68q5MnyfAAChaXkHHICLVtWq1tKkl7zmlfec87N7M0xmi5SRCTOpZjjiMIG3mjXgpNWMoGWLWsUbkT2QgFHRCTOHTqkkQ0Nqexs/XRUQdqI/WjZvNrDqkTiW1IEnDbnIMG62uKWc5HvU0Tixuzi+j1ab/zZBVhKgFD1Ng+rEolvSRFw1mzcQijYopCzr5wjFGxhzcYtXlciIu3MLqrntW3tuqdG7EfL5jUeViQS/5LiKaobb76f666Yx4TRJfg0y2+ftTnHmo1buPHm+70uRUSifOaYVdTAN97+dMbftJIJNKzsfvJRkcEsKQJOdW091/zidq/LEBHpdwflN7GtKYXt0cUoLTWdlPwiWrdv9LgykfiWFF1UIiLJquP6U2nDx0fCTZsWqBTpigKOiEgcm1NUz+vbOj4erqenRLqjgCMiEqcMx6yidjMYm4/U4rG0blnrbWEiCUABR0QkTh2Y10xt0M+mxlQAUoeNIlS3k7aWRo8rE4l/CjgiInFqTnGHx8PVPSXSYwo4IiJxanbRngtsav4bkZ5TwBERiUuO2e1acFJyCwFHuLbC27JEEoQCjohIHJqY20KwzdjQEBl/o9Ybkd6JacAxs7lmttLMVpvZtZ1szzOzJ81ssZktM7NLY1mPiEii+HR5hsjs7KnR1cNFpGdiFnDMzA/cDJwKTAbmmdnkDrtdASx3zh0CHAf8r5mlxqomEZFEMaf40/E3vrQsUnKG0LqjzOOqRBJHLFtwDgdWO+fWOudagQeAszrs44AcMzMgG9gJaHpOERnkHHPaLbCZWjKe1m3rwbV5W5ZIAollwBkJtP/nRnn0s/ZuAg4ENgMfAVc599k/wWZ2uZktMrNFLhyMVb0iInFhTFYrfp9jdV0aAGkjJtCyWY+Hi/RGLANOZ8t6uw7vTwE+BEYA04CbzCz3Mwc5d5tzbqZzbqb5A/1dp4hIXJlTXM8bu8bf+FJIHTaalq3rvC5LJKHEMuCUA6PavS8l0lLT3qXAP13EamAdcEAMaxIRiXvtF9hMLR5NqHobLtjscVUiiSWWAWchMNHMxkUHDl8APNFhn43AiQBmVgxMArTIiogMarPbjb9JK9Hj4SJ9kRKrEzvnQmZ2JfAc4AfudM4tM7OvR7ffCtwAzDezj4h0aX3POadZrERk0BqZ2UpuIMyKmnQgsjxD1ap3Pa5KJPHELOAAOOcWAAs6fHZru9ebgc/FsgYRkUQyp6ieN7Zn4zBSCopxwRbC9dVelyWScDSTsYhIHDm6uEP3lCb3E+kTBRwRkTgyp90Cm1o9XKTvFHBEROJEcXqQovQQH1Vn4MvIwZ+ZQ7Cy48OnItITCjgiInHi6KJ63tyRRZuzSOvN1nXgOk4fJiI9oYAjIhIn5hRHBhiDZi8W2VcKOCIicWLX/DeWEiAwdGRk/SkR6RMFHBGRODAkNcTorFY+2JlJatFYgju34EKtXpclkrAUcERE4sDRRfW8U5FF2BlpI/ZT95TIPlLAERGJA3N2z39jpJWM1/w3IvtIAUdEJA4cHZ3/JjB0BOGmetoaa70uSSShKeCIiHgsNxBm/9wW3qvMjE7up9YbkX2lgCMi4rFZRfUsqsyktc2nx8NF+okCjoiIx3Y9Hu7PyscC6YSqtnpdkkjCU8AREfHYrvWn0kaoe0qkvyjgiIh4KCslzOT8Zt6tyCKtZAKtCjgi/UIBR0TEQ0cUNrC4KoMWXwYpBcNp2b7B65JEkoICjoiIh+YU1/P6tmzSho8juKMMwiGvSxJJCgo4IiIemlNcz2vbs0kr2U/jb0T6kQKOiIhH0v1tTBvSxNsVOaQOH6uAI9KPFHBERDxyWGEDy6vTac0fQ7i+mrbmBq9LEkkaCjgiIh6ZE53/Ro+Hi/Q/BRwREY/MLmo3/kazF4v0KwUcEREPBHxtzCxs5N3GUZjPR6hmh9cliSSVFK8LEBEZjGYObWR1XRrNRQfgV+uNSL9TC46IiAeOLorOf6PHw0ViQgFHRMQDc4rreWPnUFJyh9K6o8zrckSSjgKOiMgA85vjyMIGFvqm0rp9A7SFvS5JJOko4IiIDLBpQxrZ2JBKQ9FUWjare0okFhRwREQG2Jyiel7fkUNq0Rhatq71uhyRpKSAIyIywGYX1/N28zhCNTtwrU1elyOSlBRwREQGkM8cs4Y1sChthrqnRGJIAUdEZABNzW9iR0sKNcMOoWWL5r8RiZWYBhwzm2tmK81stZldu5d9jjOzD81smZm9Est6RES8NqeonjeqhuHCYcJ1O70uRyRpxWwmYzPzAzcDJwPlwEIze8I5t7zdPvnALcBc59xGMyuKVT0iIvFgdnE9z4YPU+uNSIzFsgXncGC1c26tc64VeAA4q8M+FwL/dM5tBHDObY9hPSIiHnMcXdTAexlHafZikRiLZcAZCbSfnrM8+ll7+wMFZvaymb1nZl/u7ERmdrmZLTKzRS4cjFG5IiKxdWBeM/WhFLZljCVYscnrckSSWiwX27ROPnOdXH8GcCKQAbxlZm8751btcZBztwG3AfjSczqeQ0QkIcwuquethlJaataBa/O6HJGkFsuAUw6Mave+FNjcyT4VzrkGoMHMXgUOAVYhIpJk5hTX87rvMFq0erhIzMWyi2ohMNHMxplZKnAB8ESHfR4H5phZipllAkcAH8ewJhERjzhmFzfwXsaRtG5d53UxIkkvZi04zrmQmV0JPAf4gTudc8vM7OvR7bc65z42s2eBJUAb8Ffn3NJY1SQi4pUJOS2ELYU1lWFcqNXrckSSXiy7qHDOLQAWdPjs1g7vfwP8JpZ1iIh4bU5xPW+3jNfTUyIDRDMZi4gMgDlF9SwMTKdVAUdkQCjgiIgMgNklTbzVOIpwQ43XpYgMCgo4IiIxNjqrhfQUHx+X13ldisig0eOAY2ZZsSxERCRZzSmu553w/rRsXut1KSKDRrcBx8xmmdlyoo9vm9khZnZLzCsTEUkSc0a08k7bZII7t3hdisig0ZMWnN8DpwCVAM65xcAxsSxKRCSZzBnewGtbM/nsZO4iEis96qJyzpV1+Cgcg1pERJLOiIxW8gNhlqzX4GKRgdSTeXDKzGwW4KIzEn8LzTYsItIjs0uaebftAFq2bfC6FJFBpSctOF8HriCyEng5MC36XkREunHsWHizdjguHPS6FJFBpdsWHOdcBXDRANQiIpJ0Zg+t5m8r/F6XITLodBtwzOwuOhkZ55z7akwqEhFJEsPSQxSnNvLe6iqvSxEZdHoyBuepdq/TgXOAzbEpR0QkeRw7IZWFLeMINTV4XYrIoNOTLqpH2r83s/uBF2JWkYhIkjh2VJg3KvOAZq9LERl0+rJUw0RgdH8XIiKSbGbl7+DldW1elyEyKPVkDE4dkTE4Fv19K/C9GNclIpLQCguyGeX/mEUbhhH58SkiA6knXVQ5A1GIiEgyOXZiBu81DCfkFG5EvLDXgGNm07s60Dn3fv+XIyKSHGaXNEWXZxARL3TVgvO/XWxzwAn9XIuISFKw1HRmZZbx7XVpXpciMmjtNeA4544fyEJERJLFsJGlTLD3WLTjQK9LERm0ejIPDmY2FZhMZB4cAJxzf4tVUSIiiezocQHery2gta0vD6qKSH/oyVNUPwGOIxJwFgCnAq8DCjgiIh35/MweWsWrq1O9rkRkUOvJPy/OBU4EtjrnLgUOAdSxLCLSidRhozjclvHaZv2YFPFSTwJOs3OuDQiZWS6wHRgf27JERBJTwYhRHODfzLsVWV6XIjKodfWY+E3A/cC7ZpYP3A68B9QD7w5IdSIiCeaoUbCkOoumsMbfiHipqzE4nwC/BUYQCTX3AycDuc65JQNQm4hIQknJG8YRKZ/w+sb07ncWkZja6z8xnHP/55w7CjgG2AncBTwDnG1mEweoPhGRhJFWMoHD25bw2rZsr0sRGfS6bUN1zm1wzv3KOXcocCFwDrAi5pWJiCSY3BGjOTh9K2/v0PgbEa91G3DMLGBmZ5jZvURacFYBX4x5ZSIiCcSXnsWMvGpWVKdRH/J7XY7IoNfVIOOTgXnA6UQGFT8AXO6caxig2kREEkba8PHMbH6b17are0okHnTVgvMD4C3gQOfcGc65exVuREQ6lzZiP47wrdD4G5E4obWoRET2lT+FzGElzEjfwZvbi7yuRkTo4VpUIiKyd6lFY5jc8D5rgmnUBPVjVSQexHQmKjOba2YrzWy1mV3bxX6HmVnYzM6NZT0iIrGQVjKBw1oX8bq6p0TiRswCjpn5gZuJLM45GZhnZpP3st+vgOdiVYuISCyllUzgyLQ1vK4BxiJxI5YtOIcDq51za51zrUSewjqrk/2+CTxCZI0rEZGEklIwHF+wkSMKqhVwROJILAPOSKCs3fvy6Ge7mdlIIhMH3trViczscjNbZGaLXDjY74WKiPRV2ogJTKx+m/LGVCpbNP5GJF7EMuBYJ5+5Du//AHzPORfu6kTOuducczOdczPNH+iv+kRE9llayX4cHl6s8TcicSaW/9woB0a1e18KbO6wz0zgATMDKAROM7OQc+6xGNYlItIvfJm5+DOyOSq4mX+sL/C6HBFpJ5YtOAuBiWY2zsxSgQuAJ9rv4Jwb55wb65wbCzwM/H8KNyKSKNJKJtC6ZQ1HF9Vr/I1InIlZC45zLmRmVxJ5OsoP3OmcW2ZmX49u73LcjYhIvEsrmcCE7a9Q2ZLC1iZ1n4vEk5iOiHPOLQAWdPis02DjnLsklrWIiPQnS0klMHQkh1dqeQaReBTTif5ERJJVavFYgpWbmFNYqwU2ReKQAo6ISB+klUygZctqji6u5w214IjEHQUcEZFeM9JKxjO2/iMaQz7KGlO9LkhEOlDAERHppcDQEYSb6jk6d5vG34jEKQUcEZFeShuxHy1bVjOnuE4BRyROKeCIiPRSWskEWjavZnZRg+a/EYlTCjgiIr3gz87HAmmMDm7EOVhXr/E3IvFIAUdEpBfSSvajZcsa5hTXRx8P72zZPRHxmgKOiEgvpI2YQMuWNczW8gwicU0BR0SkhyyQRkp+Ma3bN0RacDTAWCRuKeCIiPRQ2vDxBHeUMTq9kXR/G6tq07wuSUT2QgFHRKSH0kZEnp46uqieNzT+RiSuKeCIiPSE+UgtHkfL1rXqnhJJAAo4IiI9kDqslHD9TtqaG5hTVK8FNkXinAKOiEgPpJZMoGXzGkoyghSkhVhene51SSLSBQUcEZEe2DX/zeyiet7cno3T+BuRuKaAIyLSDX/uUMyMUM0OZmv8jUhCUMAREenGrtYbQBP8iSQIBRwRkW7smr24MC3IiMwgi6syvC5JRLqhgCMi0gVLyyQlZyitO8qYXdTAWzuyaHMafyMS7xRwRES6kDZ8PK3bN0BbmNnF9byh8TciCUEBR0SkC2kj9qNl82ogMv5G89+IJAYFHBGRvfH5SS0aTcvWteSnhhif08L7lZleVyUiPaCAIyKyF6lFowlVb8e1NjNrWAPvVmQR0vgbkYSggCMishftHw/X+lMiiUUBR0RkL9JKJuwefzOnuJ7XFXBEEoYCjohIJ1Lyi3DhIOH6KrJTwkzKbWaRxt+IJAwFHBGRTqSN+LR76qhhDby/M5OWNv3IFEkU+tMqItKJjt1TGn8jklgUcEREOvBlZOPPzCNYuQnQ+lMiiUgBR0Skg7SSCbRsXQfOkeFv46CCJt7ZkeV1WSLSCwo4IiIdRB4Pj3RPHTGsgY+qMmgK68elSCKJ6Z9YM5trZivNbLWZXdvJ9ovMbEn015tmdkgs6xER6Y75AwQKR9K6dR2g7imRRBWzgGNmfuBm4FRgMjDPzCZ32G0dcKxz7mDgBuC2WNUjItITqcVjCe7cggu1AhpgLJKoYtmCcziw2jm31jnXCjwAnNV+B+fcm865qujbt4HSGNYjItKttJIJux8PT/O1MX1II29p/I1IwollwBkJlLV7Xx79bG8uA57pbIOZXW5mi8xskQsH+7FEEZH2jLSS8bRsjgScmUMbWVmbTn3I73FdItJbKTE8d2cr0rlOdzQ7nkjAmd3ZdufcbUS7r3zpOZ2eQ0RkXwWGlBBuaaStsQaA2eqeEklYsWzBKQdGtXtfCmzuuJOZHQz8FTjLOVcZw3pERLqUNmICrdHuKdD4G5FEFsuAsxCYaGbjzCwVuAB4ov0OZjYa+CdwsXNuVQxrERHpVvvZi1PMcXhhA29q/I1IQopZF5VzLmRmVwLPAX7gTufcMjP7enT7rcCPgaHALWYGEHLOzYxVTSIie+PPysPSMgnu3ALA9KGNrK1Lo7o1lj35IhIrMf2T65xbACzo8Nmt7V5/DfhaLGsQEemJ1JIO3VOa/0YkoWlqThERIHPijN2PhwMcXVzP6xp/I5KwFHBEZNCzQBopWfm0bNsAgM8cRw1r4PXtGn8jkqgUcERk0EstHhd5EZ1n65CCJjY3BqhoCXhYlYjsC42eE5FBK2vyLLInH737ffG51wBwQsXNvL5ds1aIJDIFHBEZlCyQhj+rgFBdFbWLFjDk+IvY9vBvADji2LU8tK3A4wpFZF+oi0pEBp3U4rEMPfkSXLCZyhfuJlj56RykhmOWnqASSXhqwRGRQcP8AbIPPo604eOoXfQsrds37N5Wv/wNACbnN1PVksKWJo2/EUlkCjgiMigECkvJnXkqwYoyKv81Hxdq3WN7w/I3gcj8N6+p9UYk4SngiEhy8/nJnjqH9FEHUvf+83vMddOZOcX1PFWeN0DFiUisaAyOiCStlIJihp70ZfyZuVT+a3634QYcRxdpgj+RZKAWHBFJPuYj68CjyBx/CHWLX6K5bEWPDts/t4XmsI+yxtQYFygisaaAIyJJxZ9bSN5hp9HW3EDlC3fT1tzQ42PnFNfzmlpvRJKCAo6IJAkjc//DyJp0GPVLX6Np3ZJen2F2UT3/3poTg9pEZKAp4IhIwvNn5ZN72Kng2qh88R7aGmv6cBbHnKJ6blhc0u/1icjAU8ARkYSWMX4a2VOOpmHF2zR+8l6fzzMuuxUzWFuv8TciyUABR0QSki8jh9yZc/EF0tj58v2E63bu0/lmF+0af2P9U6CIeEoBR0QSTvroKeQcchyNn7xHw8p3wLl9PuecYi3PIJJMFHBEJGH40jLJmf45/Nn5VL32EKHq7f1y3uEZQc4bU8X81UP75Xwi4j1N9CciCSFt5P4MOfkSwnU72fni3/st3AD8z6GbSPM7zh+7b91cIhI/zPVD0+5A8qXnuLTSGV6XISIDxAJp5Ew7icCQEmoXLdhj5e99tfOCD8lI+ezPwKaQMeSBaf12HRHZN81rXnnPOTezN8eoBUdE4lZq8ViGnnwJLthM5Qt392u4GZoW4kcfjKC8IbB7CE9jyLh/bQEHPjal364jIt7QGBwRiTvmD5B98HGkDR9H7aJnad2+oV/OOzqrlTNKqzljVA3ThjTy7605rKpNoyQzSGvISPM7aoM+tjUH+uV6IuIdBRwRiSuBwlJyZ55KsKKMyn/Nx4Va9+Fsjin5zZw5KhJqSjODLCjP5U8rinhxSw7NYR/3H7OW21cN5c5PCvnqxAqGZ4T67V5ExDsagyMi8cHnJ3vqHNJHHUjd+8/3YOXvvZzGHEcWNnDGqBrOGFVNisETZXk8UZbHWzuyCTvNcyOSaPoyBkctOCLiuZSCYvIOO41QbWWk1aa1qVfHp/naOL6kjjNKazi9tIZtzSk8UZbPha+OY0lVBpq8T2TwUcAREe+Yj6wDjyJz/CHULX6J5rIVPT40NxBm7sgazhxVw4kldSytSufJ8nx+81wx6+vTYli0iCQCBRwR8YQ/t5C8w06jrbmByhfupq25odtjSjKCnF4a6Xo6srCB17Zn82RZPt9+t5QdLRoYLCKfUsARkQFmZO5/GFmTDqP+o1dpWv9Rl3vvl9PMWdHxNPvntvDc5lzmrx7KRa+Ooz7kH6CaRSTRKOCIyIDxZ+WTe9ip4NqofPEe2hprOtnLMWNoI2eOquGM0hryU0M8UZbPDUtKeHVbNsE2Td8lIt1TwBGRAZExfhrZU46m4eO3aVz93h7bUswxp7hud6ipD/l4oiyf/3xrNIsqM3EaJCwivaSAIyIx5cvIIXfmXHyBNHa+fD/hush6T5n+MJ8bUccZo6qZO7KWNXVpPFGWx2kv7seq2nSPqxaRRKeAIyIxkz56CjmHHEfjJ+/RsPIdhqYGOW18DWeOruaYonrercjiyfI8fvTBCDY3pXpdrogkkZh2ZpvZXDNbaWarzezaTrabmf0xun2JmU2PZT1eGp4R5IWztlGcHvS6lF5J1LohcWtPhrp9aZnkHXU2mZMOI2fh37iUJ3jupFUsO2sZp46s5ZH1BUx6bApnvLQft60apnAjIv0uZgHHzPzAzcCpwGRgnplN7rDbqcDE6K/LgT/Hqh6vXTt1C0dlb+H7B23xupReSdS6IXFrT/S6f3hUE7NOOYlvZy7g6cC1/Hv2m0zJb+IPy4sY+8hBXPjaOB5YP4TqVjUgi0jsxGypBjM7Cvipc+6U6PvvAzjnftFun78ALzvn7o++Xwkc55zb60/2RFuqYecFH5KR8tnvOOSMP26L3/v4VvF7pFji1Q2JW3vS1d0Gc1+YyFs7smjT8ggisg/ibamGkUBZu/flwBE92GcksEfAMbPLibTwQEpizVA6c/lXuHHyKk7zv0OqhQk6P6vcSF6qG0dTuHfT0Q+kP9cew4k565homwgkUN2QuLUnS93NLsAz4cO4bvn+rNu+2OvyRGSQimXA6eyfbB3/mdeTfXDO3QbcBpEWnH0vbeCsXbKY7ekb8U+M/OAPEOT1Txr47sJar0vrRpD/O7yBSQlXNyRu7clT9/Y1K1m3JH5DmYgkv1gOMi4HRrV7Xwps7sM+Ca8oPcTtq4ZyTuvPuH3VUIozQl6X1COJWjckbu2qW0Skf8RyDE4KsAo4EdgELAQudM4ta7fP6cCVwGlEuq/+6Jw7vKvzJtoYnPayJs+iYfmbXpfRa4laNyRu7apbRORTfRmDE7OAA2BmpwF/APzAnc65G83s6wDOuVvNzICbgLlAI3Cpc25RV+dM5IAjIiIivRd3AScWFHBEREQGl74EHK1aJyIiIklHAUdERESSjgKOiIiIJB0FHBEREUk6CTfI2MzqgJVe1zHIFAIVXhcxyOg7H3j6zgeevvOBl6jf+Rjn3LDeHJCIq92t7O1Iatk3ZrZI3/nA0nc+8PSdDzx95wNvMH3n6qISERGRpKOAIyIiIkknEQPObV4XMAjpOx94+s4Hnr7zgafvfOANmu884QYZi4iIiHQnEVtwRERERLqkgCMiIiJJJ6ECjpnNNbOVZrbazK71up5kZ2ajzOzfZvaxmS0zs6u8rmkwMDO/mX1gZk95XctgYWb5Zvawma2I/v9+lNc1JTMz+3b0Z8pSM7vfzNK9rikZmdmdZrbdzJa2+2yImf3LzD6J/l7gZY2xlDABx8z8wM3AqcBkYJ6ZTfa2qqQXAq52zh0IHAlcoe98QFwFfOx1EYPM/wHPOucOAA5B33/MmNlI4FvATOfcVMAPXOBtVUlrPjC3w2fXAi865yYCL0bfJ6WECTjA4cBq59xa51wr8ABwlsc1JTXn3Bbn3PvR13VEfuiP9Laq5GZmpcDpwF+9rmWwMLNc4BjgDgDnXKtzrtrTopJfCpBhZilAJrDZ43qSknPuVWBnh4/PAu6Ovr4bOHsgaxpIiRRwRgJl7d6Xo79sB4yZjQUOBd7xuJRk9wfgu0Cbx3UMJuOBHcBd0a7Bv5pZltdFJSvn3Cbgt8BGYAtQ45x73tuqBpVi59wWiPwjFijyuJ6YSaSAY518pmfcB4CZZQOPAP/lnKv1up5kZWafB7Y7597zupZBJgWYDvzZOXco0EASN9t7LTrm4yxgHDACyDKz//C2KklGiRRwyoFR7d6XombNmDOzAJFwc69z7p9e15PkjgbONLP1RLpgTzCze7wtaVAoB8qdc7taJx8mEngkNk4C1jnndjjngsA/gVke1zSYbDOzEoDo79s9ridmEingLAQmmtk4M0slMijtCY9rSmpmZkTGJXzsnPud1/UkO+fc951zpc65sUT+/37JOad/2caYc24rUGZmk6IfnQgs97CkZLcRONLMMqM/Y05Eg7oH0hPAV6KvvwI87mEtMZUwq4k750JmdiXwHJFR93c655Z5XFayOxq4GPjIzD6MfvYD59wC70oSiYlvAvdG//G0FrjU43qSlnPuHTN7GHifyJOaHzCIlg8YSGZ2P3AcUGhm5cBPgF8C/zCzy4iEzfO8qzC2tFSDiIiIJJ1E6qISERER6REFHBEREUk6CjgiIiKSdBRwREREJOko4IiIiEjSUcAREczsuujqzkvM7EMzO6KXx19iZiN6eczY9qscd/i8KVrHcjO71cz6/LPKzF42s5nR1wvMLL+Lfc9uv6Csmf3MzE7q67VFxDsJMw+OiMSGmR0FfB6Y7pxrMbNCILUXx/uBS4Cl9N/s4mucc9OiizG+RGRBwN0zaZtZinMu1NuTOudO62aXs4GniE7055z7cW+vISLxQS04IlICVDjnWgCccxXOuc0AZnZidAHKj8zsTjNLi36+3sx+bGavA/OAmUQmyvvQzDLMbIaZvWJm75nZc+2mhp9hZovN7C3giu4Ki4aYN4H9oq1ED5nZk8DzZpYVrWlhtMazotfIMLMHoq1RDwIZu84Xrbsw+vrL0X0Wm9nfzWwWcCbwm+h9TDCz+WZ2bg++i+vN7P3otgP64b+JiOwjBRwReR4YZWarzOwWMzsWwMzSgfnAl5xzBxFp8f1Gu+OanXOznXP3AIuAi5xz04jMTvsn4Fzn3AzgTuDG6DF3Ad9yzh3Vk8LMLJPIVP4fRT86CviKc+4E4Doiy1kcBhxPJJhkRWtsdM4dHL3ujE7OOyV6/AnOuUOAq5xzbxKZxv4a59w059yadvt3911UOOemA38GvtOTexOR2FLAERnknHP1RELA5cAO4EEzuwSYRGRRxFXRXe8Gjml36IN7OeUkYCrwr+gSHz8ESs0sD8h3zr0S3e/vXZQ1IXrsG8DTzrlnop//yzm3M/r6c8C10f1eBtKB0dEa74ne2xJgSSfnPwF42DlXEd1vZyf7dLynrr6LXd1n7wFjuzmXiAwAjcEREZxzYSIh4WUz+4jIInwfdnNYw14+N2BZx1aa6ODenq4NsybaGtTVNQ34onNuZYfr0IPrWC9q2bV/V1qiv4fRz1WRuKAWHJFBzswmmdnEdh9NAzYAK4CxZrZf9POLgVfoXB2QE329EhgWHbyMmQXMbIpzrhqoMbPZ0f0u2sfSnwO+GV2RGjM7NPr5q7vObWZTgYM7OfZF4HwzGxrdb0gn99Feb74LEYkDCjgikg3cHX0kewkwGfipc66ZyKraD0VbddqAW/dyjvnArdHuIj9wLvArM1tMpCVoVnS/S4Gbo4OMm/ax7huAALAk+rj5DdHP/wxkR+/lu8C7HQ90zi0jMj7nlWiNv4tuegC4JjqYeEK7/XvzXYhIHNBq4iIiIpJ01IIjIiIiSUcBR0RERJKOAo6IiIgkHQUcERERSToKOCIiIpJ0FHBEREQk6SjgiIiISNJRwBEREZGko4AjIiIiSUcBR0RERJKOAo6IiIgkHQUcERERSToKOCKDmJnNMbOV7d5Piq6kXWdm39rHc79sZl/b9yq9YWbrzeykvWzb43sTkfijgCMyCOztL2vn3GvOuUntPvou8LJzLsc598eu/pIfaGY238z+p5t9bjCzj8wsZGY/7bDtODNrM7P6dr++0pdaOvneRCTOpHhdgIjElTHAA14X0ZGZ+Xu462oiIe3re9m+2TlX2j9ViUg8UwuOyCAWbdUoj75+CTgeuCnaunE/MBp4Mvr+u3s5x1lm9qGZ1ZrZGjOb227zGDN7I9rl9byZFbY77iEz22pmNWb2qplNabdtvpn92cwWmFkDcBlwEfDdaC1PdlaLc+5u59wzQN0+fjW7HGZmy82syszuMrP0aH27v7fo+/Vm9h0zWxK9nwfb7VtoZk+ZWbWZ7TSz18xMP3tFYkx/yEQEAOfcCcBrwJXOuWzn3DxgI3BG9P2vOx5jZocDfwOuAfKBY4D17Xa5ELgUKAJSge+02/YMMDG67X3g3g6nvxC4EciJXuNe4NfRWs7o420Wmdk2M1tnZr83s6xu9r8IOAWYAOwP/LCLfc8H5gLjgIOBS6KfXw2UA8OAYuAHgOtj/SLSQwo4IrIvLgPudM79yznX5pzb5Jxb0W77Xc65Vc65JuAfwLRdG5xzdzrn6pxzLcBPgUPMLK/dsY87596Inre5H2pdEb1+CXACMAP4XTfH3OScK3PO7SQStuZ1se8fnXObo/s+yaf3Goxec4xzLhgdv6OAIxJjCjgisi9GAWu62L613etGIBsiY2rM7JfRLq1aPm31KWy3f1l/Fuqc2+qcWx4NTOuIjNU5t5vD2tewARjRxb6d3ivwGyJjg543s7Vmdm0vSxeRPlDAEZGudNfSUEak+6a3LgTOAk4C8oCx0c+ti2v3d6uH63C9zoxq93o0sLnXF4m0Ul3tnBsPnAH8t5md2NvziEjvKOCIDB4BM0tv96snT1FuA8Z3sf0O4FIzO9HMfGY20swO6MF5c4AWoBLIBH7eD7VgZoHo4F4fkBK9T39023FmNtoiRgG/BB7v5ppXmFmpmQ0hMnbmwR7U2bGmz5vZfmZmQC0Qjv4SkRhSwBEZPBYATe1+/bQHx/wC+GH0CaDvdNzonHuXyCDi3wM1wCtEHjXvzt+IdPlsApYDb/fgmDuAydFaHtvLPrcTubd5wHXR1xdHt00H3gIagDeBpUB3kxneBzwPrI3+6nIenr2YCLwA1Eevf4tz7uU+nEdEesE01k1ERESSjVpwREREJOko4IiIiEjSUcARERGRpKOAIyIiIkkn4RbbNH/AWSDd6zJERERkgLiW+grn3LDeHJN4ASeQTlrpDK/LEBERkQHSvOaVDb09Rl1UIiIiknQUcERERCTpKOCIiIhI0km4MTidyc/N5ror5jFhdAk+627tPNmbNudYs3ELN958P9W19V6XIyIi0mdJEXCuu2Ieh02bSkogDRRw+s45hgwZynVXzOOaX9zudTUiIiJ9FrMuKjO708y2m9nSvWw3M/ujma02syVmNr2v15owukThpj+YkRJIY8LoEq8rEZEOhmcEef7kVRSnBwfl9b3k9b0P9uv3VSzH4MwH5nax/VQiq+xOBC4H/tzXC/nMFG76i5m6+UTi0LVTtzCrqIHvH7RlUF7fS17f+2C/fl/FdDVxMxsLPOWcm9rJtr8ALzvn7o++Xwkc55zr8hv0pee4jvPgPHbrTygsHtFvdQ92Fds2c/bXr/e6DBEBdl7wIRkpn/053Ro2zn15fMyv//Bxa0n1f/b6TSFjyAPTYn59L8Xrd+/19b34b9+85pX3nHMze3OMl2NwRgJl7d6XRz/7TMAxs8uJtPJAStpA1OapRQvf4Z677+QPN/2FV/79EuvWruaSyy7vdN+62lqeXfAU511wYa+u8Zdb/kRmZiYXX3JZf5QsIjEy+fEp/HJ6OeePrcYMwm2wrTmFVbVpXHng9phf/80dmUzKbaE4I4TPoDFkPL4xn++/PzLm1/ba5Men8H+HlXH6qBr8Hn73Rekh/L6BvL5h5uOtnbnsn9NEUVor/gT8b+9lwOmsH6TT5iTn3G3AbRBpwemvAh5cUsmXDh7aX6frVjgcxu/39+qYY48/gWOPP2Gv2+vqannowft7HXBEJDFsbQrsft0UMlL9jifL8vivhaMHrIb/O3wjl02sJNgGGX5HXmqIbc2B7g9McLOG1XPSiFoM77/7ZhcgYMG9Xt9SUiO/ApHffdHfP/N5h/edfQ7ggi1gPn6eci8X8hLNLoV0f4jQ6BnUV0+C5W8O2HfQV14GnHJgVLv3pcDmgSzgHx9V9VvA2bypnG9+4/8x9aCDWbniY0aPGcvPbvwV551zOmee/UXefusNzr/gIvLy8vjLLX+itTVI6ahR/OSGn5OZmcWbr7/G//765+QXFHDAgZN3n/fJx//J8mVL+d4PfkxlZQW/uOGnbCqPNHxd+8Of8MB997CpfCMXnnc2Rxw5i6uu/i5/u+sOXnj+GVpbWzn+hJP4zyu+BcAdt93K008+xvDhJeQXFHDg5Cn9cu8iElvjslvZ0BDg/JfH89WJFQzPCA3o9YvSQ9y+aih3flLIdQdv4eSSOv53ZjnXfTCC5nDyTaeWnRLmtzPLmVXUwMKKLD6uSePOTwoH7rs3H/7MXPxZeZQMqWR+5UgezpnH+c3/ZNSoRobkXrxncPGn4EIhXKgVF2yJ/B5qpS3UigsGd793oVbCDTW0hVpwwV2fRba3BT/dh7bwp9/FMWu5rWkIj4/9Dmet/y3DMxbSsLwy9t9BP/Ay4DwBXGlmDwBHADXdjb+JdxvWr+NH19/ItEOnc/2Pf8BDD94HQGpqGnfcfR/VVVVc8+1vcsttd5GRmcn8O2/n3r/N58uXfo0br/8Rf/7rfEaNHsP3r/l2p+f/7S9vZPqMw/jtH24iHA7T1NjIN6/6b9Z88gn3PfQYAG+/+TplG9dz930P4Zzjv7/1Dd5ftJCMzAyef/Zp7vvHo4TCYf7jS19QwBFJEM9vySXV5/ioOpNvD2DrwS7zXv10vMcFr04gPzXEHw8v4/VTV3LJ62NZWp0x4DXFymFDG7hr9npe2ZrDUQsm0RD6tNW9P797X3oW/qw8/Fn5kd8z8/BnR373pWfR1lyPA76ZdX7kAAfXp0X+bghWLaZx1cJ2wSV2Tzft+m9fPGYML3vw/96+iFnAMbP7geOAQjMrB34CBACcc7cCC4DTgNVAI3Bpf137i/eu7vd9H7lov273KR5ewrRDI0+7n3b6mTxw398B+NzcUwH4aMmHrF27msu+EulOCgaDHHTINNavW8uIkaWMHjMWgFNPP5NHH3nwM+df+O7bXH/jrwDw+/1k5+RQW1uzxz5vv/kGb7/1Bhedfw4AjY2NbNy4gcaGBo4/8WTSMyI/iI457vge3beIeG9qfhOPbsz3uozdqltT+PLrY5k3rooFJ63m10uLuXnFMFynIw8Sg88c10zZxjcm7eCqd0fxeFn+HtuzJs+ioRfdMpaSGg0w7ULMrl+ZebhQK6GGGtoaagg31hCs3ERz2XJC9TW0NdWBa9vjfMXnXsO2h3/TH7faJ/XL3/Ds2n0Vs4DjnJvXzXYHXBGLa/ckjEAk3PR0356wDo9X73qfkZEJgHOOI46cxc9//bs99lu54uN+e8rd4bjkssv54nkX7PH5fX+/O4F/9IgMblPym7lhcby1khj3rxvCWzuyuOvoDZwyopbL3xrDlqbEG5szOquFu47eQHPYmLVgEpubUj+zT/bko/cMOO26kT4TYLLyMH/KpwGmoYZQfRWt29YTjr534cSaU6Y34S5eJF/nqYe2btnMksUfAPDcM0/vbs3Z5aCDp7H4ww8o2xhZ9b25qYkN69cxdtx4Nm3aRHnZxuixT3V6/sOOOIqHH7wfiAxYrq+vJzMri8bGht37HDVrNk88+s/dn23fto2dlZVMnzGTf7/0As3NzTQ01PPaKy/3672LSGxk+NsozWxlVW2616V0an19Gic9P5G3K7J487QVnFFa7XVJvXLB2J28fuoqnijL4/Mv7rc73FggDX/uUFKLx5I+JjLTSe6MuRQc+yUKT72corOvouCY88g84AgCQ4bTFmqhZfMn1H34IhXP38X2x/6Pnf+aT/Wbj1K3+CWaVr9Py5Y1hGor+hRuErEFxWtJsVRDX51/UEG/nm/c+Ak89cRj/PxnP2HU6DGce/48Hrz/nt3bC4YM4ac3/ILrvnc1ra2tAHzjyv9izNhxXPfjn3HVFf9JfkEB0w6dwZrVqz5z/u987wfceP2PefzRh/H7/Vz7w59w8CGHcsi0Qzn/nDM4evYcrrr6u6xbu5ZL/yPSgpOZmckNv/gNB0yewsmnnMaF551NyYgRTJs+4zPnF5H4c2BeM6vr0gi5+G2DDTvjxiUlvLg5hzuP3sDnRtbyvUUjaQz37qnRrvS2i2ivzIcvPYuCnHR+d/AKpuXV8KWN57MsZzz5x2bjS8/Bn5GFcw7awvjSMncfmjHuIAAaPnmP+iUvf6YbKZYSsQXFazGd6C8W4nWiv82byvmvK7/BPx590tM6+oMm+hOJHxePr+SEkjoufWOs16X0SE4gzO9mlnP4sAYueX0sH+zM7P6gHujJGBQLpOHLyMGfkY0vIxt/Rg6+9Ozo+xx8Gdn4UtOZEVrC/2X+lRca9uOn5bOob2ymramecFM9bU11kQG+HQbuej0GZrBLtIn+RESkG1MLmlhWHZ/dU52pC/r5f2+N4bwxVTx2whr+uLyI339cRNu+tEBFBykGho6IBpZoWMn49LU/PRvn2mhrqosGlXrCTXWEanbQunUd4eY6fM21XLv/Wi7Zr4IrXhvNM5tCwKv9c+MSdxRw+smIkaVJ0XojIvFlSn4zN60Y5nUZvfbQhgLe3pHFHUdv4OQRtXztzTGUN3528C4A/kBkwG5mTuRx6axcfJm5pBaW4s/M3b3bkOMvAiBYuZnmzZ8QqtlBuKmOtmig6Wpsy/jsFuYfu56dLX6OevqAXk9UqDEwiUcBR0Qkjk3Jb2JZVXw8QdXbcTBljanMfWE/rj5oJ2+c/gnXfDKTpxom48vMjQSZzBz8mbmYP0C4sZZwU11kIrrGWlq3rqdp7WLCjbW0NdVT/MWr+9hF5Lh4/E5unL6ZX3w0nD+vLKTzifS7pjEwiUcBR0QkThWmBUn3O8oa4+PR6888Kg1gFhn3kpm7+5ev3Wt/Zi7zwyEW1S3lT/v/jVPqa/j+mpnUVGyKhJrGWlxLY0zqLUgN8acjytg/t5m5/9qP5TXxERRlYCjgiIjEqSn5zSyvTqcvLQ79yQJpBAqGA5A1ZfbubiR/Zm501t2G3WGlrbGWUPU2WjZ/svu9CwX5N3BEykh+PWMTLx1wP5e+PpaFVVk9rqG3XUTHFNfx11kbeGxjPpe9MYaWNs2KMtgo4IiIxKmp+U2eLIPgz84nMHQkgaEjSR8xEV/6p09CZR94FABN6z+i5t0Fnc66uzcNIT9XvDOas0ZV89Bxa/nzymH8ZllxjwYg97SLKOBr48cHb+HC8VX851ujeWFLbvcHSVJSwPHAooXvEAgEOGTa9O533os5R0zntXfe78eqRCTeTClo5sOdMQ44Pj+BguHRQDOC1KEjcG1hghWbaK3cRNXaDwnV7ADn+u1R6cfL8llYkcntsyIDkL/6xhg2NqTt83kn5jYz/+j1bGpM5YinJ1HREh9de+KNQRtwUpoqGP/udaw94kZC6YUDeu33Fr5LZmbmPgUcEUl+U/ObuHfNkH49py8tMxJmCiOBJpA3jFDtzuhaSB9T9+GLkVaZGNvclMrnX9yPbx24nddOXcX3Fo3kgfV9vVfHZRMr+ckhW7h+cQl3fDIUr7v1xHuDNuCUfHwH2RUfUrL8Dsqmf69fznn1VVewbesWWltbueCii/nCuV/izddf4+Y//p62tjD5+QX86Pr/4Z8PPYjP7+OZp5/kmmt/yOOPPszsY47jpM/NBT5tnWlsbODqb11BbW0toVCQb3zzvzju+BP7pVYRiW+G48C8ZpbV7NscOCm5hXsEGl9qBsHKzQQrN1G/9DWCO7dCD5cO6O9HpR3G/31czMtbc5g/ez2njKzlqndHURvs+QzIhWlBbjmyjNKsVk56fmLcLmkhA2/QBZxDH52Nr6119/uidY9QtO4R2nypfHDO6/t07h//7Eby8vJpbm7my/PO49jjT+TG63/EbXfdw8jSUmpqqsnLy+cL532JzMxMLr7kMgAef/ThTs+XmprGb/5wE9nZ2VRXVXHJf3yJY4874TOLeopI8hmb3Up1q5/q1p7/mDZ/gMDQkt3jZwJDSmhraYwEmopyGla+Q7i2ss81xepR6cVVmcxacAA/n76Jd09fwVffGMObO7K7Pe7EklpuO2oj968r4KLXxhLUQGJpJykDzoxHDu/1Mb621i6Pe++L73Z7jgfu/Tsvv/QCANu2beHRh//BoTNmMrK0FIC8vPzeFeUcN//xd3zw3iJ8Ph87tm+jsrKCwsLEm/RLRHqnswHGHeeh8WXkkFo4cnegSckpIFi9nWDlZprWfEjNwgUxewS7vzWFfXx74Sie31zDPcesY/7qofx8SUmna3Cl+dq44dDNnDO6mq++MYZXtuV4ULHEu6QMON2FkdHv/5LCdY/ifAGsLciOcV/Y526qRQvf4d133uKuvz9AekYGl3/1YibufwAb1q/r9li/P4Vda4I55wgGI83Fzyx4kuqqKu554BFSAgHOmHsCrS0t+1SniCSGKQXNey7RYD6yJx9NW2szqdFAg89HsHJTZGbfjR8TrN4GbWHviu4Hz2zK46inD+AvR23gpVNWccnrY2kM+/jb7HVc/No4hqaFmD97Patr0zn86QOo6kULlwwug/L/jJSWnewY/wUqxp1D4bpHCTT3vcl2l/r6enJzcknPyGD9urUsXbKYYLCV9xctZFN5+R5dVFlZWTTU1+8+tmTkSD5evoyTTzmVV/79IqHoIm/1dfUUDBlCSiDAonffZsvmzftcp4gkhqn5TTxVnrf7/dDPXQpASs5QWraspX7pa4Qbqj2qLra2NQc4+98T+MakCl6eu4plVenMKmrgnjnrmJTXwnXvj+Dva4eggcTSFa0m3k9aW1v5zlVXsH37NsaMHUdV1U4u/8aVtDS3cPOffo9ra6NgyFBuue1ONqxfx/euvgqfz8c11/6Q0WPHcvW3rqDNtXH4EUfy4H338to771NdVcW3v/l1QqEQ+086kMUfvs8fb7mNESNLY/qYuFYTF/HeB2cs58uvjWXtiJPInnz0Z7bXL38j6ZcP2HnBh2SkfPbvqKaQMeSBaQNfkHimL6uJK+DIZyjgiHgrzdfG1i8tofjBg2lt85Ex7mACQ0eSMXZqv8xDkyiGZwT5xfRNnDmqmswUR2PIeHxjPt9/f2SvF8uUxNaXgKMh5yIiceaAvGbW1qXRGn0qKFBYSrCi3OOqBt7WpgC1QR9pfkdTyEjzO2qDPoUb6REFHBGRODMlv2mPAcaphaW0VpT3+zw0iaAoPcTtq4Zy7LP7c/uqoRRnhLwuSRJEUgwybnMOnAPND7PvnIt8nyLimakFzbsfEfdl5IA/hXB9VdKPuenMvFfH73797YWjPaxEEk1StOCs2biFULAlEnKk75wjFGxhzcYtXlciMqhFWnAiASe1sJRgxSaPKxJJPEnRgnPjzfdz3RXzmDC6BJ9acfqszTnWbNzCjTff73UpIoPa1PxmllZFuqgG6/gbkX2VFAGnuraea35xu9dliIjss4LUEDmBMBsbUoFIC07NuiUeVyWSeJKii0pEJFns6p5yGJaaji8zh1DNdq/LEkk4CjgiInFkan4zS6NPUKUOLSVYuVnjC0X6QAFHRCSOTCloYllVZIBxYJjG34j0lQKOiEgc2aMFp3AkrQo4In2igCMiEjcck/MiY3DMH8CfW0hwp6ZtEOkLBRwRkTgxOquV+pCfqtYUAkNHEKreDm1hr8sSSUgKOCIicaJ995TmvxHZNwo4IiJxYs8ZjDX+RmRfxDTgmNlcM1tpZqvN7NpOtueZ2ZNmttjMlpnZpbGsR0Qknk0taIrMYGw+UgpKIo+Ii0ifxCzgmJkfuBk4FZgMzDOzyR12uwJY7pw7BDgO+F8zS41VTSIi8WxKfjPLqjMIFBQTbqjGBVu8LkkkYcWyBedwYLVzbq1zrhV4ADirwz4OyDEzA7KBnUAohjWJiMSlgK+N8dktrKhJ1/gbkX4Qy4AzEihr9748+ll7NwEHApuBj4CrnHNtHU9kZpeb2SIzW+TCwVjVKyLimUm5LWxoSKWlzUdqYSmtO8q6P0hE9iqWAaezZb07zjd+CvAhMAKYBtxkZrmfOci525xzM51zM80f6O86RUQ8N7XdAOPA0JEEKzZ5XJFIYotlwCkHRrV7X0qkpaa9S4F/uojVwDrggBjWJCISl6bkN7G0OgN/biFtrc20tTR4XZJIQotlwFkITDSzcdGBwxcAT3TYZyNwIoCZFQOTgLUxrElEJC5NKWhmWVU6qRp/I9IvUmJ1YudcyMyuBJ4D/MCdzrllZvb16PZbgRuA+Wb2EZEure855ypiVZOISLyaGm3BSZ1cSsu29V6XI5LwYhZwAJxzC4AFHT67td3rzcDnYlmDiEi8ywuEKEgNs74+laGFpdQve93rkkQSnmYyFhHx2JT8Zj6uSccy88GMcEO11yWJJDwFHBERj+0aYKzxNyL9RwFHRMRjU/IjA4wDWn9KpN8o4IiIeGxqgVpwRPqbAo6IiKccU/KbWd44BF96FqEaPUgq0h8UcEREPFSaGaQ5bNTmjiNYuYnPTvguIn2hgCMi4qEp0SUaNP5GpH8p4IiIeGhqfjPLqnfNYKz1p0T6iwKOiIiHphQ0saw2B3/OEIJVW70uRyRpKOCIiHhoSn4TKxlNqGobtIW9LkckaSjgiIh4JMUcE3NaWJN2gMbfiPQzBRwREY9MzG2mvDGV8NBxmv9GpJ8p4IiIeCQywDiDlILi6CPiItJfFHBERDwyJb+Jj1uGEa7diQsFvS5HJKko4IiIeGRKQRMrbRytleqeEulvCjgiIh6Zmt/MJ2mTCe5QwBHpbwo4IiIeyE4JMyw9xKa8qbRq/I1Iv0vxugARkcFoSn4zK+uzCbU24VoavS5HJOmoBUdExANT8ptY0Vqs+W9EYkQBR0TEA1Pzm1hpY7X+lEiMKOCIiHhgSkEzn6RPUQuOSIwo4IiIDDjHlIJmVoRKaGus8boYkaSkgCMiMsBKMkI487NpR63XpYgkLQUcEZEBFhlgPJxgxWavSxFJWgo4IiIDbGp+Eyt94zT+RiSGFHBERAbY1KEhVroxhGsrvC5FJGkp4IiIDLCpQ4MsrUr1ugyRpKaAIyIygPzmmJhVx9ItIa9LEUlqCjgiIgNoQk4L29vyqN6+zetSRJKaAo6IyACKjL8ZTbBKAUcklhRwREQG0MElfpY35oNr87oUkaSmgCMiMoCmDgmytDLgdRkiSS+mAcfM5prZSjNbbWbX7mWf48zsQzNbZmavxLIeERGvTc6uZokGGIvEXEqsTmxmfuBm4GSgHFhoZk8455a32ycfuAWY65zbaGZFsapHRMRrmQFjuL+Oj8vqvC5FJOnFsgXncGC1c26tc64VeAA4q8M+FwL/dM5tBHDObY9hPSIinjp4dCZrQoWEQ2rBEYm1WAackUBZu/fl0c/a2x8oMLOXzew9M/tyZycys8vNbJGZLXLhYIzKFRGJrYNLUlhWn+d1GSKDQsy6qADr5DPXyfVnACcCGcBbZva2c27VHgc5dxtwG4AvPafjOUREEsLUIa0s3aYBxiIDIZYtOOXAqHbvS4GOS+eWA8865xqccxXAq8AhMaxJRMQbZkzOrGLJllavKxEZFGIZcBYCE81snJmlAhcAT3TY53FgjpmlmFkmcATwcQxrEhHxREpeEZN8ZSzd4fe6FJFBIWZdVM65kJldCTwH+IE7nXPLzOzr0e23Ouc+NrNngSVAG/BX59zSWNUkIuKVkSVD8LeF2NykLiqRgRDLMTg45xYACzp8dmuH978BfhPLOkREvHZQiZ/l9bl0PjxRRPqbZjIWERkAUwta+Kgypv+mFJF2FHBERGLMnzOEA6yMpRUafyMyUBRwRERiLLWwlP3depZVpXtdisig0eOAY2ZZsSxERCRZpRWOYP/UHSyryfC6FJFBo9uAY2azzGw50ce3zewQM7sl5pWJiCSJ/YrSqGhJoS6oLiqRgdKTFpzfA6cAlQDOucXAMbEsSkQkWfgysjkwdRtLd6Z6XYrIoNKjLirnXFmHj8IxqEVEJOmkFpYyseVjllWre0pkIPUk4JSZ2SzAmVmqmX0HzTYsItIjgcJRTGI9y6o1wFhkIPUk4HwduILISuDlwLToexER6UZq4UgOTN3O0iq14IgMpG5nnYougnnRANQiIpJULDWdjMx0StObWFWrFhyRgdRtwDGzuwDX8XPn3FdjUpGISJJIHTqS8fVLWB1KI+S0RIPIQOrJvOFPtXudDpwDbI5NOSIiySNQWMrEliUsa1b3lMhA60kX1SPt35vZ/cALMatIRCRJBApLmRR8QgOMRTzQl6UaJgKj+7sQEZGk4g8QyCvkwLQKluoRcZEB15MxOHVExuBY9PetwPdiXJeISEILDCkhWLODKUMaWaYnqEQGXE+6qHIGohARkWSSWlhKzs5PyBjmKGsMeF2OyKCz14BjZtO7OtA5937/lyMikhwCw0qZuGMBy6vTiTSAi8hA6qoF53+72OaAE/q5FhGR5GA+AgUlTKrYqPE3Ih7Za8Bxzh0/kIWIiCSLlIJiwg3VTMmpZ7HG34h4oifz4GBmU4HJRObBAcA597dYFSUikshSC0sJVpQztbiJ+9YO8bockUGpJ09R/QQ4jkjAWQCcCrwOKOCIiHQiUDiSlo3LOXBSM8tqNAeOiBd6Mg/OucCJwFbn3KXAIUBaTKsSEUlgqUNLGdG0hppWP9WtPWooF5F+1pOA0+ycawNCZpYLbAfGx7YsEZHE5M8dSluwhckZlRpgLOKhrh4Tvwm4H3jXzPKB24H3gHrg3QGpTkQkwewef1PQrCUaRDzUVdvpJ8BvgRFEQs39wMlArnNuyQDUJiKScAKFpbRu38DUUU08VZ7ndTkig9Zeu6icc//nnDsKOAbYCdwFPAOcbWYTB6g+EZGEsqsFZ0p+E8uq1IIj4pVux+A45zY4537lnDsUuBA4B1gR88pERBKMLzMXfH5SGncyNruVlbUKOCJe6TbgmFnAzM4ws3uJtOCsAr4Y88pERBLMrtabA/KaWVuXRmtbT57jEJFY6GqQ8cnAPOB0IoOKHwAud841DFBtIiIJJVBYSmu0e2qpBhiLeKqrQcY/AO4DvuOc2zlA9YiIJKzUwlKa1n7I1PHNLNUSDSKe0lpUIiL9wFIz8KVnEarewZT8Jv6ycpjXJYkMauogFhHpB6mFpQR3bgYcU/Ob1UUl4rGYBhwzm2tmK81stZld28V+h5lZ2MzOjWU9IiKxsmv8TUFqiOyUMBsbUr0uSWRQi1nAMTM/cDORxTknA/PMbPJe9vsV8FysahERibXUwlKCOyIDjJfXZADmdUkig1osW3AOB1Y759Y651qJPIV1Vif7fRN4hMgaVyIiCcdSAvhzhxCs2qruKZE4EcuAMxIoa/e+PPrZbmY2ksjEgbd2dSIzu9zMFpnZIhcO9nuhIiL7IjB0JKGqbdAWZkpBE8v0BJWI52IZcDprn3Ud3v8B+J5zLtzViZxztznnZjrnZpo/0F/1iYj0i0DhSForygHUgiMSJ7qaB2dflQOj2r0vBTZ32Gcm8ICZARQCp5lZyDn3WAzrEhHpV6mFpTSseAdwTM5rYlm1WnBEvBbLgLMQmGhm44BNwAVE1rLazTk3btdrM5sPPKVwIyIJxecnpWA4wcrNjM5qpT7kp6o1lj9aRaQnYvan0DkXMrMriTwd5QfudM4tM7OvR7d3Oe5GRCQRBAqKCddV4UKtTC1W95RIvIjpPzOccwuABR0+6zTYOOcuiWUtIiKxEIgusAkwJV/dUyLxQjMZi4jsg9ToBH8AUwuaWFqlFhyReKCAIyLSZ0Zg6Mh2LTjNasERiRMKOCIifZSSV0hbSyNtLY0EfG2Mz25hRY1acETigQKOiEgftR9/Mym3hQ0NqbS06ceqSDzQn0QRkT7aY/yNBhiLxBUFHBGRPur4BNVSBRyRuKGAIyLSB/6sfMARbqgBYEpBM8v0BJVI3FDAERHpg0Dhp09PQaSLSi04IvFDAUdEpA8i4282AZAXCFGQGmZ9farHVYnILgo4IiJ9ECgctcf8Nx/XpOMwj6sSkV0UcEREesmXloUvLZ1QzQ5AA4xF4pECjohIL0XG32za/X5KvgYYi8QbBRwRkV5qP/4GomtQqQVHJK4o4IiI9FL7+W/ARdegUguOSDxRwBER6QVLScWfU0CwaisApZlBmsNGRUvA48pEpD0FHBGRXggUjiS0cyu4NiAywFhLNIjEHwUcEZFeaL/+FMBUdU+JxCUFHBGRXthz/A1MKWhiaZVacETijQKOiEhP+fwE8osIVm7e/ZG6qETikwKOiEgPBYaUEKqtxIWDAKSYY2JOCx/XqItKJN4o4IiI9FDH8TcTc5spb0ylKawfpSLxRn8qRUR6qOP4Gw0wFolfCjgiIj1iBIaO2GMGY61BJRK/FHBERHogJX8YtIVxrU27P5tS0MQyPUElEpcUcEREeiBQWIovLXOPz6bmN7NUXVQicUkBR0SkB1ILS/d4n50SZlh6iLX1aR5VJCJdSfG6ABGReJY1eRbZk4/e/b743GsAOGDDI6yo+YQ2Z16VJiJdUMAREelCw/I3CQwZQWtFGTlTj2Hbw78B4PT9KliWovE3IvFKXVQiIl1IHz0FX3oWjSsX7vH5VM1gLBLXFHBERPbCl5ZJzsHHUrvoGXBt1C9/Y/e2KQXNLK3SAGOReKWAIyKyFznTTqRp/VJC1duBSHdVhNMaVCJxTgFHRKQTaSP2IyW/iPrdoeZTJRkh2pyxrVnDGEXiVUwDjpnNNbOVZrbazK7tZPtFZrYk+utNMzsklvWIiPSEBdLImXYSte89B22hz2yPtN6kA3qCSiRexSzgmJkfuBk4FZgMzDOzyR12Wwcc65w7GLgBuC1W9YiI9FT2QcfSsmX1HutOtacBxiLxL5YtOIcDq51za51zrcADwFntd3DOvemcq4q+fRsoRUTEQ6lFo0kbPo76j17d6z5TNIOxSNyLZcAZCZS1e18e/WxvLgOe6WyDmV1uZovMbJELB/uxRBGRdvwBcqafQu37/8KFWve6m9agEol/sRwh11nntOt0R7PjiQSc2Z1td87dRrT7ypee0+k5RET2VfaUowlWbqZ169q97uM3x6TcZpbXqAVHJJ7FsgWnHBjV7n0psLnjTmZ2MPBX4CznXGUM6xER2auUISWkj55M3eKXutxvQk4LW5sCNIT8A1SZiPRFLAPOQmCimY0zs1TgAuCJ9juY2Wjgn8DFzrlVMaxFRGTvzEfejFOo+/AlXGtTl7tqgLFIYohZF5VzLmRmVwLPAX7gTufcMjP7enT7rcCPgaHALWYGEHLOzYxVTSIinck64EjCDTW0lK/odl8NMBZJDDGdpco5twBY0OGzW9u9/hrwtVjWICLSFX9uIZn7HUrlC3f3aP+p+U38Y31BjKsSkX2lmYxFZBAz8maeQv3S12hrqu/REVMKmliqLiqRuKeAIyKDVubE6bhQiKZ1S3q2vz9MSUaQ1XVpMa5MRPaVAo6IDEr+rDyyDjgyshxDD03Ob2ZVbTphpyUaROKdAo6IDEo5M06hYeW7hBuqe3zMlPzm6BpUIhLvFHBEZNBJH3sQvpQ0Gj9Z1Kvj9Ii4SOJQwBGRQcWXnkXOQcdQ+96z4Ho3MfqU/CaWaokGkYSggCMig0rOoSfTuPZDQjU7en3slAJ1UYkkCgUcERk00kbuT0rOEBo+frvXxxalB0kxx+amQAwqE5H+poAjIoOCpaaTM+3ESNdUW7jXx0/ZPf5GT1CJJAIFHBEZFHIOPp6W8pUEKz+z5m+PHDWsgQk5zRSnB/u5MhGJBQUcEUl6qcVjSR1WSv3S1/p8jvPGVjE8I8T3D9rSj5WJSKzEdC0qERGvWUqA3Omfo/b953Hh3re+7LzgQzJSPn3a6j8nVfKfkyppChlDHpjWj5WKSH9SC46IJLXsqcfQumMjrdvW9+n4i18bS33QCLZF3jeGjPvXFnDgY1P6r0gR6XcKOCKStAJDR5I2cn/qFr/c62P95vjhwVu46cgy3tiejc+gKWSk+R21QR/bmvU0lUg8U8ARkeTk85M74xTqPnwRF2zu1aFjs1t48XOrOLywgaMWHEBT2Mftq4Zy7LP7c/uqoRRnhGJUtIj0F43BEZGklHXgUYRqK2nZtKoXRzkuHLeTX87YzK+XFnPzimE4jHmvjt+9x7cXju7/YkWk3yngiEjSSckrInPcwVS+cHePj8lPDfHHw8uYnN/MaS/sx1KtOSWS0NRFJSLJxYzcmadQ99GrtDU39OiQ2UV1vHPaCrY3B5j9zCSFG5EkoBYcEUkqmRMPo621meYNS7vdN+Br44cHb+Xi8ZV8/e3RPL85bwAqFJGBoIAjIknDn11A1qTDqHzxnm733S+nmfmzN7C1KYUjnj6AHS16KkokmaiLSkSSRu6MU6j/+C3aGmu62MtxyX4VvHTKJ9y9egjnvjxe4UYkCakFR0SSQsb4Q8Dnp2n1B3vdZ0hqiFuO3MjY7FY+96/9WFGjsTYiyUotOCKS8HwZOWRPmU3tomcB1+k+xw+v5d3TV7CuPo1jnt1f4UYkyakFR0QSXu70k2lc/T7husrPbEv1tXH9tC2cN6aK//fWaP69NdeDCkVkoCngiEhCSx91IL6MXBpWPPaZbQfkNTH/6A2sr0/l8KcPYGerfuSJDBbqohKRhGWpGWQfcjy17z0Lrq3dFsfl++/g+ZNXc+uqQi54dZzCjcggoz/xIpKwcqadQPPG5YSqtu7+bFhakFuP2khxRpATnpvI6rp0DysUEa+oBUdEElJqyQQCQ0qoX/b67s8+N6KGd05fwdLqDI5/bn+FG5FBTC04IpJwLCWV3ENPombhAgiHSPe3ceOhm/l8aTVffn0sr2/P8bpEEfGYAo6IJJzsg46lZes6gjvKmJrfxPzZ6/m4Op0jFhxAtcbaiAjqohKRBBMYNoq0kgk0fPQyVx6wnQUnreZ3y4q4+PWxCjcisltMA46ZzTWzlWa22syu7WS7mdkfo9uXmNn0WNYjIgnOn0Lu9FPIXPYEj81Zzrljqjj22f25b91QwLyuTkTiSMwCjpn5gZuBU4HJwDwzm9xht1OBidFflwN/jlU9/WF4RpAXztpGcXpQ19f1dX0Prv/SORV8Ifwsr8x8jncqsjjx+f1ZV5/mST0iEt9i2YJzOLDaObfWOdcKPACc1WGfs4C/uYi3gXwzK4lhTfvk2qlbOCp7C98/aIuur+vr+gPsx4dVcWTaOv5nyNPMe2UcNy4pIezUaiMinTPnOl+3ZZ9PbHYuMNc597Xo+4uBI5xzV7bb5yngl86516PvXwS+55xbtLfz+tJzXFrpjJjUvDc7L/iQjJTPfk/OwU6XHfPrD7F6rJOf47q+rj+Yr98UMoY8MC3m1xcR7zWveeU959zM3hwTyxF5nf3TqmNK6Mk+mNnlRLqwIGXgm6NnLv8KN05exSn+RWRYK00uwEvhQ/nF6v3ZuHJlzK8/ZtIkvr/fSo73f0iGBXV9XX+QXn9x9M9fKs+GD+O65ROBxTG/vogkplgGnHJgVLv3pcDmPuyDc+424DaItOD0b5ndW7tkMdvTN5I6sZVmFyCVIOVr1vDeooEZi7Bj0TLKfRtJnRjU9XX9QXz9XX/+Wtm+ZgXrljQOyPVFJDHFcgzOQmCimY0zs1TgAuCJDvs8AXw5+jTVkUCNc86bDv5uFKWHuH3VUM5p/Rm3rxpKcUZI19f1df1Bcn0RSTwxG4MDYGanAX8A/MCdzrkbzezrAM65W83MgJuAuUAjcGlX42/AmzE47WVNnkXD8jd1fV1f1x+E1xcRb/RlDE5MA04seB1wREREZGD1JeBoJmMRERFJOgo4IiIiknQUcERERCTpKOCIiIhI0lHAERERkaSTcE9RmVkdEPvpU+NXIVDhdREe0v3r/nX/g9NgvnfQ/U9yzuX05oBYzmQcKyt7+6hYMjGzRbp/3b/XdXhF9z94738w3zvo/s2syznyOqMuKhEREUk6CjgiIiKSdBIx4NzmdQEe0/0Pbrr/wW0w3/9gvnfQ/ff6/hNukLGIiIhIdxKxBUdERESkSwo4IiIiknQSKuCY2VwzW2lmq83sWq/rGUhmNsrM/m1mH5vZMjO7yuuaBpqZ+c3sAzN7yutaBpqZ5ZvZw2a2Ivr/wFFe1zSQzOzb0f/vl5rZ/WaW7nVNsWRmd5rZdjNb2u6zIWb2LzP7JPp7gZc1xtJe7v830f//l5jZo2aW72GJMdXZ/bfb9h0zc2ZW6EVtA2Fv929m34xmgGVm9uvuzpMwAcfM/MDNwKnAZGCemU32tqoBFQKuds4dCBwJXDHI7h/gKuBjr4vwyP8BzzrnDgAOYRB9D2Y2EvgWMNM5NxXwAxd4W1XMzQfmdvjsWuBF59xE4MXo+2Q1n8/e/7+Aqc65g4FVwPcHuqgBNJ/P3j9mNgo4Gdg40AUNsPl0uH8zOx44CzjYOTcF+G13J0mYgAMcDqx2zq11zrUCDxC52UHBObfFOfd+9HUdkb/gRnpb1cAxs1LgdOCvXtcy0MwsFzgGuAPAOdfqnKv2tKiBlwJkmFkKkAls9riemHLOvQrs7PDxWcDd0dd3A2cPZE0DqbP7d84975wLRd++DZQOeGEDZC///QF+D3wXSOqng/Zy/98Afumca4nus7278yRSwBkJlLV7X84g+gu+PTMbCxwKvONxKQPpD0T+YLd5XIcXxgM7gLuiXXR/NbMsr4saKM65TUT+tbYR2ALUOOee97YqTxQ757ZA5B88QJHH9Xjpq8AzXhcxkMzsTGCTc26x17V4ZH9gjpm9Y2avmNlh3R2QSAHHOvksqVNsZ8wsG3gE+C/nXK3X9QwEM/s8sN05957XtXgkBZgO/Nk5dyjQQHJ3T+whOtbkLGAcMALIMrP/8LYq8YqZXUeky/5er2sZKGaWCVwH/NjrWjyUAhQQGaJxDfAPM+ssF+yWSAGnHBjV7n0pSd5M3ZGZBYiEm3udc//0up4BdDRwppmtJ9I1eYKZ3eNtSQOqHCh3zu1qsXuYSOAZLE4C1jnndjjngsA/gVke1+SFbWZWAhD9vdsm+mRjZl8BPg9c5AbXJG4TiAT8xdGfg6XA+2Y23NOqBlY58E8X8S6R1vwuB1onUsBZCEw0s3FmlkpkkOETHtc0YKJJ9Q7gY+fc77yuZyA5577vnCt1zo0l8t/9JefcoPkXvHNuK1BmZpOiH50ILPewpIG2ETjSzDKjfw5OZBANsm7nCeAr0ddfAR73sJYBZ2Zzge8BZzrnGr2uZyA55z5yzhU558ZGfw6WA9OjPxsGi8eAEwDMbH8glW5WV0+YgBMdXHYl8ByRH27/cM4t87aqAXU0cDGR1osPo79O87ooGTDfBO41syXANODn3pYzcKItVw8D7wMfEfm5ldTT1pvZ/cBbwCQzKzezy4BfAieb2SdEnqT5pZc1xtJe7v8mIAf4V/Tn362eFhlDe7n/QWMv938nMD766PgDwFe6a8XTUg0iIiKSdBKmBUdERESkpxRwREREJOko4IiIiEjSUcARERGRpKOAIyIiIklHAUdEMLProiv0Lok+gntEL4+/xMxG9PKYsXtZLXmsmTVF61huZreaWZ9/VpnZy2Y2M/p6QVerUJvZ2e0XsTWzn5nZSX29toh4J8XrAkTEW2Z2FJHZYac751rMrJDIJFo9Pd4PXAIspf9mF1/jnJsWXVzzJSILS+6evdvMUtotvNhjzrnu5o46G3iK6ESKzrnBPDW+SEJTC46IlAAV7VbprXDObQYwsxOjC3x+ZGZ3mlla9PP1ZvZjM3sdmAfMJDIR4YdmlmFmM6IL4r1nZs+1W2JghpktNrO3gCu6KywaYt4E9ou2Ej1kZk8Cz5tZVrSmhdEaz4peI8PMHoi2Rj0IZOw6X7TuwujrL0f3WWxmfzezWcCZwG+i9zHBzOab2bk9+C6uN7P3o9sO6If/JiKyjxRwROR5YJSZrTKzW8zsWAAzSwfmA19yzh1EpMX3G+2Oa3bOzXbO3QMsIrI+0DQiCyH+CTjXOTeDyAykN0aPuQv4lnPuqJ4UZpFFBk8kMoMxwFFEZjA9gcjigy855w4DjicSTLKiNTY65w6OXndGJ+edEj3+BOfcIcBVzrk3iSyHcI1zbppzbk27/bv7Liqcc9OBPwPf6cm9iUhsKeCIDHLOuXoiIeByYAfwoJldAkwissjlquiudwPHtDv0wb2cchIwleiU+sAPgVIzywPynXOvRPf7exdlTYge+wbwtHPumejn/3LO7Yy+/hxwbXS/l4F0YHS0xnui97YEWNLJ+U8AHnbOVUT329nJPh3vqavvYlf32XvA2G7OJSIDQGNwRATnXJhISHjZzD4ispjjh90c1rCXzw1Y1rGVJjq4t6drw6yJtgZ1dU0DvuicW9nhOvTgOtaLWnbt35WW6O9h9HNVJC6oBUdkkDOzSWY2sd1H04ANwApgrJntF/38YuAVOldHZCFEgJXAsOjgZcwsYGZTnHPVQI2ZzY7ud9E+lv4c8E2LJhozOzT6+au7zm1mU4GDOzn2ReB8Mxsa3W9IJ/fRXm++CxGJAwo4IpIN3B19JHsJMBn4qXOuGbgUeCjaqtMG7G0F5/nArdHuIj9wLvArM1tMpCVoVnS/S4Gbo4OMm/ax7huAALAk+rj5DdHP/wxkR+/lu8C7HQ90zi0jMj7nlWiNv4tuegC4JjqYeEK7/XvzXYhIHNBq4iIiIpJ01IIjIiIiSUcBR0RERJKOAo6IiIgkHQUcERERSToKOCIiIpJ0FHBEREQk6SjgiIiISNL5/wG1cgpTZ773qQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 576x576 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lift_chart = model_top.get_lift_chart(\"validation\")\n",
    "\n",
    "# Save the result into a pandas dataframe\n",
    "lift_df = pd.DataFrame(lift_chart.bins)\n",
    "\n",
    "bin_counts = [10, 15]\n",
    "f, axarr = plt.subplots(len(bin_counts))\n",
    "f.set_size_inches((8, 4 * len(bin_counts)))\n",
    "\n",
    "rebinned_dfs = []\n",
    "for i in range(len(bin_counts)):\n",
    "    rebinned_dfs.append(matplotlib_lift(lift_df, bin_counts[i], axarr[i]))\n",
    "\n",
    "plt.tight_layout()\n",
    "# plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROC Curve\n",
    "\n",
    "The receiver operating characteristic curve, or [ROC curve](https://docs.datarobot.com/en/docs/modeling/analyze-models/evaluate/roc-curve-tab/roc-curve.html), is a graphical plot that illustrates the performance of a binary classifier system as its discrimination threshold is varied. The curve is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "autoscroll": "auto"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1_score</th>\n",
       "      <th>false_negative_score</th>\n",
       "      <th>true_negative_score</th>\n",
       "      <th>true_positive_score</th>\n",
       "      <th>false_positive_score</th>\n",
       "      <th>true_negative_rate</th>\n",
       "      <th>false_positive_rate</th>\n",
       "      <th>true_positive_rate</th>\n",
       "      <th>matthews_correlation_coefficient</th>\n",
       "      <th>positive_predictive_value</th>\n",
       "      <th>negative_predictive_value</th>\n",
       "      <th>threshold</th>\n",
       "      <th>fraction_predicted_as_positive</th>\n",
       "      <th>fraction_predicted_as_negative</th>\n",
       "      <th>lift_positive</th>\n",
       "      <th>lift_negative</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.595890</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>59</td>\n",
       "      <td>87</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.595890</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.602740</td>\n",
       "      <td>0.033333</td>\n",
       "      <td>58</td>\n",
       "      <td>87</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.016949</td>\n",
       "      <td>0.100844</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.600000</td>\n",
       "      <td>9.982433e-01</td>\n",
       "      <td>0.006849</td>\n",
       "      <td>0.993151</td>\n",
       "      <td>2.474576</td>\n",
       "      <td>1.006897</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.609589</td>\n",
       "      <td>0.065574</td>\n",
       "      <td>57</td>\n",
       "      <td>87</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.033898</td>\n",
       "      <td>0.143109</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.604167</td>\n",
       "      <td>9.978805e-01</td>\n",
       "      <td>0.013699</td>\n",
       "      <td>0.986301</td>\n",
       "      <td>2.474576</td>\n",
       "      <td>1.013889</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.616438</td>\n",
       "      <td>0.096774</td>\n",
       "      <td>56</td>\n",
       "      <td>87</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.050847</td>\n",
       "      <td>0.175884</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.608392</td>\n",
       "      <td>9.974460e-01</td>\n",
       "      <td>0.020548</td>\n",
       "      <td>0.979452</td>\n",
       "      <td>2.474576</td>\n",
       "      <td>1.020979</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.623288</td>\n",
       "      <td>0.126984</td>\n",
       "      <td>55</td>\n",
       "      <td>87</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.067797</td>\n",
       "      <td>0.203807</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.612676</td>\n",
       "      <td>9.955975e-01</td>\n",
       "      <td>0.027397</td>\n",
       "      <td>0.972603</td>\n",
       "      <td>2.474576</td>\n",
       "      <td>1.028169</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>0.883562</td>\n",
       "      <td>0.874074</td>\n",
       "      <td>0</td>\n",
       "      <td>70</td>\n",
       "      <td>59</td>\n",
       "      <td>17</td>\n",
       "      <td>0.804598</td>\n",
       "      <td>0.195402</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.790330</td>\n",
       "      <td>0.776316</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.757119e-06</td>\n",
       "      <td>0.520548</td>\n",
       "      <td>0.479452</td>\n",
       "      <td>1.921053</td>\n",
       "      <td>1.678161</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77</th>\n",
       "      <td>0.876712</td>\n",
       "      <td>0.867647</td>\n",
       "      <td>0</td>\n",
       "      <td>69</td>\n",
       "      <td>59</td>\n",
       "      <td>18</td>\n",
       "      <td>0.793103</td>\n",
       "      <td>0.206897</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.779553</td>\n",
       "      <td>0.766234</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.139812e-06</td>\n",
       "      <td>0.527397</td>\n",
       "      <td>0.472603</td>\n",
       "      <td>1.896104</td>\n",
       "      <td>1.678161</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>0.869863</td>\n",
       "      <td>0.861314</td>\n",
       "      <td>0</td>\n",
       "      <td>68</td>\n",
       "      <td>59</td>\n",
       "      <td>19</td>\n",
       "      <td>0.781609</td>\n",
       "      <td>0.218391</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.768907</td>\n",
       "      <td>0.756410</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.501561e-07</td>\n",
       "      <td>0.534247</td>\n",
       "      <td>0.465753</td>\n",
       "      <td>1.871795</td>\n",
       "      <td>1.678161</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>0.863014</td>\n",
       "      <td>0.855072</td>\n",
       "      <td>0</td>\n",
       "      <td>67</td>\n",
       "      <td>59</td>\n",
       "      <td>20</td>\n",
       "      <td>0.770115</td>\n",
       "      <td>0.229885</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.758386</td>\n",
       "      <td>0.746835</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.079081e-07</td>\n",
       "      <td>0.541096</td>\n",
       "      <td>0.458904</td>\n",
       "      <td>1.848101</td>\n",
       "      <td>1.678161</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>0.404110</td>\n",
       "      <td>0.575610</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>59</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.404110</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.614430e-33</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>81 rows × 17 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    accuracy  f1_score  false_negative_score  true_negative_score  \\\n",
       "0   0.595890  0.000000                    59                   87   \n",
       "1   0.602740  0.033333                    58                   87   \n",
       "2   0.609589  0.065574                    57                   87   \n",
       "3   0.616438  0.096774                    56                   87   \n",
       "4   0.623288  0.126984                    55                   87   \n",
       "..       ...       ...                   ...                  ...   \n",
       "76  0.883562  0.874074                     0                   70   \n",
       "77  0.876712  0.867647                     0                   69   \n",
       "78  0.869863  0.861314                     0                   68   \n",
       "79  0.863014  0.855072                     0                   67   \n",
       "80  0.404110  0.575610                     0                    0   \n",
       "\n",
       "    true_positive_score  false_positive_score  true_negative_rate  \\\n",
       "0                     0                     0            1.000000   \n",
       "1                     1                     0            1.000000   \n",
       "2                     2                     0            1.000000   \n",
       "3                     3                     0            1.000000   \n",
       "4                     4                     0            1.000000   \n",
       "..                  ...                   ...                 ...   \n",
       "76                   59                    17            0.804598   \n",
       "77                   59                    18            0.793103   \n",
       "78                   59                    19            0.781609   \n",
       "79                   59                    20            0.770115   \n",
       "80                   59                    87            0.000000   \n",
       "\n",
       "    false_positive_rate  true_positive_rate  matthews_correlation_coefficient  \\\n",
       "0              0.000000            0.000000                          0.000000   \n",
       "1              0.000000            0.016949                          0.100844   \n",
       "2              0.000000            0.033898                          0.143109   \n",
       "3              0.000000            0.050847                          0.175884   \n",
       "4              0.000000            0.067797                          0.203807   \n",
       "..                  ...                 ...                               ...   \n",
       "76             0.195402            1.000000                          0.790330   \n",
       "77             0.206897            1.000000                          0.779553   \n",
       "78             0.218391            1.000000                          0.768907   \n",
       "79             0.229885            1.000000                          0.758386   \n",
       "80             1.000000            1.000000                          0.000000   \n",
       "\n",
       "    positive_predictive_value  negative_predictive_value     threshold  \\\n",
       "0                    0.000000                   0.595890  1.000000e+00   \n",
       "1                    1.000000                   0.600000  9.982433e-01   \n",
       "2                    1.000000                   0.604167  9.978805e-01   \n",
       "3                    1.000000                   0.608392  9.974460e-01   \n",
       "4                    1.000000                   0.612676  9.955975e-01   \n",
       "..                        ...                        ...           ...   \n",
       "76                   0.776316                   1.000000  3.757119e-06   \n",
       "77                   0.766234                   1.000000  3.139812e-06   \n",
       "78                   0.756410                   1.000000  1.501561e-07   \n",
       "79                   0.746835                   1.000000  1.079081e-07   \n",
       "80                   0.404110                   0.000000  1.614430e-33   \n",
       "\n",
       "    fraction_predicted_as_positive  fraction_predicted_as_negative  \\\n",
       "0                         0.000000                        1.000000   \n",
       "1                         0.006849                        0.993151   \n",
       "2                         0.013699                        0.986301   \n",
       "3                         0.020548                        0.979452   \n",
       "4                         0.027397                        0.972603   \n",
       "..                             ...                             ...   \n",
       "76                        0.520548                        0.479452   \n",
       "77                        0.527397                        0.472603   \n",
       "78                        0.534247                        0.465753   \n",
       "79                        0.541096                        0.458904   \n",
       "80                        1.000000                        0.000000   \n",
       "\n",
       "    lift_positive  lift_negative  \n",
       "0        0.000000       1.000000  \n",
       "1        2.474576       1.006897  \n",
       "2        2.474576       1.013889  \n",
       "3        2.474576       1.020979  \n",
       "4        2.474576       1.028169  \n",
       "..            ...            ...  \n",
       "76       1.921053       1.678161  \n",
       "77       1.896104       1.678161  \n",
       "78       1.871795       1.678161  \n",
       "79       1.848101       1.678161  \n",
       "80       1.000000       0.000000  \n",
       "\n",
       "[81 rows x 17 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc = model_top.get_roc_curve(\"validation\")\n",
    "\n",
    "# Save the result into a pandas dataframe\n",
    "roc_df = pd.DataFrame(roc.roc_points)\n",
    "\n",
    "roc_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.0, 1.0)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfkAAAHwCAYAAACluRYsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABJ/ElEQVR4nO3deXxke13n/9enKvu+diedPZWl0kv6ApfloihcBgWEQZRhVUeckWFUHPA3joz4wxmVGRX1CiriFYGBAREVBRFBXABnANm8SS9JuitrJ5100kknnX39zh9V4ebmZql0p3LqVL2fj0ceSVWdOvW5p/v2O9/vOZ/zNeccIiIiknoCXhcgIiIiiaGQFxERSVEKeRERkRSlkBcREUlRCnkREZEUpZAXERFJUQp5ERGRFKWQF/EZMxsys2UzWzCzCTP7oJkV7Nrm2Wb2D2Y2b2ZzZvZXZnZ21zZFZvbbZjYS21ck9rjiZP+LRCRRFPIi/vRS51wB8ADwFOC/br9gZg8Bfwt8EjgDNAFdwP81s+bYNlnA3wPngBcCRcCzgWngGYkq2swyErVvEXkyhbyIjznnJoDPEQ37bb8OfMg59y7n3LxzbsY59wvAV4H/FtvmR4B64OXOuavOuS3n3KRz7pedc5/Z67PM7JyZfd7MZszslpn9fOz5D5rZr+zY7rlmNrrj8ZCZ/ZyZdQOLZvYLZvZnu/b9LjN7d+znYjP7IzMbN7MxM/sVMwve35ESSU8KeREfM7Na4EVAJPY4j+iI/E/32PzjwAtiP/8r4LPOuYU4P6cQ+Dvgs0RnB1qIzgTE6zXA9wElwIeBF5tZUWzfQeCVwEdj2/4vYCP2GU8Bvgf490f4LBGJUciL+NNfmtk8cAOYBH4x9nwZ0f+vx/d4zziwfb69fJ9t9vMSYMI595vOuZXYDME/H+H973bO3XDOLTvnhoFvAd8fe+1hYMk591UzO030l5Y3O+cWnXOTwCPAq4/wWSISo5AX8afvd84VAs8Fwjwe3neALaB6j/dUA7djP0/vs81+6oD+e6o06sauxx8lOroHeC2Pj+IbgExg3MxmzWwW+APg1H18tkjaUsiL+Jhz7ovAB4HfiD1eBL4C/Js9Nn8lj0+x/x3wvWaWH+dH3QBC+7y2COTteFy1V6m7Hv8p8NzY6YaX83jI3wBWgQrnXEnsq8g5dy7OOkVkB4W8iP/9NvACM3sg9vitwL81s582s0IzK41dGPcQ8N9j23yYaKD+uZmFzSxgZuVm9vNm9uI9PuPTQJWZvdnMsmP7fWbstceInmMvM7Mq4M2HFeycmwK+AHwAGHTO9cSeHyfaGfCbsRa/gJmFzOy7j3hMRASFvIjvxQLzQ8D/H3v8f4DvBX6A6Hn3YaIXsH2nc+56bJtVohff9QKfB+4CXyM67f+kc+3OuXmiF+29FJgArgPPi738YaItekNEA/pP4iz9o7EaPrrr+R8BsoCrRE8//BlHO7UgIjHm3O5ZNBEREUkFGsmLiIikqISFvJm938wmzezyPq+bmb07divNbjN7aqJqERERSUeJHMl/kOjtMvfzIqA19vUG4PcTWIuIiEjaSVjIO+e+BMwcsMnLiN560znnvgqUmJkurhERETkmXp6Tr+GJN8gYjT0nIiIix8DLFaFsj+f2vNTfzN5AdEofLPA0y8rDNWUnsDQ5FstbXlcgIuJbgdwgeYEsABbGpnGzK3vl5oG8DPlRorfK3FYL3NxrQ+fco8CjAIGcQpdd+zTWf70WKuMsf9NB8MjHJnGSrZ693G+NUxtkvmX08O1ERORJ8vNyOfueZ+FKgjy2NAL/9hP3tB8vp+s/BfxI7Cr7ZwFzsbtdxSXw8TuwumukuL4F67smA1a34O/n49v2JJ5LtnoSUePqVvTPR0REjqwgP5eLHS24v7/Lv9wZZHlr7Z73lbCRvJn9MdHFMypia0v/ItGFJ3DOvRf4DPBioktkLgGvP8r+g19ZBGDrxyuie729+e1g2XplKZQHYTr6XPAri2xeX33S83ttexLPJVs9iahx+89HRETiV1iQx4X2EBubm1z6k0usRTLhlaX3vD/f3fFue7oeYPOh/CeFvMJFRET8qKggnwvhZtbWN+ju6Wd17fER/Er/F7/pnHvwqPv08pz8fdl8KJ+tf18OWbHzxpUZ0cegoBcREV8pLizgfHsTq2vrdPf2s7a2fiz79e1tbbdeWQrZu8rPDkSfFxER8YnS4kLOh5tZWV2jqydybAEPPh7JUxHc+/nyfZ4XERFJMmUlRXS0NrK8vEp3b4SNjc1j3b9vR/Lc3udATB/vARIREUmE8tJizrY1sbS0kpCABx+H/J4tdGrdEhERH6gsK6GjtZGFxSW6e/sTEvDg4+n6/VrodNGdiIgks1PlpbSF6plfWORy3yCbm4mbgfZtyEM06N3DhQBkvGPC42pEREQOdrqijLZQPXN3F7jcN8DWVmJv/+3b6XqIttG5UDYunM36I7VsPpTvdUkiIiJ7qj5VTluontm5+RMJePDxSF598iIi4hdnTlcQaqxl5s4cV68PcVI3ovPtSF598iIi4ge11ZWEGmuZnjnZgAcfj+TVJy8iIsmu/sxpGuqquT0zS8/1oRP/fN+O5NUnLyIiyay+poqGumomb894EvDg45BXn7yIiCSrxrpqGmqruDU1Q1//iGd1+Ha6Xn3yIiKSjJrqz1BbfYrxW7eJDI16WotvQ15ERCTZhBpqOFNVyc2JKfqHx7wux78hrxY6ERFJJq1NdVSdKmd0fJLBkZtelwP4+Jy8WuhERCRZtDXXU3WqnJGxiaQJePDxSF4tdCIikgzaQ/WcqihjeHSCkbHkusW6b0fyaqETERGvdbQ2cqqijKGRm0kX8ODjkFcLnYiIeMXMONvaREVZCYMjY9wYn/S6pD35drpeLXQiIuKFaMA3UlZaTP/QKDdv3fa6pH35NuRFREROWiAQ4FxbEyXFhUQGbzA+Oe11SQfybcirhU5ERE5SIBDgfHszxUUFXOsf4dbtGa9LOpRvz8mrhU5ERE5KMBjkQriZosJ8+iLDvgh48PFIXi10IiJyErYDviA/j97IELdn5rwuKW6+HcmrhU5ERBItIyNIZzhEQX4ePdf9FfDg45BXC52IiCRSNOBbyMvL4eq1Qabv+CvgwcfT9WqhExGRRMnMzKAzHCInJ5srfQPM3l3wuqR74tuQh2jQu4cLAch4R/LdaUhERPwnKyuTznCI7KxMLvcOMDfvz4AHH0/XQ7SNzoWyceFs1h+pZfOhfK9LEhERH8vOyuJiRwvZWZlc8nnAg49H8uqTFxGR45STnUVnRwsZwSDdvf3MLyx5XdJ98+1IXn3yIiJyXHKys7h4djvgIykR8ODjkbz65EVE5Djk5mRzsaMFM6OrJ8Li0rLXJR0b347k1ScvIiL3Ky83h4sdLQApF/Dg45BXn7yIiNyP/LxcLna04HB09URYWl7xuqRj59vpevXJi4jIvSrIz6Uz3MLm1iZdVyOsrK55XVJC+DbkRURE7kVhQR4X2kNsbEYDfnUtNQMefBzyaqETEZGjKi4s4Hx7E2vrG3T1RFhbW/e6pITy7Tl5tdCJiMhRlBRFA351bT0tAh58PJJXC52IiMSrtLiQs21NrKys0t3bz/r6htclnQjfjuTVQiciIvEoKynibFsTy8urdPVE0ibgwcchrxY6ERE5THlpMWfbmlhaWqG7N8LGRnoNBH07Xa8WOhEROUhlWQntLQ0sLC5xqXeAzc30CnjwcciDlpoVEZG9nSovpS1Uz/zCIpd6B9ja2jr8TSnIt9P1oKVmRUTkyaoqy2hvaWDu7kJaBzz4eCSvPnkREdmt+lQ5LU113Jm9y5VrgzjnvC7JU74dyatPXkREdqqpqqSlqY6ZO3MK+BjfjuTVJy8iIttqqytpqq9hemaOnsiQAj7GtyN59cmLiAhA/ZnTNNXXMDV9h6vXNYLfybchrz55ERFpqK2ioa6aydsz9EaGvS4n6fh2ul598iIi6a2xrpq6M6eZmJzm+uANr8tJSr4NeRERSV/NDTXUVFUyfus2kaFRr8tJWr4NebXQiYikp1BDDWeqKhmbmGJgeMzrcpKab8/Jq4VORCT9tDbVcaaqktHxSQV8HHw7klcLnYhIemlrrud0ZRkjYxMMj+pW5vHw7UheLXQiIukj3NLA6coyhm+MK+CPwLchrxY6EZHUZ2Z0tDZSWV7K0MhNRm7e8rokX/HtdL1a6EREUpuZcba1kbLSYgaGxxibmPK6JN/xbciLiEjqMjPOtTVRWlJE/9AoN2/d9rokX/JtyKuFTkQkNQUCAc61NVFSXMj1gREmpma8Lsm3fHtOXi10IiKpJxAIcL69meKiAq71K+Dvl29H8mqhExFJLcFgkPPtTRQW5NMXGWZqZtbrknzPtyN5tdCJiKSOYDBIZzhEYUE+vZEhBfwx8W3Iq4VORCQ1ZGQEudjRQn5+Lj3Xh7g9M+d1SSnDt9P1aqETEfG/zMwMOsMhcnKyudI3wJ25ea9LSim+DXmIBr17uBCAjHfoDkgiIn6SmZnBxY4WsrOzuNI3wOzdBa9LSjm+na6HaBudC2XjwtmsP1LL5kP5XpckIiJxyMrK5IGzrWRnZXK5VwGfKL4dyatPXkTEn7KzsujsCJGVmcGl3gHuLujf7ETx7UheffIiIv6Tk53FxbMtZGZk0N3br4BPMN+O5NUnLyLiL7k52XR2hAhYgO7eCAuLy16XlPJ8O5JXn7yIiH/k5eZwsaMlFvD9CvgT4tuQV5+8iIg/5OflcrGjBYCungiLSwr4k+Lb6Xr1yYuIJL+C/FwutIfYclt09/SzvLLqdUlpxbchLyIiya2wII8L7SE2Njfp7omwsrrmdUlpx7chrxY6EZHkVVSQz4VwM+sbm3RdjbC6poD3gm/PyauFTkQkORUXFnAh3Mza+gaPXb2ugPeQb0fyaqETEUk+JUUFnGtvZnV1ja6eCOvrG16XlNZ8O5JXC52ISHIpLS7kXHszKyurCvgkkdCQN7MXmlmfmUXM7K17vF5sZn9lZl1mdsXMXh/vvtVCJyKSPMpKijjb1sTysgI+mSQs5M0sCPwe8CLgLPAaMzu7a7OfBK465y4CzwV+08yy4tl/8CuLBN43DWsOnIOpDQLvm9ZFdyIiJ6yirJizbU0sLa3Q1RNhY0MzqskikSP5ZwAR59yAc24N+Bjwsl3bOKDQzAwoAGYA/fonIuITlWUlhFsamV9YpLu3n81NBXwySeSFdzXAjR2PR4Fn7trmd4FPATeBQuBVzrldc/B7UwudiIi3TleU0dpcx935RS73DbC1Fdc/33KCEjmStz2ec7sefy/wGHAGeAD4XTMretKOzN5gZt8ws2+4zXVALXQiIl6qqiyjLVTP3N0FBXwSS2TIjwJ1Ox7XEh2x7/R64BMuKgIMAuHdO3LOPeqce9A596AFM6NPqoVORMQTZ05X0Npcz53Zuwr4JJfIkP860GpmTbGL6V5NdGp+pxHg+QBmdhpoBwbi2rta6ERETlxNVSWhxlpm7sxx5dogzu2eoJVkkrCQd85tAD8FfA7oAT7unLtiZm80szfGNvtl4Nlmdgn4e+DnnHO349m/WuhERE5WXfUpmhtquD0zy9XrQwp4HzC//SEFcgpddu3TgNjFd1qFTkQk4erPnKahrpqp6Tv0Roa9LiftrPR/8ZvOuQeP+j7/3taW6FX07uFCADLeMeFxNSIiqamhtor6miomb8/Q1z/idTlyBP69rS3RkbwLZePC2aw/UsvmQ/lelyQiklKa6s9QX1PFxOS0At6HfDuSV5+8iEhiNTfUUFNVyfit20SGRr0uR+6Bb0fy6pMXEUmclsZaaqoqGZuYUsD7mG9H8uqTFxFJjNamOqpOlXPj5i2Gbox7XY7cB9+O5NUnLyJy/NpD9VSdKmdkbEIBnwJ8G/LqkxcROV7hlgZOVZQxfGOc4VF1LKUC307Xb19cpz55EZH7Y2aEWxqoKCthcGSM0fEpr0uSY+LbkBcRkftnZpxtbaSstJiB4THGJhTwqcS3Ia8WOhGR+2NmnGtrorSkiMjgDcYnp70uSY6Zb8/Jq4VOROTeBQIBzrc3U1pSxPWBEQV8ivLtSF4tdCIi9yYQCHAh3ExhQT59kWEmp3XBcqry7UheLXQiIkcXDAbp7Agp4NOEb0NeLXQiIkcTDAbpDIcoyM+jNzLE1Mys1yVJgvl2ul4tdCIi8cvICNIZbiE3N5ur1waZmb3rdUlyAnwb8iIiEp/MzAw6wyFycqIBf2du3uuS5IT4NuTVQicicrjMzAwudrSQnZ3Flb4BZu8ueF2SnCDfnpNXC52IyMGysjJ54Gwr2VmZXO5VwKcj347k1UInIrK/7KwsLp5tISMY5FLvAHcXNMOZjnw7klcLnYjI3nKyHw/47t6IAj6N+Tbk1UInIvJkuTnZXDzbQjAQoKsnwsListcliYd8O12vFjoRkSfKy82hMxwCoKsnwtLyiscVidd8G/IQDXr3cCEAGe/Q2scikr7y83LpDIdwztHVE2F5ZdXrkiQJ+Ha6HqJtdC6UjQtns/5ILZsP5XtdkojIiSvIjwb8lttSwMsT+HYkrz55EREoLMjjQnuIjc1NunsirKyueV2SJBHfjuTVJy8i6a6oIJ/OcIj1jQ26rirg5cl8O5JXn7yIpLPiwgLOtzexurZOd28/a2vrXpckSci3I3n1yYtIuiopKuB8uJmV1TW6eiIKeNmXb0NeffIiko7KSoo4197Mysoq3b39rK9veF2SJDHfTterT15E0k15aTEdrY0sLi5zqa+fjQ3NXMrBfBvyIiLppKKsmHBLIwuLS1zqHWBzUwEvh/NtyKuFTkTSxanyUtpC9cwvLHK5b1ABL3Hz7Tl5tdCJSDo4XVFGe0sDd+cXNYKXI/PtSF4tdCKS6qoqy2htrmd2bp4r1wbZ2to6/E0iO/h2JK8WOhFJZWdOV9DaXM+d2btc7htQwMs98W3Iq4VORFJVTVUlocZapmfmuHJtEOec1yWJT/l2ul4tdCKSiuqqT9FYf4bbM7P0XB/yuhzxOd+GPGipWRFJLfU1VTTUVjE1fYfeyLDX5UgK8O10PWipWRFJHQ210YC/NTWjgJdj49uRvPrkRSRVNNWfobb6FBOT01wfvOF1OZJCfDuSV5+8iKSC5oYaaqtPcXNiSgEvx863I3n1yYuI37U01lJ9uoKxiSkGhse8LkdSkG9H8uqTFxE/a2uup/p0BTdu3lLAS8L4NuTVJy8iftUequd0ZRnDoxMM3Rj3uhxJYb6drlefvIj4UbilgcryUoZvjDNy85bX5UiK823Ii4j4iZnR0dJIeVkxgyNjjI5PeV2SpAHfhrxa6ETEL8yMs62NlJUWMzA8xtiEAl5Ohm/PyauFTkT8IBAIcK6tibLSYiKDNxTwcqJ8O5JXC52IJLtAIMD59maKiwq41j/CrdszXpckaca3I3m10IlIMgsEAlwIN1NUmE9fZFgBL57wbcirhU5EklUwGKSzI0RhQTTgJ6f175J448DpejOrBV4NPAc4AywDl4G/Bv7GObd1wNsTSi10IpKMMjKCXGgPkZ+fS8/1IabvzHldkqSxfUPezD4A1ACfBn4NmARygDbghcDbzOytzrkvnUShIiLJLiMjSGe4hdzcbK5eG2Rm9q7XJUmaO2gk/5vOuct7PH8Z+ISZZQH1iSnrcGqhE5FkkpmZQWc4RE5ONODvzM17XZLI/ufktwPezF5iZk/azjm35pyLJLK4g6iFTkSSRVZWJhc7WsjJzuJK34ACXpJGPBfevRq4bma/bmYdiS4obmqhE5EksB3w2VmZXOodYPbugtcliXzboSHvnPsh4ClAP/ABM/uKmb3BzAoTXt1B1EInIh7LzsrigbOtZGZk0N3bz90FnSqU5BJXC51z7i7w58DHgGrg5cC3zOxNCaztQGqhExEv5WRn8cC5FjKCQbp7I8wvLHldksiTHHrHOzP718DrgRDwYeAZzrlJM8sDeoDfSWyJe1MLnYh4JTcnm86OEAEL0NUTYXFp2euSRPYUz21tXwE8srtVzjm3ZGY/lpiy4hP8yiLu4ehZg4x3THhZioikibzcHDrDIQC6eiIsLa94XJHI/uKZrh/fHfBm9msAzrm/T0hVcdp8KB8XysaFs1l/pJbNh/K9LEdEUlx+Xi4XO1oABbz4Qzwh/4I9nnvRcRdyVE/okzf7dp+8gl5EEqEgPxrwW26Lx65eZ3ll1euSRA61b8ib2X80s0tA2My6d3wNAt0nV+Le1CcvIielsCCPznALG5ubPHYlwsrqmtclicTloHPyHwX+BvifwFt3PD/vnPN+OSX1yYvICSgqyOdCuJm19Q26e/pZXVPAi38cNF3vnHNDwE8C8zu+MLOyxJd2CPXJi0iCFRcWcCHczOraOl09EQW8+M5hI/mXAN8EHGA7XnNAcwLrOlTg43ei5+R3TtmrT15EjklpcSFn25pYWVmlu7ef9fUNr0sSObJ9Q94595LY96aTKyd+6pMXkUQpKymio7WR5eVVunsjbGxohlD86dCr683sk2b2mtjNbzznmtQuJyKJU15azNm2JpaWVhTw4nvxtND9FvAcoMfM/tTMXmFmOQmu62CVGWy9oZytN1SohU5Ejk1FWTEdrY0sLC7R3duvgBffi2eBmi86536C6Dn4R4FXApOJLuxQmQHItCc+pxY6EblHp8pLCbc0Mr+wyKXeATY3FfDif/Hc1hYzywVeCrwKeCrwvxJZ1H1RC52IHNHpijLaQvXM3V3gct8AW1tbh79JxAfiWaDmT4BnAp8Ffg/4gnMuef8PUAudiBxB9alyWprqmJ2b58q1QQW8pJR4RvIfAF7rnEuu9FzfAuyJU/ZqoRORIzhzuoJQYy0zd+a4en0I55zXJYkcq31D3sweds79A5AHvMzsiee/nXOfSHBt+5va+HaYb72yNDpFP60WOhGJX211JU31NUzPzNETUcBLajpoJP/dwD8QPRe/mwO8C/mY4FcWFeoicmT1Z07TUFfN7ZlZeq4PeV2OSMIcdDOcX4z9+EvOucGdr5mZtzfIibXLAQp5ETmS+poqGmqrmLw9Q1//iNfliCRUPH3yf77Hc3923IUcmdrlROSIGuuqaait4taUAl7Sw0Hn5MPAOaDYzH5gx0tFgLc3w9mmdjkRiVNT/Rlqq08xfus2kaFRr8sROREHnZNvJ7pATQlPPC8/D/x4PDs3sxcC7wKCwPucc7+6xzbPBX6b2B3onXPfHc++AbXLiUhcQg01nKmq5ObEFP3DY16XI3JiDjon/0ngk2b2kHPuK0fdsZkFifbVvwAYBb5uZp9yzl3dsU0J8B7ghc65ETM7FfcHqF1OROLQ2lRH1alyRscnGRy56XU5IifqoOn6/+Kc+3XgtWb2mt2vO+d++pB9PwOIOOcGYvv7GPAy4OqObV4LfMI5NxLbZ3y3y4210OmiOxE5SFtzPacryxgZm2B4dMLrckRO3EHT9T2x79+4x33XADd2PB4leue8ndqATDP7AlAIvMs596ED97q8ReZbdD5NRA7WHqrnVEUZw6MTjIwp4CU9HTRd/1ex79++T72ZBYAC59zdOPZtezy3+24TGcDTgOcDucBXzOyrzrlrT9iR2RuANwDQUsr6b9VqJC8i++pobaSirIShkZvcGPd+PS0Rr8SznvxHzazIzPKJTrX3mdnPxrHvUaBux+NaYPcJsVHgs865RefcbeBLwMXdO3LOPeqce9A59yAZAS0rKyJ7MjPOtjZRUVbC4MiYAl7SXjx98mdjI/fvBz4D1AM/HMf7vg60mlmTmWUBrwY+tWubTwLPMbMMM8sjOp3fQzzUJy8iO0QDvpHysmL6h0YZHZ/yuiQRz8WzQE2mmWUSDfnfdc6tm9mhN3l2zm2Y2U8BnyPaQvd+59wVM3tj7PX3Oud6zOyzQDewRbTN7nLc1atPXkSAQCDAubYmSooLiQzeYHxy2uuSRJJCPCH/B8AQ0AV8ycwagHjOyeOc+wzR0f/O59676/E7gXfGs78nUZ+8SNoLBAKcb2+muKiAa/0j3Lo943VJIknj0JB3zr0bePeOp4bN7HmJKylO6pMXSXvBYJDz7U0UFuTTFxlmclr/JojsdGjIm1k28INA467tfylBNR1OffIiaS8YDHIh3ExBfh69kSFuz8x5XZJI0olnuv6TwBzwTWA1seWIiBwuIyPIhfYQ+fm59FwfYvqOAl5kL/GEfK1z7oUJr+QotNSsSNrKyAjSGW4hNzebq9cGmZmN6xIhkbQUTwvdl83sQsIrOSq10ImknczMDC52RAP+St+AAl7kEPGM5L8T+FEzGyQ6XW+Ac851JrSyeKiFTiRtZGVl0hkOkZ2VyeXeAebmF7wuSSTpxRPyL0p4FfdKLXQiaSE7K4vOjhBZmRlc6h3g7oJO04nE49DpeufcMNHb0z4c+3kpnvclnFroRNJCdlYWF8+2kJmRQXdvvwJe5AjiaaH7ReBBoB34AJAJ/G/gOxJb2gHUQieSFnKyowEfDATp7o2wsLjsdUkivhLPdP3LgacA3wJwzt00s8KEVnUQLTUrkhZyc7K52NGCmdHVE2FxSQEvclTxTLuvOeccsWViY6vReSc3wPojtVqBTiSF5eXmcLGjBUABL3If4gn5j5vZHwAlZvbjwN8Bf5jYsg6hpWZFUlZ+Xi4XO1pwOLp6Iiwtr3hdkohvxXPv+t8wsxcQXZSmDXi7c+7zCa/sMLE+eZ2XF0kdBfm5dIZb2NzapOtqhJXVNa9LEvG1eM7J45z7vJl9C/guIHmWeFKfvEjKKCzI40J7iI3NaMCvringRe7XvtP1ZvZpMzsf+7kauAz8GPBhM3vzyZR3CPXJi6SE4sICOsMh1jc2eOzqdQW8yDE56Jx8k3Pucuzn1wOfd869FHgm0bD3lvrkRVJCSVEB59ubWF1bp6snwtrautcliaSMg0J+5/9pzwc+A+Ccmwe2ElnUoaY2CLxvWufjRXyutLiQc+3NrKyuKeBFEuCgc/I3zOxNwCjwVOCzAGaWS/SGOCIi96yspIiO1kaWl1fp7o2wsaHTbyLH7aCR/L8DzgE/CrzKOTcbe/5ZRO985x210In4WnlpMWfbmlhaWlHAiyTQviN559wk8MY9nv9H4B8TWVRc1EIn4kuVZSW0tzSwsLjEpd4BNjcV8CKJctDV9Y9uX12/x2v5ZvZjZva6xJUWB7XQifjKqfJS2lsamF9YpLunXwEvkmAHnZN/D/B2M7tAtH1uCsgBWoEi4P3ARxJe4UHUQifiG1WVZbQ21zM7N8+Va4NsbXl7/a5IOjhouv4x4JVmVkB0FbpqYBnocc71nUx5B1ALnYhvVJ8qp6Wpjjuzd7lybZDochgikmjx3NZ2AfhC4ks5Ai01K+IbNVWVNDfUMHNnjqvXhxTwIicortvaiojci9rqSprqa5iemaMnooAXOWn+DPlYCx2g0bxIkqo/c5qGumqmpu/QGxn2uhyRtBTPUrNAEqwjv1ushU5Ekk99TRUNddVM3p5RwIt46NCQN7Nnm9lVoCf2+KKZvSfhlcVDLXQiSaexrpqG2iomJqfp6x/xuhyRtBbPSP4R4HuBaQDnXBfRJWe9pxY6kaTS3FBD3ZnTjN+6zfXBG16XI5L24l1P/oaZ7XzK+3RVC51IUgk11HCmqpKxiSkGhse8LkdEiC/kb5jZswFnZlnATxObuveMWuhEkkprUx1Vp8oZHZ9kcOSm1+WISEw8If9G4F1ADdEV6f4W+IlEFnWg5S0y3zLq2ceLyBO1NddzurKMkbEJhkcnvC5HRHaI55x8u3Pudc650865U865HwI6El3YvnIDrD9SqxXoRJJAuKWB05VlDN8YV8CLJKF4Qv534nzu5GipWRFPmRkdrY1UlpcyNHKTkZu3vC5JRPaw73S9mT0EPBuoNLOf2fFSEeB975qWmhXxhJnR0dJIeVkxA8NjjE1MeV2SiOzjoHPyWUBBbJvCHc/fBV6RyKLipj55kRNlZpxra6K0pIj+oVFu3rrtdUkicoCDVqH7IvBFM/ugcy45b1mlPnmRExMIBDjX1kRJcSHXB0aYmJrxuiQROUQ8V9cvmdk7gXNE15MHwDn3cMKqiof65EVOTCAQ4Hx7M0WF+VzrH+HWbQW8iB/Ec+HdR4BeoAn478AQ8PUE1nS4qQ0C75vW+XiRExAMBrkQjgZ8X2RYAS/iI/GEfLlz7o+AdefcF51zPwY8K8F1iUgS2A74woJ8eiNDTM3Mel2SiBxBPNP167Hv42b2fcBNoDZxJcVBS82KJFxGRpDOcAt5eTn0XB9i+s6c1yWJyBHFE/K/YmbFwP9HtD++CHhzIouKi1roRBImMzODznCInJxsrvQNcGdu3uuSROQeHBryzrlPx36cA54HYGbfkcii4qYWOpFjl5mZwcWOFrKzs7jSN8Ds3QWvSxKRe3TQzXCCwCuJ3rP+s865y2b2EuDngVzgKSdT4gHUQidyrLKyMrnY0UJWZgaXeweYm1fAi/jZQSP5PwLqgK8B7zazYeAh4K3Oub88gdoOphY6kWOVnZVFZ0eIrMwMLvUOcHdBp8JE/O6gkH8Q6HTObZlZDnAbaHHOeb8KhZaaFTlWOdlZdHa0kBEM0t3bz/zCktclicgxOCjk15xzWwDOuRUzu5YUAS8ixyo3J5vOjhABC9DdG2FhcdnrkkTkmBwU8mEz6479bEAo9tgA55zrTHh1+1ELncixyM3J5mJHC2ZGd28/i0sKeJFUclDIe7dmfDzUQidyX/Jyc7jY0YJzjq6eCEvLK16XJCLH7KAFapJzUZqd1EInck/y83LpDIfYclt09/SzvLLqdUkikgDx3AwneamFTuTICgvyuNAeYmNzk+6eCCura16XJCIJ4t+QVwudyJEVFeRzIdzM+sYmXVcjrK4p4EVSWTwL1GBmuWbWnuhi4qZV6ESOrLiwgAvhZtbWN3js6nUFvEgaODTkzeylwGPAZ2OPHzCzTyW4rv0tb5H5llEFvMgRlBQVcD7czOraOo9dvc7a2vrhbxIR34tnJP/fgGcAswDOuceAxkQVdKjcAOuP1LL5UL5nJYj4SWlxIefam1lZWaWrJ8L6+obXJYnICYnnnPyGc27OzBJeTNzUJy8Sl7KSIjpaG1leXqW7N8LGhi5WFUkn8YzkL5vZa4GgmbWa2e8AX05wXYeL9cmLyN4qyoo529bE0tIKXT0KeJF0FE/Ivwk4B6wCHyW65OybE1hT/NQnL7KnyrISwi2NzC8s0t3bz+amAl4kHcUzXd/unHsb8LZEF3Nk6pMXeZLTFWW0Ntdxd36Ry30DbG1teV2SiHgknpH8b5lZr5n9spmdS3hF8VKfvMiTVFWW0RaqZ+7uggJeRA4Peefc84DnAlPAo2Z2ycx+IdGFHUh98iJPcuZ0Ba3N9dyZvauAFxEgzpvhOOcmnHPvBt5ItGf+7YksSkSOpqaqklBjLTN35rhybRDnnNcliUgSiOdmOB1m9t/M7DLwu0SvrK9NeGUHibXQqVdeBOqqT9HcUMPtmVmuXh9SwIvIt8Vz4d0HgD8Gvsc5dzPB9cRPS82KUH/mNA111UxN36E3kvwLR4rIyTo05J1zzzqJQu6JWugkjTXUVlFfU8Xk7Rn6+ke8LkdEktC+IW9mH3fOvdLMLgE75/8McM65zoRXdxi10Emaaqo/Q231KSYmp7k+eMPrckQkSR00kv9Pse8vOYlCjkwtdJKmmhtqqKmqZPzWbSJDo16XIyJJbN8L75xz47Eff8I5N7zzC/iJkylvH2qhkzTV0lhLTVUlYxNTCngROVQ8LXQv2OO5Fx13IXHTUrOSplqb6qg+XcGNm7cYGB7zuhwR8YF9Q97M/mPsfHy7mXXv+BoEuk+uxF201KykofZQPVWnyhkZm2DoxvjhbxAR4eBz8h8F/gb4n8Bbdzw/75ybSWhVh9FSs5JGwi0NVJaXMnxjnJGbt7wuR0R85KDpeuecGwJ+Epjf8YWZlSW+tENoqVlJcWZGR2sjleWlDI6MKeBF5MgOG8m/BPgm0RY62/GaA5oTWFd81CcvKcrMONvaSFlpMQPDY4xNTHldkoj40L4h75x7Sex708mVc0Tqk5cUZGaca2uitKSIyOANxienvS5JRHwqnnvXf4eZ5cd+/iEz+y0zq098aYdQn7ykoEAgwPn2ZkpLirg+MKKAF5H7Ek8L3e8DS2Z2EfgvwDDw4YRWdRj1yUsKCgQCXAg3U1xUQF9kmIkpb69vFRH/iyfkN1x0WauXAe9yzr0LKExsWSLpJRgM0tkRorAgn77IMJPTmqUSkfsXzyp082b2X4EfBp5jZkEgM7FlHUItdJJCgsEgneEQ+fm59EaGuD0z53VJIpIi4hnJvwpYBX7MOTcB1ADvjGfnZvZCM+szs4iZvfWA7Z5uZptm9oq4qga10ElKyMgIcrGjhby8HK5eG1TAi8ixOjTkY8H+EaDYzF4CrDjnPnTY+2Ij/t8jegvcs8BrzOzsPtv9GvC5I9auFjrxtczMDC52tJCbm83Va4PMzN71uiQRSTHxXF3/SuBrwL8BXgn8c5wj7mcAEefcgHNuDfgY0fP6u70J+HNgMu6qt6mFTnxqO+BzcrK50jfAnbl5r0sSkRQUzzn5twFPd85NAphZJfB3wJ8d8r4aYOdC16PAM3duYGY1wMuBh4Gnx1lzlFroxKeysjK52NFCVmYGl3sHmJtf8LokEUlR8ZyTD2wHfMx0nO+zPZ5zux7/NvBzzrkDh+Rm9gYz+4aZfYONLbXQiW9lZ2XxwNlWMjMyuKSAF5EEi2ck/1kz+xzwx7HHrwI+E8f7RoG6HY9rgZu7tnkQ+JiZAVQALzazDefcX+7cyDn3KPAogHVU7P5FQcQXcrKz6OxoISMYpLs3wsListcliUiKOzTknXM/a2Y/AHwn0dH5o865v4hj318HWs2sCRgDXg28dte+v33LXDP7IPDp3QG/J7XQic/k5mTT2REiYAG6eiIsLingRSTx9g15M2sFfgMIAZeA/+ycG4t3x865DTP7KaJXzQeB9zvnrpjZG2Ovv/e+Ko+10CnkJdnl5ebQGQ4B0NUTYWl5xeOKRCRdHDSSfz/wIeBLwEuB3wF+4Cg7d859hl1T+/uFu3PuR4+yb0AtdJL08vNy6QyHcM7R1RNheWXV65JEJI0cFPKFzrk/jP3cZ2bfOomCjkQtdJLECvJzudAeYstt0d3Tr4AXkRN3UMjnmNlTePwq+dydj51z3oa+WugkiRUW5HGhPcTG5ibdPRFWVte8LklE0tBBIT8O/NaOxxM7Hjuive3emNog8PE7Oh8vSamoIJ8L4WbW1jfo7ulndU0BLyLe2DfknXPPO8lC4ra8ReZbRr2uQmRPxYUFnG9vYnVtne7eftbW1r0uSUTSWDw3tUkuuQHWH6ll86F8rysReYKSogLOh5tZWV2jqyeigBcRz8VzM5zkoz55STJlJUV0tDaysrJKd28/6+sbXpckIuLDkfw2LTUrSaK8tJizbU0sLa3Q1RNRwItI0ohnFTozsx8ys7fHHteb2TMSX1oc1CcvHqsoK6ajtZGFxSW6e/vZ2FBbp4gkj3hG8u8BHgJeE3s8T3SdeO+pT148dKq8lHBLI/MLi1zqHWBzU38fRSS5xHNO/pnOuaea2b8AOOfumFlWgus6nPrkxUOnK8poC9Uzd3eBy30DbG1teV2SiMiTxBPy62YWJLZMbGw9eW//RVOfvHioqrKM1uZ6ZufmuXJtUAEvIkkrnun6dwN/AZwys3cA/wf4HwmtSiRJnTldQWtzPXdm72oELyJJL56lZj9iZt8Enk/0lrbf75zrSXhlB1ELnXigpqqS5oYapmfm6IkM4ZzzuiQRkQPFc3V9PbAE/BXwKWAx9py31EInJ6iu+hTNDTXcnpnl6vVBBbyI+EI85+T/muj5eANygCagDziXwLrioxY6OQH1NVU01FYxNX2H3siw1+WIiMQtnun6Czsfm9lTgf+QsIqOQi10kmANtVXU11Rxa2qGawMjXpcjInIkR76trXPuW2b29EQUcyRqoZMEa6o/Q231KSYmp7k+eMPrckREjuzQkDezn9nxMAA8FZhKWEXxUAudJFhzQw01VZXcnJiif3jM63JERO5JPCP5wh0/bxA9R//niSlHxHstjbVUn65gbGKKAQW8iPjYgSEfuwlOgXPuZ0+onviohU4SpK25ntOVZdy4eYuhG+NelyMicl/2baEzswzn3CbR6fnkoxY6OWbtoWjAD49OKOBFJCUcNJL/GtGAf8zMPgX8KfDtYbNz7hMJru1waqGTYxJuaaCyvJThG+OM3LzldTkiIscinnPyZcA08DCP98s7wPuQVwud3Cczo6OlkfKyYgZHxhgd9/aaUhGR43RQyJ+KXVl/mcfDfZv3t/tSC53cJzPjbGsjZaXFDAyPMTahgBeR1HJQyAeBAp4Y7tu8DXm10Ml9MjPOtTVRWlJEZPAG45PTXpckInLsDgr5cefcL51YJfFa3iLzLaNeVyE+FggEON/eTHFRAdf6R7h1e8brkkREEuKgBWr2GsF7LzfA+iO1bD6U73Ul4kOBQIAL4WaKCvPpiwwr4EUkpR00kn/+iVVxVOqTl3sQDAa5EG6mID+PvsgwUzOzXpckIpJQ+47knXPJPcRRn7wcQUZGkM5wiIL8PHquDyngRSQtHHmBmqSiPnmJQzTgW8jNzebqtUFmZu96XZKIyInwd8irT14OkZmZQWc4RE5ONODvzM17XZKIyInxb8irT14OkZWVSWc4RHZWJlf6Bpi9u+B1SSIiJ8qfIa8+eTlEVlYmFztayMrM4FLvAHcX9HdFRNKPP0Ne5ADZWVlcPNtCRjBId28/8wtLXpckIuIJf4a8WuhkHznZ0YAPBoJ090ZYWFz2uiQREc8cdDOc5KYWOtklNyebi2dbCFiArh4FvIiIP0fy29RCJzF5uTl0hkMAdPVEWFpe8bgiERHv+Tvk1UInQH5eLp3hEM45unoiLK+sel2SiEhS8G/Iq4VOgIL8XDrDLWxubdJ1NcLK6prXJYmIJA1/hrxa6AQoLMjjQnuIjc1owK+uKeBFRHbyZ8hL2isqyOdCuJm19Q26e/oV8CIie/BnyKuFLq0VFxZwvr2J1bV1unv7WVtb97okEZGkpBY68ZXS4kLOh5tZWV2jqyeigBcROYA/R/Lb1EKXVspKiuhobWR5eZXu3ggbG+quEBE5iL9DXi10aaO8tJiO1kYWF5e51NevgBcRiYN/Q14tdGmjoqyYcEsjC4tLXOodYHNTAS8iEg9/hrxa6NLGqfJS2kL1zC8scrlvUAEvInIE/gv55S0y3zLqdRVyAk5XlNEWqmfu7gKX+wbY2tryuiQREV/x39X1uQHWH6ll86F8ryuRBKo+VU5bqJ7ZuXkFvIjIPfLfSB7UJ5/izpyuINRYy8ydOa5eH8I553VJIiK+5L+R/Db1yaek2upKQo21TM8o4EVE7pc/R/Lb1CefUuqqT9FYf4bbM7P0XB/yuhwREd/zd8irTz5l1NdU0VBbxeTtGfr6R7wuR0QkJfg35NUnnzIa66qpO3OaW1MzXBtQwIuIHBd/hrz65FNGU/0ZaqtPMX7rNpEhtUaKiBwnf4a8pIRQQw1nqiq5OTFF//CY1+WIiKQcf4a8Wuh8r6WxlurTFYyOTzI4ctPrckREUpJa6OTEtTXXU326gpGxCQW8iEgC+XMkv00tdL7THqrnVEUZw6MTjIxNeF2OiEhK83fIq4XOVzpaG6koK2Fo5CY3xie9LkdEJOX5N+TVQucbZkZHSyPlZcUMjowxOj7ldUkiImnBnyGvFjrfMDPOtjZSVlpM/9AoN2/d9rokEZG04b+Q11KzvhEIBDjX1kRJcSGRwRuMT057XZKISFrx39X1WmrWFwKBAOfbmykpLuRa/4gCXkTEA/4byYP65JNcMBjkfHsThQX59EWGmZzWtRMiIl7w30h+m/rkk1IwGORCuJnCgnx6I0MKeBERD/lzJL9NffJJJSMjyIX2EPn5ufRcH2L6zpzXJYmIpDV/h7z65JNGRkaQznALubnZXL02yMzsXa9LEhFJe/4NefXJJ43MzAw6wyFycrK50jfA7N0Fr0sSERH8GvLqk08aWVmZdIZDZGdlcrl3gLl5BbyISLLwZ8hLUsjOyqKzI0RWZgaXege4u6BfukREkok/Q14tdJ7Lzsri4tkWMoJBunv7mV9Y8rokERHZRS10cmQ52Vk8cG474CMKeBGRJOXPkfw2tdCduNycbC52tGBmdPVEWFxa9rokERHZh79DXi10JyovN4fOcAiArp4IS8srHlckIiIH8W/Iq4XuROXn5dIZDrHltuju6Wd5ZdXrkkRE5BD+DHm10J2ogvxcOsMtbG5t0nU1wsrqmtcliYhIHPwZ8nJiCgvyuNAeYmMzGvCrawp4ERG/SOjV9Wb2QjPrM7OImb11j9dfZ2bdsa8vm9nFuHYca6HTcrOJVVSQT2c4xPrGBo9dva6AFxHxmYSFvJkFgd8DXgScBV5jZmd3bTYIfLdzrhP4ZeDRuD9ALXQJVVJUwIVwM6tr63T1RFhbW/e6JBEROaJEjuSfAUSccwPOuTXgY8DLdm7gnPuyc2776rmvArVH+gS10CVEaXEh59qbWVldU8CLiPhYIkO+Brix4/Fo7Ln9/Dvgb470CWqhO3ZlJUWcbWtieXmVrp4I6+sbXpckIiL3KJEX3tkez7k9NzR7HtGQ/859Xn8D8AYAWmJT9GqhO3blpcV0tDayuLjMpb5+Njb0S5SIiJ8lciQ/CtTteFwL3Ny9kZl1Au8DXuacm95rR865R51zDzrnHiQjEG2he9+0WuiOUWVZCR2tjSwsLtHdq4AXEUkFiRzJfx1oNbMmYAx4NfDanRuYWT3wCeCHnXPX4trr8haZbxk95lLT26nyUtpC9cwvLHKpd4CtrS2vSxIRkWOQsJG8c24D+Cngc0AP8HHn3BUze6OZvTG22duBcuA9ZvaYmX3j0B3nBlh/pFbtc8ekqrKM9pYG5u4uKOBFRFKMObfnafKkZR0VLviBl0TPyWvK/r5UnyqnpamOO7N3uXJtEL/9XRARSRcr/V/8pnPuwaO+T0vNpqmaqkpamuqYuTOngBcRSVH+vq2t+uTvSW11JU31NUzPzNETGVLAi4ikKH+HvPrkj6z+zGka6qqZmr5Db2TY63JERCSB/Bvy6pM/svqaKhpqq5i8PUNf/4jX5YiISIL5M+S11OyRNdZVU3fmNBOT01wfvHH4G0RExPf8GfJyJM0NNdRUVTJ+6zaRId1jQEQkXfgz5GNLzQIazR8i1FDDmapKxiamGBge87ocERE5QWqhS2GtTXWcqapkdHxSAS8ikob8OZLfpha6fbU113O6soyRsQmGRye8LkdERDzg75BXC92ewi0NVJaXMnxjnJGbt7wuR0REPOLfkFcL3Z46WhupKCthaOQmN8YnvS5HREQ85M+QVwvdk5gZHS2NlJcVMzA8xtjElNcliYiIx/wZ8vIEZsa5tiZKS4roHxrl5q3bXpckIiJJwJ8hrxa6bwsEApxra6KkuJDrAyNMTM14XZKIiCQJtdD5WCAQ4Hx7M8VFBVzrV8CLiMgT+XMkvy2NW+iCwSDn25soLMinLzLM1Mys1yWJiEiS8XfIp2kLXTAY5EK4mYL8PHojQ9yemfO6JBERSUL+Dfk0baHLyAjSGW4hLy+HnutDTN9RwIuIyN78GfJp2kKXmZlBZzhETk42V/oGuDM373VJIiKSxPwX8stbZL4l/VZSy8zM4GJHC9nZWVzpG2D27oLXJYmISJLz39X1uQHWH6ll86F8rys5MVlZmdGAz8rkcq8CXkRE4uO/kTykVZ98dlYWnR0hsjIzuNQ7wN2F1P7vFRGR4+O/kfy2NOiTz8nO4uLZFjIzMuju7VfAi4jIkfhzJL8thfvktwM+YAG6eyMsLC57XZKIiPiMv0M+Rfvkc3OyudjRgpnR3dvP4pICXkREjs6/IZ+iffJ5uTlc7GjBOUdXT4Sl5RWvSxIREZ/yZ8inaJ98fl4uneEQW26L7p5+lldWvS5JRER8zJ8hn4IKC/K40B5iY3OT7p4IK6trXpckIiI+58+QT7EWuqKCfC6Em1nf2KTraoTVNQW8iIjcP7XQeay4sIAL4WbW1jd47Op1BbyIiBwbf47kt/m8ha6kqIBz7c2srq7R1RNhfX3D65JERCSF+DvkfdxCV1pcyNm2JlZWVunu7VfAi4jIsfNvyPu4ha6spIiO1kaWl1fp7o2wseHfX1ZERCR5+TPkfdxCV1FWTLilkcXFZbp7+9ncVMCLiEhi+C/kfbzUbGVZCe0tDcwvLHK5b1ABLyIiCeW/q+t9utTs6Yoy2lsauDu/yKXeAQW8iIgknP9G8uC7PvmqyjJam+uZnZvnyrVBtra2vC5JRETSgP9G8tt80idffaqc1uZ67sze5XLfgAJeREROjD9H8tuSvE++pqqS5oYaZu7McfX6EM45r0sSEZE04u+QT+I++brqUzTWn+H2zCy9kWEFvIiInDj/hnwS98nXnzlNQ101U9N36I0Me12OiIikKX+GfBL3yTfUVlFfU8Xk7Rn6+ke8LkdERNKYP0M+STXWVVN35jQTk9NcH7zhdTkiIpLm/BnySdhC19xQQ01VJeO3bhMZ8ufNekREJLWohe4YtDTWUlNVydjElAJeRESShj9H8tuSoIWutamOqlPl3Lh5i6Eb416XIyIi8m3+DnmPW+jaQ/WcqihjZGyC4dEJT2sRERHZzb8h73ELXbilgcryUoZvjDNy85ZndYiIiOzHnyHvYQudmRFuaaCirITBkTFGx6dOvAYREZF4+DPkPWJmnG1tpKy0mIHhMcYmFPAiIpK8/BnyHrTQmRnn2pooLSkiMniD8cnpE/lcERGRe6UWujgEAgHOtzdTWlLE9YERBbyIiPiCP0fy206ghW474IsK8+mLDDM5nZz3yxcREdnN3yGf4Ba6YDDIhXAzBfl59EWGmZqZTejniYiIHCf/hnyCW+iCwSCd4RD5+bn0Roa4PTOXsM8SERFJBH+GfIJb6DIygnSGW8jNzebqtUFmZu8m5HNEREQSyX8hv7xF5lsSd3/4zMwMOsMhcnKiAX9nbj5hnyUiIpJI/ru6PjfA+iO1bD6Uf+y7zszM4GJHCzk52VzpG1DAi4iIr/lvJA8J6ZPPysrkYkcLWZkZXO4dYG5+4Vj2KyIi4hX/jeS3HWOffHZWFg+cbSUzI4NLCngREUkR/hzJbzuGPvmc7Cw6O1rICAbp7o2wsLh8DIWJiIh4z98hf5998rk52XR2hAhYgK6eCItLCngREUkd/g35++yTz8vNoTMcAqCrJ8LS8spxVSYiIpIU/Bny99knn5+XS2c4hHOOrp4Iyyurx1ygiIiI9/wZ8vehID+XC+0httwW3T39CngREUlZ/gz5e2yhKyzI40J7iI3NTbp7IqysriWqQhEREc+lTQtdUUE+neEQ6xsbdF1VwIuISOrz50h+W5wtdMWFBZxvb2J1bZ3u3n7W1tYTXJiIiIj3/B3ycbTQlRQVcK69mZWVVbp7+1lf3ziBwkRERLzn35CPo4WutLiQs21NCngREUlL/gz5OFroykuLCbc0sLS0wqW+fjY27u/GOSIiIn7jz5A/REVZMeGWRhYWl7jUO8DmpgJeRETSjz9D/oAWulPlpbSF6plfWORy36ACXkRE0lZKtdCdriijLVTP3flFjeBFRCTt+XMkv21HC11VZRmtzfXMzs1z5dogW1tbHhYmIiLiPX+HfKyF7szpCkKNtdyZvcuVa4M45zwuTERExHv+DflYC11NVSXNDTVMz8zRExlSwIuIiMT4M+RjLXSNQ/k0Npzh9swsPdeHvK5KREQkqST0wjsze6GZ9ZlZxMzeusfrZmbvjr3ebWZPPXSny1tkvmWUppFCGuvPMDV9RwEvIiKyh4SFvJkFgd8DXgScBV5jZmd3bfYioDX29Qbg9w/dcW6Amvc+hbrvaeTW1Ay9keHjLVxERCRFJHIk/wwg4pwbcM6tAR8DXrZrm5cBH3JRXwVKzKz6oJ1mWwYNZ05z8wVb9JyeTkzlIiIiKSCRIV8D3NjxeDT23FG3eYKsQAZja3e4tjV5pKVmRURE0k0iL7yzPZ7bfel7PNtgZm8gOp0PmQH6XvNH335ts3/6m/deouyjArjtdRFpQMc58XSME0/H+GS038ubEhnyo0Ddjse1wM172Abn3KPAowBm9g3Xc/vB4y1VdjKzbzjndIwTTMc58XSME0/H+GSY2Tfu5X2JnK7/OtBqZk1mlgW8GvjUrm0+BfxI7Cr7ZwFzzrnxBNYkIiKSNhI2knfObZjZTwGfA4LA+51zV8zsjbHX3wt8BngxEAGWgNcnqh4REZF0k9Cb4TjnPkM0yHc+994dPzvgJ4+420ePoTQ5mI7xydBxTjwd48TTMT4Z93ScTbeBFRERSU3+XWpWREREDpS0IZ+QW+LKE8RxjF8XO7bdZvZlM7voRZ1+dtgx3rHd081s08xecZL1pYp4jrOZPdfMHjOzK2b2xZOu0e/i+Pei2Mz+ysy6YsdY11gdkZm938wmzezyPq8fPfecc0n3RfRCvX6gGcgCuoCzu7Z5MfA3RHvtnwX8s9d1++krzmP8bKA09vOLdIyP/xjv2O4fiF6/8gqv6/bbV5x/l0uAq0B97PEpr+v201ecx/jngV+L/VwJzABZXtfupy/gu4CnApf3ef3IuZesI/mE3BJXnuDQY+yc+7Jz7k7s4VeJ3sdA4hfP32OANwF/DkyeZHEpJJ7j/FrgE865EQDnnI710cRzjB1QaGYGFBAN+Y2TLdPfnHNfInrc9nPk3EvWkE/ILXHlCY56/P4d0d8gJX6HHmMzqwFeDrwXuVfx/F1uA0rN7Atm9k0z+5ETqy41xHOMfxfoIHpDs0vAf3LObZ1MeWnjyLmXrOvJH9stcWVfcR8/M3se0ZD/zoRWlHriOca/Dfycc24zOgCSexDPcc4AngY8H8gFvmJmX3XOXUt0cSkinmP8vcBjwMNACPi8mf2Tc+5ugmtLJ0fOvWQN+WO7Ja7sK67jZ2adwPuAFznntOzf0cRzjB8EPhYL+ArgxWa24Zz7yxOpMDXE++/FbefcIrBoZl8CLgIK+fjEc4xfD/yqi548jpjZIBAGvnYyJaaFI+desk7X65a4iXfoMTazeuATwA9rxHNPDj3Gzrkm51yjc64R+DPgJxTwRxbPvxefBJ5jZhlmlgc8E+g54Tr9LJ5jPEJ0pgQzO010QZWBE60y9R0595JyJO90S9yEi/MYvx0oB94TG2luOC1EEbc4j7Hcp3iOs3Oux8w+C3QDW8D7nHN7tinJk8X5d/mXgQ+a2SWi08o/55zT6nRHYGZ/DDwXqDCzUeAXgUy499zTHe9ERERSVLJO14uIiMh9UsiLiIikKIW8iIhIilLIi4iIpCiFvIiISIpSyIuIiKQohbzIAWLLvz6246vxgG0XjuHzPmhmg7HP+paZPXQP+3ifmZ2N/fzzu1778v3WGNvP9nG5HFtetOSQ7R8wsxffw+dUm9mnYz8/18zmdvxZ/N0B73vujvf9qJn97lE/O/beRjN77Y7HF8zsg/eyLxEvKORFDrbsnHtgx9fQCXzmzzrnHgDeCvzBUd/snPv3zrmrsYc/v+u1Z99/ecDjx+U80VWzfvKQ7R8gehOPo/oZ4A93PP6nHX8W/+oe9ndUjURXsAPAOXcJqI3dDVIk6SnkRY7AzArM7O9jo+xLZvakpWNjo88v7RjpPif2/PeY2Vdi7/1TMys45OO+BLTE3vszsX1dNrM3x57LN7O/NrOu2POvij3/BTN70Mx+FciN1fGR2GsLse9/snNkHZtB+EEzC5rZO83s62bWbWb/IY7D8hViK2GZ2TPM7Mtm9i+x7+2x26D+EvCqWC2vitX+/tjn/MtexzHmB4HP7vfBe33eQYWaWUPsz6879r1+x3//K3Zstz0r86tEb4f7mJm9JfbcXxG9ratI0lPIixxsOyQfM7O/AFaAlzvnngo8D/hNsyctH/da4HOx0fhF4DEzqwB+AfhXsfd+g+go9SAvBS6Z2dOI3r7ymcCzgB83s6cALwRuOucuxkbUTwhD59xbeXzE/bpd+/4YsP1LQRbRe45/huhqg3POuacDT499VtN+BZpZMPbe7fuY9wLf5Zx7CtHbIv+P2Prjbwf+JFbLnwBvA/4h9jnPA95pZvm79t0E3HHOre54+jk7/jzettfnHXA8Iboc6oecc53AR4B3H7L9W3l89uCR2HPfAJ5zyPtEkkJS3rteJIksx8IaADPLBP6HmX0X0Xug1wCngYkd7/k68P7Ytn/pnHvMzL4bOAv839jvBFlER8B7eaeZ/QIwRTR0nw/8RWwFNczsE0RD5rPAb5jZrwGfds790xH+u/4GeLeZZRP9ZeFLzrllM/seoHPHqLYYaAUGd70/18weIzqd/U3g8zu2/19m1kp0CczMfT7/e4B/bWb/OfY4B6jniYvGVMeOwU7/5Jx7yfYDM6uL8/O2PQT8QOznDwO/fsj2e5kEztzD+0ROnEJe5GheB1QCT3POrZvZENGA+jbn3JdivwR8H/BhM3sncAf4vHPuNXF8xs865/5s+4GZ7Xnu2Tl3LTbKfzHwP83sb51zvxTPf4RzbsXMvkB0DfBXAX+8/XHAm5xznztkF8vOuQfMrBj4NNFz8u8mukjJPzrnXm7RixS/sM/7DfhB51zfQZ/BrmO7h3g/bz/bi3dsEJvZjM3MZB3wnpxYbSJJT9P1IkdTDEzGAv55QMPuDcysIbbNHwJ/BDwV+CrwHWa2fY49z8za4vzMLwHfH3tPPvBy4J/M7Ayw5Jz738BvxD5nt/XYjMJePkb0NMBziK4uRuz7f9x+j5m17Z5G38k5Nwf8NPCfY+8pBsZiL//ojk3ngcIdjz8HvGn7VEfs9MNu14jOFBxkv8/bz5d5/Hz664D/E/t5CHha7OeX8fiMwO66AdoArWAnvqCQFzmajwAPmtk3iIZE7x7bPJfoefh/IXrh2Lucc1NEQ+iPzaybaOiH4/lA59y3gA8CXwP+megyqf8CXAC+Fps2fxvwK3u8/VGge/vCu13+Fvgu4O9i580B3gdcBb5lZpeJXt1/4IxfrJYuouH560RnFf4v0SVJt/0jcHb7wjuiI/DMWG2XY49373cR6N/+xWgf+33efn4aeH3sz+CHgf8Ue/4Pge82s68RvfZhMfZ8N7ARu7hx+8K75wF/HcdniXhOS82KSNIys5cTPTXyC17XAhC7huGLwHc65za8rkfkMDonLyJJyzn3F2ZW7nUdO9QDb1XAi19oJC8iIpKidE5eREQkRSnkRUREUpRCXkREJEUp5EVERFKUQl5ERCRF/T+N8g4BnNAd3wAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 576x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "dr_roc_green = \"#03c75f\"\n",
    "white = \"#ffffff\"\n",
    "dr_purple = \"#65147D\"\n",
    "dr_dense_green = \"#018f4f\"\n",
    "\n",
    "threshold = roc.get_best_f1_threshold()\n",
    "fig = plt.figure(figsize=(8, 8))\n",
    "axes = fig.add_subplot(1, 1, 1, facecolor=dr_dark_blue)\n",
    "\n",
    "plt.scatter(roc_df.false_positive_rate, roc_df.true_positive_rate, color=dr_roc_green)\n",
    "plt.plot(roc_df.false_positive_rate, roc_df.true_positive_rate, color=dr_roc_green)\n",
    "plt.plot([0, 1], [0, 1], color=white, alpha=0.25)\n",
    "plt.title(\"ROC curve\")\n",
    "plt.xlabel(\"False Positive Rate (Fallout)\")\n",
    "plt.xlim([0, 1])\n",
    "plt.ylabel(\"True Positive Rate (Sensitivity)\")\n",
    "plt.ylim([0, 1])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature Impact\n",
    "\n",
    "[Feature Impact](https://docs.datarobot.com/en/docs/modeling/analyze-models/understand/feature-impact.html) measures how important a feature is in the context of a model. It measures how much the accuracy of a model would decrease if that feature was removed.\n",
    "\n",
    "Feature Impact is available for all model types and works by altering input data and observing the effect on a model’s score. It is an on-demand feature, meaning that you must initiate a calculation to see the results. Once DataRobot computes the feature impact for a model, that information is saved with the project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "autoscroll": "auto"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAp8AAAIBCAYAAAAPhNQ7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAApJklEQVR4nO3de7hddX3n8feHhHu4CqggCGKKIoWoiICKVBm8FrBFBUFxbMuDtFU6ig9eimCdImBttS3WjFZUEBmjgwxOBYsKI4MIkRDCTUQQKBRERC5yz3f+WOuUzek5yUlIfjs75/16nvWctX7r9lv7+5zks39r7bNTVUiSJEktrDHsDkiSJGn6MHxKkiSpGcOnJEmSmjF8SpIkqRnDpyRJkpoxfEqSJKkZw6ckSZKaMXxK0gSS3JTkwST3D0xbroBj7rOi+jiF8x2X5LRW55uKJKcm+fiw+yFpeAyfkjS536+qWQPTbcPsTJKZwzy/JK0Ihk9JWgZJNkryhSS3J/m3JB9PMqNft32S7yX5VZK7kpyeZON+3VeAbYD/3Y+ifiDJ3kluHXf8/xgd7Ucu5yU5Lcm9wDuXdP4p9L2SHJnk+iT3Jfmrvs8XJ7k3yf9Msla/7d5Jbk3yof5abkpyyMCx3pDk8n6/W5IcN+5cL0/y/5Lc069/Z5LDgUOAD/Svwf9ezjJIGmGGT0laNl8CHgOeC7wQ2Bf4435dgBOALYHnA1sDxwFU1duBm3liNPWkKZ5vf2AesDFw+lLOPxWvBV4M7A58AJhLFwi3BnYCDh7Y9hnAZsBWwGHA3CQ79OseAN7R9+sNwLuTHACQZBvgX4C/BzYH5gALqmpufw0n9a/B7y9DvyWtJgyfkjS5s/qRu3uSnJXk6cDrgKOq6oGquhP4W+AggKr6WVV9t6oerqpfAp8CXvkU+3BxVZ1VVYuBDZd0/ik6saruraqrgEXAeVX186r6DV1gfOG47f+yv54LgG8DbwGoqh9U1ZVVtbiqFgJn8MS1HgL8a1WdUVWPVtWvqmrB8l2+pNWNzw9J0uQOqKp/HVtIshuwJnB7krHmNYBb+vVbAJ8BXgFs0K/79VPswy0D889e0vmn6I6B+QcnWH7GwPKvq+qBgeVf0I3qkuSlwCfoRkvXAtYGvt5vtzVwwzL0SdI04sinJE3dLcDDwGZVtXE/bVhVL+jXnwAUsHNVbQgcSncrfkyNO94DwHpjC/2zm5uP22Zwn6Wdf0XbJMn6A8vbAGMfuvoqcDawdVVtBPwTT1zrLcD2kxxz/GsgaZoxfErSFFXV7cB5wN8k2TDJGv0HdsZuN28A3A/ck2Qr4Ohxh7gDeM7A8k+BdfoP76wJfIRuBHF5z78yHJ9krSSvAN7IE6ObGwB3V9VD/Yjw2wb2OR3YJ8lbksxM8rQkc/p1418DSdOM4VOSls076G4zX013S30e8Mx+3fHAi4Df0D0f+c1x+54AfKR/hvT9/XOWRwKfB/6NbiT0VpZsSedf0f69P8dtdIHyiKq6tl93JPCxJPcBxwL/c2ynqroZeD3wPuBuYAGwS7/6C8COY8/RrqR+S1qFpco7IJKkJ0uyN3BaVT1ryF2RtJpx5FOSJEnNGD4lSZLUjLfdJUmS1Iwjn5IkSWrG8ClJkqRmDJ+SJElqxvC5kiR5bZLrkvwsyTF924lJFib58sB2b0/y3uH1VABJtk7y/STXJLlqrCZJNk3y3STX9z836dtf1tfy0iTP7ds2TnJuBr73UO0kmZHk8iTn9MvWbkT0r/+8JNf2v4N7WL/RkOQv+n8zFyU5I8k61m7VleSfk9yZZNFA24T16td9sM8x1yV5Td+2dpLv9DU/cmDbuUleOJV+GD5Xgv4r8v4ReB2wI3Bwkl2APatqZ2BGkt9Nsi7wTuCUoXVWYx4D3ldVzwd2B/40yY7AMcD5VTUbOL9fhu6PZ/8h8CHg3X3bXwJ/XX6Kb1jeC1wzsGztRsenge9U1fPo/hj9NVi/VV7/LV7vAXatqp2AGcBBWLtV2anAa8e1TViv/v/Ag4AX9Puc0ueb1wDzgZ2Bw/ttdwHWqKrLp9IJw+fKsRvws6r6eVU9AnwN2A9Yq393ty7wKN1X732mqh4dXlcF3dcWVtVP+vn76P7z2wrYH/hSv9mXgAP6+Ufp6rge8GiS7YGtquqClv1WJ8mzgDfQfVPQGGs3ApJsCOxF981HVNUjVXUP1m9UzATWTTKTria3Ye1WWVV1Id23jg2arF77A1+rqoer6kbgZ3T5ZqyOMweO8Vd033Q2JTOXvomWw1bALQPLtwIvBb4BXE73zuI3wEuq6mPtu6clSbIt8ELgEuDp/fdpU1W3J9mi3+wEYC7wIPB24JN07+A1HH8HfIDu+8bHWLvR8Bzgl8AX+9GT+XSj2NZvFVdV/5bkk8DNdPU4r6rOS2LtRstk9doK+NHAdrf2bWfR1e8S4KQk+wHzq+q2qZ7Q8LlyTPTsSlXVScBJAEk+Dxyb5I+BfYGFVfXxhn3UBJLMonuTcFRV3TvZY0hVtYDu9jxJ9qJ7t58kZ9K9K3xfVd3RpNPTXJI3AndW1fx0Xwm5RNZulTMTeBHw51V1SZJP88Rt2v/E+q06+mcD9we2A+4Bvp7k0Mm2t3YjZ7Is8xjwNoAkawLnAvsl+RSwDfDlqjp7SQf2tvvKcSuw9cDys+h+yQAYeCD3p8A7quotwE5JZrfrosbrf4m+AZxeVd/sm+9I8sx+/TOBO8ftE+AjdLccPtpPp9E9B6U2Xkb3D99NdI+4vCrJaVi7UXErcGtVXdIvz6MLo9Zv1bcPcGNV/bJ/fOybwJ5Yu1EzWb2WmGV6R9Ldqt8DeAR4K11tl8jwuXJcCsxOsl2Stege2B18FzD2bMSadA9oAyymew5GQ9D/Y/gF4Jqq+tTAqrOBw/r5w4Bvjdv1MODbVfVruvotxlo2VVUfrKpnVdW2dL9r36uqQ7F2I6Gq/h24JckOfdOrgauxfqPgZmD3JOv1/4a+mu55eWs3Wiar19nAQf2n27cDZgM/HtupH/l+I/BlnqhjAess9YxV5bQSJuD1dCObNwAfHmg/APjowPIngSvpRtuG3u/pOgEv739pFgIL+un1wNPontG9vv+56cA+6wHfB9bsl1/R13I+8DvDvqbpOAF7A+f089ZuRCZgDnBZ//t3FrCJ9RuNCTgeuBZYBHwFWNvarboTcAZwO91jDrcCf7SUen24zzHXAa8bd6y/BV7Zz68DnAdcRfcIzRL74Xe7S5IkqRlvu0uSJKkZw6ckSZKaMXxKkiSpGcOnJEmSmjF8SpIkqRnDpyRJkpoxfEqSJKkZv9u9sRlrrl01Y61hd0PLac2ZM3j0sceH3Q0tJ+s3uqzdaLN+o6sevv+uqtp8RR7T8NnYzLXXY41n7DLsbmg5bbnFptx2593D7oaWk/UbXdZutFm/0fXQDRf8YkUf09vukiRJasbwKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZgyfkiRJasbwKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZgyfkiRJasbwKUmSpGYMn5IkSWrG8ClJkqRmZg67A9PNzA03Y/MDjx52N7ScNlu3ePzBDLsbWk7Wb3RZu9Fm/UbXL068YIUf05FPSZIkNWP4lCRJUjOGT0mSJDVj+JQkSVIzhk9JkiQ1Y/iUJElSM4ZPSZIkNWP4lCRJUjOGT0mSJDVj+JQkSVIzhk9JkiQ1Y/iUJElSM4ZPSZIkNWP4lCRJUjMjHz6T3N//3DLJvIH2M5IsTPIXS9j3gCQ7Dix/LMk+K7fHkiRJ09fMYXdgRamq24ADAZI8A9izqp69lN0OAM4Bru6PcezK7KMkSdJ0N/Ijn2OSbJtkUb94HrBFkgVJXpFk+yTfSTI/yf9N8rwkewL7ASf3222f5NQkYwH2piR/neTiJJcleVGSc5PckOSIgfMeneTSfpT1+PZXLkmSNDpWm5HPcfYDzqmqOQBJzgeOqKrrk7wUOKWqXpXk7H67ef12449zS1XtkeRvgVOBlwHrAFcB/5RkX2A2sBsQ4Owke1XVhYMHSXI4cDjAWhttvjKuV5IkaSSsruHzPySZBewJfH0gXK49xd3P7n9eCcyqqvuA+5I8lGRjYN9+urzfbhZdGH1S+KyqucBcgPW3nF3LdyWSJEmjb7UPn3SPFtwzNgq6jB7ufy4emB9bnkk32nlCVX3uKfVQkiRpmlhtnvmcTFXdC9yY5M0A6ezSr74P2OApHP5c4F396CpJtkqyxVPqsCRJ0mpstQ+fvUOAP0pyBd3zmvv37V8Djk5yeZLtl/WgVXUe8FXg4iRXAvN4amFWkiRptZYqH0Fsaf0tZ9fm7/i7YXdDy+np6xZ3PPifPpimEWH9Rpe1G23Wb3T94sQ3zq+qXVfkMafLyKckSZJWAYZPSZIkNWP4lCRJUjOGT0mSJDVj+JQkSVIzhk9JkiQ1Y/iUJElSM4ZPSZIkNWP4lCRJUjOGT0mSJDVj+JQkSVIzhk9JkiQ1M3PYHZhuHrv3Lu6Yd/Kwu6HlNGOLTbnjzruH3Q0tJ+s3uqzdaLN+GuTIpyRJkpoxfEqSJKkZw6ckSZKaMXxKkiSpGcOnJEmSmjF8SpIkqRnDpyRJkpoxfEqSJKkZw6ckSZKaMXxKkiSpGcOnJEmSmjF8SpIkqRnDpyRJkpoxfEqSJKkZw6ckSZKaMXxKkiSpGcOnJEmSmjF8SpIkqRnDpyRJkpoxfEqSJKkZw6ckSZKaMXxKkiSpGcOnJEmSmjF8SpIkqRnDpyRJkpoxfEqSJKkZw6ckSZKaMXxKkiSpGcOnJEmSmjF8SpIkqRnDpyRJkpoxfEqSJKkZw6ckSZKaMXxKkiSpGcOnJEmSmjF8SpIkqZmZw+7AdDNzw83Y/MCjh90NLafN1i0efzDD7sak7ph38rC7IEnSEjnyKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZgyfkiRJasbwKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZgyfkiRJasbwKUmSpGYMnwOS/L9J2k9NcmDr/kiSJK1uVtvwmWTGsu5TVXuuoHPPXBHHkSRJWt2MZPhMsm2Sa5N8KcnCJPOSrJfkpiTHJvkh8OYkBye5MsmiJCf2+747yUkDx3pnkr/v5+/vfybJPyS5Osm3gS0Gtn9xkguSzE9ybpJn9u0/SPLXSS4A3tvw5ZAkSRoZIxk+ezsAc6tqZ+Be4Mi+/aGqejlwIXAi8CpgDvCSJAcA84A/GDjOW4Ezxx37Tf3xfxf4E2BPgCRrAn8PHFhVLwb+GfjvA/ttXFWvrKq/WUHXKEmStFoZ5dvDt1TVRf38acB7+vmxIPkS4AdV9UuAJKcDe1XVWUl+nmR34Hq6kHkRT7YXcEZVPQ7cluR7ffsOwE7Ad5MAzABuH9hvfIilP/fhwOEAa220+fJcqyRJ0mphlMNnTbL8QP8zS9j3TOAtwLXA/6qq8cea6Phjx7yqqvaY5LgPTNRYVXOBuQDrbzl7ouNKkiRNC6N8232bJGMh8GDgh+PWXwK8Mslm/YePDgYu6Nd9Ezigb5totPJC4KAkM/pnOn+vb78O2HzsvEnWTPKCFXVBkiRJq7tRDp/XAIclWQhsCnx2cGVV3Q58EPg+cAXwk6r6Vr/u18DVwLOr6scTHPt/0d2Sv7I/7gX9fo8ABwInJrkCWED/PKgkSZKWbpRvuy+uqiPGtW07uFBVXwW+OtHOVfXGCdpm9T8L+LNJ9ltA90zo+Pa9p9BnSZKkaW2URz4lSZI0YkZy5LOqbqL71LkkSZJGiCOfkiRJasbwKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZgyfkiRJasbwKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqZiS/232UPXbvXdwx7+Rhd0PLacYWm3LHnXcPuxuSJI0sRz4lSZLUjOFTkiRJzRg+JUmS1IzhU5IkSc0YPiVJktSM4VOSJEnNGD4lSZLUjOFTkiRJzRg+JUmS1IzhU5IkSc0YPiVJktSM4VOSJEnNGD4lSZLUjOFTkiRJzRg+JUmS1IzhU5IkSc0YPiVJktSM4VOSJEnNGD4lSZLUjOFTkiRJzRg+JUmS1IzhU5IkSc0YPiVJktSM4VOSJEnNGD4lSZLUjOFTkiRJzRg+JUmS1IzhU5IkSc0YPiVJktSM4VOSJEnNGD4lSZLUjOFTkiRJzRg+JUmS1IzhU5IkSc0YPiVJktTMzGF3YLqZueFmbH7g0cu17x3zTl7BvZEkSWrLkU9JkiQ1Y/iUJElSM4ZPSZIkNWP4lCRJUjOGT0mSJDVj+JQkSVIzhk9JkiQ1Y/iUJElSM4ZPSZIkNWP4lCRJUjOGT0mSJDVj+JQkSVIzhk9JkiQ1Y/iUJElSM9MufCbZOMmRw+6HJEnSdDTtwiewMWD4lCRJGoLpGD4/AWyfZEGSk5McneTSJAuTHD+2UZKzksxPclWSwwfa709yYr/uX5PsluQHSX6eZL+hXJEkSdKImI7h8xjghqqaA3wXmA3sBswBXpxkr367d1XVi4FdgfckeVrfvj7wg37dfcDHgf8CvAn42EQnTHJ4ksuSXPbYb3+zcq5KkiRpBMwcdgeGbN9+urxfnkUXRi+kC5xv6tu37tt/BTwCfKdvvxJ4uKoeTXIlsO1EJ6mqucBcgPW3nF0r/jIkSZJGw3QPnwFOqKrPPakx2RvYB9ijqn6b5AfAOv3qR6tqLEAuBh4GqKrFSab76ylJkrRE0/G2+33ABv38ucC7kswCSLJVki2AjYBf98HzecDuw+mqJEnS6mXajdRV1a+SXJRkEfAvwFeBi5MA3A8cSndb/YgkC4HrgB8Nq7+SJEmrk2kXPgGq6m3jmj49wWavm2TfWQPzx022TpIkSf/ZdLztLkmSpCExfEqSJKkZw6ckSZKaMXxKkiSpGcOnJEmSmjF8SpIkqRnDpyRJkpoxfEqSJKkZw6ckSZKaMXxKkiSpGcOnJEmSmjF8SpIkqZmZw+7AdPPYvXdxx7yTh90NSZKkoXDkU5IkSc0YPiVJktSM4VOSJEnNGD4lSZLUjOFTkiRJzRg+JUmS1IzhU5IkSc0YPiVJktSM4VOSJEnNGD4lSZLUjOFTkiRJzRg+JUmS1IzhU5IkSc0YPiVJktSM4VOSJEnNGD4lSZLUjOFTkiRJzRg+JUmS1IzhU5IkSc0YPiVJktSM4VOSJEnNGD4lSZLUjOFTkiRJzRg+JUmS1IzhU5IkSc0YPiVJktSM4VOSJEnNGD4lSZLUjOFTkiRJzRg+JUmS1IzhU5IkSc0YPiVJktSM4VOSJEnNGD4lSZLUjOFTkiRJzRg+G5u54WbD7oIkSdLQGD4lSZLUjOFTkiRJzRg+JUmS1IzhU5IkSc0YPiVJktSM4VOSJEnNGD4lSZLUjOFTkiRJzRg+JUmS1IzhU5IkSc0YPiVJktSM4VOSJEnNGD4lSZLUjOFTkiRJzSwxfCbZOMmRrTqzvJIclWS9FXi845K8fxn3+UGSXVdUHyRJklZHSxv53BgYevhMZ0l9PQpYpvCZZOZT6pQkSZKW2dLC5yeA7ZMsSHJykqOTXJpkYZLjAZJsm+TaJJ9PsijJ6Un2SXJRkuuT7NZvd1ySryT5Xt/+J2MnWcJxr0lyCvATYOskn01yWZKrBrZ7D7Al8P0k3+/b7h849oFJTu3nT03yqX67E5Nsn+Q7SeYn+b9Jnjf+BehHNE9M8uMkP03yir593SRf6/t8JrDuclVAkiRpGlna6N8xwE5VNSfJvsCBwG5AgLOT7AXcDDwXeDNwOHAp8Dbg5cB+wIeAA/rj7QzsDqwPXJ7k28BOwOxJjrsD8F+r6kiAJB+uqruTzADOT7JzVX0myX8Dfq+q7prCNf8OsE9VPZ7kfOCIqro+yUuBU4BXTfQ6VdVuSV4PfBTYB3g38Nuq2jnJznQBWZIkSUuwLLee9+2ny/vlWXSh8Wbgxqq6EiDJVcD5VVVJrgS2HTjGt6rqQeDBfvRxN7qQOtlxf1FVPxrY/y1JDu/7/UxgR2DhMlwDwNf74DkL2BP4epKxdWtPss83+5/zB65nL+AzAFW1MMmk/ej7fDjAWhttzhrrbbWMXZYkSVo9LEv4DHBCVX3uSY3JtsDDA02LB5YXjztHjTtmLeW4Dwwsbwe8H3hJVf26v5W+ziR9HTzP+G3GjrkGcE9VzZnkGIPGrudxlnw9E3emai4wF2D9LWfX4qnsJEmStBpa2jOf9wEb9PPnAu/qRwxJslWSLZbxfPsnWSfJ04C96W7RT/W4G9IFx98keTrwukn6CXBHkuf3H1J600Qdqap7gRuTvLk/b5LssgzXciFwSL/vTnSPFEiSJGkJljjyWVW/6j84tAj4F+CrwMX9ber7gUPpRgOn6sfAt4FtgL+qqtuA25I8f2nHraorklwOXAX8HLhoYPVc4F+S3F5Vv0f3rOo5wC3AIrpb+RM5BPhsko8AawJfA66Y4rV8Fvhif7t9QX9tkiRJWoJUTenO8VM/UXIccH9VfbLJCVdR6285uxb7zOfI2nKLTbntzruH3Q0tJ+s3uqzdaLN+o+uhGy6YX1Ur9O+Y+w1HkiRJaqbZH1qvquNanUuSJEmrJkc+JUmS1IzhU5IkSc0YPiVJktSM4VOSJEnNGD4lSZLUjOFTkiRJzRg+JUmS1IzhU5IkSc0YPiVJktSM4VOSJEnNGD4lSZLUjOGzscfuvWvYXZAkSRoaw6ckSZKaMXxKkiSpGcOnJEmSmjF8SpIkqRnDpyRJkpoxfEqSJKkZw6ckSZKaMXxKkiSpGcOnJEmSmjF8SpIkqRnDpyRJkpoxfEqSJKkZw6ckSZKaMXxKkiSpGcOnJEmSmjF8SpIkqRnDpyRJkpoxfEqSJKkZw6ckSZKaMXxKkiSpGcOnJEmSmjF8SpIkqRnDpyRJkpoxfEqSJKkZw6ckSZKaMXxKkiSpGcOnJEmSmjF8SpIkqRnDpyRJkpoxfEqSJKkZw6ckSZKaMXxKkiSpGcOnJEmSmjF8SpIkqRnDpyRJkpoxfEqSJKkZw6ckSZKaMXxKkiSpGcOnJEmSmjF8SpIkqRnDpyRJkpoxfEqSJKkZw6ckSZKaMXxKkiSpGcOnJEmSmjF8SpIkqRnDpyRJkpoxfEqSJKkZw6ckSZKaMXxKkiSpGcPnJJLclGSzZdh+2ySLVmafJEmSRt0qFz6TzFgdzyVJkqTG4bMfHbw2yZeSLEwyL8l6/SjjsUl+CLw5yb5JLk7ykyRfTzKr3/8TSa7u9/1k3/bmJIuSXJHkwr7tnUn+YeC85yTZu5+/P8nHklwC7JHk0CQ/TrIgyefGB9K+z9ck+R9JrkpyXpJ1+3Uv7s97MfCnDV5CSZKkkTaMkc8dgLlVtTNwL3Bk3/5QVb0c+FfgI8A+VfUi4DLgvyXZFHgT8IJ+34/3+x0LvKaqdgH2m8L51wcWVdVLgV8BbwVeVlVzgMeBQybYZzbwj1X1AuAe4A/79i8C76mqPZZ0wiSHJ7ksyWWLH314Cl2UJElaPQ0jfN5SVRf186cBL+/nz+x/7g7sCFyUZAFwGPBsuqD6EPD5JH8A/Lbf/iLg1CR/AkzlNvrjwDf6+VcDLwYu7c/1auA5E+xzY1Ut6OfnA9sm2QjYuKou6Nu/MtkJq2puVe1aVbuusebaU+iiJEnS6mnmEM5Zkyw/0P8M8N2qOnj8jkl2owuIBwF/Bryqqo5I8lLgDcCCJHOAx3hysF5nYP6hqnp84FxfqqoPLqXPg8OVjwPr9vuOvxZJkiQtwTBGPrdJMnab+mDgh+PW/wh4WZLnAvTPhP5O/9znRlX1f4CjgDn9+u2r6pKqOha4C9gauAmYk2SNJFsDu03Sl/OBA5Ns0R9r0yTPnspFVNU9wG+SjI3cTnS7XpIkSQOGMfJ5DXBYks8B1wOfBf58bGVV/TLJO4Ezkozdo/4IcB/wrSTr0I06/kW/7uQks/u284Er+vYbgSuBRcBPJupIVV2d5CPAeUnWAB6l++DQL6Z4Lf8V+OckvwXOneI+kiRJ01aq2t05TrItcE5V7dTspKuYtWdtUms8Y5dhd0PLacstNuW2O+8edje0nKzf6LJ2o836ja6HbrhgflXtuiKPucr9nU9JkiStvpredq+qm4BpO+opSZI03TnyKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZgyfkiRJasbwKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZgyfkiRJasbwKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZgyfkiRJasbwKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZgyfkiRJasbwKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZgyfkiRJasbwKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZgyfkiRJasbwKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZgyfkiRJasbwKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZgyfkiRJasbwKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZgyfkiRJasbwKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZgyfkiRJasbwKUmSpGZGJnwm2S/JMUtYPyfJ61fwOe9fxu33TnLOiuyDJEnS6mRkwmdVnV1Vn1jCJnOAZQqf6YzMayBJkjTqmgavJNsmuTbJ55MsSnJ6kn2SXJTk+iS7Jdk0yVlJFib5UZKd+33fmeQf+vk39/tfkeTCJGsBHwPemmRBkrcmOS7J+wfOvag//7ZJrklyCvATYOskRye5tD/n8RP0e+8kP0gyr+//6UnSr3tt3/ZD4A8avIySJEkjaxijfs8FPg3sDDwPeBvwcuD9wIeA44HLq2rnfvnLExzjWOA1VbULsF9VPdK3nVlVc6rqzKX0YQfgy1X1wn5+NrAb3ejpi5PsNcE+LwSOAnYEngO8LMk6wP8Afh94BfCMqbwAkiRJ09UwwueNVXVlVS0GrgLOr6oCrgS2pQuiXwGoqu8BT0uy0bhjXAScmuRPgBnL0YdfVNWP+vl9++lyupHQ59GF0fF+XFW39v1e0Pf1ef31XN9fw2kTnSzJ4UkuS3LZ4kcfXo7uSpIkrR5mDuGcg+lr8cDyYrr+PDbBPvWkhaojkrwUeAOwIMmcCfZ5jCeH63UG5h8YmA9wQlV9bhn6/ThPvHY1wbZPUlVzgbkAa8/aZKnbS5Ikra5WxQ/bXAgcAt2zlsBdVXXv4AZJtq+qS6rqWOAuYGvgPmCDgc1uAl7Ub/8iYLtJzncu8K4ks/ptt0qyxRT7ei2wXZLt++WDp7ifJEnStDSMkc+lOQ74YpKFwG+BwybY5uQks+lGLc8HrgBuBo5JsgA4AfgG8I5++VLgpxOdrKrOS/J84OL+M0T3A4cCdy6to1X1UJLDgW8nuQv4IbDTlK9UkiRpmkn3qKJaWXvWJrXGM3YZdje0nLbcYlNuu/PuYXdDy8n6jS5rN9qs3+h66IYL5lfVrivymKvibXdJkiStpgyfkiRJasbwKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZgyfkiRJasbwKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZgyfkiRJasbwKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZgyfkiRJasbwKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZgyfkiRJasbwKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZlJVw+7DtJLkPuC6YfdDy20z4K5hd0LLzfqNLms32qzf6NqhqjZYkQecuSIPpim5rqp2HXYntHySXGb9Rpf1G13WbrRZv9GV5LIVfUxvu0uSJKkZw6ckSZKaMXy2N3fYHdBTYv1Gm/UbXdZutFm/0bXCa+cHjiRJktSMI5+SJElqxvC5kiR5bZLrkvwsyTF924lJFib58sB2b0/y3uH1VABJtk7y/STXJLlqrCZJNk3y3STX9z836dtf1tfy0iTP7ds2TnJukgzzWqarJDOSXJ7knH7Z2o2I/vWfl+Ta/ndwD+s3GpL8Rf9v5qIkZyRZx9qtupL8c5I7kywaaJuwXv26D/Y55rokr+nb1k7ynb7mRw5sOzfJC6fSD8PnSpBkBvCPwOuAHYGDk+wC7FlVOwMzkvxuknWBdwKnDK2zGvMY8L6qej6wO/CnSXYEjgHOr6rZwPn9MsD7gD8EPgS8u2/7S+Cvy2dZhuW9wDUDy9ZudHwa+E5VPQ/Yha6O1m8Vl2Qr4D3ArlW1EzADOAhrtyo7FXjtuLYJ69X/H3gQ8IJ+n1P6fPMaYD6wM3B4v+0uwBpVdflUOmH4XDl2A35WVT+vqkeArwH7AWv17+7WBR4FjgY+U1WPDq+rAqiq26vqJ/38fXT/+W0F7A98qd/sS8AB/fyjdHVcD3g0yfbAVlV1Qct+q5PkWcAbgM8PNFu7EZBkQ2Av4AsAVfVIVd2D9RsVM4F1k8ykq8ltWLtVVlVdCNw9rnmyeu0PfK2qHq6qG4Gf0eWbsToO/q34vwKOnWo//CPzK8dWwC0Dy7cCLwW+AVxO987iN8BLqupj7bunJUmyLfBC4BLg6VV1O3QBNckW/WYn0H0C8EHg7cAn6d7Bazj+DvgAMPgtHNZuNDwH+CXwxX70ZD7dKLb1W8VV1b8l+SRwM109zquq85JYu9EyWb22An40sN2tfdtZdPW7BDgpyX7A/Kq6baonNHyuHBM9u1JVdRJwEkCSzwPHJvljYF9gYVV9vGEfNYEks+jeJBxVVfdO9hhSVS2guz1Pkr3o3u0nyZl07wrfV1V3NOn0NJfkjcCdVTU/yd5L297arXJmAi8C/ryqLknyaZ64TfufWL9VR/9s4P7AdsA9wNeTHDrZ9tZu5EyWZR4D3gaQZE3gXGC/JJ8CtgG+XFVnL+nA3nZfOW4Fth5YfhbdLxkAAw/k/hR4R1W9Bdgpyex2XdR4/S/RN4DTq+qbffMdSZ7Zr38mcOe4fQJ8hO6Ww0f76TS656DUxsvo/uG7ie4Rl1clOQ1rNypuBW6tqkv65Xl0YdT6rfr2AW6sql/2j499E9gTazdqJqvXErNM70i6W/V7AI8Ab6Wr7RIZPleOS4HZSbZLshbdA7uD7wLGno1Yk+4BbYDFdM/BaAj6fwy/AFxTVZ8aWHU2cFg/fxjwrXG7HgZ8u6p+TVe/xVjLpqrqg1X1rKralu537XtVdSjWbiRU1b8DtyTZoW96NXA11m8U3AzsnmS9/t/QV9M9L2/tRstk9TobOKj/dPt2wGzgx2M79SPfbwS+zBN1LGCdpZ6xqpxWwgS8nm5k8wbgwwPtBwAfHVj+JHAl3Wjb0Ps9XSfg5f0vzUJgQT+9Hnga3TO61/c/Nx3YZz3g+8Ca/fIr+lrOB35n2Nc0HSdgb+Ccft7ajcgEzAEu63//zgI2sX6jMQHHA9cCi4CvAGtbu1V3As4Abqd7zOFW4I+WUq8P9znmOuB14471t8Ar+/l1gPOAq+geoVliP/yGI0mSJDXjbXdJkiQ1Y/iUJElSM4ZPSZIkNWP4lCRJUjOGT0mSJDVj+JSkVUySx5MsGJiO6dtfkeSqvm3dJCf3yycvxzk+tOJ7LklL559akqRVTJL7q2rWBO3/BFxSVV/sl+8FNq+qh1fUOSRpZfO73SVpBCT5Y+AtwGuS7ANsAKwPXJLkBOB7wD/RfbcywFFVdVGSWcDfA7vSfZHC8cBLgHWTLACuqqpDml6MpGnNkU9JWsUkeZzuW1/GnFBVZyY5le4bnOb12/3H6GWSrwKnVNUPk2wDnFtVz09yIrB2VR3Vb7dJVf3akU9Jw+LIpySteh6sqjnLuM8+wI7dV2wDsGGSDfr2g8Yaq/s+bUkaGsOnJK0e1gD2qKoHBxvTpVFvcUlaZfhpd0laPZwH/NnYQpI5k7Rv0s8+mmTNZr2TpJ7hU5JWPeuO+1NLn5jCPu8Bdk2yMMnVwBF9+8eBTZIsSnIF8Ht9+1xgYZLTV3z3JWlyfuBIkiRJzTjyKUmSpGYMn5IkSWrG8ClJkqRmDJ+SJElqxvApSZKkZgyfkiRJasbwKUmSpGYMn5IkSWrm/wMm2Lw2MtNp7QAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "feature_impacts = model_top.get_or_request_feature_impact()\n",
    "\n",
    "# Limit size to make chart look good. Display top 25 values\n",
    "if len(feature_impacts) > 25:\n",
    "    feature_impacts = feature_impacts[0:24]\n",
    "\n",
    "# Formats the ticks from a float into a percent\n",
    "percent_tick_fmt = mtick.PercentFormatter(xmax=1.0)\n",
    "\n",
    "impact_df = pd.DataFrame(feature_impacts)\n",
    "impact_df.sort_values(by=\"impactNormalized\", ascending=True, inplace=True)\n",
    "\n",
    "# Positive values are blue, negative are red\n",
    "bar_colors = impact_df.impactNormalized.apply(lambda x: dr_red if x < 0 else dr_blue)\n",
    "\n",
    "ax = impact_df.plot.barh(\n",
    "    x=\"featureName\", y=\"impactNormalized\", legend=False, color=bar_colors, figsize=(10, 8)\n",
    ")\n",
    "ax.xaxis.set_major_formatter(percent_tick_fmt)\n",
    "ax.xaxis.set_tick_params(labeltop=True)\n",
    "ax.xaxis.grid(True, alpha=0.2)\n",
    "ax.set_facecolor(dr_dark_blue)\n",
    "\n",
    "plt.ylabel(\"\")\n",
    "plt.xlabel(\"Effect\")\n",
    "plt.xlim((None, 1))  # Allow for negative impact\n",
    "plt.title(\"Feature Impact\", y=1.04);"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Make predictions\n",
    "\n",
    "### Test predictions\n",
    "\n",
    "After determining the top-performing model from the Leaderboard, upload the prediction test dataset to verify that the model generates predictions successfully before deploying the model to a production environment. The predictions are returned as a Pandas dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_path_scoring = (\n",
    "    \"https://s3.amazonaws.com/datarobot-use-case-datasets/maintenance_predictions.csv\"\n",
    ")\n",
    "scoring_df = pd.read_csv(data_path_scoring, encoding=\"ISO-8859-1\")\n",
    "\n",
    "prediction_dataset = project.upload_dataset(scoring_df)\n",
    "predict_job = model_top.request_predictions(prediction_dataset.id)\n",
    "prediction_dataset.id\n",
    "\n",
    "predictions = predict_job.get_result_when_complete()\n",
    "pd.concat([scoring_df, predictions], axis=1)\n",
    "predictions.positive_probability.plot(kind=\"hist\", title=\"Predicted Probabilities\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy a model to production\n",
    "\n",
    "\n",
    "If you are happy with the model's performance, you can deploy it to a production environment with [MLOps](https://docs.datarobot.com/en/mlops/index.html).  Deploying the model will free up workers, as data scored through the deployment doesn't use any modeling workers. Furthermore, you are no longer restricted on the amount of data to score; score over 100GB with the deployment. Deployments also offer many model management benefits: monitoring service, data drift, model comparison, retraining, and more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "autoscroll": "auto"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Deployment(Late Shipment Predictions)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Retrieve a prediction server\n",
    "prediction_server = dr.PredictionServer.list()[0]\n",
    "\n",
    "# Get the top performing model. Uncomment if this did not execute in the previous section\n",
    "# model_top = sorted_by_metric(models, 'crossValidation', metric)[0]\n",
    "\n",
    "deployment = dr.Deployment.create_from_learning_model(\n",
    "    model_top.id,\n",
    "    label=\"Predictive Maintenance\",\n",
    "    description=\"Predict Whether a Parts Failure Will Occur\",\n",
    "    default_prediction_server_id=prediction_server.id,\n",
    ")\n",
    "deployment.id"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure batch predictions\n",
    "\n",
    "After the model has been deployed, DataRobot creates an endpoint for real-time scoring. The deployment allows you to use DataRobot's batch prediction API to score large datasets with a deployed DataRobot model. \n",
    "\n",
    "The batch prediction API provides flexible intake and output options when scoring large datasets using prediction servers. The API is exposed through the DataRobot Public API and can be consumed using a REST-enabled client or Public API bindings for DataRobot's Python client."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Set the deployment ID"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before proceeding, provide the deployed model's deployment ID (retrieved from the deployment's [Overview tab](https://docs.datarobot.com/en/docs/mlops/monitor/dep-overview.html) or from the Deployment object in the Python client: `deployment.id`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deployment_id = \"YOUR_DEPLOYMENT_ID\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Determine input and output options\n",
    "\n",
    "DataRobot's batch prediction API allows you to score data from and to multiple sources. You can take advantage of the credentials and data sources you have already established previously through the UI for easy scoring. Credentials are usernames and passwords, while data sources are any databases with which you have previously established a connection (e.g., Snowflake). View the example code below outlining how to query credentials and data sources.\n",
    "\n",
    "You can reference the full list of DataRobot's supported [input](https://docs.datarobot.com/en/docs/predictions/batch/batch-prediction-api/intake-options.html) and [output options](https://docs.datarobot.com/en/docs/predictions/batch/batch-prediction-api/output-options.html).\n",
    "\n",
    "Reference the DataRobot documentation for more information about [data connections](https://docs.datarobot.com/en/docs/data/connect-data/data-conn.html)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The snippet below shows how you can query all credentials tied to a DataRobot account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dr.Credential.list()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output above returns multiple sets of credentials. The alphanumeric string included in each item of the list is the credentials ID. You can use that ID to access credentials through the API."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The snippet below shows how you can query all data sources tied to a DataRobot account. The second line lists each datastore with an alphanumeric string; that is the datastore ID."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5e6696ff820e737a5bd78430\n"
     ]
    }
   ],
   "source": [
    "dr.DataStore.list()\n",
    "print(dr.DataStore.list()[0].id)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scoring examples"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The snippets below demonstrate how to score data with the Batch Prediction API. Edit the `intake_settings` and `output_settings` to suit your needs. You can mix and match until you get the outcome you prefer."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Score from CSV to CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scoring without Prediction Explanations\n",
    "if False:\n",
    "    results_df = deployment.predict_batch(source=Path(\"inputfile.csv\"))\n",
    "    result_df\n",
    "\n",
    "# Scoring with Prediction Explanations\n",
    "if False:\n",
    "    dr.BatchPredictionJob.score(\n",
    "        deployment_id,\n",
    "        intake_settings={\n",
    "            \"type\": \"localFile\",\n",
    "            \"file\": \"inputfile.csv\",  # Provide the filepath, Pandas dataframe, or file-like object here\n",
    "        },\n",
    "        output_settings={\"type\": \"localFile\", \"path\": \"outputfile.csv\"},\n",
    "        max_explanations=3,  # Compute Prediction Explanations for the amount of features indicated here\n",
    "    )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Score from S3 to S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if False:\n",
    "    dr.BatchPredictionJob.score(\n",
    "        deployment_id,\n",
    "        intake_settings={\n",
    "            \"type\": \"s3\",\n",
    "            \"url\": \"s3://theos-test-bucket/lending_club_scoring.csv\",  # Provide the URL of your datastore here\n",
    "            \"credential_id\": \"YOUR_CREDENTIAL_ID_FROM_ABOVE\",  # Provide your credentials here\n",
    "        },\n",
    "        output_settings={\n",
    "            \"type\": \"s3\",\n",
    "            \"url\": \"s3://theos-test-bucket/lending_club_scored2.csv\",\n",
    "            \"credential_id\": \"YOUR_CREDENTIAL_ID_FROM_ABOVE\",\n",
    "        },\n",
    "    )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Score from JDBC to JDBC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if False:\n",
    "    dr.BatchPredictionJob.score(\n",
    "        deployment_id,\n",
    "        intake_settings={\n",
    "            \"type\": \"jdbc\",\n",
    "            \"table\": \"table_name\",\n",
    "            \"schema\": \"public\",\n",
    "            \"dataStoreId\": data_store.id,  # Provide the ID of your datastore here\n",
    "            \"credentialId\": cred.credential_id,  # Provide your credentials here\n",
    "        },\n",
    "        output_settings={\n",
    "            \"type\": \"jdbc\",\n",
    "            \"table\": \"table_name\",\n",
    "            \"schema\": \"public\",\n",
    "            \"statementType\": \"insert\",\n",
    "            \"dataStoreId\": data_store.id,\n",
    "            \"credentialId\": cred.credential_id,\n",
    "        },\n",
    "    )"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}