{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "452659b1",
   "metadata": {},
   "source": [
    "File 01-descriptive.py\n",
    "\n",
    "\n",
    "Michel Bierlaire\n",
    "\n",
    "Sat Aug 02 2025, 16:32:08\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "917a762a",
   "metadata": {},
   "source": [
    "Before using a data file for modeling purposes, it is important to\n",
    "collect some information about its content. The objective of this lab is to extract some descriptive statistics\n",
    "from a database with choice data using the package `pandas`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4b29c61",
   "metadata": {},
   "source": [
    "We introduce some examples using the file `swissmetro.dat`. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f873f55",
   "metadata": {},
   "source": [
    "We first import `pandas`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "148e2afd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from IPython.core.display_functions import display\n",
    "from matplotlib import pyplot as plt\n",
    "from biogeme.data.swissmetro import read_data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f38283d",
   "metadata": {},
   "source": [
    "The data file is available at\n",
    "[http://transp-or.epfl.ch/data/swissmetro.dat](http://transp-or.epfl.ch/data/swissmetro.dat)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41ad4174",
   "metadata": {},
   "source": [
    "The\n",
    "description of the columns of the file is\n",
    "available [here](http://transp-or.epfl.ch/documents/technicalReports/CS_SwissmetroDescription.pdf)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "787591d2",
   "metadata": {},
   "source": [
    "Read the data. It is provided with the Biogeme distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c4624a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "swissmetro = read_data()\n",
    "display(swissmetro.dataframe)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d2535d5",
   "metadata": {},
   "source": [
    "- The database contains 10728 rows of data, corresponding to each\n",
    "observation in the sample.\n",
    "- It contains 28 columns, corresponding to\n",
    "the available variables."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb706a36",
   "metadata": {},
   "source": [
    "The list of columns is reported below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20a3cf2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(swissmetro.dataframe.columns)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5081a22",
   "metadata": {},
   "source": [
    "If we look at the column `ID`, we observe that it contains 1192\n",
    "unique values, corresponding to the 1192 individuals that have\n",
    "participated in the survey. Each of these respondents was asked to\n",
    "perform 9 choice exercises, for a total of 10728 observations (the\n",
    "number of rows in the file)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64a59b14",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "display(swissmetro.dataframe['ID'].unique())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6b1ea04",
   "metadata": {},
   "source": [
    "If we look at the column `PURPOSE`, corresponding to the trip\n",
    "purpose, it contains a total of 9 unique values, numbered from 1 to\n",
    "9. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea592d70",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(swissmetro.dataframe['PURPOSE'].unique())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6fe0182",
   "metadata": {},
   "source": [
    "In order to understand better the distribution of these values, we\n",
    "can calculate the frequency of each value, here sorted by decreasing\n",
    "order of frequency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22f5c2d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(swissmetro.dataframe['PURPOSE'].value_counts())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c5a0cc6",
   "metadata": {},
   "source": [
    "The histogram of this distribution is also useful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d30ce12",
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = swissmetro.dataframe['PURPOSE'].value_counts().plot(title='PURPOSE', kind='bar')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26f3ce3a",
   "metadata": {},
   "source": [
    "We do the same for the `CHOICE`variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "711dfdd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(swissmetro.dataframe['CHOICE'].value_counts())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "717a947a",
   "metadata": {},
   "source": [
    "And the histogram..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "208f4ec1",
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = swissmetro.dataframe['CHOICE'].value_counts().plot(title='CHOICE', kind='bar')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "306c1ffe",
   "metadata": {},
   "source": [
    "If we look at the column `INCOME`, we note that it is also\n",
    "coded as a discrete variables, with 5 unique values, distributed as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "490a3aa6",
   "metadata": {},
   "outputs": [],
   "source": [
    "swissmetro.dataframe['INCOME'].value_counts()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98d554a5",
   "metadata": {},
   "source": [
    "And we can represent the histogram using horizontal bars. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8b4d758",
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = swissmetro.dataframe['INCOME'].value_counts().plot(title='INCOME', kind='barh')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bad63c3",
   "metadata": {},
   "source": [
    "If we look at a continuous variable, such as `TRAIN_TT`,\n",
    "representing the travel time by train, we are interested in statistics\n",
    "such as the mean, the standard deviation, the minimum and maximum\n",
    "values, as well as some quantiles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d052d242",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "display(swissmetro.dataframe['TRAIN_TT'].describe())\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c8c0e50",
   "metadata": {},
   "source": [
    "It is interesting to note that 75\\% of the values are lesser or equal\n",
    "to 209, while the maximum is 1049. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cddcc62",
   "metadata": {},
   "source": [
    "A histogram can also be plotted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0a9726e",
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = swissmetro.dataframe['TRAIN_TT'].hist()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "910c6b05",
   "metadata": {},
   "source": [
    "A similar analysis of the variable `SM_CO` provides the\n",
    "following statistics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b236ec51",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(swissmetro.dataframe['SM_CO'].describe())\n",
    "_ = swissmetro.dataframe['SM_CO'].hist()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8e9ea54",
   "metadata": {},
   "source": [
    "It may be made more readable by using a log scale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5cc44a51",
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = swissmetro.dataframe['SM_CO'].hist(log=True)\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9033f2f1",
   "metadata": {},
   "source": [
    "It is also interesting to investigate the correlation between\n",
    "two variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "403268e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(swissmetro.dataframe['TRAIN_TT'].corr(swissmetro.dataframe['TRAIN_CO']))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0303d129",
   "metadata": {},
   "source": [
    "The correlation can also be illustrated using a scatter plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b5c3c24",
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = swissmetro.dataframe.plot(kind='scatter', x='TRAIN_TT', y='TRAIN_CO', color='r')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "902f7086",
   "metadata": {},
   "source": [
    "Now, you are asked to perform a similar analysis of the file\n",
    "[http://transp-or.epfl.ch/data/optima.dat](http://transp-or.epfl.ch/data/optima.dat). The description of the data is\n",
    "available [here](http://transp-or.epfl.ch/documents/technicalReports/CS_OptimaDescription.pdf).\n",
    "The file can also be obtained from the Biogeme distribution using `from biogeme.data.optima import read_data`"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 5
}