{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "442fe895",
   "metadata": {},
   "source": [
    "File 01-descriptive_solution.py\n",
    "\n",
    "\n",
    "Michel Bierlaire\n",
    "\n",
    "Sat Aug 02 2025, 16:38:03\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79444cc5",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import pandas as pd\n",
    "from IPython.core.display_functions import display\n",
    "from matplotlib import pyplot as plt\n",
    "from biogeme.data.optima import read_data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "578c5b9d",
   "metadata": {},
   "source": [
    "You are asked to perform a descriptive analysis of the file\n",
    "[http://transp-or.epfl.ch/data/optima.dat](http://transp-or.epfl.ch/data/optima.dat). The description of the data is\n",
    "available [here](http://transp-or.epfl.ch/documents/technicalReports/CS_OptimaDescription.pdf)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "260f17b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "optima = read_data()\n",
    "display(optima.dataframe)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e601ef04",
   "metadata": {},
   "source": [
    "We first note that there are 2265 observations, but only 1763 IDs. The reason is that some respondents have reported\n",
    "several tours for the day of reference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9bac3a87",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(len(optima.dataframe['ID'].unique()))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cf659eb",
   "metadata": {},
   "source": [
    "As there are many columns, we use a loop to enumerate them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce0cb70f",
   "metadata": {},
   "outputs": [],
   "source": [
    "for c in optima.dataframe.columns:\n",
    "    print(c)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f41b15fc",
   "metadata": {},
   "source": [
    "We first investigate the discrete variable `Education`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc4d5678",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(optima.dataframe['Education'].value_counts())\n",
    "\n",
    "_ = optima.dataframe['Education'].value_counts().plot(kind='bar')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48361987",
   "metadata": {},
   "source": [
    "We also investigate a continuous variable `TimePT`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e01fee88",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(optima.dataframe['TimePT'].describe())\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "525c6ce2",
   "metadata": {},
   "source": [
    "And the corresponding histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f55699a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = optima.dataframe['TimePT'].hist(log=True)\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da875a52",
   "metadata": {},
   "source": [
    "We note that income is available in two columns: `Income` and `CalculatedIncome`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84d0d73c",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(optima.dataframe['Income'].value_counts())\n",
    "display(optima.dataframe['CalculatedIncome'].value_counts())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94a24409",
   "metadata": {},
   "source": [
    "And we can plot the two variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60107280",
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = optima.dataframe.plot(kind='scatter', x='Income', y='CalculatedIncome', color='r')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "023bf4a2",
   "metadata": {},
   "source": [
    "When income was reported, the calculated income is simply the center of the corresponding interval.\n",
    "For instance, `Income = 2` corresponds to a monthly income from 2501 to 4000 CHF. In that case,\n",
    "`CalculatedIncome = 3250`. When income was not reported (`Income = -1`), the calculated income has been inputted.\n",
    "The method used for imputation is not reported in the documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26b6dbe6",
   "metadata": {},
   "source": [
    "It can be noted that the travel time by car and the distance in kilometers are strongly correlated. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8998800b",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(optima.dataframe['TimeCar'].corr(optima.dataframe['distance_km']))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13225c0b",
   "metadata": {},
   "source": [
    "And we can plot the two variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10a8f7e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = optima.dataframe.plot(kind='scatter', x='distance_km', y='TimeCar', color='r')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e99f423",
   "metadata": {},
   "source": [
    "The travel time by public transportation and distance are also correlated, although less so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11b953d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(optima.dataframe['TimePT'].corr(optima.dataframe['distance_km']))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31b22fb0",
   "metadata": {},
   "source": [
    "And we can plot the two variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9edbc3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = optima.dataframe.plot(kind='scatter', x='distance_km', y='TimePT', color='r')\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 5
}