{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "04271e67",
   "metadata": {},
   "source": [
    "\n",
    "File: 02-outlier.py\n",
    "\n",
    "\n",
    "Michel Bierlaire\n",
    "\n",
    "Sat Aug 02 2025, 18:18:01\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b69255d",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import pandas as pd\n",
    "from IPython.core.display_functions import display\n",
    "from biogeme.biogeme import BIOGEME\n",
    "from biogeme.database import Database\n",
    "from biogeme.expressions import Beta, Variable\n",
    "from biogeme.models import logit, loglogit\n",
    "from biogeme.results_processing import (\n",
    "    EstimationResults,\n",
    "    get_pandas_estimated_parameters,\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63c5abf8",
   "metadata": {},
   "source": [
    "The objective of this laboratory is to illustrate the outlier analysis. We use the Optima case study, for\n",
    "transportation mode choice in Switzerland."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0d4310a",
   "metadata": {},
   "source": [
    "Read the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea662f46",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('optima.dat', sep='\\t')\n",
    "display(df)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "306a53e6",
   "metadata": {},
   "source": [
    "Prepare the data for Biogeme"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2826ee7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "database = Database('optima', df)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a9130f6",
   "metadata": {},
   "source": [
    "Identification of the relevant variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70b6bd7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "Choice = Variable('Choice')\n",
    "Weight = Variable('Weight')\n",
    "age = Variable('age')\n",
    "Gender = Variable('Gender')\n",
    "TimePT = Variable('TimePT')\n",
    "TimeCar = Variable('TimeCar')\n",
    "TripPurpose = Variable('TripPurpose')\n",
    "MarginalCostPT = Variable('MarginalCostPT')\n",
    "CostCarCHF = Variable('CostCarCHF')\n",
    "distance_km = Variable('distance_km')\n",
    "Education = Variable('Education')\n",
    "LangCode = Variable('LangCode')\n",
    "OccupStat = Variable('OccupStat')\n",
    "NbTransf = Variable('NbTransf')\n",
    "Income = Variable('Income')\n",
    "WaitingTimePT = Variable('WaitingTimePT')\n",
    "CarAvail = Variable('CarAvail')\n",
    "Subscription = Variable('Subscription')\n",
    "GenAbST = Variable('GenAbST')\n",
    "OwnHouse = Variable('OwnHouse')\n",
    "NbBicy = Variable('NbBicy')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2905412d",
   "metadata": {},
   "source": [
    "Removing some incorrectly coded data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eddb736d",
   "metadata": {},
   "outputs": [],
   "source": [
    "exclude = (\n",
    "    (age == -1)\n",
    "    + (Choice == -1)\n",
    "    + (CostCarCHF < 0)\n",
    "    + (Income == -1)\n",
    "    + (CarAvail == 3) * (Choice == 1)\n",
    ") > 0\n",
    "database.remove(exclude)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b81a486",
   "metadata": {},
   "source": [
    "We first estimate a simple model:\n",
    "\\begin{align*}\n",
    "V_\\text{PT} &= \\text{Cte}_\\text{PT} + \\beta_{t, \\text{PT}} \\text{Time}_\\text{PT}\n",
    "+ \\beta_{c, \\text{PT}} \\text{Cost}_\\text{PT}, \\\\\n",
    "V_\\text{Car} &= \\text{Cte}_\\text{Car} + \\beta_{t, \\text{Car}} \\text{Time}_\\text{Car}\n",
    "+ \\beta_{c, \\text{Car}} \\text{Cost}_\\text{Car}, \\\\\n",
    "V_\\text{SM} &= \\beta_d \\text{distance}.\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "737bf409",
   "metadata": {},
   "source": [
    "Parameters to be estimated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59058374",
   "metadata": {},
   "outputs": [],
   "source": [
    "asc_pt = Beta('asc_pt', 0, None, None, 0)\n",
    "asc_car = Beta('asc_car', 0, None, None, 0)\n",
    "beta_time_pt = Beta('beta_time_pt', 0.0, None, None, 0)\n",
    "beta_time_car = Beta('beta_time_car', 0.0, None, None, 0)\n",
    "beta_cost = Beta('beta_cost', 0, None, None, 0)\n",
    "beta_distance = Beta('beta_distance', 0, None, None, 0)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af5ccdc2",
   "metadata": {},
   "source": [
    "Utility functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f2fc2f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "v_pt = asc_pt + beta_time_pt * TimePT + beta_cost * MarginalCostPT\n",
    "v_car = asc_car + beta_time_car * TimeCar + beta_cost * CostCarCHF\n",
    "v_sm = beta_distance * distance_km\n",
    "\n",
    "v = {0: v_pt, 1: v_car, 2: v_sm}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19bc3a87",
   "metadata": {},
   "source": [
    "Availability conditions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "460a1278",
   "metadata": {},
   "outputs": [],
   "source": [
    "av = {0: 1, 1: CarAvail != 3, 2: 1}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac7e325b",
   "metadata": {},
   "source": [
    "Estimation of the parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "403d7b42",
   "metadata": {},
   "outputs": [],
   "source": [
    "logprob = loglogit(v, av, Choice)\n",
    "biogeme = BIOGEME(database, logprob)\n",
    "biogeme.model_name = 'logit_optima_base'\n",
    "results: EstimationResults = biogeme.estimate()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0283284",
   "metadata": {},
   "source": [
    "General statistics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dffca1de",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(results.short_summary())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebdc3251",
   "metadata": {},
   "source": [
    "Estimated parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95034105",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(get_pandas_estimated_parameters(estimation_results=results))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3f03ab3",
   "metadata": {},
   "source": [
    "We now simulate the estimated model to obtain the contribution of each observation to the likelihood function. It is\n",
    "the probability predicted by the model to choose the actually chosen alternative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "775b4087",
   "metadata": {},
   "outputs": [],
   "source": [
    "prob_chosen = logit(v, av, Choice)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c867e20",
   "metadata": {},
   "source": [
    "We define a dictionary with the formulas to be simulated. Here, there is only one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e31847f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "simulate = {'Prob. chosen': prob_chosen}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c2de9ff",
   "metadata": {},
   "source": [
    "We perform the simulation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "addaedef",
   "metadata": {},
   "outputs": [],
   "source": [
    "biosim = BIOGEME(database, simulate)\n",
    "betas_values = results.get_beta_values()\n",
    "sim_results = biosim.simulate(the_beta_values=betas_values)\n",
    "display(sim_results)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cf205be",
   "metadata": {},
   "source": [
    "Consider as outliers all observations such that the choice probability is less than 10%.\n",
    "\n",
    "- Investigate the data in order to understand why the model is performing so poorly on those observations.\n",
    "- Use your conclusions to improve the model specification."
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 5
}