{ "cells": [ { "cell_type": "markdown", "id": "36ceaeee-37db-4050-b53d-9026b43f6562", "metadata": {}, "source": [ "# Exercise session week 2 | Preprocessing, Imputation, Statistical feature extraction\n", "\n", "\n", "The goals of this exercise are to learn following techniques:\n", "\n", "1. How to use Pandas to load and manipulate tabular and time series data\n", "2. Data preprocessing and cleaning\n", "3. Missing value imputation\n", "4. Feature extraction: statistical features, one-hot encoding\n", "5. Outlier detection\n", "6. Filtering, smoothing\n", "7. Feature normalization\n", "\n", "---" ] }, { "cell_type": "markdown", "id": "7887b546-301d-46a1-8f38-015b9ad8c0e8", "metadata": { "tags": [] }, "source": [ "## 1. Pandas and DataFrames" ] }, { "cell_type": "markdown", "id": "93bfcc97-5c6d-4040-8de3-2cb2c5252943", "metadata": {}, "source": [ "**Previously**: you learned how to use numpy." ] }, { "cell_type": "markdown", "id": "2aab9176-7ddf-4068-b545-2d6ba9d74f03", "metadata": { "jp-MarkdownHeadingCollapsed": true, "tags": [] }, "source": [ "\n", "\n", "http://pandas.pydata.org/\n", "\n", "Pandas is a widely used library to handle **structured, tabular data** using an abstraction called **DataFrame** (type : `pd.DataFrame`) : data is structured into *named columns* (as in a relational table). Allows to load and process structured data files such as CSV, Excel, Parquet, HDF5... One-dimensional data, i.e. a *column* or a *row* has type `pd.Series`.\n", "\n", "Pandas is also based on numpy! `df.values` converts a DataFrame or a Series into a 2D or 1D numpy array.\n", "\n", "**READ THE DOCUMENTATION**\n", "\n", "* http://pandas.pydata.org/pandas-docs/stable/reference/frame.html (pd.DataFrame)\n", "* http://pandas.pydata.org/pandas-docs/stable/reference/series.html (pd.Series)\n", "\n", "" ] }, { "cell_type": "code", "execution_count": null, "id": "2619c017-c2eb-48af-ae68-fe8b500a9f21", "metadata": {}, "outputs": [], "source": [ "# You should install Pandas (if not done already): `pip install pandas`\n", "import pandas as pd\n" ] }, { "cell_type": "code", "execution_count": null, "id": "60a878a1-7805-4761-bd96-5f8dae37e1f1", "metadata": {}, "outputs": [], "source": [ "# The DataFrame class is the core data representation used by Pandas\n", "# One would typically load a dataframe with a CSV, but you can also create one manually\n", "df_example = pd.DataFrame({\"A\": [\"E420912\", \"E420913\", \"E420914\"], \"B\": [1, 2, 3]})\n", "\n", "# Take a look at the representation of a DataFrame\n", "print(df_example)\n" ] }, { "cell_type": "markdown", "id": "a7c273dc-3996-4283-a7dc-b81b4341f338", "metadata": { "tags": [] }, "source": [ "## 2. Data loading and cleaning" ] }, { "cell_type": "markdown", "id": "1f19391b-834b-4813-881b-3b00a783ed31", "metadata": {}, "source": [ "Data in the real world is “dirty”:\n", "* Incomplete: lacking attribute values, lacking certain attributes of interest, or containing only aggregate data\n", "* Noisy: containing errors or outliers\n", "* Inconsistent: containing discrepancies in codes or names" ] }, { "cell_type": "code", "execution_count": null, "id": "611ab073-8254-4fc2-af96-a43ed89dd59a", "metadata": {}, "outputs": [], "source": [ "# Let's load a CSV file in a DataFrame to process it\n", "df = pd.read_csv(\"flight_data.csv\")\n", "\n", "# Let's take a look at the first few rows to understand the contents\n", "print(df.head())\n" ] }, { "cell_type": "code", "execution_count": null, "id": "baa591cd-123f-4a9a-9059-72e037d05658", "metadata": {}, "outputs": [], "source": [ "\"\"\"\n", "EXERCISE - Dimensions of a DataFrame\n", "1. Display the number of rows and columns of the DataFrame\n", "\n", "Hints:\n", "- Take a look at the documentation https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html\n", "- Consider the built-in Python functions learned last week\n", "\"\"\"\n", "print('Number of columns :', ...)\n", "print('Number of rows :', ...)\n" ] }, { "cell_type": "markdown", "id": "44fe6437-a72b-4432-95dd-7e084d5c893a", "metadata": {}, "source": [ "It is important to understand the structure of Pandas' `DataFrame`. A `DataFrame` (short DF) is made of :\n", "\n", "* Named columns (_columns_):\n", "\n", "Column names can be accessed via `df.columns`. They can be renamed. A column can be retrieved using the syntax `df['column_name']` (similar to dictionary indexing) or `df.column_name` (/!\\ the latter only works if column name has no space in it). Multiple columns can be selected using a list of column names as index : `df[['col1', 'col2']]'` (mind the double brackets).\n", "\n", "* Rows (_rows_):\n", "\n", "Rows are associated to an _index_. The default index is 0, 1, 2, etc. but can have different $keys$ (example : a time-based index). The index **is not** a standard column of the DF. Rows can be accessed in two ways. Either **index-based** : `df.loc[idx]`, or **position-based** (i-th row) : `df.iloc[i]`. A set of rows can be retrieved using a list or a _slice_ of indices, e.g. `df.iloc[2:10]` to access rows between 2 and **9**.\n", "\n", "Selection of rows and columns can be combined in loc/iloc. For instance, to select rows 10 to 14 of columns EGT_SEL and FMV_SEL:\n", "```\n", "df.loc[10:15, ['EGT_SEL', 'FMV_SEL']]\n", "```\n", "Or, using position-based indexing for rows and columns:\n", "```\n", "df.iloc[10:15, [1, 3]]\n", "```\n", "\n", "`DataFrame` columns are **strongly typed**, as in a relational database. Column types are displayed with `df.dtypes`. The main types are numerical (int32, int64, float etc.) and \"object\" (e.g. string)." ] }, { "cell_type": "code", "execution_count": null, "id": "d50c3c03-2cb6-4a81-858b-e77094717e5b", "metadata": {}, "outputs": [], "source": [ "# Let's take a look at our columns\n", "df.columns\n" ] }, { "cell_type": "code", "execution_count": null, "id": "38230498-5ae5-4fd3-ba39-ee31fe60ece0", "metadata": {}, "outputs": [], "source": [ "# Let's see how many rows we have by checking the DataFrame index\n", "df.index\n" ] }, { "cell_type": "code", "execution_count": null, "id": "d6e586e9-b925-4026-b3ef-5215ef132c95", "metadata": {}, "outputs": [], "source": [ "# Here is how to extract the values of columns \"EGT_SEL\" and \"FMV_SEL\" for rows 100 to 105\n", "df.loc[100:105, ['EGT_SEL', 'FMV_SEL']]\n" ] }, { "cell_type": "code", "execution_count": null, "id": "5d4547c8-2706-483d-8393-1663d7137627", "metadata": {}, "outputs": [], "source": [ "\"\"\"\n", "EXERCISE - Extraction and removal of the first row.\n", "Notice that the first row contains the physical units of variables.\n", "It must be removed before further processing, but we should keep this information.\n", "\n", "1. Retrieve the units and store them in a variable.\n", "2. Remove this row from the DataFrame using the method \"drop\"\n", "\"\"\"\n", "units = ...\n" ] }, { "cell_type": "code", "execution_count": null, "id": "508f8157-9da0-4e1f-b270-bdac86e023c2", "metadata": {}, "outputs": [], "source": [ "print(units)\n", "\n", "# The first row of the DataFrame should now be correct (no more the units of the columns)\n", "print(df.head())\n" ] }, { "cell_type": "markdown", "id": "3295e2cd-714c-4ac6-851b-4236d030a30e", "metadata": {}, "source": [ "Notice how all columns have been recognized as `object`, meaning strings (text), although it should be numerical values. This was caused by the first row. Columns must be converted to numerical types, with two exceptions:\n", "\n", "* Column 't' should rather be converted to `datetime`.\n", "* Column 'FLIGHT_PHASE' contains text." ] }, { "cell_type": "code", "execution_count": null, "id": "676f560f-7057-4eb5-91dd-5785ae23ecd8", "metadata": {}, "outputs": [], "source": [ "# Let's take a look at the columns' types\n", "print(df.dtypes)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "8c5adb7c-6f2b-451f-a46f-136a84e02c80", "metadata": {}, "outputs": [], "source": [ "# Let's convert the column \"t\" to the DateTime type (instead of string)\n", "df['t'] = pd.to_datetime(df['t'])\n", "\n", "# Let's convert all the other columns to digits (instead of strings)\n", "df[df.columns[1:-1]] = df[df.columns[1:-1]].apply(pd.to_numeric)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "42235233-3daf-4bcd-8b22-92cbc6c297c7", "metadata": {}, "outputs": [], "source": [ "# Now, the column types should be correct\n", "print(df.dtypes)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "14bd0d07-c6f7-4596-b838-520d2b7cb10f", "metadata": {}, "outputs": [], "source": [ "# You should install matplotlib (if not done already): `pip install matplotlib`\n", "import matplotlib.pyplot as plt\n" ] }, { "cell_type": "code", "execution_count": null, "id": "deb7638a", "metadata": {}, "outputs": [], "source": [ "# Let's create a graph to visualize the evolution of the EGT_SEL temperature during the flight \n", "plt.plot(df[\"t\"], df[\"EGT_SEL\"])\n", "plt.xlabel(\"t\")\n", "plt.ylabel(\"EGT_SEL (°C\");\n" ] }, { "cell_type": "markdown", "id": "b63c18a6-64e9-4df6-8f14-d37875b8c892", "metadata": { "tags": [] }, "source": [ "## 3. Missing value imputation" ] }, { "cell_type": "markdown", "id": "aa220874-37fa-48e1-933e-ca6920f1d8e5", "metadata": { "tags": [] }, "source": [ "**NaN = Not a Number**\n", "\n", "NaN values need to be either removed or imputated (i.e. filled with certain values) before further processing. The choice of the imputation method is crucial.\n", "\n", "From today's lecture:\n", "\n", "* Complete case analysis: Delete any record that has missing values from the\n", "data set.\n", "* Nearest neighbors: to impute variable x average the value x of the k closest\n", "data points with no missing values.\n", "* Average method: Average the value of x for the non-missing values.\n", "* Hot deck: pick a “similar” record at random and use its value of x.\n", "* Predictive: Fit a model to the data with variable x as the target and use it to\n", "predict the value (e.g. kernel regression)\n", "* Single imputation: Draw a value at random from the conditional distribution of\n", "x given the other variables\n", "27.02.23\n", "* Multiple imputation: Repeatedly draw values at random from the conditional\n", "distribution of x given the other variables (e.g. as above), creating new data\n", "sets. Make the predictions with these now complete datasets and average the\n", "predictions.\n", "\n", "In this exercise, you will learn to :\n", "\n", "* find missing values in a DataFrame (method `isna`)\n", "* remove missing values (method `dropna`)\n", "* replace missing values with a constant value, forward and backward fill (method `fillna`)" ] }, { "cell_type": "code", "execution_count": null, "id": "744a7d5b-7b74-45d3-b3f1-102fd854d70c", "metadata": {}, "outputs": [], "source": [ "# First, let's check if we have missing values\n", "df.isna()\n" ] }, { "cell_type": "code", "execution_count": null, "id": "0b19df93", "metadata": {}, "outputs": [], "source": [ "# You can check the built-in documentation of the isna() method\n", "df.isna?\n" ] }, { "cell_type": "code", "execution_count": null, "id": "ac992784-d53c-4162-9e30-cf533a8bad19", "metadata": {}, "outputs": [], "source": [ "\"\"\"\n", "EXERCISE - Method isna\n", "1. Try the isna method on DataFrame df, and on a single column or row. What is the result?\n", "2. Using any(axis=...), mean() and max()/idxmax() methods on the result of isna(), answer following questions :\n", " 2.1 Which columns contain mising values, which don't ?\n", " 2.2 What is the percentage of missing valuesin the DF (a) per column (b) overall ? Which variable containt the most NaN values ?\n", " 2.3 What is the percentage of rows where all variables are present ?\n", "\"\"\"\n" ] }, { "cell_type": "code", "execution_count": null, "id": "03b5883e-7b6e-4f89-8af7-aa8679aad7fc", "metadata": { "tags": [] }, "outputs": [], "source": [ "# You can check the built-in documentation of the dropna() method\n", "df.dropna?\n" ] }, { "cell_type": "code", "execution_count": null, "id": "4c026180-7ba3-445f-8aee-5ec58b0375cd", "metadata": {}, "outputs": [], "source": [ "\"\"\"\n", "EXERCISE - Method dropna\n", "The dropna method can be used to remove missing values (NaN). Start by reading its documentation.\n", "1. What is the meaning of arguments \"axis\" and \"how\" ?\n", "2. Remove the rows containing ONLY missing values.\n", "3. Remove the rows containing AT LEAST one missing value. How many additional rows were removed?\n", "4. Remove the columns containing AT LEAST one missing value.\n", "\n", "Note: do not change the actual DataFrame df (you can create new variables df2, df3, ...)\n", "\"\"\"\n" ] }, { "cell_type": "code", "execution_count": null, "id": "07e44cd6", "metadata": {}, "outputs": [], "source": [ "# Now, update the actual DF by removing the rows containing AT LEAST one missing value\n", "df = ...\n" ] }, { "cell_type": "code", "execution_count": null, "id": "bf434e2d-0c78-4609-a3a6-2ae93c05a49b", "metadata": {}, "outputs": [], "source": [ "\"\"\"\n", "EXERCISE - Method fillna\n", "The fillna() method can be used to fill missing values (NaN). Start by reading its documentation.\n", "1. What are the different strategies to fill missing values?\n", "2. Fill the missing value of \"age\" in the provided toy DataFrame using :\n", " - 0\n", " - the last or next non-empty value\n", " - the mean\n", " - the most frequent value (mode)\n", "\n", "Complete the exercise on the \"example\" DataFrame declared below\n", "\"\"\" \n", "example = pd.DataFrame({'nom': ['Alice', 'Bob', 'Charlie', 'David'], 'age': [24, None, 99, 24]})\n", "print(example)\n" ] }, { "cell_type": "markdown", "id": "e3cff224-1b61-4546-8bcd-1861e77e0fd2", "metadata": {}, "source": [ "**Question**: How would you handle a temporal variable, such as temperature 'EGT_SEL' in the flight data?" ] }, { "cell_type": "markdown", "id": "c37db8ee-6cfe-42a4-9441-fb466cc81042", "metadata": {}, "source": [ "## 4. Feature extraction: statistical features, one-hot encoding" ] }, { "cell_type": "markdown", "id": "c25e3d02-2be6-4281-9e20-6be1b892b335", "metadata": {}, "source": [ "Now that we have some clean data to work with, let's analyze it and extract meaningful information:\n", "\n", "* Transform raw signals into more informative signatures (or fingerprints)\n", "of a system\n", "* Reduce size / complexity of the dataset\n", "* Provide a physical description / representation\n", "* Reduce resources necessary for further processing\n", "* Achieve intended objectives\n", "* Features are known to be the most crucial point in machine learning" ] }, { "cell_type": "code", "execution_count": null, "id": "124a9a47-f873-4b94-87c9-a19d370cdd51", "metadata": {}, "outputs": [], "source": [ "\"\"\"\n", "EXERCISE - Compute statistical features for each phase of this flight\n", "For each FLIGHT_PHASE, compute and display the mean, standard deviation, max and min of the numerical features.\n", "Hint: a DataFrame or Series can be aggregated using methods such as \".mean()\".\n", "Hint: A DF can be grouped on a key using df.groupby(column), before being aggregated.\n", "\"\"\"\n" ] }, { "cell_type": "code", "execution_count": null, "id": "fec46c49", "metadata": {}, "outputs": [], "source": [ "\n" ] }, { "cell_type": "markdown", "id": "465e7863-b926-4fd3-bc32-0493c3f9fb4d", "metadata": {}, "source": [ "The `FLIGHT_PHASE` feature is categorical (text, with finite number of different values). One way to transform it into a numerical feature is through one-hot encoding.\n", "\n", "![one-hot colors](rc/onehot-color.png)" ] }, { "cell_type": "code", "execution_count": null, "id": "57af14fc-d01a-475a-a612-2776ca52d0bb", "metadata": {}, "outputs": [], "source": [ "\"\"\"\n", "EXERCISE - Perform a one-hot encoding of the categorical feature \"FLIGHT_PHASE\".\n", "The simplest way is to use the OneHotEncoder provided by the scikit-learn library.\n", "1. Fit the encoder to the FLIGHT_PHASE column.\n", "2. Print out the categories of the encoding.\n", "3. Append the new encoded features to the DF (FLIGHT_PHASE_cruise, FLIGHT_PHASE_climb, etc.).\n", "\n", "Hint: Check the OneHotEncoder class from Scikit-Learn https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html\n", "\"\"\"\n", "\n", "# You should install scikit-learn (if not done already): `pip install scikit-learn`\n", "from sklearn.preprocessing import OneHotEncoder\n" ] }, { "cell_type": "code", "execution_count": null, "id": "59aa1f8b", "metadata": {}, "outputs": [], "source": [ "# You should now have additional columns at the end of the DataFrame with the proper encoding \n", "print(df.head())\n" ] }, { "cell_type": "markdown", "id": "34bcd830-8449-4999-828a-3be4952ff12e", "metadata": {}, "source": [ "## 5. Outlier detection\n", "\n", "One way to analyze data is to look at its distribution. This is often done by computing the mean/average and the standard deviation. However, quantiles are also important metrics and help to identify outliers, which can potentially indicate abnomral operating conditions.\n", "\n", "Let's see how we can compute quantiles on a Pandas `DataFrame`." ] }, { "cell_type": "code", "execution_count": null, "id": "0bfdadb2-fcd8-431f-8d3f-f2a02a4ab82b", "metadata": {}, "outputs": [], "source": [ "\"\"\"\n", "EXERCISE - Quantiles\n", "1. Compute the 99th-percentile values for each variable in the DF.\n", "2. Compute this value for a specific variable, for example \"VIB_CN1\", and highlight the outlier values in the plot.\n", "\n", "Hint: Take a look at the documentation https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html\n", "\"\"\"\n", "\n", "# 1. Print the 99th percentile for each column\n", "print(...)\n", "\n", "# 2. Compute the 95th percentile for the VIB_CN1 column (q95 should be a single float value)\n", "var = \"VIB_CN1\"\n", "q95 = ...\n", "\n", "# Automatic plot (you shouldn't have to modify this code)\n", "plt.figure(figsize=(10,5))\n", "plt.plot(df[\"t\"], df[var])\n", "plt.plot([df[\"t\"].iloc[0], df[\"t\"].iloc[-1]], [q95, q95], \"r-\")\n", "outlier = df[df[var] > q95]\n", "plt.plot(outlier[\"t\"], outlier[var], \"ro\")\n", "plt.xlabel(\"t\")\n", "plt.ylabel(var);\n" ] }, { "cell_type": "markdown", "id": "5c99655e-326f-4ad2-9c54-8df9161184ee", "metadata": {}, "source": [ "## 6. Filtering & Smoothing\n", "\n", "As sensors are inherently noisy, their measurements often need some filtering and smoothing. In this section, we will explore a moving-average smoothing method.\n", "\n", "### 6.1 Time-based index\n", "\n", "For now, each row is identified by its index. However, the `DataFrame` contains time-series data (*i.e.* data which evolves over time). To compute a moving-average smoothing over time, we should first index each row with a time value, instead of a simple integer index." ] }, { "cell_type": "code", "execution_count": null, "id": "d055d85b-80be-411f-9054-199ef81eb6c8", "metadata": {}, "outputs": [], "source": [ "\"\"\"\n", "Exercise - Time-based index\n", "As our data are multivariate time series, we would like to use a time-based index.\n", "1. Create a copy of df, called df2, using the appropriate method.\n", "2. Set the time column ('t') the be the index of the DataFrame.\n", "3. Remove column 't' of the resulting DF.\n", "\"\"\"\n", "print(df.index)\n", "\n", "df2 = ...\n", "\n", "print(df2.index)\n", "\n", "df2.head()\n" ] }, { "cell_type": "markdown", "id": "b37d52ef-9b93-448f-9818-d105251e9a38", "metadata": {}, "source": [ "We notice that pandas has automatically recognized a `DatetimeIndex`, suited for time series processing (such as time-based window operations, etc) !\n" ] }, { "cell_type": "markdown", "id": "2e009388-0087-431e-bd83-fbc39488dd70", "metadata": {}, "source": [ "### 6.2 Moving average smoothing\n", "\n", "Now that each row is properly indexed and represents a moment in time, let's compute a 30 seconds moving-average to smooth a sensor's data. " ] }, { "cell_type": "code", "execution_count": null, "id": "211f746e-4635-4d8c-8498-30f8f8bd404e", "metadata": {}, "outputs": [], "source": [ "\"\"\"\n", "EXERCISE - Moving average smoothing.\n", "Pandas provides rolling window functions that can be applied as follows:\n", " data.rolling(window=).()\n", "Window size can be provided as a integer (number of steps), or a time unit if index is time-based.\n", "1. Perform a moving average of 30 seconds.\n", "\n", "Hint: Again, take a look at the documentation https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html\n", "\"\"\"\n", "# This is the column to smooth\n", "var = \"VIB_CN1\"\n", "\n", "# The variable below should be an array of floats\n", "var_smoothed = ...\n" ] }, { "cell_type": "code", "execution_count": null, "id": "7831d50d-da44-49f2-b8cd-37445846fcc1", "metadata": {}, "outputs": [], "source": [ "# Automatic plot (you shouldn't have to modify it)\n", "plt.figure(figsize=(10,5))\n", "plt.plot(df2.index, df2[var])\n", "plt.plot(df2.index, var_smoothed, \"r-\", label=\"moving average\")\n", "plt.xlabel(\"t\")\n", "plt.ylabel(var)\n", "plt.legend();\n" ] }, { "cell_type": "markdown", "id": "ed438236-34f5-4cee-b606-be3aaf141d9f", "metadata": {}, "source": [ "**Question** : why would you sometimes want to use a moving median instead of a moving average ? Try it yourself!" ] }, { "cell_type": "code", "execution_count": null, "id": "d0bf2483", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "86f44119-354b-4ab3-bf7c-d4d9d5c27317", "metadata": { "tags": [] }, "source": [ "### 6.3 Subsampling\n", "\n", "A DF with time-based index can be subsampled at a given frequency using method `asfreq` (http://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.PeriodIndex.asfreq.html)." ] }, { "cell_type": "code", "execution_count": null, "id": "80d0316f-307d-4311-a066-666bbec5764a", "metadata": {}, "outputs": [], "source": [ "df3 = df2.asfreq('1s')\n" ] }, { "cell_type": "code", "execution_count": null, "id": "054274da-fe38-42ad-87c6-7cd755af4326", "metadata": {}, "outputs": [], "source": [ "# This is before subsampling\n", "print(df2.index)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "6c02b486-1530-4f9a-9b6f-17b1f9ac11a7", "metadata": {}, "outputs": [], "source": [ "# This is after subsampling\n", "print(df3.index)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "97ca0415-c903-4e3a-9849-36838262a74a", "metadata": {}, "outputs": [], "source": [ "# What are now the dimensions of the DataFrame ?\n", "print(df3.shape)\n" ] }, { "cell_type": "markdown", "id": "bbc0d07b-5bd9-4774-8d60-db2d7415f7b8", "metadata": { "tags": [] }, "source": [ "## 7. Feature normalization and standardization\n", "\n", "### Motivation\n", "\n", "Different numerical variables naturally fall in different scales (e.g. N1 <0 to 100> and temperature <0 to 1000>). Many ML models are based on euclidean distance or weighted linear combination of variables and we don't want one variable to dominate the others just because they are on a different scale.\n", "\n", "Feature scaling removes the implicit dominance of variables with wider numerical ranges by transforming all numerical variables to the same range (typically from -1 to +1 or 0 to 1).\n", "\n", "Even for ML algorithms that are not based on euclidean distance or weighted linear combinations (like decision trees), scaling greatly helps convergence speed.\n", "\n", "### Scaling Strategies\n", "\n", "There are multiple strategies for feature scaling.\n", "\n", "* **Normalisation** Scaling: use when the variable is normally distributed. For example, Min-Max scaling:\n", "\n", "$$X' = \\frac{X-X_{min}}{X_{max}-X_{min}}$$\n", "\n", "* **Standardization** Scaling: use when data is not normally distributed. When in doubt use standardization scaling.\n", "\n", "$$X' = \\frac{X-\\mu}{\\sigma}$$\n", "\n", "### Fit the scaler on the training data only\n", "\n", "The scaler must be fit with the the training set only (i.e. the scaling parameters must be derived only from the training set). The test set and any prediction set must be transformed using the scaler fitted on the training set.\n", "\n", "This is the only way to guarantee that the test set and any predictions are subject to the same conditions and have been totally unobserved during the training process (**data leakage**).\n" ] }, { "cell_type": "code", "execution_count": null, "id": "fbd2c0a9-7f81-4288-bb56-d82fa7f486d4", "metadata": {}, "outputs": [], "source": [ "\"\"\"\n", "EXERCISE - Data normalization and standardization\n", "\n", "1. Using the DataFrame `df`, apply min-max normalization (create a new DataFrame to store the result) on the \"VIB_CN1\" column\n", "2. Using the DataFrame `df`, apply standardization scaling (create a new DataFrame to store the result)\n", "\n", "Hint: Take a look at the documentation https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html\n", "\"\"\"\n" ] }, { "cell_type": "code", "execution_count": null, "id": "034dd030", "metadata": {}, "outputs": [], "source": [ "\"\"\"\n", "EXERCISE - Plot normalized data\n", "\n", "1. Using the matplotlib library, create a plot of the normalized \"VIB_CN1\" with\n", " - A title\n", " - An X axis label (Time)\n", " - A Y axis label (Normalized VIB_CN1)\n", " - A plot title\n", " - \n", "\"\"\"\n" ] }, { "cell_type": "code", "execution_count": null, "id": "d11aec8d-adb2-455d-bc1f-6adbabbd50e2", "metadata": {}, "outputs": [], "source": [ "# Example: standardization in a training/testing set scenario (using scikit-learn)\n", "from sklearn.preprocessing import StandardScaler\n", "\n", "scaler = StandardScaler()\n", "\n", "# IMPORTANT: the scaler must be fitted with the training data only.\n", "X_train = scaler.fit_transform(X_train)\n", "X_test = scaler.transform(X_test)\n" ] } ], "metadata": { "kernelspec": { "display_name": "civil-332", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.7" } }, "nbformat": 4, "nbformat_minor": 5 }