{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "3HfhkfWi122_" }, "source": [ "# Exercise 7: Image Classification with Deep Learning" ] }, { "cell_type": "markdown", "metadata": { "id": "vCQWSSP3Gpin" }, "source": [ "Last week we have seen the basics of PyTorch and its relationship to NumPy and implemented a linear regression model.\n", "\n", "In this exercise, we will take a step further and implement a Convolutional Neural Network following the AlexNet architecture\n", "\n", "\n", "\n", "\n", "[image from Llamas et al., (2017) availble by Creative Commons Attribution 4.0 International](https://www.researchgate.net/publication/320052364_Classification_of_Architectural_Heritage_Images_Using_Deep_Learning_Techniques)\n", "\n", "We\n", "1. build a torch dataset to lead the [UC Merced Dataset](http://weegee.vision.ucmerced.edu/datasets/landuse.html) data\n", "2. implement common data augmentation strategies\n", "3. build a CNN model and load pre-trained weights\n", "4. evaluate a trained CNN model" ] }, { "cell_type": "markdown", "metadata": { "id": "mOAj8vIOGpio" }, "source": [ "## 1. Setup" ] }, { "cell_type": "markdown", "metadata": { "id": "A8YmgeasGpio" }, "source": [ "### 1.1 Install dependencies" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "-2l-S3DP1zyV" }, "outputs": [], "source": [ "!pip install -U -q torch=2.5.0 torchvision matplotlib tqdm gdown" ] }, { "cell_type": "markdown", "metadata": { "id": "42GDDlFtGpip" }, "source": [ "### 1.2 Check if GPU available\n", "\n", "PyTorch has full support for GPUs and we will make use of it already in this exercise. So let's first test if you have GPU availability.\n", "\n", "Run the following code block. It should be `False` when running locally (unless you have a GPU), and `True` on Colab\n", "\n", "If you get `False` and you are in Colab, click on Runtime tab and select GPU hardware accelerator" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "33FmMPWFGpip" }, "outputs": [], "source": [ "import torch\n", "\n", "print(torch.cuda.is_available())" ] }, { "cell_type": "markdown", "metadata": { "id": "oyfKdc3M3T0N" }, "source": [ "## 2. Data Loading\n", "\n", "Here, we get the dataset and write some code to make it accessible to our prospective DL model." ] }, { "cell_type": "markdown", "metadata": { "id": "RSpuJl8n5MZt" }, "source": [ "### 2.1 Download data\n", "\n", "Let's download and unzip the [UC Merced Land Use dataset](http://weegee.vision.ucmerced.edu/datasets/UCMerced_LandUse.zip) for this exercise." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "52SHOZx-4s3R" }, "outputs": [], "source": [ "import urllib.request\n", "import os\n", "if not os.path.exists(\"UCMerced_LandUse.zip\"):\n", " print(f\"downloading UCMerced_LandUse.zip\")\n", " urllib.request.urlretrieve(\"http://weegee.vision.ucmerced.edu/datasets/UCMerced_LandUse.zip\", \"UCMerced_LandUse.zip\")\n", "\n", "import zipfile\n", "with zipfile.ZipFile(\"UCMerced_LandUse.zip\", 'r') as zip_ref:\n", " zip_ref.extractall()" ] }, { "cell_type": "markdown", "metadata": { "id": "X3zthGuD5liE" }, "source": [ "### 2.3 Write a PyTorch Dataset class\n", "\n", "In previous cases, we were able to load all of our data into system memory to train machine learning models.\n", "For DL, this does not work anymore, because we have to use potentially millions of images. Furthermore, since we are dealing with images, we have no clear definition of labels – for example, a model does not know what to do with the information that an image is called \"mediumresidential00.jpg\".\n", "\n", "Hence, we need to implement a routine that (i.) keeps track of our many images and only loads them if needed, and (ii.) can prepare the dataset for usage in our prospective DL model.\n", "In PyTorch, this is done through a [Dataset](https://pytorch.org/tutorials/recipes/recipes/custom_dataset_transforms_loader.html#create-a-dataset-class) class – if you remember last week's exercise you should know what object classes are.\n", "Let us implement a Dataset class for our UCMerced dataset.\n", "\n", "Please take a look on the link to understand how this works and complete the code shell below accordingly.\n", "\n", "Tips:\n", "* Check which functions a `torch.utils.data.Dataset` class requires and implement them accordingly.\n", "* See last week's exercise on how to load images using PIL (the Python Image Library).\n", "* In the constructor you see an optional variable named `transforms`. This is for data augmentation, which we will address later. At this point all you need to know is that you can use this on the PIL images and it will return a `torch.Tensor` directly. So it suffices to use `data = self.transforms(img)`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "xrvSTpB36Z6A" }, "outputs": [], "source": [ "from torch.utils.data import Dataset\n", "\n", "from PIL import Image\n", "\n", "import os\n", "import glob\n", "\n", "\n", "class UCMerced(Dataset):\n", "\n", " # mapping between label class names and indices\n", " LABEL_CLASSES = {\n", " 'agricultural': 0,\n", " 'airplane': 1,\n", " 'baseballdiamond': 2,\n", " 'beach': 3,\n", " 'buildings': 4,\n", " 'chaparral': 5,\n", " 'denseresidential': 6,\n", " 'forest': 7,\n", " 'freeway': 8,\n", " 'golfcourse': 9,\n", " 'harbor': 10,\n", " 'intersection': 11,\n", " 'mediumresidential': 12,\n", " 'mobilehomepark': 13,\n", " 'overpass': 14,\n", " 'parkinglot': 15,\n", " 'river': 16,\n", " 'runway': 17,\n", " 'sparseresidential': 18,\n", " 'storagetanks': 19,\n", " 'tenniscourt': 20\n", " }\n", "\n", " # image indices to use for different splits\n", " SPLITS = {\n", " 'train': list(range(0, 60)), # use first 60 images of each class for training...\n", " 'val': list(range(61, 70)), # ...images 61-70 for model validation...\n", " 'test': list(range(71, 100)) # ...and the rest for testing\n", " }\n", "\n", " def __init__(self, transforms=None, split='train'):\n", " self.transforms = transforms\n", "\n", " # prepare data\n", " self.data = [] # list of tuples of (image path, label class)\n", " for labelclass in self.LABEL_CLASSES:\n", " # get images with correct index according to dataset split\n", " for imgIndex in self.SPLITS[split]:\n", " imgName = os.path.join('UCMerced_LandUse/Images', labelclass, f'{labelclass}{str(imgIndex).zfill(2)}.tif')\n", " # example format: 'baseFolder/agricultural/agricultural07.tif'\n", " self.data.append((\n", " imgName,\n", " self.LABEL_CLASSES[labelclass] # get index for label class\n", " ))\n", "\n", "\n", " #TODO: please provide the remaining functions required for the torch.utils.data.Dataset class.\n", " def __len__(self):\n", " return #TODO\n", "\n", "\n", " def __getitem__(self, index):\n", " # TODO retrieve the item from self.data at position index\n", " imgName, label =\n", "\n", " # TODO load the image array from the imgName\n", " img =\n", "\n", " # TODO apply transformation\n", " if self.transforms is not None:\n", " img =\n", " return img, label\n" ] }, { "cell_type": "markdown", "metadata": { "id": "uUszW01uGpir" }, "source": [ "Nothing to do here, if your implementation works, you should be able to plot samples now" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "fiCdqqDrGpir" }, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "\n", "# initialize the dataset (call the constructor __init__)\n", "dataset = UCMerced()\n", "print(f\"dataset of length {len(dataset)}\")\n", "\n", "# plot individual samples\n", "from ipywidgets import interact\n", "@interact(idx=range(len(dataset)))\n", "def plot_sample(idx=0):\n", " img, label = dataset[idx]\n", "\n", " plt.imshow(img)\n", "\n", " # swaps keys and values in the dictionary UCMerced.LABEL_CLASSES\n", " class_mapping = {v: k for k, v in UCMerced.LABEL_CLASSES.items()}\n", "\n", " plt.title(f\"classid {label} ({class_mapping[label]})\")" ] }, { "cell_type": "markdown", "metadata": { "id": "8HMHHZWFEjLN" }, "source": [ "### 2.4 Define transforms\n", "\n", "We have seen some `transform` operations above, e.g. `ToTensor()`. These are not only used to perform data conversion, but also data **augmentation**. This is important for DL as it artificially increases the dataset complexity. For example, randomly flipping images horizontally every now and then is a logical operation to perform on remote sensing images; doing it during training exposes the DL model to more variations.\n", "\n", "In the following, we will define transforms for training and validation separately.\n", "\n", "TODO: add the following augmentations using the [torchvision.transforms documentations](https://pytorch.org/vision/stable/transforms.html)\n", "* RandomResizedCrop\n", "* RandomGrayscale\n", "* RandomHorizontalFlip\n", "* GaussianBlur\n", "* RandomPosterize\n", "* RandomVerticalFlip\n", "* ColorJitter" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "BGmZePDcFKaR" }, "outputs": [], "source": [ "import torchvision.transforms as T\n", "import numpy as np\n", "\n", "# mean and standard deviation of the dataset\n", "mean=torch.tensor([0.504, 0.504, 0.503])\n", "std=torch.tensor([0.019 , 0.018, 0.018])\n", "\n", "# normalize image [0-1] (or 0-255) to zero-mean unit standard deviation\n", "normalize = T.Normalize(mean, std)\n", "# we invert normalization for plotting later\n", "std_inv = 1 / (std + 1e-7)\n", "unnormalize = T.Normalize(-mean * std_inv, std_inv)\n", "\n", "transforms_train = T.Compose([\n", " #TODO: add your own transforms here\n", "\n", " T.Resize((224, 224)),\n", " T.ToTensor(),\n", " normalize\n", "])\n", "\n", "# we do not augment the validation dataset (aside from resizing and tensor casting)\n", "transforms_val = T.Compose([\n", " T.Resize((224, 224)),\n", " T.ToTensor(),\n", " normalize\n", "])" ] }, { "cell_type": "markdown", "metadata": { "id": "gpMh8_tvGpis" }, "source": [ "test your transforms by executing the following cell" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "6wK1TepfGpis" }, "outputs": [], "source": [ "dataset_index = 500\n", "\n", "img, label = dataset[dataset_index]\n", "\n", "fig, axs = plt.subplots(1,2, figsize=(12,6))\n", "axs[0].imshow(unnormalize(transforms_val(img)).permute(1,2,0))\n", "axs[0].set_title(\"validation transform (no augmentation)\")\n", "\n", "axs[1].imshow(unnormalize(transforms_train(img)).permute(1,2,0))\n", "axs[1].set_title(\"training transform\")\n", "[ax.axis(\"off\") for ax in axs] # removes ticks" ] }, { "cell_type": "markdown", "metadata": { "id": "vbJz61FmGpis" }, "source": [ "Lets add the transform function to the dataset rather than manually applying it every time" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "qccd-6-5Gpit" }, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "\n", "# initialize the dataset (call the constructor __init__)\n", "dataset = UCMerced()\n", "print(f\"dataset of length {len(dataset)}\")\n", "\n", "# plot individual samples\n", "from ipywidgets import interact\n", "@interact(idx=range(len(dataset)))\n", "def plot_sample(idx=0):\n", " img, label = dataset[idx]\n", "\n", " plt.imshow(img)\n", "\n", " # swaps keys and values in the dictionary UCMerced.LABEL_CLASSES\n", " class_mapping = {v: k for k, v in UCMerced.LABEL_CLASSES.items()}\n", "\n", " plt.title(f\"classid {label} ({class_mapping[label]})\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "BKHQab9QGpit", "scrolled": false }, "outputs": [], "source": [ "# TODO add your transforms function to the dataset as argument\n", "train_dataset = UCMerced(#TODO)\n", "val_dataset = UCMerced(#TODO)\n", "\n", "\n", "# Nothing todo, plotting functions\n", "fig, axs = plt.subplots(1,5, figsize=(5*3, 3))\n", "for ax in axs:\n", " idx = np.random.randint(len(train_dataset)) # random sample\n", " image, label = train_dataset[idx]\n", " ax.imshow(unnormalize(image).permute(1,2,0))\n", " ax.set_title(f\"idx {idx}, {list(UCMerced.LABEL_CLASSES.keys())[label]}\")\n", " ax.axis(\"off\")\n", "\n", "fig.suptitle(\"training samples\")\n", "plt.tight_layout()\n", "\n", "fig, axs = plt.subplots(1,5, figsize=(5*3, 3))\n", "for ax in axs:\n", " idx = np.random.randint(len(val_dataset)) # random sample\n", " image, label = val_dataset[idx]\n", " ax.imshow(unnormalize(image).permute(1,2,0))\n", " ax.set_title(f\"idx {idx}, {list(UCMerced.LABEL_CLASSES.keys())[label]}\")\n", " ax.axis(\"off\")\n", "\n", "fig.suptitle(\"validation samples\")\n", "plt.tight_layout()" ] }, { "cell_type": "markdown", "metadata": { "id": "fapSg1Jm8tjc" }, "source": [ "## 3. Create model\n", "\n", "Let's design an deep learning model for classifying the UC Merced images. In this case, this will be a based on AlexNet." ] }, { "cell_type": "markdown", "metadata": { "id": "8vPUbB-vGpit" }, "source": [ "### 3.1 Building blocks\n", "\n", "get one val image with label\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "PvEiAKAsGpit" }, "outputs": [], "source": [ "# TODO: load one val image with label and save into x and y\n", "x, y =\n", "\n", "# we add a 1-dimension to build a \"batch\" of size 1\n", "x = x.unsqueeze(0)\n", "\n", "fig, ax = plt.subplots(1,1)\n", "ax.imshow(unnormalize(x.squeeze().detach()).permute(1,2,0).numpy(), cmap=\"gray\")\n", "ax.set_title(\"input x\")" ] }, { "cell_type": "markdown", "metadata": { "id": "jeNBahYqGpiu" }, "source": [ "### 3.1.1 Convolution\n", "\n", "\n", "\n", "animation from [Wikipedia with Creative Commons Attribution-Share Alike 3.0 Unported License](https://commons.wikimedia.org/wiki/File:2D_Convolution_Animation.gif#filelinks)\n", "\n", "**TODO** add a Conv layer following the [torch.nn.Conv2d documentation](https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html?highlight=conv2d#torch.nn.Conv2d)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "6DX0Qu6zGpiu", "scrolled": false }, "outputs": [], "source": [ "from torch import nn\n", "\n", "# TODO initialize a 5x5 conv layer with nn.Conv2d that contains 3 input dimensions and 1 output dimension,\n", "# with stride 2 and padding 2\n", "convlayer =\n", "\n", "# TODO: make a forward pass throught the convolution function\n", "x_convolved =\n", "\n", "fig, ax = plt.subplots(1,1)\n", "ax.imshow(x_convolved.detach().squeeze().numpy(), cmap=\"gray\")\n", "ax.set_title(\"conv(x)\")\n" ] }, { "cell_type": "markdown", "metadata": { "id": "a_mhgp_4Gpiu" }, "source": [ "### 3.1.2 ReLU: Rectified linear unit activation function\n", "\n", "$y = \\max(x, 0)$\n", "\n", "**TODO** add a ReLU layer following the [torch.nn.ReLU documentation](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "3SehnX7FGpiu" }, "outputs": [], "source": [ "# TODO: add a ReLU layer\n", "relu =\n", "\n", "# make forward pass through the relu function\n", "x_relu = relu(x_convolved)\n", "\n", "fig, ax = plt.subplots(1,1)\n", "ax.imshow(x_relu.detach().squeeze().numpy(), cmap=\"gray\")\n", "ax.set_title(\"relu(conv(x))\")\n" ] }, { "cell_type": "markdown", "metadata": { "id": "1brvSpEUGpiu" }, "source": [ "### 3.1.3 ReLU MaxPooling\n", "\n", "\n", "\n", "image showing 2x2 pooling with stride 2 [image from wikipedia with CC BY-SA 4.0](https://en.wikipedia.org/wiki/Convolutional_neural_network#/media/File:Max_pooling.png) license\n", "\n", "**TODO** add a MaxPool2d layer following the [torch.nn.MaxPool2d documentation](https://pytorch.org/docs/stable/generated/torch.nn.MaxPool2d.html)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Tcg7iSTsGpiv" }, "outputs": [], "source": [ "# TODO add a 3x3 maxpooling layer with stride 2\n", "maxpool =\n", "\n", "# make forward pass with x_relu through the maxpool layer\n", "x_pool = maxpool(x_relu)\n", "\n", "fig, ax = plt.subplots(1,1)\n", "ax.imshow(x_pool.detach().squeeze().numpy(), cmap=\"gray\")\n", "ax.set_title(\"maxpool(relu(conv(x)))\")" ] }, { "cell_type": "markdown", "metadata": { "id": "_T7N_SgpGpiv" }, "source": [ "### 3.1.4 Dropout\n", "\n", "in training mode (default)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "mxRcao3pGpiv" }, "outputs": [], "source": [ "# TODO: add a Dropout layer with 50% dropout probability\n", "dropout =\n", "\n", "#TODO take the original input x and apply dropout to it\n", "x_dropout =\n", "\n", "fig, ax = plt.subplots()\n", "ax.imshow(unnormalize(x_dropout.squeeze().detach()).permute(1,2,0).numpy(), cmap=\"gray\")" ] }, { "cell_type": "markdown", "metadata": { "id": "lBCbeXrLGpiv" }, "source": [ "in evaluation mode\n", "\n", "**TODO** set dropout to eval mode by calling the `.eval` function on it" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "OPG_gNNUGpiv" }, "outputs": [], "source": [ "# set dropout to eval mode\n", "dropout.eval()\n", "\n", "# call dropout again (same code as above)\n", "x_dropout = dropout(x)\n", "\n", "fig, ax = plt.subplots()\n", "ax.imshow(unnormalize(x_dropout.squeeze().detach()).permute(1,2,0).numpy(), cmap=\"gray\")" ] }, { "cell_type": "markdown", "metadata": { "id": "9ozbt61sGpiv" }, "source": [ "### 3.1.5 Adaptive Average Pooling\n", "\n", "\n", "\n", "\n", "image from [paperswithcode.com under CC-BY-SA License](https://paperswithcode.com/method/global-average-pooling)\n", "\n", "**TODO** implement adaptive (global) average pooling following the [torch.nn.AdaptiveAvgPool2d documentation](https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool2d.html) with output size 1x1" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "p4EZQhgcGpiv" }, "outputs": [], "source": [ "# TODO: implement AdaptiveAvgPool2d with output size 1x1\n", "avgpool =\n", "\n", "# TODO: apply adaptive average pooling on the input image x\n", "x_pooled =\n", "\n", "print(x_pooled.shape)\n", "# remove (squeeze) the 1x1 dimensions (\"-1\" means last dimension)\n", "x_pooled = x_pooled.squeeze(-1).squeeze(-1)\n", "\n", "print(x_pooled.shape)\n", "\n", "print(x_pooled.squeeze())" ] }, { "cell_type": "markdown", "metadata": { "id": "TIIiGe1uGpiv" }, "source": [ "### 3.1.6 Linear (Dense) Layer\n", "\n", "a dense layer is a linear transformation $y = Ax + b$\n", "\n", "**TODO** Implement a linear (dense) layer with bias with 3 input dimensions and 8 output dimensions following [torch.nn.Linear documentation](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html)\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Woc2RDX3Gpiv" }, "outputs": [], "source": [ "# TODO: imlement the Linear Layer\n", "dense =\n", "\n", "# TODO: call the linear layer on x_pooled\n", "x_dense =\n", "\n", "print(f\"dense output {x_dense}\")\n", "\n", "# FYI:access the A matrix in the equation by\n", "print(f\"A matrix: {dense.weight} and bias {dense.bias}\")" ] }, { "cell_type": "markdown", "metadata": { "id": "DX9UGxCbRnF0" }, "source": [ "### 3.2 Build a complete AlexNet\n", "\n", "Now we will use those building blocks in a CNN. To this end, PyTorch implements models again in a class, this time `torch.nn.Module`. In there, you define:\n", "1. which layers your model has in the `__init__` function\n", "2. in which order they are to be executed in the `forward` function.\n", "3. we do not worry about training for now. We can save some memory by specifying that we do not need gradients with `@torch.no_grad()` later\n", "\n", "\n", "Note that we do not implement exactly the model in the figure of the first cell. Instead, we stay with the established torchvision alexnet implementation.\n", "\n", "**TODO** Implement the AlexNet\n", "\n", "Hint:\n", "* you can either implement it line-by-line following the instructions\n", "* or search the AlexNet class in the torchvision [source code](https://github.com/pytorch/vision) and copy-paste the model definition" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Au6pLUPdGpiw" }, "outputs": [], "source": [ "class AlexNet(nn.Module):\n", " def __init__(self, num_classes: int = 1000, dropout: float = 0.5) -> None:\n", " super().__init__()\n", " self.features = nn.Sequential(\n", " # TODO 11x11 convolution from 3 to 64 channels stride 4, padding 2\n", " # TODO inplace ReLU\n", " # TODO 3x3 Maxpool with stride 2\n", " # TODO 5x5 convolution from 64 to 192 channels and padding 2\n", " # TODO inplace ReLU\n", " # TODO 3x3 Maxpool with stride 2\n", " # TODO 3x3 convolution from 192 to 384 channels and padding 1\n", " # TODO inplace ReLU\n", " # TODO 3x3 convolution from 384 to 256 channels and padding 1\n", " # TODO inplace ReLU\n", " # TODO 3x3 convolution from 256 to 256 channels and padding 1\n", " # TODO inplace ReLU\n", " # TODO 3x3 Maxpool with stride 2\n", " )\n", " self.avgpool = nn.AdaptiveAvgPool2d((6, 6))\n", " self.classifier = nn.Sequential(\n", " # TODO Dropout\n", " # TODO Linear with 256*6*6 to 4096 channels\n", " # TODO inplace ReLU\n", " # TODO Dropout\n", " # TODO Linear with 4096 to 4096 channels\n", " # TODO inplace ReLU\n", " # TODO Linear with 4096 to num_classes channels\n", " )\n", "\n", " def forward(self, x: torch.Tensor) -> torch.Tensor:\n", " x = # TODO apply features sub-module\n", " x = # TODO apply average pooling sub-module\n", " x = torch.flatten(x, 1)\n", " x = # TODO apply classifier sub.module\n", " return x\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "9WtuNphbGpiw" }, "outputs": [], "source": [ "# Load the model :\n", "model = AlexNet(num_classes=21)\n", "\n", "# Perform a forward pass\n", "logits = model(x)" ] }, { "cell_type": "markdown", "metadata": { "id": "F6I8XLRXGpiw" }, "source": [ "## 4 Evaluation with pre-trained model\n", "\n", "Download model weights `alexnet-epoch=1414-val_accuracy=0.71.pth` by running the next cell.\n", "\n", "\n", "Alternatively, you can download it here:https://drive.google.com/uc?export=download&id=1S5bI-3cSdyUKw-oknoYI3KKXnK7e96rN" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "_nO5Ft0VGpiw" }, "outputs": [], "source": [ "import gdown\n", "\n", "url = \"https://drive.google.com/uc?export=download&id=1S5bI-3cSdyUKw-oknoYI3KKXnK7e96rN\"\n", "gdown.download(url, output=\"alexnet-epoch=1414-val_accuracy=0.71.pth\", quiet=True)" ] }, { "cell_type": "markdown", "metadata": { "id": "9m0k2TIjGpiw" }, "source": [ "this should output ``" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "tgalgetiGpix" }, "outputs": [], "source": [ "# TODO: load the model state dict with torch.load()\n", "state_dict =\n", "\n", "# TODO: update the model weights by calling load_state_dict on the model\n" ] }, { "cell_type": "markdown", "metadata": { "id": "EYAnnJH2VrGH" }, "source": [ "### 4.1 Qualitative Evaluation\n", "\n", "Let's check whether our model does the right thing. If we feed it an image of the right size it should return a tensor with n elements, where n equals the number of classes we have (21 in our case of the UC Merced land use dataset)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "SXTscg5GGpix" }, "outputs": [], "source": [ "model = model.cpu()\n", "\n", "# TODO: initialize the test_dataset with tansforms_val and test split\n", "test_dataset =\n", "\n", "# TODO take the first image in the dataset\n", "img, label =\n", "img = img.unsqueeze(0) # add 1-dimension (creates a batch of one sample)\n", "\n", "# TODO predict the image with the model\n", "logits =\n", "\n", "print(f'logits tensor size: {logits.size()}')\n", "print('logits tensor (model output) values:')\n", "print(logits)\n", "\n", "# TODO package the lines above into a function\n", "@torch.no_grad()\n", "def predict(dataset, index, model):\n", " \"\"\"\n", " inputs:\n", " dataset: the test dataset,\n", " index: index of the sample in the test dataset,\n", " model: the model\n", " returns:\n", " image: the input image\n", " label: the input label\n", " logits: a vector of prediction logits\n", " \"\"\"\n", " model.eval()\n", "\n", " return img, label, logits" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "7PCkvp3cGpix" }, "outputs": [], "source": [ "# Observe the predictions :\n", "test_dataset = UCMerced(transforms=transforms_val, split='test')\n", "\n", "@interact(idx=range(len(test_dataset)))\n", "def plot_logits(idx):\n", " img, label, prediction = predict(test_dataset, idx, model)\n", "\n", " fig, axs = plt.subplots(1,2, figsize=(12,3))\n", " axs[0].imshow(unnormalize(img).permute(1,2,0))\n", " axs[0].set_title(list(UCMerced.LABEL_CLASSES.keys())[label])\n", " axs[0].axis(\"off\")\n", "\n", " ax = axs[1]\n", " ax.bar(np.arange(21), prediction[0].detach())\n", " ax.set_xticks(np.arange(21))\n", " ax.set_xticklabels(UCMerced.LABEL_CLASSES.keys(), rotation=90)\n", " ax.set_ylabel(\"logits\")" ] }, { "cell_type": "markdown", "metadata": { "id": "B91hpdxdGpix" }, "source": [ "## Normalize Logits to Probabilities with Softmax\n", "\n", "usually, we are interested in output probabilities rather than outputs.\n", "\n", "we normalize the logits $z = (z_0, z_i, \\dots, z_C)$ to probabilities with the softmax function:\n", "\n", "$\\sigma(z)_i = \\frac{\\exp(z_i)}{\\sum_j^K \\exp(z_j)}$\n", "\n", "hint:\n", "* you can use `tensor.exp()` function for the exponential\n", "* you can use `tensor.sum()` to sum over all elements" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "2X3_1QPbGpix" }, "outputs": [], "source": [ "img, label, logits = predict(test_dataset, index=0, model=model)\n", "\n", "# TODO implement the softmax function\n", "def softmax(logits):\n", " val =\n", " return val\n", "\n", "print(\"implementation check: should be all True (equal to torch softmax function)\")\n", "torch.isclose(softmax(logits), nn.functional.softmax(logits, dim=-1))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "uvWfqjeKGpix" }, "outputs": [], "source": [ "softmax(logits)" ] }, { "cell_type": "markdown", "metadata": { "id": "14XdfxlcGpix" }, "source": [ "**TODO**: go back to the previous cells and modify the `predict` function to output probabilities rather than logits" ] }, { "cell_type": "markdown", "metadata": { "id": "H8AUFceBGpix" }, "source": [ "### Quantitative Evaluation\n", "\n", "TODO try changing the device now from `cpu` to `cuda` and run the code again.\n", "\n", "Note, the difference is not extremely large, as we just predict, for training, GPU will make a massive difference" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "WUWpL9QiGpiy" }, "outputs": [], "source": [ "test_dataset = UCMerced(transforms=transforms_val, split='test')\n", "from torch.utils.data import DataLoader\n", "from tqdm.auto import tqdm\n", "dataloader = DataLoader(test_dataset, batch_size=16)\n", "\n", "# TODO change cpu to cuda and see if the prediction is faster\n", "device =\n", "model = model.to(device)\n", "\n", "y_preds, y_trues = [], []\n", "with torch.no_grad():\n", " for x,y in tqdm(dataloader, total=len(dataloader)):\n", " x = x.to(device)\n", " y = y.to(device)\n", "\n", " logits = model(x)\n", " y_pred = logits.argmax(1)\n", " y_preds.append(y_pred.cpu().numpy())\n", " y_trues.append(y.cpu().numpy())\n", "\n", "y_preds = np.hstack(y_preds)\n", "y_trues = np.hstack(y_trues)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "sVLoB7bAGpiy" }, "outputs": [], "source": [ "# TODO calculate the overall accuracy\n", "overall_accuracy =\n", "print(overall_accuracy)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "obCPFaPrGpiy", "scrolled": false }, "outputs": [], "source": [ "from sklearn.metrics import classification_report\n", "\n", "# the names of all classes\n", "target_names = list(UCMerced.LABEL_CLASSES.keys())\n", "\n", "# TODO print the classification report\n", "print(classification_report( #TODO ))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "w_w534WAGpiy", "scrolled": false }, "outputs": [], "source": [ "from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n", "\n", "display = ConfusionMatrixDisplay.from_predictions(y_preds, y_trues,\n", " display_labels=classes, xticks_rotation=90, cmap=\"Blues\")\n", "\n", "display" ] }, { "cell_type": "markdown", "metadata": { "id": "Ddwa5OAyXyQN" }, "source": [ "## Appendix: Training the AlexNet (not included in the exercise)\n", "\n", "With the next cells, you can train the Alexnet. This requires a GPU (or a lot of time) and is not part of this exercise.\n", "\n", "This is for-your-interest only and we will talk about neural network training and optimization in detail in the next exercise. Note that in order to save some time, we're only training for 50 epochs here. The resulting model will thus not be as accurate as the one you loaded the pretrained weights into earlier. Feel free to change the amount of epochs if you'd like a better model." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "w6OeS-TCGpiy", "scrolled": false }, "outputs": [], "source": [ "!pip install -q -U pytorch_lightning tensorboard\n", "import pytorch_lightning as pl\n", "from pytorch_lightning import loggers as pl_loggers\n", "\n", "from pytorch_lightning.callbacks import ModelCheckpoint\n", "\n", "import torch.nn as nn\n", "import torch.nn.functional as F" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "J27Wb1EYGpiy" }, "outputs": [], "source": [ "%load_ext tensorboard\n", "%tensorboard --logdir logs" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "kq-DfjIeGpiy" }, "outputs": [], "source": [ "class PLWrapper(pl.LightningModule):\n", " def __init__(self, model):\n", " super().__init__()\n", " self.model = model\n", "\n", " def forward(self, x):\n", " return self.model(x)\n", "\n", " def training_step(self, batch, batch_idx):\n", " x, y = batch\n", " y_hat = self(x)\n", " loss = F.cross_entropy(y_hat, y)\n", " self.log(\"train_loss\", loss)\n", " return loss\n", "\n", " def validation_step(self, batch, batch_idx):\n", " x, y = batch\n", " y_hat = self.model(x)\n", " loss = F.cross_entropy(y_hat, y)\n", " self.log(\"val_loss\", loss)\n", " self.log(\"val_accuracy\", (y_hat.argmax(1) == y).float().mean())\n", "\n", " def configure_optimizers(self):\n", " return torch.optim.SGD(self.parameters(), lr=0.01, weight_decay=0.01)\n", "\n", "\n", "plmodel = PLWrapper(model)\n", "\n", "tb_logger = pl_loggers.TensorBoardLogger(save_dir=\"logs/\")\n", "\n", "checkpoint_callback = ModelCheckpoint(\n", " dirpath='checkpoints',\n", " filename='alexnet-{epoch}-{val_accuracy:.2f}',\n", " monitor=\"val_accuracy\",\n", " mode=\"max\"\n", " )\n", "\n", "trainer = pl.Trainer(max_epochs=1500, accelerator=\"gpu\", devices=[0],\n", " logger=tb_logger, callbacks=[checkpoint_callback])\n", "train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=256, shuffle=True, num_workers=8)\n", "val_loader = torch.utils.data.DataLoader(val_dataset, batch_size=256, shuffle=False, num_workers=8)\n", "trainer.fit(plmodel, train_dataloaders=train_loader, val_dataloaders=val_loader)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "1MksQHl5Gpiy" }, "outputs": [], "source": [ "!ls checkpoints\n", "checkpoint = torch.load(\"checkpoints/alexnet-epoch=1410-val_accuracy=0.75.ckpt\")\n", "from collections import OrderedDict\n", "state_dict = OrderedDict({k.replace(\"model.\",\"\"):v for k,v in checkpoint[\"state_dict\"].items()})\n", "torch.save(state_dict,\"alexnet-epoch=1410-val_accuracy=0.75.pth\")" ] } ], "metadata": { "accelerator": "GPU", "colab": { "provenance": [] }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.8" }, "vscode": { "interpreter": { "hash": "3ef253a3d10d8481140589c30479a16b962c8c121d91b8cba7660edf77ab7301" } } }, "nbformat": 4, "nbformat_minor": 0 }