{"cells":[{"cell_type":"markdown","metadata":{"id":"tolu6dzqmyoH"},"source":["# Tensors\n"]},{"cell_type":"markdown","source":["*Adapted from `pytorch.org/tutorials/beginner/basics/tensorqs_tutorial.html`*.\n","\n","Tensors are a specialized data structure that are very similar to arrays and matrices.\n","In PyTorch, we use tensors to encode the inputs and outputs of a model, as well as the model’s parameters.\n","\n","Tensors are similar to NumPy’s ndarrays, except that tensors can run on GPUs or other hardware accelerators. Tensors\n","are also optimized for automatic differentiation (we'll see more about that later in the Autograd section). If you’re familiar with ndarrays, you’ll be right at home with the Tensor API!"],"metadata":{"id":"iB3neJGNnVnv"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"sYwZaEfNmyoJ"},"outputs":[],"source":["import torch\n","import numpy as np"]},{"cell_type":"markdown","metadata":{"id":"Dt2FRIECmyoJ"},"source":["## Initializing a Tensor\n","\n","Tensors can be initialized in various ways. Take a look at the following examples:\n","\n","**Directly from data**\n","\n","Tensors can be created directly from data. The data type is automatically inferred.\n","\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"OCVxyVJjmyoK","executionInfo":{"status":"ok","timestamp":1701379207495,"user_tz":-60,"elapsed":22,"user":{"displayName":"Alessandro Favero","userId":"04135226433552403200"}},"outputId":"2da4dc91-fe8a-423c-8ddf-b8c750fa1bed","colab":{"base_uri":"https://localhost:8080/"}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([[1., 2.],\n","        [3., 4.]])"]},"metadata":{},"execution_count":2}],"source":["data = [[1, 2],[3, 4.]]\n","x_data = torch.tensor(data)\n","x_data # Notice that all elements of a tensor must have the same data type"]},{"cell_type":"markdown","metadata":{"id":"VNwlC-7vmyoK"},"source":["**From a NumPy array**\n","\n","Tensors can be created from NumPy arrays (and vice versa - `x_data.numpy()`).\n","\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"5FfK_c-vmyoL","executionInfo":{"status":"ok","timestamp":1701379207495,"user_tz":-60,"elapsed":19,"user":{"displayName":"Alessandro Favero","userId":"04135226433552403200"}},"outputId":"e4a3f786-b59d-42a6-81cc-1b68bc050e71","colab":{"base_uri":"https://localhost:8080/"}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([[1., 2.],\n","        [3., 4.]], dtype=torch.float64)"]},"metadata":{},"execution_count":3}],"source":["np_array = np.array(data)\n","x_np = torch.from_numpy(np_array)\n","x_np"]},{"cell_type":"markdown","metadata":{"id":"FV0Z5iOumyoM"},"source":["**With random or constant values:**\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"yfZf0V7lmyoM","executionInfo":{"status":"ok","timestamp":1701379207496,"user_tz":-60,"elapsed":18,"user":{"displayName":"Alessandro Favero","userId":"04135226433552403200"}},"outputId":"04bacb24-66f0-4554-b7f1-5d00c4e24d16","colab":{"base_uri":"https://localhost:8080/"}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([[0.5454, 0.6531, 0.4457],\n","        [0.7516, 0.4656, 0.7648]])"]},"metadata":{},"execution_count":4}],"source":["rand_tensor = torch.rand(2,3)\n","rand_tensor"]},{"cell_type":"markdown","metadata":{"id":"1m789xuymyoN"},"source":["## Attributes of a Tensor\n","\n","Tensor attributes describe their shape, and the device on which they are stored.\n","\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"tXF80du2myoN","executionInfo":{"status":"ok","timestamp":1701379207496,"user_tz":-60,"elapsed":17,"user":{"displayName":"Alessandro Favero","userId":"04135226433552403200"}},"outputId":"0d877774-cbfb-49a9-d863-d730ea9c4dff","colab":{"base_uri":"https://localhost:8080/"}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["torch.Size([3, 4])"]},"metadata":{},"execution_count":5}],"source":["tensor = torch.rand(3,4)\n","tensor.shape # Shape"]},{"cell_type":"code","source":["tensor.device # Device on which the tensor is stored, e.g., cpu or gpu"],"metadata":{"id":"NSRw1U2VoWz8","executionInfo":{"status":"ok","timestamp":1701379207496,"user_tz":-60,"elapsed":15,"user":{"displayName":"Alessandro Favero","userId":"04135226433552403200"}},"outputId":"b07129ad-41aa-4466-c3bf-823ba90031e3","colab":{"base_uri":"https://localhost:8080/"}},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["device(type='cpu')"]},"metadata":{},"execution_count":6}]},{"cell_type":"markdown","metadata":{"id":"fHWzPlSBmyoN"},"source":["## Operations on Tensors\n","\n","Over 100 tensor operations, including arithmetic, linear algebra, matrix manipulation (transposing,\n","indexing, slicing), sampling and more are\n","comprehensively described [here](https://pytorch.org/docs/stable/torch.html).\n","\n","Each of these operations can be run on the GPU (at typically higher speeds than on a\n","CPU). If you’re using Colab, allocate a GPU by going to `Runtime > Change runtime type > GPU`.\n","\n","By default, tensors are created on the CPU. We need to explicitly move tensors to the GPU using\n","``.to`` method (after checking for GPU availability). Keep in mind that copying large tensors\n","across devices can be expensive in terms of time and memory!\n","\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"gditg145myoN"},"outputs":[],"source":["# We move our tensor to the GPU if available\n","if torch.cuda.is_available():\n","    tensor = tensor.to(\"cuda\")"]},{"cell_type":"markdown","metadata":{"id":"Q3BHYLEvmyoO"},"source":["Try out some of the operations from the list.\n","If you're familiar with the NumPy API, you'll find the Tensor API a breeze to use.\n","\n","\n"]},{"cell_type":"markdown","metadata":{"id":"imINZ_JCmyoO"},"source":["**Standard numpy-like indexing and slicing:**\n","\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"3qqm4BnCmyoO","executionInfo":{"status":"ok","timestamp":1701379207496,"user_tz":-60,"elapsed":12,"user":{"displayName":"Alessandro Favero","userId":"04135226433552403200"}},"outputId":"0833d792-de11-45f8-de26-71f3f4a740fa","colab":{"base_uri":"https://localhost:8080/"}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["(tensor([[1., 1., 1., 1.],\n","         [1., 1., 1., 1.],\n","         [1., 1., 1., 1.]]),\n"," tensor([1., 1., 1., 1.]))"]},"metadata":{},"execution_count":8}],"source":["tensor = torch.ones(3, 4)\n","tensor, tensor[0]"]},{"cell_type":"markdown","metadata":{"id":"HHBpWNKMmyoP"},"source":["**Arithmetic operations**\n","\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"LJ4KDJnKmyoP","executionInfo":{"status":"ok","timestamp":1701379207496,"user_tz":-60,"elapsed":11,"user":{"displayName":"Alessandro Favero","userId":"04135226433552403200"}},"outputId":"9193cac8-ee3e-43c8-dc23-10e06f4e7417","colab":{"base_uri":"https://localhost:8080/"}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([[4., 4., 4.],\n","        [4., 4., 4.],\n","        [4., 4., 4.]])"]},"metadata":{},"execution_count":9}],"source":["# This computes the matrix multiplication between two tensors.\n","y = tensor @ tensor.T\n","y"]},{"cell_type":"markdown","metadata":{"id":"1S0rugqNmyoP"},"source":["**Single-element tensors** If you have a one-element tensor, for example by aggregating all\n","values of a tensor into one value, you can convert it to a Python\n","numerical value using ``item()``:\n","\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Eo59wV5ZmyoP","executionInfo":{"status":"ok","timestamp":1701379207497,"user_tz":-60,"elapsed":10,"user":{"displayName":"Alessandro Favero","userId":"04135226433552403200"}},"outputId":"fc0586ed-5e00-49b2-9b17-3650a3455dc5","colab":{"base_uri":"https://localhost:8080/"}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["(tensor(12.), 12.0)"]},"metadata":{},"execution_count":10}],"source":["agg = tensor.sum()\n","agg_item = agg.item()\n","agg, agg_item"]},{"cell_type":"markdown","source":["# Autograd"],"metadata":{"id":"ejg-uZGTqxqZ"}},{"cell_type":"markdown","source":["When training neural networks, the most frequently used algorithm is back propagation. In this algorithm, parameters (model weights) are adjusted according to the gradient of the loss function with respect to the given parameter.\n","\n","To compute those gradients, PyTorch has a built-in differentiation engine called torch.autograd. It supports automatic computation of gradient for any computational graph.\n","\n","Let's create two tensors `a` and `b` with `requires_grad=True` to signal PyTorch that every operation on them should be tracked."],"metadata":{"id":"k7S83Gn9t12k"}},{"cell_type":"code","source":["a = torch.tensor([2.], requires_grad=True)\n","b = torch.tensor([6.], requires_grad=True)"],"metadata":{"id":"FvQyUU95rBMm"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["Let's define a new tensor `Q` from `a` and `b`,\n","$$Q = 3a^3-b^2$$"],"metadata":{"id":"WD-yXRUzuG-G"}},{"cell_type":"code","source":["Q = 3 * a**3 - b**2"],"metadata":{"id":"DBfj4A37rNVO"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["We can use PyTorch to compute $\\frac{\\partial Q}{\\partial a}$ and $\\frac{\\partial Q}{\\partial b}$.\n","\n","Calling `Q.backward()`, PyTorch computes these derivatives and stores them in the `.grad` attributes of `a` and `b` respectively."],"metadata":{"id":"4sDPJvXquWIV"}},{"cell_type":"code","source":["Q.backward()"],"metadata":{"id":"wJ-kJOYarVaZ"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["a.grad == 9 * a**2 # Compares with actual derivative"],"metadata":{"id":"TbOfJCo3rZqI","executionInfo":{"status":"ok","timestamp":1701379207497,"user_tz":-60,"elapsed":7,"user":{"displayName":"Alessandro Favero","userId":"04135226433552403200"}},"outputId":"1dd88a51-996c-4633-b21c-c76ebf9e5dd4","colab":{"base_uri":"https://localhost:8080/"}},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([True])"]},"metadata":{},"execution_count":14}]},{"cell_type":"code","source":["b.grad == -2 * b # Compares with actual derivative"],"metadata":{"id":"KS8b9uz0rig0","executionInfo":{"status":"ok","timestamp":1701379207839,"user_tz":-60,"elapsed":348,"user":{"displayName":"Alessandro Favero","userId":"04135226433552403200"}},"outputId":"804be304-eaa0-4aba-a886-eb9181130afa","colab":{"base_uri":"https://localhost:8080/"}},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([True])"]},"metadata":{},"execution_count":15}]},{"cell_type":"markdown","source":["Notice that if you compute new derivatives with respect to `a`, the values will be accumulated (and not substituted!)."],"metadata":{"id":"dvrC1Dk0vBWs"}},{"cell_type":"code","source":["a.grad"],"metadata":{"id":"52pf3-J9rnRZ","executionInfo":{"status":"ok","timestamp":1701379207839,"user_tz":-60,"elapsed":9,"user":{"displayName":"Alessandro Favero","userId":"04135226433552403200"}},"outputId":"7ddce2f4-ee3e-4770-da5a-e24ce90d27f0","colab":{"base_uri":"https://localhost:8080/"}},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([36.])"]},"metadata":{},"execution_count":16}]},{"cell_type":"code","source":["Q = 3 * a**3 - b**2\n","Q.backward()\n","a.grad"],"metadata":{"id":"SCaRuSDbryqo","executionInfo":{"status":"ok","timestamp":1701379207839,"user_tz":-60,"elapsed":7,"user":{"displayName":"Alessandro Favero","userId":"04135226433552403200"}},"outputId":"4a4123a5-47c5-482c-9d43-2afd9c1b2e9f","colab":{"base_uri":"https://localhost:8080/"}},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([72.])"]},"metadata":{},"execution_count":17}]},{"cell_type":"markdown","source":["Then, if you want to compute a new derivative, you should zero the `grad` attribute with `a.grad.zero_()`."],"metadata":{"id":"Cy3xv7rmvPyA"}},{"cell_type":"code","source":["a.grad.zero_()\n","Q = 3 * a**3 - b**2\n","Q.backward()\n","a.grad"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"qUqE_wa9tJWK","executionInfo":{"status":"ok","timestamp":1701379207840,"user_tz":-60,"elapsed":6,"user":{"displayName":"Alessandro Favero","userId":"04135226433552403200"}},"outputId":"f67d70f2-4d6f-4e39-96b5-8bfbc59d8b55"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["tensor([36.])"]},"metadata":{},"execution_count":18}]},{"cell_type":"markdown","source":["*If you want to know more about `torch.autograd` you can check `https://pytorch.org/tutorials/beginner/basics/autogradqs_tutorial.html`.*"],"metadata":{"id":"7uBKUEYFwowQ"}},{"cell_type":"markdown","source":["## Exercise"],"metadata":{"id":"4_vMqlr7vfUf"}},{"cell_type":"markdown","source":["Given the tensors `u = [1., 2., 3.]`, `v = [4., 5.]`, and the function\n","$$f(u, v) =\n","\\sum_i u_i^2 + \\sum_i \\log v_i,$$\n","compute the gradients of $f$ with respect to $u$ and $v$ and compare with your analytical predictions."],"metadata":{"id":"Dhke0-Ymvihh"}}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.13"},"colab":{"provenance":[{"file_id":"https://github.com/pytorch/tutorials/blob/gh-pages/_downloads/0e6615c5a7bc71e01ff3c51217ea00da/tensorqs_tutorial.ipynb","timestamp":1701372326056}]}},"nbformat":4,"nbformat_minor":0}