CoCalc provides the best real-time collaborative environment for Jupyter Notebooks, LaTeX documents, and SageMath, scalable from individual users to large groups and classes!
CoCalc provides the best real-time collaborative environment for Jupyter Notebooks, LaTeX documents, and SageMath, scalable from individual users to large groups and classes!
Path: blob/main/intermediate_source/optimizer_step_in_backward_tutorial.py
Views: 494
"""12How to save memory by fusing the optimizer step into the backward pass3======================================================================45Hello there! This tutorial aims to showcase one way of reducing the6memory footprint of a training loop by reducing the memory taken by7the *gradients*. Say you have a model and you're interested in ways to8optimize memory to avoid ``Out of Memory`` (OOM) errors or simply to ooze9more out of your GPU. Well, you _might_ be in luck (if gradients take up10a portion of your memory and you do not need to do gradient accumulation).11We will explore the following:12131. What takes up memory during your training or finetuning loop,142. How to capture and visualize memory snapshots to determine the bottleneck,153. The new ``Tensor.register_post_accumulate_grad_hook(hook)`` API, and finally,164. How everything fits together in 10 lines to achieve memory savings.1718To run this tutorial, you will need:1920* PyTorch 2.1.0 or newer with ``torchvision``21* 1 CUDA GPU if you'd like to run the memory visualizations locally.22Otherwise, this technique would benefit similarly on any device.2324Let us start by importing the required modules and models. We will use a25vision transformer model from torchvision, but feel free to substitute26with your own model. We will also use ``torch.optim.Adam`` as our optimizer,27but, again, feel free to substitute with your own optimizer.2829"""3031import torch32from torchvision import models33from pickle import dump3435model = models.vit_l_16(weights='DEFAULT').cuda()36optimizer = torch.optim.Adam(model.parameters())3738###############################################################################39# Now let's define our typical training loop. You should use real images when40# training, but for the purposes of this tutorial, we are passing in fake41# inputs and not worrying about loading any actual data.4243IMAGE_SIZE = 2244445def train(model, optimizer):46# create our fake image input: tensor shape is batch_size, channels, height, width47fake_image = torch.rand(1, 3, IMAGE_SIZE, IMAGE_SIZE).cuda()4849# call our forward and backward50loss = model.forward(fake_image)51loss.sum().backward()5253# optimizer update54optimizer.step()55optimizer.zero_grad()5657###############################################################################58# Memory usage during training59# """"""""""""""""""""""""""""60# We are about to look at some memory snapshots, so we should be prepared to61# analyze them properly. Typically, training memory consists of:62#63# * Model parameters (size P)64# * Activations that are saved for the backward pass (size A)65# * Gradients, which are the same size as the model parameters, so size G = P.66# * Optimizer state, which is proportional to the size of the parameters. In67# this case, the state for Adam requires 2x the model parameters, so size O = 2P.68# * Intermediate tensors, which are allocated throughout the compute. We will69# not worry about them for now as they are usually small and ephemeral.70#71# Capturing and visualizing memory snapshots72# """"""""""""""""""""""""""""""""""""""""""73# Let's get us a memory snapshot! As your code runs, consider what you may expect74# the CUDA memory timeline to look like.7576# tell CUDA to start recording memory allocations77torch.cuda.memory._record_memory_history(enabled='all')7879# train 3 steps80for _ in range(3):81train(model, optimizer)8283# save a snapshot of the memory allocations84s = torch.cuda.memory._snapshot()85with open(f"snapshot.pickle", "wb") as f:86dump(s, f)8788# tell CUDA to stop recording memory allocations now89torch.cuda.memory._record_memory_history(enabled=None)9091###############################################################################92# Now open up the snapshot in the CUDA Memory Visualizer at93# https://pytorch.org/memory_viz by dragging and dropping the94# ``snapshot.pickle`` file. Does the memory timeline match your expectations?95#96# .. figure:: /_static/img/optim_step_in_bwd/snapshot.jpg97# :alt: snapshot.png loaded into CUDA Memory Visualizer98#99# The model parameters have already been loaded in memory before the training100# step, so we see a chunk of memory devoted to the weights right off the bat.101# As we start our forward pass, memory is allocated gradually for the activations,102# or the tensors we are saving to be able to compute gradients in the backward pass.103# Once we start the backward pass, the activations are gradually freed while memory104# of the gradients starts building up.105#106# Lastly, as the optimizer kicks in, its state will be lazily initialized, so we107# should see the optimizer state memory gradually increase during the optimizer108# step of the first training loop only. In future loops, the optimizer memory109# will remain and be updated in-place. The memory for the gradients is then110# freed accordingly at the end of every training loop when ``zero_grad`` is called.111#112# Where is the memory bottleneck in this training loop? Or, in other words,113# where is the peak memory?114#115# The peak memory usage is during the optimizer step! Note the memory then116# consists of ~1.2GB of parameters, ~1.2GB of gradients, and ~2.4GB=2*1.2GB of117# the optimizer state as expected. The last ~1.2GB comes from Adam optimizer118# requiring memory for intermediates, totaling to ~6GB of peak memory.119# Technically, you can remove the need for the last 1.2GB for optimizer120# intermediates if you set ``Adam(model.parameters(), foreach=False)`` which121# would trade off runtime for memory. If switching off the ``foreach`` runtime122# optimization is sufficient in memory savings for you, nice, but please123# read on if you're curious how this tutorial can help you do better!124# With the technique we will soon introduce, we will reduce peak memory by125# removing the need for the ~1.2GB of **gradients memory** as well as **optimizer126# intermediates memory**. Now, what would you expect the new peak memory to be?127# The answer will be revealed in the `next` snapshot.128#129# DISCLAIMER: This technique is **not** for all130# """""""""""""""""""""""""""""""""""""""""""""131# Before we get too excited, we have to consider whether this technique is applicable132# for `your` use case. This is NOT a silver bullet! The technique of fusing the133# optimizer step into the backward only targets reducing *gradient* memory (and as a side effect also optimizer intermediates134# memory). Thus, the more sizable the memory taken up by the gradients, the more135# tantamount the memory reduction. In our example above, the gradients eat up 20%136# of the memory pie, which is quite sizable!137#138# This may not be the case for you, for example, if your weights are already tiny,139# (say, due to applying LoRa,) then the gradients do not take much space in your140# training loop and the wins are way less exciting. In that case, you should141# first try other techniques like activations checkpointing, distributed142# training, quantization, or reducing the batch size. Then, when the gradients143# are part of the bottleneck again, come back to this tutorial!144#145# Still here? Cool, let's introduce our new ``register_post_accumulate_grad_hook(hook)``146# API on Tensor.147#148# ``Tensor.register_post_accumulate_grad_hook(hook)`` API and our technique149# """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""150# Our technique relies on not having to save the gradients during ``backward()``. Instead,151# once a gradient has been accumulated, we will immediately apply the optimizer to152# the corresponding parameter and drop that gradient entirely! This removes the need153# for holding onto a big buffer of gradients until the optimizer step.154#155# So how can we unlock the behavior of applying the optimizer more eagerly? In our 2.1156# release, we've added a new API :func:`torch.Tensor.register_post_accumulate_grad_hook`157# that would allow us to add a hook onto a Tensor once its ``.grad`` field has been158# accumulated. We will encapsulate the optimizer step into this hook. How?159#160# How everything fits together in 10 lines161# """"""""""""""""""""""""""""""""""""""""162# Remember our model and optimizer setup from the beginning? I'll leave them commented163# out below so we don't spend resources rerunning the code.164#165# .. code-block:: python166#167# model = models.vit_l_16(weights='DEFAULT').cuda()168# optimizer = torch.optim.Adam(model.parameters())169170# Instead of having just *one* optimizer, we will have a ``dict`` of optimizers171# for every parameter so we could reference them in our hook.172optimizer_dict = {p: torch.optim.Adam([p], foreach=False) for p in model.parameters()}173174# Define our hook, which will call the optimizer ``step()`` and ``zero_grad()``175def optimizer_hook(parameter) -> None:176optimizer_dict[parameter].step()177optimizer_dict[parameter].zero_grad()178179# Register the hook onto every parameter180for p in model.parameters():181p.register_post_accumulate_grad_hook(optimizer_hook)182183# Now remember our previous ``train()`` function? Since the optimizer has been184# fused into the backward, we can remove the optimizer step and zero_grad calls.185def train(model):186# create our fake image input: tensor shape is batch_size, channels, height, width187fake_image = torch.rand(1, 3, IMAGE_SIZE, IMAGE_SIZE).cuda()188189# call our forward and backward190loss = model.forward(fake_image)191loss.sum().backward()192193# optimizer update --> no longer needed!194# optimizer.step()195# optimizer.zero_grad()196197########################################################################198# That took about 10 lines of changes in our sample model, which is neat.199# However, for real models, it could be a fairly intrusive change to switch200# out the optimizer for an optimizer dictionary, especially for those who use201# ``LRScheduler``s or manipulate optimizer configuration throughout the202# training epochs. Working out this API with those changes will be more203# involved and will likely require moving more configuration into global204# state but should not be impossible. That said, a next step for PyTorch205# is to make this API easier to adopt with LRSchedulers and other features206# you are already used to.207#208# But let me get back to convincing you that this technique is worth it.209# We will consult our friend, the memory snapshot.210211# delete optimizer memory from before to get a clean slate for the next212# memory snapshot213del optimizer214215# tell CUDA to start recording memory allocations216torch.cuda.memory._record_memory_history(enabled='all')217218# train 3 steps. note that we no longer pass the optimizer into train()219for _ in range(3):220train(model)221222# save a snapshot of the memory allocations223s = torch.cuda.memory._snapshot()224with open(f"snapshot-opt-in-bwd.pickle", "wb") as f:225dump(s, f)226227# tell CUDA to stop recording memory allocations now228torch.cuda.memory._record_memory_history(enabled=None)229230###############################################################################231# Yes, take some time to drag your snapshot into the CUDA Memory Visualizer.232#233# .. figure:: /_static/img/optim_step_in_bwd/snapshot_opt_in_bwd.jpg234# :alt: snapshot.png loaded into CUDA Memory Visualizer235#236# Several major observations:237# 1. There is no more optimizer step! Right...we fused that into the backward.238# 2. Likewise, the backward drags longer and there are more random allocations239# for intermediates. This is expected, as the optimizer step requires240# intermediates.241# 3. Most importantly! The peak memory is lower! It is now ~4GB (which I242# hope maps closely to your earlier expectation).243#244# Note that there is no longer any big chunk of memory allocated for the gradients245# compared to before, accounting for ~1.2GB of memory savings. Instead, we've freed246# each gradient very quickly after they've been computed by moving the optimizer247# step as far ahead as we can. Woohoo! By the way, the other ~1.2GB of memory savings248# comes from breaking apart the optimizer into per-parameter optimizers, so the249# intermediates have proportionally shrunk. This detail is `less important` than250# the gradient memory savings, as you can get optimizer intermediates savings251# from just turning ``foreach=False`` without this technique.252#253# You may be correctly wondering: if we saved 2.4GB of memory, why is the peak memory254# NOT 6GB - 2.4GB = 3.6GB? Well, the peak has moved! The peak is now near the start255# of the backward step, when we still have activations in memory, where before, the peak256# was during the optimizer step when the activations had been freed. The ~0.4GB difference257# accounting for ~4.0GB - ~3.6GB is thus due to the activations memory. One can then258# imagine that this technique can be coupled with activations checkpointing for more259# memory wins.260#261# Conclusion262# """"""""""263# In this tutorial, we learned about the memory saving technique of264# fusing the optimizer into the backward step through the new265# ``Tensor.register_post_accumulate_grad_hook()`` API and *when* to apply this266# technique (when gradients memory is significant). Along the way, we also learned267# about memory snapshots, which are generally useful in memory optimization.268269270