Hello @amogkam, Here https://github.com/pytorch/pytorch/issues/8741 is an old feature request for a pytorch fct to move optimizer to device. I use the optimizer_to function posted there and get around my OOM, training looks good so far. def optimizer_to(optim, device): for param in optim.stat…