The network had been training for the last 12 hours. It all looked good: the gradients were flowing and the loss was decreasing. But then…