Gradient checkpointing

Gradient checkpointing enables you to run a more powerful model on your machine - beneficial under training. A more careful analysis has to be done if applying this technique in the cloud as you are essentially trading memory for time. If a more powerful model isn’t required then it can enable a larger batch size.

Essentially many activations are recomputed on the backward pass as needed rather than having been fully stored in the original forward pass. Estimates are up to a 50% reduction in memory footprint.

#GPU memory, gradient checkpointing

Read Next

Applied Ml resource

Thought I'd share this awesome repo for applied ML from the prodigous blogger Eugene Yan :) - https://github.com/eugeneyan/applied-ml

Batch processing

A crucial technique in training neural network (NN) models. Instead of processing individual data samples one at a time, batch processing groups multiple samples into batches and processes them simultaneously.