DeepSpeed

DeepSpeed is Microsoft's open-source deep learning optimization library for training and serving very large models across many GPUs. Researchers and ML engineers use its ZeRO memory optimizations, offloading and parallelism to fit bigger models on the hardware they have.

ZeRO, ZeRO++ and ZeRO-Offload split optimizer states, gradients and parameters across devices or move them to CPU and NVMe. Pipeline and automatic tensor parallelism, mixture-of-experts support, mixed precision, autotuning and a FLOPs profiler are configured through a ds_config file.

It integrates with PyTorch and Hugging Face Transformers, supports multiple accelerator vendors including AMD GPUs, and keeps adding features such as long-sequence training and new optimizers. Tutorials, docs and the code are public on GitHub.