Colossal-AI

Colossal-AI is an open-source deep learning framework that makes training large neural networks cheaper by combining several kinds of parallelism and memory management. It targets ML engineers and researchers running big models on clusters of GPUs.

The library supports data, tensor and pipeline parallelism and lets you mix them, with a heterogeneous memory manager called Gemini that moves tensors between GPU and CPU memory. Documentation includes tutorials such as training GPT with hybrid parallelism.

A command line interface launches distributed jobs and runs tensor-parallel benchmarks, and projects are configured through a single configuration file. It is developed by HPC-AI Technology, which also offers professional services for teams that need help.