MLServer

MLServer is Seldon's open-source inference server that exposes machine learning models through REST and gRPC endpoints following the V2 Inference Protocol. It is used by ML engineers who deploy models on Kubernetes with Seldon Core or KServe.

Ready-made inference runtimes cover scikit-learn, XGBoost, LightGBM, CatBoost, Spark MLlib, MLflow, Hugging Face and Alibi, and you can write a custom runtime for anything else. Each runtime is a separate pip package, so you install only what your models need.

Several models can run inside one process, a pool of workers handles parallel inference, and adaptive batching groups incoming requests on the fly. It is the core Python server inside Seldon Core and KServe, but it also runs on its own after a pip install.