Kubeflow is an open source project that provides Machine Learning (ML) resources on Kubernetes clusters. Kubernetes is evolving to be the hybrid solution for deploying complex workloads on private and public clouds. A fast growing use case is using Kubernetes as the deployment platform of choice for machine learning. Infrastructure engineers will often spend time modifying deployments before a single model can be tested. These deployments are often bound to the clusters they have been deployed to, thus moving a model from a laptop to a cloud cluster is difficult without significant re-architecture. The open source Kubeflow project addresses these concerns by enabling Github Machine Learning stacks on Kubernetes portable across environments. The repository contains:
JupyterHub to create & manage interactive Jupyter notebooks
A Tensorflow Custom Resource (CRD) that can be configured to use CPUs or GPUs, and adjusted to the size of a cluster
A TF Serving container
There are two parts to Kubeflow on Kubernetes:
A hypervisor - Kubernetes creates clusters of containers. These clusters require compute, networking and storage. The hypervisor virtualizes the host computer’s storage, networking and compute for the Kubernetes clusters.
Packages on the host operating system to create clusters on the hypervisor and install packages on the cluster.