The growth and innovation in the Kubernetes project, since it first launched just over four years ago, has been tremendous to see. In part 1 of my blog, I talked about how Red Hat has been a key contributor to Kubernetes since the launch of the project, detailed where we invested our resources and what drove those decisions. Today, that innovation continues and we are just as excited for what comes next. In this blog, I’d like to talk about where we are going and what we’re focused on, as we continue driving innovation in Kubernetes and the broader cloud native ecosystem and building the next generation of OpenShift.

From Building the Platform to Operating the Platform

In the first few years, Red Hat’s objective for Kubernetes was to build out a production grade modern Linux container platform, which was was an extension of Linux itself. Just as we made Red Hat Enterprise Linux, the world’s leading enterprise Linux platform, we are focusing on making Red Hat OpenShift the standard for enterprise Kubernetes. We also focused on establishing open community container standards and protecting those standards by helping to launch both CNCF and the Open Container Initiative, to provide vendor neutral governance.  

While the Kubernetes platform continues to evolve, over the past year we have shifted more focus to how you operate the platform, throughout the stack, from infrastructure to applications. Once again, we took direction from our enterprise customers who have installed, upgraded and managed a number of OpenShift Kubernetes clusters and are relying on them for mission critical production applications. This was a primary driver in our decision to acquire CoreOS earlier this year, which I have often described as a $250 million investment in Kubernetes Day 2 management and automated operations. This focus has been a key driver for the OpenShift roadmap for 2018, 2019 and beyond.

Day 2 Management & Monitoring with Prometheus & Grafana

The ability to monitor your Kubernetes clusters and alert when issues occur is a critical requirement for platform administrators, to help ensure platform and application health. While Red Hat had already decided to adopt Prometheus, CoreOS had established a leadership position in the Prometheus community and brought valuable expertise to Red Hat. In OpenShift 3.11, released this fall, we delivered native Prometheus monitoring, alerting and integrated Grafana dashboards, fully supported in OpenShift. This included default Grafana dashboards and Prometheus alert definitions for Kubernetes, pre-configured on install, developed through the Prometheus monitoring mixin project. Finally, we integrated the CoreOS Tectonic console with the OpenShift console, in an effort to provide cluster administrators a comprehensive Kubernetes-native admin experience that complemented OpenShift’s existing service catalog and developer focused interfaces.

Managing Your Services with Kubernetes Operators & CRDs

Another innovation introduced by CoreOS back in 2016 was the Operator pattern for managing services that run on Kubernetes. As Brandon Phillips described then, an Operator is “an application-specific controller that extends the Kubernetes API to create, configure, and manage instances of complex stateful applications on behalf of a Kubernetes user”. Earlier this year, at KubeCon Europe, Red Hat and CoreOS launched the Operator Framework to accelerate and standardize the development of Kubernetes Operators. In OpenShift 3.11, we launched the first previews of Operator-based services from Red Hat and our ISV partners.

Why is this so important? Running complex stateful applications like databases, middleware and other services often requires operational domain knowledge that is specific to that service. Operators allow that domain knowledge to be programmed into the service, in an effort to make each service self-managing, while taking advantage of the Kubernetes API and declarative management capabilities to do so. By building operators for their applications, ISVs and service owners can provide self-managed services to enable the same 'as-a-Service' functionality that's found in the public cloud, but more consistently deliver that on any Kubernetes platform in the data center or across multiple public clouds. This a win for both application owners and customers, who can get simplified operations across an Open Hybrid Cloud environment.

Operators take advantage of Kubernetes Custom Resource Definitions (CRDs), which is a feature that Red Hat played a big role in developing. Custom resources are extensions of the Kubernetes API and CRDs allow you to define those resources. Red Hat helped lead the development of CRDs in order to build more integrated feature extensions in OpenShift, as an extension of the Kubernetes API itself. I described some of those extended features in my first blog. This also allows other contributors to build their own extensions to Kubernetes, while helping to keep the Kubernetes core slim and avoiding bloat. With Operators, which leverage CRDs, complex stateful services that run on Kubernetes likewise become an extension of the Kubernetes API and this opens up a whole new range of possibilities for managing these services in an automated fashion, as we move forward.  

Using Kubernetes to Install and Upgrade Kubernetes

Kubernetes does a great job managing a containerized application through powerful, declarative automation primitives.  Unfortunately Kubernetes doesn’t typically manage itself or its own underlying infrastructure in the same manner. The new OpenShift installer, currently being developed upstream in OKD, and being introduced in OpenShift 4, is designed to bring every aspect of the cluster under control of a cluster operator including the control plane and its underlying infrastructure.

To manage underlying infrastructure, OpenShift is adopting machine operators that are tuned for each provider environment. By representing nodes and related infrastructure as Kubernetes-style API objects, the community can develop controllers to manage Kubernetes machines by defining a desired state and reconciling to that state, across the different public cloud or data center footprints where those Kubernetes machines may run.

Starting from a single Kubernetes node, users will be able to bootstrap their Kubernetes cluster in minutes and also be able to add and remove capacity from their cluster over time.  Operator based platform services will be used to deploy self-hosted platform components like Kubernetes, etcd, Prometheus, Grafana, Elasticsearch, Kibana, software defined networking, storage, registry and other OpenShift platform components.

Deploying Kubernetes on Immutable Infrastructure

Red Hat CoreOS will provide a fully immutable Linux container host that is managed by the OpenShift 4 installer, which will also manage the configuration of the underlying cloud or data center infrastructure it runs on. We will be demonstrating this at Kubecon and the OpenShift Commons Gathering for OpenShift on AWS and then extending full stack management automation to OpenShift on OpenStack, Azure, Google Cloud Platform, Bare Metal environments, VMWare, Red Hat Virtualization and other infrastructure provider footprints where OpenShift Kubernetes clusters may run.

While Red Hat CoreOS is based on the RHEL kernel and core libraries (i.e. the RHEL “core”), OpenShift will also continue to be supported on a traditional RPM-based RHEL container host environment. In this environment, the OpenShift Ansible installer will bootstrap the initial Kubernetes environment and users will be responsible for manage the configuration and upgrades of the RHEL hosts and managing the configuration of the underlying infrastructure on their own. This can give users the flexibility to choose a fully immutable and full stack managed OpenShift environment (using Red Hat CoreOS) or choose to run OpenShift on a traditional RHEL infrastructure environment that they manage themselves.

Keeping Your Clusters Up To Date with Over the Air Updates

Once you install your Kubernetes clusters, the next big challenge is keeping them up to date.  CoreOS had pioneered the concept of “over the air updates” for Tectonic and Container Linux and Red Hat is bringing that capability to OpenShift 4. Customers that are able to connect their  OpenShift clusters to Red Hat, can receive automated notifications about new releases and critical patches that they can automatically apply to each of their clusters. Customers running in disconnected or air gapped environments will still be able to download and install those same updates from a local container registry, without relying on a connection to Red Hat.  

Over the air updates are enabled by automated Prometheus-based telemetry that reports on the state of each cluster and a remote update server that will advise users when they are out of date, in the same way your mobile phone or laptop notifies you when a new OS version or patch is available. Through Operators, over the air updates can then be extended to all of the operator-based OpenShift platform services running on the cluster. Our ultimate goal is to extend over-the air updates to all ISVs deploying certified Operators with Red Hat, so that both your cluster and all of its content can be more secure and up to date.

Multi-Cluster Management and Federation

Once organizations start using Kubernetes, we have seen usage grow and customers deploying multiple clusters. We have certainly seen that firsthand with OpenShift, where the average customer ranges from a few cluster to tens of clusters as they grow their number of users, create new lifecycle environments and deploy clusters to new data centers or public cloud footprints. This is why we’ve been growing our investment in multi-cluster management and cluster federation. Building on our work in cluster administration with Prometheus and Grafana, Red Hat is focused on expanding this to a multi-cluster environment, running across different public clouds, private clouds, virtualization platforms and bare metal environments.

The Kubernetes Federation v2 project, which Red Hat is contributing to as part of the Multi-Cluster SIG, is then focused on providing capabilities for managing application deployments across multiple clusters. These components include a cluster registry, to store and access metadata information on all your clusters, as well as multi-cluster ingress, deployment and related primitives. Through our efforts in multi-cluster management and federation, our goal is to provide better tools to operate and manage application deployments across a multi-cluster and multi-cloud environment.

Innovating Beyond Kubernetes with Istio

Istio service-mesh is a great example of how innovation is growing in the Cloud Native ecosystem beyond Kubernetes. Red Hat is contributing to Istio upstream and bringing this capability to OpenShift users. Istio adds a distributed control plane for microservices to a distributed data plane with Envoy, that routes and manages requests to those services via lightweight distributed proxies. Multiple OpenShift customers are already piloting Istio on OpenShift and with the recent preview launch in OpenShift 3.11 we are bringing this capability to a broader set of users. There is more work to do upstream as we continue to focus on core areas like performance and multi-tenancy and Red Hat is also driving capabilities around service mesh monitoring and observability through our work in related projects like Jaeger and Kiali.

From Microservices to Serverless

Knative, launched earlier this year, is another example of building innovation on top of Kubernetes. As part of our work on Knative, Red Hat is looking to bring open hybrid serverless capabilities to OpenShift and enabling Function as a Service (FaaS). Today, FaaS solutions like AWS Lambda are typically limited to a single cloud environment. Our goal is to work with the Knative and Kubernetes communities and different FaaS providers, to bring Serverless and FaaS capabilities to developers building applications across a hybrid, multi-cloud infrastructure.

Enabling More Kubernetes Services

Of course FaaS and Serverless rely on access to rich services that can be automatically triggered by your application functions. We have seen an expansion in the types of applications and services customers are running on Kubernetes, including database, analytics, machine learning and more. One recent example is the work Red Hat drove with partners like Nvidia to bring GPU-based services in Kubernetes. This was enabled by the work of the Kubernetes Resource Management working group and various SIGs to support performance sensitive workloads, new hardware devices and bringing Kubernetes to a broader set of use cases. Another example is the work Red Hat is driving through the Kubevirt project, to enable Kubernetes to orchestrate and manage traditional VM-based services together with container-based services.

By enabling both hybrid cloud infrastructure and hybrid services and new development paradigms like Serverless & FaaS, users should be able to take advantage of functions and services to build applications that span from their data centers and across multiple public cloud environments. Through OpenShift, Red Hat strives to provide access to the broadest selection of services by working with our own developers as well as with partner ISVs and public cloud providers.  


Some observers may have assumed that after 4+ years, the pace of innovation in Kubernetes would be slowing down. In fact, I believe that innovation is accelerating, as I have hopefully illustrated here. Red Hat plans to continue to drive innovation in Kubernetes and the Cloud Native ecosystem, working with both customers and the open source community to deliver that innovation in a open, hybrid, more secure and fully supported fashion through OpenShift.


Kubernetes, How-tos

< Back to the blog