Planning for and Reviewing Recent Kubernetes API Deprecations

Kubernetes has a very explicit policy of API deprecation and removal which has gained importance as v1.22 exercised this policy to finally remove APIs that have been marked as deprecated for many releases. The upcoming v1.25 will also remove more APIs.

Many of these APIs have migrated from beta to stable and overall this represented a huge step in maturity for Kubernetes…but that API deprecation policy change was almost 3 years ago in September of 2019. Now they need to go.

Red Hat OpenShift’s Remote Health Monitoring is designed to give Red Hat insight into real world usage of Kubernetes to proactively spot bugs, fix platform specific problems and forecast issues like deprecated APIs. Once the OpenShift user base started upgrading to OpenShift 4.8, we began to see some actual usage numbers for these deprecated APIs. The usage continued to grow for another 6 months, but thankfully, that growth has plateaued more recently as customers reduced their usage of the APIs.

While the usage numbers were perhaps higher than expected, this is the first large-scale Kubernetes API deprecation in the community's history. We’ll dig into more detail below, but first let’s learn about Kubernetes and OpenShift deprecation policies.

Recent Kubernetes Deprecations

Complicating the removal of v1beta1 APIs in Kubernetes v1.22 is that many of these objects were quite popular, especially as many users gained steam on their Kubernetes adoption in 2019, taking advantage of more advanced APIs like:

• CustomResourceDefinition (CRD)
• ValidatingWebhookConfiguration
• MutatingWebhookConfiguration
• CertificateSigningRequest

These are popular mechanisms for extending the platform and were used widely in many Operators, for example.

For Kubernetes admins, the first step to understanding the impact is to begin to observe and measure the usage of these APIs before they are removed. We’ll cover how OpenShift goes above and beyond to make this easy to do later in the post.

For vendors that produce Kubernetes based products, a plan needs to be put together to track which deprecations affect the supported versions of Kubernetes. The engineering team has been hard at work preparing for these changes. We’ve built tools to help with migrations and updated our documentation, hopefully in ways that will make these deprecations less painful.

Kubernetes v1.25 brings another large set of planned v1beta1 security and workload stability deprecations, including:

• PodSecurityPolicy
• PodDistruptionBudget
• RuntimeClass
• CronJobs

The worst place for a Kubernetes admin to be is choosing between an update to fix a security issue, but be held back by a third party tool that is still using a deprecated API. Kubernetes admins and vendors need to be cognizant of these upstream changes.

Later in the post we’ll see examples of how this has played out as OpenShift rolled out 4.9 which contained Kubernetes v1.22.

How OpenShift Handles Kubernetes Deprecations

Before diving into what we’re seeing across the OpenShift fleet, let’s understand how OpenShift handles Kubernetes deprecations. Overall, Red Hat strives to ship a given Kubernetes release in OpenShift three to four months after it is released. Historically, OpenShift has been very consistent at hitting this goal vs. other certified Kubernetes products.

Months between upstream and commercial release for certain products

This policy balances stability with powering innovation for our customers by shipping the latest features. Red Hat directs a large amount of development upstream, so it’s in our best interest to consume our own code down into OpenShift.

Planning for deprecations happens when our engineers are involved upstream as APIs are changing. When the Kubernetes release is out, we start communicating to both our customers and our partner community about changes they need to be aware of.

Red Hat worked with over 100 partners during the months before and after OpenShift 4.9’s general availability to help them understand how their Kubernetes API usage needed to change. This effort strengthens the entire Kubernetes community because partner software is certified for OpenShift but also impacts others.

How OpenShift Handles Other API Deprecations

OpenShift publishes an API Compatibility Guide and description of API tiers that govern their lifecycle and how they may be replaced over time. Here’s a small reproduction of that policy for illustration:

As you can see, all Kubernetes v1 “stable” APIs are covered under Tier 1, which means they will not break across the entirety of the OpenShift 4 lifecycle.

Kubernetes v1beta1 APIs are Tier 2, which is the policy that governs the APIs removals we have talked about in v1.22 and the upcoming v1.25 release. OpenShift 4.9 with Kubernetes v1.22 was beyond the 3 releases since the inclusion of Kubernetes v1.16 in OpenShift 4.3.

API Deprecation Tools for OpenShift Admins

Several tools are built into OpenShift to allow admins to observe API usage in aggregate, optionally alert on it and signal acknowledgement of zero usage of deprecated APIs before a cluster upgrade is started.

Tracking API Usage

The OpenShift API server is constantly keeping track of API request counts for all Kubernetes objects, including any deprecated APIs. You can view this in the CRD section of the OpenShift Console, or by using the CLI:

$ oc get apirequestcount.apiserver.openshift.io/validatingwebhookconfigurations.v1beta1.admissionregistration.k8s.io

NAME                              REMOVEDINRELEASE   CURRENTHOUR   LAST24H
validatingwebhookconfigurations   1.22               4             362

Usage is reported for the last hour, the last 24 hours, the request count and an indication if a future version will remove this API.

More advanced querying will even show you internal usage of these APIs from Kubernetes components like the Controller Manager. This is nothing for an OpenShift user to worry about because the cluster upgrade will modify this behavior to use the new APIs.

$ oc get apirequestcount ingresses.v1beta1.extensions -o jsonpath='{range .status.currentHour..byUser[*]}{..byVerb[*].verb}{","}{.username}{","}{.userAgent}{"\n"}{end}' | sort -k 2 -t, -u | column -t -s, -NVERBS,USERNAME,USERAGENT

VERBS  USERNAME                        USERAGENT
watch  system:kube-controller-manager  cluster-policy-controller/v0.0.0
watch  system:kube-controller-manager  kube-controller-manager/v1.21.6

Alerting on API Usage

OpenShift admins can alert on the same API request data with a built in alert. Below you can see an example of several APIs that are being removed soon and have current usage at the time of the screenshot.

An advanced implementation of this could be to send Slack messages to offending Service Accounts or Namespace owners as their software is triggering these alerts.

Example alert for deprecated API usage

Upgrade Acknowledgements

The last tool that OpenShift admins have is to acknowledge that API usage has ended before triggering an upgrade from a pre-deprecated version of Kubernetes to one that removes the APIs.

Prompting within the OpenShift Console

This is done by setting a boolean within a ConfigMap and is shown on the cluster settings screen in the Console.

apiVersion: v1
kind: ConfigMap
metadata:
  name: admin-acks
  namespace: openshift-cluster-version
data:
    ack-4.8-kube-122-api-removals-in-4.9: "true"

OpenShift will use this same mechanism for Kubernetes v1.25. Overall, Red Hat values a single-click upgrade experience and we do not want to require any sort of manual behavior unless it introduces possible impact to your applications. However, our customers frequently use CI/CD tools, security scanners, monitoring agents and other software that can expect certain behavior from the clusters they are connected to. This acknowledgment covers these cases where an admin understands that usage in a way that OpenShift itself cannot.

Evidence from the OpenShift Fleet

OpenShift 4.8 was released in August of 2021 and immediately after release OpenShift customers updated clusters and started to trigger usage of the APIs flagged for removal. This shows that all of the proactive API usage features are working as intended.

Usage of deprecated APIs detected in OpenShift clusters

As you can see the number is stabilizing and this indicates a few different things:

1. Clusters that are newly updated to OpenShift 4.8 are beginning to report deprecated API usage, causing the total to rise
2. Clusters running OpenShift 4.8 are reducing their deprecated API usage over time, causing the total to fall
3. Admins waiting as tools are slowly updated, causing the total to fall over time
4. Use of these APIs is increasing as folks onboard additional applications on their clusters while others are decreasing usage
5. Clusters are fully upgraded to OpenShift 4.9 and 4.10 and no longer have access to the deprecated APIs, causing the total to fall (to zero ultimately, in the future)

Deprecation = Success?

Across the entire Kubernetes community it’s too early to call this deprecation a success, but it seems to be trending in the correct direction. The policy was followed and notices were given years in advance of these changes, along with suitable v1 replacement APIs in most cases. There was significant adoption of the v1.22 v1beta1 APIs within the community but many projects updated their code bases to the v1 apis with minimal code changes required.

Within the OpenShift fleet, OpenShift 4.10 was just released, so the bulk of our user base is running a mix of 4.8 (v1.21) and 4.9 (v1.22), plus our Extended Update Support period for OpenShift 4.6 (v1.19), which predates these deprecations.

We are happy to see a flattening of the curve and hope that all of the features built into OpenShift make this a seamless process for our cluster administrators. Our partners also deserve praise for their tireless effort to match the progress happening in the Kubernetes community.