Time Is On Your Side: A Change to the OpenShift 4 Lifecycle

Red Hat is changing how long a minor number release (4.y) of OpenShift 4 will be supported. We are changing from our current version-based lifecycle policy to a timeline-based lifecycle of 18 months for all minor releases of OpenShift 4. This change will take effect with Red Hat OpenShift Container Platform version 4.7 and higher. Red Hat will also make all even numbers of OpenShift 4 an extended update support (EUS) release starting with OpenShift Container Platform 4.6 and higher. That means support periods will be extended for OpenShift releases 4.6, 4.8, 4.10, 4.12 and beyond.

A brief history on how we got to this topic. Starting in OpenShift 4, Red Hat moved away from a calendar driven lifecycle where we would declare a start date and end date for support when the software was released. At that time, we moved to a “rolling window,” which many in the application platform or orchestration industry have made the norm.  

Many vendors have moved to this “rolling window,” which is just an easy way of saying they will support some number of releases at any given time. With this system, the occurrence of a new release will push the oldest supported release out of support. OpenShift 4 was supporting 3 releases (current release minus 2 releases back) since OpenShift 4 came out in 2019. On average, depending on the cadence of new releases, that meant a release was supported for about 11-12 months. We also added a concept of an extended update support (EUS) release which was supported for 18 months. We have only announced OpenShift 4.6 as an EUS release up until now.

You may have heard Red Hat talking about our remote health program; OpenShift 4 clusters that have opted in are sending back upgrade and health metrics to Red Hat for analysis. We have used that connection with our customers to understand patterns regarding how they are upgrading their clusters across sample sizes in the thousands. We’ve listened to many of you about what you want to achieve from a product lifecycle standpoint, and read hundreds of request for proposals (RFPs) in the last year from all industry sectors. From all of those customer interactions, we have learned a few things and have decided to make some changes based on your feedback. Here are our lessons learned from releasing OpenShift 4 for almost 3 years. This information will help you understand our new lifecycle.

Don’t Force a Version Number

Users should be able to select what versions of OpenShift they want to remain on based on the version’s features. We found the opposite was happening when we released news of OpenShift 4.6 being supported longer than other minor releases. Users started forcing OpenShift 4.6 into their deployment projects. Selecting OpenShift 4.6 when OpenShift 4.7, 4.8, or 4.9 would have been a better feature match for their use cases. Red Hat needed to help them experience the best version of the product for them. Our changes today help return that freedom to them by offering more longer term support, more consistently across releases.

12 months is good, if it only started on month 1

Many customers have Dev/Test//Prod environments for the roll out of a new version of the OpenShift platform. They are installing OpenShift in highly regulated and controlled environments. In some cases there is a chain of vendors involved around the customer, all of whom must have their time with the software before handing the stack over to yet another vendor. Their CI/CD process takes on average one business quarter, but depending on when a new release hits during a business quarter that time can be +/- two months. So the safest or most conservative estimate of time for this software intake is six months. If the rolling window method was producing an average of 12 months (which is what the upstream Kubernetes ecosystem supports a Kubernetes release for as well) they were only getting 6 months of utility out of the release (12-6=6). Red Hat needed to figure out how to give them a solid 12 months on a release. The majority of customers can perform at least one upgrade of the platform per year by leveraging the new over the air upgrade flows found in OpenShift 4. Considering all these factors, we believe 18 months will solve this issue.

APIs Determine the Length of Support

To determine how long we could support a release of OpenShift, we looked at Kubernetes and the CNCF open source software projects. When analyzing lifecycle duration, the most critical factor a vendor must consider is what happens if a customer stays on a release as long as they can. For example, what if you stay on a release of OpenShift (and therefore Kubernetes) for 18 months? Would the APIs inside that release make it through an upgrade across all the releases between the one you are on and the one you want to get to when the time comes? The upstream is declaring 12 months to be the maximum. By being cautious about what APIs we expose as supported in OpenShift, we have determined that the longest we can safely extend support to our customers is 18 months. As Kubernetes matures from its current versions in the mid 1.20s, we might be able to extend this support range when building OCP off of the 1.3X Kubernetes versions. We know we are not there yet as an open source community and we do not want to do anything that would place our customers in a situation there is no way out of due to API longevity.

Code Changes Need to Follow a Time Maturity Schedule

Red Hat has thousands of customers on OpenShift 4 releases. We need to make sure when we introduce code changes for one customer we are not disrupting another customer. This aspect of the OpenShift 4 lifecycle is one we are not changing. Just like OpenShift 4 always had lifecycle phases such as “full support” and “maintenance” within a minor release, we will continue to do so in our new lifecycle. Red Hat will offer a 3 month overlap of full support on the current release once a new release is GA’d into our fast channel. Since we are releasing OpenShift starting in 2022 three times per year, that means the average time a release will be fully supported is 7 months. Full support is all the stuff you care about like critical bug fixes and critical security fixes, but it also includes less critical bugs and requests for enhancements. For the rest of the 18 months (18-7=11 months) we will offer maintenance support. Maintenance support is for critical bugs and critical security fixes. This allows there to be greater code changes at the beginning of a release’s cycle before it enters into its long tail of more cautious code changes in the maintenance phase. Please visit the OpenShift lifecycle page for more information on these full and maintenance support phases.

Only thing worse than no patches, is too many patches.

At 18 months for all minor releases (OpenShift 4.y), it means at any given time Red Hat will be maintaining the code branches for 6 releases (counting the current release under development).

It would not be healthy for us or our consumers to have patches coming out at the same cadence for all 6 of those releases.  For OpenShift 3 we released patches about once a month, with the exception of critical bugs and security issues which necessitated a quicker release. When we moved to OpenShift 4, we released patches once a week allowing customers to experience something closer to a continuous delivery of code.

As we look to support 6 versions of OpenShift for customers, we have decided to blend our experiences from OpenShift 3 and OpenShift 4. For the development branch (what you may know as nightlies), the current GA version, and one release older than the current, Red Hat will continue to release patches at a frequency of 1 per week. For the 3rd, 4th, and 5th oldest releases of OpenShift 4 we will release patches at a frequency of 1 per month. As with all things related to bugs there will be exceptions that cause us to release things sooner, but these will be the averages we want our users to understand and plan for based on the release they are on.

This is for core OpenShift. Not Everything that runs on it. That is a good thing.

Whenever you lengthen the lifecycle of a piece of software, you take from the innovation cycle. There is no model where supporting something longer does not take from doing more features in a new release. That is the rule of technical debt. The OpenShift ecosystem is very large. We have Serverless, Service Mesh, GitOps, Pipelines, IBM Cloud Paks, Red Hat Middleware, Data Foundation, OpenShift Virtualization and a 3rd party content ecosystem from over 150 certified ISVs running on top of OpenShift. We also run OpenShift as a managed service on Amazon, Microsoft Azure, and Google Cloud. Each one of these solutions represents targeted use cases that deserve to have the independence of making the best choice for their end users.

Each layered solution will continue to maintain its own lifecycle. A lifecycle for a specific service needs to be tailored to a specific market’s need that caused the solution to exist in the first place. Lifecycle can’t be forced upon a solution without causing undesirable outcomes. For Red Hat provided content we will ensure there is always a combination of versions that is supported on at least one version OpenShift. Many times a layered offering of OpenShift will not support 18 months. They will have a shorter lifecycle than OpenShift.  This will mean you might need to upgrade the layered solution or content while you leave the core platform alone for up to 18 months or it might mean you cannot stay on the older but still active releases of OpenShift if you are interested in a layer offering with a shorter lifecycle. As a platform for other solutions to run on, we wanted to make sure the platform itself (OpenShift) was not the limiting factor in the overall lifecycle. Please visit the OpenShift life cycle page for information on other Red Hat layered products’ lifecycles.

If all releases are 18 months, why keep the concept of an EUS.

Red Hat has decided to keep the concept of an EUS release alive for two primary reasons. We will continue to evaluate this decision if it causes too much confusion. It is our hope the Kubernetes upstream will evolve over the course of the next several years to allow us the technical ability to safely offer more than 18 months in the future. When we go past 18 months, we will only do so for specific releases. For example, moving forward we will declare even number releases (4.6, 4.8, 4.10, 4.12 ..) to be EUS releases. Should we ever have an ability to offer more than 18 months, it will be on a distant even number release. This is not a change we expect to see in the next 2 years.

The second reason we continue with the EUS naming scheme has to do with upgrades. As you know, it is not possible to skip levels of Kubernetes (i.e., Kubernetes 1.22 upgrading to Kubernetes 1.24) when you upgrade. Upgrades must be done serially. Red Hat is introducing a user experience wherein you can pause the upgrade process on the worker nodes at the beginning of an upgrade and run through 2 versions of OpenShift, then unpause them. You will still be serially upgrading the control plane, resulting in one fewer worker node reboot during the upgrade. Red Hat will be qualifying these upgrade paths between even number releases starting with OpenShift 4.8. For example, going from 4.8 to 4.10 or 4.10 to 4.12. In order to keep the test matrix size manageable for both us and our lISV vendors, we will focus this streamlined worker upgrade process only between the EUS releases (the even numbered releases).

Lastly, customer feedback on the EUS program was that it should be included in all Red Hat subscriptions (both Standard and Premium). Customers often purchase premium support subscriptions for one business reason and then standard support subscriptions for other use cases. The fact these subscriptions resulted in different lifecycles meant customers could not blend the two service levels to get what they wanted. For example, maybe test was on standard and production was on premium, but the result ended up being test might not be able to represent production depending on where the clusters were in their lifecycles. And so Red Hat has decided to make OpenShift 4 EUS be available to both standard and premium support subscriptions for OpenShift.

You spoke. We delivered.

In conclusion, a software vendor does not change the lifecycle during a major number (OpenShift 4) release very often. For the many people we have made happy with this life cycle change, there will be those that liked the current process we are replacing because it forced them to have a reason outside of their business to move faster. These people can still move fast. We are not falling behind the upstream Kubernetes with this lifecycle change. We will continue to release OpenShift one quarter behind the upstream as we have done for years. In 2022, upstream Kubernetes will move from 4 releases per year to 3 releases per year. This will in return mean OpenShift will release 3 minor versions of OpenShift 4 in 2022.

All indications of remote health telemetry, surveys and market investigations indicate that an 18 month lifecycle with a known start and stop date is what the majority of our production customers have wanted from Red Hat for OpenShift 4. You spoke and we delivered. Our hope is with the background information contained in this blog, you have a better understanding of not only what changed, but why it changed. And please keep talking to us. We are listening.