March 13, 2018 | by John Goulding
Yesterday, the Kubernetes Product Security team released information about two significant bugs in Kubernetes, which were assigned CVE-2017-1002101 and CVE-2017-1002102. OpenShift is built upon Kubernetes and as such these bugs were also present in both OpenShift Online and OpenShift Dedicated. Red Hat, along with Google and other members of the Cloud Native Computing Foundation, worked to create and coordinate the release of security fixes for these affected products.
In response to these security errata, at the time the embargo was lifted, the OpenShift SRE team worked around the clock, across three geographic regions (NASA, APAC, and EMEA) to remediate the bug on all affected clusters.
Starting at approximately noon Eastern (16:00 UTC) on Monday, March 12th remediation began with internal and test clusters prior to any updates being made in production. The usual and customary tooling for updates had been modified ahead of time in response to a prior incident post-mortem that allowed it to handle the unusual nature of the patch. Instead of being applied as a system errata, for most clusters, the individual OpenShift components were upgraded in place and restarted. For a small number of starter tier clusters, an automated product upgrade was performed to remediate and upgrade simultaneously.
All externally exposed, production clusters were remediated by 12:30 Eastern (16:30 UTC) on Tuesday, March 13th. Due to the nature of the global SRE team, all OpenShift Dedicated clusters were patched during the customer’s preferred (typically overnight for that region) maintenance window.
A small number of nodes saw isolated outages as other issues came to light, but in the vast majority of cases, no reboots or node outages were required.
As is always the case, the focus was on patching public (starter- and pro- tiers in OpenShift Online) clusters ahead of non-public (all clusters in OpenShift Dedicated) due to the increased attack surface.
The remediation process was entirely automated, including raising and lowering customer notification banners. This ensured that even though the remediation was performed on an accelerated timeline, customers were always kept informed about the status and progress. Additionally, to aid collaboration between team members, removing and re-adding cluster nodes from our maintenance systems was automated, avoiding false alerts during the process.
Some of the key elements that created the ability to respond so quickly include factors like:
For more information, please refer to:
Red Hat OpenShift SRE Team
Categories
November 17, 2023
November 16, 2023
November 15, 2023