This document provides a summary of the quay.io incidents that occurred on May 19 and May 28, 2020. This document describes the nature of the incidents, then provides information about steps being taken to reduce the likelihood of similar outages in the future.
Incidents Summary
Red Hat's quay.io container registry service experienced two periods of degraded performance and availability in May 2020. The first incident occurred from approximately 7 a.m. May 19, 2020, to 1 a.m. May 20, 2020, UTC. The second incident occurred from approximately 11 a.m. to 4 p.m. May 28, 2020, UTC.
During these periods, users experienced a range of outcomes, including slow container image access times and inability to retrieve container images. These issues affected several other Red Hat services, including OpenShift Cluster Manager, which is used to deploy and manage OpenShift clusters.
Red Hat Site Reliability Engineering teams concluded several factors combined to form the root cause of these incidents. These factors include:
- non-optimal quay.io tuning in the areas of process parallelization and database access
- traffic surges from simultaneous OpenShift platform upgrades
Following these two incidents, several actions have been defined to improve quay.io availability, reliability, and continuity. Some of these items are already complete, others are being actively worked on, and some are being researched. They are described in the sections below.
Completed Actions
A number of actions have already been completed to address these two incidents. These actions include:
- Redeploying quay.io on a 4.3.18 OpenShift Dedicated cluster.
- Indefinitely disabling garbage collection to reduce database load.
- Optimizing several aspects of quay.io in the areas of process parallelization and database access. These optimizations will prevent the database from being driven to the point of lockup.
- Doubling the underlying quay.io database size/capacity to handle more traffic.
- Suspending certain OpenShift Dedicated z-stream updates to prevent potential traffic spikes during OpenShift Dedicated upgrades.
In addition to these completed actions, many others are pending, and are described below.
Short-term Actions
The following actions are in progress and are targeted for completion by June 11, 2020:
- Creating database “read-replicas” that will distribute database load and prevent database lockups
- Appreciably accelerating the quay.io redeployment process for faster incident response
- Creating a quay.io hot standby in a separate region
- Adding pod caching for faster quay.io restart time
- Improving monitoring for more visibility into quay.io health and the ability to detect potential problems
Long-term Actions
The following actions have been proposed and are under consideration:
- Investigating potential exacerbating networking issues
- Conducting substantial load and performance analysis to identify additional parallelization/database optimizations
- Reducing quay.io logging verbosity
- Investigating additional caching methods
- Investigating per-account rate-limiting
- Upgrading/replacing various database components and sub-systems
- Implementing numerous process and documentation improvements
About the author
More like this
Browse by channel
Automation
The latest on IT automation that spans tech, teams, and environments
Artificial intelligence
Explore the platforms and partners building a faster path for AI
Open hybrid cloud
Explore how we build a more flexible future with hybrid cloud
Security
Explore how we reduce risks across environments and technologies
Edge computing
Updates on the solutions that simplify infrastructure at the edge
Infrastructure
Stay up to date on the world’s leading enterprise Linux platform
Applications
The latest on our solutions to the toughest application challenges
Original shows
Entertaining stories from the makers and leaders in enterprise tech
Products
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Cloud services
- See all products
Tools
- Training and certification
- My account
- Developer resources
- Customer support
- Red Hat value calculator
- Red Hat Ecosystem Catalog
- Find a partner
Try, buy, & sell
Communicate
About Red Hat
We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.
Select a language
Red Hat legal and privacy links
- About Red Hat
- Jobs
- Events
- Locations
- Contact Red Hat
- Red Hat Blog
- Diversity, equity, and inclusion
- Cool Stuff Store
- Red Hat Summit