Each Red Hat OpenShift Dedicated (OSD) cluster requires a minimum annual base cluster purchase, but there are two billing options available for each cluster: Standard and Customer Cloud Subscription (CCS; previously known as Bring-Your-Own-Cloud or BYOC).
Standard OpenShift Dedicated clusters are deployed in to their own cloud infrastructure accounts, each owned by Red Hat. Red Hat is responsible for this account, and cloud infrastructure costs are paid directly by Red Hat. The customer will only pay the Red Hat subscription costs.
In the Customer Cloud Subscription model, the customer pays the cloud infrastructure provider directly for cloud costs and the cloud infrastructure account will be part of a customer’s Organization, with specific access granted to Red Hat. In this model, the customer will pay Red Hat for the CCS subscription and will pay the cloud provider for the cloud costs. It is the customer's responsibility to pre-purchase or provide Reserved Instance (RI) compute instances to ensure lower cloud infrastructure costs.
Additional resources may be purchased for an OpenShift Dedicated Cluster, including:
Customers can create, scale, and delete their clusters from OpenShift Cluster Manager (OCM), provided they've pre-purchased the necessary subscriptions.
Actions available in OpenShift Cluster Manager must not be directly performed from within the cluster as this may cause adverse affects, including, but not limited to, having all actions automatically reverted.
OpenShift Dedicated offers OpenShift Container Platform clusters as a managed service on the following cloud providers:
Single-AZ clusters require a minimum of 2 worker nodes for Customer Cloud Subscription (CCS) clusters deployed to a single availability zone. A minimum of 4 worker nodes is required for non-CCS clusters. These 4 worker nodes are included in the base subscription.
Multi-AZ clusters require a minimum of 3 worker nodes for Customer Cloud Subscription (CCS) clusters, 1 deployed to each of three availability zones. A minimum of 9 worker nodes is required for non-CCS clusters. These 9 worker nodes are included in the base subscription, and additional nodes must be purchased in multiples of three in order to maintain proper node distribution.
All OpenShift Dedicated clusters support a maximum of 180 worker nodes.
Note: The default Machine Pool node type/size cannot be changed once the cluster has been created.
Master and infrastructure nodes are also provided by Red Hat. There are at least 3 master nodes that handle etcd and API related workloads. There are at least 2 infrastructure nodes that handle metrics, routing, web console and other workloads. Master and infrastructure nodes are strictly for Red Hat workloads to operate the service, and customer workloads are not permitted to be deployed on these nodes.
Note: Approximately 1 vCPU core and 1 GiB of memory are reserved on each worker node and removed from allocatable resources. This is necessary in order to run processes required by the underlying platform. This includes but is not limited to system daemons like udev, kubelet, container runtime, etc. and also accounts for kernel reservations. OpenShift core systems such as audit log aggregation, metrics collection, DNS, image registry, SDN, etc. may consume additional allocatable resources in order to maintain the stability and maintainability of the cluster. The additional resources consumed may vary based on usage.
OpenShift Dedicated offers the following worker node types and sizes:
OpenShift Dedicated offers the following worker node types and sizes on Google Cloud chosen to have a common CPU and memory capacity as other cloud instance types:
The following AWS regions are currently supported by:
The following Google Cloud regions are currently supported:
Each new OSD cluster is installed within a dedicated Virtual Private Cloud (VPC) in a single Region, with the option to deploy into a single Availability Zone (Single-AZ) or across multiple Availability Zones (Multi-AZ). This provides cluster-level network and resource isolation, and enables cloud-provider VPC settings, such as VPN connections and VPC Peering. Persistent volumes are backed by cloud block storage and are specific to the AZ in which they are provisioned. Persistent volumes do not bind to a volume until the associated pod resource is assigned into a specific AZ in order to prevent unschedulable pods. AZ-specific resources are only usable by resources in the same AZ.
Note: The region and the choice of single or multi AZ cannot be changed once a cluster has been deployed.
Any SLAs for the service itself are defined in Appendix 4 of the Red Hat Enterprise Agreement Appendix 4 (Online Subscription Services).
You may not remove or replace any native OpenShift Dedicated components or any other component installed and managed by Red Hat. If using cluster administration rights, Red Hat is not responsible for any actions taken by you or any of your authorized users including but not limited to actions that may affect infrastructure services, service availability and data loss.
If any actions that affect infrastructure services, service availability or data loss are detected, Red Hat will notify the customer of such and request either that the action be reverted or to create a support case to work with Red Hat to remedy any issues.
OpenShift Dedicated includes Red Hat Premium Support, which can be accessed by using the Red Hat Customer Portal.
OpenShift Dedicated support SLAs can be found here.
Red Hat OpenShift Dedicated provides optional integrated log forwarding to Amazon (AWS) CloudWatch.
Cluster audit logs are available through AWS CloudWatch, if the integration is enabled. If the integration is not enabled, you can request the audit logs by opening a support case. Audit log requests must specify a date and time range not to exceed 21 days. When requesting audit logs, customers should be aware that audit logs are many GB per day in size.
Application logs sent to
STDOUT are collected by Fluentd and forwarded to AWS CloudWatch through the cluster logging stack, if it is installed.
OpenShift Dedicated clusters come with an integrated Prometheus/Grafana stack for cluster monitoring including CPU, memory, and network-based metrics. This is accessible via the web console and can also be used to view cluster-level status and capacity/usage through a Grafana dashboard. These metrics also allow for horizontal pod autoscaling based on CPU or memory metrics provided by an OpenShift Dedicated user.
Red Hat communicates the health and status of OSD clusters through a combination of a cluster dashboard available in the OpenShift Cluster Manager, and email notifications sent to the email address of the contact that originally deployed the cluster.
To use a custom hostname for a route, you must update your DNS provider by creating a canonical name (CNAME) record. Your CNAME record should map the OpenShift canonical router hostname to your custom domain. The OpenShift canonical router hostname is shown on the Route Details page after a Route is created. Or a wildcard CNAME record can be created once to route all subdomains for a given hostname to the cluster's router.
Custom domains and subdomains are not available for the platform service routes, e.g., the API or web console routes, or for the default application routes.
OpenShift Dedicated includes TLS security certificates needed for both internal and external services on the cluster. For external routes, there are two, separate TLS wildcard certificates that are provided and installed on each cluster, one for the web console and route default hostnames and the second for the API endpoint. Let’s Encrypt is the certificate authority used for certificates. Routes within the cluster, e.g., the internal API endpoint, use TLS certificates signed by the cluster's built-in certificate authority and require the CA bundle available in every pod for trusting the TLS certificate.
OpenShift Dedicated supports the use of custom certificate authorities to be trusted by builds when pulling images from an image registry.
OSD uses up to five different load balancers: an internal master load balancer, an external master load balancer, an external master load balancer that is only accessible from Red Hat-owned, whitelisted bastion hosts, one external router load balancer, and one internal router load balancer. Optional service-level load balancers may also be purchased to enable non-HTTP/SNI traffic and non-standard ports for services.
appsin the URL. The default load balancer can be configured in OCM to be either publicly accessible over the Internet, or only privately accessible over a pre-existing private connection. All application routes on the cluster are exposed on this default router load balancer, including cluster services such as the logging UI, metrics API, and registry.
apps2in the URL. The secondary load balancer can be configured in OCM to be either publicly accessible over the Internet, or only privately accessible over a pre-existing private connection. If a "Label match" is configured for this router load balancer, then only application routes matching this label will be exposed on this router load balancer, otherwise all application routes will also be exposed on this router load balancer.
For non-CCS OSD clusters, network usage is measured based on data transfer between inbound, VPC peering, VPN, and AZ traffic. On a non-CCS OSD base cluster, 12TB of network I/O is provided. Additional network I/O can be purchased in 12TB increments. For CCS OSD clusters, network usage is not monitored, and it billed directly by the cloud provider.
Project admins can add route annotations for many different purposes, including ingress control via IP whitelisting.
Ingress policies can also be changed by using NetworkPolicy objects, which leverage the ovs-networkpolicy plugin. This allows for full control over ingress network policy down to the pod level, including between pods on the same cluster and even in the same namespace.
All cluster ingress traffic will go through the defined load balancers. Direct access to all nodes is blocked by cloud configuration.
Pod egress traffic control via EgressNetworkPolicy objects can be used to prevent or limit outbound traffic in OpenShift Dedicated.
Public outbound traffic from the master and infrastructure nodes is required and is necessary to maintain cluster image security and cluster monitoring. This requires the 0.0.0.0/0 route to belong only to the internet gateway, it is not possible to route this range over private connections.
OpenShift 4 clusters use NAT Gateways to present a public, static IP for any public outbound traffic leaving the cluster. Each subnet a cluster is deployed into receives a distinct NAT Gateway. For clusters deployed on AWS with multiple availability zones, up to 3 unique static IP addresses can exist for cluster egress traffic. For clusters deployed on Google Cloud, regardless of availability zone topology, there will by 1 static IP address for worker node egress traffic. Any traffic that remains inside the cluster or does not go out to the public internet will not pass through the NAT Gateway and will have a source IP address belonging to the node that the traffic originated from. Node IP addresses are dynamic, and therefore a customer should not rely on whitelisting individual IP address when accessing private resources.
Customers can determine their public, static IP address(es) by running a pod on the cluster and then querying an external service. For example:
oc run ip-lookup --image=busybox -i -t --restart=Never --rm -- /bin/sh -c "/bin/nslookup -type=a myip.opendns.com resolver1.opendns.com | grep -E 'Address: [0-9.]+'"
OpenShift Dedicated allows for the configuration of private network connection through several cloud provider managed technologies:
No monitoring of these private network connections is provided by Red Hat SRE. Monitoring these connections is the responsibility of the customer.
For OpenShift Dedicated clusters that have a private cloud network configuration, a customer may specify internal DNS server(s) available on that private connection that should be queried for explicitly provided domains.
Master nodes use encrypted-at-rest EBS storage.
EBS volumes used for persistent volumes are encrypted-at-rest by default.
Persistent volumes are backed by block storage (AWS EBS and Google Cloud persistent disk), which is Read-Write-Once. On a non-CCS OSD base cluster, 100GB of block storage is provided for persistent volumes, which is dynamically provisioned and recycled based on application requests. Additional persistent storage can be purchased in 500GB increments.
Persistent volumes can only be attached to a single node at a time and are specific to the availability zone in which they were provisioned, but they can be attached to any node in the availability zone.
The AWS CSI Driver can be used to provide RWX support for OpenShift Dedicated on AWS. A community operator is provided to simplify setup.
⚠️ It is critical that customers have a backup plan for their applications and application data.
Application and application data backups are not a part of the OpenShift Dedicated service.
All Kubernetes objects in each OpenShift Dedicated cluster are backed up to facilitate a prompt recovery in the unlikely event that a cluster becomes irreparably inoperable.
The backups are stored in a secure object storage (Multi Availability Zone) bucket in the same account as the cluster.
Node root volumes are not backed up as Red Hat Enterprise Linux CoreOS is fully managed by the OpenShift Container Platform cluster and no stateful data should be stored on a node's root volume.
The following table shows the frequency of backups:
|Full object store backup||Daily at 0100 UTC||7 days||This is a full backup of all kubernetes objects. No PVs are backed up in this backup schedule.|
|Full object store backup||Weekly on Mondays at 0200 UTC||30 days||This is a full backup of all kubernetes objects. No PVs are backed up in this backup schedule.|
|Full object store backup||Hourly at 17 minutes past the hour||24 hours||This is a full backup of all kubernetes objects. No PVs are backed up in this backup schedule.|
Node autoscaling is not available on OpenShift Dedicated at this time.
Customers may create and run DaemonSets on OpenShift Dedicated. In order to restrict DaemonSets to only running on worker nodes, use the following nodeSelector:
In a multiple availability zone cluster, master nodes are distributed across AZs and at least three worker nodes are required in each AZ.
Custom node labels are created by Red Hat during node creation and cannot be changed on OpenShift Dedicated clusters at this time.
Refer to OpenShift Dedicated Life Cycle for more information on the upgrade policy and procedures.
Window containers are not available on OpenShift Dedicated at this time.
OpenShift Dedicated runs on OpenShift 4 and uses CRI-O as the only available container engine.
OpenShift Dedicated runs on OpenShift 4 and uses Red Hat Enterprise Linux CoreOS as the operating system for all master and worker nodes.
All Operators listed in the Operator Hub marketplace should be available for installation. Operators installed from OperatorHub, including Red Hat operators, are not SRE managed as part of the OpenShift Dedicated service. Refer to the Red Hat Customer Portal for more information on the supportability of a given operator.
Authentication for the cluster is configured as part of the OpenShift Cluster Manager cluster creation process. OpenShift is not an identity provider, and all access to the cluster must be managed by the customer as part of their integrated solution. Multiple identity providers provisioned at the same time is supported. The following identity providers are supported:
Privileged containers are not available by default on OSD. The "anyuid" and "nonroot" Security Context Constraints are available for dedicated-admins, and should address many use cases. Privileged containers are only available for cluster-admin users.
In addition to normal users, OpenShift Dedicated provides access to an OSD-specific Group called dedicated-admins. Any users on the cluster that are members of the dedicated-admins group:
For more specific information on the dedicated-admin role, please see https://docs.openshift.com/dedicated/4/administering_a_cluster/dedicated-admin-role.html.
As an administrator of an OpenShift Dedicated cluster with Customer Cloud Subscriptions (CCS), you have access to the cluster-admin role. While logged in to an account with the cluster-admin role, users have mostly unrestricted access to control and configure the cluster. There are some configurations that are blocked with webhooks to prevent destablizing the cluster, or because they are managed in OpenShift Cluster Manager (OCM) and any in-cluster changes would be overwritten.
For more information on the cluster-admin role, please see https://docs.openshift.com/dedicated/4/administering_a_cluster/cluster-admin-role.html.
All users, by default, have the ability to create, update, and delete their projects. This can be restricted if a member of the dedicated-admins group removes the self-provisioner role from authenticated users:
oc adm policy remove-cluster-role-from-group self-provisioner system:authenticated:oauth
This can be reverted by applying:
oc adm policy add-cluster-role-to-group self-provisioner system:authenticated:oauth
Refer to OpenShift Dedicated Process and Security Overview for the latest compliance information.
With OSD on AWS, AWS does provide a standard DDoS protection on all Load Balancers, called AWS Shield. This provides 95% protection against most commonly used level 3 and 4 attacks on all the public facing Load Balancers used for OpenShift Dedicated. We also add a timeout for http requests coming to our haproxy router of 10 secs within which to get response or connection is closed to provide additional protection.