Cost Management: Bean Counting for OpenShift

Per-department billing with the Cost Management service for Red Hat OpenShift

Introduction

In this article, we describe why we think the Cost Management service for Red Hat OpenShift is needed, what it is, and how to address a simple cross-department billing problem with the technology.

In June 2021, the CNCF and the FinOps Foundation recently published the results of a microsurvey into Kubernetes cost management that is still relevant.  This found that despite a prevailing need to prevent cost overruns through tracking and monitoring usage, customers don’t often have the right systems in place to do this.

Importance of cost management

When it comes to container management and day 2 operations, we are often asked by customers to help with tracking, and manage platform costs & software use.  

Most customers using platforms are keen to understand the infrastructure costs these platforms incur, and larger enterprises often use more detailed information to internally bill ("chargeback")

departments. Moreover, agile project teams particularly value the ability to visualize costs and help manage expenses.

We commonly see these sorts of situations. which lead us to recommend the cost management capability of Red Hat OpenShift:

• Per department billing: understanding costs that each department has incurred around the container platform and bill each department separately.
• Software tracking: tracking usage of software on the platform, mapping CPU/Memory usage to licenses or subscriptions.
• Cross-cloud provider billing: some enterprises run multiple OCP clusters across cloud providers, so varied cost models are needed, along with the ability to combine information sources into a single view.

Per-department billing

OpenShift clusters are generally multi-tenant - that is, they host multiple applications from multiple teams.  That makes sense, as this really is one of the sweet spots of containers and container orchestration.  As clusters are scaled up, we find that our customers often consolidate applications from lots of different locations into these clusters. For example, a customer might adopt OpenShift as part of a datacenter migration, and bring in workload from other container hosting solutions.

This presents them with a unique problem - in the past, the costs associated with each of these applications was likely tracked on a “per-VM” basis, or by spinning up a new container cluster per project.  Centralizing everything under one OpenShift banner simplifies operations by consolidating all those applications’ hardware, software and cost in one place but the crude per-VM billing they did before won’t work in the more fine-grained scheduling world provided by OpenShift.

The teams that are responsible for running an OpenShift platform, henceforth the “platform” team, will likely have to recover costs from project teams, so they need some way of measuring usage.  This allows them to distribute those costs accurately and fairly.

Software tracking

Customers who are adopting OpenShift will frequently run software on their clusters that has commercial terms attached.  If they’re lucky, it’ll be a flat-rate enterprise fee, but in the majority of cases they’ll be billed by CPU, or some other metric.  Red Hat’s middleware products, such as JBoss EAP, Fuse and AMQ, are a good example of this type of software - it is subscribed based on CPU deployments.

Quite apart from the commercial aspect, customers might also want to keep track of the trends within their organization when it comes to technology usage.

Getting a good handle on that information is critical - from both a commercial and skills point of view, decision makers need to understand current requirements, and forecast needs in the future.

Cost tracking across differing infrastructures

Some of Red Hat’s customers, particularly those who operate in regulated industries, run several OCP clusters across several providers: e.g. AWS, Azure, Google Cloud Platform, or on-premises.  They need to have a unified way of measuring usage and cost, whilst still taking into consideration any individual platform’s charging rates when building their charging model. This can be achieved using the Cost Management service for Red Hat OpenShift.

What is the Cost Management service for Red Hat OpenShift?

The Cost Management service for Red Hat OpenShift (henceforth referred to as Cost Management) allows monitoring of workloads on cloud providers.  It is based on the open source Project Koku, and supports, among other capabilities, breaking down the costs of running a container platform, and the workload upon it.

This is provided as a software as a service offering, included as part of the Red Hat OpenShift platform, and can be accessed on the Red Hat cloud console.

Visualization of the Overview page in Cost Management for OpenShift

Cost Management allows us to visualize and allocate costs, and from that, we can start to influence behaviors - this allows us to start addressing the cost overrun problem we identified earlier.

To set up Cost Management, the platform administrator will install the Cost Management operator on his cluster(s).   After that, the cluster will show up in the Red Hat cloud console.  The Red Hat cloud console has a web interface, and also an API to allow access to raw numbers and reports.

This functionality is also available for non-connected (air gapped) clusters with an interim user uploading the data to Red Hat, rather than allowing direct connectivity.

Cost Management has been generally available since May 2020, and will work on any connected OpenShift cluster after version 4.3.

How does Cost Management address our use cases?

Each cluster is assigned a cost model to map metrics and bill information to costs. By default the software extracts information from the underlying cloud providers’ costs.  For on-premises systems, custom cost models can be created, based on factors such as CPU and memory.

These models can be augmented with custom markups for any overhead (for example, IT operations cost or subscriptions cost) for CPU, memory, nodes, clusters, storage, or persistent volume claims.  These markups can be on a relative, percentage-based overhead, or fixed cost basis.

By combining these two accounting models, we can use Cost Management to reflect not only the infrastructure cost, but the real cost of providing a service to project teams.

Once we’ve created a cost model for our cluster and we have the raw data coming into the system, we can start to interpret and explore it.  Cost Management gives us a number of ways to do this, allowing us to browse and filter historical time-series data based on any number of user-defined filters and axes, as well as forecast usage out to the end of the month.

By labeling our workload with department tracking information, we can address our per-department billing use case.  Cost Management can combine data from multiple clusters (and cost models) into a single view of cost, addressing our cross-platform billing use case.  While we can approximate software tracking by using the console APIs, new functionality to make this easier to work with is being explored.

We can use Cost Management to address our cross-department billing use case.

OpenShift Environment Setup

We’ve set up an example cluster to report data to the Cost Management service for Red Hat OpenShift.  We followed the instructions in the product documentation here:

1. Adding an OpenShift Container Platform source to cost management
2. Setting up a cost model for the Cost Management Service

We’ve created two projects to demonstrate per-department billing.

Cost Management monitors usage based on “tags”.  It understands different implementations of the “tagging” concept based on the underlying platform’s capabilities - when monitoring OpenShift, it examines Kubernetes labels.

In this scenario, we’ve added labels to the project to denote which department they belong to.

To avoid overwhelming the Cost Management users with superfluous tags, there’s an extra step that’s required to elect tags as relevant to users.  Thus, we’ve added the keys for these labels’ key/value pairs to the monitored tags list within the Cost Management interface, by enabling tag grouping.

For more information on tagging as part of Cost Management, visit Managing cost data using tagging.

In line with the above, we’ve created two projects, and labeled them to demonstrate per-department billing:

• Europe
• Tagged with `costmanagement_tracking_department=cs`.
• America 
• Tagged with `costmanagement_tracking_department=sales`

Additionally, we’ve labeled pods with costmanagement_tracking_software=X, denoting the software used within the pod.  We can use this to track software usage across the platform.

America: the project is labeled with costmanagement_tracking_department=sales, representing the sales department.

Europe: the project is labelled with costmanagement_tracking_department=cs, representing the customer success department.

We’ve enabled the following tag groups to allow us to visualize costs:

• costmanagement_tracking_department
• costmanagement_tracking_software

We can now use the Cost Explorer feature of the cost management service to summarize usage per-department:

Data from multiple projects with the same costmanagement_tracking_department label will be combined in this view.  A department lead could also use this interface to drill down and understand how their department’s costs were being used on a per-project basis.

This is not necessarily useful for those wanting to automate report creation. For them, a more suitable  way of managing and extracting this data would be to use the cost management service’s API.  For example, the output below is the result of calling the API on ``/api/cost-management/v1/reports/openshift/compute/?filter[tag:costmanagement_tracking_department]=cs&filter[resolution]=daily&filter[time_scope_units]=day&filter[time_scope_value]=-90`

This shows a daily breakdown of the last 90 days’ usage for the “cs” department.  If we want to extract the data in a different way, we can find detailed API documentation for this service here.

Alternatively, we can download CSV data from the Cost Explorer console, aggregated either monthly or daily.

Next steps

Setting up Cost Management for Red Hat OpenShift is a simple operation, requiring minimal information.

We suggest that you start with a simple use case - by building a small proof of concept, you can see the sorts of information that the system can reflect, and what sort of granularity this can be provided at.

Once that’s done, you can build a taxonomy of labels that addresses what you need to get out of the system.  That might just be something simple, like cross-department or project team chargeback reports, or it could be something more complex, like software usage trends.

You should bear in mind that the system can’t “backdate” labels.  For example, if you add a label to a project which assigns it to a department, that project will only “count” after the label has been applied.

Conclusion

We showed how to use the Cost Management service for OpenShift to track the real platform costs incurred by business departments across a hypothetical enterprise, and download reports that can be used by a platform or accounting team, perhaps to recover costs.

Further reading

• Red Hat OpenShift on redhat.com
• Product documentation for the Cost Management service for Red Hat OpenShift