Configuring OpenShift Logging using LokiStack on ROSA and (soon) ARO

A guide to shipping logs and metrics on OpenShift using the new LokiStack setup. Recently, the default logging system with OpenShift swapped from ElasticSearch/FluentD/Kibana to a system based on LokiStack/Vector/OCP Console. LokiStack requires an object store in order to function, and this guide is designed to walk the user through the steps required to set this up.

Overview of the components of OpenShift Cluster Logging

Prerequisites

1. OpenShift CLI (oc)
2. Rights to install operators on the cluster
3. Access to create S3 buckets (AWS/ROSA), Blob Storage Container (Azure), Storage Bucket (GCP)

Setting up your environment for ROSA

1. Create environment variables to use later in this process by running the following commands:

1. Create the relevant AWS resources for LokiStack:

   1. Create a bucket for the LokiStack Operator to consume

2. Create a policy document for the LockStack Operator to consume

3. Create the IAM Access Policy by running the following command:

Create the LokiStack installation (OpenShift 4.14 or higher on AWS (ROSA))

1. Create an IAM Role trust policy document by running the following command:

2. Create an IAM Role to link the trust policy to the IAM Access Policy by running the following command:

Save this role_arn for installation of the lokistack operator later.

Create the LokiStack installation (OpenShift 4.13 or lower on AWS (ROSA) and non-STS clusters)

1. Create an IAM user that will allow your LokiStack to access the bucket using the following command:

2. Attach your policy to your new user using the following command:

3. Create an AWS Access key and Secret key for your IAM user using the following command:

You are ready to proceed to the next step

Install the OpenShift Cluster Logging Operator

1. Create a namespace for the OpenShift Logging Operator

2. Install the Loki Operator by creating the following objects, specifying the Role ARN we generated above:

   1. Verify Operator Installation

      Example Output

If you are using OpenShift 4.14 or higher on AWS (ROSA)

1. Create a secret for the LokiStack Operator to consume by running the following command:

2. Create a LokiStack installation by creating the following object:

If you are using OpenShift 4.13 or lower on AWS (ROSA), or are using a Non-STS cluster

1. Extract the AWS Access Key and Secret key from your variable created above using the following command:

2. Create a secret for the LokiStack Operator to consume by running the following command:

3. Create a LokiStack installation by creating the following object:

Configuring LokiStack

1. Confirm your LokiStack is running successfully by running the following command:

   Note: If you see pods in Pending state, confirm that you have sufficient resources in the cluster to run a LokiStack. If you are running a small cluster, try adding one or two m5.4xlarge machines to your cluster like so: rosa create machinepool -c ${CLUSTER_NAME} --name=lokistack-mp --replicas=2 --instance-type=m5.4xlarge . An overview of lokistack sizing can be found here: https://docs.openshift.com/rosa/observability/logging/log_storage/installing-log-storage.html#loki-deployment-sizing_installing-log-storage

2. Install the Red Hat OpenShift Logging Operator by creating the following objects:

   1. The Cluster Logging OperatorGroup

   2. Subscription Object to subscribe a Namespace to the Red Hat OpenShift Logging Operator

   3. Verify the Operator installation, the PHASE should be Succeeded

      Example Output

3. Create an OpenShift Logging instance, specifying the logStore:

   1. Here, we have configured the ClusterLogging Operator to use the existing LokiStack we have created in the cluster as it’s LogStorage. If you are using ElasticSearch as your LogStore, this would point at ElasticSearch (now deprecated)

4. Edit your OpenShift Logging instance, adding the collection section to create vector collection pods:

5. Confirm you can see collector pods starting up using the following command. There should be one per node.

   Example output:

6. Edit your OpenShift Logging instance, adding the visualisation section to show logs in the console:

   1. Ensure the Console Plugin is enabled by running the following command:

      Example output:

   2. Here, we have added the console-view-plugin to allow us to view logs in the OpenShift console. You can check that the pod has been created using the following command:

      Example output:

   3. Confirm you can see the Logging section of the console under the Observe tab:

      

At this point OpenShift logging is installed and configured and is ready to receive logs.

Install the ClusterLogForwarder Custom Resource

Separately from the ClusterLogging storage system, the OpenShift Cluster Logging Operator provides the ClusterLogForwarder which allows you to describe which log types are sent where. We will now configure this to collect logs from our cluster and forward them to our log store

1. Create a basic ClusterLogForwarder using the following command:

This example selects all infrastructure logs and forwards them to “default”, which is a reference to our LokiStack Logging Store. If we go to the Console and browse to Observer -> Logs, then change the dropdown from “application” to “infrastructure” we can now see logs:

1. Adjust your ClusterLogForwarder to pick up Application logs from a specific namespace by running the following command:

This example has created a new input, specifying the openshift-dns namespace, and forwarded it to our LogStore. If you refresh your Logging tag and select “application” in the drop down you will now see your logs.

For more examples or configuration options please see the documentation here: https://docs.openshift.com/rosa/observability/logging/log_collection_forwarding/configuring-log-forwarding.html

Cleanup

1. Remove the ClusterLogForwarder Instance:

2. Remove the ClusterLogging Instance:

3. Remove the LokiStack Instance:

4. Remove the Cluster Logging Operator:

5. Remove the LokiStack Operator:

6. Cleanup the openshift-logging namespace

7. Cleanup your AWS Bucket

8. Cleanup your AWS Policy

If you are using OpenShift 4.14 or higher on AWS (ROSA)

1. Cleanup your AWS Role

If you are using OpenShift 4.13 or lower on AWS (ROSA), or are using a Non-STS cluster

1. Cleanup your AWS user