162930444-56fc18ee-90cc-464a-b38e-74166c0019ad

In this blog, I will show how to synchronize and replicate data in an application that uses the Galera cluster backend.

Galera is a technology developed specifically for MariaDB that provides a highly available replicating cluster. When I store and manage my data in a MariaDB database, it is in high priority to get high availability and replication. Galera allows us to have multiple nodes in multiple regions; thus, even if one region is down, our data will still be able to communicate.

Just to clarify, MariaDB, which is built and based on MySQL, is easy to manage, offers high performance, and provides scalability and expansion options with Galera. And best of all, it is all open source.

To see the great value in high availability, I will use a DNS server to simulate the application.

One of the options for DNS server software is PowerDNS, an open source authoritative DNS server that provides a robust set of features and added performance suited for larger server environments. PowerDNS has the option to store data in a database, and in the following steps, I will use the Galera cluster for that.

Now, all mentioned above will be visible on the platform as capabilities for virtual machines (VMs) and high availability in OpenShift Virtualization.

What is OpenShift Virtualization?

OpenShift Virtualization lets developers and administrators bring VMs, as they are, into Kubernetes and the containerized world. Once there, they can run multi-tier workloads in a single, declarative environment. This allows developers to focus on the containerized parts of the workload first, without slowing the modernization process while trying to figure out what to do with applications that are still running in VMs. With OpenShift Virtualization, containers and VMs can be managed using the same cloud native tools and processes.

OpenShift Virtualization allows organizations to work with a single modern platform simplifying their management and reducing costs. For more information, click here.

Rather than relying only on the captures, I suggest that you read the written instructions. The purpose of the captures is primarily to provide a visual aid for you and to make sure you are on the right page at each step.

I will use OpenShift virtualization to run VMs with MariaDB and PowerDNS. The following diagram shows the environment setup that needs to be a prerequisite.

The setup uses five VMs. Two VMs (mariadb-0 and mariadb-1) will host the Glare cluster, and two separate VMs (power-dns-0 and powser-dns-1) will act as DNS servers. The fifth VM is a workstation.

162931463-8221b7ac-de23-44cc-b0df-80b7982d2432

Create a Service for Galera Cluster

  1. It was necessary to create a service for each VM that is going to join our Galera cluster. Galera cluster requires network connectivity between the nodes with many ports. For this reason, a cluster-internal IP (“ClusterIP”) is allocated for each VM, which enables the VMs to communicate with each other. Here is an overview of the ports and their uses:
  • 3306 is the default port for MySQL client connections and state snapshot transfer using MySQL dump for backups.
  • 4567 is reserved for Galera cluster replication traffic. Multicast replication uses both TCP and UDP transport on this port.
  • 4568 is the port for incremental state transfer.
  • 4444 is used for all other state snapshot transfers.

NOTE: I used the galeracluster website to obtain this information. For more information, click here.

For example, a yaml configuration for the first VM in my project (mariadb-0).

apiVersion: v1
kind: Service
metadata:
name: <service name>
namespace: <namespace>
spec:
selector:
kubevirt.io/domain: <VM name>
ports:
- protocol: TCP
name: tcp-3306
port: 3306
targetPort: 3306
- protocol: TCP
name: tcp-4567
port: 4567
targetPort: 4567
- protocol: TCP
name: tcp-4568
port: 4568
targetPort: 4568
- protocol: TCP
name: tcp-4444
port: 4444
targetPort: 4444
- protocol: UDP
name: udp-4567
port: 4567
targetPort: 4567

Firewall Rules for MariaDB

  1. Opening these ports via the firewall for each VM that is going to join our Galera cluster. This step allows us to access the ports.
sudo firewall-cmd --permanent --zone=public --add-port=<port>/<tcp or udp>
  1. Reload the firewall to apply the changes.
sudo firewall-cmd --reload

Create Galera Cluster

Install and Configure the MariaDB First Instance

The MariaDB first instance is the most important instance at installation. This instance will essentially be the “primary” in our cluster. Without this instance, nothing can be started, and the cluster cannot be created. From this instance all other instances will launch, connect to and sync up with.

  1. Install the actual MariaDB and Galera packages.
sudo dnf module install mariadb/galera
  1. Configure the MariaDB instance.
sudo vi /etc/my.cnf
[galera]
wsrep_on=ON
wsrep_cluster_name=<'galera_cluster’>
binlog_format=ROW
bind-address=0.0.0.0

default-storage-engine=InnoDB
innodb_autoinc_lock_mode=2
innodb_doublewrite=1
query_cache_size=0
wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so

wsrep_cluster_address=gcomm://

wsrep_sst_method=rsync
wsrep_dirty_reads=ON
wsrep-sync-wait=0

wsrep_node_address=<'mariadb-0-ports.galera-cluster.svc.cluster.local'>

!includedir /etc/my.cnf.d

  • Enter the cluster name
  • Enter the IP address or service that allocate to the first VM
  1. Start the mariaDB service
sudo systemctl start mariadb
  1. Check the service status is active
sudo systemctl status mariadb
  1. Create a root password for MariaDB (optional)
mysql_secure_installation
  1. Stop the MariaDB service and run galera command. This command will start the service automatically
systemctl stop mariadb
galera_new_cluster

Connect to MariaDB and Check the Cluster Size

This step is important for make sure the cluster has been created properly and without any problem

  1. Run the following command to find out the size of the cluster
mysql -u root -p < > -e “SHOW STATUS LIKE 'wsrep_cluster_size';”

This is the first instance of the cluster, so it will have a cluster size of one.

162934206-886794f4-e942-4fee-8491-6ed9ef473330

Install and Configure the MariaDB Second Instance

The second instance is the instance that essentially creates high availability. This instance is a second instance that can be written to, read from, and acts like a normal DB. This instance connects to the first MariaDB instance.

  1. Install the actual MariaDB and Galera packages
sudo dnf module install mariadb/galera
  1. Configure the MariaDB instance
sudo vi /etc/my.cnf
[galera]
wsrep_on=ON
wsrep_cluster_name=<'galera_cluster'>
binlog_format=ROW
bind-address=0.0.0.0

default-storage-engine=InnoDB
innodb_autoinc_lock_mode=2
innodb_doublewrite=1
query_cache_size=0
wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so

wsrep_cluster_address=gcomm://<mariadb-0-ports.galera-cluster.svc.cluster.local,mariadb-1-ports.galera-cluster.svc.cluster.local>

wsrep_provider_options="ist.recv_bind=10.0.2.2"


wsrep_sst_method=rsync
wsrep_dirty_reads=ON
wsrep-sync-wait=0

wsrep_node_address=<'mariadb-1-ports.galera-cluster.svc.cluster.local'>

!includedir /etc/my.cnf.d
  • Enter the cluster name.
  • Enter the IP address or service for each VM that is going to join our Galera cluster.
  • Enter the Internal IP address, In OpenShift Virtualization, is always 10.0.2.2.
  • Enter the IP address or service that allocate to the second VM.
  1. Start the MariaDB service
sudo systemctl start mariadb
  1. Check the service status
sudo systemctl status mariadb
  1. As with the first instance, create a root password for MariaDB (optional)

Create the Cluster

Return to the first instance and check the cluster size. The size changed to two; that is, the cluster has two instances and thus is highly available for our database. This is important to make sure that the cluster has indeed increased and that our second instance has joined properly. This step also signifies the ability to write and read from both maria-0 and maria-1. When both instances join the cluster, they both can be accessed equally, and the replication is done automatically.

  1. Connect to MariaDB
mysql -u root -p 
  1. Run the following command to find out the size of the cluster
SHOW STATUS LIKE ‘wsrep_cluster_size’;

162935063-8149a386-570c-499f-a3c4-634058ac95bd

Setup the Cluster replication

  1. Connect to MariaDB from the first instance (mariadb-0).
mysql -u root -p 
  1. Run the following command to create a database in the first instance (mariadb-0).
create database powerdns;
  1. Run the following command to check the databases in the first instance (mariadb-0).
show databases;

162935331-7d3674c6-469e-4b8a-b900-90f4053f79c9

  1. Connect to the second instance (mariadb-1) and check if the database is synchronized. As seen the ‘powerdns’ database is shown on both instances and thus replicated. This step shows that the data has indeed been replicated and thus any data that will be created on one of the instances will replicate to the other one.

  2. Run the following command to check if the powerdns database exists in the second instance (mariadb-1)

mysql -u root -p < > -e “show databases;”

Create a User for the PowerDNS Server

Allowing access to our database remotely is important due to it being a separate instance from our PowerDNS. Creating a database user and allowing it access.

  1. Connect to the mariadb-0 instance and create a user by the following command.
create user <’username’> identified by <’password’>;  
  1. Permit the user that was created to access the instance remotely.
grant all privileges on powerdns.* to <’username’>@’%’ identified by <'password'>;
flush privileges;

Install and Configure the PowerDNS

  1. Connect to the PowerDNS instances (powerdns-0 and powerdns-1) and Install the actual PowerDNS packages.
dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
yum-config-manager --disable epel
dnf install --enablerepo=epel -y pdns pdns-backend-mysql

NOTE: If SElinux mode is enforced, create a SELinux policy that will allow that.

  1. Copy the structure of the tables for the PowerDNS database from one PowerDNS instance.
cat /usr/share/doc/pdns/schema.mysql.sql
  1. Connect to MariaDB from the first instance (mariadb-0).
mysql -u root -p 
  1. Create in mariadb-0 instance the structure of the tables for the PowerDNS database by running the following MySQL queries below.
use powerdns;
  1. Paste the schema from the ‘/usr/share/doc/pdns/schema.mysql.sql’ file.
  2. Create a label and allocate it to each mariadb instance for example mariadb1

162936296-e9001e35-5665-4dc0-baca-8058f015e320

  1. Create a service with the ports that MariaDB needed, like I did in the previous steps, and use the label form step 6
apiVersion: v1
kind: Service
metadata:
name: <service name>
namespace: <namespace>
spec:
selector:
< label >
………

162936581-b41e5217-85d8-4781-b043-6426ab173232

  1. Return to the powerdns-0 instance and configure the instance.
sudo vi /etc/pdns/pdns.conf
launch=gmysql
gmysql-host=<mariadb-ports.galera-cluster.svc.cluster.local>
gmysql-dbname=powerdns
gmysql-user=<pdns>
gmysql-password=<qwe123>
api=yes
api-key=dbpass
webserver-address=127.0.0.1
logging-facility=0
loglevel=5
log-dns-queries=yes
resolver=[::1]:53
expand-alias=yes
  • Enter the IP address or service for one of the Galera cluster member
  • Enter a user with privilege in the database
  • Enter a password for the user
  1. Start pdns service int the powerdns-0 instance
sudo systemctl start pdns
  1. Create a DNS zone and record.
sudo pdnsutil create-zone <domain_name>
sudo pdnsutil add-record <domain_name> <record> A <ip_address>

162937618-8d596cef-82ec-4693-bc9c-1a3aacd63066

  1. Once the DNS zone is created the record will be stored in our database. In this step check the record data on the first instance (mariadb-0).
mysql -u root -p
use powerdns;
select  * from domains; 
select * from records;

162937821-602c3acc-4873-492f-a315-4cc91a0cbb0b

  1. Repeat the command to check the record data on the second instance (mariadb-1). This step shows that the data has indeed been replicated and thus any data that will be created on one of the instances will replicate to the other one.

What is the value of high availability?

In this section of my blog, I want us to understand the importance and value that the high availability we have created gives us. As we saw at DB steps, the data replicated and synchronized, and the PowerDNS instances configured works with the MariaDB instances. I will stop one of the mariadb instances to simulate a failure, and we will see that our service remains available and prevents downtime to the customer.

  1. Create a label and allocate it for each PowerDNS instance, for example, powerdns-1.

162937989-6f2567ec-bc38-45a9-be7a-41e57d0fafb9

  1. Create a DNS service for PowerDNS instances
apiVersion: v1
kind: Service
metadata:
name: <service name>
namespace: <namespace>
spec:
selector:
< label >
ports:
- protocol: UDP
name: udp-53
port: 53
targetPort: 53
  1. Connect to the powerdns-1 instance and configure the instance with the same example configuration file for powedns-0 in the previous step.

  2. Start pdns service in the powerdns-1 instance

sudo systemctl start pdns
  1. The DNS zone and record that I created before are automatically replicated and synced thanks to the Galera cluster.

  2. Add the IP addresses of the DNS service to the resolve.conf file in the workstation instance.

  3. Install the actual bind-utils packages in the workstation instance.

sudo dnf install bind-utils
  1. Check the DNS record
162938565-2ca0c698-d243-4a5a-a2aa-4d422809adc8
  1. Stop the first instance (mariadb-0)

162938505-3bcb3240-7998-48b9-9110-1b8b41e3673f

  1. Connect to the workstation instance and verify if the DNS record is still available.

162938432-96820051-0b04-4a03-994c-53b596d80e0c NOTE: We saw that it still had a connection to the mariadb-1 instance, and the DNS service remains available to the customer. Due to the fact that I created a database service and a DNS service that both have lists of relevant servers' backend, our service has one address, and we are secure that it will remain accessible even if any of the databases or DNS servers go down.

Conclusion

As we have seen, the data is synchronized and replicated between the members of the cluster. As a result, the DNS service will be available even if Galera cluster members fail.

High availability is crucial when delivering a service to customers. It allows administrators to providing continuous service availability, improve service maintainability, as well as the ability to fix problems with no or minimal impact to customers.

OpenShift Virtualization allows you to run highly available VM workloads and to be sure they will keep running even in a case of VM failures.

This blog is part of a large and interesting project that allows us to set up a highly available DNS service on Kubernetes using the Galera cluster. This type of deployment benefits from Kubernetes cloud-native advantages and unified API that are used to simplify the configuration and management of containers and VMs on the same platform.


About the author

Almog Elfassy is a highly skilled Cloud Architect at Red Hat, leading strategic projects across diverse sectors. He brings a wealth of experience in designing and implementing cutting-edge distributed cloud services. His expertise spans a cloud-native approach, hands-on experience with various cloud services, network configurations, AI, data management, Kubernetes and more. Almog is particularly interested in understanding how businesses leverage Red Hat OpenShift and technologies to solve complex problems.

Read full bio