GlusterFS is a distributed scale out filesystem that allows rapid provisioning of additional storage based on your storage consumption needs. It incorporates automatic failover as a primary feature. All of this is accomplished without a centralized metadata server. GlusterFS is open source and part of Linux OS and it works well on Oracle's Cloud Infrastructure Services.
This experience is based of a real case customer proof of concept by the author testing 57 Oracle Cloud Infrastructure nodes using a 52TB GlusterFS volume with 26TB of data simultaneously along with a HPC (High Performance Computing) application.
This tutorial describes the deployment steps of a high availability GlusterFS Storage environment on Oracle Cloud Infrastructure Instances using a Distributed Glusterfs Volume. There are different GlusterFS types of volumes you can use as explained in the GlusterFS public documentation and it's important that you understand them first before proceeding with the initial configuration. This blog demonstrates how to use a Distributed GlusterFS Volume on two Oracle Linux Instances being accessed by one Linux Client.
As for the configuration option on Oracle Cloud Infrastructure, you might either keep all GlusterFS resources in a single availability domain (AD) or use all ADs as shown below. Depending on the network traffic utilization you should consider having all GlusterFS resources in a single AD or use a distributed option across all availability domains for fault tolerance. Add a Gluster Replicated Volume option in the setup which is recommended to avoid data loss and for production environments. In addition to using a replicated GlusterFS Volume for fault tolerance (Distributed and Replicated Volume) you should enable GlusterFS Trash Translator and Snapshots to assist you with file recovery if needed.
Here is an example of a GlusterFS architecture that can be used on Oracle Cloud Infrastructure
For this setup you need to create at least three instances; two instances will be used to hold the Distributed GlusterFS Volume and the third instance as a GlusterFS Client with a local mount point to GlusterFS volume. As an optional step for this tutorial, create two 100GB block volumes and attach them to each GlusterFS server (two GlusterFS Servers with two 100GB block volumes) that will be used later to create a single 200GB GlusterFS volume as shown in the above picture.
|Server||gfsserver.publicsubnetad1.baremetal.oraclevcn.com||Oracle Linux 7.x x86_64|
|Server||gfsserver1.publicsubnetad1.baremetal.oraclevcn.com||Oracle Linux 7.x x86_64|
|Client||gfsclient.publicsubnetad1.baremetal.oraclevcn.com||Oracle Linux 7.x x86_64|
Enable GlusterFS repo in your Oracle Linux 7.x x86_64 instances that will be used to hold your GlusterFS volume(s). Recommend checking Oracle Linux YUM public portal for current packages version. Below is an example for gluster310 packages.
$ sudo yum-config-manager --add-repo http://yum.oracle.com/repo/OracleLinux/OL7/developer_gluster310/x86_64
Install GlusterFS Server and Samba packages on all nodes that will be used as GlusterFS servers.
|$ sudo yum install glusterfs-server samba –y|
Skip these optional steps if your environment is already have prepared XFS bricks (partitions). Use /dev/sdb device which is the 100GB attached block volume you have created before as described above. Change the device name if necessary based on your current configuration. Now, create brick1 Logical Volumes for XFS bricks on both cluster nodes as shown below
$ sudo pvcreate /dev/sdb
$ sudo vgcreate vg_gluster /dev/sdb
$ sudo lvcreate -L 100000M -n brick1 vg_gluster
Setup XFS file systems:
|$ sudo mkfs.xfs /dev/vg_gluster/brick1|
Create a mount point and mount the XFS brick:
$ sudo mkdir -p /bricks/brick1
$ sudo mount /dev/vg_gluster/brick1 /bricks/brick1
Extend the /etc/fstab inserting the following line
|/dev/vg_gluster/brick1 /bricks/brick1 xfs defaults,_netdev,nofail 0 0|
Important Note: as described in the Public documentation about when connecting with iSCSI on linux instances block volumes, it’s important to include the "_netdev" and "nofail" options on every non-root block volume in /etc/fstab or the instance may fail to launch as the OS tries to mount the volume(s) before the iSCSI initiator has started.
For example, a single volume in /etc/fstab might look like this:
|/dev/sdb /data1 ext4 defaults,noatime,_netdev,nofail 0 2|
If you reboot without these options in fstab, the instance will fail to start after the next reboot. Instances in this state are not recoverable.
Enable and start glusterfsd.service on both nodes:
$ sudo systemctl enable glusterd.service
$ sudo systemctl start glusterd.service
Ports "TCP:24007-24008" are required for communication between GlusterFS nodes and each brick requires another TCP port starting at 24009.
Enable required ports on the firewall:
$ sudo firewall-cmd --zone=public --add-port=24007-24008/tcp --permanent
$ sudo firewall-cmd --reload
You also need to open ports "24007-24008" on the dashboard. Edit the Virtual Cloud Network Security List and either open all ports for the Internal Network (NOT PUBLIC NETWORK) as shown below for network 126.96.36.199/16 or just ports "24007-24008" as shown below respectively.
IP Protocol: All Protocols
Allows: all traffic for all ports
IP Protocol: TCP
Source Port Range: All
Destination Port Range: 24007-24008
Allows: TCP traffic for ports: 24007-24008
Now, use gluster command to connect the second GlusterFS node and create a Trusted Pool (Storage Cluster).
$ sudo gluster peer probe gfsserver1
peer probe: success
Verify cluster peer:
$ sudo gluster peer status
Number of Peers: 1
State: Peer in Cluster (Connected)
GlusterFS Volume works with Gluster File System which is a logical collection of XFS bricks. The following table shows dependencies between Volume types and sizes:
|Distributed (for maximum space)||1G + 1G = 2G|
|Replicated (for high availability)||1G + 1G = 1G|
|Striped (for large files)||1G + 1G = 2G|
|Distributed and Replicated||(1G+1G) + (1G+1G) = 2G|
|Distributed and Striped||(1G+1G) + (1G+1G) = 4G|
|Distributed, Replicated and Stripped||[(1G+1G)+(1G+1G)] + [(1G+1G)+(1G+1G)] = 4G|
NOTE: for production environments it is recommended using Distributed GlusterFS volume along with Replicated option for security reasons.
Open the required port on the firewall. Remember, each brick in the GlusterFS Volume requires a TCP port starting at 24009:
$ sudo firewall-cmd --zone=public --add-port=24009/tcp --permanent
$ sudo firewall-cmd --reload
The same port needs to be opened on the Dashboard as well in case you haven't opened all ports for internal network.
Use the /bricks/brick1 XFS partition on both nodes to create a highly available Replicated Volume. First create a sub-directory in /bricks/brick1 mount point. It will be necessary for GlusterFS.
|$ sudo mkdir /bricks/brick1/brick|
Create a Distributed GlusterFS Volume running these commands on the first node only (gfsserver)
$ sudo gluster volume create glustervol1 transport tcp gfsserver:/bricks/brick1/brick gfsserver1:/bricks/brick1/brick
$ sudo gluster volume start glustervol1
Verify the GlusterFS volume
$ sudo gluster volume info all
Volume Name: glustervol1
Volume ID: c660712f-29ea-4288-96b6-2c0a0c85a82a
Snapshot Count: 0
Number of Bricks: 1
GlusterFS volumes can be accessed using GlusterFS Native Client (OracleLinux 6.x and 7.X), NFS v3 (other Linux clients), or CIFS (Windows clients).
Open the Firewall for Glusterfs/NFS/CIFS Clients
$ sudo firewall-cmd --zone=public --add-service=nfs --add-service=samba --add-service=samba-client --permanent
$ sudo firewall-cmd --zone=public --add-port=111/tcp --add-port=139/tcp --add-port=445/tcp --add-port=965/tcp --add-port=2049/tcp --add-port=38465-38469/tcp --add-port=631/tcp --add-port=111/udp --add-port=963/udp --add-port=49152-49251/tcp --permanent
$ sudo firewall-cmd --reload
As already mentioned above the same ports need to be opened on the Dashboard.
All required GlusterFS Client packages are available by default in the Oracle Linux 7.x Base repository
Install GlusterFS Client packages:
|$ sudo yum install glusterfs glusterfs-fuse attr -y|
Mount GlusterFS Volumes on the client:
$ sudo mkdir /glusterfs
$ sudo mount -t glusterfs gfsserver:/glustervol1 /glusterfs
$ sudo mount |grep -i glusterfs
gfsserver:/glustervol1 on /glusterfs type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
$ df -kh |grep glusterfs
gfsserver:/glustervol1 196G 65M 98G 1% /glusterfs
to make it permanent across restarts you need to add the following line in /etc/fstab on all GlusterFS Linux clients
|gfsserver:/glustervol1 /glusterfs glusterfs defaults,_netdev,nofail 0 0|
Check the GlusterFS public documentation for more details about:
Try out Oracle’s Cloud Services by signing up for a free cloud trial today.