LPIC3 DIPLOMA Linux Clustering – LAB NOTES: GlusterFS

LAB for installing and configuring GlusterFS on Ubuntu

 

These are my notes made during my lab practical as part of my LPIC3 Diploma course in Linux Clustering. They are in “rough format”, presented as they were written.

 

 

Overview

 

The cluster comprises three nodes (ubuntu31, ubuntu32, ubuntu33) installed with Ubuntu Version 20 LTS and housed on a KVM virtual machine system on a Linux Ubuntu host.

 

each node has a 1gb scsi disk called /dev/sda (the root system disk is /dev/vda)
 
brick1
brick2
brick3
 
respectively (these are NOT host definitions, just gluster identities)
 

on each machine:
 

88 wget -O- https://download.gluster.org/pub/gluster/glusterfs/3.12/rsa.pub | apt-key add –
89 sudo add-apt-repository ppa:gluster/glusterfs-3.12
90 apt install glusterfs-server -y
91 systemctl start glusterd
92 systemctl enable glusterd

 
 
Created a trusted pool. This is done on ubuntu31 with the command:
 
gluster peer probe ubuntu32
 
You should immediately see peer probe: success.

 

root@ubuntu31:/home/kevin# gluster peer probe ubuntu32
 

You can check the status of peers with the command:
 
gluster peer status

 

We want the trusted pool to include all three bricks. So we do:

 
root@ubuntu31:/home/kevin# gluster peer probe ubuntu32
peer probe: success.
root@ubuntu31:/home/kevin# gluster peer probe ubuntu33
peer probe: success.
root@ubuntu31:/home/kevin# gluster peer status
Number of Peers: 2
 
Hostname: ubuntu32
Uuid: 6b4ca918-e77c-40d9-821c-e24fe7130afa
State: Peer in Cluster (Connected)
 
Hostname: ubuntu33
Uuid: e3b02490-9a14-45a3-ad0d-fcc66dd1c731
State: Peer in Cluster (Connected)
root@ubuntu31:/home/kevin#

 

Add the disk for the gluster storage on each machine:

 
Disk /dev/sda: 1 GiB, 1073741824 bytes, 2097152 sectors
Disk model: QEMU HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xffb101f9
 
Device Boot Start End Sectors Size Id Type
/dev/sda1 2048 2097151 2095104 1023M 83 Linux
 
NOTE: on these ubuntu cluster nodes the root system partition is on /dev/vda – hence the next free scsi disk is sda!

 
Format and mount the bricks
 
Perform this step on all the nodes
 
Note: We are going to use the XFS filesystem for the backend bricks.
 
But Gluster is designed to work on top of any filesystem, which supports extended attributes.
 
The following examples assume that the brick will be residing on /dev/sda1.

 

mkfs.xfs -i size=512 /dev/sda1
mkdir -p /gluster
echo ‘/dev/sda1 /gluster/brick1 xfs defaults 1 2’ >> /etc/fstab ; mount -a && mount
 

You should now see sda1 mounted at /gluster

 

root@ubuntu31:/home/kevin# mkfs.xfs -i size=512 /dev/sda1
meta-data=/dev/sda1 isize=512 agcount=4, agsize=65472 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=261888, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=1566, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
root@ubuntu31:/home/kevin#

 

do the same on the other two nodes, using
 
/gluster and /gluster respectively
 
echo ‘/dev/sda1 /gluster xfs defaults 1 2’ >> /etc/fstab ; mount -a && mount

 

/dev/sda1 on /gluster type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
root@ubuntu31:/home/kevin# d
 
/dev/sda1 1041288 40296 1000992 4% /gluster
root@ubuntu31:/home/kevin
 

root@ubuntu31:/home/kevin# gluster pool list
UUID Hostname State
6b4ca918-e77c-40d9-821c-e24fe7130afa ubuntu32 Connected
e3b02490-9a14-45a3-ad0d-fcc66dd1c731 ubuntu33 Connected
2eb4eca2-11e4-40ef-9b70-43bfa551121c localhost Connected
root@ubuntu31:/home/kevin#

 
on ubuntu31, ubuntu32, ubuntu33:
 
mkdir -p /gluster/brick

 

replica n is the number of nodes in the gluster
 

gluster volume create glustervol1 replica 3 transport tcp ubuntu31:/glusterfs/distributed ubuntu32:/glusterfs/distributed ubuntu33:/glusterfs/distributed

 

gluster volume create glustervol1 replica 3 transport tcp ubuntu31:/gluster/brick ubuntu32:/gluster/brick ubuntu33:/gluster/brick

 
root@ubuntu31:/home/kevin# gluster volume create glustervol1 replica 3 transport tcp ubuntu31:/gluster/brick ubuntu32:/gluster/brick ubuntu33:/gluster/brick
volume create: glustervol1: success: please start the volume to access data
root@ubuntu31:/home/kevin#
 

Now we’ve created the distributed volume ‘glustervol1’ – start the ‘glustervol1’ and check the volume info.
 
gluster volume start glustervol1
gluster volume info glustervol1

 
root@ubuntu31:/home/kevin# gluster volume start glustervol1
volume start: glustervol1: success
root@ubuntu31:/home/kevin#

 
root@ubuntu31:/home/kevin# gluster volume info glustervol1
 
Volume Name: glustervol1
Type: Replicate
Volume ID: 9335962f-342e-423e-aefc-a87777a5b081
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: ubuntu31:/gluster/brick
Brick2: ubuntu32:/gluster/brick
Brick3: ubuntu33:/gluster/brick
Options Reconfigured:
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.client-io-threads: off
root@ubuntu31:/home/kevin#

 

on the client machines:

 

Install glusterfs-client to the Ubuntu system using the apt command.
 
sudo apt install glusterfs-client -y
 
Now create a new directory ‘/mnt/glusterfs’ when the glusterfs-client installation is complete.
 
mkdir -p /mnt/glusterfs

And mount the distributed glusterfs volume to the ‘/mnt/glusterfs’ directory.

 

mount -t glusterfs ubuntu31:/glustervol1 /mnt/glusterfs

 

ubuntu31:/glustervol1 1041288 50808 990480 5% /mnt/glusterfs
root@yoga:/home/kevin#