Tags Archives: Gluster

LPIC3 DIPLOMA Linux Clustering – LAB NOTES: GlusterFS Configuration on Centos

How To Install GlusterFS on Centos7

 

Choose a package source: either the CentOS Storage SIG or Gluster.org

 

Using CentOS Storage SIG Packages

 

 

yum search centos-release-gluster

 

yum install centos-release-gluster37

 

yum install centos-release-gluster37

 

yum install glusterfs gluster-cli glusterfs-libs glusterfs-server

 

 

 

[root@glusterfs1 ~]# yum search centos-release-gluster
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: mirrors.xtom.de
* centos-ceph-nautilus: mirror1.hs-esslingen.de
* centos-nfs-ganesha28: ftp.agdsn.de
* epel: mirrors.xtom.de
* extras: mirror.netcologne.de
* updates: mirrors.xtom.de
================================================= N/S matched: centos-release-gluster =================================================
centos-release-gluster-legacy.noarch : Disable unmaintained Gluster repositories from the CentOS Storage SIG
centos-release-gluster40.x86_64 : Gluster 4.0 (Short Term Stable) packages from the CentOS Storage SIG repository
centos-release-gluster41.noarch : Gluster 4.1 (Long Term Stable) packages from the CentOS Storage SIG repository
centos-release-gluster5.noarch : Gluster 5 packages from the CentOS Storage SIG repository
centos-release-gluster6.noarch : Gluster 6 packages from the CentOS Storage SIG repository
centos-release-gluster7.noarch : Gluster 7 packages from the CentOS Storage SIG repository
centos-release-gluster8.noarch : Gluster 8 packages from the CentOS Storage SIG repository
centos-release-gluster9.noarch : Gluster 9 packages from the CentOS Storage SIG repository

Name and summary matches only, use “search all” for everything.
[root@glusterfs1 ~]#

 

 

Alternatively, using Gluster.org Packages

 

# yum update -y

 

 

Download the latest glusterfs-epel repository from gluster.org:

 

yum install wget -y

 

 

[root@glusterfs1 ~]# yum install wget -y
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: mirrors.xtom.de
* centos-ceph-nautilus: mirror1.hs-esslingen.de
* centos-nfs-ganesha28: ftp.agdsn.de
* epel: mirrors.xtom.de
* extras: mirror.netcologne.de
* updates: mirrors.xtom.de
Package wget-1.14-18.el7_6.1.x86_64 already installed and latest version
Nothing to do
[root@glusterfs1 ~]#

 

 

 

wget -P /etc/yum.repos.d/ http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo

 

Also install the latest EPEL repository from fedoraproject.org to resolve all dependencies:

 

yum install http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

 

 

[root@glusterfs1 ~]# yum repolist
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: mirrors.xtom.de
* centos-ceph-nautilus: mirror1.hs-esslingen.de
* centos-nfs-ganesha28: ftp.agdsn.de
* epel: mirrors.xtom.de
* extras: mirror.netcologne.de
* updates: mirrors.xtom.de
repo id repo name status
base/7/x86_64 CentOS-7 – Base 10,072
centos-ceph-nautilus/7/x86_64 CentOS-7 – Ceph Nautilus 609
centos-nfs-ganesha28/7/x86_64 CentOS-7 – NFS Ganesha 2.8 153
ceph-noarch Ceph noarch packages 184
epel/x86_64 Extra Packages for Enterprise Linux 7 – x86_64 13,638
extras/7/x86_64 CentOS-7 – Extras 498
updates/7/x86_64 CentOS-7 – Updates 2,579
repolist: 27,733
[root@glusterfs1 ~]#

 

 

Then install GlusterFS Server on all glusterfs storage cluster nodes.

[root@glusterfs1 ~]# yum install glusterfs gluster-cli glusterfs-libs glusterfs-server

 

Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: mirrors.xtom.de
* centos-ceph-nautilus: mirror1.hs-esslingen.de
* centos-nfs-ganesha28: ftp.agdsn.de
* epel: mirrors.xtom.de
* extras: mirror.netcologne.de
* updates: mirrors.xtom.de
No package gluster-cli available.
No package glusterfs-server available.
Resolving Dependencies
–> Running transaction check
—> Package glusterfs.x86_64 0:6.0-49.1.el7 will be installed
—> Package glusterfs-libs.x86_64 0:6.0-49.1.el7 will be installed
–> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================================================================
Package Arch Version Repository Size
=======================================================================================================================================
Installing:
glusterfs x86_64 6.0-49.1.el7 updates 622 k
glusterfs-libs x86_64 6.0-49.1.el7 updates 398 k

Transaction Summary
=======================================================================================================================================
Install 2 Packages

Total download size: 1.0 M
Installed size: 4.3 M
Is this ok [y/d/N]: y
Downloading packages:
(1/2): glusterfs-libs-6.0-49.1.el7.x86_64.rpm | 398 kB 00:00:00
(2/2): glusterfs-6.0-49.1.el7.x86_64.rpm | 622 kB 00:00:00
—————————————————————————————————————————————
Total 2.8 MB/s | 1.0 MB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : glusterfs-libs-6.0-49.1.el7.x86_64 1/2
Installing : glusterfs-6.0-49.1.el7.x86_64 2/2
Verifying : glusterfs-6.0-49.1.el7.x86_64 1/2
Verifying : glusterfs-libs-6.0-49.1.el7.x86_64 2/2

Installed:
glusterfs.x86_64 0:6.0-49.1.el7 glusterfs-libs.x86_64 0:6.0-49.1.el7

Complete!
[root@glusterfs1 ~]#

 

 

 

 

 

Continue Reading

LPIC3-306 COURSE NOTES: GlusterFS

 

These are my notes on GlusterFS made as part of my LPIC3 Diploma course in Linux Clustering.  They are in “rough format”, presented as they were written.

 

 

Gluster FS

 

GlusterFS is a free GNU/GPL licensed scalable network-attached storage file system now offered and developed by RedHat.

 

Servers are implemented as storage “bricks” which can be added, removed and migrated usually without interrupting service provision. Each GlusterFS server or node runs a glusterfsd daemon which exports a local file system as a Gluster volume.

 

GlusterFS provides for file-based mirroring, file replication, file-based distribution and file-striping, file-based load balancing,geo-replication, storage quotas, volume failover, disk caching and volume snapshots.

 

Types of Glusterfs Volumes

 

 

Gluster provides different types of file storage volumes which can be deployed according to the requirements of the environment.

 

Broadly these are:

 

Distributed Volumes: (suitable for scalable storage but has no data redundancy)
Replicated Volumes: (offers better reliability and also data redundancy)
Distributed-Replicated Volumes: (High availability of data through redundancy and scalable storage)

 

Distributed Glusterfs Volume

 

Distributed is the default in Gluster if no volume type is specified. Files are distributed across bricks in the volume. This means that eg file1 would be stored only on brick1 or brick2 – but not on both.

 

Thus there is no data redundancy.

 

The sole advantage of distributed is providing a lower cost and simpler means to increase the total volume storage capacity. But it does not protect against data loss.

 

In the majority of cases, this volume type is NOT advisable. Repicated volumes are much safer.

 

Replicated Glusterfs Volume

 

Replicated reduces the risk of data loss that exists with distributed volumes.

 

Copies of the data are maintained on all bricks. The number of replicas in the volume are set when creating the volume.

 

The number of bricks must be equal to the replica count for a replicated volume. In order to protect against server and disk failures, the bricks of the volume should be located on different servers.

 

A major advantage of replicated is that if one brick fails on a two node cluster, or if two bricks fail on a three node cluster, then  the data can still be accessed from the other replicated brick/s.

 

It is possible to create a replicated Glusterfs volume with two nodes, but this is not recommended because a split-brain cluster situation can develop. For this reason, a replicated volume should be used with at least three nodes.

 

Distributed Striped Glusterfs Volumes

 

Distributed striped volumes stripes files across two or more Gluster server nodes. Distributed striped volumes should be deployed where scalable storage is important and where access to very large files is required.

 

The number of bricks must be a multiple of the stripe count for a distributed striped volume.

 

Creating Distributed Replicated Glusterfs Volumes

 

Distributed Replicated distributes files across replicated bricks in the volume.

 

It should be deployed in environments which require both highly scalable storage and high-reliability. Distributed replicated volumes can also provide for better file read performance.

 

The number of bricks deployed needs to be a multiple of the replica count for a distributed replicated volume.

 

The order bricks are specified also affects data protection.

 

Each replica_count consecutive bricks in the list forms a replica set. All the replica sets are then combined into a volume-wide distributed set.

 

To ensure replica-sets are not located on the same node, you should list the first brick on each server, then list the second brick on each server, continuing in the same order.

 

Gluster also provides for:

 

Dispersed Glusterfs Volumes

 

Dispersed volumes are based on erasure codes. Erasure code (EC) stripes the encoded data of files, adds redundancy information and saves the blocks across multiple bricks in the volume.

 

This is especially suitable where high level of reliability is required with minimum space waste.

 

The number of redundant bricks in the volume determines how many bricks can be lost without any interruption in the operation of the volume.

 

Distributed Dispersed Glusterfs Volumes

 

Distributed dispersed volumes are the equivalent to distributed replicated volumes, but they use dispersed subvolumes rather than replicated volumes. The purpose is to easily scale volume size and distribute the data across multiple bricks.

 

The number of bricks must be a multiple of the first subvolume.

 

 

Which GlusterFS Volume System Is Best?

 

Before installing GlusterFS, you first need to decide what type of volume is best for your environment.  

 

Replicated volume

 

This type of volume provides file replication across multiple bricks in a cluster.

 

This is the best choice for environments which require high availability, high reliability, and also scalable storage.

 

It is especially suited if you want to be able to self-mount the GlusterFS volume. This could be for example, the web server document root at /var/www/  where all files need to be accessible on that node.

 

The value passed to replica will be the same as the number of nodes in the volume.

 

Files are copied to each GlusterFS brick in the volume, rather like with RAID 1.

 

However, you can also have three or more bricks in the cluster. The usable space will be equivalent to the size of one brick, with all files written to one brick being replicated to all the other bricks in the cluster. 

 

Replicated volumes offer improved read performance for most environments and they are the most common type of volume used when clients accessing the cluster are external to the GlusterFS nodes.

 

Distributed-replicated volume

As with a RAID-10, an even number of GlusterFS bricks must be used. The usable space in this case is the size of the combined bricks passed to the replica value.

 

As an example, if there are four bricks of 20 GB each and you pass replica 2 to the creation, then files are distributed to two nodes (40 GB) and are also  replicated to two nodes.

 

With a GlusterFS system of six bricks of 20 GB and replica 3, files are distributed to three nodes (60 GB) and also replicated to three nodes.

 

And if you used replica 2, then files are distributed to two nodes (40 GB) and are replicated to four GlusterFS nodes in pairs.

 

This distribution and replication system is useful when your clients are external to the cluster, ie are not locally self-mounted.

 

 

 

An Overview of How to Install and Configure Gluster

 

 

First you require a server cluster, preferably with three nodes.

 

Install the gluster system on each node.

 

Decide which kind of gluster volume system you wish to implement – distributed, replicated, distributed-replicated etc.

 

Usually replicated will be preferred as a minimum. Distributed is not generally recommended for production-level environments due to the higher risk of data lost involved compared to the other options.

 

Next create a trusted pool. This needs to be done on just one of the nodes.

 

Then add the disk/s for the gluster storage on each node. These consitute the storage “bricks”.

 

Format and mount the storage bricks.

 

Activate the gluster volume

 

For detailed explanation and examples of the installation and configuration process of gluster refer to my LAB page on gluster.

 

Also see https://docs.gluster.org/en/v3/Quick-Start-Guide/Quickstart/

 

Continue Reading

LPIC3 DIPLOMA Linux Clustering – LAB NOTES: GlusterFS Configuration on Ubuntu

LAB for installing and configuring GlusterFS on Ubuntu

 

These are my notes made during my lab practical as part of my LPIC3 Diploma course in Linux Clustering. They are in “rough format”, presented as they were written.

 

 

Overview

 

The cluster comprises three nodes (ubuntu31, ubuntu32, ubuntu33) installed with Ubuntu Version 20 LTS and housed on a KVM virtual machine system on a Linux Ubuntu host.

 

each node has a 1gb scsi disk called /dev/sda (the root system disk is /dev/vda)
 
brick1
brick2
brick3
 
respectively (these are NOT host definitions, just gluster identities)
 

on each machine:
 

88 wget -O- https://download.gluster.org/pub/gluster/glusterfs/3.12/rsa.pub | apt-key add –
89 sudo add-apt-repository ppa:gluster/glusterfs-3.12
90 apt install glusterfs-server -y
91 systemctl start glusterd
92 systemctl enable glusterd

 
 
Created a trusted pool. This is done on ubuntu31 with the command:
 
gluster peer probe ubuntu32
 
You should immediately see peer probe: success.

 

root@ubuntu31:/home/kevin# gluster peer probe ubuntu32
 

You can check the status of peers with the command:
 
gluster peer status

 

We want the trusted pool to include all three bricks. So we do:

 
root@ubuntu31:/home/kevin# gluster peer probe ubuntu32
peer probe: success.
root@ubuntu31:/home/kevin# gluster peer probe ubuntu33
peer probe: success.
root@ubuntu31:/home/kevin# gluster peer status
Number of Peers: 2
 
Hostname: ubuntu32
Uuid: 6b4ca918-e77c-40d9-821c-e24fe7130afa
State: Peer in Cluster (Connected)
 
Hostname: ubuntu33
Uuid: e3b02490-9a14-45a3-ad0d-fcc66dd1c731
State: Peer in Cluster (Connected)
root@ubuntu31:/home/kevin#

 

Add the disk for the gluster storage on each machine:

 
Disk /dev/sda: 1 GiB, 1073741824 bytes, 2097152 sectors
Disk model: QEMU HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xffb101f9
 
Device Boot Start End Sectors Size Id Type
/dev/sda1 2048 2097151 2095104 1023M 83 Linux
 
NOTE: on these ubuntu cluster nodes the root system partition is on /dev/vda – hence the next free scsi disk is sda!

 
Format and mount the bricks
 
Perform this step on all the nodes
 
Note: We are going to use the XFS filesystem for the backend bricks.
 
But Gluster is designed to work on top of any filesystem, which supports extended attributes.
 
The following examples assume that the brick will be residing on /dev/sda1.

 

mkfs.xfs -i size=512 /dev/sda1
mkdir -p /gluster
echo ‘/dev/sda1 /gluster/brick1 xfs defaults 1 2’ >> /etc/fstab ; mount -a && mount
 

You should now see sda1 mounted at /gluster

 

root@ubuntu31:/home/kevin# mkfs.xfs -i size=512 /dev/sda1
meta-data=/dev/sda1 isize=512 agcount=4, agsize=65472 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=261888, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=1566, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
root@ubuntu31:/home/kevin#

 

do the same on the other two nodes, using
 
/gluster and /gluster respectively
 
echo ‘/dev/sda1 /gluster xfs defaults 1 2’ >> /etc/fstab ; mount -a && mount

 

/dev/sda1 on /gluster type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
root@ubuntu31:/home/kevin# d
 
/dev/sda1 1041288 40296 1000992 4% /gluster
root@ubuntu31:/home/kevin
 

root@ubuntu31:/home/kevin# gluster pool list
UUID Hostname State
6b4ca918-e77c-40d9-821c-e24fe7130afa ubuntu32 Connected
e3b02490-9a14-45a3-ad0d-fcc66dd1c731 ubuntu33 Connected
2eb4eca2-11e4-40ef-9b70-43bfa551121c localhost Connected
root@ubuntu31:/home/kevin#

 
on ubuntu31, ubuntu32, ubuntu33:
 
mkdir -p /gluster/brick

 

replica n is the number of nodes in the gluster
 

gluster volume create glustervol1 replica 3 transport tcp ubuntu31:/glusterfs/distributed ubuntu32:/glusterfs/distributed ubuntu33:/glusterfs/distributed

 

gluster volume create glustervol1 replica 3 transport tcp ubuntu31:/gluster/brick ubuntu32:/gluster/brick ubuntu33:/gluster/brick

 
root@ubuntu31:/home/kevin# gluster volume create glustervol1 replica 3 transport tcp ubuntu31:/gluster/brick ubuntu32:/gluster/brick ubuntu33:/gluster/brick
volume create: glustervol1: success: please start the volume to access data
root@ubuntu31:/home/kevin#
 

Now we’ve created the distributed volume ‘glustervol1’ – start the ‘glustervol1’ and check the volume info.
 
gluster volume start glustervol1
gluster volume info glustervol1

 
root@ubuntu31:/home/kevin# gluster volume start glustervol1
volume start: glustervol1: success
root@ubuntu31:/home/kevin#

 
root@ubuntu31:/home/kevin# gluster volume info glustervol1
 
Volume Name: glustervol1
Type: Replicate
Volume ID: 9335962f-342e-423e-aefc-a87777a5b081
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: ubuntu31:/gluster/brick
Brick2: ubuntu32:/gluster/brick
Brick3: ubuntu33:/gluster/brick
Options Reconfigured:
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.client-io-threads: off
root@ubuntu31:/home/kevin#

 

on the client machines:

 

Install glusterfs-client to the Ubuntu system using the apt command.
 
sudo apt install glusterfs-client -y
 
Now create a new directory ‘/mnt/glusterfs’ when the glusterfs-client installation is complete.
 
mkdir -p /mnt/glusterfs

And mount the distributed glusterfs volume to the ‘/mnt/glusterfs’ directory.

 

mount -t glusterfs ubuntu31:/glustervol1 /mnt/glusterfs

 

ubuntu31:/glustervol1 1041288 50808 990480 5% /mnt/glusterfs
root@yoga:/home/kevin#

 

Continue Reading