LPIC3-306 COURSE NOTES: GlusterFS

 

These are my notes on GlusterFS made as part of my LPIC3 Diploma course in Linux Clustering.  They are in “rough format”, presented as they were written.

 

 

Gluster FS

 

GlusterFS is a free GNU/GPL licensed scalable network-attached storage file system now offered and developed by RedHat.

 

Servers are implemented as storage “bricks” which can be added, removed and migrated usually without interrupting service provision. Each GlusterFS server or node runs a glusterfsd daemon which exports a local file system as a Gluster volume.

 

GlusterFS provides for file-based mirroring, file replication, file-based distribution and file-striping, file-based load balancing,geo-replication, storage quotas, volume failover, disk caching and volume snapshots.

 

Types of Glusterfs Volumes

 

 

Gluster provides different types of file storage volumes which can be deployed according to the requirements of the environment.

 

Broadly these are:

 

Distributed Volumes: (suitable for scalable storage but has no data redundancy)
Replicated Volumes: (offers better reliability and also data redundancy)
Distributed-Replicated Volumes: (High availability of data through redundancy and scalable storage)

 

Distributed Glusterfs Volume

 

Distributed is the default in Gluster if no volume type is specified. Files are distributed across bricks in the volume. This means that eg file1 would be stored only on brick1 or brick2 – but not on both.

 

Thus there is no data redundancy.

 

The sole advantage of distributed is providing a lower cost and simpler means to increase the total volume storage capacity. But it does not protect against data loss.

 

In the majority of cases, this volume type is NOT advisable. Repicated volumes are much safer.

 

Replicated Glusterfs Volume

 

Replicated reduces the risk of data loss that exists with distributed volumes.

 

Copies of the data are maintained on all bricks. The number of replicas in the volume are set when creating the volume.

 

The number of bricks must be equal to the replica count for a replicated volume. In order to protect against server and disk failures, the bricks of the volume should be located on different servers.

 

A major advantage of replicated is that if one brick fails on a two node cluster, or if two bricks fail on a three node cluster, then  the data can still be accessed from the other replicated brick/s.

 

It is possible to create a replicated Glusterfs volume with two nodes, but this is not recommended because a split-brain cluster situation can develop. For this reason, a replicated volume should be used with at least three nodes.

 

Distributed Striped Glusterfs Volumes

 

Distributed striped volumes stripes files across two or more Gluster server nodes. Distributed striped volumes should be deployed where scalable storage is important and where access to very large files is required.

 

The number of bricks must be a multiple of the stripe count for a distributed striped volume.

 

Creating Distributed Replicated Glusterfs Volumes

 

Distributed Replicated distributes files across replicated bricks in the volume.

 

It should be deployed in environments which require both highly scalable storage and high-reliability. Distributed replicated volumes can also provide for better file read performance.

 

The number of bricks deployed needs to be a multiple of the replica count for a distributed replicated volume.

 

The order bricks are specified also affects data protection.

 

Each replica_count consecutive bricks in the list forms a replica set. All the replica sets are then combined into a volume-wide distributed set.

 

To ensure replica-sets are not located on the same node, you should list the first brick on each server, then list the second brick on each server, continuing in the same order.

 

Gluster also provides for:

 

Dispersed Glusterfs Volumes

 

Dispersed volumes are based on erasure codes. Erasure code (EC) stripes the encoded data of files, adds redundancy information and saves the blocks across multiple bricks in the volume.

 

This is especially suitable where high level of reliability is required with minimum space waste.

 

The number of redundant bricks in the volume determines how many bricks can be lost without any interruption in the operation of the volume.

 

Distributed Dispersed Glusterfs Volumes

 

Distributed dispersed volumes are the equivalent to distributed replicated volumes, but they use dispersed subvolumes rather than replicated volumes. The purpose is to easily scale volume size and distribute the data across multiple bricks.

 

The number of bricks must be a multiple of the first subvolume.

 

 

Which GlusterFS Volume System Is Best?

 

Before installing GlusterFS, you first need to decide what type of volume is best for your environment.  

 

Replicated volume

 

This type of volume provides file replication across multiple bricks in a cluster.

 

This is the best choice for environments which require high availability, high reliability, and also scalable storage.

 

It is especially suited if you want to be able to self-mount the GlusterFS volume. This could be for example, the web server document root at /var/www/  where all files need to be accessible on that node.

 

The value passed to replica will be the same as the number of nodes in the volume.

 

Files are copied to each GlusterFS brick in the volume, rather like with RAID 1.

 

However, you can also have three or more bricks in the cluster. The usable space will be equivalent to the size of one brick, with all files written to one brick being replicated to all the other bricks in the cluster. 

 

Replicated volumes offer improved read performance for most environments and they are the most common type of volume used when clients accessing the cluster are external to the GlusterFS nodes.

 

Distributed-replicated volume

As with a RAID-10, an even number of GlusterFS bricks must be used. The usable space in this case is the size of the combined bricks passed to the replica value.

 

As an example, if there are four bricks of 20 GB each and you pass replica 2 to the creation, then files are distributed to two nodes (40 GB) and are also  replicated to two nodes.

 

With a GlusterFS system of six bricks of 20 GB and replica 3, files are distributed to three nodes (60 GB) and also replicated to three nodes.

 

And if you used replica 2, then files are distributed to two nodes (40 GB) and are replicated to four GlusterFS nodes in pairs.

 

This distribution and replication system is useful when your clients are external to the cluster, ie are not locally self-mounted.

 

 

 

An Overview of How to Install and Configure Gluster

 

 

First you require a server cluster, preferably with three nodes.

 

Install the gluster system on each node.

 

Decide which kind of gluster volume system you wish to implement – distributed, replicated, distributed-replicated etc.

 

Usually replicated will be preferred as a minimum. Distributed is not generally recommended for production-level environments due to the higher risk of data lost involved compared to the other options.

 

Next create a trusted pool. This needs to be done on just one of the nodes.

 

Then add the disk/s for the gluster storage on each node. These consitute the storage “bricks”.

 

Format and mount the storage bricks.

 

Activate the gluster volume

 

For detailed explanation and examples of the installation and configuration process of gluster refer to my LAB page on gluster.

 

Also see https://docs.gluster.org/en/v3/Quick-Start-Guide/Quickstart/