GlusterFS + WebDAV Centos Setup Guide
Emad R last edited by Emad R
This will be a quick and dirty setup guide, please if you see something stupid point it out. I started from the latest Centos 7.3 minimal and I applied this on 3 nodes. This helped me understand how NAS works, and it is very rough draft. I felt bad not writing it once I learned this cause the documentation now i have is 1 month old and I it may be lacking, thus I apologize. before hand. Please read this guide with the notion of understanding and not copy/paste implementing. I mean after you understand the goal below and understand it I reckon you can implement it better by enhancing it.
I recommend before you start to partition Centos in such a way:
2 GiB boot | 50 GiB root | 2 GiB swap
Cause GlusterFS needs to be away from root / partition.
And leave 1 empty unallocated partition/space for Gluster, we will create in latter steps.
Before we start, the imaginary goal I created is create 3 nodes that replicate the Filesystem, and if 1 node can go down, users can have the option to connect to node 2 or 3 (manually by entering different IP address), and users will be using WinSCP (cause it rocks in stability and security and performance also it tries to make it easy for dumb users). So let us start:
yum search centos-release-gluster # check LTS version (if you find latest + LTS install) yum -y install centos-release-gluster310 -y sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-Gluster-3.10.repo yum --enablerepo=centos-gluster310,epel -y install glusterfs-server systemctl enable glusterd systemctl start rpcbind systemctl enable rpcbind systemctl restart glusterd
Part 1 finished, we installed and started required packages and services.
Prep the servers/nodes by changing their hostnames and configured /etc/hosts to reflect that, for example ensure hostname is something like centos.gluster.1 | centos.gluster.2 | centos.gluster.3 (one on each node)
And put the hostnames to manually point to IP of their respective machines, in each of the nodes
192.168.1.20 centos.gluster.1 192.168.1.19 centos.gluster.2 192.168.1.18 centos.gluster.3
Now on all nodes do this:
mkdir -p /glusterfs mkdir -p /replica_files
Then use cfdisk to create/write the new partition without a type. it has command line interface simply navigate using the arrow keys and create partition.
Then run the below on all 3 nodes
partprobe mkfs.xfs /dev/sda4 (replace /dev/sda4 with the your partition) partprobe mount /dev/sda4 /glusterfs/ nano /etc/fstab -> /dev/sda4 /glusterfs xfs defaults 0 0 mkdir -p /glusterfs/replicafs
Part 2 done we created the folder structure and prepared the filesystem
Now let us connect the peers:
gluster peer probe centos.gluster.1 #(do this from node 1/2/3) gluster peer probe centos.gluster.2 #(do this from node 1/2/3) gluster peer probe centos.gluster.3 #(do this from node 1/2/3)
and check that all is good by running:
gluster peer status
If you have any issues, trying pinging the hostnames from the nodes and see if they translate to IP properly, if not check your hosts file.
Let us create the replication between 3 nodes, you need to run this on 1 node only
gluster volume create vol_replica replica 3 transport tcp \ centos.gluster.1:/glusterfs/replicafs \ centos.gluster.2:/glusterfs/replicafs \ centos.gluster.3:/glusterfs/replicafs gluster volume start vol_replica gluster volume info
You can then mount this replicafs by running the below on each node respectively :
mount -t glusterfs centos.gluster.1:/vol_replica /replica_files (run only on node 1) mount -t glusterfs centos.gluster.2:/vol_replica /replica_files (run only on node 2) mount -t glusterfs centos.gluster.3:/vol_replica /replica_files (run only on node 3)
I had trouble using /etc/fstab to mount the above 3 commands, so I just scheduled the above commands to run at startup.
Now the fun starts with WebDAV you can do this on seperate node or on one of the 3 nodes or on all 3 nodes, it is up to you.
yum -y install httpd systemctl start httpd systemctl enable httpd
Create SSL/TLS key + Certificate + Certificate Signing request (3 files server.key|server.cert|server.csr)
cd /etc/pki/tls/certs make server.key openssl rsa -in server.key -out server.key make server.csr openssl x509 -in server.csr -out server.crt -req -signkey server.key -days 365 yum -y install mod_ssl nano /etc/httpd/conf.d/ssl.conf # line 59: uncomment -> DocumentRoot "/var/www/html" # line 60: uncomment and specify the server name -> ServerName 192.168.1.20:443 # line 100: change to the one created earlier -> SSLCertificateFile /etc/pki/tls/certs/server.crt # line 107: change to the one created earlier -> SSLCertificateKeyFile /etc/pki/tls/certs/server.key # line 110: comment out -> # SSLCertificateChainFile
To Change SSL/TLS 443 default port, edit 3 lines:
nano /etc/httpd/conf.d/ssl.conf Listen 7777 https <VirtualHost _default_:7777> ServerName 192.168.1.20:7777
Using WebDAV + GlusterFS:
make sure you mounted the glusterfs on /replica_files first
mkdir /replica_files/public chown apache. /replica_files/public chmod 770 /replica_files/public
DavLockDB "/tmp/DavLock" Alias /public /replica_files/public <Location /public> DAV On SSLRequireSSL Options None AuthType Basic AuthName "Public Share" AuthUserFile /etc/httpd/conf/.htpasswd <RequireAny> Require method GET POST OPTIONS Require valid-user </RequireAny> </Location>
htpasswd -B -C 10 /etc/httpd/conf/.htpasswd jane # Create Bcrypt strong pass for jane (Works with WinSCP)
What if you want to create a personal share, not public one, will you need to understand a public share needs to be accessed with username and pass, but you can share this credentials to multiple users.
However you can also separate database password files, which adds separation layer, for example even if Jane password is correct she wont access another folder (hr folder for example).
mkdir -p /replica_files/hr chown apache. /replica_files/hr chmod 770 /replica_files/hr
Alias /hr /replica_files/hr <Directory /replica_files/hr> DAV On AuthType Basic AuthName "HR Private Share" AuthUserFile /etc/httpd/conf/hr.passwd Require valid-user
The only noteable change is hr.passwd instead of .htpasswd
to create user there (hr.passwd):
htpasswd -c -B -C 10 /etc/httpd/conf/hr.passwd marc
add another user
htpasswd -B -C 10 /etc/httpd/conf/hr.passwd marie
dont use the -c twice it will re-create the database file. Other useful commands include:
htpasswd -v /etc/httpd/conf/.htpasswd jane # to verify user exist htpasswd -D /etc/httpd/conf/.htpasswd jane # delete user jane
htpasswd password encryption info:
-m default is MD5 htpasswd -B is Bcrypt which is very secure can be paired with -C valid values 4-31 (higher is more secure but slower) -s SHA encryption (insecure)
If SELinux is enabled, change rules like follows.
chcon -R -t httpd_sys_rw_content_t /webdav_folder_location semanage fcontext -a -t httpd_sys_rw_content_t /webdav_folder_location
In WinSCP you can connect using one liner:
copy and paste this in Hostname
In the end you can create structure, if you installed Apace+WebDav on all 3 nodes, that if one node goes down you can instruct users to use node 2 IP address, and it will have exactly the same files. but users need to know the 3 nodes Address before hand, I still didnt know how to group all 3 nodes location behind load balancing proxy or something similar. will learn that in the future.
Many of the article especially GlusterFs take from guides already found in ML, see scott guide: