GlusterFS + WebDAV Centos Setup Guide

  • This will be a quick and dirty setup guide, please if you see something stupid point it out. I started from the latest Centos 7.3 minimal and I applied this on 3 nodes. This helped me understand how NAS works, and it is very rough draft. I felt bad not writing it once I learned this cause the documentation now i have is 1 month old and I it may be lacking, thus I apologize. before hand. Please read this guide with the notion of understanding and not copy/paste implementing. I mean after you understand the goal below and understand it I reckon you can implement it better by enhancing it.

    I recommend before you start to partition Centos in such a way:

    2 GiB boot | 50 GiB root | 2 GiB swap

    Cause GlusterFS needs to be away from root / partition.
    And leave 1 empty unallocated partition/space for Gluster, we will create in latter steps.

    Before we start, the imaginary goal I created is create 3 nodes that replicate the Filesystem, and if 1 node can go down, users can have the option to connect to node 2 or 3 (manually by entering different IP address), and users will be using WinSCP (cause it rocks in stability and security and performance also it tries to make it easy for dumb users). So let us start:

    yum search centos-release-gluster  	# check LTS version  (if you find  latest + LTS install)
    yum -y install centos-release-gluster310 -y
    sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-Gluster-3.10.repo
    yum --enablerepo=centos-gluster310,epel -y install glusterfs-server
    systemctl enable glusterd
    systemctl start rpcbind
    systemctl enable rpcbind
    systemctl restart glusterd

    Part 1 finished, we installed and started required packages and services.

    Prep the servers/nodes by changing their hostnames and configured /etc/hosts to reflect that, for example ensure hostname is something like centos.gluster.1 | centos.gluster.2 | centos.gluster.3 (one on each node)

    nano /etc/hostname	


    nano /etc/hosts	

    And put the hostnames to manually point to IP of their respective machines, in each of the nodes centos.gluster.1 centos.gluster.2 centos.gluster.3

    Now on all nodes do this:

    mkdir -p /glusterfs
    mkdir -p /replica_files

    Then use cfdisk to create/write the new partition without a type. it has command line interface simply navigate using the arrow keys and create partition.

    Then run the below on all 3 nodes

    mkfs.xfs /dev/sda4                                (replace /dev/sda4 with the your partition)
    mount /dev/sda4 /glusterfs/
    nano /etc/fstab -> /dev/sda4       /glusterfs      xfs     defaults 0 0
    mkdir -p /glusterfs/replicafs

    Part 2 done we created the folder structure and prepared the filesystem

    Now let us connect the peers:

    gluster peer probe centos.gluster.1	 #(do this from node 1/2/3)
    gluster peer probe centos.gluster.2	 #(do this from node 1/2/3)
    gluster peer probe centos.gluster.3	 #(do this from node 1/2/3)

    and check that all is good by running:

    gluster peer status

    If you have any issues, trying pinging the hostnames from the nodes and see if they translate to IP properly, if not check your hosts file.

    Let us create the replication between 3 nodes, you need to run this on 1 node only

    gluster volume create vol_replica replica 3 transport tcp \	
    centos.gluster.1:/glusterfs/replicafs \
    centos.gluster.2:/glusterfs/replicafs \
    gluster volume start vol_replica
    gluster volume info	

    You can then mount this replicafs by running the below on each node respectively :

    mount -t glusterfs centos.gluster.1:/vol_replica  /replica_files (run only on node 1)
    mount -t glusterfs centos.gluster.2:/vol_replica  /replica_files (run only on node 2)
    mount -t glusterfs centos.gluster.3:/vol_replica  /replica_files (run only on node 3)

    I had trouble using /etc/fstab to mount the above 3 commands, so I just scheduled the above commands to run at startup.

    Now the fun starts with WebDAV you can do this on seperate node or on one of the 3 nodes or on all 3 nodes, it is up to you.

    yum -y install httpd
    systemctl start httpd 
    systemctl enable httpd 

    Create SSL/TLS key + Certificate + Certificate Signing request (3 files server.key|server.cert|server.csr)

    cd /etc/pki/tls/certs
    make server.key 
    openssl rsa -in server.key -out server.key 
    make server.csr 
    openssl x509 -in server.csr -out server.crt -req -signkey server.key -days 365
    yum -y install mod_ssl
    nano /etc/httpd/conf.d/ssl.conf
    	# line 59: uncomment -> DocumentRoot "/var/www/html"
    	# line 60: uncomment and specify the server name -> ServerName
    	# line 100: change to the one created earlier -> SSLCertificateFile /etc/pki/tls/certs/server.crt
    	# line 107: change to the one created earlier -> SSLCertificateKeyFile /etc/pki/tls/certs/server.key
    	# line 110: comment out -> # SSLCertificateChainFile

    To Change SSL/TLS 443 default port, edit 3 lines:

    nano /etc/httpd/conf.d/ssl.conf
    Listen 7777 https
    <VirtualHost _default_:7777>

    Using WebDAV + GlusterFS:

    make sure you mounted the glusterfs on /replica_files first

    mkdir /replica_files/public
    chown apache. /replica_files/public
    chmod 770 /replica_files/public

    nano /etc/httpd/conf.d/webdav.conf

    	DavLockDB "/tmp/DavLock"
    	Alias /public /replica_files/public
    	<Location /public>
    	DAV On
    	Options None
    	AuthType Basic
    	AuthName "Public Share"
    	AuthUserFile /etc/httpd/conf/.htpasswd
        Require method GET POST OPTIONS
        Require valid-user

    Create user

    htpasswd -B -C 10 /etc/httpd/conf/.htpasswd jane	# Create Bcrypt strong pass for jane (Works with WinSCP)

    What if you want to create a personal share, not public one, will you need to understand a public share needs to be accessed with username and pass, but you can share this credentials to multiple users.

    However you can also separate database password files, which adds separation layer, for example even if Jane password is correct she wont access another folder (hr folder for example).

    mkdir -p /replica_files/hr
    chown apache. /replica_files/hr
    chmod 770 /replica_files/hr	

    nano /etc/httpd/conf.d/webdav.conf

    And add:

    Alias /hr /replica_files/hr
    <Directory /replica_files/hr>
        DAV             On
        AuthType        Basic 
        AuthName        "HR Private Share"
        AuthUserFile    /etc/httpd/conf/hr.passwd
        Require         valid-user 

    The only noteable change is hr.passwd instead of .htpasswd
    to create user there (hr.passwd):

    htpasswd -c -B -C 10 /etc/httpd/conf/hr.passwd marc

    add another user

    htpasswd -B -C 10 /etc/httpd/conf/hr.passwd marie

    dont use the -c twice it will re-create the database file. Other useful commands include:

    htpasswd -v	/etc/httpd/conf/.htpasswd jane			# to verify user exist
    htpasswd -D	/etc/httpd/conf/.htpasswd jane			# delete user jane

    htpasswd password encryption info:

    -m default is MD5 htpasswd
    -B is Bcrypt which is very secure can be paired with -C valid values 4-31 (higher is more secure but slower)
    -s SHA encryption (insecure)

    If SELinux is enabled, change rules like follows.

    chcon -R -t httpd_sys_rw_content_t /webdav_folder_location
    semanage fcontext -a -t httpd_sys_rw_content_t /webdav_folder_location

    In WinSCP you can connect using one liner:

    https://marc:[email protected]:7777/hr/

    copy and paste this in Hostname

    In the end you can create structure, if you installed Apace+WebDav on all 3 nodes, that if one node goes down you can instruct users to use node 2 IP address, and it will have exactly the same files. but users need to know the 3 nodes Address before hand, I still didnt know how to group all 3 nodes location behind load balancing proxy or something similar. will learn that in the future.

    Many of the article especially GlusterFs take from guides already found in ML, see scott guide: