Doubt on a script- Backup site & DB



  • Just wanted to check a script that I am working on:

    Goal is to take complete site backup with db and then rsync to remote server, email the download link

    #Cleanup old backup files
    rm -f /home/ACCOUNT_NAME/BACKUP_FOLDER/backup-*.tgz
    
    #Insert date and a random key (key added to use it on public url to avoid guessing of public download link)
    date=$(date +%Y-%m-%d)
    key=$(head /dev/urandom | md5sum | cut -c1-10)
    
    #MySQLdump of the account to the backup folder
    mysqldump -uDB_USER -pDB_PASSWORD --single-transaction DB_NAME 2>&1 | gzip > /home/ACCOUNT_NAME/BACKUP_FOLDER/backup-${date}-${key}_SITE_NAME.sql.gz
    
    #Compress the web directory and sql backup to a single backup file
    tar czf /home/ACCOUNT_NAME/BACKUP_FOLDER/backup-${date}-${key}_SITE_NAME.tgz /home/ACCOUNT_NAME/public_html /home/ACCOUNT_NAME/BACKUP_FOLDER/backup-${date}-${key}_SITE_NAME.sql.gz 2> /dev/null
    
    #Rsync the final backup file to remote server, also cleanup existing old backup prior to the transfer by comparing source directory
    rsync -a --delete /home/ACCOUNT_NAME/BACKUP_FOLDER/backup-${date}-${key}_SITE_NAME.tgz [email protected]_SERVER_URL:/home/ACCOUNT_NAME/public_html/SITE_NAME_BACKUP/
    
    #Email the backup link
    echo "Latest backup is available at http://REMOTE_SERVER_URL/SITE_NAME_BACKUP/backup-${date}-${key}_SITE_NAME.tgz" | mail -s "Server backup available for download" [email protected]
    

    The rsync command for some reason is not cleaning up old files!

    Is this correct or are there any options to tweak this?



  • Why use a delete function, wouldn't rsync automatically remove any files that are not in the source from the target?

    From looking this up, other people have had similar issues, where using rsync -a --delete is effectively not rsyncing the files at all.



  • Here is a topic from the ubuntu forums.

    Are you trying to delete files from the source or the destination?

    If from the destination

    "rsync -avh source/ dest/ --delete"

    Might be how you need to enter the command.



  • @DustinB3403 said in Doubt on a script- Backup site & DB:

    Here is a topic from the ubuntu forums.

    Are you trying to delete files from the source or the destination?

    If from the destination

    "rsync -avh source/ dest/ --delete"

    Might be how you need to enter the command.

    The first line on the script cleans up old backup files, so during rsync i want to compare the source and destination and cleanup accordingly. I will try with the --delete at the end and see if that helps



  • @Ambarishrh said in Doubt on a script- Backup site & DB:

    @DustinB3403 said in Doubt on a script- Backup site & DB:

    Here is a topic from the ubuntu forums.

    Are you trying to delete files from the source or the destination?

    If from the destination

    "rsync -avh source/ dest/ --delete"

    Might be how you need to enter the command.

    The first line on the script cleans up old backup files, so during rsync i want to compare the source and destination and cleanup accordingly. I will try with the --delete at the end and see if that helps

    That didn't work!



  • Can you post the logs?



  • Rsync --delete doesn't delete files in target directory.

    Obviously just an example

    rsync -avz --ignore-existing --recursive --delete /var/www/ [email protected]:/var/www



  • @DustinB3403 said in Doubt on a script- Backup site & DB:

    rsync -avz --ignore-existing --recursive --delete

    Tried that as well!
    sending incremental file list
    backup-2016-10-10-16694c6b59_dur.tgz
    582776973 100% 75.96MB/s 0:00:07 (xfer#1, to-check=0/1)

    sent 582972731 bytes received 31 bytes 77729701.60 bytes/sec
    total size is 582776973 speedup is 1.00

    Copies the new file but doesnt do anything with old



  • I do have a * on the file but without that i can't get the new file created



  • But with the wildcard, you can't delete files. (Just read that in the manual)



  • @DustinB3403 said in Doubt on a script- Backup site & DB:

    Why use a delete function, wouldn't rsync automatically remove any files that are not in the source from the target?

    Yes, it will do that.



  • --delete
    			  This tells rsync to delete extraneous files from the receiving side (ones that arenât on the sending side), but only for the directories that are being synchronized.  You must have asked rsync to send the whole direcâ
    			  tory  (e.g.  "dir" or "dir/") without using a wildcard for the directoryâs contents (e.g. "dir/*") since the wildcard is expanded by the shell and rsync thus gets a request to transfer individual files, not the filesâ
    			  parent directory.  Files that are excluded from the transfer are also excluded from being deleted unless you use the --delete-excluded option or  mark  the  rules  as  only  matching  on  the  sending  side  (see  the
    			  include/exclude modifiers in the FILTER RULES section).


  • I wonder if --delete-excluded is the required approach with this, as it accounts for wildcards..

    According the manual --delete-excluded is for use cases where you're using the wildcard flag. So it would remove the folder and contents, rather than just the contents of a particular folder.



  • @scottalanmiller said in Doubt on a script- Backup site & DB:

    @DustinB3403 said in Doubt on a script- Backup site & DB:

    Why use a delete function, wouldn't rsync automatically remove any files that are not in the source from the target?

    Yes, it will do that.

    If am not mistaken, plain rsync just syncs the difference to destination but does not deletes anything



  • @DustinB3403 said in Doubt on a script- Backup site & DB:

    I wonder if --delete-excluded is the required approach with this, as it accounts for wildcards..

    According the manual --delete-excluded is for use cases where you're using the wildcard flag. So it would remove the folder and contents, rather than just the contents of a particular folder.

    Fixed it! So the issue on my case was that i was transferring a file and adding --delete switch, whereas the --delete only works with a folder.

    --delete
    
    This tells rsync to delete extraneous files from the receiving side (ones that aren’t on the sending side), but only for the directories that are being synchronized. You must have asked rsync to send the whole directory (e.g. "dir" or "dir/") without using a wildcard for the directory’s contents (e.g. "dir/*") since the wildcard is expanded by the shell and rsync thus gets a request to transfer individual files, not the files’ parent directory. Files that are excluded from the transfer are also excluded from being deleted unless you use the --delete-excluded option or mark the rules as only matching on the sending side (see the include/exclude modifiers in the FILTER RULES section).
    

    So i updated my script and added the backup files to a dir and synced that, all works fine as expected.



  • Final script:

    #Cleanup old backup files
    rm -f /home/ACCOUNT_NAME/BACKUP_FOLDER/*
    
    #Insert date and a random key (key added to use it on public url to avoid guessing of public download link)
    date=$(date +%Y-%m-%d)
    key=$(head /dev/urandom | md5sum | cut -c1-10)
    
    #MySQLdump of the account to the backup folder
    mysqldump -uDB_USER -pDB_PASSWORD --single-transaction DB_NAME 2>&1 | gzip > /home/ACCOUNT_NAME/BACKUP_FOLDER/backup-${date}-${key}_SITE_NAME.sql.gz
    
    #Compress the web directory and sql backup to a single backup file
    tar czf /home/ACCOUNT_NAME/BACKUP_FOLDER/backup-${date}-${key}_SITE_NAME.tgz /home/ACCOUNT_NAME/public_html /home/ACCOUNT_NAME/BACKUP_FOLDER/backup-${date}-${key}_SITE_NAME.sql.gz 2> /dev/null
    
    #Rsync the final backup file to remote server, also cleanup existing old backup prior to the transfer by comparing source directory
    rsync -a --delete /home/ACCOUNT_NAME/BACKUP_FOLDER/ [email protected]_SERVER_URL:/home/ACCOUNT_NAME/public_html/SITE_NAME_BACKUP/
    
    #Email the backup link
    echo "Latest backup is available at http://REMOTE_SERVER_URL/SITE_NAME_BACKUP/backup-${date}-${key}_SITE_NAME.tgz" | mail -s "Server backup available for download" [email protected]


  • @Ambarishrh said in Doubt on a script- Backup site & DB:

    @scottalanmiller said in Doubt on a script- Backup site & DB:

    @DustinB3403 said in Doubt on a script- Backup site & DB:

    Why use a delete function, wouldn't rsync automatically remove any files that are not in the source from the target?

    Yes, it will do that.

    If am not mistaken, plain rsync just syncs the difference to destination but does not deletes anything

    But you always select your flags, you never just use vanilla.



  • @scottalanmiller said in Doubt on a script- Backup site & DB:

    @Ambarishrh said in Doubt on a script- Backup site & DB:

    @scottalanmiller said in Doubt on a script- Backup site & DB:

    @DustinB3403 said in Doubt on a script- Backup site & DB:

    Why use a delete function, wouldn't rsync automatically remove any files that are not in the source from the target?

    Yes, it will do that.

    If am not mistaken, plain rsync just syncs the difference to destination but does not deletes anything

    But you always select your flags, you never just use vanilla.

    But what was mentioned earlier

    @DustinB3403 said in Doubt on a script- Backup site & DB:

    Why use a delete function, wouldn't rsync automatically remove any files that are not in the source from the target?

    From looking this up, other people have had similar issues, where using rsync -a --delete is effectively not rsyncing the files at all.



  • @Ambarishrh oh I see.



  • Thanks @DustinB3403 for the help and pointers.



  • @Ambarishrh You're welcome, sorry I didn't have the exact answer for you.



  • @DustinB3403 said in Doubt on a script- Backup site & DB:

    @Ambarishrh You're welcome, sorry I didn't have the exact answer for you.

    But that really helped me to find the right solution 🙂