Doubt on a script- Backup site & DB

Ambarishrh

Just wanted to check a script that I am working on:

Goal is to take complete site backup with db and then rsync to remote server, email the download link

#Cleanup old backup files
rm -f /home/ACCOUNT_NAME/BACKUP_FOLDER/backup-*.tgz

#Insert date and a random key (key added to use it on public url to avoid guessing of public download link)
date=$(date +%Y-%m-%d)
key=$(head /dev/urandom | md5sum | cut -c1-10)

#MySQLdump of the account to the backup folder
mysqldump -uDB_USER -pDB_PASSWORD --single-transaction DB_NAME 2>&1 | gzip > /home/ACCOUNT_NAME/BACKUP_FOLDER/backup-${date}-${key}_SITE_NAME.sql.gz

#Compress the web directory and sql backup to a single backup file
tar czf /home/ACCOUNT_NAME/BACKUP_FOLDER/backup-${date}-${key}_SITE_NAME.tgz /home/ACCOUNT_NAME/public_html /home/ACCOUNT_NAME/BACKUP_FOLDER/backup-${date}-${key}_SITE_NAME.sql.gz 2> /dev/null

#Rsync the final backup file to remote server, also cleanup existing old backup prior to the transfer by comparing source directory
rsync -a --delete /home/ACCOUNT_NAME/BACKUP_FOLDER/backup-${date}-${key}_SITE_NAME.tgz REMOTE_USER@REMOTE_SERVER_URL:/home/ACCOUNT_NAME/public_html/SITE_NAME_BACKUP/

#Email the backup link
echo "Latest backup is available at http://REMOTE_SERVER_URL/SITE_NAME_BACKUP/backup-${date}-${key}_SITE_NAME.tgz" | mail -s "Server backup available for download" [email protected]

The rsync command for some reason is not cleaning up old files!

Is this correct or are there any options to tweak this?

DustinB3403

Why use a delete function, wouldn't rsync automatically remove any files that are not in the source from the target?

From looking this up, other people have had similar issues, where using rsync -a --delete is effectively not rsyncing the files at all.

DustinB3403

Here is a topic from the ubuntu forums.

Are you trying to delete files from the source or the destination?

If from the destination

"rsync -avh source/ dest/ --delete"

Might be how you need to enter the command.

Ambarishrh

@DustinB3403 said in Doubt on a script- Backup site & DB:

Here is a topic from the ubuntu forums.

Are you trying to delete files from the source or the destination?

If from the destination

"rsync -avh source/ dest/ --delete"

Might be how you need to enter the command.

The first line on the script cleans up old backup files, so during rsync i want to compare the source and destination and cleanup accordingly. I will try with the --delete at the end and see if that helps

Ambarishrh

@Ambarishrh said in Doubt on a script- Backup site & DB:

@DustinB3403 said in Doubt on a script- Backup site & DB:

Here is a topic from the ubuntu forums.

Are you trying to delete files from the source or the destination?

If from the destination

"rsync -avh source/ dest/ --delete"

Might be how you need to enter the command.

The first line on the script cleans up old backup files, so during rsync i want to compare the source and destination and cleanup accordingly. I will try with the --delete at the end and see if that helps

That didn't work!

DustinB3403

Can you post the logs?

DustinB3403

Rsync --delete doesn't delete files in target directory.

Obviously just an example

rsync -avz --ignore-existing --recursive --delete /var/www/ [email protected]:/var/www

Ambarishrh

@DustinB3403 said in Doubt on a script- Backup site & DB:

rsync -avz --ignore-existing --recursive --delete

Tried that as well!
sending incremental file list
backup-2016-10-10-16694c6b59_dur.tgz
582776973 100% 75.96MB/s 0:00:07 (xfer#1, to-check=0/1)

sent 582972731 bytes received 31 bytes 77729701.60 bytes/sec
total size is 582776973 speedup is 1.00

Copies the new file but doesnt do anything with old

Ambarishrh

I do have a * on the file but without that i can't get the new file created

DustinB3403

But with the wildcard, you can't delete files. (Just read that in the manual)

scottalanmiller

@DustinB3403 said in Doubt on a script- Backup site & DB:

Why use a delete function, wouldn't rsync automatically remove any files that are not in the source from the target?

Yes, it will do that.

DustinB3403

--delete
			  This tells rsync to delete extraneous files from the receiving side (ones that arenât on the sending side), but only for the directories that are being synchronized.  You must have asked rsync to send the whole direcâ
			  tory  (e.g.  "dir" or "dir/") without using a wildcard for the directoryâs contents (e.g. "dir/*") since the wildcard is expanded by the shell and rsync thus gets a request to transfer individual files, not the filesâ
			  parent directory.  Files that are excluded from the transfer are also excluded from being deleted unless you use the --delete-excluded option or  mark  the  rules  as  only  matching  on  the  sending  side  (see  the
			  include/exclude modifiers in the FILTER RULES section).

DustinB3403

I wonder if --delete-excluded is the required approach with this, as it accounts for wildcards..

According the manual --delete-excluded is for use cases where you're using the wildcard flag. So it would remove the folder and contents, rather than just the contents of a particular folder.

Ambarishrh

@scottalanmiller said in Doubt on a script- Backup site & DB:

@DustinB3403 said in Doubt on a script- Backup site & DB:

Why use a delete function, wouldn't rsync automatically remove any files that are not in the source from the target?

Yes, it will do that.

If am not mistaken, plain rsync just syncs the difference to destination but does not deletes anything

Ambarishrh

@DustinB3403 said in Doubt on a script- Backup site & DB:

I wonder if --delete-excluded is the required approach with this, as it accounts for wildcards..

According the manual --delete-excluded is for use cases where you're using the wildcard flag. So it would remove the folder and contents, rather than just the contents of a particular folder.

Fixed it! So the issue on my case was that i was transferring a file and adding --delete switch, whereas the --delete only works with a folder.

--delete

This tells rsync to delete extraneous files from the receiving side (ones that aren’t on the sending side), but only for the directories that are being synchronized. You must have asked rsync to send the whole directory (e.g. "dir" or "dir/") without using a wildcard for the directory’s contents (e.g. "dir/*") since the wildcard is expanded by the shell and rsync thus gets a request to transfer individual files, not the files’ parent directory. Files that are excluded from the transfer are also excluded from being deleted unless you use the --delete-excluded option or mark the rules as only matching on the sending side (see the include/exclude modifiers in the FILTER RULES section).

So i updated my script and added the backup files to a dir and synced that, all works fine as expected.

Ambarishrh

Final script:

#Cleanup old backup files
rm -f /home/ACCOUNT_NAME/BACKUP_FOLDER/*

#Insert date and a random key (key added to use it on public url to avoid guessing of public download link)
date=$(date +%Y-%m-%d)
key=$(head /dev/urandom | md5sum | cut -c1-10)

#MySQLdump of the account to the backup folder
mysqldump -uDB_USER -pDB_PASSWORD --single-transaction DB_NAME 2>&1 | gzip > /home/ACCOUNT_NAME/BACKUP_FOLDER/backup-${date}-${key}_SITE_NAME.sql.gz

#Compress the web directory and sql backup to a single backup file
tar czf /home/ACCOUNT_NAME/BACKUP_FOLDER/backup-${date}-${key}_SITE_NAME.tgz /home/ACCOUNT_NAME/public_html /home/ACCOUNT_NAME/BACKUP_FOLDER/backup-${date}-${key}_SITE_NAME.sql.gz 2> /dev/null

#Rsync the final backup file to remote server, also cleanup existing old backup prior to the transfer by comparing source directory
rsync -a --delete /home/ACCOUNT_NAME/BACKUP_FOLDER/ REMOTE_USER@REMOTE_SERVER_URL:/home/ACCOUNT_NAME/public_html/SITE_NAME_BACKUP/

#Email the backup link
echo "Latest backup is available at http://REMOTE_SERVER_URL/SITE_NAME_BACKUP/backup-${date}-${key}_SITE_NAME.tgz" | mail -s "Server backup available for download" [email protected]

scottalanmiller

@Ambarishrh said in Doubt on a script- Backup site & DB:

@scottalanmiller said in Doubt on a script- Backup site & DB:

@DustinB3403 said in Doubt on a script- Backup site & DB:

Why use a delete function, wouldn't rsync automatically remove any files that are not in the source from the target?

Yes, it will do that.

If am not mistaken, plain rsync just syncs the difference to destination but does not deletes anything

But you always select your flags, you never just use vanilla.

Ambarishrh

@scottalanmiller said in Doubt on a script- Backup site & DB:

@Ambarishrh said in Doubt on a script- Backup site & DB:

@scottalanmiller said in Doubt on a script- Backup site & DB:

@DustinB3403 said in Doubt on a script- Backup site & DB:

Why use a delete function, wouldn't rsync automatically remove any files that are not in the source from the target?

Yes, it will do that.

If am not mistaken, plain rsync just syncs the difference to destination but does not deletes anything

But you always select your flags, you never just use vanilla.

But what was mentioned earlier

@DustinB3403 said in Doubt on a script- Backup site & DB:

Why use a delete function, wouldn't rsync automatically remove any files that are not in the source from the target?

From looking this up, other people have had similar issues, where using rsync -a --delete is effectively not rsyncing the files at all.

scottalanmiller

@Ambarishrh oh I see.

Ambarishrh

Thanks @DustinB3403 for the help and pointers.