Download and Delete from Source every 5 minutes (Ubuntu)
-
@Jstear said in Download and Delete from Source every 5 minutes (Ubuntu):
This sounds like a good use case for Rsync. Setup a cron job to run rsync every X amount of minutes.
rsync would work perfectly for this as well, cron is for scheduled tasks.
-
No problem using rsync with a ftp I assume?
-
There shouldn't be any issues.
-
Alright think I've got it
So mount my drives with fstab and then just a simple
rsync --remove-source-files -options /path/to/src/ /path/to/dest
-
I believe the function you want is --delete-after . But otherwise, yeah that should work.Nope ignore me remove-source-files would be better.
-
@Sparkum said in Download and Delete from Source every 5 minutes (Ubuntu):
@DustinB3403
Would there be a way to script it in that it checks, if it sees the file the same (source vs remote) that it deleted?Yes, that would not be too hard. MD5 hash stored in a text file to compare against would work. But you need an MD5 on the remote site to be able to compare against. Otherwise, it needs to download it to check the hash.
-
@Sparkum said in Download and Delete from Source every 5 minutes (Ubuntu):
No problem using rsync with a ftp I assume?
Rsync and FTP are different protocols. The term "using rsync with FTP" is meaningless. Imagine saying "using email with a phone call." What does that even mean?
Rsync and FTP are competing options. You can only use the technologies that the site you are downloading from provides. If they only offer FTP, you can only use FTP, it's that simple.
-
Let's start with the source, what is the actual source of this data. And why do you need it every five minutes? This is a weird task and I think we need to understand the real goals to really know what is and isn't a reasonable approach.
-
@Sparkum said in Download and Delete from Source every 5 minutes (Ubuntu):
Alright think I've got it
So mount my drives with fstab and then just a simple
rsync --remove-source-files -options /path/to/src/ /path/to/dest
Seems that this would be a good option.
-
I'm trying to mount the drive under /etc/fstab but I cant seem to get it to work.
I have my local windows drive mounted, just cant get the remote Ubuntu drive mounted.
I'm assuming my problem child will either be firewall or port, does anyone know what port mounting the drive takes on Ubuntu?
-
@Sparkum said in Download and Delete from Source every 5 minutes (Ubuntu):
I'm trying to mount the drive under /etc/fstab but I cant seem to get it to work.
I have my local windows drive mounted, just cant get the remote Ubuntu drive mounted.
I'm assuming my problem child will either be firewall or port, does anyone know what port mounting the drive takes on Ubuntu?
What is the mounting protocol in question?
-
Trying to mount using /etc/fstab
So I'm doing...
//XXX.XXX.XXX.XXX/var/www/completed /mnt/ftp cifs username=root,password=PASSWORD,iocharset=utf8,sec=ntlm,dir_mode=0777,file_mode=0777 0 0
but I'm getting error code 115
So I'm wondering if there's a certain port that needs to be opened up, or a typical thing that has to be done when connecting to remote computers (both ubuntu 14.04)
-
@Sparkum said in Download and Delete from Source every 5 minutes (Ubuntu):
Trying to mount using /etc/fstab
So I'm doing...
//XXX.XXX.XXX.XXX/var/www/completed /mnt/ftp cifs username=root,password=PASSWORD,iocharset=utf8,sec=ntlm,dir_mode=0777,file_mode=0777 0 0
but I'm getting error code 115
So I'm wondering if there's a certain port that needs to be opened up, or a typical thing that has to be done when connecting to remote computers (both ubuntu 14.04)
Can you run the mount commands by hand and have it work?
-
@Sparkum said in Download and Delete from Source every 5 minutes (Ubuntu):
Trying to mount using /etc/fstab
So I'm doing...
//XXX.XXX.XXX.XXX/var/www/completed /mnt/ftp cifs username=root,password=PASSWORD,iocharset=utf8,sec=ntlm,dir_mode=0777,file_mode=0777 0 0
but I'm getting error code 115
So I'm wondering if there's a certain port that needs to be opened up, or a typical thing that has to be done when connecting to remote computers (both ubuntu 14.04)
Why are you using CIFS between Linux machines? This seems like it is being treated in a very complicated way for something that should be really simple.
-
Why are you not just using Rsync?
-
Because thats what was suggested below, thats about it.
And what should I use to mount between linux machines?
-
NFS tends to work better between Linux boxes.
-
@Sparkum said in Download and Delete from Source every 5 minutes (Ubuntu):
Because thats what was suggested below, thats about it.
And what should I use to mount between linux machines?
Who suggested that? CIFS would be the worst option. Rsync the best. NFS in the middle.
Only NFS between LInux machines. But we need to back up. Why are you mounting anything? Mounting sounds like the wrong approach here. Hence why I keep saying Rsync instead of mounting.
-
@dafyre said in Download and Delete from Source every 5 minutes (Ubuntu):
NFS tends to work better between Linux boxes.
A LOT better...
- Native libraries.
- Faster
- More stable
- Must less to configure
- No weird permissions mismatches
-
@Sparkum said in Download and Delete from Source every 5 minutes (Ubuntu):
So mount my drives with fstab and then just a simple
rsync --remove-source-files -options /path/to/src/ /path/to/dest
This is where the error crept in. You added "mount my drives" as a step. Remove that. You ONLY Rsync, no mounting of drives at all.
