Turn server into backup storage for remote servers?

EddieJennings

There are many things you can do, but we'd need more detail about what data is to be backed up. Also, we'd want to know where these remote servers are compared to this to-be backup server.

Conceptually one of the easiest things you can do is have this server pull backups from your various Linux servers using something like rsync or just ssh / scp.

1337

@EddieJennings said in Turn server into backup storage for remote servers?:

There are many things you can do, but we'd need more detail about what data is to be backed up. Also, we'd want to know where these remote servers are compared to this to-be backup server.

Thanks Eddie.

Well, the servers are basically web servers (VMs) with not much data, say 5-10GB, in each for a full backup. Backups will run a couple of times per day but they can be incremental.

The purpose of this is to backup the customer's data and not the server instance.

Remote servers are a mixture of on-prem VMs, hosted webservers, cloud instances and such.

1337

@EddieJennings said in Turn server into backup storage for remote servers?:

Conceptually one of the easiest things you can do is have this server pull backups from your various Linux servers using something like rsync or just ssh / scp.

Maybe that's the easiest thing.

I was thinking perhaps pushing the backup from each server would be better, instead of backups being pulled by the backup server.

Perhaps taking everything that needs to be backed up, compress it and send it.

Does ssh support having many concurrent sessions going at the same time, say a hundred?

dafyre

@Pete-S SSH will pretty much only be limited by the bandwidth available for you to suck in the backups.

I actually just finished a script for doing almost exactly what you said just a few days ago. I sanitized it and dropped in in Gitlab if you want to take a look.

https://gitlab.com/dafyre/linux-utils/-/blob/master/autobackup/autobackup.sh

I think I made everything a variable, but since it's sanitized, it's quite possible I missed something.

This one is for pushing backups from the server with data out to the backup server. Once you get it working, just set it up in Cron and go.

scottalanmiller

@Pete-S said in Turn server into backup storage for remote servers?:

I have a server with lots of storage that sits in a datacenter doing nothing.

What is the easiest/best way to turn this server into a backup storage that I can use to backup remote linux servers?

I generally just install Ubuntu latest and keep it pretty vanilla.

scottalanmiller

@Pete-S said in Turn server into backup storage for remote servers?:

@EddieJennings said in Turn server into backup storage for remote servers?:

There are many things you can do, but we'd need more detail about what data is to be backed up. Also, we'd want to know where these remote servers are compared to this to-be backup server.

Thanks Eddie.

Well, the servers are basically web servers (VMs) with not much data, say 5-10GB, in each for a full backup. Backups will run a couple of times per day but they can be incremental.

The purpose of this is to backup the customer's data and not the server instance.

Remote servers are a mixture of on-prem VMs, hosted webservers, cloud instances and such.

We typically script that then backup over SSH or NFS. But you can use backup software, too. But it's hard to beat straight scripts for performance and reliability.

scottalanmiller

@Pete-S said in Turn server into backup storage for remote servers?:

I was thinking perhaps pushing the backup from each server would be better, instead of backups being pulled by the backup server.

That's what we do.

scottalanmiller

@Pete-S said in Turn server into backup storage for remote servers?:

Perhaps taking everything that needs to be backed up, compress it and send it.

Yes, that's also what we do. We do a 7zip typically before sending. All the renaming, dates, compression done when the source computer can do it. Then the smallest possible send over to the storage system. Then the storage server handles the cloud uploads for the offsite backup.

1337

@dafyre said in Turn server into backup storage for remote servers?:

@Pete-S SSH will pretty much only be limited by the bandwidth available for you to suck in the backups.

I actually just finished a script for doing almost exactly what you said just a few days ago. I sanitized it and dropped in in Gitlab if you want to take a look.

https://gitlab.com/dafyre/linux-utils/-/blob/master/autobackup/autobackup.sh

I think I made everything a variable, but since it's sanitized, it's quite possible I missed something.

This one is for pushing backups from the server with data out to the backup server. Once you get it working, just set it up in Cron and go.

Great, thanks! Looks good! I'll use that as a base going forward.

Dashrender

@dafyre said in Turn server into backup storage for remote servers?:

@Pete-S SSH will pretty much only be limited by the bandwidth available for you to suck in the backups.

I actually just finished a script for doing almost exactly what you said just a few days ago. I sanitized it and dropped in in Gitlab if you want to take a look.

https://gitlab.com/dafyre/linux-utils/-/blob/master/autobackup/autobackup.sh

I think I made everything a variable, but since it's sanitized, it's quite possible I missed something.

This one is for pushing backups from the server with data out to the backup server. Once you get it working, just set it up in Cron and go.

Why push instead of pull? Wouldn't pulling be safer - i.e. the webserver has no information about the backup server, so if it's compromised, it can't give anything to the attackers about the backups.

dafyre

@Dashrender said in Turn server into backup storage for remote servers?:

@dafyre said in Turn server into backup storage for remote servers?:

@Pete-S SSH will pretty much only be limited by the bandwidth available for you to suck in the backups.

I actually just finished a script for doing almost exactly what you said just a few days ago. I sanitized it and dropped in in Gitlab if you want to take a look.

https://gitlab.com/dafyre/linux-utils/-/blob/master/autobackup/autobackup.sh

I think I made everything a variable, but since it's sanitized, it's quite possible I missed something.

This one is for pushing backups from the server with data out to the backup server. Once you get it working, just set it up in Cron and go.

Why push instead of pull? Wouldn't pulling be safer - i.e. the webserver has no information about the backup server, so if it's compromised, it can't give anything to the attackers about the backups.

Your point is a good one. The reverse is also true. If somebody gets access to either system, though, you have bigger issues, lol.

We try to keep all of our systems the same, and this is the backup routine has been working for us. Since we haven't had any immediate need to change it, we haven't.

JaredBusch

@Dashrender said in Turn server into backup storage for remote servers?:

@dafyre said in Turn server into backup storage for remote servers?:

@Pete-S SSH will pretty much only be limited by the bandwidth available for you to suck in the backups.

I actually just finished a script for doing almost exactly what you said just a few days ago. I sanitized it and dropped in in Gitlab if you want to take a look.

https://gitlab.com/dafyre/linux-utils/-/blob/master/autobackup/autobackup.sh

I think I made everything a variable, but since it's sanitized, it's quite possible I missed something.

This one is for pushing backups from the server with data out to the backup server. Once you get it working, just set it up in Cron and go.

Why push instead of pull? Wouldn't pulling be safer - i.e. the webserver has no information about the backup server, so if it's compromised, it can't give anything to the attackers about the backups.

Pull is nothing but stupid complexity. It has no control over the remote system to know when backups are available and you have to pin hole each system to allow the incoming connection.

A push happens as soon as the valid backup is created, so you know that your backup is gone to location 2 almost immediately. Location 2 then sends to location 3 via a cloud sync tool. Preferably location 3 is immutable storage.

Mario Jakovina

@scottalanmiller said in Turn server into backup storage for remote servers?:

We do a 7zip typically before sending.

Does this mean you always do a full backup.
Is there a way to compress incremental backups?
If not, would it be faster to just rsync differences?

scottalanmiller

@Mario-Jakovina said in Turn server into backup storage for remote servers?:

@scottalanmiller said in Turn server into backup storage for remote servers?:

We do a 7zip typically before sending.

Does this mean you always do a full backup.
Is there a way to compress incremental backups?
If not, would it be faster to just rsync differences?

We typically are backing up databases, so full normally. But you can compress anything if you are the one scripting it.

LZMA2 (often referred to as 7z, but doesn't include the container format) is actually part of Rsync. So Rsync with 7zip is often a choice. Not technically 7zip, but 7z compression.

1337

@scottalanmiller said in Turn server into backup storage for remote servers?:

We typically are backing up databases, so full normally.

Just a side note but one way to make partial backups from databases is to have a "modified date/time" column for large tables.

Then you can do partial backups by looking at the "modified" timestamp and including all rows that has been modified since the last backup.

select * from tableX where modified_datetime>last_backup;

This can be done fully application independent if the database have triggers. The trigger will automatically update the "modified date/time" when changes are made and the application has no clue.

This is an example how to create a trigger that saves when a row is updated:

create trigger update_modified
before update on table tableX
for each row
   set new.modified_datetime=now();

This obviously only makes sense when backups are getting big enough to warrant the extra work.

Yonah S.

@Pete-S have you thought of selling it? there is a big market right now for getting rid of old/unused hardware. Especially if you have any SSD's in there.

1337

@Yonah-S said in Turn server into backup storage for remote servers?:

@Pete-S have you thought of selling it? there is a big market right now for getting rid of old/unused hardware. Especially if you have any SSD's in there.

Thanks, but we're keeping it. Just want to extract the maximum value out of it while it's occupying rack space