Nextcloud Backup System Idea
-
Since I'm planning on deploying NextCloud on my colo server, I want to also think through how I want to backup the data. In my environment there is a single 1U server.
Thinking through RPO / RTO this is what I think is reasonable for this data. I could handle NextCloud not being available for 24 hours as well losing 24 hours worth of data. The initial amount of data that will be stored in NextCloud will be around 1.5 TB. There will likely be far less than an a gigabyte / day of new data created.
Here is my initial idea of how to complete backups. Mind you this is a high-level picture. I haven't decided what tools would best complete this.
- One full backup followed by daily incremental backups to a VM on my cololab host to store the backups.
- Replicate the data on the Backup VM to something like Backblaze B2.
The end result will be three copies of the data (live Nextcloud VM, Backup VM, and cloud storage), on two different media (my colocation server and the cloud storage service), with one offsite (cloud storage service).
I think it's a good idea to do another full backup periodically (perhaps weekly or monthly). From what I understand to restore from an incremental backup, you restore the full and then all of the incrementals available from the last full backup. With taking more full backups, I have to start to consider how I can have data deduplication.
One thing I'm considering is not having the Backup VM, and just backing up directly to some kind of cloud storage. The potential disadvantage I see in this is with recovery, since it will be faster to copy data stored on my LAN.
-
Now this is off the top of my head.
You have restic installed on your Nextcloud VM and Backup VM.
Restic is configured to backup the following folders:- config folder
- data folder
- theme folder
You make a backup of your database. You can probably leave out the
date +"%Y%m%dsince restic timestamp your snapshots.mysqldump --single-transaction -h [server] -u [username] -p[password] [db_name] > nextcloud-sqlbkp_`date +"%Y%m%d"`.bakHave restic backup your database that you've just backed up.
- database
After restic completes it backups to the Backup VM, then setup restic on your Backup VM to backup that data to Backblaze.
-
After saying all that, now I'm interested in seeing that in action.
-
@black3dynamite said in Backup System Idea:
Now this is off the top of my head.
You have restic installed on your Nextcloud VM and Backup VM.
Restic is configured to backup the following folders:- config folder
- data folder
- theme folder
You make a backup of your database. You can probably leave out the
date +"%Y%m%dsince restic timestamp your snapshots.mysqldump --single-transaction -h [server] -u [username] -p[password] [db_name] > nextcloud-sqlbkp_`date +"%Y%m%d"`.bakHave restic backup your database that you've just backed up.
- database
After restic completes it backups to the Backup VM, then setup restic on your Backup VM to backup that data to Backblaze.
I'll take a look at that. For the database, you described what I'm thinking about doing. Backing up the database, and then backup the resultant file to the Backup VM and then to the cloud storage.
-
@EddieJennings said in Backup System Idea:
@black3dynamite said in Backup System Idea:
Now this is off the top of my head.
You have restic installed on your Nextcloud VM and Backup VM.
Restic is configured to backup the following folders:- config folder
- data folder
- theme folder
You make a backup of your database. You can probably leave out the
date +"%Y%m%dsince restic timestamp your snapshots.mysqldump --single-transaction -h [server] -u [username] -p[password] [db_name] > nextcloud-sqlbkp_`date +"%Y%m%d"`.bakHave restic backup your database that you've just backed up.
- database
After restic completes it backups to the Backup VM, then setup restic on your Backup VM to backup that data to Backblaze.
I'll take a look at that. For the database, you described what I'm thinking about doing. Backing up the database, and then backup the resultant file to the Backup VM and then to the cloud storage.
My concerns is how long you have to set Nextcloud in maintenance mode. Any new or modified data on the client will not be synced back to Nextcloud until your backup is finished. Now if you are backing up the VM itself, then it shouldn't be an issue.
-
@black3dynamite said in Backup System Idea:
@EddieJennings said in Backup System Idea:
@black3dynamite said in Backup System Idea:
Now this is off the top of my head.
You have restic installed on your Nextcloud VM and Backup VM.
Restic is configured to backup the following folders:- config folder
- data folder
- theme folder
You make a backup of your database. You can probably leave out the
date +"%Y%m%dsince restic timestamp your snapshots.mysqldump --single-transaction -h [server] -u [username] -p[password] [db_name] > nextcloud-sqlbkp_`date +"%Y%m%d"`.bakHave restic backup your database that you've just backed up.
- database
After restic completes it backups to the Backup VM, then setup restic on your Backup VM to backup that data to Backblaze.
I'll take a look at that. For the database, you described what I'm thinking about doing. Backing up the database, and then backup the resultant file to the Backup VM and then to the cloud storage.
My concerns is how long you have to set Nextcloud in maintenance mode. Any new or modified data on the client will not be synced back to Nextcloud until your backup is finished. Now if you are backing up the VM itself, then it shouldn't be an issue.
For this situation, I am the only user of Nextcloud, so I could script taking it and out of maintenance mode when I'd be sleeping. I can see the concern in an environment with multiple users that could be accessing Nextcloud at any time.
-
This person create a automatic backup using restic that sends the backup to Backblaze
It can email notifications on failure too.
https://github.com/erikw/restic-systemd-automatic-backup -
Here's a more straight forward auto backup to Google Drive using restic that can tweak for backing up Nextcloud VM to your Backup VM and then use the script from erikw or use the this same script to backup to Backblaze.
https://jakobkofod.com/automated-restic-backups-google-drive/ -
Restic does backups really well and efficiently. I use Restic on my VPS and back up to my personal OneDrive via Rclone.
https://timothygruber.com/backup-and-restore/back-up-linux-vps-to-onedrive-via-restic-and-rclone/
The initial backup will take a bit, but the incrementals... you won't even notice.
-
To ask and to be that guy;
Why does it sound as if NextCloud is going to be installed to the bare metal with the potential for a separate "Backup VM"?
As for the backup solution, you can use tools like Restic, Duplicati, UrBackup or really anything you want. Have that tool backup to a local storage media and then copy those to a cloud provider.
-
@DustinB3403 said in Backup System Idea:
To ask and to be that guy;
Why does it sound as if NextCloud is going to be installed to the bare metal with the potential for a separate "Backup VM"?
As for the backup solution, you can use tools like Restic, Duplicati, UrBackup or really anything you want. Have that tool backup to a local storage media and then copy those to a cloud provider.
NextCloud isn't going to be installed on bare metal. I could just attach another qcow2 disk to the VM, mount it somewhere, and use that as the backup target.
The idea of the Backup VM is emulating something like what I might find in a production environment using something like Veeam, where you have a dedicated server (VM) that runs the backup software and is a target for the backed up data. Of course, this is overkill for what I'm doing (just backing up my NextCloud instance).
-
@EddieJennings said in Backup System Idea:
@DustinB3403 said in Backup System Idea:
To ask and to be that guy;
Why does it sound as if NextCloud is going to be installed to the bare metal with the potential for a separate "Backup VM"?
As for the backup solution, you can use tools like Restic, Duplicati, UrBackup or really anything you want. Have that tool backup to a local storage media and then copy those to a cloud provider.
NextCloud isn't going to be installed on bare metal. I could just attach another qcow2 disk to the VM, mount it somewhere, and use that as the backup target.
The idea of the Backup VM is emulating something like what I might find in a production environment using something like Veeam, where you have a dedicated server (VM) that runs the backup software and is a target for the backed up data. Of course, this is overkill for what I'm doing (just backing up my NextCloud instance).
Why not send the backup to either Backblaze B2 or Wasabi? If you want practice doing backups, might as well do it right. My B2 is costing me ~$1.25/month
-
@travisdh1 said in Backup System Idea:
@EddieJennings said in Backup System Idea:
@DustinB3403 said in Backup System Idea:
To ask and to be that guy;
Why does it sound as if NextCloud is going to be installed to the bare metal with the potential for a separate "Backup VM"?
As for the backup solution, you can use tools like Restic, Duplicati, UrBackup or really anything you want. Have that tool backup to a local storage media and then copy those to a cloud provider.
NextCloud isn't going to be installed on bare metal. I could just attach another qcow2 disk to the VM, mount it somewhere, and use that as the backup target.
The idea of the Backup VM is emulating something like what I might find in a production environment using something like Veeam, where you have a dedicated server (VM) that runs the backup software and is a target for the backed up data. Of course, this is overkill for what I'm doing (just backing up my NextCloud instance).
Why not send the backup to either Backblaze B2 or Wasabi? If you want practice doing backups, might as well do it right. My B2 is costing me ~$1.25/month
So skip having a 3rd copy of the data (live, copy on aforementioned Backup VM, copy on Backblaze storage or some other provider)?
-
As important as backups are - I'm really surprised that NC has not integrated them into the software.
This thread is helpful as I was not aware of Restic. I normally make exports of the VM for NC backups. -
@EddieJennings said in Nextcloud Backup System Idea:
@travisdh1 said in Backup System Idea:
@EddieJennings said in Backup System Idea:
@DustinB3403 said in Backup System Idea:
To ask and to be that guy;
Why does it sound as if NextCloud is going to be installed to the bare metal with the potential for a separate "Backup VM"?
As for the backup solution, you can use tools like Restic, Duplicati, UrBackup or really anything you want. Have that tool backup to a local storage media and then copy those to a cloud provider.
NextCloud isn't going to be installed on bare metal. I could just attach another qcow2 disk to the VM, mount it somewhere, and use that as the backup target.
The idea of the Backup VM is emulating something like what I might find in a production environment using something like Veeam, where you have a dedicated server (VM) that runs the backup software and is a target for the backed up data. Of course, this is overkill for what I'm doing (just backing up my NextCloud instance).
Why not send the backup to either Backblaze B2 or Wasabi? If you want practice doing backups, might as well do it right. My B2 is costing me ~$1.25/month
So skip having a 3rd copy of the data (live, copy on aforementioned Backup VM, copy on Backblaze storage or some other provider)?
But you don't have a 3rd copy currently with 1 of the 3 being on the same server.
Apparently I missed that you already have an offsite happening, so good on you.
Being that it's your home lab, is it really worth the expense of getting a real 3rd copy? Might not matter (value of the data compared to time to restore.)
-
@travisdh1 said in Nextcloud Backup System Idea:
But you don't have a 3rd copy currently with 1 of the 3 being on the same server.
Yeah. Since this lab server is just a 1U in co-location, the best I could do for having an additional copy off of the Nextcloud VM is having another VM as a backup target. Of course, there's no real protection with that.
Apparently I missed that you already have an offsite happening, so good on you.
Nothing offsite yet. I'm trying to think all of this through before I create the Nextcloud VM and start storing data on it.
Being that it's your home lab, is it really worth the expense of getting a real 3rd copy? Might not matter (value of the data compared to time to restore.)
That's something I'm weighing as well. The only way I think I could get a real 3rd copy is have one on a second cloud storage provider, since having additional hardware in colo really isn't an option. I suppose I could store a copy at home as the 3rd copy, but that's more of make-a-way-to-have-a-3rd-copy-just-to-have-a-third-copy rather than as you suggested before "do it right."
-
@EddieJennings said in Nextcloud Backup System Idea:
@travisdh1 said in Nextcloud Backup System Idea:
But you don't have a 3rd copy currently with 1 of the 3 being on the same server.
Yeah. Since this lab server is just a 1U in co-location, the best I could do for having an additional copy off of the Nextcloud VM is having another VM as a backup target. Of course, there's no real protection with that.
So the question to ask is why add the complexity of backing up to a VM that would reside on the same host? It's more complex for no gain.
Nothing offsite yet. I'm trying to think all of this through before I create the Nextcloud VM and start storing data on it.
This is good to do.
That's something I'm weighing as well. The only way I think I could get a real 3rd copy is have one on a second cloud storage provider, since having additional hardware in colo really isn't an option. I suppose I could store a copy at home as the 3rd copy, but that's more of make-a-way-to-have-a-3rd-copy-just-to-have-a-third-copy rather than as you suggested before "do it right."
Or you could setup a NextCloud instance on something like Vultr and replicate between your 1U and the actual remote VM.
-
@DustinB3403 and @travisdh1 are both missing the fucking point that while this may be a single 1u server in colo the point is to be a full lab design.
He absolutely should have a “backup” VM as this is what exists in reality.
It is replicating the real world on site copy.
-
@brandon220 said in Nextcloud Backup System Idea:
As important as backups are - I'm really surprised that NC has not integrated them into the software.
This thread is helpful as I was not aware of Restic. I normally make exports of the VM for NC backups.I am surprised that they have not partnered with some provider.
I wonder is someone could write a backup app that would backup the DB and up it to Backblaze
-
@JaredBusch said in Nextcloud Backup System Idea:
@DustinB3403 and @travisdh1 are both missing the fucking point that while this may be a single 1u server in colo the point is to be a full lab design.
He absolutely should have a “backup” VM as this is what exists in reality.
It is replicating the real world on site copy.
This was my assumption as well - he's trying to mimic real life as best he can.
